Problem Retrieving 1 time series via Python API

snaiste · February 12, 2020, 9:54am

Hello,

I’ve been using your Python API successfully for some ttime, but I’m unable to retrieve this specific dataset: https://api.db.nomics.world/v22/series?observations=1&series_ids=BEA/NIPA-T11706/A191RX-Q.

When debugging, I found that series with data are initially retrieved fine, but then in your “fetch_series_by_api_link” method (lines 233-236) it breaks in a loop where it tries to retrieve “Industry” dimension, which, I think, maybe was not populated properly for these time-series.

Could you advise me how I could retrieve these time series with Python?

bruno.duye · February 12, 2020, 2:33pm

Hello,

Thanks for your feedback. If I understand well, the problem is:

df = fetch_series('BEA/NIPA-T11706/A191RX-Q')

KeyError  Traceback (most recent call last)
<ipython-input-9-da4c1b56e5b7> in <module>
----> 1 df = fetch_series('BEA/NIPA-T11706/A191RX-Q')

~/dev/jailbreak/dbnomics/dbnomics-python-client/dbnomics/__init__.py in fetch_series(provider_code, dataset_code, series_code, dimensions, series_ids, max_nb_series, api_base_url, editor_api_base_url, filters)
    167             raise ValueError("When you use `series_ids`, you must not specifiy `provider_code` nor `dataset_code`.")
    168         api_link = series_base_url + '?observations=1&series_ids={}'.format(','.join(series_ids))
--> 169         return fetch_series_by_api_link(api_link, filters=filters, max_nb_series=max_nb_series,
    170                                         editor_api_base_url=editor_api_base_url)
    171 

~/dev/jailbreak/dbnomics/dbnomics-python-client/dbnomics/__init__.py in fetch_series_by_api_link(api_link, max_nb_series, editor_api_base_url, filters)
    233         if 'dimensions_values_labels' in dataset_dimensions:
    234             for dimension_code, dimension_label in dataset_dimensions_labels.items():
--> 235                 dimension_value_code = series_dims_by_dataset_code[complete_dataset_code][series['series_code']][dimension_code]
    236                 flat_series[dimension_label] = dict(dataset_dimensions['dimensions_values_labels']
    237                                                     [dimension_code])[dimension_value_code]

KeyError: 'industry'

I created an issue in our Gitlab to treat this. I can’t tell when we’ll plan to treat this. Probably not this week unfortunately.

But, as DBnomics is open software, if you identify the bug (and apparently you did well ) and find a way to solve this one, we’re very open to pull requests ! You can communicate with us here to ask questions if you have some.

Have a good day

snaiste · February 13, 2020, 8:48am

Hi @bruno.duye ,

Thank you very much for your quick reply.

Yes, I believe this is the problem, but I’m not sure yet how it could be resolved (DBNomics is a new thing to me, I need to spend some time understanding how it works). But I will think about it and will let you know if I get any ideas.

In the meantime, I think I found a lot more examples that fail in the same for loop that we identified above. Not sure if it helps to resolve the issue, but I’ll summarise my findings below just in case.

I cannot retrieve any except Total index (i.e. FED/G17_IP_MARKET_GROUPS/IP.B50001.S) time series from this data group: https://db.nomics.world/FED/G17_IP_MARKET_GROUPS?dimensions={"FREQ"%3A["129"]%2C"SA"%3A["SA"]}
I cannot retrieve some time series in this data group: https://db.nomics.world/FED/G17_CAPUTL?dimensions={"FREQ"%3A["129"]%2C"SA"%3A["SA"]}. Examples:
– FED/G17_CAPUTL/CAPUTL.G211.S
– FED/G17_CAPUTL/CAPUTL.G212.S
– FED/G17_CAPUTL/CAPUTL.G2122.S
– FED/G17_CAPUTL/CAPUTL.G2123.S
– FED/G17_CAPUTL/CAPUTL.G213.S
– FED/G17_CAPUTL/CAPUTL.G311.S
– FED/G17_CAPUTL/CAPUTL.G312.S
– FED/G17_CAPUTL/CAPUTL.G313.S
– FED/G17_CAPUTL/CAPUTL.G314.S
– FED/G17_CAPUTL/CAPUTL.G315.S
– FED/G17_CAPUTL/CAPUTL.G316.S
– FED/G17_CAPUTL/CAPUTL.G325212.S
– FED/G17_CAPUTL/CAPUTL.G32522.S
– FED/G17_CAPUTL/CAPUTL.G3311A2.S
– FED/G17_CAPUTL/CAPUTL.G3341.S
– FED/G17_CAPUTL/CAPUTL.G3342.S
– FED/G17_CAPUTL/CAPUTL.G3344.S
– FED/G17_CAPUTL/CAPUTL.G336.S
– FED/G17_CAPUTL/CAPUTL.G33611.S
– FED/G17_CAPUTL/CAPUTL.HITEK2.S
– FED/G17_CAPUTL/CAPUTL.HITEK2.S
– FED/G17_CAPUTL/CAPUTL.N325211.S
– FED/G17_CAPUTL/CAPUTL.X4HTK2.S
– FED/G17_CAPUTL/CAPUTL.X4HTMV.S
– FED/G17_CAPUTL/CAPUTL.X50HTK.S
I cannot retrieve many (maybe all, haven’t tested all) from this data group: Consumer Credit Outstanding [G19_CCOUT], examples:

FED/G19_CCOUT/DTCTL.M
FED/G19_CCOUT/DTCTLN.M
FED/G19_CCOUT/DTCTLR.M

I think all of these are related to the same for loop issue.

Thanks a lot for looking into this and your work developing this amazing database! Once I get something useful out of my economics research, I’ll make sure to share the credit

Regards,
Aiste

thomasbrand · February 14, 2020, 3:46pm

Hi Aiste,

I can explain your FED problem, which is different from the BEA problem. We had to modify a bit the series code of FED series, sorry (the Fed gave sometimes more than ones series even when we specified a value for each dimension).

Now you can download all the series if you use the mask, as explained here (https://git.nomics.world/dbnomics/dbnomics-python-client/blob/master/index.ipynb, Fetch time series by code mask). In your example 1, you should write : fetch_series("FED", "G17_IP_MARKET_GROUPS", series_code="129..SA") and you will obtain the 63 different series, as in the DBnomics website.

Equivalently, you could use the Fetch time series by dimension :
fetch_series("FED", "G17_IP_MARKET_GROUPS", dimensions={ "FREQ": ["129"], "SA": ["SA"], })
and you will obtain the 63 series.

Thank you for your interest in DBnomics!
Thomas

acottrell · February 25, 2020, 10:11pm

Not sure if this is the same issue, but some things are kind of weird in the FED/G17_CAPUTL area. For example the series CAPUTL.B50001.S. If you download this (JSON) with

complete_missing_periods=1

you get 53 valid observations (out of 637 periods). The data are supposed to be monthly, but these values seem to pertain to the 28th of February in each year, with other months’ data all missing. But if you use

complete_missing_periods=0

you get 637 (apparently) non-missing observations. This is not easy to understand!

snaiste · February 26, 2020, 11:26am

Hi @thomasbrand,

Thank you very much! I will be looking into this again this week. I’ll try your workaround and will you know if it works.

Many thanks,
Aiste

snaiste · February 2, 2021, 2:28pm

Hi @thomasbrand,

I hope you’re well!

I’ve been looking at this issue, but I still can’t figure it out…

So I extracted 3 examples that either work or don’t when retrieving data using Python API:

Working:

Germany GDP: DESTATIS/81000BV007/DG.VGRPVK.X13JDKSB.VGR014

Not working:

Spain GDP: INE-SPAIN/30679/72.b1gq.l.y.00.index
US GDP: BEA/NIPA-T11706/A191RX-Q

I find that the issue with the not working series is in this method: fetch_series_by_api_link(api_link, max_nb_series=None, editor_api_base_url=default_editor_api_base_url, filters=None), lines 233-237, i.e. that’s where the code breaks for not working series.

If I look at ‘series_infos’ when trying to retrieve each of the series, I can see that in the case of Germany, ‘dimensions’ and ‘dimension_labels’ objects match, i.e.: there is a key equivalent in both the objects.:

‘dimensions’: {
‘DINSG’: ‘DG’,
‘VGRPB5’: ‘VGRPVK’,
‘WERT05’: ‘X13JDKSB’,
‘freq’: ‘Q’,
‘ind’: ‘VGR014’,
‘unit’: ‘jew_me’
}

‘dimensions_labels’: {
‘DINSG’: ‘Germany’,
‘VGRPB5’: ‘Price base (current prices / price-adjusted)’,
‘WERT05’: ‘Original and adjusted data’,
‘freq’: ‘Frequency’,
‘ind’: ‘Indicator’,
‘unit’: ‘Unit’
}

But when I look at Spain time series, I can see there are 2 additional key-value pairs in it’s dimension_labels object.:

‘dimensions’: {
‘3’: ‘72’,
‘482’: ‘b1gq’,
‘501’: ‘l’,
‘544’: ‘y’,
‘nac’: ‘00’,
‘units’: ‘index’
}
‘dimensions_labels’: {
‘3’: ‘Type of data’,
’481’: ‘DISTRIBUTION TRANSACTIONS’,
‘482’: ‘ACCOUNTING BALANCES’,
’495’: ‘GVA ACCOUNTING BALANCE’,
‘501’: ‘Prices_ESA’,
‘544’: ‘Correction of effects’,
‘nac’: ‘National Total’,
‘units’: ‘Units’
}

Similar thing happens with the US GDP:

‘dimensions’: {
‘FREQ’: ‘Q’,
‘concept’: ‘gross-domestic-product-gdp’,
‘metric’: ‘millions-of-chained-dollars’,
‘unit’: ‘level’
}

‘dimensions_labels’: {
‘FREQ’: ‘Frequency’,
‘concept’: ‘Concept’,
’industry’: ‘Industry’,
‘metric’: ‘Metric’,
’sub-industry’: ‘Industry precision’,
‘unit’: ‘Unit’
}

So because of these additional key-value pairs, the code breaks when it iterates over ‘dimensions_labels’ object and tries to find all key-values in the ‘dimensions’ object.

Unfortunatelly, I don’t know why in the case of Spain or the US these ‘dimensions’ and ‘dimensions_labels’ objects are different in terms of lenght.

There are many more time series like this, I was wondering if you guys are aware of this issue and or know the workaround?

Thank you!

Have a great day,
Aiste

snaiste · March 22, 2021, 3:13pm

Hi @enzo,

I hope you’re well!

I was wondering if there is any chance you could update me on this issue? I was wondering if maybe it’s so blocked that it’s perhaps impossible to resolve? In which case I would try to get that data somewhere else… which would be a shame because I love using DBNomics!

Thank you

Aiste

enzo · March 23, 2021, 11:27am

Hello @snaiste,

I have notified our developer of your issue and he will be working on it this week. We will keep you informed of the progress

Thank you for contacting us!

Enzo

snaiste · March 23, 2021, 4:08pm

Hi @enzo,

This is great, thank you very much!

Best,
Aiste

bruno.duye · April 1, 2021, 5:20pm

Hi @snaiste

I hope you’re well!

I’ve a good news for you (and this is not an April fool!): the problem you mention in the top of this thread has been fixed
The new version 1.2.1 should fix this problem.

I’ve to congratulate you: you isolated the problem on this comment !

Let us informed

Thanks again, and sorry for the delay !

snaiste · April 7, 2021, 12:44pm

Hi @bruno.duye,

I’m well, thank you, I hope you’re enjoying the Spring time too!

Wonderful news, thank you very much! I’ve just installed the new version and now it works very well for all examples I’ve given in this thread! Thank you very much, this is super helpful!!!

Have a great day,
Aiste

bruno.duye · April 7, 2021, 8:10pm

Great, I’m happy to read this ! Spring is strange this week, it snowed today in France

Have a good day !

snaiste · April 15, 2021, 9:16am

@bruno.duye,

O wow! London is colder than usual, but no snow since winter !

Thanks guys for your work once again, very helpful!

Topic		Replies	Views
Various Comments on Website/Python API Site Feedback	1	686	May 6, 2019
Finished call to 'dbnomics._fetch_response' Documentation fetchers	6	172	February 7, 2024
KeyError: 'ADJUSTMENT'	0	73	February 29, 2024
Using db.nomics in Grafana, Pandas _metadata, formats and stability Community	0	98	March 5, 2024
BUBA unavailable Site Feedback	3	317	May 30, 2022

Problem Retrieving 1 time series via Python API

Related topics