About the Stata plugin

Hi all,

Love this initiative! I have some experience with Stata so I would like to offer my help in developing the API plugin.

As it looks the two straightforward entry points would be the JSON API (I read somewhere that it should be already active or soon to be), on the basis of the Stata libjson library and the insheetjson command by Erik Lindsley. Alternatively, it may be interesting to integrate the API with the existing command sdmxuse from Sébastien Fontenay, which already allows to import data from WB, IMF, OECD, Eurostat (and via tweaking other services sharing the same API specs, e.g. the Belgian National Bank).

The first solution may be more elegant as sdmxuse uses a lot of shell calls which tend to make import process rather “flashy”. Another important aspect to consider is whether to allow insecure requests: only the two latest versions of Stata (14 & 15) support the https protocol, but I reckon earlier versions are still widely used.

Keep up the good work, can’t wait to see all the new sources you are planning to include up and working.

Thanks,
Simone

1 Like

Hello Simone,

Love this initiative! I have some experience with Stata so I would like to offer my help in developing the API plugin.

Thank you very much for your interest in DB.nomics. Your message pleased our team!

I’m the author of the Python client for DB.nomics which you can find here: dbnomics / dbnomics-python-client · GitLab
As stated in the README, you can open it in a dynamically generated environment by opening this link.

I don’t know Stata at all and can’t give you any hint about which library to use.

You can look at the source code of dbnomics-python-client, this Python file in particular, and reproduce the main ideas.

It calls the Web API, especially the /dataframe endpoint, which is documented here, and its source code is here. It transforms the response JSON into a Pandas DataFrame. I think you can do an equivalent transformation, but targeting a Stata tabular data structure.

Another important aspect to consider is whether to allow insecure requests: only the two latest versions of Stata (14 & 15) support the https protocol, but I reckon earlier versions are still widely used.

We did not state yet about allowing http:// requests, but please let us know when you have the actual problem.

Best regards,

Christophe

Great! Thanks a lot for the tips, I’ll get back here as soon as I can put together a workable version based on your python client.

Simone

Hi!
Just to keep you informed that we push a new version of the python client for DBnomics, with more features : https://git.nomics.world/dbnomics/dbnomics-python-client/tree/master. I hope it can help. We are currently adapting the R client.

Hi Thomas,

Thanks a lot, I had a look at the new apidocs. I have been on the Stata plugin on and off in the past weeks, I hope to push an initial release for testing soon. I’m not looking for a full port from python or R just yet but something that would look more native to the way Stata works (I might change my opinion on this in the future).

I noticed there is a {series_code}.csv endpoint now, that will make it much easier to integrate in Stata since its handling of JSON objects is rather basic without external dependencies (so far I am working only with the libjson library).

More in general, do you have a roadmap for future updates? I’d be happy to know and plan ahead the addition of new features.

Best,
Simone

Hi Simone,
It will be possible very soon (in February) to download several series in csv format (for one given frequency), so maybe it will easier for you to develop your Stata plugin. Moreover, we plan to put in production the new version of DBnomics (https://next.nomics.world/) in March.
If you want to know more about our discussions and roadmap (and participate!), you can have a look here : https://git.nomics.world/dbnomics-fetchers/management/boards.

Hi Thomas,

Many thanks for the update. I’m almost done on my side, I built routines for almost all endpoints of the API. It would be nice at some point to allow to extract series using masks (e.g. FRA+BEL.3+99…0.0.PIGOT), which is something that can be done now but requires the user to know dimension names in advance and is thus a bit more challenging to automate.

By the way, querying series at the moment seems only to work with the /{provider_code}/{dataset_code}, despite being allowed also for /series/{provider_code}/{dataset_code} (this option does not seem to work at the moment as it always returns all series allowed by the limit option).

I’ll be back soon with a first prototype version.
Best,

Simone

1 Like

Hi Simone,

We will implement masks as your example very soon. However, such masks are similar to SDMX requests and could not be extended to all the providers in DBnomics. Moreover, we will provide on the website a direct access to a dictionary of the codes and labels of dimensions for each dataset, to use more easily the API.

Best,
Thomas

Hi Simone,

These 2 endpoints of the API could be merged at some point. The big picture is that the first responds a list of series (codes and names only) and how much series correspond to each dimension (called facets counts in Apache Solr indexer), and the second responds series with their observations.

The clients call the second one to build the dataframes. I didn’t understand your problem with the limit param. Perhaps you missed the offset param, which you can use to fetch the next “pages”.

Have a look at this code section in the Python client: it loops until all the pages are downloaded, and increments the offset param accordingly.

Hope it helps.

Regards

Hi Christophe, Thomas,

It took me a while, but I am happy to say that the first release of the Stata plugin is ready! Have a look at https://dreameater89.github.io/dbnomics/.

Aside from de-bugging, as a next step it would be great in the future to try and write an article about the command (and the platform) to publish on the Stata Journal, perhaps present it at future Stata conferences. It would definitely help promoting the platform as well as the tool itself. I understand you may prefer to reach a certain milestone version before thinking about that, so I’m happy discuss more in the future. In the meantime, let me know if you spot any bug or would like to see things changed.

Best,
Simone

Hi Simone,
Great news! I will use it and go back to you soon (I have to install Stata before).
Definitely it could be very useful to present it at conferences. We expect to have a version to do that very soon (end of March).
Thanks again for your contribution.
Thomas

Hello everyone, Any news from this wonderful project? I just tried DBnomics (v1.0.2) but got some error messages. For example:

dbnomics providers, clear
Downloading list of providers…
jsoncolsArray(): 3010 attempt to dereference NULL pointer
json2table(): - function returned error
: - function returned error

Did I miss anything?

Hi Nathalex,

The reason why you get such error (pretty much on any dbnomics subcommand) is because I last updated the routine on version 0.18.0 of the API. I see now the API got updated to version 0.21.5. Likely there are some structural changes to the API response which I am not parsing correctly.

Haven’t had much time to keep the version update in the last weeks and months, but I will try to do so in the next days. Keep following this thread and the github page https://github.com/dreameater89/dbnomics for new updates!

Best,
Simone

2 Likes

Hi Nathalex,

The stata command was just updated to work with version 0.21.5 of the API. Could you please update (net install dbnomics, from("https://dreameater89.github.io/dbnomics") replace) and let me know if the updated version works? I plan a couple more improvements in the next weeks, this was just a temporary patch to make things work with the new version of the API.

Best,
Simone

Great news Simone, thank you very much for your contribution!

1 Like

Hi @signorsignore,

Thanks for your work!

I just wanted to inform you that we updated the DBnomics web-site to add a link to the Stata client in the footer.

1 Like

Hi all,

Thank you for your great work in the past months, in particular for having pushed version 22 of the API. I finally have some spare time to get back to this project and update the Stata routine. I hope to be able to push a new version by the end of the month.

In the meantime, I have been proposed to present the Stata command to the upcoming 25th UK Stata Conference in London (September 2019), which I think would be a great opportunity to showcase not only my API client but the overall project.

However, I wanted to check with you first to get your consent (I could work independently towards a draft presentation and share it with you first, if interested). Let me know if this would be interesting for you.

Best,
Simone

Hi Simone,
Thank you for your message! Your initiative is really helpful for us. I attach a basic presentation we’ve made, if it can help (DBnomics_presentation.pdf (292.1 KB)).
Best,
Thomas

Hi all,

Apologies for the long silence, but I come with good news. I have been invited to speak at the upcoming 2019 London Stata Conference (2019 London Stata Conference | Stata), where I will present the Stata plugin for dbnomics. The conference will happen this week (5th of September).

I have put together the attached slides (I believe they will be published on the conference’s website at the end of the event). If you spot anything wrong with the presentation (especially in the initial slides, where I present the platform itself) let me know! I might still have time to correct. Stata_UK2019_signore_dbnomics_handout.pdf (245.0 KB)

Thank you again for sending me the material and being very cool about the whole thing. I will report back the reaction from the conference’s audience!

Best,
Simone

Hi Simone,

What a great presentation, very convincing, congrats!
Thank you very much for your contribution. I am waiting for the reaction from the audience.

Best,
Thomas