I run The Gold Barometer (https://thegoldbarometer.com/), a daily 0-100 score of gold buying conditions for the US market. I would like to suggest it as a DBnomics source, and I am happy to write the fetcher myself with dbnomics-toolbox if the source is welcome.
Yes, I think The Gold Barometer would be a good fit for DBnomics. Thanks a lot for proposing it, and especially for offering to write the fetcher yourself — that’s greatly appreciated!
Although DBnomics primarily focuses on data from national and international institutions, we’ve already integrated sources such as Citymapper and OpenTable, so the fact that The Gold Barometer is an independent project isn’t an issue in itself. The clear CC BY 4.0 licence and the fact that the methodology is published in detail are particularly appreciated from our side.
There is also an interesting coincidence here: we recently had to remove the LBMA datasets from DBnomics because of licensing issues, and we haven’t found a suitable replacement yet. See Removal of LBMA datasets. I was wondering whether Metals.dev might provide an alternative, and I’ll have a closer look at that. If you happen to have any thoughts on possible sources for LBMA data, I’d be very interested to hear them as well.
Regarding the fetcher: the easiest way to go is to create an account on https://git.nomics.world/ then create a personal project. When finished I’ll move it to the dbnomics-fetchers · GitLab group.
Also you’re actually among the very first people, apart from myself and a few others, to build one using dbnomics-toolbox. So there will probably be a few areas where the documentation is still unclear or where the intended way of doing things isn’t obvious yet. Please don’t hesitate to ask me whenever you run into something that isn’t clear. I’ll be happy to help, and your experience will also be very useful for us: it should help us identify and document those rough edges and make the path easier for future fetcher developers.
So, yes — please go ahead with the fetcher. And thanks again for taking the initiative!
Thank you! I will go ahead and start the fetcher, and I will ask here when I hit unclear corners of the toolbox documentation.
On LBMA: from the licensing review I did for my own project, the auction prices themselves are the problem, LBMA licenses them restrictively and that reaches every redistributor (FRED dropped their LBMA-sourced gold fix series for the same reason). So a drop-in daily replacement is genuinely hard: commercial APIs like Metals.dev typically allow display but their terms deserve a close read on redistribution, which is what DBnomics does. If monthly granularity is acceptable, the World Bank Pink Sheet gold price (monthly average, CC BY 4.0) is freely redistributable and well maintained, I use it as one input. I am happy to share notes if useful.
Progress update: the fetcher is written and runs end to end locally. Provider code TGB, two datasets: daily (the published daily readings) and monthly (the overall score plus the six component part scores, back to January 1971, 7 series). I mirrored the structure of abc-fetcher with dbnomics-toolbox 0.9.1.
One onboarding blocker: my GitLab account exists now (username marcpavageau), but project creation in my personal namespace is disabled (“You cannot create projects in your personal namespace”). Could you raise my project limit, or create an empty thegoldbarometer-fetcher project and give me access? I will push right away.
Naming trap: pip install dbnomics-fetcher-toolbox serves an old 0.0.9 package from 2020 with a different API. The right package is dbnomics-toolbox. A note in the docs would save the next developer an hour.
Possible bug or misuse on my side: I create dimensions with labels (Dimension.create(“INDICATOR”, label=“Indicator”, values=[DimensionValue.create(“COMPOSITE”, label=“Overall score”)])) and the model objects keep them, but dataset.json comes out with dimensions_labels and dimensions_values_labels all null. What is the intended way to get labels written? Related question: what is the recommended command to validate the json-data with the new toolbox? (dbnomics-data-model does not build on Python 3.12 because of pysimdjson.)
Thanks for the feedback. I raised your personal project limit to 5 but also have created dbnomics-fetchers / TGB fetcher · GitLab and gave you developer access.
but they are both unmaintained as for now I’m sorry you stumbled upon them and will try to clean them ASAP.
I did not find it, could you please give the URL?
Yes, but where did you find that old package name?
That may be a bug, I’ll have a look at your code, meanwhile you can look at abc-fetcher that writes dimension labels (and dimension value labels).
The data validation command is now baked into dbnomics-cli. It’s still WIP and not yet published to PyPI, but data validation works. Here is an example (I use uvx from uv):
❯ uvx git+https://git.nomics.world/dbnomics/dbnomics-cli --storage-uri ~/Dev/dbnomics/converted-data/abc-json-data/ storage validate
(no error here)
❯ uvx git+https://git.nomics.world/dbnomics/dbnomics-cli --storage-uri ~/Dev/dbnomics/converted-data/scsmich-json-data storage validate
┌──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ code │ dataset_series.unordered_series │
│ message │ The codes of the series of the dataset are not ordered alphabetically │
│ location │ dataset: SCSMICH/MICS │
│ extra │ first_unordered_series_code: "ICC" │
│ │ latest_ordered_series_code: "ICS" │
│ cause │ UnorderedSeriesValidationError( │
│ │ msg='The codes of the series of the dataset are not ordered alphabetically', │
│ │ dataset_id=DatasetId(provider_code='SCSMICH', dataset_code='MICS'), │
│ │ first_unordered_series_code='ICC', │
│ │ latest_ordered_series_code='ICS' │
│ │ ) │
└──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ code │ dataset_series.unordered_series │
│ message │ The codes of the series of the dataset are not ordered alphabetically │
│ location │ dataset: SCSMICH/MPX1PX5 │
│ extra │ first_unordered_series_code: "PX5_MD" │
│ │ latest_ordered_series_code: "PX_MD" │
│ cause │ UnorderedSeriesValidationError( │
│ │ msg='The codes of the series of the dataset are not ordered alphabetically', │
│ │ dataset_id=DatasetId(provider_code='SCSMICH', dataset_code='MPX1PX5'), │
│ │ first_unordered_series_code='PX5_MD', │
│ │ latest_ordered_series_code='PX_MD' │
│ │ ) │
└──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[...]
I’ll add its use cases to the docs of dbnomics-toolbox when it’s ready.
Good news first: the fetcher is finished and the output passes validation. dbnomics-cli storage validate exits 0 on my tgb-json-data (2 datasets: daily, and monthly with 7 series back to 1971, alphabetically ordered).
Two things block me on the GitLab side, whichever fix is easiest for you:
I cannot push the initial commit to dbnomics-fetchers/tgb-fetcher: “You cannot push the initial commit because the default branch is protected and your role does not allow it.” An initial empty commit from you, or a role bump on that project, would both unblock me.
For information: creating a project in my personal namespace now fails with “Failed to create repository”, even though the limit was raised. Not blocking if point 1 is solved.
The old package: searching “dbnomics toolbox” on PyPI returns dbnomics-fetcher-toolbox 0.0.9 (2020), and the old wiki links to dbnomics-fetcher-toolbox.readthedocs.io, so the wrong name looks confirmed from two directions.
On the labels question, here is what I observed while trying to understand my null labels, in case it saves you time (unless I am simply holding the API wrong!): in abc-fetcher’s own output, the declared slugified value codes keep their labels (‘butane’: ‘Butane’) while the codes actually used by the series come out as None (‘Butane’: None). In my fetcher the declared codes and the used codes are identical, and every label ends up None. So it looks like values used by series get registered without their declared label. dimensions_labels come out None in abc’s output too. Happy to file an issue wherever you prefer.
The fetcher is committed locally and ready to push the minute the repo accepts it.
Pushed! The fetcher is now on dbnomics-fetchers/tgb-fetcher (main), and the output still validates with dbnomics-cli. I remain available for any adjustment during review. Thanks a lot for the quick unblocking, and curious to hear what you find on the labels question.
I’m sorry you stumbled upon them and will try to clean them ASAP.