Using db.nomics in Grafana, Pandas _metadata, formats and stability

Community
1

Hi,

Thanks for providing db.nomics. I made a little Python script that allows me to use db.nomics API results in Grafana through the JSON API plugin:

from flask import Flask, request
from dbnomics import fetch_series
import json

app = Flask(__name__)

@app.route("/series")
def fs():
    if (request.args.get('name') is None):
        return {}
    df = fetch_series(request.args.get('name'))
    return json.loads(df.to_json(date_format='iso', orient='records'))

Unfortunately the dbnomics module does not seem to forward the metadata included in the API response (information about the provider, dataset or series to cite my sources properly, data type information like eur-mn). I am not a heavy user of pandas, but I gather there is a _metadata mechanism that might allow passing the metadata on?

It also seems it would be quite easy to make a dbnomics data source plugin for Grafana, mostly transforming the API responses to Grafana dataframes.

For my personally, it seems I need a wee bit more features than what comes natively and easily with Grafana. I made a bit of a hello world dashboard in it with german public debt figures and already encountered two issues:

Screenshot_2024-03-05_23-30-29
Screenshot_2024-03-05_23-30-291438×571 90.2 KB

Like here on the left the chart consists of 23 time series due to how Destatis provides the data and I kinda would like a virtual data source for that instead of configuring a chart like that through Grafana. And I was looking for ways to turn the time series into cumulative sums, which does not seem supported out of the box.

So I am leaning towards making something like a little SQLite-based proxy where I can just make a view for the 23 time series for the first chart, have rather unlimited flexibility, and might also be able to get the metadata I want into results (there is a “format config from data” transformation for instance).

If I were to implement that, I am wondering if I should use the API, or just take the raw JSON data instead. Neither seems to have a complete documentation, and I do not know which is more stable.

Which brings me to the last thing I noticed in my experiment. Much of the data I actually want to work with comes in .xlsx files from Destatis and Buba, with no time series data for that available. With that in mind, I would actually much rather have a standardized format so I do not have to code for a moving target from both ends. SDMX would seem to be the logical candidate for that, but FOSS tooling for that seems almost non-existent and not well suited for non-professionals like myself.

Any directions?

Thanks!