Missing series when downloading large dataset

,
1

SuperMayo wrote:

When you want to download large dataset, some series are missing without any warning message.

How to reproduce this bug:

Haven’t tried this in JSON, maybe only a SDMX bug.

2

@MichelJuillard wrote:

There is in fact a limit of 1000 series on download, but there is a bug with the missing error message. Given the ease to select data in Widukind, it is best to narrow the search rather than downloading very large data sets.
We are still studying an alternative for downloading large amount of data, if possible using compression.

3

Any developments into 2020 on removing the download limit (&limit=1000) - eg zip compression ?

NB Some users have local (academic) longstanding data lookup systems that auto-query the full database (or large selection) once downloaded to obtain latest/current target data.
eg being able to download the entire OECD:MEI [Quarterly Data only] or OECD:QNA {Quarterly Data only] is very useful & time saving on data surveillance/preparation for analysis/estimation. (Keep up this great work on DB NOMICS ! - amazing project )

4

Hi @MT_MANC,
One possible solution is to use the R client (https://github.com/dbnomics/rdbnomics). It handles internally the download limit. For your example of the OECD/MEI data, you can do:

install.packages("rdbnomics")

# Load libraries
library(rdbnomics)
library(data.table)

# Download OECD/MEI data with a quarterly frequency (~8 min)
x <- rdb("OECD", "MEI", dimensions = list(FREQUENCY = "Q"))

# Save the data (~1.6 Gb)
fwrite(x, file = "C:/TEMP/OECD_MEI_Q.csv", sep = ";", dec = ",")

# Or save the data by series code (33678 series)
for (i in unique(x$series_code)) {
  fwrite(
    x[series_code == i],
    file = paste0("C:/TEMP/OECD_MEI_", i, ".csv"),
    sep = ";", dec = ","
  )
}

Hope it helps.

5

Thanks @s915 for this useful tip. @MT_MANC I hope this helps you find a short-term solution to your problem.

You might also want to check this ongoing discussion about raising the download limit in DBnomics:

Please add any suggestion or feedback you have, so we can take them into consideration.