# Missing series when downloading large dataset

**URL:** <https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29>\
**Category:** Uncategorized\
**Tags:** ui, micheljuillard\
**Created:** [February 9, 2017, 10:59am UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29 "2017-02-09T10:59:21Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![eraviart](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/eraviart/32/2_2.png) [@eraviart](https://dbnomics.discourse.group/u/eraviart)\
**Post date:** [February 9, 2017, 10:59am UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29/1 "2017-02-09T10:59:22Z")

</div>

[SuperMayo](https://github.com/SuperMayo) wrote:

When you want to download large dataset, some series are missing without any warning message.

How to reproduce this bug:

- Follow this link (This is the bilateral trade for several countries): [http://widukind-api.cepremap.org/api/v1/sdmx/IMF/data/DOT/6.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122+542.A.txg-fob-usd.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122+542](http://widukind-api.cepremap.org/api/v1/sdmx/IMF/data/DOT/6.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122+542.A.txg-fob-usd.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122+542)
- Search for “273” ( Ctrl +F `"273"`)
- If you look closely you’ll see that country `273` (Mexico) is never shown as “reference area”, which mean that we don’t have exports from Mexico. (searching for `REF-AREA="273"`gives no result)
- If you remove 1 country from the request (for example `542` for Korea) you get the following link: [http://widukind-api.cepremap.org/api/v1/sdmx/IMF/data/DOT/6.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122.A.txg-fob-usd.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122](http://widukind-api.cepremap.org/api/v1/sdmx/IMF/data/DOT/6.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122.A.txg-fob-usd.112+156+158+111+132+136+134+193+273+172+228+186+144+196+122)
- Now, searching for `REF-AREA="273"` gives you 4 returns (but we should have 14 results as there are 15 countries)

Haven’t tried this in JSON, maybe only a SDMX bug.

---

<div class="post-metadata">

**Author:** ![eraviart](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/eraviart/32/2_2.png) [@eraviart](https://dbnomics.discourse.group/u/eraviart)\
**Post date:** [February 9, 2017, 11:01am UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29/2 "2017-02-09T11:01:49Z")

</div>

@MichelJuillard wrote:

There is in fact a limit of 1000 series on download, but there is a bug with the missing error message. Given the ease to select data in Widukind, it is best to narrow the search rather than downloading very large data sets.  
We are still studying an alternative for downloading large amount of data, if possible using compression.

---

<div class="post-metadata">

**Author:** ![MT\_MANC](https://avatars.discourse-cdn.com/v4/letter/m/57b2e6/32.png) [@MT\_MANC](https://dbnomics.discourse.group/u/MT_MANC)\
**Post date:** [January 9, 2020, 9:02pm UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29/3 "2020-01-09T21:02:30Z")

</div>

Any developments into 2020 on removing the download limit (&limit=1000) - eg zip compression ?

NB Some users have local (academic) longstanding data lookup systems that auto-query the _full_ database (or large selection) once downloaded to obtain latest/current target data.  
eg being able to download the entire OECD:MEI [Quarterly Data only] or OECD:QNA {Quarterly Data only] is _very_ useful & time saving on data surveillance/preparation for analysis/estimation. (Keep up this great work on DB NOMICS ! - amazing project )

---

<div class="post-metadata">

**Author:** ![s915](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@s915](https://dbnomics.discourse.group/u/s915)\
**Post date:** [January 14, 2020, 8:51pm UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29/4 "2020-01-14T20:51:54Z")

</div>

> [@MT\_MANC](#):
>
> OECD:MEI

Hi @MT_MANC,  
One possible solution is to use the R client ([https://github.com/dbnomics/rdbnomics](https://github.com/dbnomics/rdbnomics)). It handles internally the download limit. For your example of the OECD/MEI data, you can do:

```r
install.packages("rdbnomics")

# Load libraries
library(rdbnomics)
library(data.table)

# Download OECD/MEI data with a quarterly frequency (~8 min)
x <- rdb("OECD", "MEI", dimensions = list(FREQUENCY = "Q"))

# Save the data (~1.6 Gb)
fwrite(x, file = "C:/TEMP/OECD_MEI_Q.csv", sep = ";", dec = ",")

# Or save the data by series code (33678 series)
for (i in unique(x$series_code)) {
  fwrite(
    x[series_code == i],
    file = paste0("C:/TEMP/OECD_MEI_", i, ".csv"),
    sep = ";", dec = ","
  )
}

```

Hope it helps.

---

<div class="post-metadata">

**Author:** ![johan](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/johan/32/66_2.png) [@johan](https://dbnomics.discourse.group/u/johan)\
**Post date:** [January 15, 2020, 9:24am UTC](https://dbnomics.discourse.group/t/missing-series-when-downloading-large-dataset/29/5 "2020-01-15T09:24:10Z")

</div>

Thanks @s915 for this useful tip. @MT_MANC I hope this helps you find a short-term solution to your problem.

You might also want to check this ongoing discussion about raising the download limit in DBnomics:

> **[Raise the downloaded series limit to 10000 (#480) · Issues · dbnomics-fetchers...](https://git.nomics.world/dbnomics-fetchers/management/-/issues/480)**
>
> Requested by @MichelJuillard As a user I want to download Excel...

Please add any suggestion or feedback you have, so we can take them into consideration.