Querying only monthly series

Community
#1

Is there a simple way to use python or R to walk through all providers and get only monthly series?

If possible I’m capable of writing the code and can share, but would like a general yay or neigh before investing time.

Thanks!

#2

Hi @toast,

There is a way with R to go through all providers and their datasets, and find those that have monthly series.
I just warn you that it takes some time (~1-2h) to download all “dimensions” of the datasets.

At the end, you’ll have a data.table with two columns:

  • provider: names of providers with at least one dataset with a dimension “frequency” (or freq or Freq…) that contains a frequency “M”.
  • dataset: names of datasets with monthly series.
library(data.table)
library(rdbnomics)

options(rdbnomics.progress_bar_datasets = TRUE)
options(rdbnomics.progress_bar_dimensions = TRUE)

# function
sapply_no_null <- function(l, fun) {
  l <- sapply(
    simplify = FALSE,
    l,
    fun
  )
  l <- Filter(Negate(is.null), l)
  if (length(l) > 0) {
    l
  } else {
    NULL
  }
}

# ~1-2h to download and ~1.5Go
dimensions <- rdb_dimensions()

all <- sapply_no_null(
  dimensions,
  function(x) {
    sapply_no_null(
      x,
      function(y) {
        nm <- tolower(names(y))
        if (sum(grepl("^freq", nm)) > 0) {
          z <- y[grep("^freq", names(y), ignore.case = TRUE)]
          sapply_no_null(
            z,
            function(u) {
              if ("M" %in% u[[1]]) {
                u
              } else {
                NULL
              }
            }
          )
        } else {
          NULL
        }
      }
    )
  }
)

provider_dataset <- lapply(all, function(x) { data.table(dataset = names(x)) })
provider_dataset <- rbindlist(provider_dataset, idcol = "provider")
#>       provider            dataset
#>    1:    ACOSS ECONOMIC_BAROMETER
#>    2:      AIH    GGO_IVORY_COAST
#>    3:      AIH        GGO_NIGERIA
#>    4:      AIH        GGO_SENEGAL
#>    5:      BCB                bop
#>   ---                            
#> 4335:      WTO       ITS_MTP_MMPM
#> 4336:      WTO        ITS_MTP_MXP
#> 4337:      WTO       ITS_MTP_MXPM
#> 4338:      WTO         ITS_MTV_MM
#> 4339:      WTO         ITS_MTV_MX

# Check
dimensions[["ACOSS"]][["ECONOMIC_BAROMETER"]][["freq"]]
#>    freq Frequency
#> 1:    M   Monthly
#> 2:    Q Quarterly

Best.

#3

Indeed, there is no way to query the whole DBnomics data, and you must iterate like @s915 explained.