# Open Numbers

**URL:** <https://dbnomics.discourse.group/t/open-numbers/66>\
**Category:** Community\
**Created:** [June 27, 2017, 12:41pm UTC](https://dbnomics.discourse.group/t/open-numbers/66 "2017-06-27T12:41:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![johan](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/johan/32/66_2.png) [@johan](https://dbnomics.discourse.group/u/johan)\
**Post date:** [June 27, 2017, 12:41pm UTC](https://dbnomics.discourse.group/t/open-numbers/66/1 "2017-06-27T12:41:14Z")

</div>

## Open Numbers: Crowd-sourced harmonization of global & local statistics

It’s a community for data-crunchers who harmonize statistics to make them more useful for a fact-based worldview. [Gapminder](http://www.gapminder.org/tools/) moderates the process to make sure people participating know what they are doing.

- [https://open-numbers.github.io/](https://open-numbers.github.io/)
- [https://github.com/open-numbers](https://github.com/open-numbers)

---

<div class="post-metadata">

**Author:** ![johan](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/johan/32/66_2.png) [@johan](https://dbnomics.discourse.group/u/johan)\
**Post date:** [August 30, 2017, 5:12pm UTC](https://dbnomics.discourse.group/t/open-numbers/66/2 "2017-08-30T17:12:27Z")

</div>

Dans Open Numbers, chaque jeu de données est structuré suivant le modèle DDF. 4 éléments : DataPoints (“simple data storage”), Entities (“defining single-dimensional data”), Concepts (“giving information about your column headers”), Metadata (“giving information about your data”) =\> simili SDMX.  
Les données sont stockées en csv (mais pas obligatoirement).

Usage final d’Open Numbers : assemblage de jeux de données pour visualisation dans l’[outil Gapminder](http://www.gapminder.org/tools).

Exemple : ‘[ddf–gapminder–gdp\_per\_capita\_cppp](https://github.com/open-numbers/ddf--gapminder--gdp_per_capita_cppp)’. Assemblage à partir de données World Bank (1990-2015) + données Madison (~200 ans). [Méthodo](https://www.gapminder.org/data/documentation/gd001/).

Liens utiles :

- [Systema Globalis](https://github.com/open-numbers/ddf--gapminder--systema_globalis) (main dataset used in tools on the official Gapminder website. It contains local & global statistics combined from hundreds of sources)
- [An introduction to DDF](https://open-numbers.github.io/ddf.html)
- [DDFcsv format](https://docs.google.com/document/d/1aynARjsrSgOKsO1dEqboTqANRD1O9u7J_xmxy8m5jW8/edit#)
- [DDFcsv datapackage](https://docs.google.com/document/d/10yzpTirAe-sAeMoCRzmMiCde1fEAtzMR6zfRY5E-fQs/edit)
- [DDF\_utils](https://github.com/semio/ddf_utils)

---

<div class="post-metadata">

**Author:** ![johan](https://yyz2.discourse-cdn.com/free1/user_avatar/dbnomics.discourse.group/johan/32/66_2.png) [@johan](https://dbnomics.discourse.group/u/johan)\
**Post date:** [September 28, 2017, 11:37am UTC](https://dbnomics.discourse.group/t/open-numbers/66/3 "2017-09-28T11:37:41Z")

</div>

# 2017-09-26 Discussion with Jasper Heeffer (Gapminder / Open Numbers)

Gapminder 7 people team, including 2 developers (some things outsourced).

The old Gapminder World graph used Flash and data on Google Spreadsheet. **Open Numbers** started about 1,5 years ago for the new version of [Gapminder Tools](http://www.gapminder.org/tools/).

Data is stored in CSV because it’s easy to work with, that’s what researchers want, to dig in, create their own dataset, etc.  
However CSV is not the best to fetch things from, it’s not optimised for queries.

## Architecture

 ![gapminder_data_pipeline](https://global.discourse-cdn.com/free1/uploads/dbnomics/original/1X/7655306a688e2a82ec5ababc92a721333d2f81ff.png)

Python script =\> transformed to DDF =\> data stored in CSV  
Semantic harmonization: for example country names/IDs are changed in order to be harmonized across all stored datasets, using [tables to define and match countries and territories frequently used across many other datasets](https://github.com/open-numbers/ddf--gapminder--geo_entity_domain) (see [all alternative names/IDs for countries](https://github.com/open-numbers/ddf--gapminder--geo_entity_domain/raw/master/ddf--entities--geo--country.csv)).  
Data is stored harmonized but an unchanged copy is also kept.  
“DDF Chef” : a python script (what we would call a fetcher).  
“DDF Recipe” : how to mix and harmonize (“cook”) multiple datasets.

## Challenges

- Automatise more
- Fetch more sources
- Crowdsource the harmonization
- Get an overview of fetchers (which will be updated?, when? etc.)

## Their strength is in data visualisation

[Vizabi](http://vizabi.org/) : Powerful visualisation tools developed in-house ([GitHub](https://github.com/vizabi/vizabi))

Example of dataviz project they’ve done: [Södertörnsmodellen](https://open-numbers.github.io/sodertornsmodellen/)

# Potential of collaboration

DB.nomics is kind of creating the “data architecture”, with a platform on which you could build other projects on the user-side: vizualisation, reuse, mixing, harmonization, etc.  
Users can be individuals but also other websites/platforms  
Gapminder could be one of these users, focused on the dataviz, since the architecture (fetching, agregating, etc.) is not their strength.  
[Data.world](https://data.world/) : similar idea  
See also Quandl