PYMRIO

current status and future plans
pymrio.iedl.no

Konstantin Stadler
Hazim Hussein, Mohamed Badr, Peter Maxwell, Candy Deck, Ruslan Zhuravchak

NTNU Industrial Ecology Programme

2023-06-27

Pymrio

Py(thon) and MRIO
- Open Source
- Focus: Analyzing EE MRIOs
  - But can handle domestic ones as well
- Designed to simplify common IO tasks

Agenda:
- History and current status
- Short tutorial
- Collaboration and future plans

Why did I start developing?

Began as just a personal project (2014)
DRY (Don’t repeat yourself) unheard of for IO analysis
- Novel code for each MRIO analysis
- No testing apart of: do the results look reasonable?
- No collaboration
- No real documentation
Wanted to learn Python, Git and do something useful

What did it become?

Harmonizes the handling of (EE-MR)IO databases
Ideal for data pipelines processing (multiple) MRIO databases
Integration with other Python packages
Common format for saving MRIOs (txt or binary)
Close to 90% test coverage
“Fully” documented: https://pymrio.readthedocs.io/
Citable: Stadler 2021, JORS, DOI: 10.5334/jors.251

Where we are now?

Licensed under the GNU Lesser General Public License (LGPL)
Source on GitHub at: https://github.com/IndEcol/pymrio
Over 100 GitHub stars
Dependency of 16 other repos/2 packages
7 code contributors

Tutorial

Installation

Pymrio is available from the standard Python package repositories:

pip install pymrio

conda-forge

conda install -c conda-forge pymrio

Downloading MRIOs

EXIOBASE example

import pymrio
exio3_folder = "/tmp/mrios/autodownload/EXIO3"
exio_downloadlog = pymrio.download_exiobase3( storage_folder=exio3_folder, system="pxp", years=[2015, 2016])

which gives something like:

print(exio_downloadlog)

Description: EXIOBASE3 metadata file for pymrio
MRIO Name: EXIO3
System: pxp
Version: 10.5281/zenodo.3583070
File: /tmp/mrios/autodownload/EXIO3/metadata.json
History:
20230511 15:10:33 - FILEIO -  Downloaded https://zenodo.org/record/5589597/files/IOT_2016_pxp.zip to IOT_2016_pxp.zip
20230511 15:07:07 - FILEIO -  Downloaded https://zenodo.org/record/5589597/files/IOT_2015_pxp.zip to IOT_2015_pxp.zip

OECD-ICIO example

import pymrio
oecd_folder_v2018 = "/tmp/mrios/autodownload/OECD_2018"
oecd_downloadlog = pymrio.download_oecd(storage_folder=oecd_folder_v2018, years=["2015-2018"])

which gives something like:

print(oecd_downloadlog)

Description: OECD-ICIO download
MRIO Name: OECD-ICIO
System: IxI
Version: v2021
File: /tmp/mrios/autodownload/OECD_2018/download_log.json
History:
20230619 15:22:27 - FILEIO -  Downloaded https://stats.oecd.org/wbos/fileview2.aspx?IDFile=59a3d7f2-3f23-40d5-95ca-48da84c0f861 to ICIO2 021_2015-2018.zip

Summary MRIOs download capabilities

EXIOBASE 3 … works (monetary)
OECD-ICIO … works
GLORIA … since version 0.5
EORA26 … currently broken, fix in the source code
WIOD … currently broken, fix planned
FIGARO … planned
EXIOBASE 1/2 … not implemented

Reading MRIOS

EXIOBASE

mrio_folder = "/tmp/mrios/autodownload/EXIO3/IOT_2015_pxp.zip"
mrio = pymrio.parse_exiobase3(path=mrio_folder)
print(mrio)
#IO System with parameters: Z, Y, x, A, unit, meta, satellite, impacts
print(mrio.satellite)
# Extension Satellite Accounts with parameters: name, F, F_Y, S, S_Y, M, D_cba, D_pba, unit, D_cba_reg, D_pba_reg, D_imp_reg, D_exp_reg

OECD-ICIO

mrio_folder = "/tmp/mrios/autodownload/OECD_2018/ICIO2021_2018.csv"
mrio = pymrio.parse_oecd(path=mrio_folder)
print(mrio)
#IO System with parameters: Z, Y, unit, meta, factor_inputs
print(mrio.factor_inputs)
# Extension factor_inputs with parameters: name, F, F_Y, unit

Summary Parsing/handling of MRIOs

EXIOBASE 1/2/3 … works
OECD-ICIO … works
WIOD … currently broken
EORA26 … currently broken
GLORIA … under development
FIGARO … planned

Architecture

Basic MRIO structure

Map to code

in Python:

mrio.Z
mrio.A
mrio.L
mrio.Y
mrio.unit
mrio.emissions.F
mrio.emissions.F_Y
mrio.emissions.S
mrio.emissions.unit
mrio.emissions.D_cba
mrio.factor_inputs.F
mrio.factor_inputs.S
mrio.factor_inputs.unit

Small Test MRIO

Pymrio includes a small MRIO for testing and development.

import pymrio
mrio = pymrio.load_test()
print(mrio)
# IO System with parameters: Z, Y, unit, population, meta, factor_inputs, emissions

All codes that follows works for all parsed MRIOs.

Explore the system

mrio.get_sectors()
# index(['food', 'mining', 'manufactoring', 'electricity', 'construction',
#       'trade', 'transport', 'other'],
#      dtype='object', name='sector')

Explore the system

mrio.get_sectors()
# index(['food', 'mining', 'manufactoring', 'electricity', 'construction',
#       'trade', 'transport', 'other'],
#      dtype='object', name='sector')

mrio.get_regions()
# Index(['reg1', 'reg2', 'reg3', 'reg4', 'reg5', 'reg6'], dtype='object', name='region')

Explore the system

mrio.get_sectors()
# index(['food', 'mining', 'manufactoring', 'electricity', 'construction',
#       'trade', 'transport', 'other'],
#      dtype='object', name='sector')

mrio.get_regions()
# Index(['reg1', 'reg2', 'reg3', 'reg4', 'reg5', 'reg6'], dtype='object', name='region')

list(mrio.get_extensions())
# ['factor_inputs', 'emissions']

mrio.emissions.get_rows()
# MultiIndex([('emission_type1',   'air'),
#             ('emission_type2', 'water')],
#            names=['stressor', 'compartment'])

Calculating missing accounts

calc_all function
checks available variables and calculates the “remaining” ones for a basic IO analysis
e.g. A and x avaialable, calculates L, Z etc.

import pymrio
mrio = pymrio.load_test()
print(mrio)
# IO System with parameters: Z, Y, unit, population, meta, factor_inputs, emissions
print(mrio.emissions)
# Extension Emissions with parameters: name, F, F_Y, unit

Calculating missing accounts

calc_all function
checks available variables and calculates the “remaining” ones for a basic IO analysis
e.g. A and x avaialable, calculates L, Z etc.

import pymrio
mrio = pymrio.load_test()
print(mrio)
# IO System with parameters: Z, Y, unit, population, meta, factor_inputs, emissions
print(mrio.emissions)
# Extension Emissions with parameters: name, F, F_Y, unit
mrio.calc_all()

Calculating missing accounts

calc_all function
checks available variables and calculates the “remaining” ones for a basic IO analysis
e.g. A and x avaialable, calculates L, Z etc.

import pymrio
mrio = pymrio.load_test()
print(mrio)
# IO System with parameters: Z, Y, unit, population, meta, factor_inputs, emissions
print(mrio.emissions)
# Extension Emissions with parameters: name, F, F_Y, unit
mrio.calc_all()
print(mrio)
# IO System with parameters: Z, Y, x, A, L, unit, population, meta, factor_inputs, emissions
print(mrio.emissions)
# Extension Emissions with parameters: name, F, F_Y, S, S_Y, M, D_cba, D_pba, D_imp, D_exp, unit, D_cba_reg, D_pba_reg, D_imp_reg, D_exp_r eg, D_cba_cap, D_pba_cap, D_imp_cap, D_exp_cap

Exploring accounts

All tables are stored as Pandas DataFrames

mrio.emissions.D_cba
# region                              reg1                                             ...          reg6
# 
# sector                              food         mining manufactoring   electricity  ...  construction         trade     transport         other
# stressor       compartment                                                           ...
# 
# emission_type1 air          2.056183e+06  179423.535893  9.749300e+07  1.188759e+07  ...  1.836696e+06  4.241568e+07  4.805409e+07  3.602298e+07
# emission_type2 water        2.423103e+05   25278.192086  1.671240e+07  1.371303e+05  ...  9.069515e+05  5.449044e+07  8.836484e+06  4.634899e+07

# [2 rows x 48 columns]

mrio.emissions.unit
#                            unit
# stressor       compartment
# emission_type1 air           kg
# emission_type2 water         kg

Accessing values

All tables are stored as Pandas DataFrames

mrio.emissions.D_cba.loc[('emission_type1', 'air'), 'reg2']
# sector
# food             1.793338e+06
# mining           1.914560e+04
# manufactoring    3.397832e+07
# electricity      1.692355e+07
# construction     1.706260e+06
# trade            1.469033e+06
# transport        5.175570e+06
# other            1.583615e+07
# Name: (emission_type1, air), dtype: float64

mrio.emissions.M.loc[('emission_type1', 'air'), (slice(None), 'construction')]
# region  sector
# reg1    construction    0.140381
# reg2    construction    0.104897
# reg3    construction    0.601120
# reg4    construction    5.434724
# reg5    construction    4.630920
# reg6    construction    2.975493
# Name: (emission_type1, air), dtype: float64

Simple aggregation

mrio.rename_regions({'reg1': 'EU', 'reg2':'EU'})
mrio.get_regions()
# Index(['EU', 'reg3', 'reg4', 'reg5', 'reg6'], dtype='object', name='region')

mrio.aggregate_duplicates()
mrio.calc_all()
mrio.emissions.D_cba_reg
# region                                EU          reg3          reg4          reg5          reg6
# stressor       compartment
# emission_type1 air          3.232204e+08  3.457988e+08  4.460602e+08  4.164857e+08  8.244078e+08
# emission_type2 water        1.584347e+08  3.753335e+08  1.721573e+08  1.278938e+08  2.901570e+08

More advanced aggregation methods available.

Scenarios

Lets double the consumption of the EU:

Y = mrio.Y.copy()
Y.loc[:, 'EU'] = Y.loc[:, 'EU'].values * 2

mrio = mrio.reset_all_to_coefficients()
mrio.Y = Y

mrio.calc_all()
mrio.emissions.D_cba_reg
# region                            EU      reg3      reg4      reg5      reg6
# stressor       compartment
# emission_type1 air         4.453e+08 4.294e+08 4.389e+08 4.759e+08 8.976e+08
# emission_type2 water       2.183e+08 5.730e+08 1.349e+08 1.252e+08 2.310e+08

Storing MRIOs

Storing MRIOs - structure

├── A.txt
├── L.txt
├── Y.txt
├── Z.txt
├── emissions
│   ├── D_cba.txt
│   ├── D_cba_cap.txt
│   ├── D_cba_reg.txt
│   ├── D_exp.txt
│   ├── D_exp_cap.txt
│   ├── D_exp_reg.txt
│   ├── D_imp.txt
│   ├── D_imp_cap.txt
│   ├── D_imp_reg.txt
│   ├── D_pba.txt
│   ├── D_pba_cap.txt
│   ├── D_pba_reg.txt
│   ├── F.txt
│   ├── F_Y.txt
│   ├── M.txt
│   ├── S.txt
│   ├── S_Y.txt
│   ├── file_parameters.json
│   └── unit.txt
├── factor_inputs
│   ├── D_cba.txt
│   ├── D_cba_cap.txt
│   ├── D_cba_reg.txt
│   ├── D_exp.txt
│   ├── D_exp_cap.txt
│   ├── D_exp_reg.txt
│   ├── D_imp.txt
│   ├── D_imp_cap.txt
│   ├── D_imp_reg.txt
│   ├── D_pba.txt
│   ├── D_pba_cap.txt
│   ├── D_pba_reg.txt
│   ├── F.txt
│   ├── M.txt
│   ├── S.txt
│   ├── file_parameters.json
│   └── unit.txt
├── file_parameters.json
├── metadata.json
├── population.txt
├── unit.txt
└── x.txt

Storing MRIOs - format

Standard format: csv (txt)
- 2-3 min save/load for the full EXIOBASE system
- 1.9 GB
- exiobase.save_all(‘/storage/folder’, table_format=‘txt’)
- pymrio.load_all(‘/storage/folder’)
Parquet binary format
- 10-20 sec save/load
- 1 GB
- exiobase.save_all(‘/storage/folder’, table_format=‘parquet’)
- pymrio.load_all(‘/storage/folder’)
Binary pickle format
- legacy

Other functions

diagonalize stressors (where do impacts occur)
archive MRIOs
aggregate stressors based on names
characterize stressors
merge extensions
see https://pymrio.readthedocs.io/

Integrations

Country Converter coco
- https://github.com/IndEcol/country_converter
- Aware of most MRIO (and many IAM) regional classifications
- Multiple country classifications (G7, BRIC, OECD, EU, etc.)
- Automatically builds aggreagtions between these for Pymrio
EXIOBASE extension system
- Currently under development (see ISIE presentation)
- Collaborative and modular extension system for EXIOBASE
- Pymrio “glues” custom extension to EXIOBASE

Collaboration

Contributions are very welcome
- reporting issues
- improving documentation
- adding new features
Issue/ticket driven development
- discuss ideas via github issue tracker
- contribute code via pull requests
Code and test standards
https://github.com/IndEcol/pymrio/blob/master/CONTRIBUTING.rst

Future plans

HiTEA (High Throughput Environmental Assessment Pipeline)
- https://hitea.iedl.no
- Monte Carlo sampling
  - implemented (presentation by Mohamed Badr)
- Pymrio on multi-node and cloud systems
- GPU-acceleration
Leontief inverse alternatives (Taylor series etc.)
Downloader/parsers for GLORIA, Figaro (and fix WIOD)
Linking EXIOBASE to LC-IMPACT via Pymrio
- BAMBOO Horizon project https://bamboo-horizon.eu/
- RAINFOREST Horizon project https://bamboo-horizon.eu/
pySUT

Summary Pymrio

Source code: https://github.com/IndEcol/pymrio.
Documentation: https://pymrio.readthedocs.io/
Citation: Stadler 2021, JORS, DOI: 10.5334/jors.251
Open for collaboration

Special session discussion

We presented 3 Open Source projects
Where to find Open Source IO software: IndEcol Dashboard - IO section

IndEcol Dashboard

https://github.com/IndEcol/Dashboard

IndEcol Dashboard

https://github.com/IndEcol/Dashboard

Special session discussion

How to build a community around Open Source development for MRIO analysis?
More Open Source IO projects around?

If we run out of time:

https://hackmd.io/@kstadler/openiioa/edit
Add questions/comments at the bottom (edit on the left)
Keep that open for a week (add your contact)