SISEPUEDE model configuration

Apptainer

Apptainer is a container technology that simplifies the creation and execution of containers. Apptainers represents an alternative to Docker in scientific computing.

We are going to use Apptainer in order to create a image for the SISEPUEDE model execution.

The Apptainer installations steps can be found in the next URL:

https://apptainer.org/docs/admin/main/installation.html

Download and interact with pre-built images

We can download pre-built images from repositories like:

https://hub.docker.com
https://quay.io

Docker Hub download example : apptainer pull docker://alpine

Quay download example : apptainer pull docker://quay.io/jitesoft/alpine

Image configuration from sandbox

The image configuration steps are the next

The steps to configure the image using the sandbox format are as follows:

1. Build the image in sandbox format:

apptainer build --sandbox <URI> | <imagen>

1. Request a shell in the sandbox directory:

apptainer shell --writable <DIR>

1. Make the required configurations.
1. Convert the sandbox to SIF format:

apptainer build imagen.sif <DIR>

Image construction of SISEPUEDE model from sandbox format

1) Build the image in sandbox format

We will configure the image to run SISEPUEDE from a pre-built Ubuntu 22.04.4 LTS image from Docker Hub:

apptainer pull docker://ubuntu:22.04

We will use the sandbox format to make changes to the container image. Next we will build the image defining steps in a definition file.

The downloaded image is immutable. Since we must to do configuration changes, we will create a sandbox from the image:

apptainer build --sandbox ssp_model ubuntu_22.04.sif

2) Request a shell in the sandbox directory

We request a shell in the container generated by Apptainer from the sandbox format:

apptainer shell -pfw --no-mount home ssp_model

The flags used mean the following:

-w, --writable : by default all Apptainer containers are available as read only. This option makes the file system accessible as read/write.
-f, --fakeroot : run container with the appearance of running as root
-p, --pid : run container in a new PID namespace

Inside the sandbox we have to update packages:

Apptainer> apt update

We add Apptainer> at the beginning of the prompt In order to make explicit that we are working in a new shell within the container and we can interact with it as though it were a virtual machine.

Python 3.11 installation in the sandbox:

Apptainer> apt install -y python3.11 python3.11-venv python3-venv python3-dev python3-pip

Create the file requirements.txt for the Python packages installation:

Apptainer> echo "
julia==0.6.2
geopy==2.1.0
munch==2.5.0
numpy==1.23.5
pandas==2.2.1
pyDOE2==1.3.0
PyYAML==6.0
SQLAlchemy==2.0.29
XlsxWriter==3.2.0
openpyxl
" > requirements.txt

Apptainer> pip install -r requirements.txt

Julia 1.8.1 installation:

Apptainer> cd /opt
Apptainer> apt install -y wget
Apptainer> wget https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.1-linux-x86_64.tar.gz
Apptainer> tar zxvf julia-1.8.1-linux-x86_64.tar.gz
Apptainer> export PATH=/opt/julia-1.8.1/bin:$PATH

The SISEPUEDE repository is cloned for installing the packages in the julia's environment. Also, the NemoMod repository is cloned and locally installed:

Apptainer> apt install -y git
Apptainer> git clone https://github.com/jcsyme/sisepuede.git
Apptainer> wget https://github.com/sei-international/NemoMod.jl/archive/refs/tags/v1.9.zip
Apptainer> apt install unzip
Apptainer> unzip v1.9.zip 
Apptainer> julia -e 'using Pkg; \
    cd("sisepuede/julia");\
    Pkg.activate(".");\
    Pkg.rm("NemoMod"); \
    Pkg.rm("Gurobi"); \
    Pkg.rm("GAMS"); \
    Pkg.develop(path = "/opt/NemoMod.jl-1.9"); \
    Pkg.instantiate()'

Apptainer> julia -e 'using Pkg; \
    cd("sisepuede/julia");\
    Pkg.activate(".");\
    Pkg.add("PyCall");\
    Pkg.build("PyCall")'

Install the julia package in Python:

Apptainer> python3 -c "import julia; julia.install()"

Clone the SISEPUEDE data repository:

Apptainer> git clone https://github.com/milocortes/sisepuede_data.git
Apptainer> mkdir -p /opt/ssp_input_data

Define a bash program build-ssp-templates.sh that builds the data templates needed for the SISEPUEDE execution.

Apptainer> echo '
#/bin/bash
PAIS=$1
mkdir "/opt/sisepuede/ref/ingestion/calibrated/${PAIS}"
for sector in af ce en ip se
do
    cp "/opt/sisepuede/ref/ingestion/demo/model_input_variables_${sector}_demo.xlsx" "/opt/sisepuede/ref/ingestion/calibrated/${PAIS}/model_input_variables_${PAIS}_${sector}_calibrated.xlsx"
done
' > build-ssp-templates.sh

Download the Python programs that build the input data to SISEPUEDE.

Apptainer> wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/build_real_data.py
Apptainer> wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/build_ssp_input_data.py
Apptainer> wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/merge_ssp_input_output_data.py

Execute SISEPUEDE model

So far we have already the environment for the SISEPUEDE model execution. We are going to test the model execution with Armenia. We have to follow the next steps:

Build data templates:

Apptainer> bash /opt/build-ssp-templates.sh armenia

1. Build input data for the country from the SISEPUEDE repository:
```
Apptainer> python3 /opt/build_real_data.py "$2"
```
1. Populate the templates created in step 1 with the data created in the step 2:
```
Apptainer> python3 /opt/build_ssp_input_data.py
```

We are going to use the program sisepuede_cl.py that is the command line utility to run SISEPUEDE:

Apptainer> cd /opt/sisepuede/python
Apptainer> python3 sisepuede_cl.py  --regions armenia --keys-strategy 1015,5009 --keys-design 0 --keys-future 0 --database-type csv --save-inputs

You can find additional information about the command line utility running the instruction Apptainer> python3 sisepuede_cl.py --help

The outputs of the execution are in the directory /opt/sisepuede/out:

Apptainer> apt install tree
Apptainer> tree /opt/sisepuede/out
/opt/sisepuede/out
|-- sisepuede_run_2024-05-01T23;06;30.231232
|   `-- sisepuede_run_2024-05-01T23;06;30.231232_model_attributes.pkl
`-- sisepuede_run_2024-05-01T23;06;30.916459
    |-- sisepuede_run_2024-05-01T23;06;30.916459_log.log
    |-- sisepuede_run_2024-05-01T23;06;30.916459_model_attributes.pkl
    `-- sisepuede_run_2024-05-01T23;06;30.916459_output_database
        |-- ANALYSIS_METADATA.csv
        |-- ATTRIBUTE_DESIGN.csv
        |-- ATTRIBUTE_PRIMARY.csv
        |-- ATTRIBUTE_STRATEGY.csv
        |-- MODEL_BASE_INPUT_DATABASE.csv
        |-- MODEL_INPUT.csv
        `-- MODEL_OUTPUT.csv

3 directories, 10 files

Creating Apptainer image from Definition File

For a reproducible, verifiable and production-quality container, the Apptainer documentation recommends that you build a SIF file using an Apptainer definition file. The Apptainer definition file can be thinked like a Dockerfile that contains a script of instructions.

Create the definition file ssp_model.def with the content:

Bootstrap : docker
From: ubuntu:22.04

%post
    echo "Update the available packages"
    apt update

    echo "Install Python 3.11 and packages required"
    apt install -y python3.11 python3.11-venv python3-venv python3-dev python3-pip
    echo "
          julia==0.6.2
          geopy==2.1.0
          munch==2.5.0
          numpy==1.23.5
          pandas==2.2.1
          pyDOE2==1.3.0
          PyYAML==6.0
          SQLAlchemy==2.0.29
          XlsxWriter==3.2.0
          openpyxl
          tqdm
          " > requirements.txt
    pip install -r requirements.txt

    echo "Download and install Julia 1.8.1"
    cd /opt
    apt install -y wget
    wget https://julialang-s3.julialang.org/bin/linux/x64/1.8/julia-1.8.1-linux-x86_64.tar.gz
    tar zxvf julia-1.8.1-linux-x86_64.tar.gz
    export PATH=/opt/julia-1.8.1/bin:$PATH

    echo "Clone SISEPUEDE repository"
    apt install -y git
    git clone https://github.com/jcsyme/sisepuede.git

    echo "Clone SISEPUEDE data repository"
    git clone https://github.com/milocortes/sisepuede_data.git
    mkdir -p /opt/ssp_input_data
    
    echo "Install NemoMod in Julia"
    wget https://github.com/sei-international/NemoMod.jl/archive/refs/tags/v1.9.zip
    apt install unzip
    unzip v1.9.zip 
    julia -e 'using Pkg; \
        cd("sisepuede/julia");\
        Pkg.activate(".");\
        Pkg.rm("NemoMod"); \
        Pkg.rm("Gurobi"); \
        Pkg.rm("GAMS"); \
        Pkg.develop(path = "/opt/NemoMod.jl-1.9"); \
        Pkg.instantiate()'

    julia -e 'using Pkg; \
        cd("sisepuede/julia");\
        Pkg.activate(".");\
        Pkg.add("PyCall");\
        Pkg.build("PyCall")'

    python3 -c "import julia; julia.install()"
    apt install -y rsync
    apt install -y zip

    echo '
    #!/bin/bash
    echo  >> /opt/sisepuede/docs/source/csvs/attribute_dim_time_period.csv
    for i in {2051..2070}; do indice="$(( $i - 2015))"; echo "$indice,$i" >> /opt/sisepuede/docs/source/csvs/attribute_dim_time_period.csv ; done
    ' > configura-sisepuede.sh

    echo "
    #!/bin/bash
    python3 /opt/sisepuede/python/sisepuede_cl.py \$*
    " > ejecuta-sisepuede.sh

    echo '
    #/bin/bash
    PAIS=$1
    mkdir "/opt/sisepuede/ref/ingestion/calibrated/${PAIS}"
    for sector in af ce en ip se
    do
        cp "/opt/sisepuede/ref/ingestion/demo/model_input_variables_${sector}_demo.xlsx" "/opt/sisepuede/ref/ingestion/calibrated/${PAIS}/model_input_variables_${PAIS}_${sector}_calibrated.xlsx"
    done
    ' > build-ssp-templates.sh

    echo "Download programs to generate model input data"
    wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/build_real_data.py
    wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/build_ssp_input_data.py
    wget https://raw.githubusercontent.com/milocortes/sisepuede_data/main/utils/merge_ssp_input_output_data.py

%environment
    export PATH=/opt/julia-1.8.1/bin:$PATH
%runscript

    echo "Create input data for the SISEPUEDE model execution"
    bash /opt/build-ssp-templates.sh "$2"
    python3 /opt/build_real_data.py "$2"
    python3 /opt/build_ssp_input_data.py
    
    echo "Execute SISEPUEDE model"
    bash /opt/ejecuta-sisepuede.sh $*

    echo "Merge input and output data"
    mkdir /opt/SSP_RESULTS 
    find /opt/sisepuede/out/ -type f -name "*.csv" -exec mv  "{}" /opt/SSP_RESULTS \;
    python3 /opt/merge_ssp_input_output_data.py

    echo "Compress results and retrieve them to the user file system where the container is running"
    find /opt/SSP_RESULTS/ -type f -name "*.csv" -exec zip -r -j "ssp_$2.zip" "{}"  \;

Build the SIF

apptainer build sisepuede_cl.sif ssp_model.def

We can create a symbolic link for sisepuede cl.sifin order to be executed like any other system executable:

SIF_PATH="$(pwd)/sisepuede_cl.sif"
sudo ln -sv $SIF_PATH /usr/local/bin/sisepuede_cl

We must to configure some Apptainer environment variables for the SIF execution:

export APPTAINER_WRITABLE_TMPFS="true"
export APPTAINER_NO_HOME="true"
export APPTAINER_FAKEROOT="true"

Execute the SIF for Armenia:

sisepuede_cl --regions armenia --keys-strategy 1015,5009 --keys-design 0 --keys-future 0 --database-type csv --save-inputs

The execution return a zip file ssp_armenia.zip with the SISEPUEDE model output on it.

SISEPUEDE configuration in Apptainer