Introduction

This module implements functionality for exchanging data between a Mosel model and Python 3 (CPython) and for calling Python 3 scripts.

The python3 module defines an I/O driver for exchanging data using the initializations from and initializations to Mosel constructs.

It is the Mosel run-time library that loads and runs the Python interpreter, not the other way round.

The purpose of the module is to make the extensive scientific and numeric capabilities of Python available from Mosel. This module does not implement an interactive Python shell. However, the interaction of the Mosel model and the Python interpreter is similar to an interactive shell: transferring data to Python and executing Python scripts changes the state of the interpreter.

Prerequisites

This module does not include Python binaries. In order to use Python you need a working installation of Python 3 targeting the same platform as Mosel (you won't be able to use, e.g., the Windows 32-bit version of Python from the Windows 64-bit version of Mosel). The supported Python versions are 3.4.3 to 3.7.5. Version 3.8.0 is not supported. It is recommended to download and install the Anaconda Python distribution from www.anaconda.com. Alternatively, Python binaries for Windows and Mac OS are also available at www.python.org. On Linux, Python 3 is most likely part of your distribution and provided in a package called "python3" that can be installed via the package manager. Note that Python 3 is not part of the Red Hat Enterprise Linux 7 standard repository. In order to use the latest Python release on Linux, we recommend to download the latest Anaconda Python distribution for Linux from the Anaconda website.

The Mosel module python3 tries to automatically locate the correct Python libraries on your system, applying the following rules. If the environment variable PYTHONHOME is specified, it will load the libraries of the Python installation in that directory. Otherwise, it searches for the Python executable in the directories specified in the PATH environment variable. If the Python executable has been found, the module will try to load the libraries of the Python installation of that Python executable. If the libraries could not be loaded with the help of the environment variables, then on Windows they are loaded from the latest Python installation specified in the registry (keys: HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\3.*\InstallPath). If the previous steps failed, then the python3 module will load the most recent libraries from the standard library search paths. On Windows it looks for python3*.dll and python3.dll, on Linux for libpython3.*m.so and libpython3.so, and on Mac OSX for libpython3.*m.dylib and libpython3.dylib.

If you have multiple installations of Python, or if Python could not be located automatically, or if the initialization of Python fails, you will need to set the environment variable PYTHONHOME to point to your Python installation directory or set PATH to include the location of the Python executable.

Note that the loading of Python is not influenced by Mosel statements like setparam("workdir",...) or setenv("PYTHONHOME",...) in the Mosel model that uses the Python module since these don't affect the process environment used for Python loading. The environment variables must be set before launching the Mosel instance that serves for executing Python scripts in order to influence the loading of Python.

As an example, if Python 3.7.0 is installed on Windows in "C:\opt\python370" then this directory is also the correct value for the PYTHONHOME environment variable or alternatively, add this directory to the PATH environment variable.

The module supports the conversion between Mosel types and pandas and NumPy types. The supported pandas versions are 0.19.2 to 0.24.2 and the supported NumPy versions are 1.11.3 to 1.16.2.

Windows Anaconda Setup

Set the PYTHONHOME system environment variable to the base directory of Anaconda or to the home directory of a specific Anaconda environment, e.g., C:\opt\anaconda3\envs\py373. The NumPy module that ships with Anaconda requires the Math Kernel Library (MKL). It is necessary to add

%PYTHONHOME%\Library\bin

to the PATH system environment variable such that NumPy can find the DLLs of that library. If NumPy cannot find the library, the import of NumPy and pandas will fail with an error message similar to:

Traceback ...
    from . import _mklinit
ImportError: DLL load failed: The specified module could not be found.

If you change the system environment variables, then it is necessary to restart the Insight Execution Worker and Workbench such that the changes take effect. Note that if you run an Insight Execution Worker on the same machine, it will only pass the system environment variables to the Insight app, but the user environment variables of your local user will not be applied in Insight apps, because the server runs as a different user.

Linux Anaconda Setup

Set the PYTHONHOME environment variable to the base directory of Anaconda or to the home directory of a specific Anaconda environment, e.g., /opt/anaconda3/envs/py373. The pandas module that ships with Anaconda may require a version of the C++ standard library which is more recent than the one that ships with your operating system. The required library ships with Anaconda and the loading of that library can be forced by adding

$PYTHONHOME/lib/libstdc++.so

to the LD_PRELOAD environment variable. If an incompatible version gets loaded, then the initialization of pandas will fail with an error message similar to:

ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

Python initialization

The Python environment is automatically initialized at the point where a Mosel model uses for the first time any function that requires it. So we can have the following small example that just executes a Python script:

model "Python script example"
  uses "python3"
  pyrun("my-python-script.py")
end-model

Alternatively it is possible to explicitly initialize Python using the pyinit function. If the initialization fails, then try setting the PYTHONHOME environment variable to the path of your Python installation.

At the end of the model execution, the Python environment will automatically be released. It is also possible to explicitly release the environment using pyunload. This can be useful for freeing resources allocated by Python.

It is only possible to initialize one Python interpreter per Mosel instance. For that reason it is not possible to initialize and use the Python interpreter in two models in parallel if both models are run in the same Mosel instance. However, you can initialize and use multiple interpreters in concurrent models if each model is run in a separate Mosel instance.

Data types

The types of data that can be exchanged with Python are the Mosel types boolean, integer, real, string and text, plus arrays, lists and sets of these. Nested compositions are supported. Mosel lists and sets are exported to Python lists and sets. Both, dense and sparse Mosel arrays are supported and by default they are mapped to dictionaries of the corresponding element type. If the pandas interface is initialized, then arrays and lists of arrays can also be mapped to pandas Series and DataFrames. Moreover, Mosel arrays can be initialized from NumPy ndarrays and Mosel scalars be initialized from NumPy scalars (see pyinitpandas and pyusepandas for more details).

The following example shows how to invert a matrix with NumPy:

  declarations
    I, J: range
    A, A_inverse: array(I, J) of real
  end-declarations

  writeln("Run Python script that defines invert_matrix function.")
  pyinitpandas
  pyrun("invert_matrix.py")

  I := 0..2
  J := 0..2
  A :: [1,0,3,
        0,1,2,
        0,0,1]

  writeln("Invert matrix with NumPy.")
  pycall("invert_matrix", A_inverse, A)

  writeln("Matrix A_inverse:")
  writeln("A_inverse: ", A_inverse)

At the beginning of this code snippet the pandas interface is initialized via a call to pyinitpandas. This enables the conversion of Mosel arrays from and to pandas Series and from NumPy arrays from this point onwards—this statement will typically occur at the beginning of the program, but standard python3 functionality can already be used before to it. After the pandas initialization a Python example script in a separate file invert_matrix.py is executed, which defines the invert_matrix Python function. This function is then invoked via the pycall procedure. The first parameter of this procedure is the Python function name, the second one is the Mosel array that will be used for storing the result, and the last parameter is the input parameter for the Python function. The Python function takes a pandas Series with a two-dimensional MultiIndex as input and returns a two-dimensional NumPy ndarray:

def invert_matrix(series):
    # Get pivot table of MultiIndex Series as DataFrame.
    df = series.unstack()

    # Compute and return inverse matrix as NumPy ndarray.
    return inv(df)

See the model invert-matrix.mos for a full example. At first, the function creates a pivot table of the MultiIndex Series, such that the resulting DataFrame looks like a two-dimensional matrix. This matrix-like DataFrame is used as an input value for the NumPy inv function, which returns a two-dimensional ndarray, which is then passed to Mosel.

Note that sparse Mosel arrays are exported to sparse Python dictionaries or pandas Series. In this example, the Mosel matrix A is dense, hence the pandas Series is also dense, that is, for each index tuple (i, j) in the cross product of I and J the pandas Series has a value.

When initializing a Mosel array, list, or set from a Python type, the initialization of the Mosel type is additive, which means that the elements of the Python type are added to the existing Mosel array, list, or set. In the example above, the pandas Series is dense such that all elements of the Mosel array will be overwritten. However, if the pandas Series (or dictionary) is sparse and the Mosel array is non-zero, it is necessary to manually clear its contents before initializing it with values from the sparse Python type. In that situation, the Mosel array should be cleared with reset. See Section Driver python for an example with a sparse array and reset: io_example.mos.