python3

The python3 module makes it possible to easily exchange data with Python 3 and execute Python 3 scripts.
Python is an interpreted programming language for general-purpose programming that has also become popular for scientific and numeric computing. The reference implementation (CPython) is available as Free Software under the terms of the Python Software Foundation License, which is compatible with the GNU General Public License.
To use this module, the following line must be included in the header of the Mosel model file:

  uses "python3"

Introduction

This module implements functionality for exchanging data between a Mosel model and Python 3 (CPython) and for calling Python 3 scripts.

The python3 module defines an I/O driver for exchanging data using the initializations from and initializations to Mosel constructs.

It is the Mosel run-time library that loads and runs the Python interpreter, not the other way round.

The purpose of the module is to make the extensive scientific and numeric capabilities of Python available from Mosel. This module does not implement an interactive Python shell. However, the interaction of the Mosel model and the Python interpreter is similar to an interactive shell: writing to the I/O driver and executing Python scripts changes the state of the interpreter.

Prerequisites

This module does not include Python binaries. In order to use Python you need a working installation of Python, version 3.4.0 or newer and targeting the same platform as Mosel (you won't be able to use, e.g., the Windows 32-bit version of Python from the Windows 64-bit version of Mosel). The most recent supported Python version is 3.7.x. Python binaries for Windows and Mac OS are available at www.python.org. Alternatively, you can also download and use the Anaconda Python distribution from www.anaconda.com. On Linux, Python 3 is most likely part of your distribution and provided in a package called "python3" that can be installed via the package manager. Note that Python 3 is not part of the Red Hat Enterprise Linux 7 standard repository. If your Linux distribution does not include Python 3 or if you want to use the latest Python version on Linux, we recommend to download the latest Anaconda Python distribution for Linux from the Anaconda website.

The Mosel module python3 tries to automatically locate the correct Python libraries on your system, applying the following rules. If the environment variable PYTHONHOME is specified, it will load the libraries of the Python installation in that directory. Otherwise, it searches for the Python executable in the directories specified in the PATH environment variable. If the Python executable has been found, the module will try to load the libraries of the Python installation of that Python executable. If the libraries could not be loaded with the help of the environment variables, then on Windows they are loaded from the latest Python installation specified in the registry (keys: HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\3.*\InstallPath). If the previous steps failed, then the python3 module will load the most recent libraries from the standard library search paths. On Windows it looks for python3*.dll and python3.dll, on Linux for libpython3.*m.so and libpython3.so, and on Mac OSX for libpython3.*m.dylib and libpython3.dylib.

If you have multiple installations of Python, or if Python could not be located automatically, or if the initialization of Python fails, you will need to set the environment variable PYTHONHOME to point to your Python installation directory or set PATH to include the location of the Python executable.

Note that the loading of Python is not influenced by Mosel statements like setparam("workdir",...) or setenv("PYTHONHOME",...) in the Mosel model that uses the Python module since these don't affect the process environment used for Python loading. The environment variables must be set before launching the Mosel instance that serves for executing Python scripts in order to influence the loading of Python.

As an example, if Python 3.7.0 is installed on Windows in "C:\opt\python370" then this directory is also the correct value for the PYTHONHOME environment variable or alternatively, add this directory to the PATH environment variable.

Python initialization

The Python environment is automatically initialized at the point where a Mosel model uses for the first time any function that requires it. So we can have the following small example that just executes a Python script:

model "Python script example"
  uses "python3"
  pyrun("my-python-script.py")
end-model

Alternatively it is possible to explicitly initialize Python using the pyinit function. If the initialization fails, then try setting the PYTHONHOME environment variable to the path of your Python installation.

At the end of the model execution, the Python environment will automatically be released. It is also possible to explicitly release the environment using pyunload. This can be useful for resetting the interpreter state.

It is only possible to initialize one Python interpreter per Mosel instance. For that reason it is not possible to initialize and use the Python interpreter in two models in parallel if both models are run in the same Mosel instance. However, you can initialize and use multiple interpreters in concurrent models if each model is run in a separate Mosel instance.

Data types

The types of data that can be exchanged with Python are the four Mosel elementary types boolean, integer, real and string, plus arrays, lists and sets of these. Nested compositions are supported. Both static and dynamic Mosel arrays are supported and mapped to dictionaries of the element type. Mosel lists and sets are exported to Python lists and sets.

There is no direct mapping to other Python types such as NumPy arrays, however these can be exchanged after conversion from and to supported types. For example, a Mosel array will be exported to a Python dictionary, which can be transformed to a NumPy array in Python:

  declarations
    I, J: range
    A, A_inverse: array(I, J) of real
  end-declarations

  writeln('Run Python script that defines invert_matrix function.')
  pyrun('invert_matrix.py')

  I := 0..2
  J := 0..2
  A :: [1,0,3,
        0,1,2,
        0,0,1]

  initializations to PY_IO_GLOBAL_VAR
    I
    J
    A
  end-initializations

  writeln("Invert matrix with NumPy.")
  pyexec('A_inverse = invert_matrix(A, I, J)')

In the Python environment, the exported matrix A and its index sets I and J are available as global variables A, I and J. In the pyexec procedure call, those variables are passed as input parameters to the invert_matrix Python function. This function is defined in the example script invert_matrix.py and has been imported by the pyrun procedure call. It transforms the Python dictionary to a NumPy array, before inverting the matrix:

  # Convert dictionary to NumPy array.
  np_a = np.array([[dictionary[i, j] for j in indices2] for i in indices1])

See the model invert-matrix.mos for a full example. Note that sparse Mosel arrays are exported to sparse Python dictionaries. In this example, the Mosel matrix A is dense, hence the Python dictionary is also dense, that is, for each key tuple (i, j) in the cross product of I and J the dictionary returns a value.

For the opposite operation, that is, initializing a Mosel array from a NumPy array, it is necessary to transform the NumPy array to a Python dictionary:

  # Convert NumPy array to dictionary.
  a_inverse = {(i, j): np_a_inverse[i, j] for j in indices2 for i in indices1}

where np_a_inverse is a NumPy array and a_inverse the dictionary, which is returned by the function invert_matrix. That returned dictionary is stored in the global Python variable A_inverse and can be imported to a Mosel array with:

  initializations from PY_IO_GLOBAL_VAR
    A_inverse
  end-initializations

Note that the initializations from drivers are additive, which means that the elements of the Python dictionary are added to the existing Mosel array. If the Python dictionary is dense as in the example above, then all elements of the Mosel array will be overwritten. However, if the Python dictionary is sparse and the Mosel array is non-zero, it is necessary to manually clear its contents before initializing it with values from the sparse dictionary. In that situation, the Mosel array should be declared as dynamic and be cleared with delcell. See Section Driver python for an example with a dynamic array and delcell: io_example.mos.

Procedures and functions

The procedures and functions of the python3 module fail in case of Python compile-time or run-time errors.

I/O drivers

The python3 module provides a driver that is designed to be used in initializations blocks for both reading data from and writing data to Python.

Driver python

python:module_name

The driver can only be used in `initializations' blocks. The string after the colon is the Python module name to read the data from or write the data to. Using the predefined constant PY_IO_GLOBAL_VAR = "python:__main__" as file name will initialize the Mosel variables from and to global Python variables. Alternatively, you can create your own modules by initializing data to, for example, "python:client_data". To access your own modules in Python, you need to import them like a normal Python module ("import client_data"). After importing the module, the imported Mosel variables will be the attributes of the module, for example, client_data.names.

When initializing data from Mosel to Python, a possibly existing Python objects with the same name will be replaced by a new Python object with a Python object type that matches the one of the Mosel source variable.

When initializing arrays, sets, or lists from Python to Mosel, the initialization behavior is additive: The elements of the Python structures are added to the Mosel structures and existing elements in the Mosel structures will not be deleted automatically. If, for example, the target Mosel array is dense and the source Python dictionary is sparse, then the Mosel array may contain old and new values after initialization from Python. If the Mosel array is meant to contain only the values retrieved from the Python dictionary, it is recommended to define it as dynamic and to delete the non-zero elements with delcell(dynamic_array) before initializing it from Python.

The optional label entries in an initialization block are understood as Python variable (or module attribute) names.

Type mapping to Python:

• Mosel boolean → Python bool
• Mosel integer → Python int
• Mosel real → Python float
• Mosel string → Python str
• Mosel set → Python set
• Mosel list → Python list
• Mosel array(I) → Python dictionary with scalar keys
• Mosel array(I, J, ...) → Python dictionary with tuples of scalars as keys
• Mosel nested types → Python nested type
• Mosel records: not supported
• Mosel external types: not supported

• Mosel boolean ← Python bool
• Mosel int ← Python int
• Mosel real ← Python int, Python float
• Mosel string ← String representation of Python object as returned by repr()
• Mosel list ← Python list
• Mosel set ← iterable types (e.g., set, generator, generator expression, classes that implement __iter__())
• Mosel array ← dictionary with scalar or tuple keys
• Mosel nested types ← Python nested type of supported subtypes
• Mosel records: not supported
• Mosel external types: not supported

In the following example, a sparse array is transferred to Python and then the same array is reused for retrieving data from a sparse Python dictionary.

model "Python I/O example"
  uses "python3"

  declarations
    I = 1..4
    A: dynamic array(I) of integer
  end-declarations

  A(1) := 1*2; A(3) := 3*2

  initializations to PY_IO_GLOBAL_VAR
    I as "MyRange"
    A
  end-initializations

  pyrun("io-example.py")
  delcell(A) ! Delete existing elements from array A.

  initializations from PY_IO_GLOBAL_VAR
    A
  end-initializations

  writeln("Values initialized from Python:")
  writeln("  A   = ", A)
end-model

The content of io-example.py:

print("Values initialized to Python:")
print("  MyRange =", MyRange)
print("  A =", A)
print("Modifying data in Python...")
A = {i: 2 * i for i in MyRange if i % 2 == 0}

Executing this model generates the following output:

Values initialized to Python:
  MyRange = range(1, 5)
  A = {1: 2, 3: 6}
Modifying data in Python...
Values initialized from Python:
  A   = [(2,4),(4,8)]

Troubleshooting

This section describes some known issues and possible solutions.

• Some Python extensions may not work if pyinit and pyunload are called multiple times inside of a single Mosel instance. This is due to a bug in Python's finalization function. See https://docs.python.org/3/c-api/init.html\#c.Py\_FinalizeEx for more details.
• This module does not handle Mosel restrictions, it will therefore fail to load if Mosel is run in restricted mode.