python3

Topics covered in this chapter:

The python3 Mosel module ("python3.dso") makes it possible to easily exchange data between Mosel and Python^® and to execute Python scripts from within Mosel.

Python is an interpreted programming language for general-purpose programming that has also become popular for scientific and numeric computing. The reference implementation (CPython) is available as Free Software under the terms of the Python Software Foundation License, which is compatible with the GNU General Public License. "Python" is a registered trademark of the Python Software Foundation.

To use this module, the following line must be included in the header of the Mosel model file:

  uses "python3"

Introduction

This module implements functionality for exchanging data between a Mosel model and Python 3 (CPython) and for calling Python 3 scripts.

The python3 module defines an I/O driver for exchanging data using the initializations from and initializations to Mosel constructs.

It is the Mosel run-time library that loads and runs the Python interpreter, not the other way round.

The purpose of the module is to make the extensive scientific and numeric capabilities of Python available from Mosel. This module does not implement an interactive Python shell. However, the interaction of the Mosel model and the Python interpreter is similar to an interactive shell: transferring data to Python and executing Python scripts changes the state of the interpreter.

Prerequisites

This module does not include Python binaries. In order to use Python you need a working installation of Python 3 targeting the same platform as Mosel (you won't be able to use, e.g., the Windows 32-bit version of Python from the Windows 64-bit version of Mosel). The supported Python versions are 3.7 to 3.10. Version 3.8.0 is not and cannot be supported, because of Python issue #37633. For Windows users it is recommended to get the binaries from www.python.org. Advanced users, who want to use multiple Python environments on a single machine, can download and install the Anaconda Python distribution from www.anaconda.com. On recent versions of Mac OS, Python 3 should be pre-installed. Advanced users can also install the binaries from the Python or Anaconda website or via Homebrew. On Linux, Python 3 is usually part of your distribution and provided in a package called "python3" that can be installed via the package manager. Note that Python 3 is not part of the Red Hat 6 standard repository. In order to use the latest Python release on Red Hat 6, we recommend to download the latest Anaconda Python distribution for Linux from the Anaconda website. It is not recommended to compile Python from source.

The Mosel module python3 tries to automatically locate the correct Python libraries on your system, applying the following rules. If the environment variable PYTHONHOME is specified, it will load the libraries of the Python installation in that directory. Otherwise, it searches for the Python executable in the directories specified in the PATH environment variable. If the Python executable has been found, the module will try to load the libraries of the Python installation of that Python executable. If the libraries could not be loaded with the help of the environment variables, then on Windows they are loaded from the latest Python installation specified in the registry (keys: HKEY_CURRENT_USER and HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\3.*\InstallPath). If the previous steps failed, then the python3 module will load the most recent libraries from the standard library search paths. On Windows it looks for python3*.dll and python3.dll, on Linux for libpython3.*[m].so and libpython3.so, and on Mac OSX for libpython3.*[m].dylib and libpython3.dylib.

If you have multiple installations of Python, or if Python could not be located automatically, or if the initialization of Python fails, you will need to set the environment variable PYTHONHOME to point to your Python installation directory or set PATH to include the location of the Python executable.

Note that the loading of Python is not influenced by Mosel statements like setparam("workdir",...) or setenv("PYTHONHOME",...) in the Mosel model that uses the Python module since these don't affect the process environment used for Python loading. The environment variables must be set before launching the Mosel instance that serves for executing Python scripts in order to influence the loading of Python.

As an example, if Python 3.x is installed on Windows in "C:\Program Files\Python3x" then this directory is also the correct value for the PYTHONHOME environment variable or alternatively, add this directory to the PATH environment variable.

The module supports the conversion between Mosel types and pandas and NumPy types. The supported pandas versions are 0.25 to 1.3 and the supported NumPy versions are 1.16 to 1.21. NumPy 1.19.4 is not supported on Windows, because of a problem in the Windows runtime (#1207405).

Windows Anaconda Setup

Set the PYTHONHOME system environment variable to the base directory of Anaconda or to the home directory of a specific Anaconda environment, e.g., C:\ProgramData\Anaconda3\envs\py373. The NumPy module that ships with Anaconda requires the Math Kernel Library (MKL). It is necessary to add

%PYTHONHOME%\Library\bin

to the PATH system environment variable such that NumPy can find the DLLs of that library. If NumPy cannot find the library, the import of NumPy and pandas will fail with an error message similar to:

Traceback ...
    from . import _mklinit
ImportError: DLL load failed: The specified module could not be found.

If you change the system environment variables, then it is necessary to restart Workbench such that the changes take effect. If you run an Insight 5 Execution Worker, setting the system environment variables PYTHONHOME or PATH has no effect on the Insight app, because Insight does not pass those variables to the Insight app. Section Xpress Insight 5 configuration provides information about how to set those variables in Insight.

Linux Anaconda Setup

Set the PYTHONHOME environment variable to the base directory of Anaconda or to the home directory of a specific Anaconda environment, e.g., /opt/anaconda3/envs/py373. The pandas module that ships with Anaconda may require a version of the C++ standard library which is more recent than the one that ships with your operating system. The required library ships with Anaconda and the loading of that library can be forced by adding

$PYTHONHOME/lib/libstdc++.so

to the LD_PRELOAD environment variable. If an incompatible version gets loaded, then the initialization of pandas will fail with an error message similar to:

ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

If you run an Insight 5 Execution Worker, setting the system environment variables PYTHONHOME or LD_PRELOAD has no effect on the Insight app, because Insight does not pass those variables to the Insight app. Section Xpress Insight 5 configuration provides information about how to set those variables in Insight.

Python initialization

The Python environment is automatically initialized at the point where a Mosel model uses for the first time any function that requires it. So we can have the following small example that just executes a Python script:

model "Python script example"
  uses "python3"
  pyrun("my-python-script.py")
end-model

Alternatively it is possible to explicitly initialize Python using the pyinit function. If the initialization fails, activate additional logging by setting the parameter pyinitverbose to true before initializing Python and double check the values of your environment variables (see previous sections).

At the end of the model execution, the Python environment will automatically be released. It is also possible to explicitly release the environment using pyunload. This can be useful for freeing resources allocated by Python.

It is only possible to initialize one Python interpreter per Mosel instance. For that reason it is not possible to initialize and use the Python interpreter in two models in parallel if both models are run in the same Mosel instance. However, you can initialize and use multiple interpreters in concurrent models if each model is run in a separate Mosel instance.

Data types

The types of data that can be exchanged with Python are the Mosel types boolean, integer, real, string and text, plus arrays, lists and sets of these. Nested compositions are supported. Mosel lists and sets are exported to Python lists and sets. Both, dense and sparse Mosel arrays are supported and by default they are mapped to dictionaries of the corresponding element type. If the pandas interface is initialized, then arrays and lists of arrays can also be mapped to pandas Series and DataFrames. Moreover, Mosel arrays can be initialized from NumPy ndarrays and Mosel scalars be initialized from NumPy scalars (see pyinitpandas and pyusepandas for more details).

The following example shows how to invert a matrix with NumPy:

  declarations
    I, J: range
    A, A_inverse: array(I, J) of real
  end-declarations

  writeln("Run Python script that defines invert_matrix function.")
  pyinitpandas
  pyrun("invert_matrix.py")

  I := 0..2
  J := 0..2
  A :: [1,0,3,
        0,1,2,
        0,0,1]

  writeln("Invert matrix with NumPy.")
  pycall("invert_matrix", A_inverse, A)

  writeln("Matrix A_inverse:")
  writeln("A_inverse: ", A_inverse)

At the beginning of this code snippet the pandas interface is initialized via a call to pyinitpandas. This enables the conversion of Mosel arrays from and to pandas Series and from NumPy arrays from this point onwards—this statement will typically occur at the beginning of the program, but standard python3 functionality can already be used before to it. After the pandas initialization a Python example script in a separate file invert_matrix.py is executed, which defines the invert_matrix Python function. This function is then invoked via the pycall procedure. The first parameter of this procedure is the Python function name, the second one is the Mosel array that will be used for storing the result, and the last parameter is the input parameter for the Python function. The Python function takes a pandas Series with a two-dimensional MultiIndex as input and returns a two-dimensional NumPy ndarray:

def invert_matrix(series):
    # Get pivot table of MultiIndex Series as DataFrame.
    df = series.unstack()

    # Compute and return inverse matrix as NumPy ndarray.
    return inv(df)

See the model invert-matrix.mos for a full example. At first, the function creates a pivot table of the MultiIndex Series, such that the resulting DataFrame looks like a two-dimensional matrix. This matrix-like DataFrame is used as an input value for the NumPy inv function, which returns a two-dimensional ndarray, which is then passed to Mosel.

Note that sparse Mosel arrays are exported to sparse Python dictionaries or pandas Series. In this example, the Mosel matrix A is dense, hence the pandas Series is also dense, that is, for each index tuple (i, j) in the cross product of I and J the pandas Series has a value.

When initializing a Mosel array, list, or set from a Python type, the initialization of the Mosel type is additive, which means that the elements of the Python type are added to the existing Mosel array, list, or set. In the example above, the pandas Series is dense such that all elements of the Mosel array will be overwritten. However, if the pandas Series (or dictionary) is sparse and the Mosel array is non-zero, it is necessary to manually clear its contents before initializing it with values from the sparse Python type. In that situation, the Mosel array should be cleared with reset. See Section Driver python for an example with a sparse array and reset: io_example.mos.

Xpress Insight 4 configuration

The python3 module can only be used when Mosel restrictions are disabled (MOSEL_RESTR=0). When the restrictions are disabled, any executed Mosel and Python code have the same rights (in particular for file system access) as the operating system user that runs the Insight Execution Worker. In order to use the python3 module in an Insight app, it is necessary to relax the Mosel restrictions in the Insight Execution Worker configuration file. On Windows, the default location of the worker configuration file is %XPRESSDIR%\bin\xprmsrv.cfg. After relaxing the Mosel restrictions, we strongly recommend that the Insight administrator makes sure of the following points:

• The operating system user that runs the Insight Execution Worker should only be granted the minimal rights that are necessary for running the Insight app.
• Access to the workers should be protected by a password and additionally by IP filters (see the example extract of the configuration file xprmsrv.cfg below).
• If the network is not trusted, the workers should only accept SSH connections: Set TCP_PORT=-1 (configurable via xprmsrv.cfg) and use xssh instead of the xsrv protocol (Execution Worker configuration in the Insight admin interface).
• Only trusted users should be granted the right to upload trusted Insight apps to the Insight Server.

And the Insight app developer needs to address the following points:

• The app should not execute any untrusted Python scripts that an end user may have uploaded as an app attachment (see pyrun function).
• The app should not concatenate untrusted strings entered by the end user (e.g. Insight scalars or arrays) into a Python evaluation string, because this could allow an attacker to inject and execute custom Python code. For example, the first function input parameter of pycall, pyexec and pyget is a Python evaluation string. Note that it is safe to transfer untrusted data between Mosel and Python variables. The developer just needs to avoid using untrusted strings directly in a Python evaluation string parameter.

If the Insight Execution Worker runs on the same machine as the Insight Server, it is recommended to modify the configuration settings in xprmsrv.cfg as follows:

...
# INFO: It is recommended to replace +ALL by -ALL in XPRMSRV_ACCESS when MOSEL_RESTR=0.
XPRMSRV_ACCESS=+127.0.0.1 -ALL
[insight]
# INFO: It is recommended to protect the worker with a password when MOSEL_RESTR=0.
PASS=my_password
# IMPORTANT: Replace the existing MOSEL_RESTR line to disable the Mosel restrictions.
MOSEL_RESTR=0
PYTHONHOME=C:\ProgramData\Anaconda3
PATH=${PYTHONHOME}\Library\bin;${PATH}
...

Restart the Execution Worker after changing the configuration file. Then log into the Insight admin interface, go to Execution Services, edit the Execution Worker, enter the password in the password edit field and save the changes.

Depending on your system configuration, the PYTHONHOME environment variable is optional. The PATH entry is only necessary for Anaconda on Windows. You can also specify the PYTHONHOME and PATH environment variables as system environment variables. Note that it is not sufficient to specify them for your personal user account, because the Insight service runs as a different user.

Xpress Insight 5 configuration

The python3 module can only be used when Mosel restrictions are disabled (MOSEL_RESTR=0). When the restrictions are disabled, any executed Mosel and Python code have the same rights (in particular for file system access) as the operating system user that runs the Insight Execution Worker. In order to use the python3 module in an Insight Mosel app, it is necessary to relax the Mosel restrictions in the Insight Execution Worker application.properties file. On Windows, the default location of the worker properties file is C:\ProgramData\FICO\XpressInsight\Worker\config\application.properties. To disable the restrictions, add the following line:

insight.worker.execution.environment.MOSEL_RESTR=0

After relaxing the Mosel restrictions, we strongly recommend that the Insight administrator makes sure of the following points:

• The operating system user that runs the Insight Execution Worker should only be granted the minimal rights that are necessary for running the Insight app.
• Only trusted users should be granted the right to upload trusted Insight apps to the Insight Server.

And the Insight app developer needs to address the following points:

• The app should not execute any untrusted Python scripts that an end user may have uploaded as an app attachment (see pyrun function).
• The app should not concatenate untrusted strings entered by the end user (e.g. Insight scalars or arrays) into a Python evaluation string, because this could allow an attacker to inject and execute custom Python code. For example, the first function input parameter of pycall, pyexec and pyget is a Python evaluation string. Note that it is safe to transfer untrusted data between Mosel and Python variables. The developer just needs to avoid using untrusted strings directly in a Python evaluation string parameter.

The Insight worker controls the environment variables under which the model runs. In particular, it clears all environment variables except for a list of allowed variables related to Xpress. For this reason, it may be necessary to set certain environment variables in the worker application.properties file, such that the python3.dso Mosel module can find and use Python. For example:

...
# MANDATORY: Replace the existing MOSEL_RESTR line to disable the Mosel restrictions.
insight.worker.execution.environment.MOSEL_RESTR=0

# IMPORTANT: Use one of the following example blocks to configure a Python environment:

# Windows: No configuration needed for using the latest Python.org
#          installation that is registered in the Windows registry.

# Configuration for a specific installation from Python.org.
insight.worker.execution.environment.PATH=C:\\Program Files\\Python39

# Alternative configuration for a specific installation from Python.org.
insight.worker.execution.environment.PYTHONHOME=C:\\Program Files\\Python39

# Configuration for the base environment of Anaconda on Windows.
insight.worker.execution.environment.PYTHONHOME=C:\\ProgramData\\Anaconda3
insight.worker.execution.environment.PATH=\
${insight.worker.execution.environment.PYTHONHOME}\\Library\\bin
...

Restart the Execution Worker after changing the properties file.

Depending on your system configuration, the PYTHONHOME environment variable is optional. Adding ${PYTHONHOME}\Library\bin to the PATH is only necessary for Anaconda on Windows. Note that it is not sufficient to add these as either system or user environment variables as the worker only uses the environment variables configured in its application.properties file and the allowed Xpress related variables.

Control parameters

The following parameters are defined by the module python3:

pyinitverbose	Show additional log messages when initializing Python.
pyusepandas	Enable and control pandas and NumPy conversion.

Procedures and functions

The procedures and functions of the python3 module fail in case of Python compile-time or run-time errors.

pycall	Call a Python object, a function, with optional input arguments and convert the result to a Mosel variable.
pyexec	Execute Python statements from a string.
pyget	Get the result of a Python expression as Mosel variable.
pygetdf	Initialize a list of Mosel arrays from the columns of a pandas DataFrame.
pyinit	Initialize the Python interpreter.
pyinitpandas	Set pyusepandas parameter to true and initialize the pandas interface.
pyrun	Run a Python script and wait until it is finished.
pyset	Assign a Mosel value to a global Python variable.
pysetdf	Convert a list of Mosel arrays to a pandas DataFrame.
pyunload	Release the Python interpreter and its resources.

I/O drivers

The python3 module provides a driver that is designed to be used in initializations blocks for both reading data from and writing data to Python.

Driver python

python:optional_module_name

The driver can only be used in `initializations' blocks. The optional string after the colon is the Python module name to read the data from or write the data to. If a module name is provided, then the optional item labels are understood as attribute names of the specified module. Initializing data to, for example, "python:client_data" will create a new module called client_data. To access that module in Python, you need to import it like a normal Python module ("import client_data"). After importing the module, the converted Mosel variables will be the attributes of the module, for example, client_data.demand.

If no module name is provided, i.e. if the file name is "python:", then the driver behaves like pyset and pyget: When writing data to Python, the optional labels are understood as global variable names. The driver creates or overwrites the global variables by writing the new values to the attributes of the Python __main__ module. If a variable name is not a valid Python variable identifier the driver will succeed anyway and write the value to the module attributes using the name specified in the label. When reading data from Python, the optional labels are understood as Python expressions. At first, the driver checks whether an expression is a global variable and tries to access it by getting it from the attributes of the Python __main__ module. If this fails, the expression is evaluated by Python and the result is written to the Mosel variable.

The driver throws an I/O error if the expression evaluation or the type conversion fails. Use the parameters ioctrl (see setparam) and iostatus (see getparam) to catch I/O errors.

When initializing data from Mosel to Python, a possibly existing Python object with the same name will be replaced by a new Python object with a Python object type that matches the type of the Mosel source variable.

When initializing arrays, sets, or lists from Python to Mosel, the initialization behavior is additive: The elements of the Python structures are added to the Mosel structures and existing elements in the Mosel structures will not be deleted automatically. If, for example, the target Mosel array is dense and the source Python dictionary or Series is sparse, then the Mosel array may contain old and new values after initialization from Python. If the Mosel array is meant to contain only the values retrieved from the Python dictionary, it is recommended to clear the array with reset before initializing it from Python.

Type mapping to Python

• Mosel boolean →
  • Python bool
  • NumPy bool_ if element of pandas Series
• Mosel integer →
  • Python int
  • NumPy int64 if element of pandas Series
• Mosel real →
  • Python float
  • NumPy float64 if element of pandas Series
• Mosel string/text → Python str
• Mosel set → Python set
• Mosel list → Python list
• Mosel array(I) →
  • If pyusepandas: pandas Series with one-dimensional index
  • If not pyusepandas: Python dictionary with scalar keys
• Mosel array(I, J, ...) →
  • If pyusepandas: pandas Series with multi-dimensional index
  • If not pyusepandas: Python dictionary with tuples of scalars as keys
• Mosel list of array →
  • If pyusepandas: pandas DataFrame with one- or multi-dimensional index
  • If not pyusepandas: not supported
• Mosel nested types → Python nested type
• Mosel records: not supported
• Other Mosel external types: not supported

Type mapping from Python

The NumPy and pandas types are only supported if the pandas interface is initialized.

• Mosel boolean ← Python bool; NumPy bool_
• Mosel int ← Python int, bool; NumPy integer, bool_
• Mosel real ← Python float, int, bool; Numpy floating, integer, bool_
• Mosel string/text ← String representation of Python object as returned by repr()
• Mosel list ← Python list
• Mosel set ← iterable types (e.g., set, generator expression, classes that implement __iter__())
• Mosel array ←
  • pandas Series with one- or multi-dimensional index
  • NumPy ndarray with one or multiple dimensions
  • Python dictionary with scalar or tuple keys
• Mosel list of array ←
  • pandas DataFrame with one- or multi-dimensional index
• Mosel nested types ← Python nested type of supported subtypes
• Mosel records: not supported
• Other Mosel external types: not supported

In the following example, a sparse array is transferred to Python and then the same array is reused for retrieving data from a sparse Python dictionary.

model "Python I/O example"
  uses "python3"

  declarations
    I = 1..4
    A: dynamic array(I) of integer
  end-declarations

  A(1) := 1*2; A(3) := 3*2

  initializations to "python:"
    I as "MyRange"
    A
  end-initializations

  pyrun("io-example.py")
  reset(A) ! Delete existing elements from array A.

  initializations from "python:"
    A
  end-initializations

  writeln("Values initialized from Python:")
  writeln("  A   = ", A)
end-model

The content of io-example.py:

print("Values initialized to Python:")
print("  MyRange =", MyRange)
print("  A =", A)
print("Modifying data in Python...")
A = {i: 2 * i for i in MyRange if i % 2 == 0}

Executing this model generates the following output:

Values initialized to Python:
  MyRange = range(1, 5)
  A = {1: 2, 3: 6}
Modifying data in Python...
Values initialized from Python:
  A   = [(2,4),(4,8)]

Troubleshooting

This section describes some known issues and possible solutions.

• Mosel: E-353: Module `python3' disabled by restrictions. This module does not handle Mosel restrictions, it will therefore fail to load if Mosel is run in restricted mode. Section Xpress Insight 5 configuration provides information about configuring the security restrictions.
• If the initialization of a Python extension module fails when it is imported from within Mosel, then first check which Python environment is used from within Mosel (set pyinitverbose to true) and check whether the extension module is installed in that environment. If it is installed and can be imported from within the interactive shell of that environment, but fails when it is imported from within Mosel, then check whether your environment is set up correctly: Windows Anaconda Setup, Linux Anaconda Setup.