Introduction

Topics covered in this chapter:

The xpressinsight Python package can be used to develop Python^® based web applications for Xpress Insight. The package makes it possible to easily exchange data between the web application server Xpress Insight and Python and to trigger execution mode functions in Python.

Python is an interpreted programming language for general-purpose programming that has also become popular for scientific and numeric computing. The reference implementation (CPython) is available as Free Software under the terms of the Python Software Foundation License, which is compatible with the GNU General Public License. "Python" is a registered trademark of the Python Software Foundation.

This package implements functionality for developing Python based web applications for Xpress Insight. The purpose of the package is to make the extensive scientific and numeric capabilities of Python available from Insight.

With the help of this package the user can define a Python application class that can be deployed to Insight. In this application class, the user defines the data model and execution mode functions. The package uses the data model to automatically exchange the data between the Insight server and the application. The data will be stored in the attributes of the application instance and the supported types are pandas DataFrames, Series, Index, and scalar Python types.

The Python interpreter will be executed by the Xpress Insight server. The server will automatically load the application and trigger a certain execution mode function. Depending on the type of the execution mode, the data is automatically injected into and extracted from the application instance.

Installation and Setup

Xpress Insight does not include Python binaries. In order to use Python you need a working installation of Python 3 on the developer machine, for development, offline testing, and deployment in Xpress Workbench; and a Python 3 installation on the Xpress Execution Worker machine for user web interface testing and for production. The supported Python versions are 3.8 to 3.11. For Windows users it is recommended to get the binaries from www.python.org. Advanced users, who want to use multiple Python environments on a single machine, can download and install the Anaconda Python distribution from www.anaconda.com. On recent versions of Mac OS, Python 3 should be pre-installed. Advanced users can also install the binaries from the Python or Anaconda website or via Homebrew. On Linux, Python 3 is usually part of your distribution and provided in a package called "python3" that can be installed via the package manager. Note that Python 3.8 or later is not yet part of the Red Hat Enterprise Linux 6 and 7 standard repository. Red Hat 6 is not supported. In order to use the latest Python release on Red Hat 7, we recommend to download the latest Anaconda Python distribution for Linux from the Anaconda website. It is not recommended to compile Python from source.

The package supports the conversion between Insight data types and pandas and NumPy types. The supported pandas versions are 1.0 to 2.0 and the supported NumPy versions are 1.17 to 1.24.

For Windows users who are new to Python, it is recommended to download Python from www.python.org, install it as administrator, and make sure that only a single Python version is installed on the system. The Python installer will ask whether Python should be added to the system environment and whether it should be installed for all users. It is recommended to install Python for all users and to add it to the system environment such that Workbench can find the Python installation. After adding it to the system environment, it is necessary to restart the computer.

Package installation with Administrator Privileges

If Python has been installed to a location that requires administrator privileges for write access, then it is also necessary to install additional Python packages with administrator privileges. Python packages that have been installed to the user's site-packages directory are not accessible for the system user that is running the Insight worker.

Package installation from Python Package Index (PyPI)

A PyPi package is available for Python 3.8 to 3.11 for Windows, Linux, and MacOS. It can be downloaded and installed with the following command:

python -m pip install "xpressinsight~=1.8.0" "xpress~=9.2.5"

Please read the Section Package installation with Administrator Privileges carefully. On Linux and Mac, replace python with python3. The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use xpress version 9.2.5 or later.

Run the following command to uninstall the package(s):

python -m pip uninstall xpressinsight xpress

Package installation from Conda

Before using the xpressinsight package, please make sure that the version number of the conda package in the Anaconda base environment is greater than or equal to 4.9.0:

conda install -n base "conda>=4.9.0"

This is necessary, because in versions prior to 4.8.4 the conda run command does not correctly forward the Python exit code (conda issue #9665) and version 4.9.0 has a new feature for forwarding the output more directly (conda issue #9646).

A conda package is available for Python 3.8 to 3.11 for Windows, Linux, and MacOS. We recommend installing the package into a dedicated Conda environment for Xpress, for example:

conda create -n xpress-env -c fico-xpress python=3.11 "xpressinsight~=1.8.0" "xpress~=9.2.5"

The parameter -n xpress-env specifies the name of the new environment. To choose a different environment name, replace xpress-env.

Please read the Section Package installation with Administrator Privileges carefully. The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use xpress version 9.2.5 or later.

Advanced users can create multiple Anaconda environments or install the package into a pre-existing environment. The Anaconda documentation provides more information about how to manage environments. Example:

conda install -n xpress-env -c fico-xpress "xpressinsight~=1.8.0" "xpress~=9.2.5"

The parameter -n xpress-env specifies the name of the existing environment.

Workbench setup

If the Python environment has been added to the system environment variables, then Workbench will find that Python version. Advanced users can also use the system environment variable PYTHON_EXE, to specify a specific Python executable that Workbench should use if Python has not been added to the system environment. To run Python from within Workbench, it is sufficient to add Python to the user environment, while for Insight it is required to add it to the Insight Execution Worker configuration file. Example:

PYTHON_EXE="C:\Program Files\Python39\python.exe"

Users who did not add the Anaconda base environment to the system environment, can use PYTHON_EXE and the conda run command to select a specific Anaconda environment. Please read Section Package installation from Conda carefully before using conda run. The section also provides an example that shows how to create an environment. Examples:

PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run --no-capture-output python
PYTHON_EXE="C:\Path\To\Scripts\conda.exe" run --no-capture-output -n xpress-env python

The parameter -n xpress-env in the second example specifies the name of the environment that conda will use. Note that you have to create the environment before you can use it. Alternatively, you can use the base environment. To use the base environment, omit the -n xpress-env parameter, as demonstrated in the first example.

Xpress Insight Configuration on Developer Machine

Python can only be used in Insight when Mosel restrictions are disabled (MOSEL_RESTR=0). When the restrictions are disabled, any executed Mosel and Python code have the same rights (in particular for file system access) as the operating system user that runs the Insight Execution Worker. In order to use Python in an Insight app, it is necessary to relax the Mosel restrictions in the Insight Execution Worker application.properties file. On Windows, the default location of the worker properties file is C:\ProgramData\FICO\XpressInsight\Worker\config\application.properties. To disable the restrictions, add the following line:

insight.worker.execution.environment.MOSEL_RESTR=0

After relaxing the Mosel restrictions, we strongly recommend that the Insight administrator makes sure of the following points:

• The operating system user that runs the Insight Execution Worker should only be granted the minimal rights that are necessary for running the Insight app.
• Only trusted users should be granted the right to upload trusted Insight apps to the Insight Server.

And the Insight app developer needs to address the following points:

• The app should not execute any untrusted Python scripts that an end user may have uploaded as an app attachment (e.g. by importing it or by passing the content of the untrusted file to the Python exec function).
• The app should not concatenate untrusted strings entered by the end user (e.g. Insight scalars or arrays) into a Python evaluation string, because this could allow an attacker to inject and execute custom Python code. For example, the first function input parameter of the Python eval and exec functions is a Python evaluation string. Note that it is safe to transfer untrusted data between Insight and Python. The developer just needs to avoid using untrusted strings directly in a Python evaluation string parameter.

The Insight worker controls the environment variables under which the model runs. In particular, it clears all environment variables except for a list of allowed variables related to Xpress. For this reason, it is necessary to set certain environment variables in the worker application.properties file, such that Insight can find and use Python. For example:

...
# MANDATORY: Replace the existing MOSEL_RESTR line to disable the Mosel restrictions.
insight.worker.execution.environment.MOSEL_RESTR=0

# MANDATORY: Use one of the following example blocks to configure a Python environment:

# Configuration for an installation from Python.org.
insight.worker.execution.environment.PATH=C:\\Program Files\\Python39

# Alternative configuration for an installation from Python.org.
insight.worker.execution.environment.PYTHON_EXE=\
"C:\\Program Files\\Python39\\python.exe"

# Configuration for a custom environment of Anaconda on Windows.
insight.worker.execution.environment.PATH=C:\\Windows\\System32
insight.worker.execution.environment.PYTHON_EXE=\
"C:\\Path\\To\\Scripts\\conda.exe" run --no-capture-output -n xpress-env python
# Some versions of conda require the USERPROFILE environment variable to be imported from the system environment.
insight.worker.execution.environment.USERPROFILE=${USERPROFILE}

# Configuration for a custom environment of Anaconda on Unix
insight.worker.execution.environment.PATH=/usr/bin:/bin
insight.worker.execution.environment.PYTHON_EXE=\
"/opt/anaconda3/bin/conda" run --no-capture-output -n xpress-env python
...

Restart the Execution Worker after changing the properties file. Restart the operating system after changing global system environment variables which are on Insight's allow list, e.g. XPAUTH_PATH.

PYTHON_EXE: This variable specifies the command that Insight will call to start Python. It is particularly useful for activating an Anaconda environment with the help of conda run.

conda run: In order to select a specific Anaconda environment, it is recommended to use the conda run command in the PYTHON_EXE environment variable, as shown in the examples above. Specifying only the path to the Anaconda Python executable without using conda run is not sufficient and will cause module import errors. The conda run command automatically configures all necessary environment variables. It's not necessary to set additional environment variables when using conda run. In particular, conda run automatically adds the Python executable to the PATH, such that it's sufficient to use python as the executable name, without having to specify the full path to the executable. To use the Anaconda base environment, omit the environment name parameter -n xpress-env or replace it by -n base.

Import Insight Package

To use this package in Python you first have to install it (see Section Installation and Setup). Afterwards you can import the package by adding the following line to the header of the Python file:

import xpressinsight

Because all Insight types must be accessed by prepending "xpressinsight.", it is advisable to alias the module name upon import:

import xpressinsight as xi

We assume that this is the way the module is imported from now on. It is also possible to import all types to the current name space to avoid prepending the module name or its alias, but this practice is advised against, because then it's difficult to distinguish between, e.g., the Insight Series and pandas Series type.

Limitations

The maximal total size of the data in an xpressinsight.types.Index, xpressinsight.types.Series, or xpressinsight.types.DataFrame column is currently limited to 2 GB. The limits on integer and string types can be found in the integer and string type sections.