Introduction

The xpressinsight Python package can be used to develop Python^® based web applications for Xpress Insight. The package makes it possible to easily trigger execution mode functions in Python and to exchange data between the web application server Xpress Insight and Python.

Python is an interpreted programming language for general-purpose programming that has also become popular for scientific and numeric computing. The reference implementation (CPython) is available as Free Software under the terms of the Python Software Foundation License, which is compatible with the GNU General Public License. "Python" is a registered trademark of the Python Software Foundation.

This package implements functionality for developing Python based web applications for Xpress Insight. The purpose of the package is to make the extensive scientific and numeric capabilities of Python available from Insight.

With the help of this package the user can define a Python application class that can be deployed to Insight. In this application class, the user defines the data model and execution mode functions. The package uses the data model to automatically exchange the data between the Insight server and the application. The data will be stored in the attributes of the application instance and the supported types are pandas DataFrames, Series, Index, and scalar Python types.

The Python interpreter will be executed by the Xpress Insight server. The server will automatically load the application and trigger a certain execution mode function. Depending on the type of the execution mode, the data is automatically injected into and extracted from the application instance.

Installation and Setup

Xpress Insight does not include Python binaries. In order to use Python you need a working installation of Python 3 on the developer machine, for development, offline testing, and deployment in Xpress Workbench; and a Python 3 installation on the Xpress Execution Worker machine for user web interface testing and for production. The supported Python versions are 3.7 to 3.8. For Windows users it is recommended to get the binaries from www.python.org. For advanced users, who want to use multiple Python versions on a single machine, it is recommended to download and install the Anaconda Python distribution from www.anaconda.com. On recent versions of Mac OS, Python 3 should be pre-installed. Advanced users can also install the binaries from the Python or Anaconda website or via Homebrew. On Linux, Python 3 is usually part of your distribution and provided in a package called "python3" that can be installed via the package manager. Alternatively, advanced users can download the latest Anaconda Python distribution for Linux from the Anaconda website.

The package supports the conversion between Insight data type and pandas and NumPy types. The supported pandas version is 1.0 and the supported NumPy versions are 1.17 to 1.18.

For Windows users who are new to Python, it is recommended to install a Python version from www.python.org and to make sure that they only have a single version on their system. The Python installer will ask whether Python should be added to the system environment. It is recommended to add Python to the system environment such that both Workbench and the Execution Worker can find the Python installation. After adding it to the system environment, it is necessary to restart the computer.

Package installation from Python Package Index (PyPI)

The Insight Python package is available on the PyPI server and can be installed with the following command:

python -m pip install xpressinsight "xpress>=8.10"

On Linux and Mac, replace python with python3. The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use version 8.10 or later. Run the following command to uninstall the package(s):

python -m pip uninstall xpressinsight xpress

Package installation from Conda

A conda package is available for download with the following command:

conda install -c fico-xpress xpressinsight "xpress>=8.10"

The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use version 8.10 or later. Run the following command to uninstall the package(s):

conda uninstall xpressinsight xpress

Advanced users can create a new Anaconda environment and install the package in that environment. The Anaconda documentation provides more information about how to manage environments. Example:

conda create -n xpress-env -c fico-xpress python=3.8 xpressinsight "xpress>=8.10"

The parameter -n xpress-env specifies the name of the new environment. To choose a different environment name, replace xpress-env.

Workbench setup

If the Python environment has been added to the system environment variables, then Workbench will find that Python version. Advanced users, can also use the system environment variable PYTHON_EXE, to specify a specific Python executable that Workbench should use if Python has not been added to the system environment. To run Python from within Workbench, it is sufficient to add Python to the user environment, while for Insight it is required to add it to the system environment or to the Insight Execution Worker configuration file. Example:

PYTHON_EXE="C:\Program Files\Python38\python.exe"

Advanced users, who did not add the Anaconda base environment to the system environment, can use this environment variable to select a specific Anaconda environment. Examples:

PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run python
PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run -n xpress-env python
            

The parameter -n xpress-env specifies the name of the environment that conda will use. Note that you have to create the environment before you can use it. Section Package installation from Conda provides an example that shows how to create an environment. Alternatively, you can use the base environment. To use the base environment, omit the -n xpress-env parameter.

Note that there is a bug in the conda run command such that it does not correctly forward the exit code of Python. For this reason, it is currently recommended to avoid this configuration option.

Xpress Insight Configuration on Developer Machine

Python can only be used in Insight when Mosel restrictions are disabled (MOSEL_RESTR=0). When the restrictions are disabled, any executed Mosel and Python code have the same rights (in particular for file system access) as the operating system user that runs the Insight Execution Worker. In order to use Python in an Insight app, it is necessary to relax the Mosel restrictions in the Insight Execution Worker configuration file (xprmsrv.cfg). After relaxing the Mosel restrictions, we strongly recommend that the Insight administrator makes sure of the following points:

• The operating system user that runs the Insight Execution Worker should only be granted the minimal rights that are necessary for running the Insight app.
• Access to the workers should be protected by a password and additionally by IP filters (see the example extract of the configuration file xprmsrv.cfg below).
• If the network is not trusted, the workers should only accept SSH connections: Set TCP_PORT=-1 (configurable via xprmsrv.cfg) and use xssh instead of the xsrv protocol (Execution Worker configuration in the Insight admin interface).
• Only trusted users should be granted the right to upload trusted Insight apps to the Insight Server.

And the Insight app developer needs to address the following points:

• The app should not execute any untrusted Python scripts that an end user may have uploaded as an app attachment (e.g. by importing it or by passing the content of the untrusted file to the Python exec function).
• The app should not concatenate untrusted strings entered by the end user (e.g. Insight scalars or arrays) into a Python evaluation string, because this could allow an attacker to inject and execute custom Python code. For example, the first function input parameter of the Python eval and exec functions is a Python evaluation string. Note that it is safe to transfer untrusted data between Insight and Python. The developer just needs to avoid using untrusted strings directly in a Python evaluation string parameter.

On a developer machine, the Insight Execution Worker usually runs on the same machine as the Insight Server. In this situation, it is recommended to modify the configuration settings in xprmsrv.cfg as follows:

...
# INFO: Removing +ALL from XPRMSRV_ACCESS is recommended when MOSEL_RESTR=0.
XPRMSRV_ACCESS=+127.0.0.1
[insight]
# INFO: Replace the existing MOSEL_RESTR line to disable the Mosel restrictions.
MOSEL_RESTR=0
# INFO: Optional: Specify the license file for the 'xpress' Python package.
XPRESS=${XPRESSDIR}/bin/xpauth.xpr
# INFO: PASS and PYTHON_EXE are optional.
#PASS=my_password
#PYTHON_EXE="C:\Program Files\Python38\python.exe"
#PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run python
#PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run -n xpress-env python
#PYTHON_EXE="/opt/anaconda3/bin/conda" run python
#PYTHON_EXE="/opt/anaconda3/bin/conda" run -n xpress-env python
...

Restart the Execution Worker after changing the configuration file.

PASS: If you have specified a password by uncommenting the line with the PASS variable, then it's necessary to configure Insight in such a way that it uses this password when it connects to the Execution worker: log into the Insight ADMIN interface, go to Execution Services, edit the Execution Worker, enter the password in the password edit field and save the changes.

PYTHON_EXE: If the Python environment has not been added to the system environment, then its possible to explicitly specify the path to the Python executable by uncommenting one of the lines with the PYTHON_EXE variable. You can also specify the PYTHON_EXE environment variable as system environment variable instead of defining it in xprmsrv.cfg. Note that it is not sufficient to specify it for your personal user account, because the Insight service runs as a different user. If you want to select a specific Anaconda environment with the help of the PYTHON_EXE variable, then it's necessary to use the conda run command as shown in the Example. Specifying only the path to the Anaconda Python executable is not sufficient and will cause module import errors. To use the Anaconda base environment, omit the -n xpress-env parameter.

Import Insight Package

To use this package in Python you first have to install it (see Section Installation and Setup). Afterwards you can import the package by adding the following line to the header of the Python file:

import xpressinsight

Because all Insight types must be accessed by prepending "xpressinsight.", it is advisable to alias the module name upon import:

import xpressinsight as xi

We assume that this is the way the module is imported from now on. It is also possible to import all types to the current name space to avoid prepending the module name or its alias, but this practice is advised against, because then it's difficult to distinguish between, e.g., the Insight Series and pandas Series type.