Introduction
The xpressinsight Python package can be used to develop Python® based web applications for Xpress Insight. The package makes it possible to easily exchange data between the web application server Xpress Insight and Python and to trigger execution mode functions in Python.
Python is an interpreted programming language for general-purpose programming that has also become popular for scientific and numeric computing. The reference implementation (CPython) is available as Free Software under the terms of the Python Software Foundation License, which is compatible with the GNU General Public License. "Python" is a registered trademark of the Python Software Foundation.
This package implements functionality for developing Python based web applications for Xpress Insight. The purpose of the package is to make the extensive scientific and numeric capabilities of Python available from Insight.
With the help of this package the user can define a Python application class that can be deployed to Insight. In this application class, the user defines the data model and execution mode functions. The package uses the data model to automatically exchange the data between the Insight server and the application. The data will be stored in the attributes of the application instance and the supported types are pandas DataFrames, Series, Index, and scalar Python types.
The Python interpreter will be executed by the Xpress Insight server. The server will automatically load the application and trigger a certain execution mode function. Depending on the type of the execution mode, the data is automatically injected into and extracted from the application instance.
Installation and Setup
Xpress Insight does not include Python binaries. In order to use Python you need a working installation of Python 3 on the developer machine, for development, offline testing, and deployment in Xpress Workbench; and a Python 3 installation on the Xpress Execution Worker machine for user web interface testing and for production. The supported Python versions are 3.7 to 3.8. For Windows users it is recommended to get the binaries from www.python.org. For advanced users, who want to use multiple Python versions on a single machine, it is recommended to download and install the Anaconda Python distribution from www.anaconda.com. On recent versions of Mac OS, Python 3 should be pre-installed. Advanced users can also install the binaries from the Python or Anaconda website or via Homebrew. On Linux, Python 3 is usually part of your distribution and provided in a package called "python3" that can be installed via the package manager. Note that Python 3 is not part of the Red Hat Enterprise Linux 6 and 7 standard repository. In order to use the latest Python release on Red Hat, we recommend to download the latest Anaconda Python distribution for Linux from the Anaconda website. It is not recommended to compile Python from source.
The package supports the conversion between Insight data type and pandas and NumPy types. The supported pandas versions are 1.0 to 1.1 and the supported NumPy versions are 1.17 to 1.19.
For Windows users who are new to Python, it is recommended to install a Python version from www.python.org and to make sure that they only have a single version on their system. The Python installer will ask whether Python should be added to the system environment. It is recommended to add Python to the system environment such that both Workbench and the Execution Worker can find the Python installation. After adding it to the system environment, it is necessary to restart the computer.
Package installation from Python Package Index (PyPI)
A PyPi package is available for Python 3.7 to 3.8 for Windows, Linux, and MacOS. It can be downloaded and installed with the following command:
python -m pip install xpressinsight "xpress>=8.11"
On Linux and Mac, replace python with python3. The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use xpress version 8.11 or later.
Run the following command to uninstall the package(s):
python -m pip uninstall xpressinsight xpress
Package installation from Conda
Before using the xpressinsight package, please make sure that the version number of the conda package in the Anaconda base environment is greater than or equal to 4.9.0:
conda install -n base "conda>=4.9.0"
This is necessary, because in versions prior to 4.8.4 the conda run command does not correctly forward the Python exit code (conda issue #9665) and version 4.9.0 has a new feature for forwarding the output more directly (conda issue #9646).
A conda package is available for Python 3.7 to 3.8 for Windows, Linux, and MacOS. It can be downloaded and installed with the following command:
conda install -c fico-xpress xpressinsight "xpress>=8.11"
The xpress package (the last part of the command) is optional and only needed if you want to solve optimization problems with Xpress Optimizer. It is recommended to use xpress version 8.11 or later.
Run the following command to uninstall the package(s):
conda uninstall xpressinsight xpress
Advanced users can create a new Anaconda environment and install the package in that environment. The Anaconda documentation provides more information about how to manage environments. Example:
conda create -n xpress-env -c fico-xpress python=3.8 xpressinsight "xpress>=8.11"
The parameter -n xpress-env specifies the name of the new environment. To choose a different environment name, replace xpress-env.
Workbench setup
If the Python environment has been added to the system environment variables, then Workbench will find that Python version. Advanced users, can also use the system environment variable PYTHON_EXE, to specify a specific Python executable that Workbench should use if Python has not been added to the system environment. To run Python from within Workbench, it is sufficient to add Python to the user environment, while for Insight it is required to add it to the system environment or to the Insight Execution Worker configuration file. Example:
PYTHON_EXE="C:\Program Files\Python38\python.exe"
Users who did not add the Anaconda base environment to the system environment, can use PYTHON_EXE and the conda run command to select a specific Anaconda environment. Please read Section Package installation from Conda carefully before using conda run. The section also provides an example that shows how to create an environment. Examples:
PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run --no-capture-output python PYTHON_EXE="C:\Path\To\Scripts\conda.exe" run --no-capture-output -n xpress-env python
The parameter -n xpress-env in the second example specifies the name of the environment that conda will use. Note that you have to create the environment before you can use it. Alternatively, you can use the base environment. To use the base environment, omit the -n xpress-env parameter, as demonstrated in the first example.
Xpress Insight Configuration on Developer Machine
Python can only be used in Insight when Mosel restrictions are disabled (MOSEL_RESTR=0). When the restrictions are disabled, any executed Mosel and Python code have the same rights (in particular for file system access) as the operating system user that runs the Insight Execution Worker. In order to use Python in an Insight app, it is necessary to relax the Mosel restrictions in the Insight Execution Worker configuration file (xprmsrv.cfg). After relaxing the Mosel restrictions, we strongly recommend that the Insight administrator makes sure of the following points:
- The operating system user that runs the Insight Execution Worker should only be granted the minimal rights that are necessary for running the Insight app.
- Access to the workers should be protected by a password and additionally by IP filters (see the example extract of the configuration file xprmsrv.cfg below).
- If the network is not trusted, the workers should only accept SSH connections: Set TCP_PORT=-1 (configurable via xprmsrv.cfg) and use xssh instead of the xsrv protocol (Execution Worker configuration in the Insight admin interface).
- Only trusted users should be granted the right to upload trusted Insight apps to the Insight Server.
And the Insight app developer needs to address the following points:
- The app should not execute any untrusted Python scripts that an end user may have uploaded as an app attachment (e.g. by importing it or by passing the content of the untrusted file to the Python exec function).
- The app should not concatenate untrusted strings entered by the end user (e.g. Insight scalars or arrays) into a Python evaluation string, because this could allow an attacker to inject and execute custom Python code. For example, the first function input parameter of the Python eval and exec functions is a Python evaluation string. Note that it is safe to transfer untrusted data between Insight and Python. The developer just needs to avoid using untrusted strings directly in a Python evaluation string parameter.
On a developer machine, the Insight Execution Worker usually runs on the same machine as the Insight Server. In this situation, it is recommended to modify the configuration settings in xprmsrv.cfg as follows:
...
# INFO: It is recommended to remove +ALL from XPRMSRV_ACCESS when MOSEL_RESTR=0.
XPRMSRV_ACCESS=+127.0.0.1
[insight]
# IMPORTANT: Replace the existing MOSEL_RESTR line to disable the Mosel restrictions.
MOSEL_RESTR=0
# OPTIONAL: Uncomment to set XPAUTH_PATH to the full path of the Xpress license file.
#XPAUTH_PATH=${XPRESSDIR}/bin/xpauth.xpr
# OPTIONAL: Uncomment PASS and/or one of the PYTHON_EXE lines.
#PASS=my_password
#PYTHON_EXE="C:\Program Files\Python38\python.exe"
#PYTHON_EXE="C:\ProgramData\Anaconda3\Scripts\conda.exe" run --no-capture-output python
#PYTHON_EXE="C:\Path\To\Scripts\conda.exe" run --no-capture-output -n xpress-env python
#PYTHON_EXE="/opt/anaconda3/bin/conda" run --no-capture-output python
#PYTHON_EXE="/opt/anaconda3/bin/conda" run --no-capture-output -n xpress-env python
... Restart the Execution Worker after changing the configuration file.
PASS: If you have specified a password by uncommenting the line with the PASS variable, then it's necessary to configure Insight in such a way that it uses this password when it connects to the Execution worker: log into the Insight ADMIN interface, go to Execution Services, edit the Execution Worker, enter the password in the password edit field and save the changes.
PYTHON_EXE: If the Python environment has not been added to the system environment, then its possible to explicitly specify the path to the Python executable by uncommenting one of the lines with the PYTHON_EXE variable. You can also specify the PYTHON_EXE environment variable as system environment variable instead of defining it in xprmsrv.cfg. Note that it is not sufficient to specify it for your personal user account, because the Insight service runs as a different user.
conda run: In order to select a specific Anaconda environment, it is recommended to use the conda run command in the PYTHON_EXE environment variable, as shown in the example above. Specifying only the path to the Anaconda Python executable without using conda run is not sufficient and will cause module import errors. The conda run command automatically configures all necessary environment variables. It's not necessary to set additional environment variables when using conda run. In particular, conda run automatically adds the Python executable to the PATH, such that it's sufficient to use python as the executable name, without having to specify the full path to the executable. To use the Anaconda base environment, omit the environment name parameter -n xpress-env or replace it by -n base.
Import Insight Package
To use this package in Python you first have to install it (see Section Installation and Setup). Afterwards you can import the package by adding the following line to the header of the Python file:
import xpressinsight
Because all Insight types must be accessed by prepending "xpressinsight.", it is advisable to alias the module name upon import:
import xpressinsight as xi
We assume that this is the way the module is imported from now on. It is also possible to import all types to the current name space to avoid prepending the module name or its alias, but this practice is advised against, because then it's difficult to distinguish between, e.g., the Insight Series and pandas Series type.