Data Model

Topics covered in this chapter:

All entities in the data model must be declared with the help of type hints for the class attributes.

The data model consists of container classes (xpressinsight.Param, xpressinsight.Scalar, xpressinsight.Index, xpressinsight.PolarsIndex, xpressinsight.Series, xpressinsight.Column, xpressinsight.DataFrame and xpressinsight.PolarsDataFrame), which hold the configuration of a given entity. But rather than instantiate these classes directly, you will apply type hints to a class definition using the xpressinsight.types or xpressinsight.data sub-modules.

When configuring the entities of your application, use xpressinsight.types, e.g.:

import xpressinsight as xi

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    MY_PARAM: xi.types.Param(100)
    MyScalar: xi.types.Scalar(dtype=xi.integer)
    MyIndex: xi.types.Index(dtype=xi.integer)
    MySeries: xi.types.Series(dtype=xi.real, index='MyIndex')
    MyDataFrame: xi.types.DataFrame(
        columns=[
            xi.types.Column('col1', dtype=xi.integer),
            xi.types.Column('col2', dtype=xi.real)
        ], index='MyIndex')

When configuring the entities of another application that you will be reading, use xpressinsight.data, e.g.:

import xpressinsight as xi

@xi.ScenarioData()
class AnotherApp:
    MyScalar: xi.data.Scalar(dtype=xi.integer)
    MyIndex: xi.data.Index(dtype=xi.integer)
    MySeries: xi.data.Series(dtype=xi.real, index='MyIndex')
    MyDataFrame: xi.data.DataFrame(
        columns=[
            xi.data.Column('col1', dtype=xi.integer),
            xi.data.Column('col2', dtype=xi.real)
        ], index='MyIndex')

Parameters

Parameters are scalar values that can be used to configure an Insight scenario. When parameters are declared, their name, data type, and default value must be specified. The data type can be omitted if the default value is specified and vice versa. Parameter values are automatically passed from the scenario to Insight app such that their value from the scenario is available in all execution modes. In particular, the parameter values are available in the standard load mode. Within the Insight execution mode, parameters should be treated as runtime constants. They can only be changed in the Insight user interface and any changes in an execution mode will not be transferred back to the Insight server, i.e., the scenario parameters remain unchanged. Some examples include:

import xpressinsight as xi

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):

    # Examples where data type is inferred from the default value.
    # Parameter "P" of type "xi.integer" with default value 100.
    P: xi.types.Param(100)
    # Parameter "DEBUG" of type "xi.boolean" with default value False.
    DEBUG: xi.types.Param(False)
    # Parameter "PI" of type "xi.real" with default value 3.14.
    PI: xi.types.Param(3.14)
    # Parameter "STR_PARAM" of type xi.string with default value 'My String Param'.
    STR_PARAM: xi.types.Param('My String Param')

    # Examples where data type is explicitly given.
    BOOL_PARAM: xi.types.Param(dtype=xi.boolean)   # Default value False.
    INT_PARAM: xi.types.Param(dtype=xi.integer)    # Default value 0.
    REAL_PARAM: xi.types.Param(dtype=xi.real)      # Default value 0.0.
    STRING_PARAM: xi.types.Param(dtype=xi.string)  # Default value "".

Parameters are typically read-only.

Scalars

Scalar values are used to represent single-valued entities in the data model. Like parameters, when scalars are declared, their name, data type, and default value must be specified. The data type can be omitted if the default value is specified and vice versa. Scalars can either be input values or result values. By default, scalars are treated as input values. Some examples include:

import xpressinsight as xi

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):

    # Examples where data type is inferred from the default value.
    # Input scalar "NumFactory" of type "xi.integer" with default value 100.
    NumFactory: xi.types.Scalar(100, manage=xi.Manage.INPUT)
    # Result scalar "IsOn" of type "xi.boolean" with default value True.
    IsOn: xi.types.Scalar(True, manage=xi.Manage.RESULT)

    # Examples where data type is explicitly given.
    StringScalar: xi.types.Scalar(dtype=xi.string)  # Input scalar with default value "".
    RealScalar: xi.types.Scalar(dtype=xi.real)      # Input scalar with default value 0.0.

Index

An Index is used for the Python representation of the Insight set - a sequence of distinct, unordered values, typically used to inex an Insight array. The Insight data-model requires that all Series, DataFrame and PolarsDataFrame entities are indexed by one of more Index (or PolarsIndex) entities. An Index entity must be assigned a pandas.Index value. Some examples include:

import xpressinsight as xi
import pandas as pd

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    Months: xi.types.Index(dtype=xi.integer, manage=xi.Manage.INPUT)
    Employees: xi.types.Index(dtype=xi.string, manage=xi.Manage.INPUT)

    @xi.ExecModeLoad()
    def load(self):
        self.Months = pd.Index([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
        self.Employees = pd.Index(dtype=str)

Polars Index

A PolarsIndex allows an index entity to be represented as a polars.Series rather than a pandas.Index. In other aspects, they behave the same as an Index. Some examples include:

import xpressinsight as xi
import polars as pl

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    Months: xi.types.PolarsIndex(dtype=xi.integer, manage=xi.Manage.INPUT)
    Employees: xi.types.PolarsIndex(dtype=xi.string, manage=xi.Manage.INPUT)

    @xi.ExecModeLoad()
    def load(self):
        self.Months = pl.Series('Months', [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12])
        self.Employees = pl.Series('Employees', dtype=pl.Utf8())

Both PolarsIndex and Index entities can be used in Series, DataFrame and PolarsDataFrame indexes - it's not the case that DataFrame must be indexed by Index and PolarsDataFrame by DataFrame.

Series

A Series is used as the Python representation of a single Insight array - a sequence of values indexed by tuples of one or more index values, drawn from an Index or PolarsIndex entity. A Series entity must be assigned a pandas.Series value with a compatible index. Some examples include:

import xpressinsight as xi
import pandas as pd

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    Months: xi.types.Index(dtype=xi.integer, manage=xi.Manage.INPUT)
    Employees: xi.types.Index(dtype=xi.string, manage=xi.Manage.INPUT)
    DaysPerMonth: xi.types.Series(index='Months', dtype=xi.integer, manage=xi.Manage.INPUT)
    HoursWorkedPerMonth: xi.types.Series(index=['Employees', 'Months'], dtype=xi.real, manage=xi.Manage.RESULT)

    @xi.ExecModeLoad()
    def load(self):
        self.DaysPerMonth = pd.Series(
            [31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
            index=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
        )
        self.Months = self.DaysPerMonth.index
        self.Employees = pd.Index(['Paul', 'Ringo', 'John', 'George')

    @xi.ExecModeRun()
    def run(self):
        self.HoursWorkedPerMonth = pd.Series(
            [35, 35, 40, 20],
            index = pd.MultiIndex.from_tuples([
                [1, 1, 2, 2],
                ['Paul', 'John', 'Paul', 'Ringo']
            ], names=['Months', 'Employees'])

DataFrame

A DataFrame is used as the Python equivalent of multiple Insight arrays with the same index sets. Each DataFrame defines a number of Index or PolarsIndex entities and one or more Columns which contain the values. A DataFrame entity must be assigned a pandas.DataFrame value with a compatible index. Some examples include:

import xpressinsight as xi
import pandas as pd

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    Months: xi.types.Index(dtype=xi.integer, manage=xi.Manage.INPUT)
    Employees: xi.types.Index(dtype=xi.string, manage=xi.Manage.INPUT)
    EmployeeInfo: xi.types.DataFrame(index='Employees', columns=[
        xi.types.Column('FullName', dtype=xi.string),
        xi.types.Column('HourlyWage', dtype=xi.real)])
    MonthlyBreakdown: xi.types.DataFrame(index=['Employees', 'Months'], columns=[
        xi.types.Column('MaxHours', dtype=xi.real, manage=xi.manage.INPUT),
        xi.types.Column('HoursWorked', dtype=xi.real, manage=xi.manage.RESULT)])

    @xi.ExecModeLoad()
    def load(self):
        self.EmployeeInfo = pd.DataFrame({
                'FullName': ['John Lennon', 'Ringo Starr'],
                'HourlyWage': [12.5, 10.0],
            }, index=['John', 'Ringo'])
        self.MonthlyBreakdown = pd.DataFrame({
                'MaxHours': [37.5, 37.5, 37.5, 35.0],
            }, index = pd.MultiIndex.from_tuples([
                [1, 1, 2, 2],
                ['John', 'Ringo', 'John', 'Ringo']
            ], names=['Months', 'Employees'])

Polars DataFrame

A PolarsDataFrame allows a DataFrame entity to be represented as a polars.DataFrame instead of a pandas.DataFrame. Although Polars data-frames are not indexed, you must still declare one or more Index or PolarsIndex entities as the indexes for the arrays in the Insight data model; these will then be represented as additional columns in the Polars DataFrame value. A DataFrame entity must be assigned a polars.DataFrame value with the expected columns; assigning a polars.LazyFrame is not supported. Some examples include:

import xpressinsight as xi
import pandas as pd

@xi.AppConfig(name="My First Insight Python App", version=xi.AppVersion(0, 1, 2))
class MyApp(xi.AppBase):
    Months: xi.types.Index(dtype=xi.integer, manage=xi.Manage.INPUT)
    Employees: xi.types.Index(dtype=xi.string, manage=xi.Manage.INPUT)
    EmployeeInfo: xi.types.PolarsDataFrame(index='Employees', columns=[
        xi.types.Column('FullName', dtype=xi.string),
        xi.types.Column('HourlyWage', dtype=xi.real)])
    MonthlyBreakdown: xi.types.PolarsDataFrame(index=['Employees', 'Months'], columns=[
        xi.types.Column('MaxHours', dtype=xi.real, manage=xi.manage.INPUT),
        xi.types.Column('HoursWorked', dtype=xi.real, manage=xi.manage.RESULT)])

    @xi.ExecModeLoad()
    def load(self):
        self.EmployeeInfo = pl.DataFrame({
            'Employees': ['John', 'Ringo'],
            'FullName': ['John Lennon', 'Ringo Starr'],
            'HourlyWage': [12.5, 10.0]
        })
        self.MonthlyBreakdown = pl.DataFrame({
            'Employees': ['John', 'Ringo', 'John', 'Ringo'],
            'Months': [1, 1, 2, 2]
            'MaxHours': [37.5, 37.5, 37.5, 35.0]
        })

General Notes

In the data model, entities may be assigned an alias, which allows to specify a more "user-friendly" label for the entity which will be used in the Insight UI. For example:

MyInteger: xi.types.Scalar(dtype=xi.integer, alias='My Integer')

An entity MyInteger is declared, but will appear as My Integer in the Insight app UI such as table column headers, entity explorer, etc..

To separate the data model across multiple files, you can declare entities in superclasses of your application class.

Polars entity types will only be available if you explicitly install a compatible version of the polars library in your Python environment - xpressinsight will not install this automatically.

Legacy Syntax

In earlier versions of the xpressinsight package (pre-1.4.0), entities were declared in a slightly different syntax that didn't use the xi.types.* functions, e.g:

MyInteger: xi.Scalar(dtype=xi.integer, alias='My Integer')

This syntax is now deprecated. It will continue to work in Python versions up to 3.11 but will not be supported in Python 3.12 and later. Additionally, the legacy syntax is incompatible with the Python deferred annotation functionality (from __future__ import annotations) and with declaring entities in a superclass.

The legacy and current syntaxes may not be used in the same class.