mmjobs

Topics covered in this chapter:

Thanks to this module it is possible to load several models in memory and execute them concurrently. In addition, other instances of Mosel might be started (either locally to the running system or remotely on another machine through the network) and used to run additional models controlled by the model that has started them. This means that the computing capacity of the running model is not restricted to the executing process. A general synchronization mechanism based on event queues as well as two specialized IO drivers are also provided in order to ease the implementation of parallel algorithms in Mosel.
To use this module, the following line must be included in the header of the Mosel model file:

 uses 'mmjobs'

Example

The following example shows how to compile, load, and then run a model from another model. After having started the execution, it waits for 60 seconds before stopping the secondary model if the latter has not yet finished.

model "mmjobs example"
uses "mmjobs","mmsystem"

declarations
 mymod: Model
 event: Event
end-declarations

                              ! Compile 'mymod.mos' to memory
 if compile("","mymod.mos","shmem:bim")<>0
 then
  exit(1)
 end-if

 load(mymod,"shmem:bim")      ! Load bim file from memory...
 fdelete("shmem:bim")         ! ... and release the memory block

                              ! Disable model output
 setdefstream(mymod,"","null:","null:")
 run(mymod)                   ! Start execution and
 wait(60)                     ! wait 1 min for an event

 if waitexpired then          ! No event has been sent...
  writeln("Model too long: stopping it!")
  stop(mymod)                 ! ... stop the model then wait
  wait
 end-if
                              ! An event is available: model finished
 event:=getnextevent
 writeln("Exit status: ", getvalue(event))
 writeln("Exit code  : ", getexitcode(mymod))

 unload(mymod)
end-model

Data sharing between models

A model may share data with its submodels under certain conditions: any initialisation performed by the master model on these shared entities is available to the submodels at their startup and any modification carried out by both the master model and its submodels are effective for all models.

Entities to be shared must be global and identified by the declaration qualifier shared (they do not need to be public). Only scalars of basic types and native types supporting sharing, as well as sets, lists and arrays of basic types can be shared. For the arrays, index sets must be either shared or constants of basic types, shared hashmap arrays cannot have more than 1 dimension.

declarations
 sci: shared integer
 ss: shared set of string
 sa: shared dynamic array(ss,1..2) of real
end-declarations

Data sharing is possible only between a model (the master model) and its clones (i.e. submodels loaded from the running model see load). The master model can manipulate its shared entities just like any other data structure as long as no compatible submodel is running. However, as soon as a submodel using shared data is started the sharing mode is enabled and access to shared entities is altered as follows: sets and lists behave as if they were constant, the structure of arrays is locked (i.e. it is no longer possible to add or remove cells of sparse arrays). Normal access to shared entities is restored when all submodels using them are reset (reset) or unloaded (unload). The current status of the sharing mode can be obtained from the sharingstatus control parameter (getparam).

model "shared example"
uses 'mmjobs'

declarations
 a: shared array(1..3) of integer
 m: Model
end-declarations

if getparam("sharingstatus")<>2 then
                            ! in master model ('a' is empty)
 forall(i in 1..3) a(i):=i  ! initialise 'a'
 writeln("master:",a)       ! output: master:[1,2,3]
 load(m)                    ! clone master then run it
 run(m)
 waitforend(m)              ! wait for its termination
 writeln("aftersub:",a)     ! output: aftersub:[2,3,4]
else
                            ! in submodel ('a' is already initialised)
 writeln("sub:",a)          ! output: sub:[1,2,3]
 forall(i in 1..3) a(i)+=1  ! modify 'a'
end-if

end-model

I/O drivers

The mmjobs module provides a modified version of the mem IO driver designed to be used in a multithreaded environment: memory blocks allocated by the shmem IO driver are persistent (i.e. they are not released after the model terminates) and can be used by several models. Thanks to this facility, models running concurrently may exchange data through memory by means of initialization blocks for instance.

The driver mempipe offers another communication mechanism between models: a memory pipe may be open by two models simultaneously. One of them for writing and the other one for reading. This driver also supports initialization blocks.

The drivers rcmd, xsrv and xssh allow to start additional Mosel instances: they have to be used to build host specifications as expected by the connect function. Finally, thanks to the rmt driver a Mosel instance can access files available from the environment of another instance.

Driver shmem

shmem:label[/minsize[/incstep]]

The file name for this driver is a label: this is the identifier (the first character must be a letter) of the memory block. A label is not local to a particular model and remains valid after the end of the execution of the model having created it. All memory blocks are released when the module mmjobs is unloaded but a given memory block may also be deleted explicitly by calling the fdelete procedure of module mmsystem or by using the fremove C-function of the Native Interface. Note also that deleting the special file "shmem:*" has the effect of releasing all memory blocks handled by the driver.

Several models may open a given label at the same time and several read operations may be performed concurrently. However, writing to a memory block can be done by only one model at a time: if several models try to read and write from/to the same label, only one (it becomes the owner of the memory block) performs its IO operations for writing and the others are suspended until the owner closes its file descriptor to the specified label. Then, one of the waiting models is restarted and becomes the new owner: this process continues until all file descriptors to the label are closed.

The memory block is allocated dynamically and resized as necessary. By default the size of the memory block is increased by pages of 4 kilobytes: the optional parameter incstep may be used to change this page size (i.e. the default setting is "label/0/4k"). The special value 0 modifies the allocation policy: instead of being increased of a fixed amount, the block size is doubled. In all cases unused memory is released when the file is closed.

Driver mempipe

mempipe:[nonblock,]name[/minsize]

A memory pipe is characterized by its name. Only one model may open a pipe for reading but several models may open the same pipe for writing. However, if several models try to write to the same pipe, only one (it becomes the owner of the memory pipe) performs its IO operations and the others are suspended until the owner closes its file descriptor to the specified pipe. Then, one of the waiting models is restarted and becomes the new owner: this process continues until all file descriptors to the pipe are closed. This locking logic is changed if the option nonblock is used for the reading part of the pipe (this option is silently ignored for the other end): all models can write concurrently to the pipe (i.e. after writing a block of data a model looses its ownership of the pipe) and the reader may receive interlaced data.

Each pipe is associated with a buffer: a model can write to a pipe without blocking until this buffer is full. The default size of such a buffer is of 2 kilobytes but a different size may be specified by appending the required size to the pipe name (e.g. "mempipe:mypipe/256" sets the size of the pipe to 256 bytes). This setting will be ignored if the buffer has already been allocated and is larger than the requested size, the buffer will be reallocated anyway if it is not large enough to store a write operation.

There is no notion of 'end of file' in a pipe: if a model tries to read from an empty pipe (i.e. no model is writing to the other end and the pipe buffer is empty) no error is raised and the model is suspended until something is available. Similarly trying to write to a pipe for which no model is reading from the other end might be a blocking operation if its buffer is full. In order to avoid lock ups, it is usually good practice to synchronize the models relying on events, for instance using the pipenotify function. Also, getting the size of a memory pipe with getfsize will return the amount of data currently available in the pipe internal buffer and deleting a pipe with fdelete will release the data structure allocated for the pipe. The content of a pipe may also be cleared thanks to function pipeflush.

Driver rcmd

rcmd:[command]

This driver starts the specified command in a new process and connects its standard input and output streams to the calling Mosel instance. The created process is executed in the same current working directory as the controlling model and inherits the environment variables defined using setenv. The default command is "mosel -r". A typical use for this driver is to start an instance on the current machine or on a remote computer through an external program. For instance:

rcmd:rsh sunbox mosel -r

When Mosel is running in restricted mode (see Section mosel command: restricted mode), the restriction NoExec disables this driver.

Driver xsrv

xsrv:hostname[(port)][/ctx[/pass]][|var=val...]

This driver connects to the host hostname running the Mosel Remote Launcher (see Section The Mosel Remote Launcher xprmsrv) through a TCP socket on port port (default value: 2513) asking for the context ctx (default: xpress) using the password pass (default: no password). Additional environment variables can be specified: assignments of the form var=val must be separated by the symbol | and variable values may include variable references noted ${varname} (expansion is performed on the remote host in the context of its environment). The special environment variable MOSEL_CWD defines the current working directory for the newly created instance.

xsrv:winbox(3211)/xpr64|MOSEL_CWD=C:\workdir|MYDATA=${MOSEL_CWD}\data

Driver xssh

xssh:hostname[(port,kwf)][/ctx[/pass]][|var=val...]

This driver is the secure version of the xsrv driver decribed above: it establishes the connection to the xprmsrv server through an encrypted SSH tunnel (using 2515 as the default TCP port number). In addition to the port number, the driver can also take a file name (kwf) used as the known host file for server authentication: this file contains the list of known hosts with their corresponding public keys. When the connection is established to the remote host, the public key stored in this file is compared with the key provided by the server. The connection is canceled if keys do not match. Generating this known hosts file requires running the command xprmsrv -key public on the remote server in order to retrieve its public key (see Section Running the xprmsrv command).

For instance, the following command will include the server mysun in the knownhosts.txt file (the command must be run on the server):

xprmsrv -key public -hn mysun >>knownhosts.txt

Then after having moved the file to the machine(s) from where connections are initiated, the following connection string may be used to open secure connections with server authentication:

xssh:mysun(knownhosts.txt)

The remote connection is handled by a separate process. By default the program xprmsrv is used as the helper program but it can be replaced by another SSH client by changing the control parameter sshcmd.

Driver rmt

rmt:[node,bs]filename

This driver can be used with any routine expecting a physical file for accessing files on remote instances. By default, the file is located on the instance running on the node identified by the parameter defaultnode but a particular instance may be specified by prefixing the file name by its node number enclosed in square brackets. The special node number -1 designates the parent node of the current instance.

load(mi,mo,"rmt:[-1]model.bim")

By default the driver creates a buffer of 8 kilobytes for its communication operations. The size of this buffer might be changed by specifying the desired buffer size (in kilobytes) after the node number (for instance "rmt:[0,4]filename" to use a 4096 bytes buffer). If only the buffer size has to be stated the node number can be omitted (e.g. "rmt:[,4]filename"). Note that a buffer size must be between 2 and 63.

In addition to physical files, this driver also emulates the behaviour of drivers cb, sysfd, tmp, shmem, mempipe (for writing only) and java such that it can transfer streams from one instance to another. For instance, "rmt:sysfd:2" is the standard error stream of the process running the default node.

The Mosel Remote Launcher xprmsrv

The xprmsrv program is the server part of the "xsrv:" and "xssh:" IO drivers: it must be running on each computer on which instances will be started using these drivers. The communication between two Mosel instances is achieved through a single TCP stream. Mosel instances are started in the context of execution environments: such an environment consists in a set of environment variables as well as the name of the program to start with its initial working directory. The server can manage different execution environments which are identified by a name and optionally protected by a password. Thanks to this feature a single server can offer several versions of Xpress or dedicated settings for particular distributed applications.

This program is also used as an SSH client by mmjob and XPRD when connecting to an xprmsrv server through a secure tunnel. Therefore it must be available when using the "xssh:" IO driver even if no server is to be run on the host machine.

Running the xprmsrv command

Main command line options

The first argument of the command that is not identified as an option is used as the name for a configuration file. The following options are accepted:

-h

Display a short help message and terminate.

-V

Display the version number and terminate.

-tc

Display the current configuration and terminate.

-tm #

The server will terminate after # seconds of inactivity.

-f

Force automatic setting of environment variable XPRESSDIR even if it is already defined.

-v [#]

Set the verbosity level of the communication protocol. The default value is 1 (report only errors) when the server is running in background (service/daemon) and 2 (report activity) when the server is run from a console.

-l fname

Set a logfile to record all messages.

-li addr

Set the address of the interface to use (default: 0.0.0.0 for all interfaces).

-p port

Set the TCP port to listen to (default port is 2513, -1 to disable).

-bp port

Set the UDP port for broadcast (default port is 2514, -1 to disable).

-pf pfname

Define a file name for recording the process number of the server. This file is removed when the server exits.

-d

Start the server in background (or as a daemon on Posix systems).

The following options are used by the Windows version of the server:

-service install

Install the server as a service. All other provided options (including configuration file) are recorded and will be used by the server. If the corresponding service has already been installed, its execution settings are updated with the provided options.

-service remove

Remove the previously installed service.

-service start

Start the previously installed service.

-service reload

Reload configuration.

-service stop

Stop the previously started service.

-service status

Check whether the service is already running.

-u user

This option is used only when installing the service: it selects the user running the service.

-pwd pwd

This option specifies the password required for the user indicated by the -u option.

The following options are used by all other platforms:

-u user

User that should be running the server.

-g group

Group that should be running the server.

When the server is run as a service (under Windows) or as a daemon (on Posix systems) that are usually started by a privileged user, it is recommended to use the appropriate option to run the process as an unprivileged user for security reasons. For instance, under Windows, installing the service can be done using the following command in order to use the network service account:

xprmsrv -service install -u "NT AUTHORITY\NetworkService" conffile

Similarly on a Posix system, the server can be run as the nobody user:

xprmsrv -d -u nobody conffile

Secure server

xprmsrv can also accept secure connections through SSH tunnels: this is the protocol used by the xssh IO driver. The following options are used to setup the secure server:

-sp port

Set the TCP port for SSH connections (default port is 2515, -1 to disable).

-k fname

Private key file name.

-sc cilst

Set the list of accepted ciphers in order of preference (default: "aes256-ctr aes192-ctr aes128-ctr aes256-cbc aes192-cbc aes128-cbc blowfish-cbc 3des-cbc").

The secure server requires a private key to authenticate itself (see following section). By default it will use the file "xprmsrv_rsa.pem" located in the same directory as the xprmsrv executable. It is important to store this file in a secure location as it identifies the server, in particular it must not be readable by Mosel models started by the server. If this file is missing or the provided file name cannot be accessed the secure server will be disabled.

Private key management

A new private key can be generated with the following command:

xprmsrv -key new

Additionally, option -k filename can be specified to change the default key file location. Note that this procedure does not remove an existing key file.

The following command loads and check the validity of a key file:

xprmsrv -key check

When executed on a valid key file this command displays the fingerprint of the public part of the key as well as its properties.

The SSH protocol makes possible authentication of a server by a client. This optional feature, supported by the IO driver xssh, requires a known host file on the client side: this text file consists in a list of host server names with their associated public key. The command xprmsrv -key public generates the required data for such a file using the hostname reported by the operating system to identify the server. Often this hostname does not correspond to the public name of the machine. In such a case, it is possible to replace the label in the file or use the option -hn name to select a different name. For instance, the following command will append to the file knownhosts.txt the public data key for the server using keyfile mykey.pem with host name srvname:

xprmsrv -key public -k mykey.pem -hn srvname >>knownhosts.txt

Mode of operation

The server proceeds as follows:

1. If the environment variable XPRESSDIR is not defined or if the -f option is in use, the value of this environment variable is deduced from the location of the program itself. Under Posix operating systems, the environment variable XPAUTH_PATH is also set up.
2. The environment variables MOSEL_DSO and MOSEL_BIM (see Section Directive uses), MOSEL_TMP (see Section Working directory and temporary directory), MOSEL_EXECPATH (see system), MOSEL_RWPATH, MOSEL_ROPATH (see Section mosel command: restricted mode) and XPRMSRV_ACCESS (see Section Access control list) are cleared and the environment variable MOSEL_RESTR is initialised with value "NoReadNoWriteNoExecNoDBWDOnly" (see Section mosel command: restricted mode).
3. The default execution environment xpress is created: it refers to the Xpress installation detected at the first step.
4. If available, the configuration file is read (see Section Configuration file): it can be used to define global settings (e.g., defining the logfile) or/and create and modify execution environments by defining environment variables.
5. The process then starts its main loop listening to the specified TCP and UDP ports.
6. When a connection is requested, a new session is started to process commands from the client. These commands are used to authenticate the client, select an environment and finally start the Mosel program in a separate process. This process inherits all the environment variables defined in the context and starts in the specified working directory (by default: the location pointed by XPRESSDIR). In addition, on Posix systems, the path ${XPRESSDIR}/lib is added to the dynamic library path of the operating system. Once the process is started, xprmsrv detaches itself from the client — the communication is established directly between the two Mosel instances.

Configuration file

The configuration file consists in a list of variable definitions of one of the following forms:

varname=value

varname?=value

Each statement is recorded in the current environment. The value may contain variable references noted ${varname}, the expansion is executed when the environment is processed except for self references that are expanded at the time of defining the variable (e.g.PATH=${PATH}:otherpath). When the first syntax is used, the variable cannot be changed by a remote host; the second syntax (using ?=) allows a remote host to modify the corresponding variable before starting the Mosel instance.

Switching to a different environment is done by giving the name of the environment enclosed in square brackets:

[newenv]

If the environment name has not yet been used, a new environment is created unless the line ends with the symbol '+' (e.g. [myenv]+). In this case the following definitions are included only if the environment already exists. If the line ends with the symbol '=' (e.g. [myenv]=) the previous definitions for this context are cleared. These markers can be combined (e.g. [myenv]+=) such that the definition block replaces the corresponding context only if it exists.

Upon startup, two environments are automatically created: "global" to store general configuration and settings shared by all environments and "xpress" (it can also be referred to as * or default) the default execution environment. When the reading of the configuration file begins, the global environment is selected: in this environment all variable definitions are processed immediatly and added to the xprmsrv process environment. In this context, some variables have a special meaning and are not handled as ordinary environment variables (all paths must be absolute):

LOGFILE

the file to be used for recording all messages. Messages are sent to the standard error stream when this parameter is not set.

LISTEN

address of the interface to use (default value: 0.0.0.0 for all interfaces).

TCP_PORT

the port number to use for TCP connections (default value: 2513, -1 to disable).

UDP_PORT

the port number to use for UDP connections (default value: 2514, -1 to disable). The server listen to this port for broadcast messages (see procedure findxsrvs).

SSH_PORT

the port number to use for SSH connections (default value: 2515, -1 to disable).

KEYFILE

private key file name used by the SSH protocol (default value: xprmsrv_rsa.pem located in the same directory as the xprmsrv executable).

SSH_CIPHERS

the list of accepted ciphers in order of preference (default: "aes256-ctr aes192-ctr aes128-ctr aes256-cbc aes192-cbc aes128-cbc blowfish-cbc 3des-cbc").

VERBOSITY

verbosity level for the communication protocol (default value: 1 if the server is running in background and 2 if it is run from a console).

GROUPMASK

Bit mask to select what broadcast requests to accept (default value: ANY). The server replies to a request of group grp only if bit test grp&GROUPMASK is not 0 (see procedure findxsrvs). The mask value can be given as an integer ( e.g. 3 to allow groups 1 and 2), an hexadecimal number ( e.g. 0xFF for groups 1 to 128) or the special keyword ANY (all groups allowed).

MAXAUTHTIME

a connection is closed if the authentication procedure takes more than the specified amount of time in seconds (default value:30).

MAXSESSIONS

maximum number of concurrent sessions (the default value 0 disables this limitation).

XPRMSRV_ACCESS

access control list (see Section Access control list).

CONFDIR

a configuration directory path. The server includes each of the files stored in this directory (sorted in alphabetical order) after it has finished reading the main configuration file.

If the corresponding command line options are used (namely options -l, -p, -bp, -sp, -k, -sc and -v) the settings of the configuration file are ignored.

In other contexts, the following variables have a special meaning:

MOSEL_CMD

the command to execute. The default value is "${XPRESSDIR}/bin/mosel -r"

MOSEL_CWD

default working directory. The default value is "${XPRESSDIR}"

RUN_BEFORE

command to be run before MOSEL_CMD. This command is executed in the same environment as MOSEL_CMD after all variables have been defined.

RUN_AFTER

command to be run after MOSEL_CMD. This command is executed in the environment of the server but the variable itself is expanded in the context of MOSEL_CMD before its execution.

PASS

password required to use this environment (empty by default). If this variable is set to the special value "*", the associated environment is disabled.

MAXSESSIONS

maximum number of concurrent sessions running under this context (by default there is no limit; a maximum of 0 or less disables the environment).

XPRMSRV_ACCESS

context specific access control list (applied after the global access list).

XPRMSRV_SID

session ID: if not explicitly defined this variable is automatically set by the server.

XPRMSRV_PEER

IP address of the remote host: if not explicitly defined this variable is automatically set by the server.

For instance, the following configuration file sets the logfile to "/tmp/logfile.txt"; adds the password "hardone" to the default context and defines an additional context named xptest pointing to a different installation of xpress:

# simple xprmsrv config file
LOGFILE=/tmp/logfile.txt

[xpress]
PASS=hardone

[xptest]
XPRESSDIR=/opt/xpressmp/testing
XPAUTH_PATH=/opt/xpressmp/lic
MOSEL_RESTR=NoWriteNoDBNoExecWDOnly
MOSEL_CWD?=${XPRESSDIR}/workdir

Assuming the server using this configuration is running on the machine mypc, the following statements will create two instances on this machine, one for each of the defined execution environments:

 r1:=connect(m1, "xsrv:mypc/xpress/hardone")
 r2:=connect(m2, "xsrv:mypc/xptest")

Since MOSEL_CWD has been initialised with the ?= symbol, the remote host can change its working directory. For instance:

 r2:=connect(m2, "xsrv:mypc/xptest|MOSEL_CWD=/tmp")

While the server is running it is possible to request a reload of its configuration: this procedure consists in reading again the configuration file(s) in order to update the definition of the contexts. During this operation only context specific definitions are processed (all global definitions are silently ignored). Under Windows configuration change can only be requested on a running service using the reload command:

xprmsrv -service reload

On a Unix system configuration change is performed after reception of a signal USR1. For instance if PID is the process ID of a running xprmsrv server:

kill -USR1 PID

The configuration update can only be executed when the server is not monitoring any Mosel instance. If a request cannot be processed immediately it is delayed until the server is idle. Moreover if an error is detected while reading the configuration an error is reported but the server continues running with its current settings.

Access control list

The environment variable XPRMSRV_ACCESS may be defined in each context of the configuration file. This variable defines which hosts are allowed to connect to the server or use a particular context. The restriction applies to the server itself when the variable is defined in the global context and as a supplementary restriction when it is included in any other context (i.e. a host cannot be allowed in a context if it is rejected by the global context).

The value of the variable must consist in a list of hosts and subnetworks separated by spaces. Each entry of this list can optionally be preceded by the + sign (for accepting the host; this is the default if no policy is specified) or - sign (to reject connection). Order of the list members is important: when checking authorisation for a given host the list is processed from left to right. The first matching entry will decide whether access is allowed or denied. A given host will be rejected if no matching entry can be found.

A host is identified by its name (e.g. myhost) or its IP address (e.g. 192.168.1.1). A subnetwork is defined by a routing prefix that can be expressed as a partial address (e.g. 192.168.1); or using the CIDR notation - the first address of the network followed by the bit-length of the prefix, separated by a slash "/" character (e.g. 192.168.1.0/24). The subnet mask may also be used instead of the bit-length which is a quad-dotted decimal representation like an address (e.g. 192.168.1.0/255.255.255.0). The special identifier ALL is replaced by the subnetwork definition 0.0.0.0/0 (any host) and the identifier SELF is replaced by the hostname of the server.

In the first example below, host uranus is rejected and subnetwork 192.168.1.0/24 is allowed to connect. Note that uranus will be rejected even if it is part of the autorised subnetwork because its reference appears first in the list. In the second example, all hosts are allowed except 2 subnetworks (192.168.1.0/24 and 192.168.2.0/24):

 XPRMSRV_ACCESS=-uranus 192.168.1
 XPRMSRV_ACCESS=-192.168.1.0/255.255.255.0 -192.168.2.0/24 +ALL

All defined control lists are preprocessed just after the configuration file has been read in order to resolve host names and check for syntax errors. Unresolved host names are ignored (although a warning is displayed in such a case) but a syntax error on a control list will cause the server to abort its processing.