Generalized file handling

The notion of (data) file encountered repeatedly in this user guide seems to imply a physical file. However, Mosel language statements (such as initializations from / to, fopen and fclose, exportprob) and the Mosel library functions (e.g., XPRMcompmod, XPRMloadmod, or XPRMrunmod) actually work with a much more general definition of a file, including (but not limited to)

• a physical file (text or binary)
• a block of memory
• a file descriptor provided by the operating system
• a function (callback)
• a database

The type of the file is indicated by adding to its name the name of the I/O driver that is to be used to access it. In Section Reading data from spreadsheets and databases we have used mmodbc.odbc:blend.mdb to access an MS Access database via the ODBC driver provided by the module mmodbc. If we want to work with a file held in memory we may write, for instance, mem:filename. The default driver (no driver prefix) is the standard Mosel file handling.

More generally, an extended file name has the form driver_name:file_name or module_name.driver_name:file_name if the driver is provided by the module module_name. The structure of the file_name part of the extended file name is specific to the driver, it may also consist of yet another extended file name (e.g. zlib.gzip:tmp:myfile.txt).

Displaying the available I/O drivers

The Mosel core drivers can be displayed from the command line with the following command (the listing will also include any drivers that are provided by currently loaded modules):

mosel exam -i

The drivers provided by modules are displayed by the exam command for the corresponding module (in this example: mmodbc)

mosel exam -i mmodbc

Library drivers (in particular the Java module mmjava that is embedded in the Mosel core and also the mmdotnet module on Windows platforms) can be displayed with the help of the corresponding program mmdispdso.[c|cs|java] in the subdirectory examples/mosel/Library of the Xpress distribution. The command for running the Java version might look as follows (please refer to the provided makefile):

java -classpath $XPRESSDIR/xprm.jar:. mmdispdso mmjava

List of I/O drivers

The standard distribution of Mosel defines the following I/O drivers (tee, null, bin, and tmp are documented in the 'Mosel Language Reference Manual'; the drivers sysfd, mem, cb, and raw that mainly serve when interfacing Mosel with a host application are documented in the `Mosel Libraries Reference Manual'):

bin

Write (and read) data files in a platform independent binary format. The bin driver can only be used for initializations blocks as a replacement of the default driver. Files in bin format are generally smaller than the ASCII equivalent and preserve accuracy of floating point numbers.
Example: the following outputs data to a text file in binary format

initializations to "bin:results.txt"

Another likely use of bin is in combination with mem or shmem for exchanging data in memory (see Section Exchanging data between models):

initializations to "bin:shmem:results"

cb

Use a (callback) function as a file ( e.g., for reading and writing dynamically sized data in initializations blocks, see the examples in Section Dynamic data, or to write your own output or error stream handling functions when working with the Mosel libraries, see Section Redirecting the Mosel output for an example).

mem

Use memory instead of physical files for reading or writing data ( e.g., for exchanging data between a model and the calling C application as shown in Section Exchanging data between an application and a model or for compiling/loading a model to/from memory when working with the Mosel libraries).
Example: the following lines will compile the Mosel model burglar2.mos to memory and then load it from memory (full example in file ugcompmem.c).

XPRMmodel mod;
char bimfile[2000];              /* Buffer to store BIM file */
char bimfile_name[64];           /* File name of BIM file */

XPRMinit())                      /* Initialize Mosel */

/* Prepare file name for compilation using 'mem' driver: */
/*   "mem:base_address/size[/actual_size_of_pointer]"    */
sprintf(bimfile_name, "mem:%p/%d", bimfile, (int)sizeof(bimfile));

                                 /* Compile model file to memory */
XPRMcompmod(NULL, "burglar2.mos", bimfile_name, "Knapsack example"))

                                 /* Load a BIM file from memory */
mod = XPRMloadmod(bimfile_name, NULL);

null

Disable a stream.
Example: adding the line

fopen("null:", F_OUTPUT)

in a Mosel model will disable all subsequent output by this model (until the output stream is closed or a new output stream is opened).

raw

Implementation of the initializations block in binary mode, typically used in combination with mem for data exchange with a C host application (see Section Exchanging data between an application and a model).

sysfd

Working with operating system file descriptors (for instance, file descriptors returned by the C function open).
Example: in a C program, the line

XPRMsetdefstream(NULL, XPRM_F_ERROR, "sysfd:1");

will redirect the Mosel error stream to the default output stream.

tee

Output into up to 6 files simultaneously ( e.g., to display a log on screen and write it to a file at the same time).
Example: adding the line

fopen("tee:result.txt&tmp:log.txt&", F_OUTPUT)

in a Mosel model will redirect all subsequent model output to the files result.txt and tmp:log.txt, and at the same time display the output on the default output (screen), the latter is denoted by the & sign at the end of the filename string. The output to both locations is terminated by the statement

fclose(F_OUTPUT)

after which output will again only go to default output.

tmp

Extension to the default driver that locates the specified file in the temporary directory used by Mosel.
Example: adding the line

fopen("tmp:log.txt", F_OUTPUT+F_APPEND)

redirects all subsequent output from a models to the file log.txt that is located in Mosel's temporary directory. It is equivalent to

fopen(getparam("TMPDIR") + "/log.txt", F_OUTPUT+F_APPEND)

Some modules, listed below in alphabetical order, define additional I/O drivers. The drivers are documented with the corresponding module in the `Mosel Language Reference Manual':

• mmdotnet

  dotnet

  Use a C# stream or object in place of a file in Mosel.
  Example: The following C# program extract uses the dotnet driver to send data in standard initializations text format via a stream from the C# host program to a model (the model file burglar13.mos is the same as in Section Dynamic data, it uses the parameter DATAFILE as the filename for an initializations from block that expects to read data with the label 'DATA'.)

  // String containing initialization data for the model
  const string BurglarDat =
    "DATA: [(\"camera\") [15 2] (\"necklace\") [100 20] " +
    "(\"vase\") [90 20] (\"picture\") [60 30] (\"tv\") [40 40] " +
    "(\"video\") [15 30] (\"chest\") [10 60] (\"brick\") [1 10] ]\n";
  
  static void Main(string[] args) {
    // Initialize Mosel
    XPRM mosel = XPRM.Init();
  
    // Compile and load the Mosel model
    XPRMModel model = mosel.CompileAndLoad("burglar13.mos");
  
    // Bind a stream based on the BurglarDat data to the name 'BurglarIni'
    // where the model will expect to find its initialization data
    model.Bind("BurglarIni", new StringReader(BurglarDat));
  
    // Pass data location as a parameter to the model
    model.SetExecParam("DATAFILE","dotnet:BurglarIni");
  
    // Run the model
    model.Run();
  }

  dotnetraw

  Exchange of data between a Mosel model and the C# application running the model using C# array structures; C# version of raw.
  Example: Send the data held in the two arrays vdata and wdata to a Mosel and retrieve solution data into the array solution. We use the option noindex with the dotnetraw driver to indicate that all data are saved in dense format ( i.e. as arrays containing just the data values, without any information about the indices).

  // Arrays containing initialization data for the model
  static double[] vdata = new double[] {15,100,90,60,40,15,10, 1};
  static double[] wdata = new double[] { 2, 20,20,30,40,30,60,10};
  
  // Main entry point for the application
  static void Main(string[] args) {
    // Initialize Mosel
    XPRM mosel = XPRM.Init();
  
    // Compile and load the Mosel model
    XPRMModel model = mosel.CompileAndLoad("burglar8d.mos");
  
    // Associate the .NET objects with names in Mosel
    model.Bind("vdat", vdata);
    model.Bind("wdat", wdata);
  
    // Create a new array for solution data and bind it to the name 'sol'
    double[] solution = new double[8];
    mosel.Bind("sol", solution);
  
    // Pass data location as a parameter to the model
    model.ExecParams =
      "VDATA='noindex,vdat',WDATA='noindex,wdat',SOL='noindex,sol'";
  
    // Run the model
    model.Run();
  
    // Print the solution
    Console.WriteLine("Objective value: {0}", model.ObjectiveValue);
    for (int i=0;i<8;i++)
      Console.Write(" take({0}): {1}", (i+1), solution[i]);
    Console.WriteLine();
  }

  The model file burglar8d.mos uses the driver name as the file name in the initializations sections:

   initializations from 'dotnetraw:'
    VALUE as VDATA  WEIGHT as WDATA
   end-initializations
  ...
  
   initializations to 'dotnetraw:'
    soltake as SOL
   end-initializations

• mmetc

  diskdata

  Access data in text files in diskdata format (see Sections Data input with diskdata and Data output with diskdata).

• mmhttp

  url

  Access files that are stored on an HTTP enabled file serverfor reading, writing, or deletion (via fdelete).
  Example 1: the following command downloads and executes the Mosel BIM file mymodel.bim that is stored on the web server myserver:

  mosel run mmhttp.url:http://myserver/mymodel.bim

  Example 2: the following lines of Mosel code save data held in the model object results to an XML format file on the server myserver that needs to be able to accept HTTP PUT requests.

  uses "mmxml"
  declarations
    results: xmldoc
  end-declarations
  save(results, "mmhttp.url:http://myserver/myresults.xml")  

• mmjava

  java

  Use a Java stream or a ByteBuffer in place of a file in Mosel ( e.g. for redirecting default Mosel streams to Java objects, see the example in Section Redirecting the Mosel output).
  Example 1: in a Java program, the line

  mosel.setDefaultStream(XPRM.F_ERROR, "java:java.lang.System.out");

  (where mosel is an object of class XPRM) will redirect the Mosel error stream to the default output stream of Java.
  Example 2: the following lines will compile the Mosel model burglar2.mos to memory and then load it from memory (full example in the file ugcompmem.java).

  XPRM mosel;
  XPRMModel mod;
  ByteBuffer bimfile;                  // Buffer to store BIM file
  
  mosel = new XPRM();                  // Initialize Mosel
  
                              // Prepare file names for compilation:
  bimfile=ByteBuffer.allocateDirect(2048);  // Create 2K byte buffer
  mosel.bind("mybim", bimfile);        // Associate Java obj. with a
                                       // Mosel name
  
                                       // Compile model to memory
  mosel.compile("", "burglar2.mos", "java:mybim", "");
  
  bimfile.limit(bimfile.position());   // Mark end of data in buffer
  bimfile.rewind();                    // Back to the beginning
  
  mod=mosel.loadModel("java:mybim");   // Load BIM file from memory
  mosel.unbind("mybim");               // Release memory
  bimfile=null;

  jraw

  Exchange of data between a Mosel model and the Java application running the model; Java version of raw. See Section Exchanging data between an application and a model for examples.

• mmjobs

  shmem

  Use shared memory instead of physical files for reading or writing data ( e.g., for exchanging data between several models executed concurrently—one model writing, several models reading—as shown in Section Exchanging data between models, or for compiling/loading a model to/from memory from within another model, see Section Compiling to memory).

  mempipe

  Use memory pipes for reading or writing data ( e.g., for exchanging data between several models executed concurrently—one model reading, several models writing; see Section 'Dantzig-Wolfe decomposition' of the whitepaper 'Multiple models and parallel solving with Mosel' for an example).

  rcmd

  Starts the specified command in a new process and connects its standard input and output streams to the calling Mosel instance.
  Example: The following line starts a Mosel instance on a remote computer with the name some_other_machine connecting to it via rsh. The command mosel -r starts Mosel in remote mode.

  rcmd:rsh some_other_machine mosel -r

  rmt

  Can be used with any routine expecting a physical file for accessing files on remote instances. A particular instance can be specified by prefixing the file name by its node number enclosed in square brackets. See the distributed computing versions of the models in the whitepaper Multiple models and parallel solving with Mosel for further examples.
  Example: the following loads the BIM file mymodel.bim that is located at the parent node (-1) of the current instance into the model myModel of the Mosel instance myInst.

  load(myInst, myModel, "rmt:[-1]mymodel.bim") 

  The rmt driver can be combined with cb, sysfd, tmp or java, such as

  fopen("rmt:tmp:log.txt", F_OUTPUT)

  xsrv

  Connects to the specified host running the Mosel Remote Launcher xprmsrv. Optionally, the port number, a context name, a password, and environment variable settings can be specified.
  Example: the following starts a new Mosel instance (possibly using a different Xpress version) on the current machine, redefining Mosel's current working directory and the environment variable MYDATA.

  xsrv:localhost/myxpress|MOSEL_CWD=C:\workdir|MYDATA=${MOSEL_CWD}\data 

  xssh

  Secure version of the xsrv driver to connect to the specified host running the Mosel Remote Launcher xprmsrv through a secure SSH tunnel.

• mmoci

  oci

  Access an Oracle database for reading and writing in initializations blocks (see the whitepaper Using ODBC and other database interfaces with Mosel for further examples).
  Example: the OCI driver is used with a connection string that contains the database name, user name and password in the following format

  initializations from "mmoci.oci:myusername/mypassword@dbname

• mmodbc

  odbc

  Access data in external data sources via an ODBC connection (see Section Reading data from spreadsheets and databases for an example).

• mmsheet

  csv

  Access spreadsheets in CSV format. In addition to the standard options supported by other Mosel spreadsheet drivers (such as grow for dynamic sizing of output ranges and noindex for dense data format), this driver can be configured with field and decimal separators and also with the values representing 'true' and 'false'.
  Example: the following Mosel code for reading data from the file mydata.csv sets the separator sign to ';', a comma is used as decimal separator, and the Boolean values true and false are represented by 'y' and 'n' respectively:

  initializations from "mmsheet.csv:dsep=,;fsep=;;true=y;false=n;mydata.csv
    A as "[B2:D8]"
    B as "[E2:Z10](#3,#1)"       ! 3rd and 1st column from the range
  end-initializations

  Notice that with CVS format files there is no notion of range or field names and the cell positions need to be used to specify the data location.

  excel

  Access data in MS Excel spreadsheets directly (see the example in Section Excel spreadsheets).

  xls

  Access spreadsheets in Excel's XLS format (see Section Generic spreadsheet example for an example).

  xlsx

  Access spreadsheets in Excel's XLSX and XLSM formats (usage in analogy to the XLS example shown in Section Generic spreadsheet example).

• mmsystem

  pipe

  Open a pipe and start an external program which is used as input or output stream for a Mosel model.
  Example: the following will start gnuplot and draw a 3-dimensional sphere with data for the radius R and position (X,Y,Z) defined in the Mosel model:

  fopen("mmsystem.pipe:gnuplot -p", F_OUTPUT+F_LINBUF)
  writeln('set parametric')
  writeln('set urange [0:2*pi]')
  writeln('set vrange [0:2*pi]')
  writeln('set pm3d depthorder hidden3d 3')
  writeln("splot cos(u) * cos(v) *", R, "+", X,
          ", sin(u) * cos(v)*", R, "+", Y,
          ", sin(v)*", R, "+", Z, " notitle with pm3d")
  writeln("pause mouse keypress,close")

  text

  Use a (multiline) text as a file (see Section Text handling and regular expressions for further detail on the type 'text').
  Example: the following will compile the model source held in the text source_of_model to a BIM file in a temporary directory (file ugcompfrommem.mos):

   public declarations
    source_of_model=`SUBMODELSOURCE
   model Burglar
    uses 'mmxprs'
  
    declarations
     WTMAX = 102                    ! Maximum weight allowed
     ITEMS = 1..8                   ! Index range for items
     VALUE: array(ITEMS) of real    ! Value of items
     WEIGHT: array(ITEMS) of real   ! Weight of items
     take: array(ITEMS) of mpvar    ! 1 if we take item i; 0 otherwise
    end-declarations
  
    VALUE :: [15, 100, 90, 60, 40, 15, 10,  1]
    WEIGHT:: [ 2,  20, 20, 30, 40, 30, 60, 10]
  
   ! Objective: maximize total value
    MaxVal:= sum(i in ITEMS) VALUE(i)*take(i)
  
   ! Weight restriction
    sum(i in ITEMS) WEIGHT(i)*take(i) <= WTMAX
   ! All variables are 0/1
    forall(i in ITEMS) take(i) is_binary
  
    maximize(MaxVal)                ! Solve the problem
    writeln("Solution:\n Objective: ", getobjval)
   end-model
  SUBMODELSOURCE`
   end-declarations
  
  ! Compile the model from memory
   if compile("", "text:source_of_model", "tmp:burglar.bim")<>0 then
     exit(1)
   end-if

• zlib

  deflate

  Handles files compressed using the zlib compression format.
  Example: decompress the file myfile.gz to myfile:

  fcopy("zlib.deflate:myfile.gz", "myfile")

  gzip

  Handles files compressed using the gzip compression format.
  Example: the following statement creates the compressed file myfile.gz from myfile.txt:

  fcopy("myfile.txt", "zlib.gzip:myfile.gz")

The reader is referred to the whitepaper Generalized file handling in Mosel that is provided as a part of the Xpress documentation in the standard distribution and also on the Xpress website under `Product Documentation' for further explanation of this topic and a documented set of examples, including some user-written I/O drivers.