Programming Techniques for User Functions

This section is principally concerned with the programming of large or complicated user functions, perhaps taking a potentially large number of input values and calculating a large number of results. However, some of the issues raised are also applicable to simpler functions.

The first part describes in more detail some of the possible arguments to the function. The remainder of the section looks at function instances, function objects and direct calls to user functions.

FunctionInfo

The array FunctionInfo is primarily used to provide the sizes of the arrays used as arguments to the functions, and to indicate how many derivatives are required.

In particular:
FunctionInfo[0] holds the number of input values supplied
FunctionInfo[1] holds the number of return values required
FunctionInfo[2] holds the number of sets of derivatives to be calculated.

In addition, it contains problem-specific information which allows the user function to access problem data such as control parameters and attributes, matrix elements and solution values. It also holds information about function objects and function instances.

See XSLPgetfuncobject for a more detailed description.

InputNames

The function may have the potential to take a very large number of input values but in practice, within a particular problem, not all of them are used. For example, a function representing the model of a distillation unit may have input values relating to external air temperature and pressure which are not known or which cannot be controlled by the optimization. In general, therefore, these will take default values except for very specialized studies.

Although it would be possible to require that every function call had every input value specified, it would be wasteful in processing time to do so. In such cases, it is worth considering using named input variables, so that only those which are not at default values are included. The user function then picks up the input values by name, and assigns default values to the remainder. InputNames is an array of character strings which contains the names of the input variables. The order of the input values is then determined by the order in InputNames. This may be different for each instance of the function (that is, for each different formula in which it appears) and so it is necessary for the function to check the order of the input values. If function instances are used, then it may be necessary to check only when the function instance is called for the first time, provided that the order can be stored for future calls to the same instance.

ReturnNames

The function may have the potential to calculate a very large number of results but in practice, within a particular problem, not all of them are used. For example, a detailed model of a process unit might calculate yields and qualities of streams, but also internal flow rates and catalyst usage which are not required for a basic planning problem (although they are very important for detailed engineering investigations).

Although it would be possible to calculate every value and pass it back to the calling function every time, it could be wasteful in processing time to do so. In such cases, it is worth considering using named return values, so that only those which are actually required are included. The user function then identifies which values are required and only passes those values to its caller (possibly, therefore, omitting some of the calculations in the process).

ReturnNames is an array of character strings which contains the names of the return variables. The order of the values is then determined by the order in ReturnNames. This order may be different for different instances of the function (that is, for different formulae in which it is used). If the function does use named return values, it must check the order. If function instances are used for the function, then it may be necessary to check the order only when the function instance is called for the first time, if the order can be stored for subsequent use.

If the user function is being called by Xpress NonLinear to calculate values during matrix generation or optimization, the list of return values required is created dynamically and the names will appear in the order in which they are first encountered. It is possible, therefore, that changes in the structure of a problem may change the order in which the names appear.

Deltas

The Deltas array has the same dimension as InputValues and is used to indicate which of the input variables should be used to calculate derivatives. If Deltas[i] is zero, then no derivative should be returned for input variable i. If Deltas[i] is nonzero, then a derivative is required for input variable i. The value of Deltas[i] can be used as a suggested perturbation for numerical differentiation (a negative sign indicates that if a one-sided derivative is calculated, then a backward one is preferred). If derivatives are calculated analytically, or without requiring a specific perturbation, then Deltas can be interpreted simply as an array of flags indicating which derivatives are required.

Return values and ReturnArray

The ReturnArray array is provided for those user functions which return more than one value, either because they do calculate more than one result, or because they also calculate derivatives. The function must either return the address of an array which holds the values, or pass the values to the calling program through the ReturnArray array.

The total number of values returned depends on whether derivatives are being calculated. The FunctionInfo array holds details of the number of input values supplied, the number of return values required (nRet) and the number of sets derivatives required (nDeriv). The total number of values (and hence the minimum size of the array) is nRet*(nDeriv+1). Xpress NonLinear guarantees that ReturnArray will be large enough to hold the total number of values requested.

A function which calculates and returns a single value can use the ReturnArray array provided that the declarations of the function in Xpress NonLinear and in the native language both include the appropriate argument definition.

functions which use the ReturnArray array must also return a status code as their return value. Zero is the normal return value. A value of 1 or greater is an error code which will cause any formula evaluation to stop and will normally interrupt any optimization or other procedure. A value of -1 asks Xpress NonLinear to estimate the function values from the last calculation of the values and partial derivatives. This will produce an error if there is no such set of values.

Returning Derivatives

A multi-valued function which does not calculate its own derivatives will return its results as a one-dimensional array.

As already described, when derivatives are calculated as well, the order is changed, so that the required derivatives follow the value for each result. That is, the order becomes:
A,

∂A

∂X1

,

∂A

∂X2

, ...

∂A

∂Xn

, B,

∂B

∂X1

,

∂B

∂X2

, ...

∂B

∂Xn

, ...

∂Z

∂Xn

where A, B, Z are the return values, and X₁, X₂, X_n, are the input (independent) variables (in order) for which derivatives have been requested.

Not all calls to a user function necessarily require derivatives to be calculated. Check FunctionInfo for the number of derivatives required (it will be zero if only a value calculation is needed), and Deltas for the indications as to which independent variables are required to produce derivatives. Xpress NonLinear will not ask for, nor will it expect to receive, derivatives for function arguments which are actually constant in a particular problem. A function which provides uncalled-for derivatives will cause errors in subsequent calculations and may cause other unexpected side-effects if it stores values outside the expected boundaries of the return array.

Function Instances

Xpress NonLinear defines an instance of a user function to be a unique combination of function and arguments. For functions which return an array of values, the specific return argument is ignored when determining instances. Thus, given the following formulae:
 f(x) + f(y) + g(x,y : 1)
 f(y)*f(x)*g(x,y : 2)
 f(z)
the following instances are created:
f(x)
f(y)
f(z)
g(x,y)
(A function reference of the form g(x,y:n) means that g is a multi-valued function of x and y, and we want the n^th return value.)

Xpress NonLinear regards as complicated any user function which returns more than one value, which uses input or return names, or which calculates its own derivatives. All complicated functions give rise to function instances, so that each function is called only once for each distinct combination of arguments.

Functions which are not regarded as complicated are normally called each time a value is required. A function of this type can still be made to generate instances by defining its ExeType as creating instances (set bit 9 when using the normal library functions, or use the "I" suffix when using file-based input through XSLPreadprob or when using SLPDATA in Mosel).

Note that conditional re-evaluation of the function is only possible if it generates function instances.

Using function instances can improve the performance of a problem, because the function is called only once for each combination of arguments, and is not re-evaluated if the values have not changed significantly. If the function is computationally intensive, the improvement can be significant.

There are reasons for not wanting to use function instances:

• When the function is fast. It may be as fast to recalculate the value as to work out if evaluation is required.
• When the function is discontinuous. Small changes are estimated by using derivatives. These behave badly across a discontinuity and so it is usually better to evaluate the derivative of a formula by using the whole formula, rather than to calculate it from estimates of the derivatives of each term.
• Function instances do use more memory. Each instance holds a full copy of the last input and output values, and a full set of first-order derivatives. However, the only time when function instances are optional is when there is only one return value, so the extra space is not normally significant.

Function Objects

Normally, a user function is effectively a free-standing program: that is, it requires only its argument list in order to calculate its result(s). However, there may be circumstances where a user function requires access to additional data, as in the following examples:

1. The function is actually a simulator which needs access to specific (named) external files. In this case, the function needs to access a list of file names (or file handles if the files have been opened externally).
2. The function uses named input or output values and, having established the order once, needs to save the order for future calls. In this case, the function needs to use an array which is external to the function, so that it is not destroyed when the function exits.
3. The function returns an array of results and so the array must remain accessible after the function has returned. In this case, the function needs to use an array which is external to the function, so that it is not destroyed when the function exits.
4. The function determines whether it needs to re-evaluate its results when the values of the arguments have not changed significantly, and so it needs to keep a copy of the previous input and output values. In this case, the function needs to use an array which is external to the function, so that it is not destroyed when the function exits.
5. The function has to perform an initialization the first time it is called. In this case, the function needs to keep a reference to indicate whether it has been called before. It may be that a single initialization is required for the function, or it may be that it has to be initialized separately for each instance.

There is a potential difference between examples (3) and (4) above. In example (3), the array is needed only because Xpress NonLinear will pick up the values when the function has returned and so the array still needs to exist. However, once the values have been obtained, the array is no longer required, and so the next call to the same function can use the same array. In example (4), the argument values are really required for each instance of the function: for example, if f(x) and f(y) are both used in formulae, where f() is a user function and x and y are distinct variables, then it only makes sense to compare input argument values for f(x) (that is, the value of x) against the previous value for x; it does not make sense to compare against the previous value for y. In this case, a separate array is needed for each function instance.

Xpress NonLinear provides three levels of user function object. These are:

• The Global Function Object. There is only one of these for each problem, which is accessible to all user functions.
• The User Function Object. There is one of these for each defined user function.
• The Instance Function Object. There is one of these for each instance of a function.

The library functions XSLPsetuserfuncobject, XSLPchguserfuncobject and XSLPgetuserfuncobject can be used to set, change and retrieve the values from a program or function which has access to the Xpress NonLinear problem pointer.
The library functions XSLPsetfuncobject, XSLPchgfuncobject and XSLPgetfuncobject can be used by a user function to set, change or retrieve the Global Function Object, the User Function Object for the function, and the Instance Function Object for the instance of the function.

XSLPgetfuncobject can also be used to obtain the Xpress NonLinear and Xpress Optimizer problem pointers. These can then be used to obtain any problem data, or to execute any allowable library function from within the user function.

Example:

A function which uses input or return names is regarded as a complicated function, and will therefore generate function instances. All the calls for a particular instance have the same set of inputs in the same order. It is therefore necessary to work out the order of the names only once, as long as the information can be retained for subsequent use. Because each instance may have a different order, as well as different variables, for its inputs, the information should be retained separately for each instance.

The following example shows the use of the Instance Function Object to retain the order of input values

NOTE
 1  typedef struct tagMyStruct {
       int InputFromArg[5];
     } MyStruct;

     static char *MyNames[] = {"SUL", "RVP", "ARO", "OLE", "BEN"};
     static double Defaults[] = {0, 8, 4, 1, 0.5};

     double XPRS_CC MyUserFunc(double *InputValues, int *FunctionInfo,
                               char *InputNames) {
       MyStruct *InstanceObject;
       void *Object;
       char *NextName;

       int i, iArg, nArg;
       double Inputs[5], Results[10];

 2    XSLPgetfuncobject(FunctionInfo,XSLP_INSTANCEFUNCOBJECT,&Object);
 3    if (Object == NULL) {
         Object = calloc(1,sizeof(MyStruct));
 4       XSLPsetfuncobject(FunctionInfo,XSLP_INSTANCEFUNCOBJECT,Object);
         InstanceObject = (MyStruct *) Object;
         NextName = InputNames;
         nArg = FunctionInfo[0];
 5       for (iArg = 1;iArg<=nArg;iArg++) {
           for (i=0;i<5;i++) {
             if (strcmp(NextName,MyNames[i])) continue;
             InstanceObject->InputFromArg[i] = iArg;
             break;
           }
           NextName = &NextName[strlen(NextName)+1];
         }
       }
       InstanceObject = (MyStruct *) Object;
 6     if (InstanceObject == NULL) {
 7       XSLPgetfuncobject(FunctionInfo,XSLP_XSLPPROBLEM,&Object);
 8       XSLPsetfunctionerror(Prob);
         return(1);
       }
 9     for (i=0;i<5;i++) {
         iArg=InstanceObject->InputFromArg[i];
        if (iArg) Inputs[i] = InputValues[iArg-1];
        else Inputs[i] = Defaults[i];
       }
       MyCalc(Inputs, Results);
       .....
     }

Notes:

1. A structure for the instance function object is defined. This is a convenient way of starting, because it is easy to expand it if more information (such as results) needs to be retained.
2. XSLPgetfuncobject recovers the instance function object reference from the FunctionInfo data.
3. On the first call to the function, the object is NULL.
4. After the object has been created, its address is stored as the instance function object.
5. The names in InputNames are in a continuous sequence, each separated from the next by a null character. This section tests each name against the ordered list of internal names. When there is a match, the correspondence is stored in the InputFromArg array. A more sophisticated version might fault erroneous or duplicate input names.
6. If InstanceObject is NULL then the initialization must have failed in some way. Depending on the circumstances, the user function may be able to proceed, or it may have to terminate in error. We will assume that it has to terminate.
7. XSLPgetfuncobject recovers the Xpress NonLinear problem.
8. XSLPsetfunctionerror sets the error flag for the problem which will stop the optimization.
9. If the initialization was successful, the correspondence in InputFromArg is now available on each call to the function, because on subsequent calls, Object is not NULL and contains the address of the object for this particular instance.

If there are different instances for this function, or if several problems are in use simultaneously, each distinct call to the function will have its own object.

A similar method can be used to set up and retain a correspondence between the calculated results and those requested by the calling program.

The User Function Object can be used in a similar way, but there is only one such object for each function (not for each instance), so it is only appropriate for saving information which does not have to be kept separate at an instance level. One particular use for the User Function Object is to provide a return array which is not destroyed after the user function returns (an alternative is to use the ReturnArray argument to the function).

Note that one or more arrays may be allocated dynamically by each function using this type of approach. It may be necessary to release the memory if the problem is destroyed before the main program terminates. There is no built-in mechanism for this, because Xpress NonLinear cannot know how the objects are structured. However, there is a specific callback (XSLPsetcbdestroy) which is called when a problem is about to be destroyed. As a simple example, if each non-null object is the address of an allocated array, and there are no other arrays that need to be freed, the following code fragment will free the memory:

int i, n;
void *Object;
XSLPgetintattrib(Prob, XSLP_UFINSTANCES, &n);
for (i=1;i<=n;i++) {
  XSLPgetuserfuncobject(Prob, -i, &Object);
  if (Object) free(Object);
  XSLPsetuserfuncobject(Prob, -i, NULL);
}

When used in the "destroy" callback, it is not necessary to set the instance function object to NULL. However, if an object is being freed at some other time, then it should be reset to NULL so that any subsequent call that requires it will not try to use an unallocated area of memory.

Calling user functions

A user function written in a particular language (such as C) can be called directly from another function written in the same language, using the normal calling mechanism. All that is required is for the calling routine to provide the arguments in the form expected by the user function.

Xpress NonLinear provides a set of functions for calling between different languages so that, for example, it is possible for a program written in Mosel to call a user function written in C. Not all combinations of language are possible. The following table shows which are available:

User function	Calling program
	Mosel	C/Fortran	VBA (Excel)
Mosel	1	3	3	3
C/Fortran	1	1	1	1
VBA (Excel macro)	2	2	2	2
Excel spreadsheet	2	2	2	2
COM	2	2	2	2

1: User functions available with full functionality
2: User functions available, but with reduced functionality
3: User functions available if Mosel model is executed from main program
X: User functions not available.

In general, those user functions which are called using OLE automation (Excel macro, Excel spreadsheet and COM) do not have the full functionality of user functions as described below, because the calling mechanism works with a copy of the data from the calling program rather than the original. Mosel user functions can only be called from problems which are created in the same Mosel model; however, because Mosel can itself be called from another program, Mosel functions may still be accessible to programs written in other languages.

XSLPcalluserfunc provides the mechanism for calling user functions. The user function is declared to Xpress NonLinear as described earlier, so that its location, linkage and arguments are defined. In this section, we shall use three example user functions, defined in Extended MPS format as follows:

UF MyRealFunc ( DOUBLE , INTEGER ) .....
UF MyArrayFunc ( DOUBLE , INTEGER ) DLLM .....
UF MyRetArrayFunc ( DOUBLE , INTEGER , , , , DOUBLE ) .....

These all take as arguments an array of input values and the FunctionInfo array. MyArrayFunc is declared as multi-valued (using the suffix M on the linkage). MyRetArrayFunc returns its results in ReturnArray; thus usually means that it is multi-valued, or calculates its own derivatives.

double Values[100];
double ReturnArray[200];
integer FunctionInfo[XSLP_FUNCINFOSIZE];
integer RealFunc, ArrayFunc, RetArrayFunc;
double ReturnValue;

The calling program has to provide its own arrays for the function calls, which must be sufficient to hold the largest amount of data required for any call. In particular, ReturnArray may need to allow space for derivatives.
FunctionInfo should always be declared as shown.

XSLPgetindex(Prob, XSLP_USERFUNCNAMES, "MyRealFunc", RealFunc);
XSLPgetindex(Prob, XSLP_USERFUNCNAMES, "MyArrayFunc", ArrayFunc);
XSLPgetindex(Prob, XSLP_USERFUNCNAMES, "MyRetArrayFunc", RetArrayFunc);

As XSLPcalluserfunc needs the function number, we get this for each function by using the function XSLPgetindex. If you are not sure of the upper- or lower-case, then use XSLP_USERFUNCNAMESNOCASE instead. If the functions are set up using library functions, the function indices can be obtained at that time.

               ...
/*... set up Values array .....*/
               ...
XSLPsetuserfuncinfo(Prob,ArgInfo,1,n,1,0,0,0);

The input data for the function call is set up. The contents of the input array Values obviously depend on the nature of the function being called, so we do not include them here. The function information array FunctionInfo must be set up. XSLPsetuserfuncinfo will fill in the array with the items shown. The arguments after FunctionInfo are:

• CallerFlag. This is always zero when the function is called directly by Xpress NonLinear, and so if set nonzero it indicates a call from the user application; its value can be used for any purpose in the calling and called functions.
• The number of input variables: this is the number of elements used in the input array Values.
• The number of return values required for each calculation.
• The number of sets of partial derivatives required.
• The number of items in the array of input argument names.
• The number of items in the array of return value names.

This structure actually allows more flexibility than is used when the function is called directly by Xpress NonLinear because, for example, there is no requirement for the number of input names to be the same as the number of input arguments. However, such usage is beyond the scope of this manual.

ReturnValue = XSLPcalluserfunc(Prob,RealFunc,Values,FunctionInfo,
                               NULL,NULL,NULL,NULL);

XSLPcalluserfunc calls the function using the appropriate linkage and calling mechanism. The arguments to XSLPcalluserfunc are:

• The Xpress NonLinear problem.
• The index of the function being called.
• Six arguments corresponding to the six possible arguments to a user function. If the user function requires an argument, then the corresponding argument in the call must contain the appropriate data in the correct format. If the user function does not require an argument, then it can be NULL in the call (in any case, it will be omitted from the call). The FunctionInfo argument is always required for function calls using XSLPcalluserfunc.

ReturnValue will contain the single value returned by the user function.

ReturnValue = XSLPcalluserfunc(Prob,ArrayFunc,Values,FunctionInfo,
                               NULL,NULL,NULL,NULL);

This time, ReturnValue will contain the first value in the array of results returned by the function. This is because the function is multi-valued and there is nowhere for the other values to go.

Multi-valued functions must be called using the ReturnArray argument. Even if the user function itself does not recognize it, XSLPcalluserfunc does, and will transfer the results into it.

ReturnValue = XSLPcalluserfunc(Prob,ArrayFunc,Values,FunctionInfo,
                               NULL,NULL,NULL,ReturnArray);

The difference between this call and the previous one is the presence of the additional argument ReturnArray. This will be used to hold all the values returned by the function. The function will behave in exactly the same way as in the previous example, and ReturnValue will also be the same, but ReturnArray will be filled in with the values from the function.

ReturnValue = XSLPcalluserfunc(Prob,RetArrayFunc,Values,FunctionInfo,
                               NULL,NULL,NULL,ReturnArray);

As MyRetArrayFunc is defined as returning its results in an array, the ReturnArray argument is a required argument for the function anyway. In this case, ReturnValue is the value returned by the function, which indicates success (zero), failure (1) or not calculated (-1).