Using the DMP module

Topics covered in this chapter:

Accessing DMP environmental data

One of the simplest uses of the dmp module is to access information about the component instance that's executing your model. You can use getdmpcompid, getdmpsolid and getdmplifecycleenv to retrieve the DMP component ID, solution ID and lifecycle stage respectively. You can also use the isdmp function to check whether or not the model is being executed by a DMP component. For example:

if isdmp then
  writeln('Component ID:', getdmpcompid)
  writeln('Solution ID:', getdmpsolid)
  writeln('Lifecycle Stage:', getdmplifecycleenv)
else
  writeln('Not within DMP')
end-if

Making HTTP Requests

You can use the functions dmphttpget, dmphttppatch, dmphttppost, dmphttpput, dmphttpdel and dmphttphead to send HTTP requests to a DMP component, webservice or DMP Manager. These functions work much the same as the corresponding functions in mmhttp, except instead of passing a destination URL, you pass a destination dmpresource and path relative to this.

In event of an unexpected failure, the request can be retried a number of times until it succeeds, the delay between each retry growing progressively larger. When a dmphttp request returns an HTTP status code found in dmp_retry_error_codes, it will be retried, up to the number of times specified by dmp_max_retries-a value of zero disables the feature.

Calling a DMP Component

The most common use of the dmp module will be to send HTTP requests to another component in the same DMP solution. To do this, you first initialize a dmpresource value with the details of the target component by calling dmpinitcomp, then use the dmphttp functions to send HTTP requests in a similar way to using plain mmhttp (see section Making HTTP Requests for details). For example, in an Xpress Insight app, you can send a webservice request to the status endpoint of an Xpress Executor component in the same solution as follows:

declarations
  mycomp: dmpresource
end-declarations
dmpinitcomp(mycomp,'Xpress Executor')
if mycomp.status<>DMP_OK then
  writeln('Failed to find component due to error:', mycomp.lasterror)
  exit(1)
end-if

httpStatusCode := dmphttpget(mycomp,'/rest/runtime/status','result.json')
if httpStatusCode<>200 then
  writeln('Error returned by component: ',httpStatusCode)
  exit(1)
end-if

The path you pass to the the dmphttp function will be relative to the root of the component instance URL.

When executed from a DMP component, the dmp module will take care of obtaining appropriate authorization credentials and including them in the outgoing HTTP requests.

Finding DMP Components

In most cases, when you want to communicate with a DMP component in the same solution, you will either know the name of the component or you know there will only be a single component of that type, and you can pass those details directly to dmpinitcomp. But sometimes you may need to use more complex logic for finding the component; for these cases, getdmpcomponents can be used to return a list of all the components in the DMP solution, and you can iterate through them. For example, if you want to initialize an "Xpress Executor" component with a name that ends with "Simulation":

declarations
  comps: list of dmpcomponent
  mycomp: dmpresource
end-declarations

comps := getdmpcomponents
forall (c in comps) do
  if endswith(c.type,"Xpress Executor") and endswith(c.name,"Simulation") then
    dmpinitcomp(mycomp,c)
    if mycomp.status<>DMP_OK then
      writeln('Failed to find component due to error:', mycomp.lasterror)
      exit(1)
    end-if
    break
  end-if
end-do

Calling a DMP Webservice

You can also use the dmp module to send HTTP requests to a SERVERLESS_REST type webservice in the same solution. To do this, you first initialize a dmpresource value with the details of the webservice you want to talk to by calling dmpinitwebservice, then use the dmphttp functions to send HTTP requests in a similar way to using plain mmhttp (see section Making HTTP Requests for details). For example, you can send a request to a webservice called processLoanApps as follows:

declarations
  myservice: dmpresource
end-declarations
dmpinitwebservice(myservice,'processLoanApps')
if myservice.status<>DMP_OK then
  writeln('Failed to find service due to error:', myservice.lasterror)
  exit(1)
end-if

httpStatusCode := dmphttppost(myservice,'','request.json','result.json')
if httpStatusCode<>200 then
  writeln('Error returned by webservice: ',httpStatusCode)
  exit(1)
end-if

When calling a webservice, you will typically leave the path field as an empty string. In Xpress Insight 4 and Xpress Executor you may only make HTTP POST requests to this path and HTTP GET requests to the paths /api/swagger.json and /api/log; in Xpress Insight 5 there is no restriction on which paths may be accessed.

Calling DMP Manager

You can also use the dmp module to send HTTP requests to DMP Manager. Initialize a dmpresource value with the details of DMP Manager by calling dmpinitmanager, then use the dmphttp functions to send HTTP requests in a similar way to using plain mmhttp (see section Making HTTP Requests for details). For example, you can send a request to create a commit of the current solution as follows:

public declarations
  dmpmanager: dmpresource
  REQUEST_BODY = '{"label":"EXAMPLE","comment":"this is an example"}'
end-declarations
dmpinitmanager(dmpmanager)
if dmpmanager.status<>DMP_OK then
  writeln('Failed to find DMP Manager due to error:', dmpmanager.lasterror)
  exit(1)
end-if

httpStatusCode := dmphttppost(dmpmanager,'/rest/dmp/runtime/solutions/'+getdmpsolid+'/revisions',
  'text:REQUEST_BODY','result.json')
if httpStatusCode<>200 then
  writeln('Error returned by DMP Manager: ',httpStatusCode)
  exit(1)
end-if

In Xpress Insight 4 and Xpress Executor, the set of paths you can call on DMP Manager is restricted as follows:

• You may only make requests to paths pertaining to the solution containing the component executing the model.
• You may make HTTP POST and HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/revisions endpoint.
• You may make HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/revisions/REVISIONID endpoint.
• You may make HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/revisions/REVISIONID/descriptor endpoint.
• You may make HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/lifecycle endpoint.
• You may make HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/functions endpoint.
• You may make HTTP POST and HTTP GET requests to the /rest/dmp/runtime/solutions/SOLUTIONID/services endpoint.
• You may make HTTP DELETE requests to the /rest/dmp/runtime/solutions/SOLUTIONID/services/SERVICEID endpoints.
• You may make any requests to paths starting /rest/adm/
• You may not make any HTTP requests to any other paths.

In Xpress Insight 5 there are no restrictions on the paths that can be called, but the requests will be made using the solution token for a given lifecycle environment, which may alter the data returned from some endpoints. In this case, a different environment may be specified as an argument of dmpinitmanager.

When calling DMP Manager using the dmphttp functions, it is not necessary to start the path with '/com.fico.dmp.manager'.

Calling the Data Pipelines service

This functionality is not available in Insight 5.

You can use the dmp module to send HTTP requests to a FICO Data Pipelines service within the solution. To do this, you first initialize a dmpresource value by calling dmpinitdatapipelines. Then you can read the Data Pipelines service instance ID from the id attribute of the dmpresource, and use the dmphttp functions to send HTTP requests in a similar way to using plain mmhttp (see section Making HTTP Requests for details). For example, to retrieve the list all the Data Pipelines jobs:

declarations
  datapipelines: dmpresource
end-declarations
dmpinitdatapipelines(datapipelines)
if datapipelines.status<>DMP_OK then
  writeln('Failed to find Data Pipelines service due to error:', datapipelines.lasterror)
  exit(1)
end-if

httpStatusCode := dmphttpget(datapipelines,'/rest/service/dmp-system/'+
    datapipelines.id+'/api/v1/jobs','request.json')
if httpStatusCode<>200 then
  writeln('Error returned by Data Pipelines: ',httpStatusCode)
  exit(1)
end-if

When calling Data Pipelines, the query path is relative to the root of the Data Pipelines service provider.

The dmpresource.status value will be DMP_NOT_FOUND if a Data Pipelines service instance does not exist within the solution containing the component.

Use Outside the Cloud

When not being called from within a DMP Component, the dmp module does not give you additional functionality over that supplied by mmhttp. However, some users may find it useful to use the dmp module outside the cloud when writing libraries to work inside and outside the cloud.

You can use dmpiniturl to initialize a DMP resource with a set of user-supplied authorization headers, and then make HTTP requests to that resource using dmphttp functions. For example, if you have a local webservice with the URL http://localhost:8860/ that requires a cookie for authorization, you can call it as follows:

declarations
  myurl: dmpresource
  headers: dynamic array(set of string) of text
end-declarations
headers("Cookie") := "SESSIONID=8364825249"
dmpiniturl(myurl, "http://localhost:8860/", headers)
if myurl.status<>DMP_OK then
  writeln('ERROR: ',myurl.lasterror)
  exit(1)
end-if

httpStatusCode := dmphttpget(myurl,'/rest/runtime/status','result.json')
if httpStatusCode<>200 then
  writeln('Error returned from webservice request: ',httpStatusCode)
  exit(1)
end-if

If you need to specify dynamic authorization headers, then you can supply a function pointer instead, and the dmp module will automatically call your function when it requires fresh authorizaition headers. For example:

function getheaders(res:dmpresource, headers:array(set of string) of text): integer
  headers("X-Request-Timestamp") := string(datetime(SYS_NOW))
  headers("Cookie") := "SESSIONID=8364825249"
  returned := 300  ! use for the next 300 seconds, then call getheaders again
end-function

declarations
  myurl: dmpresource
end-declarations
dmpiniturl(myurl, "http://localhost:8860/", ->getheaders)
if myurl.status<>DMP_OK then
  writeln('ERROR: ',myurl.lasterror)
  exit(1)
end-if

You should report a failure within the 'getheaders' function by setting the error status on the dmpresource, e.g.:

function getheaders(res:dmpresource, headers:array(set of string) of text): integer
  if AUTHORIZATION_TOKEN='' then
    seterror(res, DMP_AUTH_ERROR, "AUTHORIZATION_TOKEN not set")
  else
    headers("Authorization") := "Bearer "+AUTHORIZATION_TOKEN
    returned := 1800  ! Use for 30 minutes
  end-if
end-function

Absolute Request Paths

Normally, the path provided to any of the dmphttp* functions is relative to the URL of the dmpresource. For example, if the dmpresource 'res' references a component with root URL https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei/, then the following code:

httpStatusCode := dmphttpget(res,'/rest/runtime/executions','result.json')

will initiate a request to https://app.dms.usw2.ficoanalyticcloud.com/ 16m0jgaeei/rest/runtime/executions. When you are constructing the request path in the model, you will usually be constructing the path relative to the resource. But in some cases, you may want to use a path that was returned by a previous call to the resource, and that path may be relative to the hostname rather than the resource's root. (e.g. /16m0jgaeei/rest/runtime/executions) To support this case, if you specify a path that starts with the path from the resource URL, this will be treated as an absolute path and resolved relative to the hostname.

For example:

! 'res' is a dmpresource with URL https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei

! Fetch https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei/rest/runtime/executions
httpStatusCode := dmphttpget(res,'/16m0jgaeei/rest/runtime/executions','result.json')

! Fetch https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei/rest/runtime/executions
httpStatusCode := dmphttpget(res,'/rest/runtime/executions','result.json')

! Fetch https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei/16m0j/rest/runtime/executions
httpStatusCode := dmphttpget(res,'/16moj/rest/runtime/executions','result.json')

This behaviour can be disabled by setting the abspath attribute of the dmpresource to 'false', e.g.:

! Fetch https://app.dms.usw2.ficoanalyticcloud.com/16m0jgaeei/16m0jgaeei/rest/runtime/executions
res.abspath := false
httpStatusCode := dmphttpget(res,'/16m0jgaeei/rest/runtime/executions','result.json')

Error handling

You should always check the status attribute of the dmpresource after every dmpinit call, to see if it succeeded or not. This status will be DMP_OK on success, DMP_NOT_FOUND if the resource you requested did not exist, DMP_ACCESS_DENIED if you don't have access to the resource you requested; any other values indicate internal errors. When the status is not DMP_OK, you can find a human-readable error message in the lasterror attribute. For example:

declarations
  myservice: dmpresource
end-declarations
dmpinitwebservice(myservice,'processLoanApps')
if myservice.status=DMP_OK then
  writeln('Service found OK')
elif myservice.status=DMP_NOT_FOUND then
  writeln('Service not found')
else then
  writeln('Failed to find service due to error:', myservice.lasterror)
end-if

httpStatusCode := dmphttppost(myservice,'','request.json','result.json')
if httpStatusCode<>200 then
  writeln('Error returned by webservice: ',httpStatusCode)
  exit(1)
end-if

Any previous error will always be removed from the dmpresource on the next call to a dmpinit* or dmphttp* subroutine. Code can also use the seterror and clearerror procedures to set and clear the error status of a dmpresource directly.

Limitations

Use within Submodels

Within an Insight 4 or Executor component, you can only use the dmp module from a submodel if the master model also uses the dmp module. In the event that your master does not need the dmp module, I recommend setting uses 'dmp' to the top of it and accessing one of the parameters to prevent the Mosel compiler removing it as an unreferenced module, e.g.

uses 'dmp'
setparam('dmp_verbose',false)

This is not necessary when the model is executing in Insight 5.

Use within Xpress Workbench

The 'dmp' module cannot be used if the model is running locally in an Xpress Workbench component on the cloud. However, the 'dmp' module will function correctly when Workbench is being used to debug an Xpress Insight scenario.

Differences between Insight 4 and 5

Xpress Insight 5 executes scenarios in a new 'trusted code execution' model; as a result, there are small differences between how the DMP module behaves in Insight 5 compared to how the same code behaved in Insight 4.

• Existing Insight 4 apps must be recompiled with at least Xpress 9.2.0 to use the DMP module functionality in Insight 5.
• Insight 4 uses the "component token" to authorize requests to DMP Manager. Insight 5 uses the "solution token" for a single lifecycle, which may affect the data returned by certain endpoints. By default the current lifecycle is used, but a different one may be specified to dmpinitmanager.
• Insight 5 does not restrict which endpoints of DMP manager or a DMP solution webservice may be accessed, whereas Insight 4 will only allow calls to a small number of whitelisted paths.
• When the dmp module is used from a submodel in Insight 4, it must also be referenced from the master model (see section Use within Submodels above). In Insight 5, this is not required.