REST API Endpoints

In addition to the Process Endpoint designed for triggering the ingestion of data into the Transformer, there is a selection of built-in exposed endpoints for you to call.

API	HTTP Request	URL Template	Description
Process	GET	/process?inputFileURL=<input-file-url>&logicalSources=<logical-source>	Tells the Transformer to ingest the file located at the specified URL location using the correlating logical source
/process?inputFileURL=<input-file-url>&logicalSources=<logical-source>&mappingURL=<mapping-url>	When an input file needs to be transformed against a specific mapping file, an additional optional parameter for the mapping file or mapping directory URL can be given.
/process?inputFileURL=<input-file-url>&logicalSources=<logical-source>&inputFileURL=<input-file-url>&logicalSources=<logical-source>	You also have the ability to process multiple input source files at once, this is for instances where there are joins in your mapping, creating links between your data. Each input file requires a correlating logical source pairing it with the source in the mapping. (This approach can also use the specific mapping parameter).
Config	GET	/config	Displays configuration as JSON string
GET	/config?paths=<config-options>	Displays all Transformer configuration specified in the comma-separated list
Update Config	PUT	/updateConfig?configEntry=<config-entry>&configValue=<config-value>	Update configuration options on a running Transformer
Upload Config Backup	PUT	/uploadConfigBackup	Uploads the current configuration to the specified config backup location so that it can be restored at a later date.
License	GET	/license	Displays license information
RML	GET	/rml?fileName=<file-name>	Displays the RML mapping file at the specified location, this is displayed in Turtle RDF serialisation
PUT	/rml?fileName=<file-name>	Deploys a new mapping file into Transformer specified in the request body
Functions	GET	/updateCustomFunctions	Allows for the deployment of new custom functions to the Transformer. It will download the files set in the CUSTOM_FUNCTION_JAR_URL and CUSTOM_FUNCTION_TTL_URL config options.
Restart Kafka	GET	/restartKafka	Turns the Transformer Kafka connection on or off depending on its current state.

Process

GET /process

As previously outlined, using the process endpoint is one way of triggering the Transformer to ingest your source data. When an execution of the Transformer fails after being triggered in this way, the response will be a status 400 Bad Request and contain a JSON response message similar to that sent to the dead letter queue.

{
    "name": "Semi-Structured-Transformer",
	"time": "2022-04-21T11:36:10.002",
	"type": "Semi-Structured-Transformer",
	"error": "Record 'file:///var/local/basic-input.xyz' could not be processed due to: Input file is of an invalid file type: (.xyz) - Expected XML, JSON, CSV, XLSX, or ODS.",
	"version": "2.0.3",
	"entity": "file:///var/local/basic-input.xyz",
	"provenance": "...[prov]..."
} 

Config

GET /config

The config endpoint is a GET request that allows you to view the configuration settings of a running Transformer. By sending GET http://<transformer-ip>:<transformer-port>/config (for example http://127.0.0.1:8080/config), you will receive the entire configuration represented as a JSON, as seen in this small snippet below. All confidential values (such as credentials) are hidden. This endpoint is also useful as a means of Health Checking the Transformer.

{
    "graphBuildTransformerConfig": {
        "provenance": {
            "kafka.kafkaBrokers": "localhost:9092",
            "outputDirUrl": "s3://graphbuild/testing/output/",
            "kafka.topicNameSuccess": "prov_success_queue",
            "recordProvo": true,
            "kafka.topicNameDLQ": "prov_dead_letter_queue",
            "switchedOffActivities": ""
        },
        "kafka": {
            "topicNameSource": "source_urls",
            "topicNameDLQ": "dead_letter_queue",
         ............    

Update Config

PUT /updateConfig

The configuration on a running Transformer can now be edited without having to restart. This is done through the update config endpoint. For example, by running the following /updateConfig?configEntry=friendlyName&configValue=GraphBuilder we have changed the friendly name of the Transformer to GraphBuilder. To see a list of the configuration entry names, consult the Semi Structured Transformer Configurable Options.

License

GET /license

The license endpoint is a GET request that allows you to view information about your license key that is in use on a running Transformer. By sending GET http://<transformer-ip>:<transfomer-port>/license (for example: http://127.0.0.1:8080/license), you will receive a JSON response containing the following values.

{
    "license": "# graphbuild License (id: xxxx)xxxxxxxxx",
    "licenseInformation": {
        "registeredTo": "<name> - <company> - <e-mail>",
        "expires": "Sat Feb 31 00:01:00 GMT 2023 - (xxx days to go)",
        "productName": "graphbuild"
    },
    "validationStatus": "LICENSE_VALID"
}

RML

The RML endpoint is all about the mapping file that you would have created. It consists of a GET and a PUT endpoint, allowing you to get the current master mapping file currently in use on the Transformer, and well as replacing the master mapping file with a new one.

GET /rml

By sending GET http://<transformer-ip>:<transformer-port>/rml?fileName=<file-name> you will receive a response containing the contents of the mapping file at the location specified written in RDF/Turtle.

PUT /rml

By sending PUT http://<transformer-ip>:<transformer-port>/rml?fileName=<file-name> with a turtle mapping file in the body of the request, it will upload it to the file location specified. If a file already exists in that location, it will be replaced, otherwise a new file will be created. The mapping file should be in RDF/Turtle format and the declared HTTP Content-Type should be text/turtle. The successful upload is then indicated by an empty response with HTTP status OK (Ref. RFC-7231) and will be functional immediately.

Custom Functions

If when designing your mapping file for your Transformer, you require a function to be executed that cannot perform your required operation simply by using the built-in functions, it is possible to create and use your own. This can be done by setting your CUSTOM_FUNCTION_JAR_URL and CUSTOM_FUNCTION_TTL_URL config options to point at your jar and ttl files, and calling this endpoint to download and set these files. For further instructions on how to correctly carry out this process, please see the walkthrough.