Create Custom Functions

The Graph.Build Transformers allow for functions to be included within the mappings. These functions allow for raw data to be transformed or filtered on its way to being translated to Graph Data. A wide array of functions are currently supported, including the majority of the GREL string functions - a full list of the supported functions can be found here. If however, when designing your mapping file for your Transformer, you require a function to be executed that cannot perform your required operation simply by using the built-in functions, it is possible to create and use your own.

Steps

1. Create your functions in Java

The logic for your function is carried out using Java code. First, create yourself a Java Class, this Class can contain as many functions as you wish; a function exists as a method with both input values and an output value. For example, let’s create a simple function to append a greeting to the input value.

public class CustomFunctions {
    public static String appendGreeting(String value) {
        return value + " Hello World";
    }
}

2. Create the functions Turtle mapping

Next, you must create the mapping to your Java functions, this is done in Turtle (ttl). Below is the mapping required for our Greeting Function, along with a line by line description. The filename is irrelevant as this will be copied into the body of the API request.

3. Build your Jar

The next step is to build your Java functions Class file into an executable Jar. To do this via the command line, simply execute the following two commands:

javac CustomFunctions.java

Followed by:

jar -cvf CustomFunctions.jar CustomFunctions.class

4. Push your files to your running Transformer

Now we have our Turtle and Jar, we can push these files to a running Transformer. This is done by setting the CUSTOM_FUNCTION_JAR_URL and CUSTOM_FUNCTION_TTL_URL configuration options to point at your jar and ttl files, either locally (file://) or on S3, for example s3://graphbuild/custom-functions/CustomFunctions.jar. If set before the Transformer is started, these files will be downloaded and set at startup. If however your Transformer is already running, or these files were edited after startup, you must call the GET /updateCustomFunctions endpoint to download and reset these function files, for example http://localhost:8080/updateCustomFunctions.

5. Creating your mapping file using your new function

Once all the previous steps have been completed, you are now able to use your new function in your mapping files. For an in-depth look at creating a mapping file from scratch, see our guide.

Strings passed into functions may be sourced from an rml:reference (The entire value of a field in the source document), an rr:template (A string that includes values of fields in the source document, as well as manually specified strings), as well as outputs from other Functions (functions may therefore be nested). Note that, when a function is used and its output is a NULL value, a triple will not be generated.

To see the full breakdown, refer to the functions section in our guide. However, using our previously created example function and a simple CSV input file with an id and name column, the mapping file will look like the following:

6. Using this example in your Transformer

All example files created in this document can be downloaded from our Bitbucket. The easiest way to test custom functions on your Transformer is to first use our example files. Steps 1, 2, 3, and 5, have already been completed for you, so all you need to do is trigger the API endpoints of you Transformer as seen in step 4.

  1. PUT request to http://<transformer-ip-address>:8080/functions?path=https://bitbucket.org/data-lens/datalens-examples/raw/5be1841fda5dcdfd8ded84e12a3877a688ab4840/custom-functions/CustomFunctions.jar with header Content-Type: text/turtle, and body as ttl example.

Now you can test the new functions in your Transformer by triggering the Process endpoint using our example as input:

  1. GET request to http://<transformer-ip-address>:8080/process?inputFileURL=https://bitbucket.org/data-lens/datalens-examples/raw/5be1841fda5dcdfd8ded84e12a3877a688ab4840/custom-functions/input-file.csv

Your output RDF should be similar to:

PreviousFunctionsNextList of Available Functions

Last updated