Triggering the Transformer / Ingesting Data

While the mapping file previously created handles the querying of your target databases, triggering the Transformer to start the ingestion of data can be done in a number of ways.

RESTful API Endpoint

Besides the Graph.Build Studio, the easiest way to trigger the database ingestion in the SQL Transformer is to use the built-in APIs. Using the process endpoint will trigger the Transformer, for example, using a GET request: <transformer-ip>:<transformer-port>/process, for example, http://127.0.0.1:8080/process. The completed process is confirmed with a success report.

Cron Job

In addition to the RESTful service, there is also a built-in Quartz Time Scheduler. This uses a user-configurable Cron Expression to set up a time-based job scheduler which will schedule the Transformer to ingest your specified data from your database(s) periodically at fixed times, dates, or intervals.

For example, 0 */30 * ? * * * translates to triggering the Transformer every 30 minutes starting at :00 or :30 minutes after the hour. A more detailed explanation can be found on the Quartz Scheduler website.

By default, cron executions are turned off. To turn it on and set a cron expression, use the /updateCron endpoint as explained below.

Kafka

The Transformer can also be triggered by sending a message to a Kafka queue. To utilise this you must ensure TRANSFORMER_RUN_STANDALONE is set to false and the Kafka configuration is configured correctly for you brokers. Once set up, you can simply push an empty message to the topic specified by the KAFKA_TOPIC_NAME_SOURCE.