Post-mapping transformations (End User-defined schema)

In this article, you will go through some common data transformation challenges that you might run into while building ETL (extracting from source service A, transforming it using end user-defined schema, loading into destination service B) integrations.

For each example that follows:

• We will be using dummy data which you would get from the source service.

• We have the end user-defined schema.

• We will be showing possible pathways (Core connectors, JSON transformer, Javascript) for each example. You are free to take a different route as well.

1 - Preparing input for a Service

Copy

In this example, we are going to map the input data/dummy data which could be coming from a source service using the end user-defined mapping in the configuration step (config wizard/custom form). The basic idea here is to write a code that can respond dynamically to any mapping provided by the end user.

We have the following dummy data:

The mapping and expected input schema for two end users is shown below:

You can import this project here. It contains two ways you could solve this:

The same code works for both end users and you can test this by editing the 'mappings' config value in workflows.

You can also play around with data and mappings in the playground. Here are the JSONata exerciser solution links for:

• End user 1

• End user 2

Here is the playground link if you want to play with the JavaScript solution.

2 - Preparing input for Mailchimp batch subscribe operation

Copy

We have the following dummy data:

Mailchimp Connector's Raw HTTP operation to batch subscribe members will expect an input schema which depends on the user-defined mapping.

The mapping and expected input schema for two end users is shown below:

You can import this project here. It contains two ways you could solve this: