Templates / Sales / SalesOps | Paginate through Salesforce Records

SalesOps | Paginate through Salesforce Records

Overview

This template demonstrates how you can pull records from Salesforce and process them in batches using a process called pagination.

This is a standard method which you will need to use any time you wish to pull large numbers of records (accounts, contacts etc.) from Salesforce, using the Salesforce 'Find records' operation.

The screenshot below shows a set of results that exceeds 20,000 records.

When this happens, Salesforce returns a 'next_page_offset' token.

In this case, you can then run a pagination loop which pulls SFDC results in batches and runs until the 'next_page_offset' token is equal to Null.

If, for example, you have 6000 records, you can pull these in three batches of 2000 (Salesforce allows you to pull a maximum of 2000 records in one call). Rather than making 6000 individual calls which would overwhelm your Salesforce API limits.

Connectors Used

The following connectors are used in this template. This is provided for reference only - there is no need to read through the linked connector pages, as all you need to know for this template is explained on this page:

End Result

The data fetched from SFDC will be available in batches for whatever processing you need to carry out. In this case, we are using Data Storage as a generic placeholder to store the fetched Salesforce data.

This template only shows you how to get records out of Salesforce. It does not assume anything about what you want to do with these records.

Please see our Paginate through Salesforce Records and Email Resulting CSV template for an example of sending paginated Salesforce records to a particular destination

Prerequisites

This workflow assumes the following:

  • You have a Salesforce instance and can authenticate as an API user.

Getting Live

In order to configure this workflow for use simply:

  1. Authenticate to Salesforce and any other needed systems. Ensure that your authentication will have access to the records required by your automation goals. This authentication must be added to both Salesforce connector steps within the workflow.

  2. Identify a relevant trigger based on your requirements:

    • If this is a standalone process to be triggered in the Tray platform, you can use a Manual trigger.
    • If it is to initiate daily or on specific intervals then use a Scheduled Trigger.
    • If you wish to fire the workflow when something happens in another system, then you can use a relevant Service Trigger or a Webhook Trigger if there is no pre-built Tray Service Trigger.
  3. Determine which type of record you wish to fetch from Salesforce, and what fields from each record. In the Find Records (salesforce-1) step you can specify the type of record, and also the fields you want to be returned for each record:

  4. Determine how you wish to process the SFDC records. If the process needs to run quickly, you may wish to leverage the Call/Callable Workflow concept. For detailed instructions, refer to the Implementation notes below.

  5. The workflow uses a Manual trigger; click Run workflow in the bottom left of the builder anytime you wish to run the workflow.

Workflow Logic

The overall logic of the workflow is:

  1. A forever loop runs until there are no more batches of records.
  2. Within the loop, the 'next_page_offset' token from the last batch of records is fetched using the Data Storage 'Get Value' operation (on the first run, this will be null).
  3. The Salesforce 'Find records' operation then pulls a batch of records, using the 'next_page_offset' token to start from where the last batch finished. Each batch is then available for further processing.
  4. A boolean condition checks if the 'next_page_offset' token ($.steps.salesforce-1.next_page_offset) received from salesforce is equal to null.
    • If the 'next_page_offset' token = null, the loop will break using the Break Loop step.
    • If the 'next_page_offset' token is not equal to null, it means there are more batches of records to process, and the loop will continue.
  5. If the loop continues the final step is to use the Data Storage 'Set Value' operation to set the 'next_page_offset' token for the next batch of records.

Implementation notes

The Process SFDC Records step within the workflow is a generic placeholder for whatever you may wish to do with each batch of SFDC records.

For simplicity's sake, we are just using data storge to store the records.

There are various ways in which you can use this fetched Salesforce data, such as:

Step-by-step Explanation

For educational purposes, the following will take you through how the workflow paginates through the Salesforce records and email resulting CSV.

This will help deepen your understanding of Tray.io and will give you the power to maintain and edit your workflow as necessary.

You will see that the steps below have both a step name (Get Offset, Find Records etc.) and a programmatic name (storage-1 salesforce-1 etc.)

The programmatic name is what enables the use of jsonpaths such as $.steps.storage-3.result to pull data from previous steps in your workflow.

Please note that the numbering of these may appear to be out of sequence in your workflow. This is because your workflow may contain nested loops, multiple branches, booleans etc. which means that a strict top-to-bottom sequence of steps does not always occur.

Steps may also be moved or deleted after they have been created.