Tray Platform / Standard/Best Practices / Pagination/Batch processing / Intro to Pagination
Intro to Pagination
An important concept to understand when using Tray is that of Pagination.
This comes into play when you are using connector operations (typically 'list' operations such as 'list customers', 'list contracts' etc.) which might return long lists of results (which could be in the dozens, hundreds or even thousands) which you then want to process in some way.
In this case, it is necessary to create a 'pagination' system which breaks the results down into manageable batches (of e.g. 100 records each) and knows when to stop.
The options for pagination will depend on the service connector you are using.
How to paginate
Pagination is typically set up by creating a main loop which separates results into batches. How this will be done depends on how the service connector operation returns results:
If the operation returns lists of records in limited batches accompanied by pagination information such as a has_more property, a next_page property or a page_offset property - all of which will eventually return as False or null - then you can start a Loop Connector which breaks when a False or null value is returned.
If the service has a 'count records' type of operation, an alternative approach is to use this in conjunction with a List Helper to create a set number of batches and run a loop connector for a set number of times.
Data Storage will need to be used in order to retrieve and store the next page details at the beginning and end of your loops.
Within your loops it is then likely that you will want to set up a nested sub-loop in order to process each record one-by-one and take necessary actions (e.g. using boolean connectors to update missing values, or creating new records in other services)
Some operation examples which cater for pagination are:
The Stripe connector has a List customers operation which returns a has_more property so you know whether to make another request. It also lets you pass in a value called Starting After so that you can enter the ID of the last customer in the previous list of 100; thus Stripe will know to start the next batch of 100 from the customer which comes after this ID.
The Salesforce connector has a Find Records operation which has a Limit parameter, as well as a Page offset parameter which allows you to set the record to start from (i.e. after the last record from the previous batch)
the Salesforce connector also has a Count Records operation which can help in creating batches for processing.
You will find two pagination tutorials in this menu section:
The Updating missing customer info tutorial uses Stripe and Salesforce. This shows you how to deal with an operation such as the Stripe 'List customers' operation which returns limited numbers of results at a time. This is typical of many operations, and is dealt with by setting up a main continuous loop which processes batches until the operation indicates there are no more to be returned, whereby a 'break loop' connector is activated.
The Processing large volumes of data tutorial shows an example of sending the data produced by each batch to a separate workflow for sub-processing. This technique will allow you to process large volumes of data much quicker than if you were to do the sub-processing within the main workflow - especially if the sub-processing is relatively complex. This tutorial also shows a slightly different way of dealing with pagination, in that Salesforce has a 'Count Records' operation which can be used to split the records up into a set number of batches and so does not need to run with a continuous loop as in the first tutorial.