Intro to Pagination

Introduction

Copy

An important concept to understand when using Tray is that of Pagination.

This comes into play when you are using connector operations (typically 'list' operations such as 'list customers', 'list contracts' etc.) which might return long lists of results (which could be in the dozens, hundreds or even thousands) which you then want to process in some way.

In this case, it is necessary to create a 'pagination' system which breaks the results down into manageable batches (of e.g. 100 records each) and knows when to stop.

The options for pagination will depend on the service connector you are using.

How to paginate

Copy

The Loop Connector and Data Storage connector are your friends when it comes to pagination!

Pagination is typically set up by creating a main loop which separates results into batches. How this will be done depends on how the service connector operation returns results:

• If the operation returns lists of records in limited batches accompanied by pagination information such as a has_more property, a next_page property or a page_offset property - all of which will eventually return as False or null - then you can start a Loop Connector which breaks when a False or null value is returned.

• If the service has a 'count records' type of operation, an alternative approach is to use this in conjunction with a List Helper to create a set number of batches and run a loop connector for a set number of times.

Data Storage will need to be used in order to retrieve and store the next page details at the beginning and end of your loops.

Within your loops it is then likely that you will want to set up a nested sub-loop in order to process each record one-by-one and take necessary actions (e.g. using boolean connectors to update missing values, or creating new records in other services)

Example operations

Copy

Some operation examples which cater for pagination are:

• The Stripe connector has a 'List customers' operation which returns a has_more property so you know whether to make another request. It also lets you pass in a value called Starting After so that you can enter the ID of the last customer in the previous list of 100; thus Stripe will know to start the next batch of 100 from the customer which comes after this ID.

• The Salesforce connector has a 'Find Records' operation which has a Limit parameter, as well as a Page offset parameter which allows you to set the record to start from (i.e. after the last record from the previous batch)

• The Salesforce connector also has a 'Count Records' operation which can help in creating batches for processing.

• If you have received an array of data e.g. via a webhook or a callable workflow trigger, the List Helpers connector has a 'Chunk' operation which can be used to divide large lists of data into smaller batches. The result can then be processed using a loop connector