Pulling data in batches (pagination)

                                        Basic pagination architecture
                                        Copy

                                        We also have a detailed Academy course on pagination, including an interactive lab to help you put it into practice!

                                        An important concept to understand when using Tray is that of Pagination.

                                        This comes into play when you are using connector operations (typically 'list' operations such as 'list customers', 'list contracts' etc.) which might return long lists of results (which could be in the dozens, hundreds or even thousands) which you then want to process in some way.

                                        In this case, it is necessary to create a 'pagination' system which breaks the results down into manageable batches (of e.g. 100 records each) and knows when to stop.

                                        The options for pagination will depend on the service connector you are using.

                                        The following example is taken from the Pagination Architecture snippet which shows how to paginate results from a table in Airtable:

                                        It illustrates the key steps involved in basic pagination:

                                        1. At the start of the loop you retrieve the offset token using Data Storage 'Get value' (Scope - Workflow) (Default empty string for first run)

                                        2. You retrieve the records using a 'List records'-type operation. Passing the offset token where appropriate and the batch size / page size (stored in your Project Config as best practice)

                                        3. You then process the records as required (in this example each record is being added to a Google Sheet)

                                        4. At the end of each loop you check if the service (Airtable) has returned an offset token, indicating there are more records to be pulled

                                        5. If so then you store the offset token using Data storage 'Set value' (Scope - workflow) to be retrieved at the start of the next run

                                        6. If not then you break the forever loop as there are no more records to process

                                        Investigating pagination requirements
                                        Copy

                                        When building and testing in Tray you can inspect the input and output logs of connector operations to look for key fields such as 'has_more' or 'offset_token'.

                                        To save time when scoping out your project requirements, you can also make use of our Ops Explorer (beta) dev tool to explore sample input and output payloads for different operations to get a quick idea of any pagination requirements:

                                        Aside from the Airtable example above, some other connector operations which require pagination are:

                                        • The Stripe connector has a 'List customers' operation which returns a has_more property so you know whether to make another request. It also lets you pass in a value called Starting After so that you can enter the ID of the last customer in the previous list of 100; thus Stripe will know to start the next batch of 100 from the customer which comes after this ID.

                                        • The Salesforce connector has a 'Find Records' operation which has a Limit parameter, as well as a Page offset parameter which allows you to set the record to start from (i.e. after the last record from the previous batch)