Documentation
Log inAcademyWhat's new?
Getting Started
Moving Data
Logs and Troubleshooting
Services and Authentications
Standard/Best Practices
Use Cases
Version Control
Advanced Tray.io usage
Useful scripts
Connecting to on-prem systems
Accounts
Security

Data storage queue

Overview
Copy

As explained in Intro to API limits it is possible to set up a Data Storage-based FIFO queue to manage a scenario whereby too many requests are coming in to a webhook-triggered workflow such that if you were to process them straight into Google Sheets, Salesforce, etc. it would exceed your limits.

This page will take you through the steps involved in setting up such a queue.

Downloadable workflows
Copy

To accompany the explanation below you can download the workflows involved to import and test for yourself:

To configure these workflows for use, you will need to:

  1. In Workflow 1, for the Send to queue step enter the webhook url of Workflow 2 (found in workflow settings)

  2. In workflows 2 and 3 edit the Key of the prospects list from 'prospects_d5ceaa746fc6' so that the suffix is taken from the url of workflow 2 to identify it as the source of the prospects list:

This is good practice to help you keep track of account-level lists. Note that you can also programmatically pick up the workflow uuid for this purpose, as explained in our documentation on workflow threads

  1. In Workflow 3, add an authentication for the Google Sheets Add row step and enter the correct spreadsheet ID (you will need to have set up a Google Sheet with 4 columns "id", "first_name", "last_name" and "email")

Running the workflows
Copy

To run this test setup:

  1. Enable Workflow 3

  2. Click 'Run workflow' in workflow 1

  3. Wait for the records to be added to your Google Sheet

  4. Inspect the logs of each workflow

  5. Disable Workflow 3

You should only have Workflow 3 enabled for when you are running the test - i.e. before you click 'Run workflow' in Workflow 1 and until all the records have been processed. Otherwise Workflow 3 will be needlessly initiating itself every 3 minutes

Workflows explained
Copy

The purpose of these workflows is to process a list of prospects such as:

Into Google Sheets, in a controlled manner so that we end up with a full list of 100 prospects in a sheet:

Workflow 1
Copy

This workflow replicates an external service sending requests to your webhook.

It:

  1. Creates a dummy list of 100 records

  2. Chunks them into batches of 10

  3. For each batch, sends the records immediately one after the other

  4. Waits 30 seconds before sending the next batch

Workflow 2
Copy

All this workflow does is create an account-level list for each record that comes through the webhook to be added to:

If this workflow were to just immediately send each record that comes through to Google Sheets, it would risk breaking your rate limit if e.g. 200 requests a minute were coming through.

Workflow 3
Copy

This workflow then gets the prospects list and adds each one as a new row in Google Sheets.

The key thing you will see here is that the workflow kicks off every 3 mins to process the prospects that have come through.

This can of course be adjusted to make sure the rate is within Google's limits:

After adding, each one is deleted from the data storage list:

Note that the index is 0 - each time you are processing the first (earliest) entry in the list and deleting from data storage, ready to process the next.

Important notes
Copy

The data storage limit under a single key is 400kb so if you are processing massive amounts of data you should be aware that data storage might get overloaded. In which case we recommend either using a CSV queue, or Amazon SQS. When consuming a lot of data you should set the scheduled trigger interval as close to the minimum of 1 minute as possible, wiithout exceeding your rate limit

  • Note that some records might come in to Google Sheets slightly out of order. This is due to the rate at which they are being sent from Workflow 1 and received by the Workflow 2 webhook. This illustrates that, even if you implement a FIFO queue, you are still at the mercy of the order in which requests are sent by the source service

  • A further tweak can be made by adding a delay to the loop which processes each prospect to Google Sheets

  • Depending on how you adjust the timing of the batch delay in Workflow 1 and the trigger in Workflow 2, you may find the check list is empty step at the end of workflow 3 may show sometimes shows records in the list, as more will have been added since the workflow was triggered and before it finished. These will just be processed the next time the workflow runs!