Connectors / Service / Azure Blob Storage

Azure Blob Storage

Azure Blob Storage

Azure Blob Storage is a feature of Microsoft Azure. It allows users to store large amounts of unstructured data on Microsoft's data storage platform.,

Overview

Azure Blob storage is a service for storing large amounts of unstructured object data, such as text or binary data. Common uses of Blob storage include: Serving images or documents directly to a browser. Storing files for distributed access.

When using the Azure Blob Storage connector, the first thing you will need to do is go to your Tray.io account page, and select the workflow you wish to work on. Once in the workflow builder itself, search and drag the Azure Blob Storage connector from the connectors panel (on the left hand side) onto your workflow.

With the new Azure Blob Storage connector step highlighted, in the properties panel on the right, click on the Authenticate tab and 'Add new authentication' (located under the 'Authentication' field).

This will result in a Tray.io authentication pop-up window. The first page will ask you to name your authentication, and state which type of authentication you wish to create ('Personal' or 'Organisational').

As you can see, the next page asks you for your 'Instance URL', 'Directory ID', 'Client ID', and 'Client Secret', credentials. Also please note that the scopes have to be ticked in order for your authentication to work as expected.

In order to get these fields, you will have to create an integration application in Azure.

  1. First sign in to the Azure portal.

  2. If you have access to multiple tenants, use the Directory + subscription filter in the top menu to select the tenant in which you want to register an application.

  3. Search for and select Azure Active Directory.

  1. Under Manage, select App registrations, then New registration.
  1. Enter a name for your application and the redirect URL and select Register to complete the initial app registration.

When registration completes, the Azure portal displays the app registration's Overview pane, which includes its Application (client) ID. Also referred to as just Client ID, this value uniquely identifies your application in the Microsoft identity platform. Please copy your 'Client ID' (Application ID) and 'Directory ID' (Tenant ID) to add them in the Tray authentication pop-up.

Next, we have to configure the Azure Application:

  1. Select your application in 'App registrations' in the Azure portal.
  1. Under Manage, select 'API permissions', then click on the 'Add permission' button and choose 'Azure Storage' box.
  1. Next tick the 'user_impersonation' check box and click the 'Add permissions' button.
  1. Notice your Azure Storage user impersonation was added.

Get you 'Client Secret':

  1. Select your application in App registrations in the Azure portal.

  2. Select 'Certificates & secrets' and then 'New client secret'.

  1. Add a description for your client secret, select a duration, then select 'Add'.
  1. Record the secret's value for use in your client application code - it's never displayed again after you leave this page.

Configure RABC role for the user:

  1. Select your Subscription from Microsoft Azure resources.
  1. Select the subscription you need.
  1. Choose 'Access control (IAM)', click on the 'Add' button, then select 'Add role assignment'.
  1. Then choose 'Storage Blob Data Owner' from the dropdown list and search for the user you want to give permissions to and click 'Save'.

The 'Instance URL' can be found in the API documentation on every endpoint page. 'myaccount' has to be changed with the name of the storage you want to retrieve the blobs from.

PLEASE NOTE: The first scope of the authentication is required so if this scope is not ticked, the authentication will fail, the second scope in not required, but its purpose is to refresh the authentication every hour. If not ticked, your authentication will stop working in one hour after it was created, when the 'access token' expires.

Once you have added these fields to your Tray.io authentication popup window, click on the 'Create authentication' button. Go back to your settings authentication field (within the workflow builder properties panel), and select the recently added authentication from the dropdown options now available.

Your connector authentication setup should now be complete.

Available Operations

PLEASE NOTE: All of the operations have under the 'Show advanced properties' button, in the properties panel, the header input fields that can be filled in.

The examples below show two of the available connector operations in use.

Please see the Full Operations Reference at the end of this page for details on all available operations for this connector.

Note on Operations Usage

Using the Raw HTTP Request ('Universal Operation')

As of version 1.0, you can effectively create your own operations.

This is a very powerful feature which you can put to use when there is an endpoint in Azure Blob Storage which is not used by any of our operations.

To use this you will first of all need to research the endpoint in the Azure Blob Storage API documentation v1.0, to find the exact format that Azure Blob Storage will be expecting the endpoint to be passed in.

Note that you will only need to add the suffix to the endpoint, as the base URL will be automatically set (the base URL is picked up from the value you entered when you created your authentication).

The base URL for Azure Blob Storage is: https://test-storage.blob.core.windows.net.

For example, say that the 'List Blobs' operation did not exist in our Azure Blob Storage connector, and you wanted to use this endpoint, you would use the Azure Blob Storage API docs to find the relevant endpoint - which in this case is a GET request called: /mycontainertest?restype=container&comp=list.

More details can be found here.

As you can see there is also the option to include a query parameter, should you wish to do so. Note that the 'x-ms-version' header is required in order to make a successful request. So if you know what your method, endpoint and details of your query parameters are, you can get the Blob storage information with the following settings:

Method: GET

Endpoint: /mycontainertest?restype=container&comp=list

Query Parameter: Key: prefix Value: a

Final Example outcome being: https://test-storage.blob.core.windows.net/mycontainertest?restype=container&comp=list

Example Usage

TRAY POTENTIAL: Tray.io is extremely flexible. By design there is no fixed way of working with it - you can pull whatever data you need from other services and work with it using our core and helper connectors. This demo which follows shows only one possible way of working with Tray.io and the Azure Blob Storage connector. Once you've finished working through this example please see our Introduction to working with data and jsonpaths page and Data Guide for more details.

Below is an example of a way in which you could potentially use the Azure Blob Storage connector, to get the existing Blobs from your storage account one by one.

The steps will be as follows:

  1. Setup using a manual trigger and List Blobs operation.
  2. Add a Loop connector to iterate through the blobs found.
  3. Get all the available data for each blob, individually.

The final outcome should look like this:

1 - Setup Trigger & List Blobs operation.

Once you have clicked 'Create new workflow' from your main Tray.io dashboard named it, select the Manual trigger from the trigger options available:

After you have been redirected to the Tray.io workflow dashboard, from the connectors panel on the left, add a Azure Blob Storage connector to your second step. Set the operation to 'List Blobs'.

Feel free to re-name your steps as you go along to make things clearer for yourself and other users.

When run, this workflow will list all the blobs and all the snapshots available within the Azure Blob Storage under your credentials.

2 - Add a Loop connector

Next, search for the Loop collection connector within your connector panel, and drag it into your workflow as your next step. Set your operations to 'Loop list'.

The Loop Collection connector allows you to iterate through a list of results. In this example, we will use it to iterate through the data found within the previous Azure Blob Storage connector step.

In order to specify the list you want to loop through, start by using the 'List' mapping icon (found next to the list input field, within the properties panel) to generate the connector-snake.

While hovering over the 'List Blobs' step (with the tail end of the connector-snake), select Blob from the list of output properties displayed. This will auto-populate a jsonpath within your 'List' input field, and update the type selector to jsonpath.

For more clarification on the pathways you have available, open the Debug panel to view your step's Input and Output.

JSONPATHS: For more information on what jsonpaths are and how to use jsonpaths with Tray.io, please see our Intro page and Data Guide for more details.
CONNECTOR-SNAKE: The simplest and easiest way to generate your jsonpaths is to use our feature called the Connector-snake. Please see the main page for more details.

When run, this workflow will iterate through the list of blobs provided by the earlier step.

3 - Download Blob

The last step is to drag a Azure Blob Storage connector inside of the Loop Collection step itself. Set the operation to 'Download Blob'. As you can see, the 'Container name' and the 'Blob name' fields are required.

INTERPOLATION: When you wish to include JSON generated data within another input/ output/ result, use our Interpolation method as described here.

You can also check the Debug panel for more details on the provided output:

When run, the workflow will now iterate through the list of blobs provided, and get all the details available for each individual one.

BEST PRACTICES: Whenever you do decide to create your own workflow, please make sure you take a look at our Managing data best practices guide.

All Operations

Latest version:

1.0