Connectors / Service / AWS Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. (updated: 1633093264604)

AWS Kinesis

Collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information

Overview

The AWS Kinesis connector currently allows you to load data onto a Kinesis firehose.

Authentication

To use the Kinesis firehose operations you need to create a user with the following permissions to put records onto the firehose:

"firehose:ListDeliveryStreams"
"firehose:PutRecord"
"firehose:PutRecordBatch"

Adding records to a Kinesis firehose

An AWS Kinesis firehose allows you to send data into other AWS services, such as S3, Lambda and Redshift, at high scale. The Kinesis connector includes 2 operations that allow you to either send a single item of data (Put Record) or send multiple items (Put Batch Record) to a Kinesis firehose.

To get started with a Kinesis firehose, its recommended that you follow the AWS guide to setting one up.

Putting Records

Using the put record operations is as simple as setting an AWS region, choosing a delivery stream and then sending some raw text content.

The raw text content can be of any type, but usually depends on how you have setup your firehose. For example if you want to load your data directly into redshift the content can be a comma seperated list of column values. Alternatively you may want to copy raw JSON into redshift.

PLEASE NOTE: If you are planning putting a JSON object onto your firehose, you need to useInterpolated Mode to send the JSON as raw string. If you try and map the output of a previous stepdirectly into the Data field, it will not stringify the JSON object.

All Operations

Latest version:

1.1

List Firehose Delivery Streams (DDL)

List the available Kinesis firehose streams

(Firehose) Put Record

Puts a record onto a AWS Kinesis Firehose

(Firehose) Put Batch Records

Puts a batch of records onto an AWS Kinesis firehose

List Streams (DDL)

List the Kinesis streams available in the specified region

Put Record

Put a single record onto a Kinesis stream.

Put Batch Records

Put a number of records onto a Kinesis stream.