AWS For Manufacturing: What Is Kinesis?

When working with integrated systems, you will find at least some form of communications protocol and data structure.

In automation, this was originally Modbus followed by more modern protocols like Ethernet-IP, Profinet, MQTT, OPC-UA, BACNet and DNP3 as technologies evolved. Commonly, on the web this would be JSON.

With large enterprise level data quantities—or more than a handful of devices—you need methods of managing and sending large amounts of data to many different systems.

In the world of Amazon Web Services (AWS) where systems can be on a global scale with thousands or millions of users and devices, this protocol is Kinesis.

What is Kinesis?

Kinesis packages data for use in AWS services. At its most basic level, Kinesis data is known as a data stream consisting of a sequence of data records. Each data record has a sequence number, a partition key, and a data blob. This allows the data to be streamed in sequential order, simplifying event management and tracking for all of the data on the stream.

Kinesis Data Streams are split across shards, with each providing a fixed unit of throughput capacity. You can scale your overall capacity by adding more shards.

Other systems and services can read data streams, or you can aggregate them into Kinesis Firehoses which can send a larger amount of data to specific tools for analysis and storage. Kinesis Data Streams are also powerful because they can store historical data directly in the stream, saving you the additional overhead of a separate process historian for short term data storage.

Using the Best Tool for the Job

Similar to using a Unified Namespace, you might use a different communication protocol for data inside the process control network before converting it to MQTT before it goes into the Unified Namespace. Kinesis is a protocol designed for use within AWS. This approach lets you utilize the best strengths of each protocol at each level of the automation pyramid and convert to a better protocol as you move up and down between layers.

At a very high level, you can think of Kinesis as a type of Unified Namespace where the data is used by AWS and the tools, technology, and services under the AWS umbrella.

In addition to Kinesis as a general tool, it offers three different flavors focused on different use cases.

Kinesis For Every Use Case

Kinesis currently has different models: Kinesis Data Streams, Kinesis Firehose, and Kinesis Video Streams.

Kinesis Data Streams

Kinesis Data Streams form the most basic level of Kinesis data in AWS. They are useful for getting data into AWS, and can also be used by systems like Ignition read data from AWS. You can write to and read from them.

Data passes through the streams in sequence, allowing you to manage and track events and data in chronological order. Kinesis Data Streams can also be encrypted as data enters the stream, providing security for the entire flow of data throughout the system.

From a manufacturing perspective, data typically flows from a SCADA system like Ignition through either AWS IoT Greengrass to AWS IoT Core, or directly into AWS IoT Core— where it would enter the Kinesis Data Stream infrastructure.

Kinesis Firehose

The easiest way to conceptualize a Kinesis Firehose is to think of it as a larger, specialized form of a Kinesis Data Stream.

Kinesis Firehoses are write-only, and deliver real-time streaming data to specific destinations. These destinations are typically “big data” endpoints like Amazon Redshift, Amazon Opensearch, MongoDB, or more generalized large storage systems like Amazon S3. They can also be analytics or machine learning platforms that will further process the data and output useful information.

From a manufacturing perspective, a Kinesis Firehose is similar to a Process Historian. The data goes through the Process Historian’s pipeline into a database and after the data has transfered, it will be accessible by other systems through the database. With a Kinesis Firehose, you can send data to many different systems instead of simply a database.

Kinesis Video Streams

Much like how Kinesis Firehoses act as a specialized form of Kinesis Data Streams, Kinesis Video Streams collect video data and make it accessible in the cloud. This can be used for watching live data feeds and collecting images or video snippets from cameras.

From the manufacturing perspective, Kinesis Video Streams are very useful for managing image-based quality control, watching process equipment, and collecting video events surrounding alarms or process upsets. It could also be integrated with pick and place operations to enhance robotic systems, or move goods through a robotic process without human intervention.

Wrapping Up

The AWS environment provides access to many different tools and technologies for interacting with process data. In addition to storing process data and logs, you can take advantage of machine learning tools, advanced analytics packages, and storage options available in AWS.

AWS provides options like Greengrass and AWS IoT Core to easily get data from a manufacturing management system into AWS. By using Kinesis Data Streams, or data output from tools inside AWS, you can read data back into your SCADA system—and use the insights and information from AWS to better inform your operations and improve your processes.

There is a lot to learn with AWS, and how you can better take advantage of the power available in a manufacturing environment. If you have any questions on anything AWS related or how you can utilize it in your facility please reach out and let us know!