loading...
Cover image for Gently Down the Stream with Amazon Kinesis

Gently Down the Stream with Amazon Kinesis

helenanders26 profile image Helen Anderson Originally published at helenanderson.co.nz Updated on ・4 min read

Gone are the days of batch processing and simply loading new tables into the database every 12 hours. With more and more platforms offering event streams of data, the infrastructure that stores these events and draws meaning from it needs to change too.

This post is an overview of how AWS Kinesis can be built into new or existing architecture to solve this problem. There are several options that allow you to run analytics on the fly, shard the data streams for scalability or simply stream the data into an S3 bucket for later processing.


Stream v Batch
Kinesis Streams
Kinesis Firehose
Kinesis Analytics
Is it secure?
How do I pay for all this?
Getting Started



Stream v Batch

Kinesis allows data to be streamed in real time from a Producer to a Processer or Storage option. More on these concepts in a bit.

This is a huge change from Batch Processing that has been the traditional way to land data from one location to another.

Batch Processing - Data, usually stored in a database, is landed in chunks and analysed when the transfer is complete.

Stream Processing - Streams of data pour in, in realtime and don't have an end... unless you create one. This allows us to act on the data and make decisions faster.



Kinesis Data Streams

Kinesis Data Streams

Back to the concepts, using Kinesis Data Streams as an example.

Input/Producer: The application that generates the events we want to capture. This can be log files, media, website clicks or transactional data.

Data Stream: This is a shard, or group of shards, that ingest records at 1000 records per shard, per second. Data is then available for 24 hours.

Consumer/Processer: This is the AWS service, which can be another Kinesis service, that retrieves the events from the shards. In most cases, this is happening in real-time. AWS Lambda can be triggered to transform the event data into more usable data or push it into a database like DynamoDB or Aurora.


Use cases for Kinesis Data Streams:

  • Streaming data like website clicks and transactional data
  • Migrating data from databases
  • Applications with specialised data pipelines



Kinesis Firehose

Kinesis Firehose

Kinesis Firehose differs from Kinesis Data Streams as it takes the data, batches, encrypts and compresses it. Then persists it somewhere such as Amazon S3, Amazon Redshift, or Amazon Elasticsearch Service.


Use cases for Kinesis Firehose:

  • IoT events
  • Security monitoring such as Splunk can be configured as a destination
  • Auto Archiving



Kinesis Analytics

Kinesis Analytics

Kinesis Data Analytics allows us to both process events and analyse them using SQL queries on-the-fly. The service recognises formats like JSON and CSV, then sends the output on to analytics tool for visualisation or action.


Use cases for Kinesis Analytics:




Is it secure?

Kinesis automatically encrypts events and access can be managed using IAM from the console.



How do I pay for all this?



Getting started

This service is not included in the Free Tier but many of the other core services are.



Useful Links


Photo by Tobias Bjørkli from Pexels

Discussion

pic
Editor guide
Collapse
perigk profile image
Periklis Gkolias

Hi Helen,

It looks like it is 4 months in your study journey.

I suppose you are studying part-time. How much time does it take to feel confident for the certification?

Why did you pick the developer one and not the architect which I suppose ( and correct me if I am wrong) is a superset in terms of what you learn?

Collapse
dvddpl profile image
Davide de Paolis

I wanted to ask the same thing. I am currently evaluating if diving into some AWS certification and wondering what would be best / more useful ( Solutions Architect or Developer Associate ).
How much effort is it required - considering a full time job?

Besides this. Another amazing post. Thanx!

Collapse
helenanders26 profile image
Helen Anderson Author

Hi Periklis and Davide

Thank you for the nice comments :)

I've been doing these posts for about four months but studying very, very slowly towards the Certification. I'm balancing a bit of study, a full-time job and have been posting here once a week at least for the last year.

I could certainly have gone faster with the study if I wasn't trying to put effort into writing blog posts too. I've got posts on Cloudwatch, CloudFront, Lambda and Dynamo lined up next in between my SQL ones. By then I'm hoping to feel prepared for the exam. Very slow progress indeed :)

A colleague of mine crammed for the Architect Associate exam and passed with high marks after studying for only a couple of months. It just depends on your experience on the tools and if you want to cram it all in. Every blog post I've read has said 120 hours or three months of study is recommended. I've really only had exposure to S3 and Aurora in my day job so have had a steep learning curve as I'm not trained in IT either. Before I headed off on this journey I didn't know much about networking in the conventional sense, let alone the way AWS do things.

As far as Architect v Developer goes, I think it's purely because the content appeals to me more.

For an overview of what is covered in each I really like the Cloud Academy pages:

cloudacademy.com/learning-paths/de...

cloudacademy.com/learning-paths/cl...

Hope that helps!

Thread Thread
perigk profile image
Periklis Gkolias

thank for your prompt reply Helen. Looks like I will need both eventually.

Thread Thread
dvddpl profile image
Davide de Paolis

thanks Helen. it's really amazing what you are doing. working full time, AWS certification and blog posts every week ( i am been writing a blogpost each week since february and it requires a lot of effort and time..). Since I imagine you also have a private life, I am wondering how many hours you sleep? :-)

right now i am pretty busy with the blog and with public speaking training + always a little bit of German + i recently bought a guitar and i would like to learn how to play.. I have definitely too many interests and goals... maybe over the winter or beginning next year i will jump onto the Solutions Architect.. will see .

Thread Thread
helenanders26 profile image
Helen Anderson Author

Sleep is definitely lacking at times :)

I set myself a goal to publish a quality post a week for a year, and while it's been incredibly rewarding I could do with some time to put into the Certification study. I'll be dropping back to once a fortnight after August to get this done by the end of the year. Hopefully.

That's awesome that you are committing to language studies and public speaking training as well. Sometimes I find it hard to narrow down what I want to learn about because it all looks so interesting. About a month ago I realised just how many articles, udemy courses, and videos on were on my to-do list. I've deleted or archived all of them to get rid of the mental pressure that I have all this work to do and no time to do it all.

Thread Thread
dvddpl profile image
Davide de Paolis

yeah well... I moved from Milan (Italy) to Hamburg (Germany) 6 years ago and learning the language is a must - even though I work in a fully English speaking company (actually that is exactly the problem with learning German - I speak English at work, and Italian at home, so exposure to German is very limited) . I try to read and watch as much content as I can in German - not really "deliberate practice" so far.
Yep. Everythings looks so interesting. ( i wrote about it here). At some point, you have to choose and prioritise.
Keep up with it! :-)

Thread Thread
helenanders26 profile image
Helen Anderson Author

Oh wow ... we're on exactly the same page. Great post!