DEV Community: Sandeep Reddy Alalla

Orchestrating a Seamless Data Ingestion Workflow with AWS Step Functions

Sandeep Reddy Alalla — Wed, 21 Feb 2024 16:40:00 +0000

Introduction:

In today's data-driven world, businesses rely heavily on efficiently processing and analyzing vast amounts of data. However, orchestrating complex workflows involving data ingestion, transformation, and analysis can be challenging. Enter AWS Step Functions, a powerful service that allows you to easily coordinate and visualize workflows while simplifying the development and maintenance process. In this tech blog, we will delve into how AWS Step Functions can be employed effectively to orchestrate a seamless data ingestion workflow, ensuring reliable and scalable data processing.

What are AWS Step Functions?

AWS Step Functions is a managed service that enables you to build serverless workflows using a visual interface. It allows you to coordinate multiple AWS services and custom applications, making it ideal for orchestrating data ingestion workflows. With Step Functions, you can create complex workflows with conditional branching, retries, error handling, and parallel processing, all while providing a clear visualization of the entire process.

Data Ingestion Workflow Overview:

Before we dive into Step Functions, let's establish a high-level understanding of the data ingestion workflow we aim to orchestrate. Typically, a data ingestion workflow involves the following stages:

1. Data Collection: Gathering data from various sources like databases, APIs, or external systems.

2. Data Validation/Cleansing: Performing data validation to ensure quality and consistency.

3. Data Transformation: Converting data into a suitable format or structure for downstream processing.

4. Data Storage: Storing processed data into a suitable storage solution like Amazon S3 or a database.

5. Notification: Sending notifications to relevant stakeholders upon workflow completion or failure.

AWS Step Functions Orchestration:

Now, let's explore how AWS Step Functions fits into this data ingestion workflow and enables seamless orchestration:

1. Defining States:

AWS Step Functions allows you to define each stage of the workflow as an individual state. For example, states can include "Collect Data," "Validate Data," "Transform Data," "Store Data," and "Notify Stakeholders." Each state can be associated with specific AWS services or custom code, giving you flexibility in integrating different components.

2. Conditions and Branching:

Step Functions supports conditional branching, allowing you to define different paths based on specific conditions. This feature is particularly useful when handling different data sources or error scenarios. For instance, if data validation fails, the workflow can take an alternative branch for error handling and notifying concerned parties.

3. Error Handling and Retries:

AWS Step Functions provides built-in error handling, enabling you to define how the workflow should handle errors or exceptions. You can configure retries with exponential backoff to handle transient failures in data collection or transformation stages. Additionally, error-handling states can be defined to gracefully handle and recover from failures.

4. Parallelism:

Parallelizing tasks within a workflow can significantly improve processing time. With Step Functions, you can leverage parallel state types, enabling concurrent execution of independent tasks. For example, you can simultaneously validate and transform multiple datasets in parallel, enhancing overall efficiency.

5. Visualization and Monitoring:

One of the key advantages of Step Functions is its visual representation of workflows. You can easily visualize the data ingestion workflow, understand its progress, and identify any bottlenecks or inefficiencies. Step Functions also integrates with AWS CloudWatch, allowing you to monitor and gain insights into the workflow's performance and resource utilization.

Conclusion:

AWS Step Functions simplify the orchestration of complex data ingestion workflows, providing a flexible and scalable solution. With Step Functions, you can easily define, manage, and visualize workflows, offering a clear understanding of data processing stages. Its built-in error handling, parallel execution, and visual monitoring capabilities allow for reliable and efficient data ingestion. By leveraging AWS Step Functions, businesses can streamline their data processing pipelines, leading to improved data quality, reduced manual efforts, and enhanced decision-making capabilities.

References:

- AWS Step Functions Documentation: https://docs.aws.amazon.com/step-functions/

- AWS Step Functions Developer Guide: https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html

Demystifying AWS API Gateway Part01

Sandeep Reddy Alalla — Sat, 29 Jul 2023 14:49:25 +0000

In this series of blog posts, I will cover API gateway in general and particularly AWS API gateway, what it is, how we can use it to the full extent and what are some best practices we can follow while doing so. Before going into the details let's first discuss what is API Gateway.

What is API Gateway in general?

In recent times, microservices have gained a lot of popularity with that the use of API gateway, what is it, why do we need it, and why can't we hit the microservices directly? I am assuming that you know the basics about microservices architecture and monolith architecture, if not here is a good place to know more about them.

so what is an API gateway, this is nothing but a reverse proxy that sits between your microservices or your backend services, it acts like a single point of entry to your backend services, and it offers a lot of functionalities

Single Entry Point: It acts as a single point of entry to the client applications, rather than clients dealing with multiple backend or microservices they can just hit the API gateway which in turn will route the client request to the appropriate microservice.
Request Routing: Based on the route URL, headers and other things API gateway will route the client requests to the correct microservice, it will also act as a load balancer.
Security and Authentication: It can act as a security layer, it is capable to validate the access token by which it can handle authorization and authentication, by this it will handle invalid or unauthorized access to the backend services.
Rate Limiting and Throttling: It can also help implement rate limiting on certain resources, which in turn avoids the** DDoS **attacks or it can avoid overloading of the system by which we can make sure that the system is more stable.
Caching: It can also act as a cache, where it will cache the redundant calls to microservices and serves the response without actually making the call to the microservice, which will improve the latency and performance of the application.
Logging and Monitoring: It can also log incoming requests and responses so that we will have a better understanding of the system behavior, this will help in troubleshooting or future enhancements
Version Management: When introducing changes to microservices, version management can be challenging. The API Gateway can handle versioning and ensure that clients interact with the correct version of the API.

I hope you got a good basic understanding of what API gateway is and how it can be useful, in upcoming posts in this series I will cover how we can achieve all these functionalities using the AWS API gateway.

I used chmod 777 -R / and why you should be careful or avoid doing it

Sandeep Reddy Alalla — Fri, 17 Mar 2023 15:03:00 +0000

As an application developer, I never cared much about Linux commands, but since I am working at a very small company now I get to do all sorts of things that I really enjoy, if you are like me then this article is for you.

I recently ran chmod 777 -R / command on one of our admin AWS EC2 instances, and this resulted in a lot of unwanted and some not-so-good things. I will detail the things that happened and some details about chmod and permissions in general in this article and by the end of this article, you will have a good understanding of these and will execute these commands with some confidence.

As an application developer, I do not have 100% knowledge of Linux systems and how they work internally, but after I did this mistake I did some research and wrote this up so that I might not do the same in the future if you are reading this I might save you as well.

File System, user and user groups in Linux:

Before going into the details let us, in short, discuss file systems, user and user groups in Linux, basically, on Unix based system there are permissions attached to each of the files created, you might have heard like read, write and execute. Based on these permissions a user will have access to these files and there is something called a super user who can grant or revoke access to any files on the file system. This superuser will be particularly useful when a user owns a very important directory and leaves the organization then in this case a superuser can come and change the owner of that directory(if a normal user is permitted to run some commands at the root level then this can be done using sudo.)

About Access Permissions on files

Now let's discuss something related to access that can be set on files, these are either represented with alphabets or numbers

ls -l your_file_name -- Command 
-rw-------@ 1 SRA 4238108 Apr 21  2022 IMG_0237.jpg --- output
# here my file name is IMG_0237.jpg
ls -ld your_folder_name -- Command

So here, the line below ls - l gave us the permissions that we have on the file, which are read and write(rw). if you want to know about a directory we can use ls -ld. you can find more about what ls command can do using man ls command.

About how you can change Permissions, user or user group

Now let's discuss how we can modify permissions are assign a new user or group to a certain folder or a file. This can be done using chown and chmod commands. Firstly, let's discuss chmod and the access codes that come with chmod

r: used to give read permission
w: used to give write permission
x: used to give execute permission
-r: used to remove read permission
-w: used to remove write permission
-x: and you guess it right to remove execute permission

Alternate CodeAlternatively, we can also you numbers to represent these permission.

let says you used ls -l on file and you were not satisfied with the permission on the file then you use chmod.

chmod permission_you_want your_filename
chmod rwx your_filename

Now let's discuss chown, this can be used to change user and or group on a file or directory. let's say as i said earlier a file or folder is owned by someuser that has left the company and we own want to change it to a different user and group.

chown user_name:user_group_name file_or_folder_name

Once you run this command and now want to check the user and group, you can just use ls -l or ls -ld.

Now what is the fuss about -R

Up to this point, we are just either dealt with a folder or a single file, but what will you do if you want to change permission on every single file in a folder and any files in a subfolder in a folder, this is where -R(Recursive) option will come in hand.

chmod -R your_access_modes folder_name
chmod -R 777 your_folder_name/

Why you should be careful

Now let's go into the details of why we should be extra careful while running chmod 777 in general, 777 means you are giving read, write and execute permission to the user which might be ok to give to a single file, but giving to a complete folder is too risky but can be accepted to an extent but the command I ran

chmod -R 777 /

This is too risky and I think should be avoided and should be never run especially if you are an application developer, what this does is it changes permissions on every single file under root, which is every single file on your Linux machine, this gives read, write and execute permission to all the files to all the user who have access to that machine, this is a very serious security issue(these issues can be an article in itself so I am not discussing them here.). In my case how it effected us is, the machine I ran this command on, we use to run our jenkins and it impacted your CI/CD pipeline for almost two days, it messed up with the sudo permissions that other users have on this machine. In simple words it got the attention of multiple people and multiple people had to work on fixing it, so that was the impact of it in simple words.

In Conclusion

But yeah you don't have to worry about it if you already ran it, there is nothing you can do about it other than fixing it. This is what my manager said to me when I confessed that I am the reason for the pipeline being done and I quote his exact words.

I've done worse over my career -- if someone hasn't messed something up in Linux, they probably are not doing enough development.

I felt very good after reading these words but at the same time felt that I should be extra careful before doing something like this again. So, that is it for this article hope i help someone out there by writing this up, and if you want to know more about these commands I discussed above there are some great articles in fact, I read those before writing this article.

why-is-chmod-r-777-destructive

change-folder-and-content-permissions

advanced-file-permissions