DEV Community: Mohammad Fuzail

Modern Cloud Infrastructure as Code with Sceptre: Building and Deploying a Simple Python Web Service

Mohammad Fuzail — Wed, 16 Jul 2025 09:25:10 +0000

As cloud-native architectures become the norm, managing infrastructure as code (IaC) is no longer a luxury, it's a necessity. While tools like the Serverless Framework are popular for abstracting infrastructure, Sceptre stands out for its tight integration with AWS CloudFormation and its flexibility in complex, real-world use cases.

In this post, we’ll:

Use Sceptre to provision infrastructure
Build a simple Python service
Containerize it with Docker
Set up Jenkins for CI/CD
Briefly explore why Sceptre may be preferable to the Serverless Framework

Why Sceptre over Serverless Framework?

Sceptre is especially suited for teams who:

Need infrastructure flexibility (beyond Lambda/APIs)
Want full access to AWS resource definitions
Deploy to multiple environments (dev, test, prod) in a structured way

Step 1: Write a Simple Python Script

Here’s the application logic — a basic Python script to be run in the cloud:

#script.py
import time

print("Python task started...")
time.sleep(3)
print("Task completed successfully!")

Use cases:

Automation tasks
Data processing
Scheduled jobs
Infrastructure hooks

Step 2: Dockerize It

To run this in ECS, we need to containerize the script:

# Dockerfile
FROM python:3.10-slim

WORKDIR /app
COPY script.py .

CMD ["python", "script.py"]
Build and test:
docker build -t python-task .
docker run python-task

Step 3: Use Sceptre to Deploy to ECS Fargate

Sceptre lets us manage AWS infrastructure using CloudFormation templates and environment-specific configurations.

Directory Structure infra/ ├── config/ │ └── dev/ │ └── python-task.yaml └── templates/ └── ecs-task.yaml

#config/dev/python-task.yaml
template_path: templates/ecs-task.yaml
stack_name: python-task
parameters:
  TaskName: python-script-task
  ContainerImage: <YOUR_ECR_IMAGE_URL>

templates/ecs-task.yaml (CloudFormation)
Resources:
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Ref TaskName
      Cpu: '256'
      Memory: '512'
      RequiresCompatibilities: [FARGATE]
      NetworkMode: awsvpc
      ExecutionRoleArn: arn:aws:iam::<ACCOUNT_ID>:role/ecsTaskExecutionRole
      ContainerDefinitions:
        - Name: python-container
          Image: !Ref ContainerImage
          Essential: true
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: /ecs/python-task
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: python-task

Deploy with Sceptre :

cd infra sceptre launch dev/python-task.yaml

Once launched, you can run this task using AWS CLI or trigger it via EventBridge for scheduled executions.

Step 4: Automate Deployment with Jenkins

Here's a minimal Jenkinsfile to automate the entire flow:

pipeline {
  agent any
  environment {
    ECR_REPO = "<ACCOUNT_ID>.dkr.ecr.<REGION>.amazonaws.com/python-task"
  }
  stages {
    stage('Build') {
      steps {
        sh 'docker build -t python-task .'
      }
    }
    stage('Push to ECR') {
      steps {
        sh '''
          aws ecr get-login-password | docker login --username AWS --password-stdin $ECR_REPO
          docker tag python-task:latest $ECR_REPO:latest
          docker push $ECR_REPO:latest
        '''
      }
    }
    stage('Deploy Infra') {
      steps {
        dir('infra') {
          sh 'sceptre launch dev/python-task.yaml'
        }
      }
    }
  }
}

One push to Git, and Jenkins takes care of:

Building your container
Publishing it to ECR
Updating your ECS TaskDefinition via Sceptre

Summary

You now have a repeatable, production-grade deployment pattern for running Python tasks in AWS with full control and no unnecessary abstraction.
Sceptre is not a replacement for Serverless Framework, but an empowering alternative when you want low-level control and native AWS CloudFormation integration. For teams who treat infrastructure as a first-class citizen, Sceptre is a great fit.

Useful References :

Disclaimer: This is a personal blog. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.

Navigating Concurrency for Large-Scale Systems

Mohammad Fuzail — Tue, 19 Nov 2024 05:35:57 +0000

Introduction

As a developer, one of the most exciting challenges I face is developing large-scale systems that can handle massive workloads efficiently. In this blog post, I'll share insights on leveraging Python's concurrent programming features to build robust, high-performance systems while optimizing resource usage.

Understanding Concurrency Options in Python

Python offers three main approaches to concurrency: asyncio, multithreading, and multiprocessing. Each has its strengths and use cases, and choosing the right one is crucial for system performance.

Asyncio: Event-driven, single-threaded concurrency

Multithreading: Concurrent execution within a single process

Multiprocessing: Parallel execution across multiple CPU cores

Choosing the Right Concurrency Model

The decision between asyncio, multithreading, and multiprocessing depends on the nature of your workload:

Asyncio: Ideal for I/O-bound tasks with many concurrent operations, such as handling numerous network connections or file operations.

Multithreading: Suitable for I/O-bound tasks where you need to maintain shared state or work with libraries that aren't asyncio-compatible.

Multiprocessing: Best for CPU-bound tasks that require true parallelism and can benefit from utilizing multiple cores.

Determining the Optimal Number of Workers

Finding the right number of workers (threads or processes) is crucial for maximizing performance without overwhelming system resources. Here are some guidelines:

For I/O-bound tasks (asyncio or multithreading):

Start with number of workers equal to 2-4 times the number of CPU cores.

Gradually increase and monitor performance improvements.

Stop increasing when you see diminishing returns or increased resource contention.

For CPU-bound tasks (multiprocessing)

Begin with a number of workers equal to the number of CPU cores.

Experiment with slightly higher numbers (e.g., number of cores + 1 or 2) to account for any I/O operations.

Monitor CPU usage and adjust accordingly.

Practical Example: Log Processing System

Let's consider a large-scale log processing system that needs to handle millions of log entries per day. This system will read logs from files, process them, and store the results in a database.

Here's how we might approach this using Python's concurrency features:

Log Reading (I/O-bound): Use asyncio for efficient file I/O operations. This allows us to read multiple log files concurrently without blocking.

Log Processing (CPU-bound): Use multiprocessing to parallelize the actual processing of log entries across multiple CPU cores.

Database Storage (I/O-bound): Use multithreading for database operations, as most database libraries are not asyncio-compatible but can benefit from concurrent access.

Putting it all together

This example demonstrates how we can leverage different concurrency models in Python to build a large-scale system that efficiently handles I/O-bound and CPU-bound tasks.

Conclusion

Building large-scale systems with Python requires a deep understanding of its concurrency features and how to apply them effectively. By carefully choosing between asyncio, multithreading, and multiprocessing, and optimizing the number of workers, we can create systems that make the best use of available resources and scale to handle massive workloads.

Remember, there's no one-size-fits-all solution. Always profile your application, experiment with different approaches, and be prepared to adjust your design based on real-world performance data. Happy scaling!

Disclaimer

This is a personal [blog, post, statement, opinion]. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.

Leveraging Serverless To Build Scalable Soluions

Mohammad Fuzail — Fri, 06 Jan 2023 09:44:52 +0000

Introduction

Serverless is a cloud computing application development and execution model that enables developers to build and run application code without provisioning or managing servers or backend infrastructure.

A serverless provider allows users to write and deploy code without worrying about the underlying infrastructure. A company that gets backend services from a serverless provider is charged based on their computation and do not have to reserve and pay for a fixed amount of bandwidth or number of servers, as the service is auto-scaling.

Note: Despite the name serverless, physical servers are still used but developers do not need to be aware of that.

So, to leverage serverless computing, we will be exploring and implementing serverless framework in this blog.

What is serverless framework?

Serverless framework is an open-source web framework written using Node.js which enables us to build, package and deploy scalable applications without worrying about provisioning and managing infrastructure. It supports different Public Cloud vendors and satisfies the definition of an Infrastructure As A Code.

It's a CLI that offers structure, automation and best practices out-of-the-box, allowing you to focus on building sophisticated, event-driven, serverless architectures, comprised of Functions and Events.

Functions :

The code of a serverless application is deployed and executed in AWS Lambda functions.
Each function is an independent unit of execution and deployment, like a microservice. A function is merely code, deployed in the cloud, that is most often written to perform a single job such as:

• Saving a user to the database
• Processing a file in a database
• Performing a scheduled task

Events :

Functions are triggered by events. Events come from other AWS resources, for example:
• An HTTP request on an API Gateway URL (e.g. for a REST API)
• A new file uploaded in an S3 bucket (e.g. for an image upload)
• A CloudWatch schedule (e.g. run every 5 minutes)
• A message in an SNS topic
• A CloudWatch alert
• And more...

When you configure an event on a Lambda function, Serverless Framework will automatically create the infrastructure needed for that event (e.g. an API Gateway endpoint) and configure your functions to listen to it.

The entry point of a Serverless Framework application is the serverless.yml file in a source root that describes the serverless application in its own syntax. Just like an AWS SAM template, it allows configuring the whole serverless application. It also has its own blueprints that can be used to bootstrap your own application.
Check more details here.

Why serverless framework?

The Serverless Framework is different from other application frameworks because:

• It manages your code as well as your infrastructure
• It supports multiple languages (Node.js, Python, Java, and more)
• It supports different cloud providers which supports function as a service.

We should focus on developing the business logic and let the infrastructure related stuff abstracted and handled by a managed service.

How to leverage serverless framework?

Let’s use serverless framework to develop and deploy a simple serverless API using AWS Lambda with Python.
Since serverless framework is a node.js CLI tool, at first we need to install Node.js
Now, just to verify Node.js installation use version check command.
Note: Serverless runs on Node v6 or higher

node -v

Now, the next step would be to install serverless framework using following command :

npm install -g serverless

We can verify the serverless framework installation by using version check command.

serverless -v

Next, we need to have an AWS account and an AWS profile is setup for the same. If not, you can use this guide. to set it up.

Basically, you need to create an IAM user and access role to secure the service deployment on AWS. Here is how we can config AWS credentials :

serverless config credentials --provider aws --key <AccessKey> --secret <secretKey> --profile <profile name>

Now, let’s create a python service using the template available in serverless framework. Use following command with service name as “backend” here :

serverless create --template aws-python --path backend

To see a list of available templates run serverless create --help
Most commonly used templates are:

• aws-clojurescript-gradle
• aws-clojure-gradle
• aws-nodejs
• aws-nodejs-typescript
• aws-alexa-typescript
• aws-nodejs-ecma-script
• aws-python
• aws-python3
• aws-ruby
• aws-provided

Note: To check the available templates, follow this link.

This will create two files in backend project directory , one serverless.yml and another one as handler.py, we can see below :

The serverless.yml file includes all the configurations related to cloud resources, language and functions. While handler.py have codes or logic related to lambda function.

Since we have the boilerplate for both the files created, let’s modify these files to create a simple API which gives us the product of two number.

We will start with the serverless.yml file :

• First, define the name of the service you want to create: mybackendservice (This should be a unique service name for a particular serverless account).
• If you want to monitor the serverless project in the serverless dashboard(dashboard.serverless.com), you can mention the app and org.
• Now, we have to mention the configuration of the cloud provider, resources and permissions. In our case, we are using AWS as our cloud service provider.
• At last, we include the lambda function configurations. In our case, we will define a POST API.

Note : For more details on provider configurations, please visit here

Here, we can see the configurations added for POST API in serverless.yml file.

Note: Add app/org name same as in your serverless dashboard.

Now, let’s add the service logic in handler.py file.

We need to write a python function which returns product of two numbers in the json response.

Since, we are done with the changes. Let’s deploy our service using below command :

serverless deploy

once we execute this command it will be provisioning all the necessary resources in AWS and deploy our code.

Here, we can see after deployment, we got dashboard link to our app and the endpoint URL.

Invoking A Lambda Function :

Events are the things that trigger your functions to run.

If you are using AWS as your provider, all events in the service are anything in AWS that can trigger an AWS Lambda function, like an S3 bucket upload, an SNS topic, and HTTP endpoints created via API Gateway.

Other way to invoke a lambda function is using CLI:

serverless invoke [local] --function functionName

It invokes a deployed function. You can send event data, read logs and display other important information of the function invocation.

check this link for more details about invocation

Logs :

One way to get the logs of a lambda function invocation is the AWS cloudwatch which we can access for a particular log set through monitoring tab in AWS Console.

Another way is using CLI :

serverless logs -f hello

This lets you watch the logs of a specific function. This command returns as many log events as can fit in 1MB (up to 10,000 log events).

This will fetch the logs from last 10 minutes as startTime was not given.

For more details on logs, check this link

Now, let’s try out our API service using Postman.

Here, we are using an HTTP request as an event to invoke serverless lambda function which we have developed.

We can see here, that the service is working as expected. So, now it’s your turn. Write your own service using serverless.

Removing A Serverless App :

The sls remove command will remove the deployed service, defined in your current working directory, from the provider.

Use this command from your CLI :

serverless remove --stage dev --region us-east-1

This example will remove the deployed service of your current working directory with the stage “dev” and the region “us-east-1”.

The alternate way is to delete the CloudFormation Stack corresponding to your service using AWS Console.

Final Thoughts:

This was a very basic example, but we can create a full stack and very sophisticated application as well with ease using serverless. So, with serverless we as developer can focus mostly at the logic part without caring about configuring and managing infrastructures. Serverless computing is evolving day by day and it’s adoption is also growing. This basic understanding will help you to develop and deploy your own service using serverless.

References:

• https://tinyurl.com/4sxwwwfs
• https://tinyurl.com/mtdruy88

Disclaimer: