DEV Community: Jorge Contreras

Understanding MCP Servers: The Model Context Protocol Explained

Jorge Contreras — Fri, 09 May 2025 14:01:39 +0000

As AI technologies continue to evolve, developers are constantly seeking ways to enhance and streamline interactions with large language models (LLMs). One significant advancement in this space is the Model Context Protocol (MCP), recently introduced by Anthropic, and its implementation through MCP servers. In this article, I'll share my findings and experiences with MCP servers, exploring what they are, why they matter, and how you can get started with them.

What is the Model Context Protocol?

The Model Context Protocol (MCP) is a standardized communication protocol that defines how applications interact with language models while efficiently managing context. It was designed to address the limitations of traditional API interactions with LLMs, particularly around context management.

The protocol enables developers to create applications that can have more natural, ongoing conversations with AI models without constantly hitting context limitations.

What is an MCP Server?

An MCP server is a specialized server that implements the Model Context Protocol, designed to enhance interactions with LLMs by managing contextual information more efficiently. In essence, it serves as a middleware between your application and language models, providing optimized context handling, memory management, and state persistence across interactions.

Unlike traditional API calls to language models where context is handled within each request, an MCP server maintains context as a first-class citizen, allowing for more sophisticated and efficient AI interactions.

Why Would I Need an MCP Server?

The need for MCP servers arises from several limitations in standard LLM interactions. Here is a comparison between traditional LLM calls and MCP Server Approach.

Traditional LLM Calls 🆚 MCP Server Approach

Feature	Traditional LLM API Calls	MCP Server Approach
Context Management	Manually append full history; prone to overflows	Context is intelligently summarized and pruned by the server
Memory Efficiency	Sends entire conversation each time	Only sends what's relevant, with long-term memory persistence
Cost Optimization	High token usage due to repeated context	Reduced tokens via smart context windows and state reuse
Enhanced Functionality	Limited to basic Q&A or single-turn logic	Supports retrieval-augmented generation, tool use, and multi-turn sessions
Consistent Performance	Degrades with long interactions or context resets	Maintains consistent response quality via controlled context

If you're building applications that require ongoing conversations with AI models, persistent memory, or sophisticated context management, an MCP server could be invaluable to your architecture.

What Makes a Server an MCP Server?

A server becomes an MCP server when it implements the Model Context Protocol specifications. Key characteristics include:

Context Management: Implements strategies for context maintenance, summarization, and relevance determination.
Protocol Implementation: Adheres to the MCP API specifications for communication with clients.
State Persistence: Maintains conversation state across multiple interactions.
Memory Management: Implements strategies for what to remember, summarize, or forget.
Message Routing: Efficiently routes messages between clients and language models.
Optional Features: May include retrieval-augmented generation, tool use, or other enhanced capabilities.

The core differentiator is the server's implementation of the protocol's standards for handling context as a managed resource rather than merely passing messages.

What Company or Organization is Behind It?

The Model Context Protocol was created by Anthropic, the AI research company behind the Claude language model. They introduced MCP to provide developers with a standardized way to build applications that could more effectively maintain conversations with their AI models.

Anthropic has made the MCP specification open for implementation, encouraging the development of various server implementations and client libraries across the ecosystem.

What are the Known Limitations of MCP Servers?

Despite their advantages, MCP servers do have several limitations:

Implementation Complexity: Setting up and maintaining an MCP server requires more infrastructure than simple API calls.
Latency Concerns: The additional processing layer can introduce latency in some implementations.
Resource Requirements: Running an MCP server requires additional computational resources compared to direct API calls.
Standardization Challenges: As the protocol evolves, different implementations may support different feature sets.
Model Compatibility: Not all language models are fully optimized for MCP-style interactions.
Security Considerations: Persistent state management introduces additional security concerns regarding data storage and access.
Cost Structure: While potentially reducing token usage, running the server itself adds operational costs.

These limitations don't diminish the value of MCP servers but are important considerations when deciding whether to implement one in your AI architecture.

Are There Similar Alternatives That Solve the Same Kind of Problem?

Several alternatives address similar context management challenges:

LlamaIndex: Offers advanced retrieval and indexing capabilities for managing context with LLMs.
Custom RAG Solutions: Many teams build custom retrieval-augmented generation systems that manage context through vector databases.
Context Compression Techniques: Some solutions focus on compressing context rather than managing it through a protocol.
Function Calling Frameworks: Frameworks that orchestrate complex interactions between models and tools.

🛠️ Function Calling Frameworks for LLMs

Framework / Tool	Description	Ecosystem	Good For
OpenAI Function Calling	Native support for structured function execution via JSON schema	OpenAI API	Simple apps, single-agent LLM + tools setup
LangChain	Python/JS framework to build chains, tools, and autonomous agents with function calling	Open-source	Agent workflows, tool orchestration, RAG
Semantic Kernel	Microsoft’s SDK for building AI copilots using plugins, skills, and memory	Microsoft / Azure	Enterprise apps, plugin-based agent design
AutoGen	Multi-agent system framework from Microsoft for LLM collaboration	Open-source (MS Research)	Agent collaboration, task decomposition
CrewAI	Orchestrates LLM agents as a “crew,” each with a role and tools	Open-source	Role-based multi-agent workflows
Google Vertex AI Extensions	Allows function/tool integration in Vertex AI models (tools like web search, APIs)	Google Cloud Platform	GCP-native AI tool chaining
Hugging Face Transformers Agents	Tool-using agents powered by Hugging Face pipelines	Hugging Face	Open-source experimentation, local models

The main difference is that MCP represents a standardized protocol specifically designed for context management, while many alternatives are frameworks or techniques that can be used to implement similar functionality but lack the standardization benefits.

How Can I See an MCP Server in Action?

To experience an MCP server firsthand, you can take a look at these amazing resources:

Github
MCP.so

To get a quick feel for it, I recommend starting with a simple local implementation that connects to a model you already have access to, such as Claude via Anthropic's API.

Can I Build My Own MCP Server and What Would I Need?

Yes, you can build your own MCP server with the following requirements:

Technical Requirements:

Programming Knowledge: Familiarity with a language suitable for server implementation (Python is common).
API Access: Access to an LLM API (Anthropic's Claude, OpenAI's GPT, etc.). Or if you prefer to run the LLM locally you should definitely checkout ollama
Server Environment: A system to host your server (can be local for development).
Database: For persistent storage of conversation history and context (PostgreSQL, MongoDB, etc.).
Understanding of Protocol: Familiarity with the MCP specification.

Implementation Steps:

Start with one of the reference implementations or starter repositories.
Set up your development environment and dependencies.
Configure your model API credentials.
Implement the core protocol endpoints.
Add context management strategies based on your use case.
Test with simple client applications.
Optimize for your specific requirements.

Building a basic MCP server can be accomplished in a weekend for experienced developers, but creating a production-ready implementation with advanced features will require more significant investment.

Links

https://www.anthropic.com/news/model-context-protocol
https://modelcontextprotocol.io/introduction
https://mcp.so/
https://www.serverless.com/framework/docs/guides/mcp
https://github.com/modelcontextprotocol/servers

Building AWS Lambda Functions

Jorge Contreras — Sun, 23 Mar 2025 15:58:57 +0000

From your Local to the Cloud

Building software is one of the most rewarding creative activities you can do. One the reasons I love building tech solutions is that you can create awesome stuff with little resources. But building is not always straightforward. The process of creating a great piece of software involves testing, a lot.

In order to build something that solves a real problem and becomes robust, it needs to go through a lot of changes, adjustments, improvements and tuning. For this reason, it is crucial to have a quick and easy way to make changes and see the result as fast as possible. Developing locally reduces this feedback loop and you can iterate faster.

Now enter AWS Lambda. Suddenly, testing locally isn't so simple anymore. Your code is running in the cloud, inside an AWS-managed environment that you don't control. And if your Lambda interacts with other AWS services like S3, DynamoDB, or API Gateway, things get even trickier.

I ran into this problem recently while working on a project, and I had to dig deep to find a good local testing approach. If you're struggling with this too, let me save you some time and share what I've learned.

So, How Do You Test AWS Lambda Locally?

Option 1: AWS SAM – The Official Way

My first stop was AWS SAM (Serverless Application Model), which is AWS's official tool for local development. With SAM, you can invoke a Lambda function on your machine using sam local invoke, or even spin up a local API Gateway using sam local start-api. This means you can test API calls before deploying to AWS, which is super useful.

But there's a catch. Since SAM relies on Docker to simulate the AWS environment, it can feel a bit heavy at times. Plus, setting it up can take a while if you're new to it. Still, if you want an AWS-supported way to test locally, SAM is a solid option.

If you are interested in learning more about this approach, I would recommend to watch this awesome tutorial, where Jonathan goes through the whole development and deployment cycle.

Option 2: LocalStack – Running AWS on Your Machine

Then I came across LocalStack, which is basically a fully emulated AWS environment that runs entirely on your local machine. This approach is quite interesting. If your Lambda function interacts with S3, DynamoDB, or SQS, you can test everything without even touching AWS.

I spun up LocalStack, deployed my Lambda, and boom! my function was running locally as if it were in the cloud. But again, there's a tradeoff. The free version has some limitations, and running a full AWS-like environment locally can be resource-intensive.

Option 3: AWS Lambda Runtime Interface Emulator

Another option I explored was the AWS Lambda Runtime Interface Emulator. This is an official AWS tool that lets you run Lambda inside a Docker container, mimicking the AWS runtime. It's particularly useful if you're deploying container-based Lambdas.

While this was helpful, I found that for most of my use cases, I didn't need to go that deep. If you're working with Lambda containers, though, this is worth checking out.

Option 4: The Good Old Mocking Approach

Sometimes, the simplest solution is the best one. Instead of running a full AWS stack locally, I started writing unit tests with mocked AWS SDK responses. Using tools like moto (for Python) or aws-sdk-mock (for Node.js), I could simulate DynamoDB, S3, and other AWS services without spinning up anything locally.

This approach made my tests run way faster, and it worked great for logic validation. Of course, it didn't fully replace integration testing, but for quick feedback, it was a lifesaver.

So, What’s the Best Approach?

Honestly, there's no one-size-fits-all solution. For me, a combination of AWS SAM for local execution and unit tests with mocks gave me the best of both worlds. I could run real-world tests when needed but still keep my feedback loop fast.

I'm curious: how do you test your Lambda functions locally? Are you using one of these methods, or do you have a different approach? Let's chat in the comments! 🚀

AWS Lambda Performance Tuning: How to Reduce Cold Starts

Jorge Contreras — Wed, 12 Mar 2025 21:52:59 +0000

Hey fellow builders! If you've worked with AWS Lambda, you've probably run into cold starts. Yes! those annoying delays when your function spins up after being idle. While AWS is great at handling infrastructure for us, cold starts can slow things down, especially for latency-sensitive apps.

So, how do we reduce them? Let's dive in!

What is a cold start? 🥶

When a Lambda function is invoked after being idle, AWS needs to:

Spin up a new execution environment
Load your function’s code
Initialize dependencies

This whole process takes milliseconds to seconds, which might not seem like much, but it's an eternity from the user's perspective. As customer-obsessed builders, we cannot let that be!

Ok so, cold starts are bad. What can we do about it?

As with any cloud computing challenge, there are multiple approaches, each with its own trade-offs. Let's explore a few ideas, one of them might be the perfect fit for your use case.

1. Provisioned concurrency.

This is the most straightforward solution. AWS introduced Provisioned Concurrency to keep function instances warm. It preloads execution environments so your function is ready to go instantly. It is perfect for low-latency, high traffic workloads. However, it will incur an extra cost, so keep that in mind.

2. Optimize Function Memory & CPU.

Finding the right balance between the allocated resources should be an art form. You can experiment and make adjustments based on observations. Luckily, there are awesome folks out there who have paved the way for us. Checkout this AWS Lambda Power Tuning tool. In this video, Daniel clearly explains how to use it.

3. Keep your deployment package small.

AWS Lambda layers let you package dependencies separately from your function code. Instead of bundling large libraries (like bcrypt or Puppeteer) inside your deployment package, you can put them in a layer and reference them in multiple functions.

✅ Benefits:

Smaller function size (faster cold starts 🚀)
Easier updates (update the layer instead of redeploying everything)
Code reuse (share layers across multiple Lambdas)

Example: A Node.js layer with dependencies

Create a nodejs folder and install packages inside it:

mkdir -p nodejs && cd nodejs
npm install sharp
cd ..
zip -r layer.zip nodejs

Upload layer.zip to AWS Lambda as a new layer and add the layer to your Lambda function in the AWS Console or using AWS CLI.

4. Avoid Heavy Initialization.

A common mistake in Lambda development is initializing database connections or loading heavy dependencies inside the handler. This slows down every invocation! Instead, move initialization outside the handler so that it runs only once per execution environment.

❌ Don't do this! (DB connection inside the handler)

const { MongoClient } = require("mongodb");

exports.handler = async (event) => {
  const client = new MongoClient(process.env.MONGO_URI);
  await client.connect(); // Establishes a new connection on every invocation ❌

  const db = client.db("mydb");
  const collection = db.collection("users");
  const users = await collection.find().toArray();

  return { statusCode: 200, body: JSON.stringify(users) };
};

✅ Do this! (Persistent DB connection & Lazy Loading)

const { MongoClient } = require("mongodb");

let client; // Store DB connection outside the handler

async function getDBClient() {
  if (!client) {
    client = new MongoClient(process.env.MONGO_URI);
    await client.connect();
  }
  return client;
}

exports.handler = async (event) => {
  const dbClient = await getDBClient();
  const db = dbClient.db("mydb");
  const collection = db.collection("users");

  return {
    statusCode: 200,
    body: JSON.stringify(await collection.find().toArray()),
  };
};

Why is this better?

The connection is reused across multiple invocations.
Lazy loading (getDBClient()) ensures the connection initializes only when needed.

5. Warmers

A warmer is a simple mechanism that sends a request to your Lambda function at regular intervals, preventing it from being idle. It's like giving your lambda a nudge just to keep it awake. You can use CloudWatch Events to run the warmer call every 5 minutes, for example.

Wrapping Up

Cold starts are annoying but manageable with the right strategies. If you need instant response times, Provisioned Concurrency is your best bet. Otherwise, keeping things lightweight, optimizing resources, and using warmers can make a big difference. Your users will thank you.

What tricks have worked for you? Drop your thoughts in the comments! 💬👇

AWS Machine Learning Certified? Let’s Do This!

Jorge Contreras — Fri, 07 Feb 2025 13:00:00 +0000

Whether you're a seasoned machine learning pro or someone just starting to dip your toes into the field, the AWS Certified Machine Learning - Specialty exam is a whole different beast. AWS offers an incredible array of ML tools and services—but figuring out how they all fit together can feel overwhelming.

With the vast amount of available material, figuring out what's truly important versus what's just "nice to know" feels like a puzzle in itself.

In this series, I'll break things down step-by-step, cutting through the noise and focusing on what really matters for the exam. Let's create a roadmap together that makes tackling this certification feel manageable, not overwhelming.

Why This Certification?

AWS certifications are more than just badges of honor; they're proof that you know your stuff. But why did I pick the Machine Learning - Specialty exam out of all the AWS certifications? A few reasons:

Machine Learning is everywhere:

ML is shaping everything—from personalized recommendations to autonomous cars. This certification is a chance to dive into AWS's take on it and how they power real-world ML solutions.

Real-World Impact:

AWS isn't just for passing exams. It's a powerful ecosystem used in production systems across industries. Mastering these tools gives you an edge in delivering scalable, secure, and effective ML solutions.

Career Edge:

Whether you're a data scientist, engineer, or just curious about ML, this certification proves you're ready to take on complex ML challenges. It's a way to stand out in a competitive field.

Hands-On Fun:

Certifications aren't just about theory—they push you to use tools, services, and techniques that you might not have explored otherwise. And trust me, there's nothing more satisfying than spinning up a SageMaker model that actually works.

Joining the Community:

One of the most underrated perks of getting certified is the community. AWS certifications are a conversation starter—they're a way to connect with like-minded people who share your passion for machine learning and cloud technologies. Whether it's at meetups, on forums, or through LinkedIn, being certified signals to others that you speak the same language and are serious about your craft.

What You Need to Know Before Starting

The certification is a journey, you should take your time and be patient. Here's what helps to have under your belt:

Fundamental ML Concepts:

Familiarize yourself with the following concepts and algorithms:

Linear Regression
Logistic Regression
Decision trees
Clustering (K-means)
Gradient Boosting
Random Forest
K-nearest neighbors (KNN)
Support Vector Machines
Neural Networks
Convolutional Neural Networks

Python Skills:

Many AWS ML tools lean on Python, so familiarity with the language is essential. Bonus points if you've dabbled in libraries like Pandas, NumPy, or Scikit-learn.

AWS Basics:

If you've set up an S3 bucket or launched an EC2 instance before, you're already ahead. Understanding IAM roles, permissions, and basic cloud concepts will help.

What This Series Will Cover

I've planned this series to be your friendly guide through the topics you'll need to master. Expect posts on:

Core AWS ML services like SageMaker, Rekognition, Polly, and more.
How to process and prepare data for ML models on AWS.
Tips for training, tuning, and deploying models.
Insights from my own study process and hands-on labs.

Think of this as part blog, part study group, and part diary of my experience. If you're on the same journey or thinking about it, feel free to share your thoughts, questions, or tips in the comments.

What's Next?

In the next post, we'll dive into the heart of AWS ML: SageMaker. We'll break down what it is, why it's so powerful, and how you can start using it right away.

Final Thoughts:

If you've been waiting for the right time to get started, this is your sign. AWS Machine Learning certification isn't just a test—it's a chance to level up your skills and explore a fascinating field.

Ready? Let’s do this together!

Kubernetes CRDs: The Backbone of Kubernetes Extensibility

Jorge Contreras — Tue, 03 Dec 2024 13:01:00 +0000

Kubernetes is a highly extensible platform, and at the core of its extensibility lies the concept of Custom Resource Definitions (CRDs). CRDs empower developers to extend Kubernetes' capabilities by creating custom resource types that operate just like native Kubernetes objects. In this article, we’ll explore what CRDs are, why they’re important, and highlight some real-world examples of how they’re used.

What Are CRDs (Custom Resource Definitions)?

Kubernetes is already packed with built-in resources like Pod, Deployment, and Service. But what if you need to manage something Kubernetes doesn’t natively understand—like a database, an application workflow, or a set of cloud resources?

Enter CRDs!

A CRD lets you create and manage custom resource types, allowing Kubernetes to recognize and handle them just like its native objects. By registering a CRD in your cluster, you’re essentially saying:

“Hey Kubernetes, here’s a new resource type. Treat it like one of your own.”

For example, if you define a Database CRD, you can create resources like Postgres or MySQL directly in Kubernetes, complete with their own schemas and validation rules.

Why Should You Care About CRDs?

Alright, here’s the deal: CRDs make Kubernetes amazingly flexible.

With CRDs, you can:

Add New Features: Need Kubernetes to manage your app-specific workflows? No problem.

Stay Declarative: Manage everything using YAML—just like the built-in stuff.

Automate All the Things: Pair your CRD with a custom controller (or operator) to handle the heavy lifting for you.

Basically, CRDs let you turn Kubernetes into your personal Swiss Army knife for app and infrastructure management.

Let's see an example!

Here’s a quick example of a CRD that defines a custom resource type called Database.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.awesomeproject.com
spec:
  group: awesomeproject.com
  names:
    kind: Database
    plural: databases
    singular: database
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                engine:
                  type: string
                version:
                  type: string
                storage:
                  type: integer

After applying this CRD, Kubernetes knows how to handle a Database resource. You can now create custom databases using YAML—how cool is that?

Are they common?

You bet they are! CRDs are everywhere in the Kubernetes ecosystem. Here are some popular ones that you might bump into:

1. Cert-Manager

Automates the boring stuff around TLS certificates.

CRDs: Certificate, Issuer, ClusterIssuer
Example: Automatically get certificates from Let’s Encrypt.

2. ArgoCD

Powers your GitOps workflows (syncing Git to your cluster).

CRDs: Application, AppProject
Example: Define an application that ArgoCD will deploy from Git.

3. Prometheus Operator

Makes monitoring a breeze by automating Prometheus setup.

CRDs: ServiceMonitor, PodMonitor, Alertmanager
Example: Define which services Prometheus should monitor.

CRDs in Action: How They Work

When you combine CRDs with a custom controller or operator, things get really exciting. Imagine this:

You define a Database custom resource to manage your Postgres instance.
A controller watches for changes to Database resources.
The controller handles all the nitty-gritty details—provisioning storage, creating users, setting up backups, you name it!
This is how tools like ArgoCD and Cert-Manager work behind the scenes. CRDs let Kubernetes handle the what, while the controller handles the how.

Wrapping Up

CRDs are what make Kubernetes extensibility easy and fun. They let you manage almost anything: databases, certificates, cloud resources, with Kubernetes’ familiar declarative style. Whether you’re using popular tools like Istio and Prometheus Operator, or building your own custom workflows, CRDs are a must-know feature.

AWS Step Functions: Your Workflow's Best Friend

Jorge Contreras — Fri, 29 Nov 2024 13:00:00 +0000

Let’s talk about automation because, let’s face it, nobody has time to babysit workflows. That’s where AWS Step Functions comes in. Think of it as the air traffic controller for your applications, making sure every task gets done in the right order, at the right time, and without you breaking a sweat.

In this article, we’ll explore what Step Functions is, why it’s so handy, and how you can use it to orchestrate all kinds of workflows—whether you’re processing data, training machine learning models, or just keeping your microservices from stepping on each other’s toes.

What Exactly Are AWS Step Functions?

AWS Step Functions is like a super-organized assistant for your cloud applications. It helps you manage workflows by breaking them down into individual tasks and stringing them together in a way that makes sense. You can think of it as flowcharts that actually do work.

Rather than writing custom code to handle retries, errors, and task coordination, Step Functions takes care of all of that for you. You just tell it what to do (using a JSON file in Amazon States Language) and let it run the show.

Why are they so useful?

Step Functions can save you a ton of time and hassle. Here’s how:

1. Tames Complexity

Got a bunch of services that need to work together? Step Functions makes them play nice by managing the sequence, timing, and decisions between tasks.

2. Grows with You

Whether you’re running 10 workflows or 10,000, Step Functions scales automatically. And because it’s serverless, you don’t have to worry about managing infrastructure.

3. Keeps Costs in Check

You only pay for what you use. Every time a task moves from one state to the next, that’s a state transition—and that’s all you’re billed for. No hidden costs here.

4. Helps You Sleep at Night

With built-in monitoring and logging through AWS CloudWatch, you can keep tabs on your workflows and troubleshoot without the guesswork.

Understood but, where can Step Function be applied in the real world?

Step Functions shines in pretty much any scenario where tasks need to happen in a certain order (or at the same time). Here are a few examples:

1. Data Processing Pipelines

Orchestrate an ETL workflow: pull raw data from S3, transform it with AWS Glue, and load it into a database.

2. Machine Learning Workflows

Automate model training with SageMaker: preprocess the data, train the model, and deploy it—without lifting a finger.

3. Microservices Coordination

Tie together multiple microservices, making sure each service completes its task before the next one starts.

4. Real-Time Event Handling

Process streaming data with Amazon Kinesis and trigger alerts or actions based on what’s happening in real time.

Can I see an example please?

Let’s say you’re building a photo app, and every time someone uploads a picture, you need to:

Resize the image.
Analyze it (e.g., detect objects using Rekognition).
Save the metadata to a database.

Here’s how you’d build that in Step Functions:

Step-by-Step Workflow

The user uploads an image to an S3 bucket.
Step Functions triggers a Lambda function to resize the image.
A second Lambda function analyzes the image.
The final Lambda function saves the analysis results to DynamoDB.

{
  "Comment": "Photo processing workflow",
  "StartAt": "ResizeImage",
  "States": {
    "ResizeImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:resizeImageFunction",
      "Next": "AnalyzeImage"
    },
    "AnalyzeImage": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:analyzeImageFunction",
      "Next": "SaveMetadata"
    },
    "SaveMetadata": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:saveMetadataFunction",
      "End": true
    }
  }
}

Wrapping Up

AWS Step Functions is one of those services that makes life easier the more you use it. Whether you’re building simple task sequences or managing complex pipelines, it’s got the tools to keep everything running smoothly. And the best part? No more custom code for retries, errors, or orchestration.

So, what workflow will you automate with Step Functions? Let’s get building!

How I Built a Serverless URL Shortener on AWS Using Terraform

Jorge Contreras — Mon, 25 Nov 2024 16:05:32 +0000

Hey there, fellow devs! 👋

I’ve been practicing on AWS as preparation for my certification exams. I wanted to share a project I worked on: a serverless URL shortener. It’s a simple, real-world application that demonstrates the power of AWS services like Lambda, API Gateway, and DynamoDB, while also introducing Terraform for managing infrastructure as code.

By the end of this guide, you’ll have a fully functional URL shortener that:

Shortens long URLs with a POST request.
Redirects users to the original URL with a GET request.
Uses AWS's serverless stack for scalability and low cost.

So, grab a coffee ☕ and let’s get started!

AWS Services Used

AWS Lambda: To execute the backend logic for shortening and redirecting URLs.
Amazon DynamoDB: To store mappings between short and long URLs.
Amazon API Gateway: To expose RESTful API endpoints.
Terraform: To define and deploy the infrastructure as code.

Prerequisites

Before we start building, make sure you have the following tools installed and configured:

AWS Account:
You’ll need an AWS account to deploy resources.

AWS CLI:
Install the AWS Command Line Interface (CLI).

After installing, configure the CLI with your AWS credentials:

   aws configure

You’ll be prompted to enter:

Access Key ID
Secret Access Key
Default AWS Region (e.g., us-east-1)
Output Format (leave as json)

Run this command to verify the configuration:

   aws s3 ls

If it lists your S3 buckets (or none if you don’t have any), the
setup is complete.

Terraform:
Install Terraform (v1.5+ recommended).

Python:
Install Python 3.9 or higher to write the Lambda functions.

cURL or Postman:
These tools will help you test the API endpoints after deployment.

Step 1. Create the project structure:

project-root/
├── application/
│   ├── backend/
│   │   ├── lambda_create/          
│   │   │   └── create_short_url.py
│   │   ├── lambda_get/             
│   │       └── get_original_url.py
├── terraform/                      # Terraform configuration files
    └── main.tf                     # Defines AWS resources

Step 2: Writing the Lambda Functions

AWS Lambda is a serverless compute service that lets you run code without managing servers. You simply write your code, upload it to Lambda, and AWS handles the rest—like provisioning, scaling, and running the infrastructure. In this section, we'll write the logic in python and package it as zip to be uploaded.

We will create two lambdas: one for creating the short url, and the other to get the original url back from a short url.

1. Create the `CreateShortURL` Function

File: `application/backend/lambda_create/create_short_url.py`

import json
import boto3
import uuid

# Initialize DynamoDB client
dynamodb = boto3.client('dynamodb')
table_name = 'URLMapping'  # Replace with your DynamoDB table name

def lambda_handler(event, context):
    try:
        # Parse the request body
        body = json.loads(event.get('body', '{}'))
        original_url = body.get('OriginalURL')

        if not original_url:
            return {
                'statusCode': 400,
                'body': json.dumps({'message': 'OriginalURL is required'})
            }

        # Generate a unique short URL identifier (6-character UUID)
        short_url = str(uuid.uuid4())[:6]

        # Store the mapping in DynamoDB
        dynamodb.put_item(
            TableName=table_name,
            Item={
                'ShortURL': {'S': short_url},
                'OriginalURL': {'S': original_url}
            }
        )

        # Return the short URL
        return {
            'statusCode': 200,
            'body': json.dumps({'ShortURL': short_url})
        }

    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({'message': 'Internal server error', 'error': str(e)})
        }

2. Create the GetOriginalURL Function

File: application/backend/lambda_get/get_original_url.py

import json
import boto3

# Initialize DynamoDB client
dynamodb = boto3.client('dynamodb')
table_name = 'URLMapping'  # Replace with your DynamoDB table name

def lambda_handler(event, context):
    try:
        # Get the short URL from the path parameter
        short_url = event['pathParameters']['short_url']

        # Query DynamoDB for the original URL
        response = dynamodb.get_item(
            TableName=table_name,
            Key={'ShortURL': {'S': short_url}}
        )

        # Check if the item exists
        if 'Item' not in response:
            return {
                'statusCode': 404,
                'body': json.dumps({'message': 'Short URL not found'})
            }

        original_url = response['Item']['OriginalURL']['S']

        # Return a 302 redirect to the original URL
        return {
            'statusCode': 302,
            'headers': {
                'Location': original_url
            },
            'body': ''
        }

    except Exception as e:
        return {
            'statusCode': 500,
            'body': json.dumps({'message': 'Internal server error', 'error': str(e)})
        }

3. Package the lambda functions

cd application/backend/lambda_create
zip create_short_url.zip create_short_url.py

cd ../lambda_get
zip get_original_url.zip get_original_url.py

After packaging, ensure your project looks like this:

project-folder/
├── application/
│   ├── backend/                  # Backend logic for API
│       ├── lambda_create/        # CreateShortURL Lambda
│       │   ├── create_short_url.py
│       │   ├── create_short_url.zip
│       ├── lambda_get/           # GetOriginalURL Lambda
│           ├── get_original_url.py
│           ├── get_original_url.zip
│
├── terraform/                    # Terraform configurations
    ├── main.tf

Step 3: Define your Infrastructure as Code

In main.tf, define the AWS resources. In this section, we define what resources will be created in the AWS cloud, from IAM roles and policies to Lambdas, DynamoDB and the API Gateway. It allows for managing your infrastructure as code, isn't that awesome?

The file is somewhat long so I just included a snippet here. You can find the whole contents of the file in my github repo

# DynamoDB Table
resource "aws_dynamodb_table" "url_mapping" {
  name           = "URLMapping"
  hash_key       = "short_url"

  attribute {
    name = "short_url"
    type = "S"
  }

  billing_mode = "PAY_PER_REQUEST"
}

# IAM Role for Lambda
resource "aws_iam_role" "lambda_exec_role" {
  name = "url_shortener_lambda_role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Action = "sts:AssumeRole"
      Effect = "Allow"
      Principal = {
        Service = "lambda.amazonaws.com"
      }
    }]
  })
}

# Lambda Functions
resource "aws_lambda_function" "create_short_url" {
  filename         = "${path.module}/../application/backend/lambda_create/create_short_url.zip"
  function_name    = "CreateShortURL"
  role             = aws_iam_role.lambda_exec_role.arn
  handler          = "create_short_url.lambda_handler"
  runtime          = "python3.9"

  environment {
    variables = {
      TABLE_NAME = aws_dynamodb_table.url_mapping.name
    }
  }
}

Step 4: Deploy your infrastructure!

Run the following commands:

cd terraform
terraform init
terraform plan
terraform apply

Step 5. Test

1. Shorten a URL

Send a POST request:

curl -X POST -H "Content-Type: application/json" \
-d '{"OriginalURL": "https://www.example.com"}' \
https://<api-gateway-id>.execute-api.<region>.amazonaws.com/shorten

2. Redirect using the short URL

curl -I https://<api-gateway-id>.execute-api.<region>.amazonaws.com/<short_url>

Conclusion

And that’s it! 🎉 You’ve built a fully functional URL shortener using AWS and Terraform. Checkout the full project here

Here’s what we achieved:

Learned how to set up AWS resources with Terraform.
Created Lambda functions for shortening and redirecting URLs.
Integrated everything with API Gateway and DynamoDB.

Stay tuned for more AWS articles! next, we’ll explore securing APIs and adding custom domain names. Happy coding!

Concurrency in Go

Jorge Contreras — Fri, 01 Mar 2024 02:19:25 +0000

What is concurrency and why does it matter?

Concurrency is handling several tasks within a given timeframe. Most modern applications, such as web servers, databases, and user interfaces, require real-time processing and responsiveness. Concurrency allows to handle multiple tasks, like API requests or database calls, simultaneously. This helps us developers improve the performance of the applications our users love. Concurrency allows for maximum optimization of computing resources, making our systems highly efficient while lowering infrastructure costs.

Concurrency optimizes computing resources, maximizes performance and minimize infrastructure costs.

Why is concurrency hard to implement?

Concurrency is challenging to implement for several reasons, largely due to the complexities and unpredictable nature of managing multiple tasks that run at overlapping times. Very often, these concurrent tasks need to share resources and it is very easy to make mistakes and interfere with other tasks. For example, if one task attempts to read from the memory while another task is writing. All these tasks being executed at the same time need an efficient way to synchronize and ensure there are no conflicts between them. Deadlocks, livelocks, starvation and race conditions are just a few examples of these complexities[1].

How is concurrency implemented in different programming languages?

Concurrency is not a new concept. The idea of concurrency in computing began with the development of multiprogramming systems. The Atlas Computer, developed at the University of Manchester and operational in 1962, is often credited as one of the first machines to provide a form of multiprogramming. The semaphore, invented by Dijkstra and colleagues[2], is one of the first primitives for all things related to synchronization.

While it is possible to implement these primitives in languages like C or C++, it can be challenging, as these languages provide low-level control over hardware and memory, but it is up to the developer to control the threads and shared resources, keep track of memory usage, free up space and so on.

Over time, there has been a development of techniques to facilitate the implementation of concurrent systems. In the world of Javascript, NodeJS implements concurrency using a non-blocking, event-driven architecture, primarily centered around the event loop and callbacks. In PHP, the OpenSwoole framework enables PHP developers to write asynchronous, concurrent, and highly scalable applications without needing extensive knowledge of non-blocking I/O programming.

A few programming languages and their concurrency approach

What makes Go an excellent choice for concurrent applications?

Go's concurrency model, based on the Communicating Sequential Processes (CSP) theory, makes concurrent programming intuitive [3]. Go abstracts many of the complexities that arise from concurrent system architectures like distributed systems and cloud computing. The go runtime, via its scheduler, takes care of managing goroutines, which are Go's lightweight threads of execution [4].

Goroutines are much more lightweight than OS threads. They have a smaller memory footprint, and the overhead of creating and destroying a goroutine is significantly lower than that of an OS thread. This allows Go programs to easily handle thousands or even millions of goroutines.

Go by itself does not eliminate the complexities associated to concurrent programming, but it definitely makes them easier to manage. It provides composable primitives like goroutines, channels and waitgroups to facilitate synchronization. Its efficient garbage collector takes care of freeing up resources for you. Additionally, Go’s standard library includes a wealth of packages that are well-suited for building cloud-native applications. Additionally, Go provides the performance benefits of a compiled language, which is critical for cloud and network services that need to handle high loads efficiently.

How is concurrency achieved in Go?

Let's take a look at how goroutines are created:

package main

import (
    "fmt"
    "time"
)

func Pink() {
    fmt.Println("Hi, Pink Goroutine!")
}

func Purple() {
    fmt.Println("Hello, Purple Goroutine!")
}

func Green() {
    fmt.Println("Hello, Green Goroutine!")
}

func Orange() {
    fmt.Println("Hello, Orange Goroutine!")
}

func main() {
    // start the goroutines
    go Pink() 
    go Purple() 
    go Green()
    go Orange()
}

The main process triggers the goroutines using the go keyword before the function call. These goroutines or threads start execution immediately and they run concurrently.

goroutines executing concurrently.

Just imagine all the time-consuming tasks that you could execute at the same time! Pulling from different external resources, reading from a database, calling an API. As long as they don't depend on each other, there is no need to wait for one task to complete to start working on the next.

In some cases though, you may need to actually wait for some goroutines to complete before starting the next one. For that, Go provides waitgroups:

package main

import (
    "fmt"
    "sync"
    "time"
)

func Pink(wg *sync.WaitGroup) {
    defer wg.Done() // Decrement counter when done
    fmt.Println("Hi, Pink Goroutine!")
}

func Purple(wg *sync.WaitGroup) {
    defer wg.Done() // Decrement counter when done
    fmt.Println("Hello, Purple Goroutine!")
}

func Green(wg *sync.WaitGroup) {
    defer wg.Done() // Decrement counter when done
    fmt.Println("Hello, Green Goroutine!")
}

func Orange() {
    fmt.Println("Hello, Orange Goroutine!")
}

func main() {
    var wg sync.WaitGroup

    // Increment the WaitGroup counter for each goroutine
    wg.Add(3)

    // Start the first three goroutines
    go Pink(&wg)
    go Purple(&wg)
    go Green(&wg)

    // Wait for the first three goroutines to complete
    wg.Wait()

    // Now start the Orange goroutine
    go Orange()
}

waitgroups: The process waits for the first 3 goroutines to complete.

Waitgroups gives you the ability to wait for goroutines to be completed before moving on.

Why is synchronization important, you ask? Let's take a closer look the animation above. Did you notice the orange goroutine was not completed? That's because the process continues its execution and the goroutine may not get enough time to complete whatever it was doing. Adding that goroutine to the waitgroup would solve the issue. There could be many places in your system that requires this type of synchronization: use it wisely!

Finally, let's see what channels can do for us. Go's philosophy to prevent common issues like data races goes something like this:

Do not communicate by sharing memory; instead, share memory by communicating.

What does this mean? Instead of explicitly using locks to mediate access to shared data, Go encourages the use of channels to pass references to data between goroutines. Let's see a code example:

package main

import (
    "fmt"
)

func main() {
    messages := make(chan string)

    go func() { messages <- "Hello" }()
    go func() { messages <- "World" }()

    msg1 := <-messages
    msg2 := <-messages

    fmt.Println(msg1, msg2)
}

In this example, two goroutines communicate strings through a channel. Instead of accessing a shared variable, they use the channel to send and receive messages.

This principle is one of the key reasons why Go's concurrency model is considered more straightforward and robust compared to traditional thread-based models.

Wrapping up

Concurrency enables applications to handle multiple tasks efficiently, essential in applications driven by real-time data, high-performance demands, and the need for scalable systems that make the most optimal use of computing resources. Go's design principles, its simplicity and efficient communication over shared memory, makes it a great choice for distributed systems and cloud-native technologies. Go was designed with concurrency in mind since its inception, making it stand out from other programming languages.

References

[1] https://www.cs.yale.edu/homes/aspnes/pinewiki/Deadlock.html

[2] https://pages.cs.wisc.edu/~remzi/OSTEP/threads-sema.pdf

[3] https://www.cs.cmu.edu/~crary/819-f09/Hoare78.pdf

[4] Cox-Buday, K. (2017). Concurrency in go: Tools and techniques for developers. O'Reilly Media.

DEV Community: Jorge Contreras

Understanding MCP Servers: The Model Context Protocol Explained

What is the Model Context Protocol?

What is an MCP Server?

Why Would I Need an MCP Server?

Traditional LLM Calls 🆚 MCP Server Approach

What Makes a Server an MCP Server?

What Company or Organization is Behind It?

What are the Known Limitations of MCP Servers?

Are There Similar Alternatives That Solve the Same Kind of Problem?

🛠️ Function Calling Frameworks for LLMs

How Can I See an MCP Server in Action?

Can I Build My Own MCP Server and What Would I Need?

Technical Requirements:

Implementation Steps:

Links

Building AWS Lambda Functions

From your Local to the Cloud

So, How Do You Test AWS Lambda Locally?

Option 1: AWS SAM – The Official Way

Option 2: LocalStack – Running AWS on Your Machine

Option 3: AWS Lambda Runtime Interface Emulator

Option 4: The Good Old Mocking Approach

So, What’s the Best Approach?

AWS Lambda Performance Tuning: How to Reduce Cold Starts

What is a cold start? 🥶

Ok so, cold starts are bad. What can we do about it?

1. Provisioned concurrency.

2. Optimize Function Memory & CPU.

3. Keep your deployment package small.

4. Avoid Heavy Initialization.

5. Warmers

Wrapping Up

AWS Machine Learning Certified? Let’s Do This!

Why This Certification?

What You Need to Know Before Starting

What This Series Will Cover

Kubernetes CRDs: The Backbone of Kubernetes Extensibility

What Are CRDs (Custom Resource Definitions)?

Why Should You Care About CRDs?

Let's see an example!

Are they common?

1. Cert-Manager

2. ArgoCD

3. Prometheus Operator

CRDs in Action: How They Work

Wrapping Up

AWS Step Functions: Your Workflow's Best Friend

What Exactly Are AWS Step Functions?

Why are they so useful?

Understood but, where can Step Function be applied in the real world?

Can I see an example please?

Wrapping Up

How I Built a Serverless URL Shortener on AWS Using Terraform

AWS Services Used

Prerequisites

Step 1. Create the project structure:

Step 2: Writing the Lambda Functions

1. Create the CreateShortURL Function

File: application/backend/lambda_create/create_short_url.py

2. Create the GetOriginalURL Function

File: application/backend/lambda_get/get_original_url.py

3. Package the lambda functions

Step 3: Define your Infrastructure as Code

Step 4: Deploy your infrastructure!

Step 5. Test

1. Shorten a URL

2. Redirect using the short URL

Conclusion

Concurrency in Go

What is concurrency and why does it matter?

Why is concurrency hard to implement?

How is concurrency implemented in different programming languages?

What makes Go an excellent choice for concurrent applications?

How is concurrency achieved in Go?

Wrapping up

References

1. Create the `CreateShortURL` Function

File: `application/backend/lambda_create/create_short_url.py`