DEV Community: Monika Prajapati

Quick Guide: Installing Conda on EC2 (The Right Way)

Monika Prajapati — Sun, 26 Jan 2025 07:33:45 +0000

Look, installing Conda on EC2 doesn't need to be complicated. After setting this up across hundreds of instances, here's my approach that just works.

The Fast Track

Skip the fluff. Here's what actually works:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
~/miniconda/bin/conda init bash
source ~/.bashrc

Done. That's it. If you're in a hurry, you can stop reading here.

When Things Break

The Command Not Found Thing

Got the "conda: command not found" error? Yeah, we've all been there. The -b flag skips initialization. Just run:

~/miniconda/bin/conda init bash
source ~/.bashrc

Fixed. Moving on.

Existing Installation

ERROR: File or directory already exists: '/home/ec2-user/miniconda'

Two options:

# Quick update
bash ~/miniconda.sh -u -b -p $HOME/miniconda

# Or nuke it and start fresh
rm -rf $HOME/miniconda && bash ~/miniconda.sh -b -p $HOME/miniconda

Production Setup

Here's my actual production setup script. Copy-paste and thank me later:

#!/bin/bash
sudo yum update -y

# Skip if you've handled these in your AMI
sudo yum install -y bzip2 wget git

# Base install
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
~/miniconda/bin/conda init bash
source ~/.bashrc

# The stuff you actually need
conda config --add channels conda-forge
conda config --set channel_priority strict
conda clean -a -y  # Trust me on this one

# Create your env
conda create -n prod python=3.11 -y
conda activate prod
conda install numpy pandas scikit-learn -y  # Add your packages

For the DevOps Folks

Throw this in your user data and forget about it:

#!/bin/bash
cat <<'EOF' > /home/ec2-user/setup_conda.sh
#!/bin/bash
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda
~/miniconda/bin/conda init bash
EOF

chmod +x /home/ec2-user/setup_conda.sh
sudo -u ec2-user /home/ec2-user/setup_conda.sh

The "Why" Behind the Choices

Miniconda over Anaconda: Smaller footprint, faster installs. You don't need the bloat.
conda-forge first: Better packages, fewer headaches.
Always clean after big installs: Saves space, prevents weird caching issues.
Separate envs: Just do it. Future you will be grateful.

Storage Notes

Quick reality check before you start:

df -h

Miniconda needs about 400MB. Your envs will grow. Plan accordingly or attach an EBS volume.

Container World

If you're using containers (and you probably should be), here's your Dockerfile snippet:

FROM amazonlinux:2

RUN yum update -y && \
    yum install -y bzip2 wget && \
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda.sh && \
    bash /miniconda.sh -b -p /opt/conda && \
    rm /miniconda.sh

ENV PATH="/opt/conda/bin:${PATH}"

RUN conda install -y python=3.11 && \
    conda clean -a -y

WORKDIR /app

One Last Thing

Always export your environment:

conda env export > environment.yml

Don't be that person who can't reproduce their env six months later.

That's it. No fluff, just what works. If you've got questions, you know where to find me.

What is Serverless Computing?

Monika Prajapati — Mon, 15 Jul 2024 19:05:11 +0000

Serverless computing has emerged as a revolutionary paradigm in cloud computing, offering a new way to build and deploy applications without the need for traditional server management. Function-as-a-Service (FaaS) is a serverless way to execute modular pieces of code on the edge, it focuses on code execution and implementing certain functions to accomplish the target actions. FaaS is often used to deploy microservices and may also be referred to as serverless computing.

Here, we will explore the concept of serverless computing, its inner workings, the backend services it can provide, the advantages and disadvantages it offers, how it compares to other cloud backend models, what the future holds, and the scenarios where it is most suitable or should be avoided.

I. What is Serverless Computing?

Serverless computing is a cloud computing execution model where the cloud provider dynamically manages the allocation and provisioning of computing resources. A company that gets backend services from a serverless vendor is charged based on their computation and do not have to reserve and pay for a fixed amount of bandwidth or number of servers, as the service is auto-scaling. In this model, developers no longer have to worry about provisioning, scaling, or managing servers. Instead, they can focus solely on writing and deploying code, while the cloud provider takes care of executing that code in response to specific events or requests.

The term "serverless" is somewhat misleading because servers are still involved in the process, but developers are abstracted away from their management and maintenance. The cloud provider handles the server infrastructure, automatically scaling resources up or down based on the workload demand.

II. How Does Serverless Computing Work?

In a serverless architecture, developers package their code into functions or small, independent units of computation. These functions are then deployed to the cloud provider's platform, where they remain dormant until triggered by specific events or requests.

When a triggering event occurs, the cloud provider automatically allocates the necessary computing resources, executes the function, and scales resources up or down as needed to handle the workload. This process is entirely managed by the cloud provider, freeing developers from the complexities of server provisioning, scaling, and maintenance.

Serverless functions are typically stateless and ephemeral, meaning they do not maintain any persistent state or long-running connections. Each function execution is independent and self-contained, with the function's input and output data being stored and retrieved from cloud-based storage services or databases.

III. What Kind of Backend Services Can Serverless Computing Provide?

Serverless computing can provide a wide range of backend services, including but not limited to:

API Gateways: Serverless functions can be used to build and deploy APIs, acting as lightweight and scalable backends for web and mobile applications.
Data Processing: Serverless functions can be triggered by events from data sources, such as object uploads to cloud storage or database updates, enabling real-time data processing and transformation.
Event-Driven Architectures: Serverless functions can be integrated with various event sources, including message queues, IoT devices, and cloud services, enabling event-driven architectures and real-time processing.
Microservices: Serverless functions can be used to build and deploy microservices, providing a scalable and cost-effective way to decompose applications into smaller, independent components.
Scheduled Tasks: Serverless functions can be configured to run on a schedule, enabling automated tasks such as data backups, report generation, or periodic maintenance tasks.
Chatbots and Conversational Interfaces: Serverless functions can be used to build and deploy chatbots and conversational interfaces, leveraging natural language processing (NLP) and machine learning (ML) capabilities.

IV. How Does Serverless Compare to Other Cloud Backend Models?

Serverless computing differs from traditional cloud backend models, such as Infrastructure-as-a-Service (IaaS) and Platform-as-a-Service (PaaS), in several ways:

IaaS: In an IaaS model, developers are responsible for provisioning and managing the underlying infrastructure, including virtual machines, storage, and networking. Serverless computing abstracts away these infrastructure management tasks, allowing developers to focus solely on writing and deploying code.
PaaS: In a PaaS model, developers deploy their applications to a platform managed by the cloud provider, but they still have to manage the application lifecycle and scaling. Serverless computing takes this abstraction a step further by also managing the execution environment and automatically scaling resources based on demand.

Serverless computing offers a higher level of abstraction and automation compared to IaaS and PaaS models, making it easier for developers to build and deploy applications without worrying about infrastructure management.

V. What are the Advantages of Serverless Computing?

Serverless computing offers several advantages over traditional server-based architectures, making it an attractive choice for developers and organizations alike:

No Server Management: With serverless computing, developers are freed from the burden of provisioning, scaling, and managing servers. This allows them to focus solely on writing and deploying code, increasing productivity and reducing operational overhead.
Automatic Scaling: Serverless platforms automatically scale resources up or down based on the workload demand, ensuring optimal performance and cost-efficiency. Developers no longer need to worry about over-provisioning or under-provisioning resources.
Pay-per-Use Pricing: Serverless computing follows a pay-per-use pricing model, where you only pay for the compute time and resources used during function execution. This can lead to significant cost savings, especially for applications with intermittent or spiky workloads.
High Availability and Fault Tolerance: Serverless platforms are designed to be highly available and fault-tolerant, ensuring that your applications remain up and running even in the face of infrastructure failures or spikes in demand.
Rapid Deployment and Iteration: Serverless functions can be deployed and updated quickly, enabling rapid iteration and faster time-to-market for new features and applications.
Integration with Cloud Services: Serverless platforms seamlessly integrate with a wide range of cloud services, such as databases, storage, messaging, and analytics, enabling developers to build sophisticated applications without managing complex infrastructure.
Simplified DevOps: Serverless computing simplifies DevOps processes by eliminating the need for server provisioning, patching, and maintenance. Developers can focus on writing and deploying code, while the cloud provider handles the underlying infrastructure.
Increased Developer Productivity: By abstracting away infrastructure management tasks, serverless computing allows developers to concentrate on writing business logic, leading to increased productivity and faster time-to-market for applications.

VI. What are the Disadvantages of Serverless Computing?

While serverless computing offers numerous advantages, it also has some potential drawbacks that developers should consider:

Cold Start Latency: Serverless functions may experience a "cold start" delay when initially invoked, as the cloud provider allocates and initializes the necessary resources. This can impact the performance of time-sensitive applications.
Execution Duration Limits: Most serverless platforms impose limits on the maximum execution duration for functions, which may not be suitable for long-running or compute-intensive tasks.
Vendor Lock-In: Serverless platforms are typically proprietary and tied to a specific cloud provider, which can lead to vendor lock-in and potential migration challenges if the need arises.
Monitoring and Debugging Challenges: Monitoring and debugging serverless applications can be more challenging compared to traditional architectures, as the functions are ephemeral and distributed across multiple execution environments.
Security Considerations: While cloud providers implement security measures, developers must still ensure that their serverless functions and associated resources (e.g., databases, storage) are properly secured and follow best practices for data protection and access control.

VII. What is Next for Serverless?

The serverless computing landscape is rapidly evolving, and several trends and developments are shaping its future:

Improved Tooling and Frameworks: As serverless adoption increases, we can expect to see more robust tooling and frameworks that simplify the development, testing, and deployment of serverless applications.
Serverless Orchestration: With the increasing complexity of serverless applications, there is a growing need for orchestration tools and services that can manage the coordination and execution of multiple serverless functions.
Serverless Databases and Storage: While serverless computing has primarily focused on compute resources, we are starting to see the emergence of serverless databases and storage solutions that offer similar pay-per-use and automatic scaling benefits.
Edge Computing and Serverless: The combination of serverless computing and edge computing is gaining traction, enabling low-latency and highly distributed applications by bringing computation closer to the data source or end-user.
Serverless Machine Learning: As machine learning and artificial intelligence become more prevalent, serverless platforms are adapting to support the deployment and execution of machine learning models, enabling scalable and cost-effective AI solutions.

VIII. Who Should Use a Serverless Architecture?

Serverless architectures are well-suited for a variety of use cases and scenarios, including:

Event-Driven Applications: Applications that need to respond to events or triggers, such as file uploads, database updates, or IoT sensor data, can benefit from the event-driven nature of serverless computing.
Microservices and APIs: Serverless functions can be used to build and deploy microservices and APIs, providing a scalable and cost-effective way to decompose applications into smaller, independent components.
Intermittent or Spiky Workloads: Applications with unpredictable or spiky workloads can take advantage of the automatic scaling capabilities of serverless computing, ensuring optimal performance and cost-efficiency.
Rapid Prototyping and Experimentation: The ease of deployment and pay-per-use pricing model make serverless computing an attractive choice for rapid prototyping, experimentation, and proof-of-concept projects.
Startups and Small Teams: Serverless computing can be particularly beneficial for startups and small teams with limited resources, as it eliminates the need for infrastructure management and allows them to focus on building their core products.

IX. When Should Developers Avoid Using a Serverless Architecture?

While serverless computing offers many advantages, there are certain scenarios where it may not be the ideal choice:

Long-Running or Compute-Intensive Tasks: Serverless functions are designed for short-lived, event-driven computations and may not be suitable for long-running or compute-intensive tasks due to execution duration limits and potential cost implications.
Strict Latency Requirements: Applications with strict latency requirements may struggle with the cold start latency associated with serverless functions, making it necessary to explore alternative architectures or implement mitigation strategies.
Proprietary or Legacy Systems: Migrating proprietary or legacy systems to a serverless architecture can be challenging and may require significant refactoring or integration efforts.
Highly Regulated Industries: Certain industries with stringent regulatory requirements, such as finance or healthcare, may have concerns about the shared execution environments and potential security risks associated with serverless computing.
Situations with Stringent Data Locality Requirements: If an application requires strict data locality or has specific data residency requirements, serverless computing may not be the best choice, as cloud providers may store and process data across multiple regions or data centers.

In conclusion, serverless computing represents a paradigm shift in how applications are built and deployed in the cloud. By offloading server management to cloud providers, developers can focus on writing and deploying code, leading to increased productivity, cost-efficiency, and scalability. While serverless computing offers numerous advantages, it is crucial to carefully evaluate its potential drawbacks and suitability for specific use cases and workloads. As the serverless ecosystem continues to evolve, it is likely to become an increasingly popular choice for building modern, event-driven, and scalable applications across various industries.

I referred to these pages to write this article.

The Art of Writing Effective Git Commits

Monika Prajapati — Wed, 29 May 2024 10:37:52 +0000

Version control systems like Git are essential tools for developers, allowing them to track changes to their codebase, collaborate with others, and manage project history effectively. One crucial aspect of using Git is writing clear and meaningful commit messages. A well-written commit message not only helps you understand the changes you've made but also assists your team members and future contributors in comprehending the project's evolution. In this blog post, we'll explore the art of writing effective Git commits.

The Importance of Good Commit Messages

A commit message serves as a concise summary of the changes introduced in a particular commit. It should provide enough context for anyone reviewing the commit history to quickly grasp the reasoning behind the changes and their impact on the codebase. Well-crafted commit messages can:

Facilitate code review: Clear commit messages make it easier for reviewers to understand the purpose and scope of the changes, streamlining the code review process.
Improve collaboration: When working in a team, commit messages help other developers understand the changes made by their colleagues, reducing confusion and enabling smoother collaboration.
Simplify debugging: If a bug is introduced, well-documented commit messages can help pinpoint the source of the issue more efficiently by providing context about the changes made in specific commits.
Enhance project maintenance: As projects evolve and new contributors join, descriptive commit messages serve as a historical record, making it easier to understand the rationale behind past decisions and changes.

The Structure of a Commit Message

A commit message typically consists of two parts: a subject line and an optional body. Here's the recommended structure:

<type>(<scope>): <subject>

<body>

Type: This is a concise description of the kind of change the commit introduces. Common types include feat (a new feature), fix (a bug fix), docs (documentation changes), style (formatting changes), refactor (code refactoring), test (addition or modification of tests), chore (maintenance tasks), and more.
Scope (optional): This specifies the area of the codebase that the commit is focused on, such as a specific component, module, or feature. For example, user-auth or dashboard.
Subject: A brief, imperative summary of the changes (e.g., "Add user authentication" or "Fix typo in README").
Body (optional): A more detailed description of the changes, including the motivation behind them, any potential side effects, and any relevant issues or pull requests. The body should be wrapped at 72 characters per line.

Here's an example of a well-structured commit message:

feat(user-auth): add email verification

- Implement email verification flow
- Add verification email templates
- Update user model to include 'verified' field

Closes #123

In this example, the commit introduces a new feature related to user authentication (feat(user-auth)), specifically the addition of email verification functionality. The body provides more context about the changes, including the implementation details and any relevant issues or pull requests.

Commit Types

As mentioned earlier, the commit type is a concise description of the kind of change the commit introduces. Here are some common commit types:

feat – a new feature is introduced with the changes
fix – a bug fix has occurred
chore – changes that do not relate to a fix or feature and don't modify src or test files (e.g., updating dependencies)
refactor – refactored code that neither fixes a bug nor adds a feature
docs – updates to documentation such as the README or other markdown files
style – changes that do not affect the meaning of the code, likely related to code formatting such as white-space, missing semi-colons, and so on.
test – including new or correcting previous tests
perf – performance improvements
ci – continuous integration related
build – changes that affect the build system or external dependencies
revert – reverts a previous commit

Here are some examples of commit messages using different types:

fix(user-auth): correct email validation regex

docs: update README with installation instructions

chore: upgrade dependencies to latest versions

refactor(auth-service): improve code readability

test(user-model): add unit tests for email validation

Best Practices for Writing Effective Commit Messages

To ensure your commit messages are clear, concise, and effective, follow these best practices:

Use the imperative mood: Write the subject line in the imperative mood, as if giving instructions to the codebase. For example, "Add user authentication" instead of "Added user authentication" or "Adds user authentication."
Keep the subject line short and descriptive: The subject line should be no longer than 50 characters and should clearly summarize the changes made in the commit.
Capitalize the subject line: Capitalize the first letter of the subject line for better readability.
Use the body for detailed explanations: If the commit requires more context or explanation, use the body section to provide additional details, such as the motivation behind the changes, potential side effects, and any relevant issues or pull requests.
Separate the subject from the body with a blank line: For better readability, leave a blank line between the subject and the body of the commit message.
Reference issues or pull requests: If the commit is related to an issue or pull request, include a reference to it in the body of the commit message (e.g., "Closes #123" or "Fixes #456").
Avoid generic messages: Avoid using generic or ambiguous commit messages like "update" or "fix bug." Instead, be specific about the changes made.
Follow project or team conventions: If your project or team has specific conventions or guidelines for commit messages, make sure to follow them for consistency.

By following these best practices and writing clear and descriptive commit messages, you'll not only improve the overall quality and maintainability of your codebase but also facilitate collaboration and streamline the development process.

I used this source to write this blog, please refer to Natalie Pina on this page

Best Practices for dynamodb

Monika Prajapati — Fri, 29 Mar 2024 06:12:18 +0000

Amazon DynamoDB is currently experiencing rapid growth as a database service. Nonetheless, being a NoSQL database, it demands a distinct approach to data modeling compared to SQL databases.

Although DynamoDB can provide performance in the range of single-digit milliseconds regardless of scale, it's important to follow best coding practices to ensure efficient and effective use of the database.

Here are some coding best practices for DynamoDB:

Use Batching for Read and Write Operations: Batch operations can significantly reduce the number of requests sent to DynamoDB, improving performance and reducing costs. Use BatchGetItem for reading multiple items and BatchWriteItem for writing multiple items.

   # Batch read example
   keys = [{'id': {'S': 'item1'}}, {'id': {'S': 'item2'}}]
   response = dynamodb.batch_get_item(RequestItems={table_name: {'Keys': keys}})

   # Batch write example
   items = [{'PutRequest': {'Item': {'id': {'S': 'item3'}, 'name': {'S': 'Item 3'}}}},
            {'PutRequest': {'Item': {'id': {'S': 'item4'}, 'name': {'S': 'Item 4'}}}}]
   response = dynamodb.batch_write_item(RequestItems={table_name: items})

Use Conditional Updates: DynamoDB supports conditional updates, which can help prevent race conditions and ensure data consistency. Use the ConditionExpression parameter when updating items to ensure updates only occur if specific conditions are met.

   response = dynamodb.update_item(
       TableName=table_name,
       Key={'id': {'S': 'item1'}},
       UpdateExpression='SET name = :val1',
       ConditionExpression='attribute_exists(id)',
       ExpressionAttributeValues={':val1': {'S': 'Updated Name'}}
   )

Implement Pagination for Query and Scan Operations: When retrieving large result sets, use pagination to avoid overwhelming your application with too much data at once. DynamoDB provides LastEvaluatedKey and ExclusiveStartKey parameters to handle pagination.

   response = dynamodb.scan(TableName=table_name)
   items = response.get('Items', [])
   last_evaluated_key = response.get('LastEvaluatedKey')

   while last_evaluated_key:
       response = dynamodb.scan(
           TableName=table_name,
           ExclusiveStartKey=last_evaluated_key
       )
       items.extend(response.get('Items', []))
       last_evaluated_key = response.get('LastEvaluatedKey')

Use Exponential Backoff for Retries: DynamoDB can throttle requests if they exceed the provisioned throughput capacity. Implement exponential backoff with jitter for retrying throttled requests to avoid overwhelming the database with retries.

   import random
   import time

   def exponential_backoff(base_delay, max_delay, attempt):
       delay = min(base_delay * (2 ** attempt), max_delay)
       jitter = random.uniform(-0.5, 0.5)
       delay = delay * (1 + jitter)
       return delay

   # Example usage
   base_delay = 0.1  # Initial delay in seconds
   max_delay = 5  # Maximum delay in seconds
   attempt = 0
   while True:
       try:
           # Perform DynamoDB operation
           ...
           break
       except dynamodb.exceptions.ProvisionedThroughputExceededException:
           delay = exponential_backoff(base_delay, max_delay, attempt)
           time.sleep(delay)
           attempt += 1

Use DynamoDB Streams for Data Replication and Event-Driven Architecture:
DynamoDB Streams capture a time-ordered sequence of item-level modifications in a table and store them for up to 24 hours. You can use DynamoDB Streams to replicate data across DynamoDB tables or trigger downstream actions in response to data modifications.
Implement Caching for Frequently Accessed Data:
Caching frequently accessed data can significantly improve performance and reduce the load on DynamoDB. Consider using in-memory caching solutions like Redis or leveraging Amazon ElastiCache for caching DynamoDB data.
Use DynamoDB Transactions for Atomic Operations:
DynamoDB transactions allow you to perform multiple operations (Put, Update, Delete) as an all-or-nothing transaction. Use transactions to maintain data integrity and consistency when multiple operations need to be executed atomically.
Leverage DynamoDB Local for Development and Testing:
DynamoDB Local is a client-side database that emulates the DynamoDB service on your local machine. Use DynamoDB Local for development and testing purposes to avoid incurring charges on the actual DynamoDB service and to enable offline development.
Implement Monitoring and Logging:
Monitor your DynamoDB usage, performance metrics, and costs using Amazon CloudWatch. Additionally, implement robust logging mechanisms to help with debugging and troubleshooting issues.
Follow Security Best Practices:
- Use AWS Identity and Access Management (IAM) policies to control access to DynamoDB resources and operations.
- Encrypt data at rest and in transit using AWS Key Management Service (KMS) and SSL/TLS connections.
- Avoid embedding AWS credentials in your code and follow the principle of least privilege when granting permissions.

By following these coding best practices, you can ensure efficient, scalable, and secure use of DynamoDB in your applications, while also optimizing performance and minimizing costs.

If you want to dive deep into best practices, feel free to check this documentation from aws

Amazon DynamoDB: Scalable NoSQL for High-Performance Applications

Monika Prajapati — Wed, 27 Mar 2024 06:29:08 +0000

DynamoDB is a cloud-native NoSQL database service offered by Amazon Web Services (AWS). DynamoDB offers a fast persistent key–value datastore with built-in support for replication, autoscaling, encryption at rest, and on-demand backup among other features. Designed for scalability and predictable performance, it caters to applications with demanding data access requirements.

Use Cases for DynamoDB

Mobile Backends: Due to its high availability and scalability, DynamoDB is a popular choice for storing and managing data in mobile applications.
IoT Applications: The real-time nature of DynamoDB makes it ideal for storing and processing data streams generated by Internet of Things (IoT) devices.
Gaming Leaderboards: With its ability to handle massive read and write requests per second, DynamoDB is well-suited for maintaining dynamic leaderboards in online games.
E-commerce Platforms: E-commerce applications can leverage DynamoDB's scalability to manage product catalogs, shopping carts, and user data efficiently.

These are just a few examples, and DynamoDB's flexibility allows it to be adapted to a wide range of use cases.

DynamoDB Pricing

Unlike traditional database services with fixed costs, DynamoDB utilizes a pay-as-you-go pricing model. You are charged based on the provisioned capacity (read and write capacity units) and the amount of data stored and accessed. This allows for cost-effective scaling based on your application's specific needs. More information on pricing details can be found on the AWS DynamoDB pricing page https://aws.amazon.com/dynamodb/pricing/.

Limitations of DynamoDB

While DynamoDB offers numerous advantages, it's essential to consider its limitations before implementation:

Eventual Consistency: Unlike traditional relational databases, DynamoDB utilizes eventual consistency, meaning reads might not always reflect the latest data after a write operation. This may not be ideal for scenarios requiring strict data consistency.
Limited Query Capabilities: Querying in DynamoDB is primarily based on the primary and sort keys. Complex queries involving joins or aggregations may require additional design considerations or workarounds.

For a deeper understanding of DynamoDB's design principles and eventual consistency model, refer to the original Dynamo paper https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf.

There are other limitations as well that are not straightly disadvantageous:

Item size limit: A single DynamoDB item cannot exceed 400KB in size. This encourages modeling data appropriately, storing large blobs in object storage like S3, and avoiding denormalizing unbounded data.
Page size limit for Query and Scan operations: Query and Scan operations are limited to returning a maximum of 1MB of data per request. If the result set exceeds 1MB, it must be paginated. This forces developers to account for pagination in their application code.
Partition throughput limits: There are limits on the maximum throughput that can be consumed on a single DynamoDB partition per second: 3,000 Read Capacity Units (RCUs) and 1,000 Write Capacity Units (WCUs). This helps guide data modeling to avoid hotspots and ensures predictable performance.

These limitations in DynamoDB are imposed for a few key reasons:

To guide proper data modeling:
- The item size limit prevents denormalizing unbounded data into single items, which could degrade performance.
- The pagination requirement for large result sets encourages efficient access patterns.
- The partition throughput limits nudge developers to distribute data and traffic evenly across partitions.
To provide predictable performance:
- With explicit limits, DynamoDB can guarantee consistent performance up to those limits.
- This binary performance profile (works or doesn't) makes capacity planning more straightforward compared to traditional databases with variable performance.
To reduce operational complexity:
- Rather than having to estimate ideal hardware configs, DynamoDB offloads performance tuning to the service.
- Features like adaptive capacity further reduce the need to micro-optimize for hot partitions.
To match the strengths of DynamoDB's architecture:
- As an OLTP (online transaction processing) database, DynamoDB excels at high volumes of small read/write operations.
- The limits steer usage toward this strength, preventing large blob storage or analytics workloads that are better suited for other purpose-built services.

In essence, the limitations shape usage of DynamoDB as an ultra-performant, scale-out key-value store while offloading complexity to the service itself.

For deeper understanding of these limitation, feel free to read this blog by
Alex DeBrie

In conclusion, DynamoDB is a powerful NoSQL database solution for applications requiring high scalability, performance, and flexibility. Understanding its use cases, pricing structure, and limitations will help you determine if it's the right fit for your project.