DEV Community: Ahmed Helmi

How to Use a Terraform Provider Locally

Ahmed Helmi — Fri, 28 Feb 2025 15:21:10 +0000

Recently, I was working on a project that needed Terraform, but there was a catch; I had to work in an environment where there is no access to the Terraform Registry. So, I had to figure out how to use a Terraform provider locally, without relying on Terraform’s online registry. If you’re in a similar boat whether for testing, security, or any other reason then this guide is for you!

Terraform doesn’t just pick up providers from your directory automatically. You need to set things up manually, and that’s where my scripts come in.

Step 1: Get Your Provider Binary

First, grab your provider binary file (in this case, we’re using vsphere with version 2.10.0). Make sure it's in your working directory before running the script.

Step 2: Run the PowerShell Script (Windows)

Save the following script as setup-local-provider.ps1 and execute it in your PowerShell terminal:

# PowerShell script to setup local Terraform provider in current directory

$version = "2.10.0"
$provider = "vsphere"
$providerBinary = "terraform-provider-$provider`_v$version.exe"

# Function to create terraform.rc file
function Create-TerraformRC {
    $rcContent = @"
provider_installation {
  filesystem_mirror {
    path = "$PWD\.terraform\plugins"
  }
  direct {
    exclude = ["registry.terraform.io/*/*"]
  }
}
"@

    $rcPath = "$env:APPDATA\terraform.rc"
    $rcContent | Out-File -FilePath $rcPath -Encoding UTF8 -Force
    Write-Host "Created terraform.rc at: $rcPath"
}

# Function to create provider.tf file
function Create-ProviderConfig {
    $tfContent = @"
terraform {
  required_providers {
    $provider = {
      source = "hashicorp/$provider"
      version = "$version"
    }
  }
}
"@

    $tfPath = "provider.tf"
    $tfContent | Out-File -FilePath $tfPath -Encoding UTF8 -Force
    Write-Host "Created provider.tf configuration"
}

try {
    # Verify provider binary exists in current directory
    if (-not (Test-Path $providerBinary)) {
        throw "Provider binary not found in current directory: $providerBinary"
    }

    # Create directory structure
    $providerDir = ".terraform/plugins/registry.terraform.io/hashicorp/$provider/$version/windows_amd64"
    New-Item -ItemType Directory -Force -Path $providerDir | Out-Null
    Write-Host "Created provider directory structure"

    # Copy provider binary
    Copy-Item -Path $providerBinary -Destination "$providerDir/$providerBinary" -Force
    Write-Host "Copied provider binary to: $providerDir/$providerBinary"

    # Create terraform.rc file
    Create-TerraformRC

    # Create provider configuration
    Create-ProviderConfig

    Write-Host "`nSetup completed successfully!`n"
    Write-Host "Next steps:"
    Write-Host "1. Run terraform init -plugin-dir='.terraform/plugins' to initialize your working directory"
    Write-Host "2. Create your terraform configuration files (.tf)"
    Write-Host "3. Run 'terraform plan' to verify the setup"

} catch {
    Write-Host "Error: $_" -ForegroundColor Red
    exit 1
}

Step 3: Run the Linux Script (Linux/macOS)

Save the following script as setup-local-provider.sh and execute it in your terminal:

#!/bin/bash

version="2.10.0"
provider="vsphere"
provider_binary="terraform-provider-$provider_v$version"

# Function to create terraform.rc file
create_terraform_rc() {
    cat <<EOF > ~/.terraformrc
provider_installation {
  filesystem_mirror {
    path = "$PWD/.terraform/plugins"
  }
  direct {
    exclude = ["registry.terraform.io/*/*"]
  }
}
EOF
    echo "Created terraform.rc at ~/.terraformrc"
}

# Function to create provider.tf file
create_provider_config() {
    cat <<EOF > provider.tf
terraform {
  required_providers {
    $provider = {
      source = "hashicorp/$provider"
      version = "$version"
    }
  }
}
EOF
    echo "Created provider.tf configuration"
}

# Main setup
if [ ! -f "$provider_binary" ]; then
    echo "Error: Provider binary not found in current directory: $provider_binary" >&2
    exit 1
fi

# Create directory structure
provider_dir=".terraform/plugins/registry.terraform.io/hashicorp/$provider/$version/linux_amd64"
mkdir -p "$provider_dir"
echo "Created provider directory structure"

# Copy provider binary
cp "$provider_binary" "$provider_dir/"
echo "Copied provider binary to: $provider_dir/$provider_binary"

# Create terraform.rc file
create_terraform_rc

# Create provider configuration
create_provider_config

echo -e "\nSetup completed successfully!\n"
echo "Next steps:"
echo "1. Run twtrito initialize your working directory"
echo "2. Create your terraform configuration files (.tf)"
echo "3. Run 'terraform plan' to verify the setup"

Step 4: Initialize Terraform

Run:

terraform init -plugin-dir='.terraform/plugins'

This tells Terraform to use the local plugin directory instead of trying to download the provider.

Step 5: Verify Your Setup

Try running:

terraform plan

If everything is set up correctly, Terraform should recognize the provider and not try to download it from the registry.

Unlocking the Potential of Cloudflare Workers for Small Projects

Ahmed Helmi — Sat, 27 Jul 2024 09:57:29 +0000

Cloudflare Workers is a serverless platform that allows developers to run code at the edge, close to the end users. Unlike some other serverless platforms that run in centralized data centers, Cloudflare Workers run in over 200 locations worldwide, providing lower latency and high performance. This article explores why Cloudflare Workers is a fantastic choice for smaller projects and offers a practical use case of creating a Telegram bot using Cloudflare Workers.

Why Choose Cloudflare Workers for Smaller Projects?

1. Low Latency and High Performance

Imagine your code running in data centers closest to your users, reducing latency significantly and providing a fast, responsive experience. Cloudflare Workers make this possible by running at the edge. This advantage is particularly crucial for smaller projects with a distributed user base. Learn more about Cloudflare's edge network.

2. Cost Efficiency

Budget constraints are a common concern for small projects. Cloudflare Workers offers a generous free tier that includes up to 100,000 requests per day, often more than enough for smaller projects. Beyond that, the pricing remains competitive, helping you keep costs minimal. See Cloudflare Workers pricing.

3. Ease of Deployment and Maintenance

One of the most appealing aspects of Cloudflare Workers is the simplicity of deployment. There's no need to manage servers or worry about infrastructure. This ease of use allows you to focus on what matters most: developing and deploying your project quickly. Get started with Cloudflare Workers.

4. Scalability

Even small projects can experience unexpected traffic spikes. Cloudflare Workers scale automatically to handle increased load, ensuring your project remains responsive and reliable without requiring any intervention. Learn about Cloudflare’s scalability.

5. Built-in Security

Security is crucial, especially for web applications. Cloudflare provides built-in DDoS protection, SSL/TLS encryption, and other security features to protect your application without additional configuration or cost. For smaller projects, this means robust security without the hassle. Explore Cloudflare’s security features.

Key-Value Storage and Database Services

Cloudflare offers additional services that complement Cloudflare Workers, making it even more powerful for small projects.

Cloudflare Workers KV

Workers KV is a global, low-latency key-value storage system. It allows you to store and retrieve data quickly, making it perfect for caching, configuration, and session management. With Workers KV, you can:

Store data globally: Data is replicated across Cloudflare's edge network, providing high availability and low latency access.
Manage configurations: Store configuration data that needs to be quickly accessible by your Worker scripts.
Cache responses: Cache static assets or frequently accessed data to improve performance.

Learn more about Workers KV.

Cloudflare D1

Cloudflare D1 is a managed SQL database built on SQLite, designed for serverless applications. It integrates seamlessly with Cloudflare Workers, enabling you to use a familiar SQL interface for data management. D1 is ideal for:

Prototyping and development: Quickly spin up a database for your applications without managing infrastructure.
Small to medium-sized applications: Use a scalable SQL database for applications that require relational data storage.

Learn more about Cloudflare D1.

Perfect for Prototyping and Hackathons

Cloudflare Workers shine in environments where rapid development and deployment are crucial, such as prototyping and hackathons.

1. Rapid Deployment

In hackathons and prototyping, time is of the essence. Cloudflare Workers enable you to deploy your code almost instantly without setting up servers or worrying about infrastructure, allowing you to focus on developing features and iterating quickly.

2. Minimal Setup

Getting started with Cloudflare Workers requires minimal setup, making it perfect for hackathons where simplicity and speed are vital.

3. Real-Time Testing

Cloudflare Workers run at the edge, allowing you to test your application in real-time and see how it performs under different network conditions. This immediate feedback is invaluable for refining your prototype or hackathon project on the fly.

4. Scalable Infrastructure

During a hackathon, you may not know how many users will interact with your project. Cloudflare Workers' ability to scale automatically ensures that your application can handle unexpected spikes in traffic, providing a seamless experience for users and judges alike.

5. Cost-Effective Experimentation

Hackathons and prototypes often operate on a limited budget. With Cloudflare Workers' generous free tier, you can experiment with different ideas without worrying about incurring significant costs. This allows for more creative freedom and innovation.

USECASE: Creating a Telegram Bot with Cloudflare Workers

Let's create a simple Telegram bot using Cloudflare Workers, integrating Workers KV for storing user data and Cloudflare D1 for managing a simple database. This bot will respond to user messages with a predefined response and store user messages in a database.

Step 1: Set Up Your Cloudflare Worker

Create a Cloudflare Account: If you don’t have one, sign up at Cloudflare.
Set Up a Worker: In the Cloudflare dashboard, navigate to the Workers section and create a new Worker.

Step 2: Configure Workers KV

Create a KV Namespace: In the Workers section of the Cloudflare dashboard, go to the "KV" tab and create a new namespace. Note the KV_NAMESPACE_ID.
Bind the KV Namespace to Your Worker: In the Worker configuration, add a binding for the KV namespace:

{
  "bindings": [
    {
      "type": "kv_namespace",
      "name": "MY_KV_NAMESPACE",
      "namespace_id": "KV_NAMESPACE_ID"
    }
  ]
}

Step 3: Set Up Cloudflare D1

Create a D1 Database: In the Cloudflare dashboard, go to the D1 section and create a new database. Note the D1_DATABASE_ID.
Bind the D1 Database to Your Worker: In the Worker configuration, add a binding for the D1 database:

{
  "bindings": [
    {
      "type": "d1",
      "name": "MY_D1_DATABASE",
      "database_id": "D1_DATABASE_ID"
    }
  ]
}

Step 4: Write the Bot Code

Here’s a basic example of a Telegram bot using Cloudflare Workers, Workers KV, and Cloudflare D1:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url)
  const pathname = url.pathname

  if (pathname === '/webhook') {
    const data = await request.json()
    const message = data.message
    const chatId = message.chat.id
    const text = 'Hello from Cloudflare Workers!'

    // Store user message in KV
    await MY_KV_NAMESPACE.put(chatId, message.text)

    // Insert user message into D1 database
    await MY_D1_DATABASE.query(
      `INSERT INTO messages (chat_id, message) VALUES (?, ?)`,
      [chatId, message.text]
    )

    await sendMessage(chatId, text)
    return new Response('OK')
  }

  return new Response('Not Found', { status: 404 })
}

async function sendMessage(chatId, text) {
  const token = 'YOUR_TELEGRAM_BOT_TOKEN'
  const url = `https://api.telegram.org/bot${token}/sendMessage`

  const response = await fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      chat_id: chatId,
      text: text
    })
  })

  return response.json()
}

Step 5: Configure Telegram Webhook

Create a Telegram Bot: If you haven't already, create a bot by talking to BotFather on Telegram.
Set the Webhook URL: Use the following command to set your webhook URL to your Cloudflare Worker URL:

   https://api.telegram.org/bot<YOUR_BOT_TOKEN>/setWebhook?url=https://<YOUR_WORKER_SUBDOMAIN>.workers.dev/webhook

Step 6: Test Your Bot

Send a message to your bot on Telegram, and it should respond with "Hello from Cloudflare Workers!" It will also store the message in the Workers KV and insert it into the D1 database.

Conclusion

Cloudflare Workers provide a powerful, cost-effective, and scalable solution for smaller projects. With low latency, ease of deployment, built-in security, and scalability, Cloudflare Workers are an excellent choice for developers looking to build and deploy applications quickly. The example of creating a Telegram bot demonstrates how simple and effective it can be to use Cloudflare Workers, Workers KV, and Cloudflare D1 for your projects.

Whether you're prototyping a new idea or participating in a hackathon, Cloudflare Workers offer the speed, simplicity, and flexibility needed to turn your vision into reality swiftly and efficiently. And with the added power of Cloudflare’s key-value storage and managed database services, you have all the tools you need to build robust and responsive applications.

Google Chrome DevTools: Handy Tips

Ahmed Helmi — Mon, 25 Dec 2023 15:26:46 +0000

Google Chrome DevTools is a powerful set of web development tools built directly into the Chrome browser. It provides a wide range of features that help developers debug, optimize, and understand web applications. In this article, we'll unravel more features, sprinkle in additional tips. Let's dive in!

Gateway to DevTools Magic

To open devtools just right-click anywhere on a webpage, choose "Inspect," and voila! If you're into shortcuts, hit Ctrl+Shift+I (Windows/Linux) or Cmd+Opt+I (Mac). Also you can find it in the Chrome menu under "More Tools" > "Developer Tools."

Dark Mode: Click the gear icon in the top-right corner of DevTools and choose "Preferences." Embrace the darkness by enabling the "Dark Theme" for a sleek and magical appearance.

Play with Devices Touch and Feel: Toggle the "Toggle device toolbar" button to simulate touch interactions. Feel the of touch events without getting fingerprints on your screen.

Console tricks!

a. Magic Commands
$_ to the Rescue: Use $_ in the console to reference the result of the last expression. It's like summoning the spirit of your previous command.

b. Fancy Logging
Styling Your Logs: Add %c to your console.log and apply CSS styles to your log messages. Spice up your console with colors and fonts.

c. Table Talk
Multidimensional Tables: Nest arrays and objects within your data for a table. :)

The Story of Your Code in the Source Panel

a. Hide and Seek
Navigate with Precision: Use Ctrl+P (Windows/Linux) or Cmd+P (Mac) to quickly navigate and find files in the source panel.

It's like having a file finder of vscode.

b. Breakpoints with Conditions

Logpoint : Instead of stopping at a breakpoint, set a "Logpoint" to log a message to the console. Keep the magic flowing without interrupting the spell ;)

Network Adventures

a. Speed Bumps

Online, Offline, and Beyond: Simulate offline experiences by checking the "Offline" option in the Network panel. Experience the magic of how your website behaves without an internet connection.

b. Copy-Paste Wizardry

Preserve the Magic: When copying an object from the console, use copy(myObject) to preserve the object in its original form. No more losing properties during the copy-paste ritual.

Performance Magic Show

a. Lights, Camera, Action!
Frame Mode: In the Performance panel, switch to "Frames" mode to analyze your website's performance frame by frame. It's like directing a movie of your web page.

Application Playground

a. Storage Treasure Hunt
IndexedDB Explorer: Under the "Application" panel, explore IndexedDB with the "IndexedDB" tab.]

b. Application Manifest Magic
Manifest Insights: Check the "Manifest" tab to inspect your web app's manifest file.

Miscellaneous Enchantments

Search Everywhere: Press Ctrl+Shift+F (Windows/Linux) or Cmd+Opt+F (Mac) to search across all DevTools panels. A magical way to find what you seek without going too far.

In Conclusion
Google Chrome DevTools is a versatile tool that can greatly improve your web development workflow. By incorporating these tips and tricks into your routine, you'll be able to debug more efficiently, optimize performance, and gain a deeper understanding of your web applications.

Cloud Computing Made Easy: A Beginner's Guide

Ahmed Helmi — Thu, 09 Nov 2023 07:45:52 +0000

In the digital age, cloud computing has become a fundamental component of our daily lives. Whether we're storing files, streaming content, or running business applications, the cloud plays a crucial role in how we access and manage data and services. In this article, we will explore the basic concepts of cloud computing, demystifying the technology that powers the digital world.

What is Cloud Computing?

At its core, cloud computing refers to the delivery of computing services, including storage, processing power, and software, over the internet. Instead of relying on local servers or personal devices, users access these resources remotely through data centers maintained by cloud service providers.

Key Characteristics

a. On-Demand Self-Service: Cloud users can provision and manage resources as needed, without requiring human intervention from the service provider.

b. Broad Network Access: Cloud services are accessible over the internet from a variety of devices, such as laptops, smartphones, and tablets.

c. Resource Pooling: Cloud providers pool resources to serve multiple customers. These resources are dynamically allocated and reassigned as needed.

d. Rapid Elasticity: Cloud resources can be quickly scaled up or down to accommodate changing demands, ensuring efficient resource utilization.

e. Measured Service: Cloud usage is metered, allowing users to pay only for the resources they consume. This pay-as-you-go model is cost-effective and flexible.

Service Models

Cloud computing offers various service models to cater to different user needs:

a. Infrastructure as a Service (IaaS): IaaS provides virtualized computing resources, such as virtual machines, storage, and networking. Users have control over the operating system and applications.

b. Platform as a Service (PaaS): PaaS offers a platform that allows developers to build, deploy, and manage applications without worrying about the underlying infrastructure.

c. Software as a Service (SaaS): SaaS delivers software applications over the internet on a subscription basis. Users access the software through a web browser without installing or maintaining it locally.

Deployment Models

Cloud computing can be deployed in various ways to meet different requirements:

a. Public Cloud: Public cloud services are provided by third-party cloud service providers and are available to anyone over the internet. They are cost-effective and scalable but may raise security and privacy concerns.

b. Private Cloud: Private clouds are built and maintained by organizations for their exclusive use. They offer greater control, security, and customization but can be more expensive to set up and manage.

c. Hybrid Cloud: Hybrid cloud combines public and private cloud services, allowing data and applications to be shared between them. This model provides flexibility and scalability while addressing security and compliance concerns.

d. Community Cloud: A community cloud is shared by a specific group of organizations with common interests, such as regulatory compliance. It offers a balance between public and private clouds.

Benefits of Cloud Computing

Cloud computing offers several advantages, including:

a. Cost Savings: Organizations can reduce capital expenses and only pay for the resources they use.

b. Scalability: Businesses can easily scale up or down based on demand, ensuring efficient resource allocation.

c. Accessibility: Cloud services are accessible from anywhere with an internet connection.

d. Reliability: Leading cloud providers offer high availability and redundancy, reducing downtime.

e. Security: Reputable cloud providers invest heavily in security measures to protect data and infrastructure.

Challenges and Considerations

While cloud computing provides numerous benefits, it also comes with its own set of challenges:

a. Security and Privacy: Organizations must trust cloud providers with their data, and data breaches or unauthorized access can be a concern.

b. Compliance: Companies in regulated industries may face challenges in meeting compliance requirements when using the cloud.

c. Downtime: Internet connectivity issues or outages on the provider's end can disrupt services.

d. Data Transfer Costs: Moving data in and out of the cloud can result in additional expenses.

Conclusion

Cloud computing is a transformative technology that has revolutionized the way individuals and organizations access and manage their digital resources. By understanding the basic concepts, service models, deployment options, and potential benefits and challenges, you can make informed decisions about how to leverage the power of the cloud in your personal or business endeavors.

🌟 Keeping It Simple: Usable Security for the Everyday User 🌟

Ahmed Helmi — Tue, 15 Aug 2023 16:08:43 +0000

Hey there, folks! Let's dive into a topic that affects all of us in the digital world: usable security. Usable security refers to the design and implementation of security measures and systems in a way that prioritizes the user experience and usability, while still maintaining a high level of protection against security threats. It aims to strike a balance between effective security practices and the ease of use for individuals interacting with those security measures.

🔒 The Struggle is Real:
We've all been there – faced with complicated security measures that make us want to pull out our hair. Who can blame us? Remembering long, convoluted passwords filled with a mix of uppercase, lowercase, symbols, and a secret handshake? It's enough to give anyone a headache. But fear not, my friends, because usable security is here to save the day!

🔐 Putting the "User" in "Usable":
Usable security is all about understanding us, the users. It's about designing security systems that fit our needs and capabilities like a glove. After all, security should be for everyone, not just tech gurus. So, let's bring on the user-friendly interfaces, straightforward instructions, and intuitive workflows. We want security measures that won't make us feel like we need a degree in rocket science to navigate them!

In the context of usable security, the focus is on creating security solutions that are accessible, intuitive, and user-friendly, reducing the cognitive load and frustration often associated with traditional complex security measures. The goal is to encourage users to adopt and comply with security practices by making them understandable, straightforward, and seamlessly integrated into their digital interactions.

Usable security takes into account human factors, such as cognitive abilities, behavioral patterns, and user expectations. It involves designing clear and concise user interfaces, providing understandable instructions and feedback, and aligning security measures with user needs and capabilities. By doing so, usable security aims to enhance user acceptance and cooperation, ultimately leading to improved security outcomes.

💡 Tips for a User-Friendly Security Experience:
1️⃣ Keep It Simple, Silly: Let's simplify the complex. No one wants to feel like they're decoding the Da Vinci Code just to access their accounts. Streamline security processes and present information in plain language. We're all about keeping it simple and stress-free.

2️⃣ Be Transparent: Trust is the name of the game. Be open and honest about security risks, privacy policies, and why certain security measures are necessary. When we understand the "why" behind the hoops we're jumping through, we're more likely to cooperate.

3️⃣ Show Us the Way: User-friendly interfaces are our best friends. Invest in designs that guide us through security processes with ease. No more guessing games or feeling like we're wandering in a digital labyrinth. Let's make security a walk in the park!

4️⃣ Educate and Empower: Knowledge is power. Educate us about common security threats, best practices, and why security matters. Teach us how to fish, so we can protect ourselves even when you're not around. We're in this together!

5️⃣ Evolve and Adapt: The digital landscape is ever-changing, and so are the threats we face. Stay on top of the latest trends, research, and industry standards. Continuously improve security measures based on user feedback and emerging technologies. Let's keep our defenses up-to-date and our security game strong.

🔒 A Brighter, User-Friendly Future:
Usable security is our ticket to a safer digital world without the headaches. By prioritizing user experience and making security accessible to everyone, we can create a future where protection and simplicity coexist in harmony.

So, my friends, let's champion usable security! Share your thoughts, funny security anecdotes, or tips in the comments below. Together, we can make security a friend, not a foe. 🔒💻

Recording and Replaying HTTP Interactions with Ease: A Guide to VCR.py

Ahmed Helmi — Wed, 05 Jul 2023 03:19:49 +0000

Introduction

Testing code that interacts with external services can be tricky. You need to make sure that your code is handling all the possible responses correctly, and that it's not making too many unnecessary requests. But how do you test this kind of code reliably and efficiently?

That's where VCR.py comes in. The idea behind VCR.py is simple: it records HTTP interactions made by your code, and then replays them when your tests run. That way, your tests can simulate HTTP requests without actually hitting the external service. Pretty neat, right?

In this article, we'll take a closer look at VCR.py and some of its features. We'll cover how to use VCR.py to record and replay HTTP interactions, how to customize its behavior with matchers and filters, and how to work with the cassette file that VCR.py uses to store recorded interactions.

Getting Started

To get started with VCR.py, you'll need to install it first. You can do this using pip:



pip install vcrpy

Once you have VCR.py installed, you can start using it in your tests. Here's a simple example:



import requests
import vcr

with vcr.use_cassette('test_api.yaml', record_mode='once'):
    def test_api():
        response = requests.get('https://api.example.com')
        assert response.status_code == 200

In this example, we're using VCR.py to intercept a GET request to https://api.example.com. The first time this test is run, VCR.py will record the HTTP interaction in a cassette file named test_api.yaml. On subsequent runs, VCR.py will replay the recorded interaction from the cassette file instead of making a new request.

Record_mode that passed in with the VCR cassette determines how the interactions with an external API are recorded. There are four record modes available:

once: This mode will record the interactions with the external API the first time the test is run. Subsequent runs of the test will use the recorded interactions instead of making new requests to the API.
new_episodes: This mode will record any interactions with the external API that haven't been previously recorded. If an interaction has already been recorded, it will be replayed instead of making a new request.
all: This mode will record all interactions with the external API, regardless of whether they have been previously recorded or not. This is useful for building up a complete set of recordings for an API.
none: This mode will turn off recording altogether and will only replay previously recorded interactions. This mode is useful when you want to ensure that your tests are only using recorded interactions and not making any new requests to the external API.

In the previous example, the record_mode argument is set to 'once', which means that the interactions with the external API will be recorded the first time the test is run and subsequently replayed on all subsequent runs.

Working with the Cassette File

The cassette file is where VCR.py stores recorded HTTP interactions. By default, the cassette file is a YAML file, but you can also use JSON or SQLite. The cassette file can be easily read and edited, which makes it easy to inspect the contents of the file and make changes if necessary.

Here's an example of a cassette file:



- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "123", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK

In this example, we have a single interaction that was recorded. The interaction consists of a POST request to https://api.example.com/foo with a JSON request body, and a response with a JSON body containing an ID and a foo value.

One important thing to keep in mind when working with the cassette file is that it can become quite large if you have a lot of interactions recorded. To help with this, VCR.py provides several options for controlling how interactions are recorded and stored in the cassette file.

Matchers

By default, VCR.py matches HTTP interactions based on the HTTP method, URL, and request body. But what if you want to match on other criteria, like headers or query parameters?

That's where matchers come in. Matchers allow you to specify custom criteria for matching HTTP interactions. Here's an example:



import vcr

def custom_matcher(request1, request2):
    # Your custom matching logic here
    return True

with vcr.use_cassette('my_test.yaml', 
                      match_on=['method', 'uri', custom_matcher]):
    def test_my_function():
        # Code that makes HTTP requests

In this example, we define a custom matcher function custom_matcher that takes two requests and returns True if they match based on our custom criteria. We then use this custom matcher in our use_cassette call, along with the default matchers for HTTP method and URI.

Filters

Sometimes you might want to filter or modify requests or responses before they are recorded or played back. For example, you might want to filter out sensitive data from the response before it gets stored in the cassette file. Or you might want to modify the request headers before the request is sent.

VCR.py provides two options for filtering: before_record and before_playback. These options allow you to specify functions that are called before requests are recorded or played back, respectively. Here's an example:



import vcr

def filter_request(request):
    request.headers['Authorization'] = 'Bearer xxxxxxx'
    return request

def filter_response(response):
    response.headers.pop('Set-Cookie', None)
    return response

with vcr.use_cassette('my_test.yaml',
                      before_record=filter_request,
                      before_playback=filter_response):
    # Your test code here

In this example, we define two filter functions filter_request and filter_response that modify the request and response respectively. We then pass these filter functions to the before_record and before_playback options of use_cassette, respectively.

Placeholders

Sometimes you might have dynamic data in your responses that you don't want to hard-code in your tests. For example, if you have a test that creates a new resource and returns a unique ID, you might want to use a placeholder to represent that ID in the cassette file.

VCR.py supports placeholders in the cassette file. Placeholders are strings that are replaced with actual values during playback. Here's an example:



- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "{{ ID }}", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK

In this example, we're using the {{ ID }} placeholder in the response body to represent the unique ID that is generated by the API. When VCR.py replays the response, it will substitute the placeholder with the actual ID value that was returned by the API during the recording.

Basic Illustration

Let's assume that we have a simple Flask web application that acts as an external web service API. It returns a JSON response when we make a GET request to the /hello endpoint:



from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/hello')
def hello():
    return jsonify({'message': 'Hello, World!'})

if __name__ == '__main__':
    app.run()

Now, let's write a vcrpy test to test this API. We'll use the requests library to make the HTTP request to the API:



import vcr
import requests

def test_hello_api():
    with vcr.use_cassette('hello_api.yaml', record_mode='once'):
        response = requests.get('http://localhost:5000/hello')
        assert response.status_code == 200
        assert response.json() == {'message': 'Hello, World!'}

In this test, we're using the use_cassette context manager to wrap the HTTP request to our Flask app. We're specifying that we want to use the cassette file hello_api.yaml to record the interactions with the API, and we're using the record_mode argument to specify that we only want to record the interactions once (i.e., on the first run of the test).

When we run this test for the first time, vcrpy will record the interaction with the Flask app and save it to the hello_api.yaml file. On subsequent runs of the test, vcrpy will replay the interaction from the cassette file instead of making a new HTTP request to the Flask app.

To run this test, we can simply execute the test file using a test runner such as pytest:



$ pytest test_hello_api.py

This will run the test and output the results of the assertions. If the test passes, we should see output similar to the following:



============================ test session starts ============================
collected 1 item

test_hello_api.py .                                                   [100%]

============================= 1 passed in 0.13s =============================

If the test fails, we'll see an error message indicating which assertion failed.

here's what the hello_api.yaml cassette file might look like after vcrpy records the HTTP interaction with the Flask app (ie. our external API):



interactions:
- request:
    body: null
    headers:
      Accept-Encoding: identity
      Connection: keep-alive
      Host: localhost:5000
      User-Agent: python-requests/2.25.1
    method: GET
    uri: http://localhost:5000/hello
  response:
    body:
      encoding: utf-8
      string: '{"message": "Hello, World!"}'
    headers:
      Connection: Keep-Alive
      Content-Length: '25'
      Content-Type: application/json
      Date: Mon, 05 Jul 2023 12:34:56 GMT
      Keep-Alive: timeout=5, max=100
      Server: Werkzeug/2.0.2 Python/3.9.5
    status:
      code: 200
      message: OK
  recorded_at: 2023-07-05T12:34:56.789012Z
  recorded_with: vcrpy/4.1.1

This file contains a YAML representation of the HTTP interaction between the test and the Flask app. The interactions section contains a list of all the HTTP interactions that were recorded, in this case just one. The request and response sections contain the request and response headers and bodies, respectively. The recorded_at and recorded_with fields indicate when and with what version of vcrpy the interaction was recorded.

Conclusion

VCR.py is a powerful tool for testing code that interacts with external services. Its ability to record and replay HTTP interactions makes it easy to write tests that are reliable and efficient. With VCR.py, you can customize how interactions are matched and filtered, and work with the cassette file to inspect and modify recorded interactions.

I hope this article has given you a good introduction to VCR.py and its features. If you're interested in learning more, I encourage you to check out the official documentation and start experimenting with VCR.py in your own tests.

Test-Driven Development in Python: Best Practices, and Detailed Explanation

Ahmed Helmi — Thu, 29 Jun 2023 15:47:37 +0000

Hey there, Python lovers! Are you looking for a way to improve your software development process and build more maintainable code? Look no further than test-driven development (TDD) in Python! In this article, we’ll explore the benefits of TDD, best practices for implementing it, and some fun example to get you started.

Why You Should Give TDD a Try:

TDD can provide a variety of benefits when developing Python applications. By writing tests before writing code, you can catch errors and bugs early, reducing the time and cost required for testing and debugging. This leads to better code quality, faster development time, better collaboration among team members, and improved confidence in your code. Who wouldn’t want those benefits?

Best Practices for TDD in Python:

The TDD process is often described as “Red-Green-Refactor”. The “Red” phase involves writing a test that fails, which means that the code being tested hasn’t been written yet. The “Green” phase involves writing the minimum amount of code required to pass the test. Once the test passes, the “Refactor” phase involves improving the code’s design and efficiency without changing its behavior.

Write tests first: This ensures that your code meets the requirements of the project and works as expected.
Write small and simple tests: Testing one aspect of the code at a time makes it easier to identify and fix errors and bugs.
Test edge cases: Testing unexpected inputs and outputs ensures that your code can handle anything that comes its way.
Use unittest: Python’s built-in testing framework is easy to use and helps you write and run tests quickly.
Run tests frequently: Catch errors and bugs as soon as they are introduced by running tests frequently during development.

Tools that maybe handy:

Test Coverage: Test coverage is a metric that measures the percentage of the code that is covered by tests. Achieving high test coverage is important, as it ensures that all parts of the code are thoroughly tested. Python provides several tools, such as coverage.py, that can be used to measure test coverage.

Test Doubles: Test doubles are objects that are used in place of other objects during testing. For example, a mock object can be used to simulate the behavior of a real object. Python provides several libraries, such as unittest.mock, that can be used to create test doubles.

Test-Driven Data Analysis (TDDA): Test-driven data analysis is a variant of TDD that is used in data science. TDDA involves writing tests for data cleaning, data preprocessing, and statistical models. By using TDDA, data scientists can ensure that their results are reproducible and that their code is correct.

HTTP interaction replay: It is like recording and playing back conversations between your Python program and a web server. It’s useful for testing, debugging, and mocking web-based applications. When you use a library like VCR.py it records the HTTP requests and responses, and then plays them back later when you need them.

TDD with python in action:

Let’s consider a feature for an e-commerce application where a customer can add a product to their cart. Here’s an example of how you might use TDD to develop this feature in Python:

Write a failing test: The first step in TDD is to write a test that fails. In this case, we might write a test that checks whether a customer’s cart is updated correctly when they add a product:

def test_add_to_cart():
 customer = Customer()
 product = Product(‘123’, ‘Widget’, 9.99)
 customer.add_to_cart(product)
 assert customer.cart == {‘123’: {‘name’: ‘Widget’, ‘price’: 9.99, ‘quantity’: 1}}

Run the test: When you run the test, it should fail because we haven’t implemented the add_to_cart method yet.

Write the implementation: Now we can write the implementation code for the add_to_cart method. It might look something like this:


class Customer:
    def __init__(self):
        self.cart = {}

    def add_to_cart(self, product):
        if product.id in self.cart:
            self.cart[product.id]['quantity'] += 1
        else:
            self.cart[product.id] = {'name': product.name, 'price': product.price, 'quantity': 1}

Run the test again: Now that we’ve implemented the add_to_cart method, we can run the test again. This time, it should pass.

Refactor: If necessary, we can now refactor our code to make it more efficient, readable, and maintainable. For example, we might extract the cart item creation logic into a separate method:

class Customer:
    def __init__(self):
        self.cart = {}

    def add_to_cart(self, product):
        if product.id in self.cart:
            self.cart[product.id]['quantity'] += 1
        else:
            self.cart[product.id] = self._create_cart_item(product)

    def _create_cart_item(self, product):
        return {'name': product.name, 'price': product.price, 'quantity': 1}

Repeat: We can now continue this cycle of writing failing tests, implementing the code to make them pass, and refactoring as necessary until we’ve fully developed and tested the feature.

Conclusion:

In conclusion, test-driven development is a great way to develop Python programs that are reliable, maintainable, and bug-free. By writing test cases before writing the implementation, we can catch errors and bugs early in the development process and avoid costly debugging later on. With practice, you’ll find that TDD becomes an intuitive and efficient way to develop software.