DEV Community: Keshav Malik

How to set up Django with Postgres from Neon

Keshav Malik — Sat, 10 Feb 2024 07:22:23 +0000

In this blog post, you’ll learn how to connect Django to Neon Serverless Postgres. We’ll start with a simple Django CRUD app as our example and demonstrate how to integrate Neon as the backing database.

If you’re unfamiliar with Neon, consider it an open-source, standard Postgres version of AWS Aurora. It is PostgreSQL delivered as a cloud service, but it goes a step further than other managed Postgres services. Under the hood, Neon has separated the compute and storage layers of the database so that, as users, we simply get a URL to our database, and Neon handles the rest. When we’re not using the DB, Neon scales compute to zero, and we pay nothing; when usage goes up, Neon scales up to handle the load. You can sign up for their free tier here to follow along.

Prerequisites 🤔

Before getting started with this tutorial, you’ll need:
Git installed on your local machine - to install it, follow this guide.

Python 3.* Installed - follow this guide if you don't have it already
Pip - Python package manager
Django 5.0 - follow this guide to install Django.
Basic knowledge of Python and Django concepts and conventions

Getting Started with Django Setup 💡

Let’s start by setting up a CRUD application using Django before we dive into the Neon part. If you already know Django, you can proceed to the next section. Here, we will create a basic Django project, set up an app, define our model, and perform our first migration.

Our CRUD app is a note-taking application with creation, reading, updating, and deleting capabilities. Please find the source code for the Django project available here.

To follow along with the tutorial, clone the project by running:



git clone https://github.com/theinfosecguy/neon-django-api.git

From your terminal, open up the project in your IDE of choice. We’ll first set up a virtual environment with pipenv. Install it with the pip install pipenv command.

Create a new virtual environment using the pipenv shell command. Now that our virtual environment is created let’s install Django using the pipenv install django command.

Lastly, install Django REST Framework using pipenv install django_rest_framework command.

We’re done with initial dependencies we’ll require. To test your progress so far, run:



python3 manage.py runserver

You’ll see an error saying Error loading psycopg2 or psycopg module which means you’re all set to follow along. We’ll resolve this further in the article.

Another problem is that Django uses sqlite database by default. We’ll switch to Postgres after integrating Neon in the upcoming sections.

Neon Setup and Integration ⚡

Let’s walk you through the process of configuration of Neon with Django. This is how we’ll hook Neon up to our Django project, set up the initial database and tables, and map our Django models into Neon’s schema.

Firstly, you need to have a Neon account. Sign up at neon.tech and create a Neon Project. Each Project is its own Postgres Cluster.

Once done, it will open up a prompt with the connection details for your newly created project. Select Django from the dropdown and copy the database configuration to your clipboard.

Then, head to the project directory, open the file neon_project\settings.py, and find the variable DATABASES. It’ll look somewhat like this as per typical Django boilerplate code where, by default, it uses a SQLite database under the hood.



DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    }
}

Replace the value of this with the new configuration.



DATABASES = {
  'default': {
    'ENGINE': 'django.db.backends.postgresql',
    'NAME': 'neon_example_db',
    'USER': 'your-username-here',
    'PASSWORD': 'your-password-here',
    'HOST': 'ep-steep-recipe-44764255.us-east-2.aws.neon.tech',
    'PORT': '5432',
    'OPTIONS': {'sslmode': 'require'},
  }
}

Follow through the rest of the article however we strongly advise that you revisit this after local testing. Switch to either environment variables or adopt a secret management service such as Hashicorp Vault or AWS KMS to store these secrets. Here’s an example of an environment variable-based approach with Python.



DATABASES = {
  'default': {
    'ENGINE': os.environ.get('db_engine', 'django.db.backends.postgresql'),
    'NAME': os.environ.get('db_name'),
    'USER': os.environ.get('db_user'),
    'PASSWORD': os.environ.get('db_password'),
    'HOST': os.environ.get('db_host'),
    'PORT': os.environ.get('db_port', '5432'),
    'OPTIONS': {'sslmode': 'require'},
  }
}

For instance, the .env file could look like the below example.



db_engine=django.db.backends.postgresql
db_name=neon_example_db
db_user= # your-username-here
db_password= # your-password-here
db_host=ep-steep-recipe-44764255.us-east-2.aws.neon.tech
db_port=5432

Lastly, we need a Postgres database driver for Django. We’ll install [psycopg](https://pypi.org/project/psycopg2/), which should serve the purpose of this tutorial. This will resolve the “Error loading psycopg2 or psycopg module” error we faced while setting up the project initially.



pipenv install psycopg2-binary

Once installed, run the following command so that Django can read the new settings. Django will automatically make the necessary migrations to initialize our Neon database.



python3 manage.py migrate

Now that our database is ready, run the following command to run the development server.



python3 manage.py runserver

Django provides a built-in admin panel that we can use for read and write API operations. Head to http://127.0.0.1:8000/notes/api/ and create a new note entry.

Once the new note entry is created, refresh the page. You’ll see that the new record is created and is being fetched with the GET All Notes API call on the top.

Next, we can verify the same from the Neon interface. Head to https://console.neon.tech/app/projects/{{your_project_id_here}}/tables and scroll down in the tables section to our notes table. You can see that the records are visible in the interface, too.

We have now successfully integrated NeonDB with our Django app 🔥. NeonDB is incredibly easy to integrate into a Django project, providing smooth access to perform database tasks.

For instance, in this tutorial, we've strategically switched from our ordinary SQLite database to a Postgres one to perfectly match the features NeonDB offers. This transition also highlights the adaptability and integration that NeonDB provides for Django development. Stay tuned for part two of this tutorial where we demonstrate some of the unique branching features of Neon with our CRUD app.

How to Secure Your CI/CD Pipelines with GitGuardian Honeytokens

Keshav Malik — Mon, 10 Jul 2023 06:56:11 +0000

In the realm of software development, Continuous Integration and Continuous Deployment (CI/CD) pipelines have become integral. They streamline the development process, automate repetitive tasks, and enable teams to release software quickly and reliably. But while CI/CD pipelines are a marvel of modern development practices, they also present potential security vulnerabilities.

With the integration of various tools, systems, and environments, CI/CD pipelines often deal with sensitive information, making them a potential target for cyber attacks. Consequently, it's crucial to weave in robust security measures to protect these pipelines and maintain the integrity of your development process.

One such proactive measure is the deployment of honeytokens. Created within GitGuardian's Honeytoken module, honeytokens are essentially digital baits designed to attract and detect unauthorized access or activity within your systems. By adding honeytokens to your pipelines, you can add an extra layer of defense against potential intruders.

In the following sections, we will delve into what honeytokens are, how they work to detect intrusions, and how you can effectively implement them in your CI/CD pipelines to enhance security. Stay tuned to uncover how you can leverage the power of Honeytokens to protect your digital assets better.

Decoding Honeytokens: An Extra Layer of Defense

A honeytoken is a deceptive instrument, a digital decoy strategically positioned to attract potential intruders.

These tokens have no legitimate operational value. Their sole purpose is to serve as bait and alert you if someone tries to use them. When an unauthorized user attempts to use a honeytoken, it triggers an alert, letting you know of the intrusion. It's like having a silent alarm in your house that triggers when a burglar tries to unlock the safe.

Not only does a honeytoken alert you of the intrusion, but it also provides crucial information about the attacker. You get details like the IP address and the user agent of the intruder, helping you in further securing your system and even tracking down the threat.

Introducing honeytokens to your CI/CD pipelines, especially within your platform's environment variables, echoes the essence of a diligent watchman, standing guard around the clock. This ever-alert sentinel stands its ground, much like during the infamous CircleCI security incident in early 2023 (the detailed account can be read here).

Imagine if an intruder had an audacious scheme to exploit these 'secret' variables, say your AWS Keys. The very moment they attempted, your trusty honeytoken watchman would sound the alarm. The alert would reach you in a flash, giving you ample opportunity to intervene, halt the dubious activity, and glean crucial lessons to fortify your digital infrastructure.

In the upcoming sections, we will guide you on how to implement honeytokens in various CI/CD pipelines, including Jenkins, CircleCI, Travis CI, GitLab CI/CD, Azure DevOps Pipelines, and AWS CodePipeline.

➡️ Before getting started, follow the step-by-step instructions below to get started creating your first honeytoken! 👇

How to Create and Use Honeytokens: Step-by-Step Instructions

Boosting Travis CI Security using Honeytokens

If you're involved in software development, you're likely familiar with Travis CI. As a popular Continuous Integration (CI) service, Travis CI is used to build and test software projects hosted on GitHub and Bitbucket. While Travis CI already has its own measures to ensure the security of your software projects, honeytokens can provide an extra layer of defense to help identify and respond to potential threats.

To enhance your Travis CI security, you can strategically place honeytokens in your build scripts or configuration files. Files like .travis.yml, environment variables, or even in your test scripts are some of the places you can consider. Remember, a honeytoken's value lies in its ability to deceive potential intruders, so placing them where sensitive data might usually reside can increase your chances of detecting a breach.

Here's how you can start boosting your Travis CI security with honeytokens:

Begin by creating a honeytoken using GitGuardian's Honeytoken module. Keep it handy for the upcoming steps.
Navigate to your Travis CI configuration file (.travis.yml or similar) or any script files where you wish to place your honeytoken.
Place the honeytoken in the selected location, treating it as you would any sensitive data.
Save the changes and ensure the updated files are pushed to your SCM repository.

Travis CI also uses environment variables to store secrets. You can add your honeytoken to your repository settings under the "Environment Variables" section. If someone tries to misuse these variables, the honeytoken will serve as an alarm bell.

Securing CircleCI Pipelines with Honeytokens

CircleCI is a widely adopted Continuous Integration and Delivery (CI/CD) platform. It simplifies the automation of your software development process, from building and testing to deploying your applications. Despite CircleCI's built-in security measures, enhancing it with honeytokens can further protect your CI/CD pipeline from malicious intrusions.

A honeytoken can be strategically added to several parts of your CircleCI pipeline. Some common locations include configuration files like config.yml, shell scripts, or even within environment variables. Similar to a secret key, an attacker accessing and using a honeytoken will trigger an alert, providing you with an early warning of a potential security breach.

Here's how you can integrate honeytokens into your CircleCI pipeline:

First, create a honeytoken in the GitGuardian Honeytoken module. Remember to note down the generated token for later use.
Go to your project repository and find the CircleCI configuration file, typically named config.yml and located in the .circleci directory.
Embed the honeytoken in this file. Once the honeytoken is in place, save your changes and push the updated file to your repository.

CircleCI uses environment variables as secure storage for secrets. Go to your project settings, find the "Environment Variables" section, and add your honeytoken there. If someone unauthorized gets ahold of these environment variables, your honeytoken will alert you right away.

Beefing up Jenkins Security with Honeytokens

Jenkins is an open-source automation server that allows developers to build, test, and deploy their applications. It's a popular choice for implementing Continuous Integration and Continuous Deployment (CI/CD) pipelines.

Honeytokens can be added to your Jenkins configuration files, script files, or even injected into your pipeline as a fake environment variable.

Here's a simple guide to integrating honeytokens into your Jenkins setup:

Start by creating a honeytoken in the GitGuardian Honeytoken module. Remember to take note of the generated token.
Navigate to your Jenkins pipeline script, typically a Jenkinsfile found in your project repository.
Add the honeytoken into the script. It could be disguised as a decoy API key, a dummy password, or another type of attractive bait for an attacker.
Once you've positioned your honeytoken, save the changes and commit the updated script to your repository.

Should an attacker stumble upon and use this honeytoken, an alert will be triggered. The alert includes crucial information about the attempted security breach, enabling you to quickly address the threat.

Jenkins offers a plugin called "Credentials" that's perfect for storing sensitive data. It's an ideal place to add your honeytoken as it's designed to keep secrets, well, secret! By doing this, should an attacker ever gain unauthorized access to Jenkins and attempt to exploit these credentials, you'll be instantly alerted.

Also read: How to Handle Secrets in Jenkins

Upping GitLab CI/CD Security Ante with Honeytokens

GitLab is a powerful DevOps tool that provides a unified platform for source code management, CI/CD, and much more. GitLab CI/CD is a part of GitLab that provides the tools necessary for software development and deployment.

As with any CI/CD platform, securing your GitLab pipelines is crucial. One method of enhancing your security measures is by utilizing honeytokens. These tokens can be placed in configuration files, like .gitlab-ci.yml, or even within your pipeline scripts as faux environment variables.

Here's how to integrate honeytokens into your GitLab CI/CD pipeline:

Create a honeytoken through the GitGuardian Honeytoken module. Ensure to note down the token that is generated.

Navigate to your .gitlab-ci.yml file or any other configuration files within your project repository.
Place the honeytoken in the chosen location. It can masquerade as a decoy API key, a dummy password, or another tempting bait for an intruder.
Save the changes and commit the updated file to your repository.

In GitLab, you can store your honeytoken in CI/CD variables. Navigate to your project settings, find the "CI/CD" section, and under "Variables," add your honeytoken. Should anyone infiltrate these variables, the honeytoken will let you know.

Empowering Azure Repos Security with Honeytokens

Azure DevOps Pipelines, a cloud service from Microsoft, is used for automatically building, testing, and deploying applications to any platform. Despite the powerful security features, it provides, including access controls and protection against DDoS attacks, integrating honeytokens into your Azure DevOps Pipelines can fortify your line of defense.

Honeytokens can be added to Azure DevOps Pipelines within several files, such as pipeline configuration files (azure-pipelines.yml), or environment variables.

Below are steps to add a honeytoken to your Azure DevOps Pipelines:

Kickstart the process by generating a honeytoken on the GitGuardian dashboard. Ensure you record the generated token safely.
Navigate to your pipeline configuration file (e.g., azure-pipelines.yml) or any other relevant file within your Azure DevOps Pipelines.
Add the honeytoken to a location within the selected file, where it can convincingly act as a fake credential or sensitive data. This could be within a section of the file specifying environment variables.

Azure DevOps keeps secrets safe through "Secret Variables." You can add your honeytoken here to monitor for any unauthorized access. Open your pipeline, go to "Variables," and choose to create a new "Secret Variable."

Strengthening AWS CodePipeline with Honeytokens

AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines. As secure as AWS services are, the addition of honeytokens can boost your security strategy.

Adding honeytokens to your buildspec.yml or other AWS CodePipeline configuration files can provide early intrusion detection.

Here's how to deploy a honeytoken in your AWS CodePipeline:

Create a honeytoken in the GitGuardian Honeytoken module. Remember to record the generated token.
Go to your buildspec.yml or another configuration file within your project repository.
Insert the honeytoken in the chosen location, appearing as an attractive piece of data for potential attackers.
Save the changes and commit the updated file to your repository.

AWS CodePipeline offers two options for storing secrets: AWS Secrets Manager or the Parameter Store. You can add your honeytoken to either of these services. This way, if either service is compromised, you'll receive a prompt notification.

Optimizing Honeytoken Usage: Best Practices

While honeytokens are a potent weapon in your cybersecurity arsenal, their effectiveness significantly depends on how you deploy and manage them. Here are a few best practices to help you maximize the benefits of using honeytokens in your CI/CD pipelines:

Placement is Key: Place honeytokens in locations where they mimic sensitive data. This increases the chances that an intruder might use them.
Variety in Deployment: Use honeytokens across various files and stages of your CI/CD pipeline. Diversifying the honeytoken deployment can cast a wider net for detecting intrusions.
Uniqueness for Each Pipeline: Use unique honeytokens for each pipeline and repository. This can help pinpoint the exact location of an intrusion when a honeytoken is triggered.
Routine Checks: Regularly verify that your honeytokens are in place and functioning as expected. This ensures that they are always ready to detect any suspicious activities.
Immediate Action: As soon as a honeytoken is triggered, take immediate action. Investigate the incident, identify the scope of the potential breach, and strengthen your defenses accordingly.
Replacements after Trigger: Once a honeytoken has been triggered, replace it with a new one. A triggered honeytoken can lose its effectiveness if not replaced promptly.

Remember, the goal of a honeytoken isn't just to catch a potential intruder but to learn from the incident and fortify your defenses. Hence, a well-planned and executed deployment of honeytokens can not only detect intrusions but also provide valuable insights to enhance your overall security strategy.

Conclusion

In the vast digital ocean where malicious entities lurk, the right defenses are critical for a secure voyage. And the innovative Honeytokens by GitGuardian provides just that - an extra layer of security that not only detects intrusions but also aids in understanding potential breach points.

By placing these deceptive honeytokens in your CI/CD pipelines, you create a security net that actively monitors and notifies you of any unauthorized activity. From Travis CI to AWS CodePipeline, these tokens seamlessly integrate into various environments, enhancing the security posture of your development processes.

So, why wait? Begin your journey towards a more secure development process today. Sign up for GitGuardian and start deploying Honeytokens in your CI/CD pipelines. Secure your digital assets, gain peace of mind, and focus on what truly matters - creating excellent software.

Remember, in cybersecurity, it's always better to be proactive than reactive, and with GitGuardian Honeytokens, you're already one step ahead.

How to Handle Secrets in Terraform

Keshav Malik — Wed, 31 May 2023 14:46:16 +0000

In today's fast-paced world of cloud computing and infrastructure as code (IaC), efficiently managing your resources and keeping them secure has become crucial. Terraform, an open-source infrastructure as a code tool, is widely used by DevOps teams to provision and manage cloud infrastructure. While Terraform provides a powerful, flexible, and easy-to-use platform, it also poses certain challenges when it comes to managing sensitive information, such as API keys, passwords, and other secrets.

Proper management of secrets is essential to maintaining the security and integrity of your infrastructure. Inadequate handling of secrets can lead to disastrous consequences, including unauthorized access, data breaches, and regulatory non-compliance.

This blog post aims to provide a comprehensive guide on secrets management in Terraform, offering insights into best practices, tools, and techniques to help you protect your infrastructure and keep your secrets safe.

Prerequisites

Before diving into the depths of secrets management in Terraform, it's important to ensure that you have the necessary foundation to build upon. The following prerequisites will help you get the most out of this guide:

Basic understanding of Terraform: Having a fundamental knowledge of Terraform and its core concepts, such as providers, resources, and modules. You can start here: 9 Extraordinary Terraform Best Practices That Will Rock Your Infrastructure
Installed Terraform: To follow along with this guide and try out the examples provided, make sure you have Terraform installed on your local machine.
Access to a cloud provider account: Having access to a cloud provider's account, such as AWS, Azure, or Google Cloud Platform, will enable you to put the concepts and strategies discussed in this guide into practice.
Familiarity with basic security concepts: Understanding the fundamentals of information security, such as encryption, access control, and authentication. Here is an introduction to Identity and Access Management.

Understanding Secrets Management

Secrets refer to sensitive information such as API keys, passwords, access tokens, and encryption keys, which require restricted access to maintain the security and integrity of your infrastructure. Properly managing secrets is crucial to prevent unauthorized access and minimize potential security risks.

Types of Secrets in Terraform

In the context of Terraform, secrets can be classified into various types, such as:

API keys used for authentication with cloud providers and other services
Database credentials like usernames and passwords
SSH keys for secure server access

Risks of not managing secrets properly

Inadequate secrets management can lead to several potential risks, including:

Compliance violations: Many industries are subject to strict regulations regarding the handling of sensitive information, and failing to manage secrets properly can lead to hefty fines and reputational damage.
Unauthorized access: Improperly managed secrets can allow unauthorized individuals to access sensitive resources, resulting in data leakage and potential security violations.
Security breaches: Exposed secrets can be exploited by attackers to steal or tamper with data or even to completely compromise entire systems.

Understanding these risks underscores the importance of implementing effective secrets management practices in your Terraform deployments.

Also Read: Security in Infrastructure as Code with Terraform

Best Practices for Managing Secrets in Terraform

Managing secrets securely in Terraform is crucial to protect sensitive information and prevent unauthorized access. In this section, we will discuss several best practices for handling secrets in Terraform, including using environment variables, storing secrets in secure external storage, and encrypting sensitive data.

Use environment variables

Storing secrets as environment variables keeps them out of your Terraform code and version control systems. Environment variables can be easily accessed in Terraform using the var keyword. For example, to set up an AWS provider using an API key stored as an environment variable:

To set the environment variable, use the export command in your terminal:

export TF_VAR_aws_access_key=<your_access_key>

Store Secrets in a Secure External Storage

Instead of storing secrets directly in your Terraform code or environment variables, consider using a secure external storage service designed for secrets management, such as HashiCorp Vault or AWS Secrets Manager. These services provide advanced features like access control, auditing, and automatic rotation of secrets. In the following sections, we will discuss integrating Terraform with Vault and AWS Secrets Manager in more detail.

Encrypt sensitive data

When storing secrets in remote backends or transferring them over the network, ensure they are encrypted. Many cloud providers offer Key Management Services (KMS) to encrypt and decrypt data. For instance, with AWS KMS, you can encrypt sensitive data using the aws_kms_secrets data source:

Secret management tools can significantly enhance the security of your Terraform deployments by providing centralized, secure storage for sensitive information. Here we will discuss various secret management tools and provide step-by-step instructions for integrating them with Terraform.

Overview of secrets management tools

There are several secrets management tools available, each offering different features and levels of integration with Terraform. Some popular options include:

HashiCorp Vault: A comprehensive secrets management solution designed to handle a wide range of secret types, with tight integration with Terraform.
AWS Secrets Manager: A managed service provided by AWS that offers seamless integration with other AWS services and Terraform.

Integrating Terraform with Vault

HashiCorp Vault is a widely-used secret management solution that allows you to store, manage, and access secrets securely. To integrate Terraform with Vault, follow these steps:

1. Install and configure Vault: Follow the official Vault documentation to install and set up Vault on your local machine or a dedicated server.

2. Enable the kv secrets engine in Vault: Run the following command which allows you to store key-value pairs:

vault secrets enable -path=my-secrets kv

3. Write secrets to Vault: Store your secrets in Vault using the following:

vault kv put my-secrets/aws aws_access_key_id=<your_access_key> aws_secret_access_key=<your_secret_key>

4. Configure the Terraform Vault provider: In your Terraform configuration, set up the Vault provider and authenticate using a token or other supported authentication methods:

5. Access secrets from Vault in Terraform: Use the vault_generic_secretdata source to read secrets from Vault:

Integrating Terraform with AWS Secrets Manager

AWS Secrets Manager is a managed service that helps you protect access to your applications, services, and IT resources. To integrate Terraform with AWS Secrets Manager, follow these steps:

1. Store secrets in AWS Secrets Manager: Log in to the AWS Management Console, navigate to Secrets Manager, and create a new secret containing your sensitive data (e.g., API keys and database credentials).

2. Configure the Terraform AWS provider: In your Terraform configuration, set up the AWS provider with the appropriate credentials:

3. Access secrets from AWS Secrets Manager in Terraform: Use the aws_secretsmanager_secret_version data source to read secrets:

You can now use the local.example_secret variable to access the stored secret as a JSON object. For example, if your secret contains a database username and password, you can reference them like this:

Implementing Role-Based Access Control (RBAC) in Terraform

Role-Based Access Control (RBAC) is an effective approach to managing access to sensitive information in your Terraform deployments, allowing you to grant permissions based on roles and responsibilities. In this section, we will discuss the fundamentals of RBAC and provide guidance on configuring RBAC in Terraform.

Understanding RBAC

RBAC is a security model that assigns permissions to users based on their roles within an organization instead of assigning permissions directly to individual users. This approach simplifies access management and improves security by ensuring that users only have the permissions they need to perform their job functions. The main components of RBAC are:

Roles: Collections of permissions that define what actions users can perform on specific resources.
Users: Individuals or entities that interact with the system and are assigned roles based on their responsibilities.
Permissions: Access rights that determine what actions can be performed on specific resources.

As you guessed, implementing RBAC in Terraform involves defining roles, assigning them to users, and managing permissions for accessing sensitive data, such as secrets.

Configuring RBAC in Terraform

To configure RBAC in Terraform, you can leverage features provided by your cloud provider, secret management tool, or Terraform Enterprise. The following example demonstrates how to configure RBAC using HashiCorp Vault:

1. Create policies in Vault: Define Vault policies that outline the permissions for each role. For example, you can create a policy that allows read access to a specific path within Vault:

To write the policy to Vault, use the vault policy write command:

vault policy write read-only read-only-policy.hcl

2. Assign policies to users: After creating the necessary policies, assign them to users by attaching the policy to an authentication method or a specific user:

vault write auth/userpass/users/john password="johnspassword" policies="read-only"

3. Access secrets in Terraform based on RBAC: Users can now access secrets in Terraform according to their assigned policies. For example, a user with the "read-only" policy can read secrets from the "my-secrets" path:

By implementing RBAC in Terraform, you can ensure that users only have access to the secrets they need, minimizing the risk of unauthorized access and improving overall security.

Continuous Monitoring and Auditing

Continuous monitoring and auditing play a crucial role in secrets management, as they help identify potential security vulnerabilities, unauthorized access, and policy violations. In this section, we will introduce some tools that can assist in tracking Terraform secrets.

Tools for Monitoring and Auditing Terraform Secrets

There are several tools available that can help you monitor and audit Terraform secrets, including:

Cloud provider tools: Many cloud providers offer monitoring and auditing services, such as AWS CloudTrail, Azure Monitor, and Google Cloud's Stackdriver. These tools can track changes in your cloud resources and provide logs for analysis.
HashiCorp Vault: Vault includes built-in auditing capabilities, allowing you to track and log all access and modifications to secrets stored in Vault.
GitGuardian: GitGuardian is a powerful secrets detection tool that can automatically scan your code repositories for exposed secrets, including those in Terraform configurations. By integrating GitGuardian into your development workflow, you can identify and remediate exposed secrets early in the development process.

By implementing continuous monitoring and auditing in your Terraform deployments, you can enhance the security of your secrets management strategy and reduce the likelihood of unauthorized access or data breaches.

Conclusion

Managing secrets in Terraform is a critical aspect of ensuring the security and integrity of your infrastructure deployments. By following best practices such as using environment variables, storing secrets in secure external storage, and implementing Role-Based Access Control (RBAC), you can significantly reduce the risks associated with handling sensitive information.

Additionally, leveraging secret management tools like HashiCorp Vault and AWS Secrets Manager can provide advanced features that further enhance security.

Continuous monitoring and auditing, using tools like GitGuardian and its CLI tool ggshield, are essential in maintaining a secure environment and identifying potential vulnerabilities. The ggshield CLI allows developers, Site Reliability Engineers, and DevOps Engineers to scan Infrastructure as Code (IaC) configurations from the command line, helping to prevent misconfigurations from being deployed.

By adopting these practices and tools, especially integrating ggshield as a pre-commit check or CI pipeline integration, you can establish a robust secrets management strategy in Terraform, protecting your sensitive data and ensuring the security of your infrastructure deployments.

Learn how to use ggshield as a git hook with these resources:

Creating a pre-commit git hook to detect secrets

https://www.youtube.com/embed/ObksvAZyWdo?feature=oembed

How to Handle Secrets in Kubernetes

Keshav Malik — Thu, 18 May 2023 11:21:20 +0000

Kubernetes has become the de facto standard for container orchestration, enabling organizations to build, deploy, and scale modern applications with efficiency and agility. As more organizations adopt Kubernetes, the need for proper security and management of sensitive data within these environments becomes paramount. One crucial aspect of ensuring a secure Kubernetes infrastructure is the effective management of secrets, such as API keys, passwords, and tokens.

In this blog post, we will delve into the world of secret management in Kubernetes, exploring various approaches and best practices to keep your sensitive data safe and accessible only to authorized components.

We will discuss how to create, store, and use secrets in Kubernetes, along with the importance of encryption, role-based access control, and auditing. By understanding and implementing these practices, you can significantly enhance the security and reliability of your Kubernetes cluster, safeguarding your applications and data from unauthorized access and potential breaches.

What are Secrets in Kubernetes (K8s)?

In the context of K8s, secrets are objects that store sensitive information, such as passwords, API keys, OAuth tokens, and Secure Shell (SSH) keys. These secrets are crucial for the proper functioning of applications and services within a Kubernetes cluster, as they often grant access to essential resources and authentication mechanisms.

Understanding and managing secrets is vital for maintaining the security and integrity of your Kubernetes environment. Inadequate management of secrets can lead to unauthorized access, data breaches, and compromised infrastructure. Therefore, it is essential to ensure that secrets are stored, transmitted, and utilized securely within your cluster.

Different Ways to Manage Secrets in K8s

A variety of methods are available to manage secrets in Kubernetes. In this section, we will focus on Kubernetes Secrets, a native solution for storing and managing sensitive data within your cluster.

Kubernetes Secrets

Kubernetes Secrets are built-in objects that enable you to store and manage sensitive information securely. They provide a convenient way to distribute confidential data to containers and maintain the separation of concerns between the application and its credentials.

Creating Kubernetes Secrets

There are several ways to create Kubernetes Secrets:

Using kubectl

You can create a secret using the kubectl command-line tool by providing the required data as key-value pairs. For example:

kubectl create secret generic my-secret --from-literal=username=myuser --from-literal=password=mypassword

This command creates a secret named my-secret containing two key-value pairs for the username and password.

Using a configuration file

Another way to create a Kubernetes Secret is through a YAML configuration file. First, encode your sensitive data in base64 format, then create a YAML file with the following structure:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  username: bXl1c2Vy
  password: bXlwYXNzd29yZA==

Apply the YAML file using kubectl apply -f my-secret.yaml

Using the Kustomize tool

Kustomize is a standalone tool for customizing Kubernetes objects through a kustomization.yaml file. You can create a secret generator with it like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
- secret.yaml

secretGenerator:
- name: my-secret
  literals:
  - username=admin
  - password=secret

Then to generate the secret run: kubectl apply -k .

Ways to employ secrets within Pods

Secrets can be used within Pods in the following ways:

Mounting secrets as files in a volume for containers: You can mount a secret as a volume inside a container. Each key-value pair in secret will be presented as a separate file within the mounted volume.
Setting secrets as container environment variables: Secrets can be exposed as environment variables in containers. This approach allows your application to read the secret value without changing the code, making it more secure and manageable.
Leveraging kubelet for image pulling with secrets: Kubelet can use secrets to pull container images from private registries. By specifying the secret in the Pod's imagePullSecrets field, Kubernetes will authenticate to the registry using the provided credentials.

Also read: Kubernetes Hardening Tutorial Part 1: Pods

Accessing secrets in containers

Once a secret is mounted as a file or exposed as an environment variable, the application running in the container can access the secret data. If an attacker gains access to the container, they may be able to view the secret data and potentially use it to compromise other parts of the system. Therefore, it is important to ensure that only authorized containers and users can access these secrets.

This can be achieved in several ways, such as using RBAC (Role-Based Access Control) to restrict access to the secrets or encrypting the secrets at rest and in transit.

Also read: Kubernetes Hardening Tutorial Part 3: Authn, Authz, Logging & Auditing

Updating and rotating secrets

Regularly updating and rotating secrets is a crucial security practice. While you can update a secret using kubectl edit, this approach is not recommended because it can be error-prone and can potentially lead to unintended consequences.

When you edit a secret with kubectl edit, you are modifying the existing secret in place. This means that any existing references to the secret (such as environment variables or volume mounts) will continue to use the old secret data until the application is restarted or the pod is deleted and recreated.

If you need to rotate a secret, you would have to update any references to the old secret to use the new secret instead!

That's why past a certain size, it becomes more interesting to either implement app-specific mechanisms that reload configuration at runtime or deploy a sidecar container that monitors for changes and restarts the main container when necessary.

Limitations of Kubernetes Secrets

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations:

Limited encryption: by default, secrets are stored unencrypted in etcd. K8s does support encryption, but the encrypting key needs separate management.

Limited rotation: K8s secrets are designed to be immutable, which means that they cannot be modified or versioned once they are created. This, as we have seen, makes it difficult to rotate secrets. They can't be audited either.

Limited access control: While Kubernetes provides RBAC (Role-Based Access Control) to control access to secrets, it is still possible for unauthorized users to gain access to secrets if they are able to compromise the cluster or the underlying infrastructure.

They may not be suitable for large-scale or highly regulated environments, where more advanced secret management solutions might be necessary.

Despite these limitations, Kubernetes Secrets remain a handy tool for managing secrets when you don't need to scale immediately your cluster.

Kubernetes External Secrets

Kubernetes External Secrets offer an alternative approach to managing secrets in Kubernetes by integrating with external secret management solutions. This allows you to maintain sensitive data outside of your Kubernetes cluster while still providing seamless access to applications running within the cluster.

How does it work?

Kubernetes External Secrets are custom resources that act as a bridge between your Kubernetes cluster and external secret management systems.

Instead of storing secrets directly in Kubernetes, External Secrets fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets. This ensures that your applications can access sensitive data without any code changes while benefiting from the security features provided by external secret management solutions.

Integrating with external secret management solutions

Kubernetes External Secrets can integrate with a variety of external secret management solutions, such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

To integrate External Secrets with your chosen secret management system, you need to deploy the corresponding External Secrets controller in your cluster and configure it to communicate with the external system.

For example, to integrate with HashiCorp Vault, you would deploy the Kubernetes External Secrets controller for Vault and configure it with the necessary Vault authentication and connection details.

Creating and using External Secrets in Kubernetes

To create an External Secret, you need to define a custom resource in a YAML file, specifying the reference to the secret stored in the external system:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the YAML file using kubectl apply -f my-external-secret.yaml

The External Secrets controller will fetch the secret data from the external system and create a native Kubernetes Secret with the same name. This generated secret can be used by your applications in the same way as regular Kubernetes Secrets.

Advantages of Kubernetes External Secrets

Using Kubernetes External Secrets offers several benefits:

Enhanced security by leveraging the features of external secret management solutions, such as encryption, access control, and auditing.
Reduced risk of exposing sensitive data within the Kubernetes cluster.
Simplified secret management for organizations already using external secret management systems.
Centralized secret management across multiple Kubernetes clusters and other platforms.

By integrating Kubernetes External Secrets with your chosen secret management solution, you can achieve a higher level of security and control over your sensitive data while maintaining compatibility with your existing Kubernetes applications.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. In this section, we will discuss some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Encrypting secrets at rest and in transit

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus, Grafana, and complimentary solutions such as GitGuardian that provide detection against hard-coded secrets.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Conclusion

In this blog post, we have discussed the importance of secrets management in Kubernetes and explored various methods and best practices for securely managing sensitive data in your applications. Kubernetes Secrets, Kubernetes External Secrets, and GitOps workflows provide powerful tools and methodologies for managing secrets effectively, ensuring the security and integrity of your sensitive information.

We encourage you to follow the best practices outlined in this post, such as implementing RBAC, encrypting secrets at rest and in transit, rotating secrets regularly, and auditing and monitoring your secrets. Continuously exploring additional resources and tools can further enhance your secret management processes.

One such tool is GitGuardian, which helps you protect your sensitive data by monitoring your Git repositories for leaked secrets and credentials. GitGuardian can be integrated with your GitOps workflows to ensure that secrets are not accidentally exposed in your repositories, providing an additional layer of security for your Kubernetes environment.

In conclusion, managing secrets securely in Kubernetes is critical for maintaining the confidentiality and integrity of your sensitive data. By following best practices and utilizing the right tools and methodologies, you can create a robust and secure Kubernetes environment, safeguarding your applications and data from potential threats.

Introducing DependAware: Automating Dependency Updates for NPM Repositories 🚀

Keshav Malik — Sat, 13 May 2023 14:14:08 +0000

What I built

Introducing DependAware, an automated dependency update tool for NPM repositories that keeps your projects up-to-date and secure by creating pull requests for updating package dependencies.

Category Submission:

Wacky Wildcards

App Link

DependAware on GitHub

Screenshots

Image: Branches created by DependAware 👇

Image: Pull Requests Tab running test cases 👇

Image: Pull Requests Description updated after running tests 👇

Image: Pull Requests created by DependAware 👇

Image: GitHub Actions Tab 👇

Description

DependAware is designed to help developers maintain the health of their NPM projects by automatically scanning, detecting, and updating outdated or vulnerable package dependencies. With DependAware, you can easily integrate it into your GitHub Actions and keep your repositories up-to-date with the latest dependency versions.

Link to Source Code

DependAware Repository

Permissive License

MIT License

Background (What made you decide to build this particular app? What inspired you?)

In today's fast-paced software development world, managing dependencies can be a daunting task. Outdated or vulnerable dependencies can lead to security risks, performance issues, and general instability. I was inspired by the need for a seamless, automated solution that can help developers stay on top of their dependencies while maintaining focus on building amazing products.

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

I built DependAware using GitHub Actions to automatically scan NPM repositories for outdated dependencies, create branches for each update, and then generate pull requests with detailed titles, descriptions, and the "dependencies" label.

I also learned how to integrate DependAware with various GitHub workflows, such as running tests and updating PR descriptions with the test results.

Throughout the development process, I picked up new skills in GitHub Actions and deepened our understanding of dependency management in NPM projects. I also explored the importance of communication between different workflows and ensuring smooth integration of DependAware into existing CI/CD pipelines.

Additional Resources/Info

For more information on how to set up and use DependAware, check out the DependAware GitHub Repository. I encourage you to contribute, provide feedback, and help us improve this invaluable tool for developers. Together, we can make software development more secure and efficient! 🚀

How to Handle Secrets in Jenkins

Keshav Malik — Tue, 18 Apr 2023 18:30:33 +0000

Jenkins is a popular open-source automation server that is widely used for building, testing, and deploying software. It allows developers to automate many aspects of their software development process, including continuous integration and continuous deployment.

As with any continuous integration software, managing secrets in Jenkins is crucial to ensure the security and integrity of the applications being built and deployed. In this blog post, we will be discussing the best practices for managing secrets in Jenkins, including configuring and managing secrets and how to handle potential security breaches.

The blog post also covers storing secrets with Hashicorp Vault and integrating it with Jenkins.

Prerequisites

Before diving into the specifics of this tutorial, managing secrets in Jenkins, it’s important to have the following:

Familiarity with Jenkins and its basic concepts, including how to create jobs and build pipelines.
Access to an already set up Jenkins instance and a Hashicorp Vault for testing and implementation, which will allow you to follow along with the tutorial.
Installed plugins in your Jenkins instance: Credentials | Jenkins plugin, Credentials Binding | Jenkins plugin, and HashiCorp Vault | Jenkins plugin.

Additionally, you can have an application codebase ready to test the secrets management flow in your workflow. We will take as an example a NodeJS application that has Mocha tests and a Docker build setup.

Next, we have a continuous integration (CI) job that is triggered when commits are pushed to the stage branch. The code is then tested and merged into the main branch. After that, a continuous deployment (CD) job is triggered and builds a Docker image before publishing it to DockerHub.

Please note that this blog post is not contingent on this CI/CD workflow: although the presented flow can look pretty basic, you can follow along without any issues, no matter how complex or simple your workflow is!

Understanding Jenkins Secrets

Jenkins secrets, also known as credentials, are used for various authentication and authorization purposes and can include things like passwords, API keys, and private SSH keys.

When it comes to managing these secrets, it’s important to understand there are two scopes in Jenkins: system-wide and job-specific. System-wide secrets are available to all jobs and projects within a Jenkins instance, such as credentials for a shared database or Git repository. Job-specific secrets are only available to a specific job or project and are typically used for specific tasks, like deploying to a particular environment.

It is important to follow best practices when managing secrets in Jenkins. This includes using different secrets for different purposes, restricting their access, and regularly rotating and updating them to minimize the impact of a potential breach.

Configuring Jenkins Secrets

Now that we understand the basics of Jenkins secrets, let’s take a look at how to set up and configure them. The process of configuring secrets in Jenkins will vary depending on the type of secret and the specific use case.

One way to configure Jenkins secrets is through the Jenkins web interface. To do this, navigate to the “Credentials” page in the Jenkins settings.

This is Jenkins’ official credential management tool. You can add, modify, and delete secrets as needed. You can also specify which jobs or projects a secret is available for, and assign permissions to specific users or groups. There are multiple ways to store secrets using the Credentials Plugin.

One good practice when configuring a secret is to give it a descriptive and unique name that clearly indicates what the secret is used for. Avoid using generic or ambiguous names that could cause confusion or make it difficult to identify the correct secret.

Also, it is important to scope credentials appropriately based on the intended usage. For example, if a credential is only needed for a specific job or pipeline, it should be scoped to that job or pipeline rather than being made available globally. This helps to minimize the potential for unauthorized access or misuse of the credential.

Here we have stored DockerHub username and password with a global scope:

Here is an example of storing SSH keys (GitHub credentials in this case):

We can also store a text-based secret like an OAuth client secret for example:

If you have an .env holding environment variables needed in multiple steps in the build pipeline:

This is where the “Credentials Binding” plugin comes into play. This plugin allows you to bind secrets to environment variables, making them available to your build scripts. It gives you an easy way to package up all a job’s secret files and passwords and access them using a single environment variable during the build.

To use this plugin, you’ll need to specify the ID of the secret you want to use, as well as the environment variable it should be bound to. The following example is for a freestyle project in Jenkins.

And below is an example for injecting the credentials PORT into a pipeline project in Jenkins. (full config here)

pipeline {
  …
  stages{
    …
    stage('Building Docker Image') {
      steps {
        script {
          withCredentials([string(credentialsId: 'PORT', variable: 'PORT')]) {
            dockerImage = docker.build("$registry:$BUILD_NUMBER", "--build-arg PORT=$PORT .")
          }
        }
      }
    }
  …
  }
  …
}

How to Use Vault Secrets in Jenkins

In addition to the built-in Jenkins Secrets manager, it is also possible to integrate with third-party secret management tools such as Hashicorp Vault, AWS KMS, etc. This can be particularly useful if you are already using Vault to manage secrets for other applications or services.

In this section, we will walk through the process of configuring Jenkins to retrieve secrets from Vault and use them in your pipeline.

Run your Vault with the following command and leave this terminal session open as dev servers run in-memory, i.e., shutting them off would wipe the slate clean.

# we are running the vault inside an ec2 machine thus the private ip
$ vault server -dev -dev-listen-address=”<private ip of ec2>:8082”

Now, run the following commands in a separate terminal session:

$ export VAULT_ADDR='http://<private ip of ec2>:8082'
# check the status just to be sure
$ vault status

It should print information similar to this:

Next, our Jenkins instance needs to communicate with the Vault. The following commands will take care of the auth-related formalities. Jenkins will later use this role to talk to the Vault.

# firstly enable the vault approle authentication method
$ vault auth enable approle

# ensure a v2 kv secrets engine is in use
$ vault secrets enable kv-v2

# create a local file “jenkins-policy.hcl” that houses a policy allowing “read” access to the path “secret/data/jenkins” under which we’ll store our secret

$ tee jenkins-policy.hcl <<"EOF"
  path "secret/data/jenkins/*" {
    capabilities = [ "read" ]
  }
  EOF

# create a vault policy "jenkins" from the local policy file "jenkins-policy.hcl"

$ vault policy write jenkins jenkins-policy.hcl

# create a vault approle “jenkins” and associate the vault policy “jenkins” with it

$ vault write auth/approle/role/jenkins token_policies=”jenkins” token_ttl=1h token_max_ttl=4h

# Next, we need the strings roleId and secretId with 
$ vault read auth/approle/role/jenkins/role-id
$ vault write -f auth/approle/role/jenkins/secret-id

These commands will ensure that apps and/or services like Jenkins can connect with the Vault. The access policy we’ve defined allows the path secret/data/jenkins/* to be accessed by the policyholder. Then, this policy is registered with the Vault and a role is attached to this policy.

Copy the role_id and secret_id for later use. These are what Jenkins will use to identify itself while communicating with the Vault, and to access secrets in the path secret/data/jenkins/*. We’ll store these as a credential in Jenkins.

Finally, on Vault’s end, we need to store the secret. There are multiple ways to achieve this, but we’ll store it as a secret text here. The following command is used to store a secret identified as my-secret in the kv secret engine in the path we earlier declared in the approle policy.

$ vault kv put secret/data/jenkins/my-secret PORT=4000

Then create credentials in Jenkins via the Jenkins web interface. To do this, navigate to the “Credentials” page in the Jenkins settings. Add a new credential under Global credentials. Select the “Kind” as “Vault App Role Credential”. And paste in the IDs role and secret we copied from the terminal. The form should look like the following:

Once the credentials are added, copy the credential’s ID for later reference in the Jenkins pipeline.

Lastly, the following example is for a pipeline project in Jenkins. (full config here)

def secrets = [
  [
    path: 'secret/data/jenkins/my-secret',
    engineVersion: 2,
    secretValues: [
      [envVar: 'PORT', vaultKey: 'PORT']
    ]
  ],
]

def configuration = [
  vaultUrl: 'http://172.31.36.198:8082',
  vaultCredentialId: 'ccc2cf0c-1b1a-40a1-a7bd-81ef1dc764f1',
  engineVersion: 2
]

pipeline {
  …
  stages{
    …
    stage('Building Docker Image') {
      steps {
        script {
          withVault([configuration: configuration, vaultSecrets: secrets]) {
            dockerImage = docker.build("$registry:$BUILD_NUMBER", "--build-arg PORT=${env.PORT} .")
          }
        }
      }
    }
  …
  }
  …
}

In this section, we have demonstrated how to integrate Jenkins with Hashicorp Vault to securely manage your secrets. By leveraging the power of Vault’s secret management capabilities and Jenkins’ flexible pipeline configuration, you can ensure that your sensitive information remains protected throughout your CI/CD process.

Managing Secrets in Jenkins

Managing secrets in Jenkins is crucial for maintaining the security of your applications. To ensure this, it is important to follow best practices such as rotating secrets regularly, using strong passwords, and avoiding storing plaintext secrets in Jenkins or in version control.

In addition, regular auditing and monitoring of secrets are also important. This involves keeping track of access to secrets, and their usage, and updating them as necessary. Alerts should also be set up to notify of potential breaches, such as unauthorized access to secrets or failed login attempts.

In case of a security breach or secrets leakage, having a plan in place for revoking access to the compromised secrets, rotating all secrets, and conducting a thorough investigation to prevent similar incidents in the future is crucial.

Conclusion

As we have seen, Jenkins secrets are used to store sensitive information such as passwords, tokens, and keys that are needed by your applications to function properly. However, if not managed properly, these secrets can become a security risk.

Throughout this blog post, we have discussed the importance of managing secrets in Jenkins, the types of secrets that can be managed, and best practices for securing and managing these secrets. We also provided step-by-step instructions for setting up and configuring secrets in Jenkins natively or with an external secrets management solution.

It is important to remember that hard-coding secrets or storing them in plaintext can lead to security breaches. Therefore, it is crucial to follow best practices for secret management and to have a plan in place for handling potential breaches.

We hope this blog post has provided you with a better understanding of how to manage secrets in Jenkins and keep your applications secure.

How to Handle AWS Secrets

Keshav Malik — Fri, 07 Apr 2023 15:07:48 +0000

Secure management of AWS secrets is essential for protecting sensitive data and preventing unauthorized access to critical systems and applications. In today’s rapidly escalating threat landscape, organizations must ensure their secrets are appropriately managed and safeguarded.

The AWS SDK also referred to as the AWS Software Development Kit, is a set of software development tools and libraries created to make it easier for developers to utilize AWS services in their applications. It provides an accessible interface for accessing resources like EC2, S3, and DynamoDB on AWS with ease.

However, when using AWS SDK to interact with AWS services, it’s essential that secrets used for authentication and authorization are managed appropriately. In this blog post, we’ll cover some best practices for managing AWS secrets when using the AWS SDK in Python.

Prerequisites

Before using the AWS SDK for Python to securely manage your AWS secrets, ensure that:

Basic understanding & knowledge of Python and the ability to install packages using pip
An AWS account with appropriate permissions to access AWS services
An IAM user or role with necessary access rights
Boto3, the AWS SDK for Python, should also be installed on your system using pip.

The Problem with Long-Lived Access Keys and Secret Keys in Code

When using AWS SDK with Python, hard-coding long-lived access keys and secret keys is not recommended. These credentials are used for authentication to AWS resources, and these keys pose a security risk since they aren’t automatically rotated.

Here are some potential risks of hard-coding long-lived access keys and secret keys into your code:

Code sharing increases the risk of accidental exposure of sensitive information to those with access to it, whether through public sharing or accidental committing to a public repository.
It can be challenging to rotate access keys and secret keys, which could lead to version control issues and the need to update all instances of those keys within a codebase.

In the following section, we’ll see how you can overcome this problem by using temporary keys.

Using Temporary Access Keys Instead

For better security when using AWS SDK with Python, temporary access keys are the better solution. Temporary keys are short-lived credentials that allow secure access to AWS resources.

Here are some advantages of using temporary access keys:

They expire after a specified period (e.g., one month or one week), decreasing the risk of unauthorized access and making it easier to manage resource access.
Temporary access credentials can be generated on demand, making it simpler & easier to provide end users with access to AWS resources without defining an AWS identity for each user.

Note: The AWS Security Token Service (STS) is a utility that generates temporary access keys.

Using AWS CLI to Manage AWS Secrets

AWS CLI is a command-line tool that enables engineers to interact with AWS services by using CLI commands. Also, AWS CLI can be utilized for managing AWS secrets.

One of the advantages of using AWS CLI is that it automatically fetches AWS credentials (access & secret keys) from a credentials file created by AWS CLI, so there’s no need to manually supply access keys and secret keys when creating an AWS client.

Here’s an example of creating an AWS client without specifying access keys and secret keys when using AWS CLI:

import boto3
client = boto3.client('s3')

In this example, the boto3.client() function is called with the s3 argument to create a client for Amazon S3. Since access keys and secret keys are not specified, the AWS SDK will automatically retrieve them from the credentials file created by AWS CLI.

To create the credentials file, run the following command in the terminal:

aws configure

This command will prompt you to enter your access key, secret key, default region, and output format. Once executed, a credentials file will be created on your machine which the AWS SDK can automatically search for and retrieve when creating an AWS client.

Manual Way to Configure AWS Secrets

Another way to create a credentials file is to do it manually. The default location for the file is ~/.aws/credentials. The credentials file should have, at minimum, the access key and secret access key specified.

In the sample file provided below, the access key and secret key for the account are specified in the default profile:

[default]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY

When you use the aws configure command, the configuration options that are not sensitive (such as region & output format) are saved in a file named config. This file is also stored in the .aws folder in your home directory.

[default]
region=us-west-2
output=json

Creating Multiple Named Profiles

Developers can create & configure additional profiles to manage different sets of AWS credentials by using the aws configure command with the --profile option. Alternatively, you can manually add entries to the config and credentials files. These files store configurations and access keys for each profile.

To add new profiles, you can create separate named profiles in the config and credentials files.

Here's an example of the credentials file with two profiles:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY 
[user1]
aws_access_key_id=AKIAI44QH8DHBEXAMPLE
aws_secret_access_key=je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY

In this example, the default profile ([default]) is used when the AWS CLI command is used without specifying a profile. The second profile ([user1]) is used when you run a command with the --profile user1 parameter. The file can be found in ~/.aws/credentials on Linux and Mac systems.

Note: Credentials location for a Windows system is %USER%\.aws\credentials.

Managing AWS CLI Configuration Settings

AWS CLI provides several commands to manage the configuration settings. You can use the aws configure set command to modify or set the configuration settings, and the aws configure get command to retrieve the configuration settings. Here's how you can use them:

Setting Configuration Settings

To set any configuration settings, you can use the aws configure set command. Specify the profile you want to modify using the --profile option. For example, to set the region for the USER profile, run the following command:

$ aws configure set region me-south-1 --profile USER

You can remove a configuration setting by using an empty string as the value or deleting the setting manually from the config and credentials files.

Retrieving Configuration Settings

You can retrieve the configuration settings that you've set using the aws configure get command. To retrieve the region setting for the USER profile, run the following command:

$ aws configure get region --profile USER

Importing CSV Credentials

You can import the CSV credentials generated from the AWS web console using the aws configure import command. The CSV file must contain the following headers:

User Name
Access key ID
Secret access key

To import the credentials from the credentials.csv file, run the following command:

$ aws configure import --csv file://credentials.csv

Listing Profiles

You can list all your profile names using the aws configure list-profiles command.

$ aws configure list-profiles --region <<YOUR_REGION>

Best Practices for Secure Credential Management in AWS

When working with AWS, it's essential to adhere to best practices for credential management in order to protect your resources. Here are six top tips for AWS SDK credential management:

Use the AWS CLI to Configure AWS Keys: Avoid hardcoding AWS access keys and secret keys into your code. Instead, utilize the AWS CLI to configure your keys and store them securely.
Limit access to secrets with IAM policies and roles: Use AWS Identity and Access Management (IAM) policies and roles to limit access to your secrets only to the users and services that require them.
Regularly rotate secrets to minimize impact: Regularly rotate your access keys, passwords, and other secrets to minimize the impact of potential exposure.
Use Parameter Store to store secrets: Parameter Store is a secure and scalable AWS service that allows you to store and manage secrets securely.
Use AWS Secrets Manager for more advanced management: AWS Secrets Manager provides advanced secret management features, such as automatic rotation and integration with Amazon RDS.
Use tools like GitGuardian to detect leaked secrets: Leaked secrets can put your AWS resources at risk. Use tools like GitGuardian to detect and prevent leaks of your secrets in code repositories and other sources.

Conclusion

Properly managing AWS credentials is crucial to maintaining the security of your AWS resources. By using AWS's configuration and credential files, you can keep your AWS access and secret keys secure and separate from your code. Additionally, following best practices such as limiting access to secrets with IAM policies and roles, and regularly rotating secrets can further enhance your AWS credential management.

As always, it's essential to stay vigilant against potential security breaches. That's where GitGuardian comes in. GitGuardian is a tool that can scan your GitHub repositories for exposed secrets, such as AWS keys, and alert you when it finds them. By using GitGuardian in conjunction with proper AWS credential management practices, you can ensure that your AWS resources remain secure.

So, whether you're new to AWS or a seasoned pro, remember the importance of proper AWS credential management and take steps to keep your AWS resources secure. And don't forget to give GitGuardian a try for an extra layer of security.

If you are interested in learning more about AWS IAM security best practices check this post.

We hope this blog post has provided you with a better understanding of how to manage AWS secrets and keep your applications secure.

How to Handle Secrets in Docker

Keshav Malik — Thu, 16 Feb 2023 07:51:38 +0000

Secrets management in Docker is a critical security concern for any business. When using Docker containers, it is essential to keep sensitive data such as passwords, API keys, and other credentials secure.

This blog post will discuss some best practices for managing secrets in Docker, including how to store them securely and minimize their exposure. We will explore multiple solutions: using Docker Secrets with Docker Swarm, Docker Compose, or Mozilla SOPS. Feel free to choose what’s more appropriate to your use case. But most importantly is to remember to never hard-code your Docker secrets in plaintext in your Dockerfile!

Following these guidelines ensures that your organization's sensitive information remains safe even when running containerized services.

4 Ways to Store & Manage Secrets in Docker

Using Docker Secrets & Docker Swarm

Docker Secrets and Docker Swarm are two official and complimentary tools allowing to securely manage secrets when running containerized services.

Docker Secrets provides a secure mechanism for storing and retrieving secrets from the system without exposing them in plaintext. It enables users to keep their credentials safe by encrypting the data with a unique key before passing it to the system.

Docker Swarm is a powerful tool for managing clusters of nodes for distributed applications. It provides an effective means of deploying containerized applications at scale. With this tool, you can easily manage multiple nodes within a cluster and automatically distribute workloads among them. This helps ensure that your application has enough resources available at all times, even during peak usage periods or unexpected traffic spikes.

Together, these two tools provide an effective way to ensure that your organization's sensitive information remains safe despite ever-evolving security needs.

Let’s see how create and manage an example secret.

Creating a Secret

To create a secret, we need to first initialize Docker Swarm. You can do so using the following command:

docker swarm init

Once the service is initialized, we can use the docker secret create command to create the secret:

ssh-keygen -t rsa -b 4096 -N "" -f mykey
docker secret create my_key mykey
rm mykey

In these commands, we first create an SSH key using the ssh-keygen command and write it to mykey. Then, we use the docker secret command to generate the secret. Ensure that you delete the mykey file to avoid any security risks.

You can use the following command to confirm that the secret is created successfully:

docker secret ls

We can now use this secret in our Docker containers. One way is to pass this secret with –secret flag when creating a service.

docker service  create --name mongodb --secret my_mongodb_secret redis:latest

We can also pass this secret to docker-compose.yml file. Let’s take a look at an example file:

version: '3.7'
services:
  myapp:
    image: mydummyapp:latest
    secrets:
      - my_secret
    volumes:
      - type: bind
        source: my_secret_key
        target: /run/secrets/my_secret
        read_only: true
secrets:
  my_secret:
    external: true

In the example compose file, the secrets section defines a secret named my_secret_key (discussed earlier). The myapp service definition specifies that it requires my_secret_key , and mounts it as a file at /run/secrets/my_secret in the container.

Using Docker Compose

Docker Compose is a powerful tool for defining and running multi-container applications with Docker. A stack is defined by a docker-compose file allowing you to define and configure the services that make up your application, including their environment variables, networks, ports, and volumes.

With Docker Compose, it is easy to set up an application in a single configuration file and deploy it quickly and consistently across multiple environments.

Docker Compose provides an effective solution for managing secrets for organizations handling sensitive data such as passwords or API keys. You can read your secrets from an external file (like a TXT file). But be careful not to commit this file with your code!

version: '3.7'
services:
  myapp:
    image: myapp:latest
    secrets:
      - my_secret
secrets:
  my_secret:
    file: ./my_secret.txt

Using a Sidecar Container

A typical strategy for maintaining and storing secrets in a Docker environment is to use sidecar containers. Secrets can be sent to the main application container via the sidecar container, which can also operate a secrets manager or another secure service.

Let’s understand this using a Hashicorp Vault sidecar for a MongoDB container:

First, create a Docker Compose (docker-compose.yml) file with two services: mongo and secrets.
In the secrets service, use an image containing your chosen secret management tool, such as a vault.
Mount a volume from the secrets container to the mongo container so the mongo container can access the secrets stored in the secrets container.
In the mongo service, use environment variables to set the credentials for the MongoDB database, and reference the secrets stored in the mounted volume.

Here is the example compose file:

version: '3.7'

services:
  mongo:
    image: mongo
    volumes:
      - secrets:/run/secrets
    environment:
      MONGO_INITDB_ROOT_USERNAME_FILE: /run/secrets/mongo-root-username
      MONGO_INITDB_ROOT_PASSWORD_FILE: /run/secrets/mongo-root-password
  secrets:
    image: vault
    volumes:
      - ./secrets:/secrets
    command: ["vault", "server", "-dev", "-dev-root-token-id=myroot"]
    ports:
      - "8200:8200"

volumes:
  secrets:

Using Mozilla SOPS

Mozilla SOPS (Secrets Ops) is an open-source platform that provides organizations with a secure and automated way to manage encrypted secrets in files.

It offers a range of features designed to help teams share secrets in code in a safe and practical way. The following assumes you are already familiar with SOPS, if that's not the case, start here.

Here is an example of how to use SOPS with docker-compose.yml :

version: '3.7'

services:
  myapp:
    image: myapp:latest
    environment:
      API_KEY: ${API_KEY}
    secrets:
      - mysecrets

  sops:
    image: mozilla/sops:latest
    command: ["sops", "--config", "/secrets/sops.yaml", "--decrypt", "/secrets/mysecrets.enc.yaml"]
    volumes:
      - ./secrets:/secrets
    environment:
      # Optional: specify the path to your PGP private key if you encrypted the file with PGP
      SOPS_PGP_PRIVATE_KEY: /secrets/myprivatekey.asc

secrets:
  mysecrets:
    external: true

In the above, the myapp service requires a secret called API_KEY. The secrets section uses a secret called mysecrets, which is expected to be stored in an external key/value store, such as Docker Swarm secrets or HashiCorp Vault.

The sops service uses the official SOPS Docker image to decrypt the mysecrets.enc.yaml file, which is stored in the local ./secrets directory. The decrypted secrets are mounted to the myapp service as environment variables.

Note: Make sure to create the secrets directory and add the encrypted mysecrets.enc.yaml file and the sops.yaml configuration file (with SOPS configuration) in that directory.

Scan for secrets in your Docker images

Hard coding secrets in Docker is a significant security risk, making them vulnerable to attackers. We have seen different best practices to avoid hard-coding secrets in plaintext in your Docker images, but security doesn't stop there.

You should also scan your images for secrets.

All Dockerfiles start with a FROM directive that defines the base image. It's important to understand that when you use a base image, especially from a public registry like Docker Hub, you are pulling external code that may contain hardcoded secrets.

More information is exposed than visible in your single Dockerfile. Indeed, it's possible to retrieve a plaintext secret hard-coded in a previous layer starting from your image.

In fact, many public Docker images are concerned: in 2021, we estimated that 7% of the Docker Hub images contained at least one secret.

Fortunately, you can easily detect them with ggshield (GitGuardian CLI). For example:

ggshield secret scan docker ubuntu:22.04

Conclusion

To sum up, managing secrets in Docker is a crucial part of preserving the security of your containerized apps. Docker includes several built-in tools for maintaining secrets, such as Docker Secrets and Docker Compose files.

Additionally, organizations can use third-party solutions like HashiCorp Vault and Mozilla SOPS to manage secrets in Docker. These technologies offer extra capabilities like access control, encryption, and audit logging to strengthen the security of your secret management.

Finally, finding and limiting accidental or unintended exposure of sensitive information is crucial to handling secrets in Docker. Companies are invited to use secret scanning tools such as GitGuardian to scan the Docker images built in their CI/CD pipelines as mitigation to prevent supply-chain attacks.

If you want to know more about Docker security, we also summarized some of the best practices in a cheat sheet.

Time Management Techniques for Developers to Boost Productivity⏳

Keshav Malik — Fri, 10 Feb 2023 07:56:07 +0000

Developers are often tasked with managing their time effectively to ensure the successful completion of projects. However, good time management isn't just about setting deadlines; it involves discipline and organization to manage workloads efficiently.

By implementing specific techniques related to planning, tracking progress, and following through on tasks, developers can boost their productivity and stay organized throughout their workday.

In this blog post, we'll discuss various strategies for effective time management that will help developers stay on top of their workloads while ensuring quality work is produced in a timely manner.

Importance of Time Management 💡

Time management is an essential skill for any successful developer, as it allows them to stay organized and use their time better. Good time management helps developers produce work that meets deadlines without sacrificing quality due to being overwhelmed with too many tasks or having insufficient organizational skills.

By implementing specific techniques related to planning, tracking progress, and following through on tasks, developers can boost their productivity and stay organized throughout their workday. This leads to higher-quality output in less time and increases overall job satisfaction by reducing stress levels associated with tight deadlines.

Effective time management allows developers to stay organized, make better use of their time, and ensure that they produce high-quality work efficiently.

5 Ways to Manage Time as a Developer 👨‍💻

Time management is an essential skill for any successful developer. Without proper planning, tracking progress, and following through on tasks, developers can quickly become overwhelmed and struggle to complete their work in a timely manner.

Setting Goals and Prioritizing Tasks

Planning ahead and setting specific goals with deadlines is essential for successful time management. It is vital to break down big projects into smaller tasks and prioritize them to ensure that the most critical tasks are done first. This will help developers stay organized, focused, and motivated throughout their workday.

Additionally, creating a timeline or checklist of tasks can help keep the developer on track and ensure that the tasks are completed on time without sacrificing quality.

Staying Focused and Avoiding Distractions 🎯

Being mindful of how one spends time is essential for effective time management. For example, limiting distractions such as checking emails too often or spending too much time on social media can help create an environment where productivity is maximized.

Additionally, developing strategies for staying focused, such as taking regular breaks or setting aside specific times during the day for tasks, can help keep motivation high while ensuring that all work assignments are completed on schedule.

Managing Email and Communication 📧

Managing emails and communication is a critical time management technique for developers to boost productivity. Emails contain valuable information that can be used to plan tasks, track progress, and make sure deadlines are met without sacrificing quality.

By filtering out unnecessary emails or setting up automatic replies, developers can save themselves valuable time each day while also staying organized with their workloads.

Additionally, effective communication helps create clear expectations between the developer and their team members or clients regarding when certain tasks should be completed. With proper email management strategies, developers can maximize efficiency throughout their workday while ensuring all assignments are finished on schedule.

Create Visual Timelines 📈

Having a visual timeline of when specific tasks need to be completed can also be beneficial to staying organized. This timeline should encompass everything from long-term projects to small daily tasks so that developers can effectively gauge their progress and stay on track throughout their workday.

Additionally, setting realistic timelines for projects will ensure that the developer does not overextend themselves by giving an unrealistic deadline for completing each task or project.

Taking Care of Your Physical and Mental Health 😷

The importance of taking care of one's physical and mental health when managing time as a developer cannot be overstated. Not only is it important to stay focused and organized while working, but prioritizing self-care is also essential for boosting productivity in the long run.

When it comes to time management, ensuring enough sleep, exercising regularly, eating healthy meals, and taking regular breaks throughout the day are all key strategies for staying productive and energized.

Additionally, taking the time to practice mindfulness or setting aside some time each day for activities that bring joy can help reduce stress levels while allowing developers to stay focused during their workday.

Conclusion

Time management is essential for any successful developer, and one can employ many strategies to help manage time effectively. Setting goals and prioritizing tasks, staying focused and avoiding distractions, managing emails and communication wisely, creating visual timelines of when certain tasks should be completed, and taking care of one's physical and mental health all boost productivity.

By following these simple steps daily, developers can ensure they stay organized while completing their work assignments within the desired timeframe.

How to Handle Secrets in Python

Keshav Malik — Thu, 26 Jan 2023 06:11:15 +0000

We live in a world where applications are used to do everything, be it stock trading or booking a salon, but behind the scenes, the connectivity is done using secrets. Secrets such as database passwords, API keys, tokens, etc., must be managed appropriately to avoid any breach.

The need for managing secrets is critical for any organization. Secrets can be leaked in many ways, including through version control systems (never hardcode any secrets in your code!), private messages, email and other communication channels. If secrets are leaked, it can lead to a loss of trust, credibility, and even business (you can learn why here). In some cases, leaked secrets can also lead to legal action. That's why it's so important to have a plan for managing secrets.

In this post, we will discuss 4 different ways to manage secrets in Python efficiently.

Pre-requisites

Before getting started, below are a few things to keep in mind to avoid any issues later.

Python & pip installed on your machine
Understanding of Python & CLI
Python IDE such as PyCharm or VS Code
Basic understanding of Cloud

💡 We will be using MacOS for this device. Please use the commands as per your OS.

4 Ways to Manage Secrets in Python

In this section, we will discuss 4 different ways to manage your secrets in Python.

1. From a file

Using an .env file

The .env file is a file used to store environment variables in Python. Environment variables are variables set outside of the Python code and are used to configure the Python code. The .env file is typically used to store secret keys and passwords.

We will use the python-dotenv package for accessing the content of the .env file. To get started, first install the package using the following command.

$ pip install python-dotenv

Create a .env file for testing purposes and paste the following secrets:

API_KEY=test-key
API_SECRET=test-secret

Of course, this file should not be committed to your git repo! Otherwise, it would be versioned and readable even after you delete it.

Add this line to your .gitignore file:

.env

Already committed the file? No worries, follow the instructions here.

Exposing secrets in a git repository is bad but mistakes happen. This is a complete guide and cheatsheet to rewrite and remove files from git history

Once done, create a main.py file and paste the below-mentioned code snippet. In this code, we are using the load_dotenv() function to load content from the .env file.

from dotenv import load_dotenv
import os

load_dotenv()

api_key = os.getenv("API_KEY")
api_secret = os.getenv("API_SECRET")

print("API_KEY: ", api_key)
print("API_SECRET: ", api_secret)

We can also use dotenv_values() function, which converts the secrets into a dictionary. Secrets can be accessed by using the following code snippet:

from dotenv import dotenv_values

secrets = dotenv_values(".env")

def main():
   print(secrets["API_KEY"])
   print(secrets["API_SECRET"])

if __name__ == "__main__":
   main()

When working on a large project, you may find that you need multiple .env files. For example, you may have a .env file for your local development environment and a .env.dev file for your cloud development production environment. The following code snippet can be helpful if you have multiple env files.

from dotenv import dotenv_values

secrets = dotenv_values(".env")
local_secrets = dotenv_values(".env.dev")

def main():
   print(secrets["API_KEY"])
   print(local_secrets["API_SECRET"])

if __name__ == "__main__":
   main()

Using a JSON file

To keep the secrets more organized, you can also use a JSON file. Let's create a secrets.json file and paste the following secrets into it.

{
   "db": {
       "host": "localhost",
       "port": 27017,
       "name": "test"
   },
   "server": {
       "host": "localhost",
       "port": 3000
   }
}

💡 The same as above applies, do not commit this file!

Now that we have the JSON file ready, let’s write a function to access secrets from the file:

# main.py
import json

def get_value_from_json(json_file, key, sub_key):
   try:
       with open(json_file) as f:
           data = json.load(f)
           return data[key][sub_key]
   except Exception as e:
       print("Error: ", e)

print(get_value_from_json("secrets.json", "db", "host")) # prints localhost

2. Using Environment Variables

Environment variables are variables that are set by the operating system or a specific user and are used by programs to determine various settings.

We can use these variables to store our secrets and then access them in our program. You can use the following syntax to create an environment variable in macOS or a Linux machine.

$ export variable_name=value
$ export API_KEY_TEST=dummykey

On a Windows machine, you use GUI to add environment variables or use the following command to add a variable.

$ setx [variable_name] “[value]”

You can use the os package to access the os environment variable. Below mentioned is the sample code:

import os

# Get the secret key from the environment
secret_key = os.environ.get('api_key_test')
print(secret_key) // prints dummykey

Secrets at the command line should be handled with special care too! Have a look at the best practices.

Secrets at the Command Line [cheat sheet included]
Developers need to prevent credentials from being exposed while working on the command line. Learn how you might be at risk and what tools and methods to help you work more safely.

3. Use a Cloud Secrets Manager

Most cloud service providers offer an inbuilt secrets manager that can be used to create and use secrets across cloud infrastructure. Following the secret managers offered by cloud providers:

AWS Secrets Manager is widely used across the industry. Let’s write a function to create and access a secret in AWS using Boto3.

import boto3

def fetch_secret_from_aws(secret_name):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       get_secret_value_response = client.get_secret_value(SecretId=secret_name)
       return get_secret_value_response['SecretString']
   except Exception as e:
       print(e)
       return None

def create_secret_in_aws(secret_name, secret_value):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       client.create_secret(Name=secret_name, SecretString=secret_value)
       return True
   except Exception as e:
       print(e)
       return False

4. Using a KMS

A KMS is a key management system that is used to manage cryptographic keys. It is typically used in organizations in order to centrally manage and secure keys. A KMS can be used to generate, store, and distribute keys. It can also be used to revoke keys and to monitor key usage.

KMS is a convenient way to centrally manage the keys used by your applications and services and helps to ensure that only authorized users have access to them.

Hashicorp Vault is one of the best open-source KMS available in the market that offers a number of features and benefits, including the ability to manage secrets and keys across multiple environments, strong security controls, and good scalability.

Let’s write a function to read and write secrets to a specific path in the vault.

💡 Note: Please ensure you have hvac (Python client for Vault) installed and have a Hashicorp Vault setup.

import hvac

def read_secret_from_vault(secret_path, token, secret_name):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       read_response = client.secrets.kv.read_secret_version(path=secret_path)
       return read_response['data']['data'][secret_name]
   except Exception as e:
       print(e)
       return None

def write_secret_to_vault(secret_path, token, secret_name, secret_value):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       create_response = client.secrets.kv.v2.create_or_update_secret(
           path=secret_path,
           secret={secret_name: secret_value},
       )
       return create_response
   except Exception as e:
       print(e)
       return None

Wrapping it up

Managing secrets is an essential part of application development. When developers hardcode secrets in cleartext into their applications, it creates a potential security breach. An attacker can use those secrets to access sensitive data if those secrets are discovered.

Another alternative to the proposed methods here is to check secrets into source code and share them with your team encrypted. This can be a very flexible solution if you learn how to leverage a tool such as Mozilla SOPS. To learn more about SOPS and get started using it, read this tutorial.

Did you know that GitGuardian detects more than 16000 secrets hardcoded into GitHub commits daily? When a secret is detected, GitGuardian notifies the repository owner so they can take action to protect their data.

If you want a better view of how many secrets could be hidden inside your repositories, sign up (for free) or book a demo to the GitGuardian Platform to scan every single commit and take preventive action now!

9 Things to Consider When Choosing an SCA Tool

Keshav Malik — Wed, 07 Dec 2022 06:02:36 +0000

In the past, the development of software was something that required a lot of effort and resources. Basically, every piece of code was developed in-house, and code reuse was quite limited. The situation is now the opposite. Open-source packages are so widely used that they make up the bulk of the total amount of software produced by passionate hobbyists and virtually all the software professionals in tech companies. The convenience of reusing and fine-tuning components made open-source is just too strong for most software engineers to ignore it and keep "reinventing the wheel".

To get a better idea of how big open source has become, we have some recent insights: according to a survey from Gartner, over 90% of the respondents stated that they rely on open source components. In another report from Synopsis, 98% of the audited codebases contained at least one open-source component, and 75% of the source code came from open-source. The report also noted that 85% of the audited codebases contained components "more than four years out of date".

This last data reveals the growing concern about the reliability and security of all this open-source material found in private code bases. Packages that are not maintained anymore cannot be patched for recently discovered vulnerabilities. Therefore it's become critical for organizations to be able to inventory open-source components and assess their vulnerabilities, making the use of software composition analysis mandatory. But not all SCA tools are created equal.

This blog post will provide eight factors to consider when choosing an SCA tool.

What is SCA (Software Composition Analysis)?

SCA is the process of analyzing an application's dependencies to determine if it is affected by known security vulnerabilities. Organizations can more effectively manage security risks by understanding the dependencies between components.

SCA can be performed either manually or using automated tools. Manual SCA is not scalable as the engineers must constantly check a vulnerability database such as NVD (National Vulnerability Database, maintained by NIST) and then compare the vulnerable versions with their existing dependencies. Much more efficient is to use automated SCA scanning tools, which can be triggered manually or integrated into the CI/CD pipeline for continuous checks.

Importance of Software Composition Analysis

SCA is integral to application security, as it helps identify and mitigate risks associated with using third-party components. SCA can help identify vulnerabilities in third-party components that attackers could exploit. It can also help track the versions of third-party components and ensure that they are up to date. By keeping track of the components used in an application, SCA can also help ensure compliance with license and security policies.

In short, SCA is an essential part of the code security toolkit, focused on third-party components.

How is SCA different from SAST?

SAST is a software testing technique that involves analyzing the source code of a software application to identify potential security vulnerabilities such as injection attacks, cross-site scripting (XSS), improper error handling, and insecure use of cryptographic functions.

SAST aims to identify security vulnerabilities early in the software development cycle to mitigate them before the application is compiled and deployed.

While SAST is an analysis technique used to check for known vulnerabilities in source code, SCA is used to scan dependencies for security vulnerabilities and license issues.

Both are usually performed pre-build (against source code), or post-build (against binaries), as they don't require an application's execution to identify potential vulnerabilities. Both are equally important in ensuring the security of a software application.

How to Choose an SCA tool?

The many SCA tools on the market today can make it hard to decide which is the right one for your needs. To help you select the right one, we have curated a list of 9 things you should look out for in an SCA tool:

1. Language Support

When opting for an SCA tool, it's vital to check what all languages are supported by the tool. Software composition analysis is language, end even ecosystem-dependent (package managers, build system, etc.): for instance, most SCA tools rely on lock files such as package-lock.json or Pipfile.lock to find dependencies and their respective versions. So you need to be careful here.

2. Accessible to Use/Developer Friendliness

The SCA tool you choose should make your life easier, not harder. It should be intuitive and easy to use so that you can focus on your work, not on learning the tool. It should also be developer friendly so that you can easily integrate it into your existing development process. Also, it should be scalable to grow with your organization.

The vendor should also have proper technical documentation available for the developers and having technical support for the tool is always a plus.

3. Support for Binary scanning

When looking for a Software Composition Analysis (SCA) tool, choosing one that supports the scanning of binary files is essential. Many SCA tools don't support this type of scanning, which can leave vulnerabilities in the binaries used by the developers unchecked.

Scanning binary files such as wheel files(.whl) are essential because it can find vulnerabilities that would otherwise be missed by scanning the dependencies. If you are not doing binary scanning used by your developers, you are not getting the complete picture of your code's security.

4. Direct vs. Transitive Dependencies

There are two types of dependencies in software development: direct and transitive.

A direct dependency is when one piece of software directly depends on another piece of software. For example, if software A directly uses software B, then A has a direct dependency on B. Conversely if software A uses software B, and software B uses software C, then A has a transitive dependency on C.

The SCA tool should be able to provide you with the information if the vulnerability is in the direct dependency or transitive dependency. This is important because both types of dependencies can pose a security threat to a software project.

5. False positives / False negatives

If you are not familiar with the concept of false positives and false negatives, you should read this article about accuracy, precision, and recall.

In a nutshell, false positives are vulnerabilities wrongly flagged as such by the tool, and false negatives are genuine vulnerabilities that the detection tool silently skipped. This is an important factor to consider when choosing a software composition analysis tool because false positives can lead to wasted time and resources.

Developers may spend a significant amount of time manually reviewing and investigating these results, even though most are not security threats. This (also known as "alert fatigue") can be frustrating for developers and decrease their trust in the tool, potentially leading them to ignore or not use the tool as frequently as they otherwise would.

The numbers reported by vendors can be challenging to verify. The best way to ensure that an SCA tool can accurately identify security threats while minimizing the number of false positives is to test it (for free, when possible).

6. Webhooks & API Support

When looking for an SCA tool, it's essential to check if the tool has proper API and webhook support so that you can easily integrate it into your CI/CD pipeline.

Having proper API and webhook support will allow you to automate SCA scanning as part of your CI/CD process, which will help ensure that your applications are always up-to-date and compliant with security standards. Without proper API and webhook support, you may have to manually trigger SCA scans, which can lead to delays in your pipeline.

7. Rich Vulnerability Database

A good SCA vendor should have a rich vulnerability database to detect vulnerabilities in your open-source packages. Such a database would enable the SCA vendor to provide you with customized alerts and recommendations on the best way to fix the identified vulnerabilities.

In addition, the SCA vendor should have a team of expert analysts who can help you understand the nature of the vulnerabilities and their potential impact on your organization in case you need any support.

8. Time to Onboard New CVEs

SCA tools are used by many organizations to keep track of vulnerabilities and potential security risks. It's essential to check how much time the SCA tool takes to onboard new CVEs on their platform from the vulnerability database. This allows organizations to plan for and respond to new security risks promptly. Additionally, it helps to ensure that SCA tools are up-to-date and provide accurate information about the latest security threats.

9. Detailed Reporting/Remediation

The scanning should provide detailed reporting that helps the security team understand the scan results of the open-source packages. The report should include a list of all the packages that were scanned, as well as the results of the scan, including

Vulnerability description
CVSS score
CVSSv3 score
Version impacted

The SCA tool should also provide proper remediation steps so that the developers can fix the issues.

Conclusion

Open source has become the norm in the software development world, with nearly 80% of companies using open-source software in some form or another. For many companies, open source is the preferred choice for software development due to its flexibility, cost-effectiveness, and vast ecosystem of available tools and resources.

However, with the increasing popularity of open source, there is also an increase in the number of vulnerabilities of open-source packages. This can create several risks and challenges for companies, including vulnerabilities in developed applications, licensing issues, and potential security breaches.

Having SCA tools in your build and deployment pipeline can help you avoid security risks that might arise from any open-source dependency. You should now be more informed about the important factors to consider before selecting such as tool.

Best practices for managing developer teams in GitHub Orgs

Keshav Malik — Wed, 02 Nov 2022 07:49:13 +0000

Are you looking for ways to manage your developer team better? GitHub Orgs is a great way to keep track of repositories, branches, and collaborators all in one place. In this article, we'll share some best practices for managing developer teams in GitHub Orgs.

There's a lot to love about GitHub, and the organizational structure it can give you is definitely one of them. When you share your code with other developers, make it public, or collaborate with a large team (or even a small one), the circle expands faster than you would imagine. Like it or not, a managed hierarchy of access and well-thought rules become imperative for the sustainability and security of your projects.

What could go wrong if you do not manage access? It's a legitimate question considering you trust the people you collaborate with. Well, all it takes is an honest intern who rewrites the history in your master branch to wreak havoc. Let us say it is safe to have a managed organizational structure, and that's what we'll talk about.

In this blog, I will guide you through the best practices for managing developer teams in GitHub Orgs and provide recommendations to help you secure your GitHub organization.

What is an organization in GitHub?

A GitHub organization is a shared platform where you can put one or more repositories and share controlled access with members and collaborators who can perform specific tasks. These organizations can be companies, non-profit organizations, some coalitions, or independent developers working together on a project for fun. An unlimited number of people can collaborate in an organization across multiple projects.

Usually, an organization has five levels of role-based permissions

Owner: has complete control of the organization along with full administrative access. They can determine the team's access level to the project code.
Billing manager: can manage payment settings for the organization. This includes setting up payment methods, managing subscriptions, and viewing invoices.
Member: is granted role-based access to control what they can see and do within the organization. Depending on their role, they may access the repositories or projects.
Moderator: have all the permissions granted to members by default. On top of that, they can block or unblock outside collaborators, set interaction limits, and hide comments on public repositories that belong to the organization.
Outside collaborator: a contractual member who's granted access to one or more repositories to perform specific tasks but doesn't have the full array of permissions granted to the members.

What is a team in GitHub?

Teams are groups of members that reflect the organization's structure. The teams are maintained by organization owners and team maintainers who can manage the level of access granted to a team and add or remove members. Teams have pages where you can see the members, child teams, the team profile picture, etc.

Managing a team's access to repositories in GitHub

The owners can give a team access to a repository, remove the access, or modify the permission level to a specific repository based on the need.
If a team has direct access to a repository, the access can be simply removed. However, if a team has inherited access from a parent team, then the maintainers have to remove the access for the parent team to remove access for the child team.

8 Best Practices for Managing Teams in GitHub

Now that we have somewhat established organizations and teams in GitHub let's jump right into the practices that make it easier to manage teams safely and sustainably.

Configure the Organization

Creating an organization on GitHub is extremely simple; it takes just a few clicks. But you should take the time to configure the organization based on the group's specific needs. This could mean restricting access to certain repositories, setting up review cycles, or modifying other settings to strengthen the organization's security. This will help to keep things organized and make it easier for team members to find the information they need.

One Branch for Each Feature or Bug

One of the key advantages of a GitHub organization is allowing an unlimited number of developers to work on different projects simultaneously. Having a branch for each bug or feature helps you make the most out of this advantage, allowing multiple people to work on various features without worrying about overwriting the code written by someone else. This also keeps things well organized.

Create Nested Teams

As an organization owner, you can create child teams or nested teams that inherit the permissions from their parent teams. It simplifies access management, as adding or removing permissions for the parent teams automatically means doing the same for the child teams.

When working with a large number of groups, having nested teams simplify communication.
Nested teams can also be given different permissions than their parent teams so that you can tailor the team's access to your organization's needs.

Assign Code Owners

The success of a project relies heavily on code reviews. Code owners are a special type of GitHub user responsible for reviewing and approving code changes in a repository. Code owners can be assigned to specific files or directories and will be notified whenever a change is made to those files. This makes pull requests safer, removes the confusion pertaining to ownership and reviews, and ensures the smooth functioning of a team.

Rebasing the Feature Branches

As we have discussed earlier, creating a branch for each feature or bug is a good practice. But if a team is working on a bunch of features simultaneously, it creates a complicated graph of branches that becomes very difficult to track. To avoid that, the ideal practice is to rebase the feature branches with the master branch.

Rebasing can help resolve merge conflicts more efficiently and make it easier to track changes and keep your codebase up to date.

Squash Commits Before Merging

Commits are like milestones in the journey of a Git project. Squashing commits is a way of rewriting the commit history before you share it with your team. It keeps the commit history clean and makes it easier to associate a bug with a specific commit when it comes to testing or debugging.

Setup Pre-Merge & Post-Merge Jobs with GitHub Repositories

If you're using GitHub for your project development, you can easily set up pre-merge and post-merge jobs to run automatically on your repositories.

This also helps to ensure that the code integrates properly and that any potential conflicts are resolved before they cause problems. Pre-merge jobs can be used to run tests and verify that the code compiles correctly. Post-merge jobs can be used to deploy the code to a staging environment for further testing.

Use Secret Scanning Tools

Secret scanning tools like GitGuardian or gitleaks can help you avoid accidentally leaking secrets like API keys and passwords through your GitHub repositories. These tools work by scanning your repositories and pull requests for potentially sensitive information and then alerting you so you can take action to remove or protect the information.
You can install the GitGuardian App through the GitHub Marketplace for free on public repositories here.

Using a secret scanning tool is a good first step in preventing accidental leaks, but it's also important to be aware of the potential risks and to review your code regularly to ensure that no sensitive information is accidentally left exposed.

Track Progress using GitHub Projects

It is essential to keep track of team progress. This can be done by creating a GitHub project board and adding issues and milestones. This feature allows you to create Kanban-style boards for your repositories to see what's being worked on and what is still pending. You can also add notes and tags to your cards to keep all the relevant information in one place. Tracking progress using GitHub Projects is a great way to keep your project organized and on track.

Best practices to secure your GitHub Organization at a glance

To keep your GitHub Organization secure, it is essential to follow best practices. Some of these best practices include:

Backup & restore for your repos

As a best practice, it is always recommended that you have a backup and recovery plan for critical repositories and ever-changing metadata. This will help ensure that you can recover your data in the event of a disaster and that your metadata is always up-to-date.

Supervise access controls

One of the most crucial security steps is setting access controls on an as-needed basis. Make sure that no one has more permissions than they need to have. This will help to prevent unauthorized access to sensitive information. You should also have a process in place for revoking access when someone no longer needs it.

MFA for all

Enforce multi-factor authentication for all members of the GitHub organization, moderators & billing managers. MFA (multi-factor authentication) adds an extra layer of security to your account by requiring you to enter a code from your mobile phone or another device in addition to your password. This makes it much harder for threat actors to gain access to your account, even if they have your password.

Security policies

Organizations on GitHub have several different security settings that can be adjusted to fit the organization's needs. It is essential to review these settings regularly and audit your member's activity to ensure that your organization's data is safe.

To sum it up

GitHub organizations are fantastic if you can maintain a few habits and imbibe some practices into the way your teams function. The responsibility is shared between the owners and the team members. As the owner, you can create nested teams, set security policies, and assign code owners to make things easier. Then if the team members can pull their share of the weight by being responsible for squashing the commits, rebasing the branches, and testing code, everything falls into place.