DEV Community: Kat Cosgrove

When Projects Fail: Why Companies Should Treat Open Source as Infrastructure

Kat Cosgrove — Wed, 18 Mar 2026 14:39:24 +0000

Maintaining an open source project is hard. It requires managing a group of people who are largely working for free to build something that other people profit off of, usually distributed across the globe, with limited resources.

The whole time you’re doing this, you’re receiving demands from users and businesses alike for features or bug fixes on a timeline that works for them, not you and your (possibly very limited) group of contributors that you can’t exactly order around, since they aren’t being paid. It’s stressful, and it can be overwhelming.

When one of these projects is the victim of an attack that takes advantage of the fact that there are only one or two maintainers, or eventually has to shut down due to rising technical debt and falling contributor numbers, the public blame falls on us, not on the businesses that didn’t offer contributors in time.

The Open Source Dependency Problem

According to a 2022 Linux Foundation study, open source software makes up 70-90% of any given modern software solution. So, if that much of the world’s digital infrastructure is reliant on open source software, why do so many projects both large and small struggle to attract and maintain contributors?

We’ve all heard the morality and community arguments for contributing back to open source projects, but we’re still seeing critical projects fall victim to attacks that take advantage of a small, overworked team of maintainers or shut down entirely because they didn’t get help until it was too late. So maybe it’s time we look at the business value of open source contribution from another angle.

Not contributing to open source is a security problem for your business.

The success of your business and the safety of your users’ data relies heavily on the stability of the open source software you build your product on top of. So, you should be motivated to ensure those open source projects remain healthy and secure, right? However, historically, only the very largest of projects has received this treatment.

We’ve seen it time and time again. A small but widely-used project has only one or two maintainers. Because they’re overworked, unpaid, and desperate for help, they’re easy to take advantage of. They may not review a PR as thoroughly as they should, and it may be easier for a bad actor to slip into a trusted position over time. Technical debt piles up, and with it, unresolved CVEs. Sometimes this is a one-off issue and a wake-up call for an industry to begin contributing to that specific project, or at least begin doing more investigation into the health of a project before relying on it. Sometimes, that offer of help comes too late and the project is unrecoverable.

A Real-World Example: Ingress NGINX

Ingress NGINX is an example of this. Roughly 50% of cloud native environments rely on this small but mighty utility for managing Kubernetes traffic. Despite its popularity and importance to the ecosystem, it has been steadfastly maintained by only two people for years. Calls for help have long been ignored, even in the face of a vulnerability with a 9.8 CVSS score. There is now no choice but to archive the project.

The fundamental architecture of Ingress NGINX is, to quote Kubernetes Security Response Committee member Tabitha Sable, “a never-ending CVE piñata.” No number of maintainers could reasonably be expected to keep it in a secure state, and it is no longer possible to get it to a point where it would be easier to maintain. If the project received a donation of an entire engineering team today, that engineering team would still just be used to ensure a more graceful shutdown.

This wasn’t always the case. There was once active development on a more modern implementation that would have resolved a lot of Ingress NGINX’s design issues, but the project maintainers were never able to attract more contributors and development stalled.

Had the businesses who are operating large cloud native environments reliant on Ingress NGINX begun contributing back to the project two years ago when asked, we might not be where we are today. Instead, the only responsible option left is to archive the project. Help came too late, and now users are faced with the choice between running an outdated Ingress NGINX deployment (and accepting the attack risk that comes with it) or spending the time and money necessary to migrate to another solution.

If You Rely on Open Source, Help Maintain It

Contributing upstream isn’t just for feel-good community points. It isn’t charity. It isn’t for clout. It’s a necessary part of maintaining a secure and reliable software supply chain. Businesses that rely on open source should treat upstream projects as part of their infrastructure. That means dedicating engineering time to contribute fixes and improvements, funding maintainers or foundations that support critical projects, prioritizing contributions to the dependencies their products rely on most, or hiring engineers or DevRel practitioners whose role includes contributing upstream.

Individual engineers can help too. Contributing documentation, reviewing pull requests, reporting bugs, improving tests, or helping with issue triage are all meaningful ways to reduce the burden on small maintainer teams. Engineers can also advocate within their organizations and push the companies they work for to support the projects their products depend on.

Open source projects rarely fail overnight. They usually show warning signs long before that point: maintainer burnout, unanswered issues, stalled development, and growing technical debt. Contributing upstream helps address those problems before they turn into security incidents, project shutdowns, or expensive migrations.

Because when a critical open source project fails, the impact doesn’t stay upstream. It spreads to every company, product, and user that depends on it.

Cloud Systems Part Three: Deploying to Amazon ECS

Kat Cosgrove — Mon, 31 Jan 2022 15:25:41 +0000

Cloud engineering is taking over software development. In a lot of ways, this is great; it allows us to build and deploy more complicated applications with less difficulty, and maintaining those applications becomes less troublesome too. We can release smaller updates more quickly than ever, ensuring that we can stay on top of feature requests and security issues. That said, the rise of cloud engineering has also introduced a lot of complexity in the form of dozens of services even within just one cloud provider. Figuring out where to start can be tough, so let’s take a practical tour! In this series, I’ll walk you through building a personal website and deploying it using modern cloud engineering practices.

Elastic Container Service

In the previous tutorial, we extended our personal website to use the Flask web framework, add server-side routing, and package everything up into a Docker container. It's still only running locally, though, and we want to deploy it. To do that, today we'll be learning to use Pulumi to deploy to Amazon's Elastic Container Service. If you completed part two of this series, we'll be picking up right where we left off. If you're just now joining me, you can get the completed code by forking and cloning this repository.

Prerequisites:

Configuring Amazon ECS

Amazon’s ECS (Elastic Container Service) is a tool used for container orchestration. It allows you to deploy containerized applications into a cluster, defined as tasks. We’re going to use Pulumi to define all of the necessary rules and settings to do this in a secure manner, then create our infrastructure and deploy the containerized website.

First, run pulumi new python -y to start a new Pulumi project if you don't already have one for this tutorial. From now on, we'll be working within __main__.py

import json
import base64
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

First, we have our imports. We’re working with both AWS and Docker, so we need both of those Pulumi providers. Launch the virtual environment set up for you by Pulumi with source venv/bin/activate and run pip3 install pulumi-aws pulumi-docker to get both. If you want to make sure your dependencies are saved for later, you can run pip3 freeze > requirements.txt .

app_cluster = aws.ecs.Cluster("app-cluster")

app_vpc = aws.ec2.Vpc("app-vpc",
    cidr_block="172.31.0.0/16",
    enable_dns_hostnames=True)

app_vpc_subnet = aws.ec2.Subnet("app-vpc-subnet",
    cidr_block="172.31.32.0/20",
    vpc_id=app_vpc.id)

In this codeblock, we're first creating our ECS cluster and naming it app-cluster. Then create a VPC (Virtual Private Cloud), and a subnet for it. VPCs are like miniature local networks, with subnets providing the addressing within that mini network. Using a VPC means you can wire up your system as you like it without worrying about how that network setup might collide with other AWS resources, since you define how that network can connect outward. If you've ever used Docker networking, you'll be familiar with the idea that an internal network is different than the external interface.

It is possible to use the default VPC here rather than defining one, but AWS networking is complex, so it's worth knowing that it is possible to define a custom VPC.

app_gateway = aws.ec2.InternetGateway("app-gateway",
    vpc_id=app_vpc.id)

app_routetable = aws.ec2.RouteTable("app-routetable",
    routes=[
        aws.ec2.RouteTableRouteArgs(
            cidr_block="0.0.0.0/0",
            gateway_id=app_gateway.id,
        )
    ],
    vpc_id=app_vpc.id)

app_routetable_association = aws.ec2.MainRouteTableAssociation("app_routetable_association",
    route_table_id=app_routetable.id,
    vpc_id=app_vpc.id)

Next, we need a gateway and a route table. The gateway's job is to provide a target in that route table we just created and associated with our VPC. This allows our VPC to communicate with the public internet.

app_security_group = aws.ec2.SecurityGroup("security-group",
    vpc_id=app_vpc.id,
    description="Enables HTTP access",
    ingress=[aws.ec2.SecurityGroupIngressArgs(
        protocol='tcp',
        from_port=80,
        to_port=80,
        cidr_blocks=['0.0.0.0/0'],
    )],
    egress=[aws.ec2.SecurityGroupEgressArgs(
        protocol='-1',
        from_port=0,
        to_port=0,
        cidr_blocks=['0.0.0.0/0'],
    )])

Just about everything you do in AWS requires a security group. The purpose of a security group is to control traffic, both inbound and outbound. This one enables HTTP access.

# Creating an IAM role used by Fargate to execute all our services
app_exec_role = aws.iam.Role("app-exec-role",
    assume_role_policy="""{
        "Version": "2012-10-17",
        "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": ""
        }]
    }""")

# Attaching execution permissions to the exec role
exec_policy_attachment = aws.iam.RolePolicyAttachment("app-exec-policy", role=app_exec_role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy")

# Creating an IAM role used by Fargate to manage tasks
app_task_role = aws.iam.Role("app-task-role",
    assume_role_policy="""{
        "Version": "2012-10-17",
        "Statement": [
        {
            "Action": "sts:AssumeRole",
            "Principal": {
                "Service": "ecs-tasks.amazonaws.com"
            },
            "Effect": "Allow",
            "Sid": ""
        }]
    }""")

# Attaching execution permissions to the task role
task_policy_attachment = aws.iam.RolePolicyAttachment("app-access-policy", role=app_task_role.name,
    policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_FULL_ACCESS)

We're going to be using AWS Fargate to actually run our container. Fargate is a serverless compute platform designed for running containers without requiring you to provision and manage clusters of EC2 (Elastic Compute Cloud) instances. Since our website is containerized and very small, it's perfect for us. Services also require IAM roles, and in this case, we need to allow Fargate to execute our services and manage our tasks, so we create new roles for both and apply execution permissions to them.

In the context of Fargate, a task is a collection of containers, and a task definition tells AWS what needs to be running all at once in order to make an application run. The task definitions will be written later onm but first we need to define policies. It's important that these policies be created first, or the task definitions won't create correctly later on.

# Creating storage space to upload a docker image of our app to
app_ecr_repo = aws.ecr.Repository("app-ecr-repo",
    image_tag_mutability="MUTABLE")

# Attaching an application life cycle policy to the storage
app_lifecycle_policy = aws.ecr.LifecyclePolicy("app-lifecycle-policy",
    repository=app_ecr_repo.name,
    policy="""{
        "rules": [
            {
                "rulePriority": 10,
                "description": "Remove untagged images",
                "selection": {
                    "tagStatus": "untagged",
                    "countType": "imageCountMoreThan",
                    "countNumber": 1
                },
                "action": {
                    "type": "expire"
                }
            }
        ]
    }""")

The last thing we need to do for our infrastructure before we can start deploying to it is create a repository on Amazon ECR (Elastic Container Registry) where our Docker image will live, then attach an application lifecycle policy to that repository. This makes sure that we expire and remove any untagged images in the repository, keeping things from geting clogged up.

Deploying a Dockerized Flask Application to ECS

We're now ready to deploy our website and wire all of this up!

flask_targetgroup = aws.lb.TargetGroup("flask-targetgroup",
    port=80,
    protocol="TCP",
    target_type="ip",
    vpc_id=app_vpc.id)

flask_balancer = aws.lb.LoadBalancer("flask-balancer",
    load_balancer_type="network",
    internal=False,
    security_groups=[],
    subnets=[app_vpc_subnet.id])

flask_listener = aws.lb.Listener("flask-listener",
    load_balancer_arn=flask_balancer.arn,
    port=80,
    protocol="TCP",
    default_actions=[aws.lb.ListenerDefaultActionArgs(
        type="forward",
        target_group_arn=flask_targetgroup.arn
    )])

First, we need to make it possible for our Flask application to communicate with the internet. That requires three pieces of configuration: a target group for port 80, a load balancer to spread out incoming requests and make sure our website doesn't get overwhelmed as easily, and a listener to forward public traffic to the defined target group. The target group is directed at the endpoint for the VPC we created earlier, defining the port as it is unknown until runtime.

def get_registry_info(rid):
    creds = aws.ecr.get_credentials(registry_id=rid)
    decoded = base64.b64decode(creds.authorization_token).decode()
    parts = decoded.split(':')
    if len(parts) != 2:
        raise Exception("Invalid credentials")
    return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

flask_image = docker.Image("flask-dockerimage",
    image_name=app_ecr_repo.repository_url,
    build="./website",
    skip_push=False,
    registry=app_registry
)

A small helper function is required here. It's grabbing some of our AWS credentials, specificaly an authorization token, so that we can talk to the registry. Next, we build the Docker image for our website and push it to the repository we created in Amazon ECR earlier. Remember, the Dockerfile is in ./website.

flask_task_definition = aws.ecs.TaskDefinition("flask-task-definition",
    family="frontend-task-definition-family",
    cpu="256",
    memory="512",
    network_mode="awsvpc",
    requires_compatibilities=["FARGATE"],
    execution_role_arn=app_exec_role.arn,
    task_role_arn=app_task_role.arn,
    container_definitions=pulumi.Output.all(flask_image.image_name).apply(lambda args: json.dumps([{
        "name": "flask-container",
        "image": args[0],
        "memory": 512,
        "essential": True,
        "portMappings": [{
            "containerPort": 80,
            "hostPort": 80,
            "protocol": "tcp"
        }],
    }])))

We need a task definition for the Flask instance. AWS Fargate will be managing this for us, using the roles we defined earlier. We're also handing it a container definition, including the image name and a little bit of information about it, like the container and host ports.

flask_service = aws.ecs.Service("flask-service",
    cluster=app_cluster.arn,
    desired_count=1,
    launch_type="FARGATE",
    task_definition=flask_task_definition.arn,
    wait_for_steady_state=False,
    network_configuration=aws.ecs.ServiceNetworkConfigurationArgs(
        assign_public_ip=True,
        subnets=[app_vpc_subnet.id],
        security_groups=[app_security_group.id]
    ),
    load_balancers=[aws.ecs.ServiceLoadBalancerArgs(
        target_group_arn=flask_targetgroup.arn,
        container_name="flask-container",
        container_port=80,
    )],
    opts=pulumi.ResourceOptions(depends_on=[flask_listener]),
)

Finally, we actually launch our website. We need to create a new service definition and hand it all of the resources we created earlier, beginning with the ECS cluster itself. It has the task definition, our network configurations, and our load balancers, and we make sure that this particular piece of code isn't executed until the listener we created to watch for traffic is online.

As a shortcut to finding our website, export the DNS name of our load balancer as an output from Pulumi as the last part of the program:

pulumi.export("app-url", flask_balancer.dns_name)

We're ready to go! Set your AWS region in your terminal:

pulumi config set aws:region us-west-2

Then run pulumi up to watch it go! This time, we're going to see way more services created than before.

     Type                                  Name                        Status      Info
 +   pulumi:pulumi:Stack                   part-three-dev              created
 +   ├─ docker:image:Image                 flask-dockerimage           created
 +   ├─ aws:ec2:Vpc                        app-vpc                     created
 +   ├─ aws:ecs:Cluster                    app-cluster                 created
 +   ├─ aws:iam:Role                       app-exec-role               created
 +   ├─ aws:iam:Role                       app-task-role               created
 +   ├─ aws:ecr:Repository                 app-ecr-repo                created
 +   ├─ aws:iam:RolePolicyAttachment       app-exec-policy             created
 +   ├─ aws:iam:RolePolicyAttachment       app-access-policy           created
 +   ├─ aws:ecr:LifecyclePolicy            app-lifecycle-policy        created
 +   ├─ aws:ec2:Subnet                     app-vpc-subnet              created
 +   ├─ aws:ec2:InternetGateway            app-gateway                 created
 +   ├─ aws:ec2:SecurityGroup              security-group              created
 +   ├─ aws:lb:TargetGroup                 flask-targetgroup           created
 +   ├─ aws:ecs:TaskDefinition             flask-task-definition       created
 +   ├─ aws:lb:LoadBalancer                flask-balancer              created
 +   ├─ aws:ec2:RouteTable                 app-routetable              created
 +   ├─ aws:ec2:MainRouteTableAssociation  app_routetable_association  created
 +   ├─ aws:lb:Listener                    flask-listener              created
 +   └─ aws:ecs:Service                    flask-service               created

Outputs:
    app-url: "flask-balancer-a90d260-057e5a40e0267b63.elb.us-west-2.amazonaws.com"

Resources:
    + 20 created

Duration: 3m36s

Pulumi has created 20 resources for you, and helpfully returned the DNS address of your load balancer. I've snipped out the diagnostics section of this output for the sake of keeping things brief, but you will also see the step-by-step process Docker goes through building and running your image. Go to that URL after a couple of minutes, and your website is online!

We now have a functional, multi-page website with server-side routing, packaged up into a Docker container and deployed to a fully-configured AWS ECS cluster in just a couple hundred lines of Python, all without leaving the same repository our website is stored in. What if we want to be able to scale really large, though? Some resiliency would be nice. Stay tuned for the next blog for an introduction to Kubernetes, where we'll learn to deploy our website to Amazon's Elastic Kubermetes Service!

Cloud Systems Part Two: Containerizing a Website

Kat Cosgrove — Wed, 26 Jan 2022 20:43:54 +0000

In part one of this series, we built a personal website and deployed it to AWS S3. That works perfectly well for a static, single-page application with minimal interactivity, but if you want server-side routing or database interactivity, things have to get a little bit more complicated. In part two of this series, we’ll be adding a couple more pages to our personal website, adding server-side routing, and containerizing it with Docker.

Containers

So, what’s a container and why would you want to use one? You can think of a container here the exact same way you think of containers on a shipping barge. On that barge is a bunch of shipping containers, and inside each shipping container is a bunch of packages. The barge itself is your computer (or your cloud environment), and the shipping containers house your applications.

It may also help to think of them as smaller, more lightweight incarnations of virtual machines. While a virtual machine virtualizes the machine’s physical hardware through the use of a hypervisor, a container virtualizes only the operating system. A virtual machine is usually several gigabytes in size, whereas a container is usually less than a gigabyte. The small size and speed of deployment as compared to a traditional virtual machine has given rise to the popularity of designing applications as collections of microservices (several single-purpose services running in containers that, when working together, make up your complete application) instead of monoliths (all of your application code and services running as a single unit, often within a single file).

Deploying your application using containers allows you to package your application code alongside everything required to run it, including its dependencies and an operating system. You know how sometimes you hand code off to someone else, but it doesn’t run for them, so you go “Well, it works on my machine,” and just shrug? Using a container is pretty similar to just deploying your machine. The most popular container engine today is Docker, so that’s what we’ll be using today.

Prerequisites:

Docker
Python3

Before we go into containerizing and deploying our website, let’s make it a little bit more useful. For this, we’re going to be using Flask, a lightweight web framework for Python. Fork and clone this GitHub repository to get the full sample code for part two of this series.

Expanding the Website

Our file structure gets more complicated now that we are building a more dynamic website with server-side routing and throwing it into a Docker container. You will have something like this:

container-tutorial
├── Pulumi.yaml
├── requirements.txt
├── __main__.py
└── website
    ├── Dockerfile
    ├── requirements.txt
    └── app
        ├── server.py
        ├── static
        │   ├── normalize.css
        │   ├── style.css
        │   └── background.jpg
        └── templates
            ├── base.html
            ├── index.html
            ├── portfolio.html
            └── about.html

Everything contained within the app directory is the website itself. We now have server-side routing, and a few different pages. To run it locally and see what the new website looks like, pip3 install flask to install the Flask web framework and then run python3 server.py from the app directory. Flask will return an IP address where your website is running; navigate there in your browser and click around a bit!

server.py

from flask import Flask, render_template

app = Flask(__name__)

@app.route("/")
def home():
    return render_template("index.html")

@app.route("/about")
def about():
    return render_template("about.html")

@app.route('/portfolio')
def portfolio():
    return render_template("portfolio.html")


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=80, debug=True)

Where before our entire website was made up of a single HTML file, we now have several, and we're using Flask to handle the routing between them. In this server file, three routs are defined: home, about, and portfolio. Flask handles serving these up for us, and associating a route with a particular HTML file. This way, we get a nice www.mywebsite.com/about URL instead of something like www.mywebsite.com/about.html. Building a website this way also means we have the ability to apply some logic to each of these routes, such as adding database interaction, user login, and passing conditional variables from the server to the templates that will be rendering each page. We aren't doing any of that yet, but we will!

At the bottom, we're binding the Flask application to 0.0.0.0:80. You can change that to any other unoccupied local IP address and port you like.

Templating

Flask can make use of a templating engine called Jinja2 to make adding pages to your website a bit less repetitive. You start with one template, in this case called base.html, that contains any HTML you want to exist on every single page.

base.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Hello world!</title>
    <link rel="stylesheet" href={{ url_for('static', filename='style.css') }}>
    <link rel="stylesheet" href={{ url_for('static', filename='normalize.css') }}>
</head>
<body>
    <header>
        <div class="logo"><i class="fas fa-cat"></i></div>
        <nav>
                <ul>
                    <li {% if request.path == url_for('home') %} class="active" {% endif %}>
                        <a href="{{ url_for('home') }}">Home</a>
                    </li>
                    <li {% if request.path == url_for('about') %} class="active" {% endif %}>
                        <a href="{{ url_for('about') }}">About</a>
                    </li>
                    <li {% if request.path == url_for('portfolio') %} class="active" {% endif %}>
                        <a href="{{ url_for('portfolio') }}">Portfolio</a>
                    </li>
                </ul>
            </nav>
        <ul class="social">
                <li><a href="http://github.com/katcosgrove" target="_blank"><i class="fab fa-github-alt"></i></a></li>
                <li><a href="http://twitter.com/Dixie3Flatline" target="_blank"><i class="fab fa-twitter"></i></a></li>
                <li><a href="http://linkedin.com/in/katcosgrove" target="_blank"><i class="fab fa-linkedin-in"></i></a></li>
            </ul>
    </header>
{% block content %}{% endblock %}
</body>
<script src="https://kit.fontawesome.com/b4747495ea.js" crossorigin="anonymous"></script>
</html>

This file includes some metadata, stylesheets, the header, navigation, and any Javascript we want to link in. All of the content in this file will exist on every page. Below the header, in the body, you'll see {% block content %} {% endblock %}. That's where additional, page-specific content will be added.

index.html

{% extends 'base.html' %}

{% block content %}

<div class="content">
    <h1>Your Name</h1>
    <h3>Job Title at Company</h3>
</div>

{% endblock %}

This is all that exists in our index.html now. The first directive tells Jinja2 that this bit of HTML goes in the base.html template, and the remaining two wrap the content we want to insert. The same thing is happening in about.html and portfolio.html. Neat, right? Jinja2 is not unique to Flask, so you can use this templating engine with another web framework if you prefer.

While we could deploy this as-is, it's a bit unwieldy. Let's wrap it up into a container.

Dockerizing our Website

In the website directory, we have something called a Dockerfile. Dockerfiles tell the Docker engine what to do with your application, how your container should behave with respect to the wider internet or other containers, and what needs to be done inside of the container for your application to run. Ours looks like this:

FROM ubuntu:20.04

COPY app /app

WORKDIR /app

EXPOSE 80

RUN apt-get update && \
    apt install -y gcc python3-dev python3-pip python-markupsafe

COPY requirements.txt /app

RUN pip3 install -r requirements.txt

ENTRYPOINT [ "python3" ]

CMD [ "server.py" ]

This Dockerfile says that our container should be running ubuntu:20.04 as its operating system. It will pull the Ubuntu image from Docker Hub, their public image registry, though you can pull from another registry here as well. The COPY directive moves our website’s files (the app directory) into the container. We then set our working directory for all future commands run inside of the container to /app to reduce the amount of typing we have to do later on, update Ubuntu, and install some dependencies using both apt and from the requirements.txt file we also copy over. Lastly, an entrypoint and command to actually start our application are defined. There is a slightly less verbose way to start the application, but this way of defining an ENTRYPOINT and CMD is fairly common, so it’s worth seeing here.

To see this run locally, we first need to build and tag the image. From the same directory as your Dockerfile, run the following command:

docker build --tag container-tutorial:latest .

We are calling the docker CLI, telling it we want to build a container, and indicating that we want to tag it at the same time. The tag isn't strictly necessary for the container to run, but it does make the container easier to identify and interact with. We're also assigning it a version of latest. Note the trailing dot at the end of the command, which is the part of the command that indicates the location of the Dockerfile we want to build.

Run docker images to see a list of all images you have built. Something like this should be in the list:

cloud-systems                                                       latest                                                             0ce569ac0360   10 days ago    408MB

Now the container is built, we need to run it and forward the container’s port to a local port so we can see our site.

docker run -d -p 80:80 container-tutorial

There's a lot going on in that command, so let's take a closer look. the -d flag is short for --detach, and it tells the Docker engine that we want to print the container's ID and run it in the background. The -p flag is short for --publish, and it tells the Docker engine to publish the container's port to the host. In this case, port 80.

Run docker ps to get a list of all running containers, and you'll see something like this:

CONTAINER ID   IMAGE                COMMAND               CREATED       STATUS       PORTS                                       NAMES
f95362faf3cd   container-tutorial   "python3 server.py"   7 days ago    Up 7 days    0.0.0.0:80->80/tcp, :::80->80/tcp           affectionate_jackson

That's our website! You can see the container ID, the image tag, the commands that ran for it to start the application, its host, the container port, and the port it forwarded to. Go to localhost in your browser, and the website is up!

We're hosting this locally, though. It's not accessible by the wider internet, and even if it was, our machines would be dealing with all of the traffic. There's nothing in place to distribute traffic or restrict access. Worry not, cloud services exist for that, too. In the next in this series, we'll take our containerized website and deploy it to AWS Elastic Container Service, complete with AWS networking configuration, IAM roles, and AWS Fargate!

Cloud Systems Part One: Static Sites and AWS S3

Kat Cosgrove — Tue, 25 Jan 2022 18:48:50 +0000

The simplest website you can build is a static site. That means no web server, just HTML, CSS, and maybe some front-end JavaScript. Static sites can still be beautiful and interactive though, and for many people, they’re more than enough.

Amazon S3 (Simple Storage Service) is an AWS service that provides cloud storage for files, stored as objects in buckets. You can think of it like an external harddrive: plug it into your computer, and you have more storage space for your files, photos, and movies that you can unplug at will and move to another computer. An S3 bucket behaves very similarly in that you can throw just about anything in it, from data lakes to images to logs, and connect it to an application. This also means that you can use it to host a static website! Let’s look at how.

Prerequisites:

An AWS account
Pulumi account
Pulumi installed and configured for AWS
Python3
NodeJS (optional if you want to preview the site locally)

There are a lot of ways to provision an S3 bucket and set its permissions, but for this I’m going to use Pulumi, an Infrastructure as Code tool that allows you to provision, configure, and deploy a variety of cloud services and tools programmatically, without learning a new language. That way, we can stay in our code, and the syntax will be more familiar for developers. It also saves us the headache of going through the AWS Console. All of the code for this, both the website and the infrastructure code, can be found in the GitHub repository for the series, although I'll be walking you through everything here too.

Building a Static Site

Create a new directory called s3-tutorial and navigate into it. From there, run pulumi new python -y to create a new Pulumi project using Python. You’ll see Pulumi create a few files for you, but the one we’ll be touching is __main__.py.

Before we move on, we need to build out our website. Create a subdirectory and call it website. In that directory, add index.html, style.css, and normalize.css.

This one is built using only HTML and CSS, plus one background image file. Replace the name and bio with your own, as well as the social links in the header.

index.html

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8">
    <title>Hello world!</title>
    <link rel="stylesheet" href="style.css">
    <link rel="stylesheet" href="normalize.css">
</head>
<body>
    <header>
        <div class="logo"><i class="fas fa-cat"></i></div>
        <ul class="social">
                <li><a href="http://github.com/katcosgrove" target="_blank"><i class="fab fa-github-alt"></i></a></li>
                <li><a href="http://twitter.com/Dixie3Flatline" target="_blank"><i class="fab fa-twitter"></i></a></li>
                <li><a href="http://linkedin.com/in/katcosgrove" target="_blank"><i class="fab fa-linkedin-in"></i></a></li>
            </ul>
    </header>
<div class="banner">
    <h1>Your Name</h1>
    <h3>Job Title at Company</h3>
    <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam blandit tristique arcu, sed commodo ligula. Ut ullamcorper velit non luctus mollis. Sed placerat quam sed tellus aliquam, ac eleifend quam lacinia. Praesent bibendum velit at metus auctor, vitae feugiat quam luctus. Morbi eu imperdiet metus, et tincidunt tellus. Aliquam non ullamcorper justo. Cras ornare nulla vel pellentesque vulputate. Mauris in elit neque. Aliquam tempor aliquam libero, elementum luctus nisl imperdiet sit amet. Praesent urna ante, scelerisque non tellus mollis, laoreet sodales felis.</p>
</div>
</body>
<script src="https://kit.fontawesome.com/b4747495ea.js" crossorigin="anonymous"></script>
</html>

style.css

@import url('https://fonts.googleapis.com/css?family=News+Cycle|Teko&display=swap');
ul {
    list-style-type: none;
}
ul li {
    display: inline-block;
}
a {
    color: white;
    -webkit-transition: color .5s ease-out;
    transition: color .5s ease-out;
    text-decoration: none;
}
a:hover, a:active {
    color: rgb(255, 157, 112);
}
header {
    background-color: rgba(0, 0, 0, .8);
    height: 80px;
    position: absolute;
    top: 0;
    width: 100%;
}
header li {
    color: white;
}
.active a {
    color: rgb(255, 157, 112);
}
.social {
    position: absolute;
    right: 50px;
    top: -5px;
    font-size: 30px;
}
.social li {
    margin: 0 5px 0 5px;
}
.logo {
    position: absolute;
    left: 10px;
    margin-left: 20px;
    font-size: 60px;
    color: white;
}
body {
    background: url('./background.jpg');
    background-size: cover;
    background-repeat: no-repeat;
}
.banner {
    width: 60vw;
    font-family: Teko;
    font-size: 2vw;
    text-align: center;
    margin-top: 15vw;
    margin-left: 20vw;
}
.banner p {
    font-family: News Cycle;
}

Your normalize.css can be copied from this respository, and the expected background.jpg can be downloaded from this repository.

To see what this looks like before we deploy it, we're going to use a tool called live-server. This isn't a real, functional webserver, but it's a useful tool during development for static websites. It will reload automatically when a change is detected, so you don't have to wait for deploys.

Install it with this:

npm install -g live-server

And run it from the same directory as your index.html file:

live-server

Your browser should automatically open to localhost:8080, but if it doesn’t, navigate there yourself and you’ll see your website.

Configuring an S3 Bucket

It’s time to deploy this thing to S3. From your project’s root, run source venv/bin/activate to make sure we’re in a virtual environment, and run pip3 install pulumi-aws. From now on, we’ll be working in __main__.py

For imports, we need the following:

import json
import mimetypes
import os
from pulumi import export, FileAsset
from pulumi_aws import s3

We’re going to be reading files from the operating system and working with JSON, so we need the first three packages from the standard Python library. We’re also going to be using Pulumi. You will already have the pulumi package installed, but we also need the AWS provider, which we just installed.

web_bucket = s3.Bucket('s3-website-bucket',
    website=s3.BucketWebsiteArgs(
        index_document="index.html",
    ))

Here, we are instantiating an s3 Bucket object and assigning it to the web_bucket variable. S3 buckets are capable of hosting websites, but we have to tell the bucket that’s what it’s going to be doing and what it should expect the index document to be called. Ours is just called index.html.

content_dir = "website"
for file in os.listdir(content_dir):
    filepath = os.path.join(content_dir, file)
    mime_type, _ = mimetypes.guess_type(filepath)
    obj = s3.BucketObject(file,
        bucket=web_bucket.id,
        source=FileAsset(filepath),
        content_type=mime_type)

This codeblock defines our content directory (the website folder containing your site files, from earlier) and reads each of them in, before creating using the Pulumi AWS module to create an S3 bucket object for that file. We are also including a little bit of metadata for each file.

def public_read_policy_for_bucket(bucket_name):
    return json.dumps({
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": [
                f"arn:aws:s3:::{bucket_name}/*",
            ]
        }]
    })

This function sets some permissions for our bucket. Since this should be publicly-accessible, we need to allow anyone the ability to read the files.

bucket_name = web_bucket.id
bucket_policy = s3.BucketPolicy("bucket-policy",
    bucket=bucket_name,
    policy=bucket_name.apply(public_read_policy_for_bucket))

Now we assign the ID of our bucket as the bucket's name, then create a policy for it that includes its name and the policy we defined in the previous step. We're almost done!

export('bucket_name', web_bucket.id)
export('website_url', web_bucket.website_endpoint)

These last two lines are using Pulumi to export some data for us to interact with. The first is the ID of the bucket so we know which one we're working with, and the second is the website endpoint created by S3 once the bucket is online, so we know where to go to see our site!

Deploying

All that's left to do is deploy! From your project's root directory, define the AWS region you want to work in by running pulumi config set aws:region us-west-2, then run pulumi up!

Pulumi will first give you an overview of the expected changes and ask if you want to perform the update. In my case, it looks like this:

     Type                    Name                Plan
 +   pulumi:pulumi:Stack     static-site-s3-dev  create
 +   ├─ aws:s3:Bucket        s3-website-bucket   create
 +   ├─ aws:s3:BucketObject  index.html          create
 +   ├─ aws:s3:BucketObject  style.css           create
 +   ├─ aws:s3:BucketObject  normalize.css       create
 +   ├─ aws:s3:BucketObject  background.jpg      create
 +   └─ aws:s3:BucketPolicy  bucket-policy       create
Resources:
    + 7 to create
Do you want to perform this update?  [Use arrows to move, enter to select, type to filter]
> yes
  no
  details

If everything goes smoothly, you'll see more feedback from Pulumi showing each resource that was created:

     Type                    Name                Status
 +   pulumi:pulumi:Stack     static-site-s3-dev  created
 +   ├─ aws:s3:Bucket        s3-website-bucket   created
 +   ├─ aws:s3:BucketObject  index.html          created
 +   ├─ aws:s3:BucketObject  style.css           created
 +   ├─ aws:s3:BucketObject  normalize.css       created
 +   ├─ aws:s3:BucketObject  background.jpg      created
 +   └─ aws:s3:BucketPolicy  bucket-policy       created
Outputs:
    bucket_name: "s3-website-bucket-89f0341"
    website_url: "s3-website-bucket-89f0341.s3-website-us-west-2.amazonaws.com"
Resources:
    + 7 created
Duration: 6s

Notice the Outputs section there. Those are the two things we exprted at the very end of our __main__.py: the ID of the bucket, and the website URL. Click on the website URL, and you'll see your personal site, deployed to AWS S3 and publicly accessible by anyone! If you don't want to keep this up, run pulumi destroy to tear down all of these resources, and you're done!

We're not quite done yet, though—this site is a little slow, and it's not very interactive. Maybe we want some kind of database associated with it, or the convenience of containers, or the scalability of Kubernetes. Stay tuned for the next tutorial in this series, where we'll keep building on this website to introduce more modern cloud engineering tools until it looks like something you'd happily call production!

Dodging Docker Hub Limits with Artifactory

Kat Cosgrove — Wed, 03 Feb 2021 21:16:54 +0000

There’s been some chatter about the partnership between JFrog and Docker, most of it around this bit here:

JFrog Artifactory as a pull-through cache for Docker Hub

By taking advantage of JFrog Artifactory as a local container cache, coupled with limitless Docker Hub access, enterprise developers will reap a variety of benefits, including:

A boost to developer productivity. Enterprise developers will get faster and more responsive access to containers by standing up JFrog Artifactory as their local container cache with no Docker Hub limitations.

Optimized usage of IT resources. By caching Docker images locally on JFrog Artifactory, external traffic on the developers’ networks is reduced, lowering their companies’ bandwidth consumption. In addition, it lessens the load on Docker Hub’s infrastructure, which benefits the overall DevOps community.

It sounds cool. Useful. Punchy. Everyone loves words like “productivity,” “faster,” and “optimized.” If you like the sound of all of this but aren’t super sure what it actually means or how to implement it, I’m here to help!

What’s a Docker registry pull-through cache?

In the context of Docker, a pull-through cache is a registry that acts as an intermediary between you and Docker Hub. A pull-through cache is self-populating; that is, if you attempt to pull an image from the cache that it doesn’t already have, it will go get it from Docker Hub for you, cache a copy, and then hand it over to you. From then on, all requests for that image come from your cache first, rather than straight from Docker Hub.

Neat. Why would I want to do that though?

Speed and resource management. If you have a bunch of different things pulling that image, going out to Docker Hub every time is a lot of internet traffic. With a local pull-through cache, the request doesn’t have to go out to Docker Hub unless you try to pull an image tag that your cache doesn’t have. There’s also a security aspect to this, since you’re now in control of your dependencies.

Since you are now only pulling an image from Docker Hub when you request something that doesn’t exist in your pull-through cache, it will take much longer for you to hit Docker Hub’s rate limits, whether you’re pulling anonymously or have your pull-through cache configured to authenticate with Docker Hub.

What does JFrog Artifactory have to do with this?

Simply using a pull-through cache, regardless of the registry you use to do it, means you get more mileage out of Docker Hub’s rate limit on image pulls. However, because of our fancy new partnership with Docker, using Artifactory as your pull-through cache means you’re completely exempt from Docker Hub’s rate limiting. Pull to your heart’s content. Or your CI’s content, whatever.

Rad. How do I set that up?

In Artifactory, you’ll need Docker repositories. The Quick Setup wizard will automatically create three of them for you: docker-local (for your images), docker-remote (the pull-through cache), and docker (a virtual repository that kind of acts as an envelope around your local and remote repositories).

To treat Artifactory as a pull-through cache, from here, make your requests against the virtual repository. For example, if I wanted to pull a specific version of Ubuntu, I would do this:

docker pull katc.jfrog.io/docker/ubuntu:16:04

To break that down a bit, katc.jfrog.io is the address of my JFrog platform, docker is the name of my virtual repository, ubuntu is the image I want, and 16.04 is the specific tag.

When that command runs, it will first check my Docker repositories in Artifactory for a matching image and tag. If it isn’t there, Artifactory will go out and pull it from Docker Hub for me, cache it in the docker-remote repository, then serve it to me. No additional configuration is required for this behavior, and because we’ve partnered with Docker, the request won’t count against your rate limit whether your remote repository is authenticated with Docker Hub or not. Cool, right? This partnership does also apply to the cloud free tier of the JFrog Platform -- sign up here to see for yourself!

DevOps 101: Package Management

Kat Cosgrove — Wed, 25 Nov 2020 16:14:39 +0000

When you’re new to an industry, you encounter a lot of new concepts. This can make it really difficult to get your feet underneath you on an unfamiliar landscape, especially for junior engineers. What’s all this jargon? What does DevOps really mean? What’s all this software? Is DevOps a methodology, or a toolset? Is any of this actually going to make my life easier, or is it just a bunch of industry buzzwords? A lot of the documentation out there assumes you already have additional context and experience, or are proficient in some related tooling, and that doesn’t exactly make it easy to learn. DevOps has a ton of jargon in it, though. We're absolutely swimming in abbreviations and abstractions, and sometimes it's difficult to define a term satisfactorily without needing to define three more for context. It’s like running into a brick wall.

Here, I'll explain package managers.

What even is that?

From Wikipedia, a package manager or package-management system is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer's operating system in a consistent manner.

If you’re a developer, you have probably already used one of these. For instance, if you’re writing Node.js, you’re probably using NPM. That’s a package manager. If you’re using Linux and you’ve ever installed something with apt-get, APT is a package manager. In plain English, a package manager is what handles the heavy lifting involved in installing something (including that thing’s dependencies), validating that it is what it says it is (to an extent, by comparing checksums), keeping track of versions, upgrades, and uninstalling. This is accomplished in part through the use of a large amount of metadata, defining characteristics about a particular package, alongside the actual application binary. Modern package managers are why you can reliably run pip install Django=-1.3.3 or whatever and be pretty sure you are getting version 1.3.3 of Django, or just pip install Django and take whatever is the most recent version.

The Before Times

We didn’t always have this, though. The earliest versions of what you could call a package manager are from the mid to late 90s, with Debian introducing dpkg in 1994 and RedHat’s RPM in 1997. Before then, and until package management really took off, we had to do a lot of things manually and we had way less information about the things we were installing.

You would get a compressed directory, usually in the form of a .tar file. Decompress that, and you would find a readme file with instructions to follow. Some kind of config script would be there which, when run, would tell your C compiler what to do, where new binaries should go, what application dependencies to look for and where, etc. If anything went wrong, it would exit and you would have to go install some more dependencies. If it found everything it needed and the config script completed, it would spit out a Makefile. Run the make command and, if everything compiles correctly, run make install to finally install the application. Updates were just as involved, if not more so. Obviously, this is a time-consuming process. Imagine having to do that for every piece of software on your computer, and every piece of software required to run all of it. A lot of things could go wrong on your journey from downloading a .tar file to actually getting the thing installed.

Understandably, everyone thought this was exhausting and the release of the first package managers for Linux was A Big Deal. It immediately changed computing and application development for the better, forever. Those early package managers pretty much just gave you install, update, and uninstall, but over time, package managers have bundled all of that work and more into simpler commands and encoded extra information alongside the application binary and its dependencies, like version numbers, checksums, dependency graphs, and more. As the popularity of package managers for Linux grew, so did the demand for something similar for other languages. Thus, a whole series of package managers for other languages were born, from CPAN for Perl to PyPi for Python to Cargo for Rust, all with the goal of making it easier to distribute and use software.

This introduced a whole new problem, though. Applications are easier to install, update, manage, and remove, so we start building more complex applications, and with Continuous Integration becoming popular in the 90s, we start releasing more often, too. One organization might be using multiple languages, and we also start caring about security. This is where binary repository managers enter the playing field.

What’s a binary repository manager?

A binary repository manager is there to manage all of those binaries we now have into a system of repositories. Some of them support just one or two package types for very specialized applications, but the one I’ll talk about supports more. This isn’t the same thing as source control, which is where your code lives, but more of an extension of it. While your code might live in a repository on GitHub or BitBucket or whatever, the result of that code being built or compiled -- your artifacts -- live in a binary repository manager like JFrog Artifactory. For DevOps to really work, this is an important part of the equation. We’re delivering updates far more often now, and we need a way to organize our build artifacts in a sensible way so they’re easier for our other tools, like our CI/CD system, to interact with them and ultimately deploy the updates to the user. Without one, it’s much more difficult to track version numbers, control access, promote builds from testing to production, collect metadata, or detect security problems.

Try to imagine writing code and keeping it organized without tools like Git and GitHub. That sounds miserable, right? It’s the same for binaries and a binary repository manager, especially on teams that have multiple languages in one house. If you use a binary repository manager like Artifactory, you can store your Go, JavaScript, Docker images, Python, and Java binaries all in one tool, plus 22 other package types.

Artifactory breaks up the repositories for each package type into three different classes: Local, remote, and virtual. Local repositories are what they sound like: repositories for your build artifacts resulting from local code that exists on your machine. Remote repositories are also fairly self-explanatory; they contain remote build artifacts, like your project's dependencies. This functions sort of like a cache, so that after the first download, your project pulls its dependencies from the associated remote repository rather than from NPM or PyPi or whatever. If you are using Docker, this is particularly helpful, since it limits the number of pulls you need to make against Docker Hub and you won't hit their new limit for pulls from anonymous users as quickly. Virtual repositories are a little weirder -- they create a kind of envelope around the local and remote repositories for your project, and this is what you'll be interacting with most frequently.

A lot of headaches are saved here, from an organizational standpoint. Things get released faster because things are more organized and easier to integrate with a CI/CD solution, and there’s no jumping around between a dozen tools that all do the same thing but for different package types. This improvement alone decreases the likelihood of a bad build making it out into the wild, because we humans are really bad at repetitive tasks, and this takes over a lot of repetition for us.

Cool cool cool, how do I try using one of these things? Sounds enterprisey.

It's definitely a thing that's more beneficial to whole companies or dev teams than for an individual person on a side project, but it's still good to learn. There are a handful available, but Artifactory is free up to a certain amount of storage and data transfer. Try it here on the cloud provider of your choice. It also comes with Xray, a vulnerability detection tool, so throw in a CI/CD system and you're getting pretty close to an end-to-end DevOps solution to play with and learn on. If you don't quite understand CI/CD (or you just need some recommendations for where to start!), check out the previous article in this series: DevOps 101: CI/CD

Summarize this for me, high school essay style.

In conclusion, package managers are a set of tools that make it easier for you to install, use, update, and remove applications. They go further than just automating the steps we used to have to take manually, with config scripts and Makefiles, by also installing your dependencies and managing a bunch of extra metadata we didn’t have clear access to before. The next leap from there is the use of a binary repository as an extension of our source code repositories, to manage all of these binaries and build artifacts produced by our package managers. Doing so gives us more insight into what’s going on with our builds, a simpler way to control who has access to what, and a place to keep all of our build artifacts regardless of the languages or tools involved in our applications. This is a boon to your developers from a sanity and organization standpoint, to your users in the form of faster updates, and to your legal team in the form of faster detection of critical vulnerabilities. The invention of the package manager is possibly one of the most important innovations in computing in decades, and a universal binary repository manager is one of the most important parts of a functional DevOps pipeline.

I hope I’ve helped you understand what package management is and what it does for you. If you’re still confused, that’s okay too -- there's a lot going on in this space. Stay tuned for more!

Oh, and if you have specific requests, reach out to me on Twitter! My DMs are open.

AWS CodeArtifact: Lol, who needs build integrations anyway

Kat Cosgrove — Fri, 19 Jun 2020 21:41:01 +0000

This article is part of a series written by JFrog's Developer Advocates. The index can be found here:

JFrog Artifactory vs AWS CodeArtifact: Comparison in 10-ish parts

JBaruch 🎩 for JFrog ・ Jun 19 '20

#jfrog #artifactory #aws #codeartifact

AWS has released CodeArtifact, an artifact repository manager, and as an employee of JFrog I’ve obviously got some opinions. My boss said I didn’t have to censor snark for this, so before I get to the point, I just want to say this:

I know naming things is hard. There are a million jokes about the difficulties of naming things in computing. AWS already has CodeCommit, CodeBuild, CodeDeploy, CodePipeline, and CodeStar, so maybe they felt backed into a corner because a convention had been established. But, y’all… there is already a completely unrelated product called AWS Artifact. Obviously this doesn’t impact the quality or usability of the tool and it’s just another example of Amazon’s tendency to forget about the existence of its own tools (understandable, there are SO MANY of them), but it’s confusing and made me chuckle so I had to bring it up.

Anyway, onwards to the point.

Do you like vendor lock-in? So does Amazon. As far as I can tell, CodeArtifact didn’t ship with any integrations for the CI/CD tools you’re probably already using and there’s no word on whether or not integrations will eventually be added. If you want to use CodeArtifact, but you’re already committed to Jenkins and moving everything over would be a bear, you’re in an awkward spot. Yes, there’s a CLI. You can still kludge something together. That’s extra work for something that should be easy, though. Your alternative is to migrate everything over to their ecosystem so you can use AWS CodePipeline, the literal only thing it integrates directly with. Either way, it’s inconvenient at best and a huge pain in the ass at worst. Maybe it’s fine if you’re starting a new project from scratch and are okay with living in Jeff’s house forever, but it’s kind of… short-sighted.

Yes, JFrog has its own CI/CD tool. It’s called Pipelines, and its integration with Artifactory is really, really smooth. We’ve got a CLI and a REST API, too. However, we also have direct integrations for the tools you might already be using -- Jenkins, TeamCity, Bamboo, Azure DevOps, GitHub Actions, and BitBucket Pipelines -- because we want you to be able to use Artifactory and everything it offers regardless of whatever else is going on in your ecosystem. Like, it’d be cool if you used Pipelines instead, but we’re not going to make it difficult to keep using the tool you already like if you don’t want to switch, and we’re not going to cut you off from some of the best features of Artifactory as some kind of punishment for choosing to stick with Jenkins.

What’s that feature I’m talking about? Build-info.

Build-info is all the information collected by the build agent, which includes details about the build. The build-info includes a list of project modules, artifacts, dependencies, environment variables and more. When using one of the JFrog clients to build the code, the client can collect the build-info and publish it to Artifactory. When the build-info is published to Artifactory, all the published details become visible in the Artifactory UI.

You get access to all of that data right in Artifactory as long as you’re using JFrog Pipelines, the JFrog CLI, or one of the six other CI/CD tools we provide integrations for. Which is six more options than AWS is giving you with CodeArtifact, and a whole lot more context for what’s going on with your builds.

Return to the index:

JFrog Artifactory vs AWS CodeArtifact: Comparison in 10-ish parts

JBaruch 🎩 for JFrog ・ Jun 19 '20

#jfrog #artifactory #aws #codeartifact

DevOps 101: CI/CD

Kat Cosgrove — Thu, 16 Apr 2020 21:55:01 +0000

When you’re new to an industry, you encounter a lot of new concepts. This can make it really difficult to get your feet underneath you on an unfamiliar landscape, especially for junior engineers. In this series, I’ll cover tools and terminology common to the DevOps space, plus the occasional newbie-friendly tutorial for emerging or established technologies. If you have a request or suggestion, let me know!

Today, I’ll break down CI/CD.

What does that even stand for?

The CI stands for Continuous Integration, and the CD can stand for either Continuous Delivery or Continuous Deployment. Yes, they do mean different things. I know, I know, this sounds like it’s already getting complicated, but I promise it’s not so bad.

What does Continuous Integration mean?

Practicing Continuous Integration means merging all developers’ working codebase with the source, multiple times a day. Doing this requires a series of automated build and unit tests to ensure none of the proposed changes cause problems, but the result is that bugs and integration issues are discovered much earlier in the development process. It also forces engineers to write code that’s more modular, making it easier to support later on.

Continuous Integration has been a thing since the early 90s. Though it hasn’t always been called that, and some of the implementation has changed, the spirit remains the same: merge changes into source in smaller but more frequent increments, test that the project still builds and runs with those changes, and make sure your engineers are all working on the most recent version of the source. Do this, and you won’t end up with a ton of merge conflicts or surprise problems when it comes time to build.

Okay, what’s the difference between Continuous Delivery and Continuous Deployment?

Continuous Delivery means what it says on the box: your software updates are continuously delivered. In concert with Continuous Integration, this means you should have the ability to deploy a new build very rapidly because you’ve automated some quality gates that would otherwise need to be performed manually, like building and testing. That reduction in manual labor means you get to release a bunch of small changes, rather than one huge update every couple of months. Since you’re now making smaller, incremental changes, you can also be more confident that your release isn’t going to break when you deploy to your users.

Continuous Deployment is similar, but it goes one step further -- deployment is automated, too. In Continuous Delivery, there is still a manual quality gate involved before an update is out in the wild. This is a controversial step for some, and requires a lot of trust in your system, but I’m personally a huge fan of it. For a modern DevOps pipeline (and thus, you) to be as efficient as possible, human involvement has to be removed wherever possible. I say this a lot, but we are really, really bad at repetitive tasks -- we get bored, we get distracted, and we’re SLOW. Write good, comprehensive tests and automate everything you can, then accept that you absolutely are going to deploy a bad update eventually, whether a human is involved in clicking the Big Green Button or not.

Wait, so what’s CI/CD? What does it get me? Go high-level.

CI/CD is just the term for the marriage of these concepts. It’s an important part of DevOps, since automation and efficiency is what we’re all about, and one doesn’t really work in the context of DevOps without the other.

Implementing CI/CD practices gets you a faster, more reliable release cycle. You can add new features or bug fixes way, way faster, since you know your engineers are all working from the most recent source, you know there are unit and integration tests and you know it builds. There aren’t a bunch of manual steps involved for the engineers or QA or whoever else you might have managing quality gates; instead, somebody pushes code or opens a pull request and those steps are taken care of by your CI/CD tooling.

How does this actually work?

To start, you need a CI/CD tool. This is what’s going to automate a bunch of manual processes for you. It does take some time to set up at the start of a new project, but personally, I’m the flavor of lazy where I’m willing to spend extra time at the beginning to make sure I don’t have to do a bunch of repetitive, manual tasks every single time I push code later on. The specifics of configuring any of these tools varies, so check the documentation for your chosen tool, but broadly they all work the same way:

You set something as a trigger, like telling it to watch your source repository for a commit or a merge. You then configure a series of steps, each with pass/fail conditions, like telling it how to run your unit tests, build, scan for vulnerabilities, or deploy your application. With a sufficiently detailed CI/CD pipeline, you don’t have to do anything but write code and push it -- the system handles everything else for you. It’s great.

Word, sounds fantastic, sign me up.

If you’re already using some JFrog tools like Artifactory or Xray, it makes sense to stay in the same ecosystem and give Pipelines a try. That way, everything is accessible from one UI. Pipelines integrates with most other DevOps tools and handles a ton of actions natively, so it minimizes the possibility of ending up with the infrastructure version of Frankenstein’s Monster. Configuration is just YAML. If you want to try Pipelines, along with Artifactory and XRay, there’s a cloud-hosted trial here.

For freemium solutions (free usage is limited, but generally fine for personal projects or small stuff), I really like CircleCI and TravisCI. Both are cloud solutions with a good number of built-in integrations, both are pretty easy to configure, and both have support for a wide range of popular programming languages. There are differing limitations in what you can use as your version control and build environments.

Summarize this like you’re writing an essay in high school.

In conclusion, CI/CD is a combination of methodology and tooling with the goal of increasing your speed and efficiency as a developer by automating tasks like building, testing, and deploying so that you can do all of those things more often. The benefit to you is more frequent software releases, earlier detection of bugs, and bad releases making it to production less often. There are several tools out there that help you accomplish this goal, from freemium tools like TravisCI or CircleCI to enterprise-scale tools with additional features like JFrog Pipelines.

I hope I’ve helped you understand what CI/CD is and what it does for you. If you’re still confused, that’s okay too -- this can be kind of a “big” problem and some of it is still in flux. Feel free to get in touch through the comments or on Twitter at @Dixie3Flatline if you have more questions. Stay tuned for the next article in this series, and let me know if you have a request!

Easy Automatic Vulnerability Detection in the JFrog Platform

Kat Cosgrove — Wed, 04 Mar 2020 02:38:04 +0000

JFrog has released a Unified DevOps Platform, putting the entire product suite under one roof. You’ll have a comprehensive view of what’s going on with your artifacts, more clear permissions control, quicker access to your build pipelines, and my personal favorite: vulnerability scanning baked right in. Let me show you around!

Looks pretty clean, right? One login gets you access to the entire platform. Your dashboard provides some high-level information on the status of your services, plus usage trends across package types. On the left, you have direct access to your JFrog services: Artifactory, Distribution, Pipelines, and the one I’ll be talking about: Xray.

JFrog Xray is a universal component DevSecOps tool that protects you from security vulnerabilities and license compliance issues that may be hidden in your software. Let’s take a look at the Security and Compliance tab to see how we can automatically trigger warnings, fail a build, or even block downloads based on a threshold you set.

JFrog Xray functions on a system of Policies and Watches. Policies allow us to define security and license compliance behaviors specific to your organization. Once they are defined, they are enforced by applying them to Watches.

In this case, the Policy is triggered when a vulnerability categorized as “high” is found. Xray’s scan is recursive, so it looks at the smallest components affecting your packages. Its vulnerability database is multi-faceted, using public information, its own private database, and Risk Based Security’s VulnDB. If the vulnerability is known, it’ll find it. This information is presented to you in a couple of places, but the easiest to digest is in the form of an Impact Analysis Graph, which I’ll show you later.

Once a Policy has been defined, you must assign it to a Watch. A Watch is a collection of repositories, builds, and release bundles that Xray should be scanning. I like to organize these by project, but it’s up to you how to handle that. Policies can be assigned to multiple Watches, so there’s no need to define a ton of them.

Now that you have a Policy and it’s assigned to a Watch, you’ll start to see Xray data in a few different places elsewhere within the Platform. The fact that everything is now organized within a single pane of glass makes staying on top of any potential security or license issues really, really easy. Let’s hop over to the Artifactory tab and take a look at what’s going on with some of our builds.

We can already see from here that there are some problems. Each of these builds has an Xray status of “high” -- something risky is happening with some component here. We need more information, though. What’s the actual vulnerability? Is this something we really need to act on? What specific component is causing the problem, and what version do we need to update it to if we want to resolve the vulnerability? Click on the build and find out.

Under the Xray Data tab for the most recent build, we get a TON of information. Everything that violates our set Security and License Policies is here. We get some high-level information immediately: a summary of the vulnerability, the severity, which Policy was triggered, the component in question, the affected versions, and the fix versions, if any. Click on one of them, and you get even more information.

On the right is the Impact Analysis Graph I mentioned earlier. Xray has peeled open our package and drilled down through the layers to find an issue with Jackson. To assist in resolving the vulnerability, we also get a description of the problem, references, CVEs, and more. You know exactly which component is causing a problem, and very clearly what the problem is. There’s no guesswork involved in fixing it, and this information is readily available to you no matter where you are in the Platform.

Collecting all of JFrog’s products into one platform gives your team a clear view into every aspect of your artifact’s lifecycle. Everything you want to know is easily within reach, in just a click or two. If you really care about automating away the trouble of dealing with security vulnerabilities and license violations, Xray is pretty compelling. Stay tuned for walkthroughs of the rest of the new JFrog Platform, and don’t hesitate to reach out to me if you have questions!

DevOps 101: Container Registries

Kat Cosgrove — Mon, 16 Dec 2019 20:42:45 +0000

When you’re new to an industry, you encounter a lot of new concepts. This can make it really difficult to get your feet underneath you on an unfamiliar landscape, especially for junior engineers. In this series, I’ll cover tools and terminology common to the DevOps space, plus the occasional newbie-friendly tutorial for emerging or established technologies. If you have a request or suggestion, let me know!

Today, I’ll break down container registries.

What’s a container registry?

A container registry is a tool for hosting, versioning, and distributing container images within repositories. They can be public (like Docker Hub) or private (like JFrog Container Registry). If you’ve used Docker, you’ve pulled an image from a container registry before. There are several out there, both public and private, free and paid, but I’ll mostly be focusing on JFrog Container Registry since it’s free, can run locally or on a cloud provider, and provides a few other nice-to-have features that other services don’t, like a Helm repository and advanced metadata.

What’s the difference between public and private container registries?

Public registries allow anyone to create a repository and host container images there for other people to use. Getting started with a public registry is fast, since you don’t have to deal with any of the infrastructure. However, public registries do sacrifice security in exchange for that ease of use. If using standard images works for your project, and you aren’t concerned about security, a public registry might be the right choice for you.

Private registries are hosted by the organization that will be using them. JFrog Container Registry can be hosted either locally on your hardware (on-prem) or with a cloud provider like Google, Amazon, or Microsoft. Hosting your own container registry gives you fine-grained access control and the ability to define your own security policies. Since the organization is in complete control of the images, repositories, and access control for those things, they can be certain that container images are what they say they are and that no one has access to things they shouldn’t. In the case of JFrog Container Registry, you also have the option to organize your images into various local, remote, generic, and virtual repositories, depending on your needs. For larger organizations, this additional reliability and security is worth the effort involved in setting up the infrastructure itself.

What are all these different types of repositories?

Local repositories are exactly what they sound like -- locally-managed repositories into which you deploy artifacts. Pretty simple.

Remote repositories are used as caching proxies for remotely-managed repositories. These are a little more complicated, as you must define the behavior of the caching and proxying. It is not the same as a mirror; an artifact only exists in a remote repository once it is requested by something else, according to the configuration you define.

Generic repositories have no specific package type associated with them. Any time you create a repository, you must define a package type. JFrog Container Registry uses this to do some behind-the-scenes work for you, indexing the artifacts and calculating metadata for your packages to optimize performance. Generic repositories have no package type, so you can put pretty much whatever you want in here, but you do lose out on some features you would see otherwise.

Virtual repositories bundle up a collection of various remote and local repositories with the same package type into a single URL.

Neat. How do I try one of these things?

The quickest and easiest way is with the cloud hosted version of JFrog Container Registry, since it’s still free-ish (you pay for usage on the cloud provider over a certain amount), mostly configures itself, and doesn’t require you to set up a reverse proxy. Sign up here with your preferred flavor of cloud provider and follow along with the automated setup wizard to configure some initial options and create your default repositories.

Once that’s done, it’s really easy to interact with. You can use the native Docker client, just like you would with Docker Hub. Just log in:

docker login ${server-name}-{repo-name}.jfrog.io

To push an image, give it a tag and push:

docker tag ${server-name}-{repo-name}.jfrog.io/
docker push ${server-name}-{repo-name}.jfrog.io/

Pulling an image is easy, too:

docker pull ${server-name}-{repo-name}.jfrog.io/

Setup and usage is a bit more complicated with the on-prem version, but fortunately, the official documentation is pretty explicit.

Summarize this for me.

If your organization or team is getting serious about cloud native development, a private container registry is probably something you’re going to work with at some point. The goal of a private registry is to provide you with a place to organize, version, and deploy your container images with increased security options and better integration into your existing CI/CD workflows when compared to a public registry, whether it’s hosted on a local server or out on the cloud. JFrog Container Registry is free on-prem or freemium on any of the three major cloud providers and gives you everything the other services on the market do, plus some extra goodies like a Helm repository and rich metadata.

I hope I’ve helped you understand what this tool does for you and given you a reasonable pushing-off point into using one. If you’re still confused, that’s okay too -- feel free to get in touch if you have more questions. Stay tuned for the next article in this series, and let me know if you have a request for deep dives or demos!