DEV Community

Cover image for AWS CodeArtifact - the long-awaited feature
Paweł Piwosz
Paweł Piwosz

Posted on • Edited on

AWS CodeArtifact - the long-awaited feature

#aws #cicd #devops

The background

I have to admit, I am not a fan of AWS Codepipeline. Even now, after a few project where I heavily used CodePipeline, I am still not big fan. For me it is rather simple solution. Of course it can do the job, but in my private world, I used to use TravisCI before and now… Azure DevOps for AWS deployments.

And one of great advantages of Azure DevOps is Artifacts, where you can store and publish your packages. Something what we missed in AWS Codepipeline for long time. Until now.

AWS CodeArtifact

Not long time ago AWS announced this service. Now it is official. And it is a great news, indeed. Until today we had to use workarounds (let’s call it this way). S3 buckets, or build the artifact solution by our own.

This is not longer needed. AWS CodeArtifact is taking a great place in the CodePipeline services. First 2G of stored packages is free (monthly), also first 100K requests are free.

Let’s get started

AWS CodeArtifacts splash screen

What is important to understand, it is the domain concept. More explanation is in the Steve Roberts’ blog, here https://aws.amazon.com/blogs/aws/software-package-management-with-aws-codeartifact/. Important to remember is that Domain aggregates all repositories. So, we are consuming the package from the repository, or even multiple repositories, but in fact, this package is stored only once across the Domain. This saves space and possible costs.

Domain

So, let’s create the Domain! We can do it from AWS GUI, but this time, I will work with CLI. Of course, I assume you know how to configure CLI, if not, there is plenty of articles how to do it. Remember, you need the newest version of awscli, I recommend to use version 2.

AWSCLI version 2. Notice, in my example it is Powershell version

Let’s check our domains now:

aws codeartifact list-domains
{
    "domains": []
}
Enter fullscreen mode Exit fullscreen mode

You should receive empty JSON.

So, it is time to create the first Domain. The process is very easy

aws codeartifact create-domain --domain mytestdomain 
{
    "domain": {
        "name": "mytestdomain",
        "owner": "012345678",
        "arn": "arn:aws:codeartifact:eu-west-1:0123456789:domain/mytestdomain",
        "status": "Active",
        "createdTime": "2020–06–28T10:46:50.193000+02:00",
        "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11",
        "repositoryCount": 0,
        "assetSizeBytes": 0
    }
}
Enter fullscreen mode Exit fullscreen mode

And… That’s it! Let’s check it again

aws codeartifact list-domains 
{
    "domains": [
    {
        "name": "mytestdomain",
        "owner": "0123456789",
        "status": "Active",
        "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11"
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode 
aws codeartifact describe-domain --domain mytestdomain 
{
    "domain": {
    "name": "mytestdomain",
    "owner": "0123456789",
    "arn": "arn:aws:codeartifact:eu-west-1: 0123456789:domain/mytestdomain",
    "status": "Active",
    "createdTime": "2020–06–28T10:46:50.193000+02:00",
    "encryptionKey": "arn:aws:kms:eu-west-1:0123456789:key/aaaa1111-aa11–1234–1234-aa11aa11aa11",
    "repositoryCount": 0,
    "assetSizeBytes": 0
    }
}
Enter fullscreen mode Exit fullscreen mode

Repository

Domain is already created, so we can create the repository.

But first, we need (again) to look on the construction here. We are in the Domain. Now, we can create Repository, and then Upstream Repository. By this approach you can separate your project repositories (by upstreams) and still be able to access all of them with one endpoint.

So, let’s create the repository first.

aws codeartifact create-repository --domain mytestdomain --repository "mymainrepo" --description "Main repository" 
{
    "repository": 
    {
        "name": "mymainrepo",
        "administratorAccount": "01234567890",
        "domainName": "mytestdomain",
        "domainOwner": "01234567890",
        "arn": "arn:aws:codeartifact:eu-west-1: 01234567890:repository/mytestdomain/mymainrepo",
        "description": "Main repository",
        "upstreams": [],
        "externalConnections": []
    }
}
Enter fullscreen mode Exit fullscreen mode

Of course, we can look on it, using:

aws codeartifact list-repositories

aws codeartifact describe-repository --domain mytestdomain --repository mymainrepo
Enter fullscreen mode Exit fullscreen mode

In order to login to the repository, you can follow the instruction available in the GUI console

AWS is giving very straightforward connection instruction

The connection command is simple (but long)

aws codeartifact login --tool pip --repository mymainrepo --domain mytestdomain --domain-owner 01234567890

Successfully logged in to codeartifact for pip.
Enter fullscreen mode Exit fullscreen mode

For configuring pip itself, we need to run two additional commands. First to get authorization token:

aws codeartifact get-authorization-token --domain mytestdomain --domain-owner 0123456789 --query authorizationToken --output text
Enter fullscreen mode Exit fullscreen mode

second, to use it in our pip config

pip config set global.index-url [https://aws:HEREGOESTHETOKEN@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/](https://aws:HEREGOESTHETOKEN@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/)

Writing to C:\Users\pawel\AppData\Roaming\pip\pip.ini
Enter fullscreen mode Exit fullscreen mode

As you can see, I’m using Powershell here.

Now we can check, if our pip is updated, by running pip config list. You should be able to see your token there.

Upstream repository

Let us follow the good practice, and now it is the time to create upstream repositories. We will add npm and pip in one shot,

aws codeartifact update-repository --repository mymainrepo --domain mytestdomain --domain-owner 0123456789 --upstreams repositoryName=pypi-store repositoryName=npm-store
{
    "repository": {
        "name": "mymainrepo",
        "administratorAccount": "0123456789",
        "domainName": "mytestdomain",
        "domainOwner": "0123456789",
        "arn": "arn:aws:codeartifact:eu-west-1:0123456789:repository/mytestdomain/mymainrepo",
        "description": "Main repository",
        "upstreams": [
            {
                "repositoryName": "pypi-store"
            },
            {
                "repositoryName": "npm-store"
            }
        ],
        "externalConnections": []
    }
}
Enter fullscreen mode Exit fullscreen mode

Thanks to that, we are able to work with one endpoint for all of ours npm and pip packages. If it works for you — it is up to you.

Be aware of search order here. If you are requesting the package, your request is going through the upstreams in the create-repository order (or update — repository). If you are curious, what is the order, simply run describe-repository, and you will see upstreams in the search order.

External connections

When we finish our repos, we can create external connections to them. Let’s do it for both of our repos:

For pip:

aws codeartifact associate-external-connection --domain mytestdomain --domain-owner 0123456789 --repository pypi-store --external-connection "public:pypi"
Enter fullscreen mode Exit fullscreen mode

And for npm:

aws codeartifact associate-external-connection --domain mytestdomain --domain-owner 0123456789 --repository npm-store --external-connection “public:npmjs”
Enter fullscreen mode Exit fullscreen mode

Although, it is possible you will receive an error, as this connection already exists.

Let’s test our new shiny repository

So. We are ready to test our repo. We gonna install some pip package. I’ve created empty virtualenv, so we can install for example awscli. What we expect?

What is happening when we want to install package?

Let me go through this schema. We want to install package X. We configured our pip in the way, it is looking to our AWS CodeArtifact first. If the requested package is not there, it will connect to main public one and download it for us (with all dependencies). And then, the package will be installed from our AWS CodeArtifact repo.

Let’s install our package then:

pip install awscli
Looking in indexes: [https://aws:****@mytestdomain-0123456789.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/](https://aws:****@mytestdomain-043144402336.d.codeartifact.eu-west-1.amazonaws.com/pypi/mymainrepo/simple/)
Collecting awscli
Enter fullscreen mode Exit fullscreen mode

You easily can spot, pip is trying to collect the package from our repository. Now, the package is installed, let’s check our repository:

The requested packages are now available from our repository

And voila! We successfully created, configured and used our CodeArtifact service!

Of course, our packages can be published into our repository. AWS CodeBuild and CodeDeploy can use our repository. In the essence, AWS CodePipeline finally has the proper solution for artifacts!

Summary

I really like the AWS CodeArtifact. It is easy to configure, easy to use. The idea with Domains and Upstreams is pretty nice.

The service is fast, reliable and managed by AWS. What else to expect? I strongly recommend CodeArtifact.

Important is, it can be used externally. not only cross-account inside AWS, but also as a proper external packages repository.

Currently it covers Python, Nodejs and Java repositories (edit: and C#), but I’m sure, we will receive more soon!

I wrote this article on Medium about one year ago. Time for migration :)

Top comments (0)

Read next

tomjohnson3 profile image

Application Dependency Mapping: A Real-World Example

Thomas Johnson -

kanav_gathe profile image

Day 2/90: Essential Linux Commands for DevOps 🐧 #90DaysOfDevOps

Kanav Gathe -

dm8ry profile image

Getting Started with MongoDB Compass: A GUI for MongoDB

Dmitry Romanoff -

luqmanbello profile image

Building Production-Grade ECS Anywhere Infrastructure with Custom Capacity Providers

Luqman Bello -