DEV Community: Batel Zohar

Get Unlimited Container Pulls from Docker Hub with JFrog Cloud

Batel Zohar — Tue, 26 Jan 2021 14:26:42 +0000

My previous blog described Docker’s new download rate limit policy that introduced new pull request limitations starting November 1st, 2020. These limitations restrict us to 100 pulls for anonymous users and 200 pulls for authorized users every 6 hours.

Today, I’m excited to share the latest news about the JFrog and Docker partnership that will provide us with an unlimited number of pulls from Docker hub.

As you already know, millions of users all over the world are using Docker and Docker Hub to manage their applications, as part of their development cycle. Thanks to an amazing partnership between JFrog and Docker, now we don’t need to beg for a budget anymore. These limitations will not apply to JFrog Cloud Platform users, including free tier subscriptions.

JFrog will support unlimited pulls from Docker hub out of the box without any additional configurations. JFrog customers will automatically be excluded from Docker hub’s image rate limit and enjoy endless access to Docker Hub.

When we are using Artifactory we’ll continue to get high-performance, automatic updates with new features, and now most importantly the freedom and flexibility to test, try and not worry about the number of downloads.

If you haven’t tried Artifactory yet and are looking for a solution to the Docker Hub limits sign up for our free subscription - we’ve got you covered.

Time to make some order with GoCenter

Batel Zohar — Wed, 16 Dec 2020 12:42:46 +0000

Go is becoming one of the world’s fastest-growing software languages. To keep increasing my skill set as a developer I started learning Go a few months ago. Here is a snapshot of my journey and some insights I learned along the way.

Dependency Management

Learning a new language can be overwhelming so I decided to start with the basics - dependency management. So let’s start from the beginning the management of the dependencies, from version 1.11 Go supports modules, this feature makes dependency version information explicit and easier to maintain.

Go module

A module is a collection of Go packages stored in a file with a go.mod file at its root. The go.mod file defines the module’s module path, which is also the import path used for the root directory, and its dependency requirements, which are the other modules needed for a successful build. Each dependency requirement is written as a module path and a specific semantic version.

Let’s start with a simple example: hello world. In this example the go.mod file will look like the following:

module "rsc.io/hello"

require "rsc.io/quote" v1.5.1

After completing a simple go run and go build we now have a hello world example which is basic, but let’s try to make it a bit more complicated by adding yaml support. To do this we will use the following commands (I found that version 2.2.7 is recommended) so let’s give it a go:

gopkg.in/yaml.v2 v2.2.7

Then I figured that I used a vulnerable package and I found GoCenterthat provided me an amazing way to better understand Go packages. GoCenter has the following features:

Proxy my dependencies

First we can use GoCenter as a GOPROXY and we will redirect all module download requests to GoCenter which can be faster than directly from the VCS.

To change the GoProxy path just use the following commands:

For mac and linux:

export GOPROXY=https://gocenter.io

For Windows:

'''set GOPROXY=https://gocenter.io'''

For powershell:

'''$env:GOPROXY=https://gocenter.io'''

Protect your binaries

I’ve tried to learn a bit more about the yaml packages and this is how it looks on GoCenter:

First I found out that my version is vulnerable and contains CVE-2019-11254 like the following:

Also I noticed the feature that scans the dependencies in a go.mod file held by GoCenter and identifies every vulnerability. Under the dependencies tab we will get the detailed information about vulnerable components at every level of the dependency tree, once we will click on the orange triangle we will forward to the package and we can check the vulnerability page like the following example of

Learn more about your packages

So I clicked on the versions tab and saw that version 2.2.8 contains a fix and I upgraded to the latest version 2.4.0 now seems like they added some documentation and examples:

I love metrics. GoCenter’s metrics are colorful and provide a lot of information in a great visual way so I can easily see that there are a lot of downloads of the packages and 37 Contributors:

Advanced mode private GOPROXY

Another advantage for developers is the ability to improve our resolution tie by integrating our JFrog Artifactory server and create our Go private repository. We want to create a private Go repository to make sure that we are pulling directly from a virtual repository that contains a remote repository that points to GoCenter and our local repository with our project. A benefit of this method is that we don’t need to manage Artifactory we can just use the SaaS version which is free and limited.

Conclusion

To sum it all up, as I learn to write in Go I will continue to use GoCenter as a proxy for my dependencies, vulnerability scanning of my binaries, version control of my packages, beautiful metrics to give me a great visualization of the data

Docker new Download Rate Limits

Batel Zohar — Fri, 23 Oct 2020 10:31:44 +0000

The new Docker announcement could be a bit confusing, but in this blog post, I’ll try to summarize it and make it simpler to understand. On November 1st, Docker is planning to add a new subscription level, and here’s how this may affect us.

The Problem

There are two main issues Docker users will now be facing: new pull request limitations, and the image retention policy.

Pull request limitations

This new limitation means that we can create 100 pulls for anonymous users and 200 pulls for authorized users every 6 hours.

To better understand why rate limits were introduced, Docker found that most Docker users pulled images at a rate you would expect for normal workflows. However, there is an outsized impact from a small number of anonymous users. For example, roughly 30% of all downloads on Hub come from only 1% of our anonymous users.

The challenge is when pulling an existing image. Even if you don’t download the layers, this pull request will still be counted.

From Scaling Docker to Serve Millions More Developers: Network Egress

Image retention policy

Images stored in free Docker Hub repositories that have not had their manifest pushed or pulled in the last 6 months, will be removed at the mid of 2021. This policy does not apply to images stored by paid Docker Hub subscription accounts, Docker verified publishers or official Docker Images.

Let's take the following example of a free subscription user who pushed a tagged image called "batelt/froggy:v1" to Docker Hub on Oct 21, 2019. If this tagged image was never pulled since it was pushed, it will be considered inactive by mid 2021 when the new policy takes effect. The image and any tag pointing to it will be subject to deletion.

According to their wiki, Docker will also be providing tooling, in the form of a UI and APIs, that will allow users to easily manage their images.

The Solution

My favorite solution is to use JFrog Artifactory which is an artifact repository manager. This will allow us to store and protect our Docker images within a private Docker registry, keeping them in our cache using a remote repository,, reducing our requests to Docker hub and using our local images kept in cache as shown in the following diagram.

Why should we use Artifactory and how it works

JFrog provides us with the ability to host our own secure private Docker registries and proxy external Docker registries. It even provides us with a smart checksum-based storage.
storage, which utilizes storage to maximum potential.

Choose Artifactory version (Cloud or On-prem)
Create Docker repositories (local remote and virtual)
Configure repository advanced configuration

Artifactory version

If you don’t want to manage Artifactory you can just use the SaaS version which is free and limited. Or you can use the JFrog container registry that supports Docker and Helm repositories

Create a Docker repository

Configure repository advanced configuration

Now we can easily Configure repository advanced options like deciding how long before Artifactory checks for a newer version of a requested artifact in a remote repository and use our local caching so we will save our docker requests:

Learn more about Artifactory Pro that contains 27 different package types like Maven NPM and much more.

How to use Helm to Support Cloud-Native Development

Batel Zohar — Mon, 21 Sep 2020 14:55:42 +0000

One day my boss informed me that we are moving towards developing our software on the cloud. As a developer, this is one of my pet peeves as I hate spending time setting up complicated environments and prefer to focus on writing code and quickly releasing their software. And then I immediately got my first assignment to create a new software product using the CI/CD pipeline on the cloud. The questions that immediately arose were: When? How? And Why do we need it?

Since then I have learned many lessons and would like to share them with you on how to prepare and configure your environment to achieve DevOps automation in the cloud for your containerized secured applications using a CI/CD pipeline in Kubernetes.

Why go to the cloud?

Let’s first look at the main reasons for going to the cloud:

Achieve unlimited scalability:

Setting up your environment on the cloud provides unlimited scalability allowing you to grow according to your needs and is easily achieved by using cloud storage providers (Amazon S3, Google GCS or Microsoft Azure) in your environment.

Hardware costs are no longer an issue:

A huge advantage of going to cloud computing is the decrease in hardware costs whereby you pay only for exactly what you use. Instead of purchasing in-house equipment, hardware needs are now left to the cloud vendor. Adding new hardware every release or every quarter to meet your increasing needs can be very expensive and inconvenient, on the other hand, cloud computing alleviates these issues because resources can be acquired quickly and easily.

Gain redundancy:

And the most important thing you gain is redundancy. A number of cloud storage providers can ensure your data is stored on multiple machines and complies with any regulations that need to be applied (like keeping two copies of all the data in two different regions).

This blog post shows you how to prepare and configure a cloud-based environment to achieve an automated CI/CD pipeline for your containerized secured applications using Kubernetes.

So let’s get started.

Configuring a cloud-based automated CI/CD pipeline

Before you start, prepare the following:

A version control system (such as Git)
Kubernetes to deploy and manage your containerized applications
A CI/CD pipeline tool like Jenkins, TeamCity, Bamboo etc.
A Binary repository manager
A Compliance auditor

Step 1: Setting up your cloud environment

Let's start by creating a Kubernetes cluster. You can easily choose one of the available Kubernetes platforms in the cloud. Then isolate your environment by creating separate environments for development, staging, and production as displayed in the following diagram:

We recommend applying the "shift left” principle, whereby you move back tasks typically performed at later stages to earlier stages in the pipeline. For example, when running security testing or selecting the FOSS license. As mentioned earlier, the aim is to move faster to reduce delivery time while improving the quality of each release. At the same time, you are also faced with increasing pressure to reduce testing, so it means that the developer’s team needs to be integrated with the testing cycle earlier.

Step 2: Configuring the VCS server

When you finish developing the app, proceed to configure the VCS server VCS server in order to trigger the build after a successful pull request.

Step 3: Running the build

Run the build on your CI server (for example Jenkins), to create the application’s Docker image and proceed to run the unit test against the Docker container.

Step 4: Testing the build

After the builds tested, create a Docker image that will be uploaded to your private Docker registry and private helm repository, then run a number of tests against the running Docker container including integration tests.

Step 5: Scanning the build

Approximately three decades ago, Richard Stallman changed the developer’s world forever, by introducing the GNU Project, the first open-source coding project that included requirements for scanning external code. In that case, I recommend using the ChartCenter as our new source for our helm chart, when checking the chart view, in that case let’s talk about our DB we can easily get the chart information for the dependencies like the following:

Afterward, I can check the security report very easily and calculate the risk on this specific version

Step 6: Deploying to development

Now we can proceed to deploy your app to the staging environment and perform additional tests against this cluster.

Add the Chartcenter Helm repository

helm repo add center https://repo.chartcenter.io

Step 7: Deploying to staging

After running a full test cycle in the development environment, deploy the application to the isolated Kubernetes staging cluster, run the staging tests, and proceed to the next step.

Step 8: Deploying to production

Run a set of sanity tests and deploy them to the isolated production cluster. Be ready to perform a fast rollback, if necessary.

It’s time to have some fun!

So you see it ain’t that scary and the steps are out there, so go ahead and develop your next app in the cloud. To make it even easier, follow our 6-step CI/CD pipeline for a simple static website application based on of official nginx Docker image.

Securing your software releases with the JFrog Platform

Batel Zohar — Fri, 01 May 2020 07:23:36 +0000

Continuing from my previous blog post about distributing software releases, there is an additional (just as important) security consideration that you need to take into account.

As you may already know, the Xray service represents our security and compliance service which easily integrates with Artifactory and our CI/CD service for license & security compliance.

Let’s start with licenses. One of the most important responsibilities of an open-source program is to ensure that your organization meets its legal obligations when integrating open source code with proprietary and third-party source code in your products. But we also want to make sure that we don’t have any security vulnerabilities in our code, so we can also create security policies like in the following screenshot:

Let’s explain a bit more about security and license policies. Security policies have two rules for scanning:

Minimal severity - Which is defined on the JFrog vulnerabilities database, in case that an artifact or build contains a vulnerability with the selected severity or higher.

CVSS score - The CVSS score ranges between 1 to 10 to apply to the policy.

Now on license policy, it will define a bit different by the following rules:

Allowed Licenses - To create a whitelist of OSS licenses that may be attached to a component.

Banned Licenses - Create a blacklist of OSS licenses that may not be attached to a component.

Optional Disallow Unknown License - Specifies the wanted behavior for components whose license cannot be determined. When enabling this feature a violation will be triggered if a component with an unknown license is found.

Now after we created a policy let’s add our release bundle to the indexed resources like the following screenshot:

That's it we are good to go. We can add our release bundle as part of the policy and keep our binaries :)

Distributing software with the JFrog Platform

Batel Zohar — Mon, 09 Mar 2020 20:27:05 +0000

Hey! In case you haven’t heard yet, JFrog just released the JFrog Platform. If your world is all about DevOps, this platform gives you an end-to-end solution that combines all JFrog products in one simple place. So, the platform includes:

Universal package management with all major packaging formats, build tools, and CI servers.
Security and Compliance that's fully integrated into the JFrog Platform, providing full trust of your pipeline from code to production.
Radically simplified administrationwith all configurations in one place.
Complete trust in your pipeline
all the way from code to production.
Seamless DevOps experience for on-prem, cloud, hybrid or multi-cloud of your choice.

Let’s take a closer look at how we can safely distribute our software using JFrog Distribution.

JFrog Distribution is a service that lets you provision software releases using immutable release bundlesto many target destinations (edge nodes).

Distribution in 3 Steps

1.Create a release bundle using REST APIor the Query Builder.

2.Secure your release bundle by signing it using a GPG key, ensuring that its contents cannot be edited. This will also ensure consistency of distribution among target instances.
Note: The same GPG key is used by the Artifactory Edge to validate the Release Bundle before it is accepted.

3.Distribute your release bundle. Distribution is responsible for triggering the replication process that happens from the source Artifactory to the edge nodes. First, it replicates the Release Bundle info to each edge node and then initiates the replication process in the source Artifactory.

Check out more information on our website and documentation.

CircleCI integration with container registries

Batel Zohar — Wed, 11 Dec 2019 12:41:38 +0000

CircleCI is a cloud-based continuous integration and delivery platform, It’s free for open-source repositories and supports most languages. From version 2.1, CircleCI supports orbs, that are shareable packages of configuration elements, including jobs, commands, and executors. Therefore we don’t need to reinvent the wheel as we can easily use the artifactory-orb which contains the JFrog CLI to store our build artifacts for future distribution in our Docker Container Registry. In this blog, I will show you how to use CircleCI by using a private JFrog Container Registry to store your builds, artifacts, and build information.

Let’s take a closer look at how this actually works. The following example describes how to easily configure CircleCI to push Docker builds as artifacts to JFrog container registry using orbs.
Step 1: Clone the sample repository
Create a repository in GitHub and clone this sample GitHub project into your git repository.
Step 2: Create the CircleCI configuration files
In this example, we use the artifactory-orbs to download the JFrog CLI to perform thee following under the hood:


    Description: Install the JFrog CLI
Steps:


run:
  command: |
    echo "Checking for existence of CLI"
    set +e
    jfrog -v
    if [ $? -gt 0 ]; then
      echo "Not found, installing latest"
      curl -fL https://getcli.jfrog.io | sh
      chmod a+x jfrog && sudo mv jfrog /usr/local/bin
    else
      echo "CLI exists, skipping install"
    fi
  name: Install JFrog CLI

This is the full config file.:


    
orbs:
  artifactory: circleci/artifactory@1.0.0
version: 2.1
workflows:
  simple-docker-example:
    jobs:
      - artifactory/docker-publish:
          docker-registry: batelt-docker.jfrog.io
          docker-tag: 'batelt-docker.jfrog.io/hello-world:1.0-${CIRCLE_BUILD_NUM}'
          name: Docker Publish Simple
          repository: docker-local

Here also in the background, we are using the Building Docker Imagescommands via the JFrog CLI.
Step 3: Enable Pipelines
To orbs, we have to enable Pipelines via the UI under Project Settings -> Advanced Settings as follows.

Step 4: Configure the environment variables
Enable your project build in CircleCI by adding the following variables to your build settings:
ARTIFACTORY_URL
ARTIFACTORY_USERNAME
ARTIFACTORY_API_KEY

Step 5: View your build in the JFrog Container Registry
You can now view build in details in the JFrog Container RegistryBuild Browser. Click any build to drill down and view all the build info that the JFrog Container Registry captures.

Great! That’s it. You have successfully completed integrating the JFrog Container Registry in CircelCI done.