The Story
This post continues from How I Setup A Private PyPI Server Using Docker And Ansible
In this post, I will try to detail how to set up a private local PyPI server using Docker And Ansible.
TL;DR
Deploy/destroy devpi server running in Docker container using a single command.
The How
After my initial research, I wanted to ensure that the deployment is deterministic and the PyPI repository can be torn down and recreated ad-hoc by a single command. In our case, a simple make pypi
deploys an instance of PyPI server through an Ansible playbook.
According to the docs:
Ansible Playbooks offer a repeatable, re-usable, simple configuration management and multi-machine deployment system, one that is well suited to deploying complex applications. If you need to execute a task with Ansible more than once, write a playbook and put it under source control. Then you can use the playbook to push out new configuration or confirm the configuration of remote systems.
A basic Ansible playbook:
- Selects machines to execute against from inventory
- Connects to those machines (or network devices, or other managed nodes), usually over SSH
- Copies one or more modules to the remote machines and starts execution there
You can read more about Ansible here
The Walk-through
The setup is divided into two sections, Containerization and Automation.
This post-walk-through mainly focuses on automation. Go here for the containerisation.
Containerization
I didn't want the post to be too long.
Post continues here
Automation
See The How for the justification of opting for Ansible for the automation.
Prerequisite
If you already have Ansible installed and configured you can skip this step else you can search for your installation methods.
python3 -m pip install ansible paramiko
Ensure dependency plugins have been installed as well.
ansible-galaxy collection install \
ansible.posix \
community.docker
Directory Structure
In this section, I will go through each file in our pypi_server
directory, which houses the configurations.
├── ansible.cfg
├── ansible-requirements-freeze.txt
├── host_inventory
├── Makefile
├── README.md
├── roles
│ └── pypi_server
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ └── simple_test-1.0.zip
│ ├── tasks
│ │ └── main.yml
│ └── templates
│ └── nginx-pypi.conf.j2
└── up_pypi.yml
Ansible Configuration
Certain settings in Ansible are adjustable via a configuration file (ansible.cfg). The stock configuration is sufficient for most users, but in our case, we wanted certain configurations. Below is a sample of our ansible.cfg
cat >> ansible.cfg << EOF
[defaults]
inventory=host_inventory
# https://github.com/ansible/ansible/issues/14426
transport=paramiko
[ssh_connection]
pipelining=True
EOF
If installing Ansible from a package manager such as apt
, the latest ansible.cfg
file should be present in /etc/ansible
.
If you installed Ansible from pip
or the source, you may want to create this file to override default settings in Ansible.
wget https://raw.githubusercontent.com/ansible/ansible/devel/examples/ansible.cfg
Selecting machine to run your commands from inventory
Ansible reads information about which machines you want to manage from your inventory. Although you can pass an IP address to an ad hoc command, you need inventory to take advantage of the full flexibility and repeatability of Ansible.
cat >> host_inventory << EOF
vagrant ansible_host=192.168.50.4 ansible_user=root ansible_become=yes
[pypi_server]
vagrant
EOF
For this post, I will be using a Vagrant box.
According to the docs:
Vagrant is a tool for building and managing virtual machine environments in a single workflow. With an easy-to-use workflow and focus on automation, Vagrant lowers development environment setup time, increases production parity, and makes the "works on my machine" excuse a relic of the past.
Vagrant will isolate dependencies and their configuration within a single disposable, consistent environment, without sacrificing any of the tools you are used to working with (editors, browsers, debuggers, etc.).
Below is a Vagrantfile
, used for local development which you may use, you would just need to run vagrant up
and everything is installed and configured for you to work.
cat >> Vagrantfile << EOF
# -*- mode: ruby -*-
# vi: set ft=ruby :
# set up the default terminal
ENV["TERM"]="linux"
Vagrant.configure(2) do |config|
config.vm.box = "opensuse/Leap-15.2.x86_64"
config.ssh.username = 'root'
config.ssh.password = 'vagrant'
config.ssh.insert_key = 'true'
config.vm.network "private_network", ip: "192.168.50.4"
config.vm.network "forwarded_port", guest: 3141, host: 3141 # devpi Access
# consifure the parameters for VirtualBox provider
config.vm.provider "virtualbox" do |vb|
vb.memory = "1024"
vb.cpus = 1
vb.customize ["modifyvm", :id, "--ioapic", "on"]
end
config.vm.provision "shell", inline: <<-SHELL
zypper --non-interactive install python3 python3-setuptools python3-pip
zypper --non-interactive install docker
systemctl enable docker
usermod -G docker -a $USER
systemctl restart docker
SHELL
end
EOF
Thereafter run the following command which allows you to install an SSH key on a remote server's authorized keys and it facilitates SSH key login, which removes the need for a password for each login, thus ensuring a password-less, automatic login process.
# password: vagrant
ssh-copy-id vagrant@192.168.50.4
Once the vagrant box is up, use the ping module to ping all the nodes in your inventory:
ansible all -m ping
You should see output for each host in your inventory, similar to the image below:
Ansible Roles
According to the docs:
Roles let you automatically load related vars, files, tasks, handlers, and other Ansible artefacts based on a known file structure. After you group your content into roles, you can easily reuse them and share them with other users.
An Ansible role has a defined directory structure with eight main standard directories. You must include at least one of these directories in each role. You can omit any directories the role does not use.
Using the ansible-galaxy
CLI tool that comes bundled with Ansible, you can create a role with the init
command. For example, the following will create a role directory structure called pypi_server
in the current working directory:
ansible-galaxy init pypi_server
See Directory Structure above.
By default Ansible will look in each directory within a role for a main.yml
file for relevant content:
-
defaults/main.yml
: default variables for the role. -
files/main.yml
: files that the role deploys. -
handlers/main.yml
: handlers, which may be used within or outside this role. -
meta/main.yml
: metadata for the role, including role dependencies. -
tasks/main.yml
: the main list of tasks that the role executes. -
templates/main.yml
: templates that the role deploys. -
vars/main.yml
: other variables for the role.
Playbook
We defined our playbook which deploys the PyPI server below.
cat >> up_pypi.yml <<EOF
---
- name: configure and deploy a PyPI server
hosts: pypi_server
roles:
- role: pypi_server
vars:
fqdn: # Fully qualified domain name
fqdn_port: 80
host_ip: "{{ hostvars[groups['pypi_server'][0]].ansible_default_ipv4.address }}"
nginx_reverse_proxy: reverse_proxy
EOF
I found these posts relevant to the way we set up our nginx_reverse_proxy
:
- NGINX Reverse Proxy
- How to Set Up an Nginx Reverse Proxy
- How to setup Nginx Reverse Proxy
- How to Deploy NGINX Reverse Proxy on Docker
- Setting up a Reverse-Proxy with Nginx and docker-compose
Plays
files/
:
This is a simple tester for PyPI upload procedures. I modified simple_test package that was downloaded from https://pypi.org
Download: simple_test-1.0.zip
defaults/main.yml
:
These are default variables for the role and they have the lowest priority of any variables available and can be easily overridden by any other variable, including inventory variables. They are used as default variables in the tasks
cat >> defaults/main.yml << EOF
---
container_name : pypi_server
base_image : << Your Docker Registry>>/pypi_server:latest
devpi_client_ver: '5.2.2'
devpi_port: 3141
devpi_user: devpi
devpi_group: devpi
devpi_folder_home: ./.devpi
devpi_nginx: /var/data/nginx
EOF
tasks/main.yml
:
In this main.yml
file we have a list of tasks that the role executes in sequence (and the whole play fails if any of these tasks fail):
- Install
apt
andpython
packages.- Update apt cache and install
python3-pip
. - Install
ansible-docker
dependencies.
- Update apt cache and install
- Start
devpi
and configurenginx
routings.- Start
devpi
server ondocker
container. - Pause for 30 seconds to ensure server is up.
- Confirm if
docker
container is up. - Create PyPI user and an index.
- Template
nginx
reverse proxy config. - Check if
nginx
reverse proxy is up. - Reload
nginx
reverse proxy.
- Start
- Check if PyPI server is running!
- Install
python
dependencies locally in a virtual environment. - Check if
devpi
index is up and confirmnginx
routing! - Login to
devpi
as PyPI user. - Find path to
simple-test
package. - Upload
simple-test
package todevpi
. - Check if package was uploaded.
- Install
python
package from PyPI server. - Garbage cleaning.
- Install
Note: These tasks are executed on the remote server, in this case, a vagrant box.
Below is the main.yml
which details the configuration, deployment and testing of the PyPI server (in a vagrant box).
cat >> tasks/main.yml << EOF
---
- name: Install apt and python packages
block:
- name: update apt-cache and install python3-pip.
apt:
name: python3-pip
state: latest
update_cache: yes
- name: install ansible-docker dependencies.
pip:
name: docker-py
state: present
become: yes
tags: [devpi, packages]
- name: start devpi and configure Nginx routings
block:
- name: start devpi server on the docker container.
community.docker.docker_container:
name: "{{ container_name }}"
image: "{{ base_image }}"
volumes:
- "{{ devpi_folder_home }}:/root/.devpi"
ports:
- "{{ devpi_port }}:{{ devpi_port }}"
restart_policy: on-failure
restart_retries: 10
state: started
- name: pause for 30 seconds to ensure server is up.
pause:
seconds: 30
- name: "confirm if {{ container_name }} docker is up"
community.docker.docker_container:
name: "{{ container_name }}"
image: "{{ base_image }}"
state: present
- name: create pypi user and an index.
shell: "docker exec -ti {{ container_name }} /bin/bash -c '/data/create_pypi_index.sh'"
register: command_output
failed_when: "'Error' in command_output.stderr"
- name: template nginx reverse proxy config
template:
src: "nginx-pypi.conf.j2"
dest: "{{ devpi_nginx }}/{{ fqdn }}.conf"
- name: "check if {{ nginx_reverse_proxy }} is up"
community.docker.docker_container_info:
name: "{{ nginx_reverse_proxy }}"
register: result
- name: "reload {{ nginx_reverse_proxy }}: nginx service"
shell: "docker exec -ti {{ nginx_reverse_proxy }} bash -c 'service nginx reload'"
when: result.exists
- name: pause for 30 seconds to ensure nginx is reloaded.
pause:
seconds: 30
tags: [docker, nginx]
- name: check if pypi server is running!
delegate_to: localhost
connection: local
block:
- name: install python dependencies locally in a virtual environment
pip:
name: devpi-client
version: "{{ devpi_client_ver }}"
virtualenv: /tmp/venv
virtualenv_python: python3
state: present
- name: "check if devpi's index is up and confirm nginx routing!"
shell: "/tmp/venv/bin/devpi use http://{{ fqdn }}/pypi/trusty"
- name: login to devpi as pypi user
shell: "/tmp/venv/bin/devpi login pypi --password="
- name: find path to simple-test package
find:
paths: "."
patterns: '*.zip'
recurse: yes
register: output
- name: upload simple-test package to devpi
shell: "/tmp/venv/bin/devpi upload {{ output.files[0]['path'] }}"
- name: check if package was uploaded
shell: "/tmp/venv/bin/devpi test simple-test"
- name: install python package from pypi server
pip:
name: pip
virtualenv: /tmp/venv
extra_args: >
--upgrade
-i http://{{ fqdn }}/pypi/trusty
--trusted-host {{ fqdn }}
- name: garbage cleaning
file:
path: "/tmp/venv"
state: absent
tags: [tests]
EOF
templates/
:
Ansible uses Jinja2 templating to enable dynamic expressions and access to variables.
Below is an Nginx templated config file used for routing from localhost to a dedicated FQDN (Fully qualified domain name
)
cat >> nginx-pypi.conf.j2 <<EOF
server {
server_name {{ fqdn }};
listen 80;
gzip on;
gzip_min_length 2000;
gzip_proxied any;
gzip_types application/json;
proxy_read_timeout 60s;
client_max_body_size 70M;
# set to where your devpi-server state is on the filesystem
root {{ devpi_folder_home }};
# try serving static files directly
location ~ /\+f/ {
# workaround to pass non-GET/HEAD requests through to the named location below
error_page 418 = @proxy_to_app;
if ($request_method !~ (GET)|(HEAD)) {
return 418;
}
expires max;
try_files /+files$uri @proxy_to_app;
}
# try serving docs directly
location ~ /\+doc/ {
# if the --documentation-path option of devpi-web is used,
# then the root must be set accordingly here
root {{ devpi_folder_home }};
try_files $uri @proxy_to_app;
}
location / {
# workaround to pass all requests to / through to the named location below
error_page 418 = @proxy_to_app;
return 418;
}
location @proxy_to_app {
proxy_pass http://{{ host_ip }}:{{ devpi_port }};
proxy_set_header X-outside-url $scheme://$http_host;
proxy_set_header X-Real-IP $remote_addr;
}
}
EOF
Makefile
Below is a snippet from our Makefile, which makes it a lot easier to install dependencies and set up a PyPI server. This means that instead of typing the whole pip
or ansible-playbook
commands to install dependencies and bring up a PyPI server, we can run something like:
make install_pkgs pypi
You can also check out my over-engineered Makefile here.
cat >> Makefile << EOF
.DEFAULT_GOAL := help
define PRINT_HELP_PYSCRIPT
import re, sys
print("Please use `make <target>` where <target> is one of\n")
for line in sys.stdin:
match = re.match(r'^([a-zA-Z_-]+):.*?## (.*)$$', line)
if match:
target, help = match.groups()
if not target.startswith('--'):
print(f"{target:20} - {help}")
endef
export PRINT_HELP_PYSCRIPT
help:
python3 -c "$$PRINT_HELP_PYSCRIPT" < $(MAKEFILE_LIST)
install_pkgs: ## Install Ansible dependencies locally.
python3 -m pip install -r ansible-requirements-freeze.txt
lint: *.yml ## Lint all yaml files
echo $^ | xargs ansible-playbook -i host_inventory --syntax-check
pypi: ## Setup and start PyPI server
ansible-playbook -i host_inventory -Kk up_pypi.yml
EOF
Final Testing
To ensure deterministic pypi_server
builds, I ran/did the following:
- Stopped
pypi_server
container, deletepypi_server
on server - Ran a CI job that builds and pushes Docker images to our local docker registry.
- Started
pypi_server
container by executingmake pypi
whilst in the current working directory (ansible roles) on an ansible dedicated server. - Verified if
pypi.domain
FQDN is up (curl http://pypi.domain && dig pypi.domain
) - In a virtual environment, installed a random Python package then rebuilt the wheel before pushing it to
pypi.domain
Conclusion
Congratulations!!!
Accessing your FQDN you should see the devpi home page listing your indices:
Assuming that everything was set up correctly. You now have a local/private PyPI server running in a Docker container that is under config management, thus ensuring deterministic builds and a single command can tear it down or bring it up.
This was a great Ansible and Nginx learning curve for me and if you have reached the end of this post. I appreciate you!
Top comments (0)