DEV Community

Raphael Schubert
Raphael Schubert

Posted on

How to create a PIP package hosted on private Github repo

#python #pip #packages #github

I have few projects with few classes I've developed to integrate multiples services and servers. Those classes are private to my business and now I need an easy way to allow all projects use it and download it and all its updates.

What is the best way? Well, we all use pip everytime to download thousands of packages to our projects and is awesome.

So what can we do with our own private repos? Lets host its project on Github and see what happens.

This articles I'm writing and will share my experience on how to do this. Until the day of this article, I have no idea like you, on HOW it will works.

Lets start our study case.

First steps

Based on this tutorial: https://python-packaging.readthedocs.io/en/latest/minimal.html I'll create my first ever package ready to go to my private pip repo.

Creating root path and start a git

First, I'll create an path, called ptolemaios wich is my main class name and start a new git on it.

$ mkdir ptolemaios
$ cd ptolemaios
$ git init
> Initialized empty Git repository in '...ptolemaios/.git/'
Enter fullscreen mode Exit fullscreen mode

Folder structure

On the tutorial it says I need this folder structure:

ptolemaios/
    ptolemaios/
        __init__.py
    setup.py
Enter fullscreen mode Exit fullscreen mode

so, lets create again one more folder called ptolemaios inside my ptolemaios folder... sounds a little confusing.

$ mkdir ptolemaios
Enter fullscreen mode Exit fullscreen mode

Creating basic files

and lets create the files setup.py and __init__.py

$ touch setup.py
$ touch ptolemaios/__init__.py
Enter fullscreen mode Exit fullscreen mode

ok, now I have the exactly structure on the tutorial. Let's commit it and go to next steps.

$ git add .
$ git commit -am "created basic package structure files and folders"
> 2 files changed, 0 insertions(+), 0 deletions(-)
> create mode 100644 ptolemaios/__init__.py
> create mode 100644 setup.py
Enter fullscreen mode Exit fullscreen mode

MVP Package

As I want to learn how to do it, and check if is working, I'll start by creating a function that just prints something on my screen.

lets open our __init__.py file and edit it

def my_cool_test_method():
    print('It works!')
Enter fullscreen mode Exit fullscreen mode

and also lets open our setup.py and configure the basics about the package

from setuptools import setup

setup(
    name='ptolemaios',
    version='0.0.1',
    description='My private package from private github repo',
    url='git@github.com:rfschubert/ptolemaios-sdk-package.git',
    author='Raphael Schubert',
    author_email='raphael.schubert@digitalbankscorp.com',
    license='unlicense',
    packages=['ptolemaios'],
    zip_safe=False
)
Enter fullscreen mode Exit fullscreen mode

you can customize those infos with all infos you want.

lets commit it

$ git commit -am "updated setup.py and created my_cool_test_method"
> 2 files changed, 19 insertions(+)
Enter fullscreen mode Exit fullscreen mode

I've created my private repo on github already, so now, I'll add it to project and push all commits to my repo

$ git remote add origin git@github.com:rfschubert/ptolemaios-sdk-package.git
$ git push -u origin master
> Counting objects: 9, done.
> Delta compression using up to 8 threads.
> Compressing objects: 100% (6/6), done.
> Writing objects: 100% (9/9), 1.00 KiB | 514.00 KiB/s, done.
> Total 9 (delta 0), reused 0 (delta 0)
> To github.com:rfschubert/ptolemaios-sdk-package.git
>  * [new branch]      master -> master
> Branch 'master' set up to track remote branch 'master' from 'origin'.
Enter fullscreen mode Exit fullscreen mode

now, I want to add it on my running project to see if will work, I'll open any project I have on my machine and try installl it from GitHub using pip.

Installing with pip

lets open our terminal inside the project we want to install it and run this command

$ pip install git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
> Collecting git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
>   Cloning ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git to > /private/var/folders/7z/....
> Installing collected packages: ptolemaios
>   Running setup.py install for ptolemaios ... done
> Successfully installed ptolemaios-0.0.1
Enter fullscreen mode Exit fullscreen mode

Here we have few important things. Where I finded this URL?

Well, it is the basic clone url with few modifications.

My ssh clone url is: git@github.com:rfschubert/ptolemaios-sdk-package.git, look at : on ssh url, change it to / and add git+ssh:// on your ssh url

Now you should have git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git, that's it! just do pip install git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git and be happy!

Lets test it on my project.

I'll go on Django console to test it easy.

import ptolemaios

ptolemaios.my_cool_test_method()
# It works!
Enter fullscreen mode Exit fullscreen mode

AWESOME! I can have now private packages on my project.

Step back

Well, on my computer, all works like a charm, but that's why I have the ssh keys to fetch data from private repository. How my Heroku app will download it ?

first I'll run the command to update my requirements.txt file

$ pip freeze > 'requirements.txt'
Enter fullscreen mode Exit fullscreen mode

now, if I look inside of it, it only has my package name, and not the GIT url

...
ptolemaios==0.0.1
...
Enter fullscreen mode Exit fullscreen mode

you will need change this line to work like expected to:

...
git+ssh://git@github.com/rfschubert/ptolemaios-sdk-package.git
...
Enter fullscreen mode Exit fullscreen mode

so, when your server deploy, it will fetch from repository the current version.

I've founded an answer that can be usefull to some of you, but not to me, I'll need do some more research and update this file later.

You can create a Machine User on GitHub add it to your private repos, generate an ssh key on your server and add it to Machine User, every time you deploy, your server will have the ssh key and will be allowed to download it.

How to deploy to Heroku?

I dont know yet, but I'll find out and I'll be back... let's do some more research...

Latest comments (5)

Collapse
 
mlussati_ profile image
M@n!|$0n •

Great post. But I have a question I am doing this for a private repository but I want the whole team to use it, how can they not ask for a login and password?

Collapse
 
matitotias profile image
Matias Galleguillos •

Great post Raphael! This was really helpful, did you ever discover a way to publish different versions of the package? So some apps can keep downloading older versions if you make big changes

Collapse
 
rf_schubert profile image
Raphael Schubert •

Sorry for the giant delay! The pip package has this for default. You will just need to something like: pip install myawesomepackage==1.0.1 where the version should be the exact version that you need, even if is old.

Collapse
 
karfau profile image
Christian Bewernitz •

But are you aware of combining the approach that you describe (using git URL) with setting a fixed version? I would hope that it works using git tags. Would be nice if anybody has tried this can report here :)

Collapse
 
caicedodavid profile image
David Caicedo •

Great post, thanks!

Read next

shaheryaryousaf profile image

Start with Python Dictionaries

Shaheryar -

spiff profile image

Employee Management System using Python.

Ebikara Dandeson Spiff -

epakconsultant profile image

AI Enchantment with Python

sajjad hussain -

shaheryaryousaf profile image

An Introduction to Python Functions

Shaheryar -