DEV Community: Nicolas Sebastian Vidal

Python script for cloning repositories

Nicolas Sebastian Vidal — Mon, 27 Jul 2020 12:02:11 +0000

A couple times it happened to me this year that I had to pull again the repositories from a given organization.

Mostly it happened to me because my computer broke, so a friend of mine lend me his computer so I could work. After mine was fixed I had to pull everything again.

So for not having to clone the repositories manually I wrote a Python 3 script for doing it for me:

The name of the organization is the GitHub username. So for example Facebook GitHub URL is https://github.com/facebook, if you would like to clone all their repositories you will have to replace organization-name-goes-here with facebook.

If you don't know how to create a personal access token, click the link for going to the GitHub documentation. It is well explained with screenshots of where to go and everything. Once you get it, you will have to place it where it says token-goes-here.

If you don't know how to install different versions of python on your machine, you could take a look to:

You will need to install PyGithub the library that I'm importing in the script:

pip install PyGithub

If you don't have pip installed, take a look here for setting it up properly.

Finally for running the script you just need to write in the terminal of your choice:

python cloner.py

Where cloner.py is the name of the file where we wrote our script. Be aware that all repositories will be cloned in the path where you are executing the script.

Have a great week everyone!!

Create GitHub PR based on a shared component upgrade?

Nicolas Sebastian Vidal — Sun, 04 Aug 2019 05:05:01 +0000

Private shared React component living here and installed cross several projects. Each project handled by different teams. I want to create a PR on all repositories using the component every time a new version of the shared component is published.

Does anyone know of an existent solution? Checking dependabot now, but not sure if that will work with private things :P.

How to create a gem from scratch?

Nicolas Sebastian Vidal — Fri, 03 May 2019 15:35:42 +0000

We will walk through the basic steps needed to create a gem and publish it.

Introduction

RubyGems has been included with Ruby itself since Ruby 1.9, and is central to Ruby code distribution. It allows Ruby developers to package their code up into a single compressed file, called a "gem". Gems have filenames that end in the .gem extension, and they contain both the Ruby code that makes up the library in question and metadata about the library, including author and version information.

Hal Fulton.

We will create a gem that will MonkeyPatch the String class in the Ruby language. It will introduce two new methods.

These methods will allow us to modify a string by randomly removing half the characters of it. We will call our gem thanoscase.

The name of the gem must be unique within the RubyGems ecosystem. So if you want to follow along with the steps for creating the gem, remember to change its name.

The only requirement for you is to have Ruby 1.9 or greater is in your system. A little bit of GitHub won't hurt either.

Crafting the gem

The first thing will be to create the structure we need to have a gem published in RubyGems. As you can see in the gist below:

Basic file structure

We will type a couple of commands in our console:

mkdir -p thanoscase/lib/thanoscase

touch thanoscase/lib/thanoscase/string.rb

touch thanoscase/lib/thanoscase.rb

touch thanoscase/thanoscase.gemspec

Commands for creation of basic file structure

string.rb

It will hold the logic for the methods that are going to be added to the String class. As can be seen here:

thanoscase.gemspec

The gemspec defines what’s in the gem, who made it, and the version of the gem. It’s also your interface to RubyGems.org. All of the information you see on a gem page (like jekyll’s) comes from the gemspec.

thanoscase.rb

At this point, we will use thanoscase.rb as a placeholder for importing other libraries. In this case, we are importing the class String.

Pushing changes to RubyGems

Once all the files are created, we need to build the gem. We will build it from the thanospec.gemspec specifications.

Inside your root folder run the following command:

gem build thanoscase.gemspec

Command used for building the gem

Installing and using our newly created gem

Run this for installing the gem locally:

gem install thanoscase-0.0.1.gem

Command used for installing

Now we will trigger the irb console and we will play along a little bit with our gem.

irb stands for interactive ruby is a tool to interactively execute Ruby expressions read from the standard input.

require 'thanoscase' is used for importing our library so we can play with it.

Pushing our gem to RubyGems

First, you will need to login into RubyGems:

gem signin

Command used for signing into RubyGems

And now what we have been waiting for!! Time to push your gem!!!

gem push thanoscase-0.0.1.gem

We are done!! Now your newly created gem can be installed from anywhere!!!

gem install thanoscase

Gem creation ends here. If you want to keep reading, we will add tests to our gem and continuous integration for tests to run automatically on every build.

RSpec for testing

It is a good practice to have tests in place for covering the functionality we have created.

Being said that, we will install RSpec, a tool for testing.

Add this line to your gemspec file:

s.add\_development\_dependency "rspec", "~\> 3.8"

It should look like this:

We will need to create a Gemfile, inside our root folder:

touch Gemfile

Add the following lines to it:

echo "source 'https://rubygems.org'" \>\> Gemfile

echo "gemspec" \>\> Gemfile

We are all set. Run the bundle command in order to install the RSpec dependency:

bundle

If you don't have bundler installed, you will need to run:

gem install bundler

If you want to install a specific bundler version:

gem install bundler -v 1.17.3

A new file will be created. Gemfile.lock will keep track of the library versions we are using.

When your co-developers (or you on another machine) check out your code, it will come with the exact versions of all the third-party code your application used on the machine that you last developed on (in the Gemfile.lock). When **they** run bundle install, bundler will find the Gemfile.lock and skip the dependency resolution step. Instead, it will install all of the same gems that you used on the original machine.

Once RSpec is installed. Run rspec --init to set up your project to use RSpec.

rspec --init

A folder spec will be created with a file inside called spec\_helper.rb. This file holds the basic and default configuration for RSpec to work. All tests will be expected to be placed under the spec folder.

Following this convention, we proceed to emulate our lib folder structure. We will create a file called string\_spec.rb as follows:

mkdir -p spec/lib/thanoscase

touch spec/lib/thanoscase/string.rb

And our file structure should look something like this:

File structure for specs

Inside our string\_spec.rb file, we will write a couple of tests:

Now we can run our test with:

bundle exec rspec --format documentation

The command for running specs

CircleCI 2.1 for Continuous Integration

Once you have created a GitHub repository and pushed all your changes to the cloud, your code will be able to receive contributions from all around the globe.

In order to ease the way to handle contributions (from other people or your own), we can automate the tests for running every time a new branch is created, or a new commit is pushed to such a branch.

Create a folder called .circleci with a configuration file inside config.yml :

mkdir .circleci

touch .circleci/config.yml

And inside this configuration file copy the following lines:

Once you have pushed these changes to your GitHub repository. And added the project to CircleCI, your tests will be running every time a contribution is made. You can take a look here for seeing how the specs were run.

Links of interest

The Sieve of Eratosthenes

Nicolas Sebastian Vidal — Fri, 27 Apr 2018 11:48:54 +0000

Generating a multiplication table with the first N prime numbers.

Introduction

This post is for walking you through the steps I have taken for creating a command line tool for printing out a multiplication table of the first N prime numbers. The result is a gem called primes_table and for the related code, you can take a look to this repository.

Stack:

Methadone an awesome project for creating command-line tools;
Ruby version 2.5.0;
RSpec as a testing framework;
Simplecov for code coverage;
CodeClimate for engineering process insights and automated code review;
SemaphoreCI for continuous integration;

Input

The command line tool will receive two parameters rows and columns.

Specifications:

By default the table will be generated as a matrix of 10X10;
Only values greater or equal than 10 will be considered for specifying rows or columns;
If you enter a value minor than 10, it will default to 10;
Only integer values are considered. For example, if you enter a string, it will be converted to an integer, the result will be 0 (zero) and will default to 10 because zero is minor than 10;

Output

A multiplication table with the first N prime numbers. The first row and column are the respective prime numbers.

For example, the default output will have as the first column, the first prime numbers until number 10. This means numbers 2, 3, 5 and 7. The same with the first row. The following image should help you visualize the desired output:

Creating the gem

With methadone installed you just need to run:

methadone --readme --rspec --license apache primes\_table

That command will generate the skeleton of your command line tool.

I have created an extra couple of options to add to the ones already included when you create the project. After that, all the options are:

-h, --help to see the available options;
-r, --rows ROWS to specify how many rows;
-c, --columns COLUMNS to specify how many columns;
--version it will give you the version of the gem you are using;

The following code handles the case where no options are entered. The default option is always going to be 10 for rows and columns.

Naive approach

The first approach was to loop over each number from 2 to N. This means that per each number I was iterating over it in order to check whether there was not another divisor than himself.

On the other hand, we have the Matrix class with a private method called header that returns the list of primes numbers I want, until the given N.

Making improvements

Once I got that working, I thought that it has to be a better approach to solving it. I'm iterating from 2 to N and over that list of numbers, I'm iterating again form 2 to N[i] checking for primality. So I checked in one of the books I have entitled Cracking the Coding Interview and there it was. Two improvements that can be done.

The first improvement was to iterate not from 2 to N[i] but to the square root of this number. So the code will look something like this:

The sqrt(n) is sufficient because for every number a which divides n evenly, there is a complement b , where a * b = n . If a > sqrt(n) , then b < sqrt(n) (since (sqrt(n))**2 = n). We, therefore, don't need to check n 's primality, since we would have already checked with b .

The next improvement is to implement The Sieve of Eratosthenes. This is a highly efficient way to generate a list of primes. It works by recognizing all non-prime numbers are divisible by a prime number.

We start with a list of all numbers up through some maximum value. First, we cross off all numbers divisible by 2. Then, look for the next prime (the next non-crossed off number) and cross off all numbers divisible by it. By crossing off all numbers divisible by 2, 3, 5, 7, 11, and so on, we wind up with a list of prime numbers from 2 through a maximum.

As you can see the method header calls process\_flags at the end. This is not part of The Sieve of Eratosthenes calculations since we are using it only to get the primes numbers based on the flags we have:

At the end our class Matrix will look something like this:

A couple of things to notice here is that I'm not using the class Prime anymore. I have taken this decision because the actions I want to perform are related to the Matrix itself and at the moment I don't really see the need for spreading the logic into another class. In the future, if the project grows in some way, I could evaluate how to scale it. But at the moment, I don't have enough information to do it.

Notes

As I mentioned at the beginning of this post, all code is Open Source and it can be found in this repository.

If you think that I should have taken a different approach or maybe you have a couple of ideas that you would like to implement, don’t doubt it and make a PR right away ☺.

One thing to notice is that right now I’m calculating the list of prime numbers for both, rows and columns. I think this could be improved. If rows are equal to columns it should be enough with calculating the list of primes only once.

Another improvement could be to only calculate the list of primes of the biggest number that was entered as row or column. In the process of calculating the list of prime numbers for the biggest number entered, we should be able to collect the prime numbers for the minor value as well.

Markdown files as static content

Nicolas Sebastian Vidal — Sun, 15 Apr 2018 12:26:01 +0000

How to use markdown files from a GitHub repository as static content in your website?

Context

We will be talking about a feature implemented on Agile Ventures that allow us to use markdown files as static content of the website. This idea was born when thinking about how to let developers contribute to the content of the site in an easier way while taking advantage of the GitHub ecosystem.

What we were using for editing the static content on the website was the mercury editor. But the problem with that editor is that has fallen out of maintenance. I have done a couple changes to that gem in order to make it work with rails 5. We are now using a custom version in a forked repository, but that is another story ;).

As the main goal, we want to get rid of that editor from our system. So, we have started from removing the functionality that involves static pages, for, later on, remove it from the projects as well. All the code related to the changes I have done in order to implement this feature can be found here.

Preparations

We need the repository from where we want to pull the content. Currently, our static pages live in this repository and we are pulling data from the root path, and working only with the markdown files.

The application is making requests to the GitHub API by using the octokit gem. So, we need to have the octokit gem in our Gemfile:

gem 'octokit', '~\> 4.8'

And the kramdown gem is being used for converting the markdown content into HTML. For adding the kramdown gem to the Gemfile:

gem 'kramdown', '~\> 1.16', '\>= 1.16.2'

Creating the service

As we will be using the kramdown gem for converting the content in a markdown file into HTML, we need to isolate the gem behavior. For doing this, we are going to create a class that will serve as a wrapper for the kramdown functionality.

The path to changeable and maintainable object-oriented software begins with classes that have a single responsibility. Classes that do one thing isolate that thing from the rest of your application. This isolation allows change without consequence and reuses without duplication.

Sandi Metz.

Crafting the rake task

Now we need to create the rake task that is gonna be scheduled by using the Heroku Scheduler add-on. It will look something like this:

I have created this rake task by following the code styles that the project is already implementing in order to preserve consistency within the system. So this rake task will call the logic that resides in the GithubStaticPages module.

The following code is the logic that we are using for pulling the content from markdown files, convert it to HTML and save it using the StaticPages model. This means that all HTML code related to a particular static content lives in the database.

Rendering the content

Since we have to change the way we CRUD static pages on the system, and we now will be using the GitHub ecosystem for doing it, there is no change to be implemented in the way we get the content from the database. The only difference is that now we have been able to remove the actions that were tightly coupled to the mercury editor.

Finally, we just get the HTML from the body attribute in the page object.

What's next?

Instead of having this rake task running daily within our system, we could be using GitHub Webhooks for achieving the same goal. But for doing this, we need first to create the functionality within our system that will handle those requests.

Talking about priorities, I will keep working on the removal of the mercury editor from the system since that is the general goal. Once removed, with the AV (Agile Ventures) team, we will decide what to do next. I'll keep you posted;)!

Notes

Something that it worth to mention is that this feature only creates and updates the static pages on the database. It doesn't remove pages that no longer exist in the repository from where we pull the content. This was a decision that we discussed within the team to be implemented in this way.

All code shared here is Open Source and you will find it in the WebsiteOne repository. If you want to contribute to the code, even if it is a small change where you are thinking "this could have been done in this other way". I invite you to create a pull request. It doesn't matter how small the change is, it will help us to improve and make our code better, while you will be contributing to an Open Source project ;).

How to create a GitHub repository?

Nicolas Sebastian Vidal — Mon, 12 Feb 2018 13:20:03 +0000

A simple guide to get you started with GitHub.

Context

This blog will walk you through the different steps you need in order to have your GitHub account setup and ready for creating amazing things :)! We will set up the SSH key and also the GPG key for having signed commits. The operative system that I will be using is Linux, distribution Ubuntu 16.04.3 LTS. If you are using Windows or MAC, you can follow this post as a guide but the respective commands for your operative system will be slightly different. You can post on the comments so I can help you. We will install GIT and push our first signed commit.

Creating a GitHub account

First, you just need to go here:

Complete the form with the requested info, and you will have your account created immediately. It will look something like this:

Creating your first repository

Notice that you have different tabs in your profile view: Overview, Repositories, Stars, Followers, and Following. One of our interest is Repositories, click there :D!

It will appear a view with the new button in it, click it:

After that you will be presented with the following form:

Notice that I completed the Repository name, I added a little Description and I clicked Initialize this repository with a README. This last step is important since we will be creating a repository with a file in it and it is different from when we create an empty repository (we will cover the empty repository later).

Regarding the name of the repository, I always use downcase only or combined with _.

Once you created the repository by hitting the button Create repository you will see something like this:

A couple of things to notice here since we are gonna configure the SSH key and the GPG key, we will clone the repository with the Clone with SSH option.

GIT installation

Now, if we want to work locally (meaning with our computer) with the repository we just created, we will need to install GIT.

Open a console Ctrl+T or go your programs an click the Terminal icon:

Once you are with the opened terminal, write the following command:

sudo apt-get update

This will update your local package index.

Now we will proceed to install GIT with:

sudo apt-get install git

We also need to tell GIT who we are, and for that, we are going to set up our name and email with the following commands:

git config --global user.name "Nicolas Sebastian Vidal"

git config --global user.email "nicolas.s.vidal@gmail.com"

Of course, where I have put my name it will go yours and the same with the email :).

Configuring your SSH key

Before cloning our repository, and in order to be able to commit and push our changes to the GitHub repository, first, we need to generate our SSH key.

Open a console Ctrl+T or go your programs an click the Terminal icon;
Paste the text below, substituting in your GitHub email address. ssh-keygen -t rsa -b 4096 -C "your\_email@example.com" This creates a new ssh key using the provided email as a label. Generating a public/private rsa key pair.
When you’re prompted to "Enter a file in which to save the key", press Enter. This accepts the default file location. Enter a file in which to save the key (/home/you/.ssh/id\_rsa):[Press enter]
At the prompt, type a secure passphrase. I encourage you to type a passphrase and not to leave it empty since it will give you more security. Enter passphrase (empty for no passphrase): [Type a passphrase] Enter same passphrase again: [Type passphrase again]

Start the ssh-agent in the background.
Add your SSH private key to the ssh-agent. If you created your key with a different name, or if you are adding an existing key that has a different name, replace id_rsa in the command with the name of your private key file.

These last two steps are for not entering the passphrase every time you want to push your changes to your repository.

These mentioned steps can be found also here.

Adding the SSH key to your GitHub account

While being in your terminal, write the following command:

cat ~/.ssh/id\_rsa

This command will show you your SSH public key:

Copy this key from ssh-rsa to your email nicolas.s.vidal@gmail.com.

Now we need to go to our GitHub profile, and click Settings:

Once you are there click the option that says SSH and GPG keys, click New SSH key after that, it will appear a form where we will paste our SSH key, add also a title to identify the machine that it will be using the mentioned key:

Press Add SSH key and finally our key it will be configured to be used:

Configuring your GPG key

If you want to have all your commits signed, follow these steps. Otherwise, you can completely avoid this.

Back in your terminal type the following command:

gpg --gen-key

Some options will appear, here the detail of each of them and the usual configuration that I use:

Please select what kind of key you want I use the default option RSA and RSA;
What keysize do you want?(20148) here you will see that I selected "2048", however, I only selected the default option because I'm using a virtual machine and it is kind of troublesome to generate the necessary entropy for getting all the needed bytes. I always go with 4096 as it is the recommended key size.
Key is valid for?(0) for this example, I have chosen that the key should not expire but what I always change the GPG key every month :D!
The next step will be to set up your Real name, Email address and Comment, this last one I use it to put my username :P!
The last step, you will have to generate enough entropy for getting the needed bytes, so for this, you will have to start doing some work on your computer. you could start moving your mouse like crazy (that works for me), also typing on a text editor a lot of random characters. One thing, do it fast!.

Once we have generated our GPG key we still need to add it to our GitHub account, the following steps need to follow:

Use the gpg --list-secret-keys --keyid-format LONG command to list GPG keys for which you have both a public and private key. A private key is required for signing commits or tags:

From the list of GPG keys, copy the GPG key ID you’d like to use. In this example, the GPG key ID is 77384044A63D6CFF ;
Configure Git on your machine for using this GPG key to sign your commits: git config user.signingkey 77384044A63D6CFF;
Paste the text below, substituting in the GPG key ID you’d like to use. In this example, the GPG key ID is 77384044A63D6CFF: gpg --armor --export 77384044A63D6CFF

Copy your GPG key, beginning with -----BEGIN PGP PUBLIC KEY BLOCK----- and ending with -----END PGP PUBLIC KEY BLOCK-----;

Adding the GPG key to your GitHub account

Once again as we did with the SSH key, you need to go to your GitHub profile and click on the Settings option.

Once you are there, click the option that says SSH and GPG keys and click New GPG key, a form will appear where we will paste it:

Press Add GPG key and finally our key it will be configured to be used:

The steps that I have mentioned to generate a GPG key can also be found here.

Cloning our repository

Back in our terminal, we will go to Documents and there we will clone our GitHub repository:

We will clone our repository by using SSH, so our command line would look something like this:

git clone git@github.com:nisevi/staticwebsite.git

It will ask us if we want to continue since the authenticity of the host github.com can't be established, but! as we trust GitHub we will say yes.

After that, you will have the repository cloned within the Documents folder, and you will find the README.md file that we have selected for initializing our repository:

Pushing your first commit

In order to create a commit, first we need to generate some changes to the files that we have in our repository, otherwise, we won't have anything to commit.

In order to check the status of the files and see if any of them has changed, we will use the following command:

git status

This is the actual content of the README.md file:

After the modification, we can see that our file was changed by typing the following command again:

git status

And we can take a look at the changes we have done with the following command:

git diff

As we can see Static website. has been deleted, space has been added, and Hello World. was written.

Now, we will add this file to our commit by doing:

git add README.md

After running this command, we will be able to see that our changes have been added to the commit we want to create. We can check this by doing:

git status

Our next step is to create a signed commit with a proper message about the changes we have done to the file. Please notice that here you will be asked to type the passphrase you used when creating your GPG key (if you have skipped that section you won't need to use the -S flag):

git commit -S -m "Update README.md"

-S flag is for signing our commit;
-m flag is for the message we want to write;

Once the commit has been created we will proceed to push our changes :D !!

Type the following command:

git push origin master

Done!!! we have pushed our changes. If you want to see them, go back to your GitHub repository. You will see that the README.md file has changed and if you want to know if your commit has been signed click where it says 2 commits:

In the following view, you will find the commit that we just did with the comment we have added; the green label that says Verified means that our commit has been signed :D!!!

Something to notice here is that I have already a signed commit that says Initial commit. This commit was created when we decided to initialize our repository with a README; for me is signed or Verified because I already have other GPG key and SSH key working on my laptop. What it matters is the commit that you just pushed :)!

This post ends here.

What you will find next are some alternatives to things that I have done in this tutorial. I hope you have enjoyed it!!

Creating an empty repository

If you decided to create an empty repository, you will be presented with different options. Here, we will walk through the steps for having this repository cloned and working on your computer.

Notice that I haven't selected the option Initialize this repository with a README. After you hit Create repository you will be presented with the following view:

Please notice that I have selected SSH as we will clone the created repository by doing:

git clone git@github.com:nisevi/staticwebsite.git

As we can see the repository has been cloned but there is nothing inside the folder (except the .git configuration folder of the repository). At this point, you can just create any file and follow the steps we have described on Pushing your first commit.

Cloning a repository by using HTTPS

Other option, instead of doing all the SSH and GPG configuration for pushing commits, will be to clone the repository with HTTPS:

Back in your terminal, we will clone the repository by doing:

git clone [https://github.com/nisevi/staticwebsite.git](https://github.com/nisevi/staticwebsite.git) https-staticwebsite

The last part that I just added was only to give the folder a name, in this case https-staticwebsite. I have done this because I didn't want to have conflicts with the same repository that I cloned with SSH:

If we take a look at the folders inside Documents, we will see:

We will get inside https-staticwebsite and do some changes to the README.md file and create a signed commit. But! as this is another folder with another Git configuration, we will have to setup our GPG key again by doing:

git config user.signinkey 77384044A63D6CFF

If you don't remember where this number 77384044A63D6CFF comes from, you might take a look back to the section where we have configured the GPG key.

After we have created our signed commit we will push our changes:

git push origin master

Notice that by using HTTPS you will always have to enter your username and password. In case you are using two-factor authentication instead of a password, you will need to enter a token.

How to generate a GitHub token

Go to your profile and click Settings. After that, click Developer settings and select the option that says Personal access tokens, press Generate new token and give it rights only for repo, as displayed here:

After you press Generate token, you will see the following:

Copy the token and save it in a safe place since this is the last time you will see it here. Once copied you can use it as password if you have activated two-factor authentication and want to push commits by using HTTPS.