DEV Community: Ana Jimenez Santamaria

Dockerizing your first web app with python and flask

Ana Jimenez Santamaria — Sun, 19 Jul 2020 21:02:28 +0000

Second part of how to measure covid-19 impact on Jitsi project with python and elasticsearch. This time, my colleagues (David, Gerardo) and I (Ana) built a web app with Python technology and Flask framework and dockerized it. Here we explain you how

Docker architecture

First, we have to build a docker application with three containers:

ElasticSearch image
Kibana image
Web app image

For ElasticSearch and Kibana, we use pre-built images from DockerHub using 7.8.0 version. Web application uses a redis-alpine image.

Note: The reason why using alpine, is because this alternative is smaller and more resource efficient than traditional GNU/Linux distributions (such as Ubuntu)

Required files and configuration

You can get the required materials on GitLab. The image below shows the different files you will get by cloning the repo

Docker-compose.yml structure

docker-compose.yml file shows the structure explained in previous section


version: '3'
services:
  web:
    build: .
  ports:
    - 5000:5000
  networks:
    - elastic
redis:
   image: "redis:alpine"
es01:
   image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
   container_name: es01
   environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
  ulimits:
    memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
networks:
  elastic:
    driver: bridge

Let's take a look at the main tags stated at docker-compose.yml

Ports: configures docker port with host machine port where docker runs. In this case, 5000, 5601 and 9200.
Image: docker image that is downloaded for the required service
Networks: this is named as 'elastic' in order to connect the three services
Environment: configures environment variables essential to operate among services, such as RAM memory parameters. Within Kibana service, it is necessary to define the variables ELASTICSEARCH_URL and ELASTICSEARCH_HOSTS, in order to link it with the Elasticsearch service.

Dockerfile configuration

Dockerfile has the required steps to configure the web application to link with our python script that extract and store the data in an Elasticsearch cluster, that at the same time imports impact_dashboard.ndjson on Kibana for previous visualization.

It runs on alpine distribution to copy the required folders to get the app running. Moreover, and thanks to requirements.txt, you can add all the dependencies that the python script needs.

FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP jitsi-data.py
ENV FLASK_RUN_HOST 0.0.0.0
RUN apk add --no-cache git
RUN apk add --no-cache tk-dev
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
COPY mapping.json mapping.json
COPY templates templates
COPY impact_dashboard.ndjson impact_dashboard.ndjson
RUN pip install -r requirements.txt
EXPOSE 5000
COPY . .
CMD ["flask", "run"]

How to start containters

In order to start the application (with docker and docker-compose previously installed) open your terminal and execute the following command

$ docker-compose up

This will initiate the different commands stated in dockerfile and download the required docker images. Once that's finished, you can check that Elasticsearch (localhost:9200) and Kibana (localhost:5601) are succesfully running.

Now, it's time to go where python Flask application web app is located, using port 5000

Clicking on start button will initialize data extraction from Jitsi git data (check jitsi-data.py for more details) and will store such data to our ElasticSearch. Finally, it will import impact_dashboard.ndjson to our Kibana, allowing us to interactively play with the data.

Once the process is finished, the browser will show the next message:

Of course, we can see if our Elasticsearch index and Kibana dashboard have successfully being added to our instances:

By default the time filter is set to the last 15 minutes. You can use the Time Picker to change the time filter or select a specific time interval or time range in the histogram at the top of the page.

et voilà now you have a cool dashboard up and running to analyze how a pandemic can impact Jitsi software development activity.

Bonus point: uploading a Docker image to Docker Hub

With a Docker Hub account, you can build an image where dockerfile is located. Simply type:

$ docker build -t dockerhubID/DockerHubreponame:imagetag .

Then, upload the image to your Docker Hub repo (you can create the repo using the Docker Hub UI

$ sudo docker push dockerhubID/DockerHubreponame:imagetag

Once the image is uploaded to Docker Hub, any user can use and run the app with the following docker-compose.yml

version: '3'
services:
  web:
    image: daviddp92/jitsi-data-extraction:1.0.0
    ports:
      - 5000:5000
    networks:
      - elastic
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.8.0
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1024m -Xmx1024m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:7.8.0
    container_name: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: http://es01:9200
    networks:
      - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
networks:
  elastic:
    driver: bridge

Closing thoughts

We have learned how to dockerize a web app with Python technology and Flask framework using docker-compose. We also saw how to use and run the application using Docker Hub images.

What do you think about this project? What should we improve? Feedback is more than welcome :)

How to measure Covid-19 impact on Jitsi project with Python and ElasticSearch

Ana Jimenez Santamaria — Fri, 03 Jul 2020 13:13:32 +0000

The situation with Covid-19 limits the possibility of face-to-face communication. Remote work is now a reality, and the demand for video conferencing such as Jitsi Meet continues breaking records.

Could it be possible that Jitsi’s software development activity and community are on high-demand too?

Jitsi

@jitsinews

👋 We are seeing high volumes of traffic from our friends all over the world (specially in Italy) and we just wanted to let everyone know that we tripled our deployment size in the busiest regions! 📈
Happy Meetings! ❤️

13:15 PM - 06 Mar 2020

This post summarizes one of the analysis that my colleagues David Duran, Gerardo Aguirre and I did for our M.S. in Data Science. Following the GQM approach, we defined a clear goal and some related questions. Then, we decided the best metrics to answer those questions.

Goal

Analyze the impact that a global pandemic such as Covid-19 has on Jitsi

Questions

Did Covid-19 countermeasures change:

Jitsi’s activity patterns?
Jitsi’s diversity patterns?
Jitsi’s community risk?

As we are trying to measure impact, we need to see potential changes. For instance, did software development activity change during covid-19 outbreak compared to last year?

We selected Feb 1st- May 31st 2019 and 2020 as our timeframe to measure such impact.

Metrics

Contributor Activity	Contributor Diversity	Community Risk
Total Contributors	Organization Diversity	Bus Factor
Total Commits	Gender diversity
	Timezone Diversity

Total Contributors: number of authors that made a commit over a period of time
Total Commits: number of commits over a period of time

Organization Diversity: different email domains over a period of time. Here we can find company’s domains (e.g jitsi, 8x8, etc) or individual’s (gmail, users, etc)

Gender Diversity: total share of female and male author names that made a commit

Timezone diversity: existing authors on each timezone

Bus Factor: minimum number of contributors needed to make the 50% of git code on Jitsi over a period of time

Tools used

Data gathering and storage

For data gathering we used an open source tool called Perceval. Perceval offers a Python module for retrieving data from repositories related to software development. It works with many data sources, from Git repositories and GitHub projects to mailing lists, Gerrit or StackOverflow.

For this specific analysis, we will just need Jitsi Git data.

If you are running Jitsi-data.py available in this Gitlab repo, make sure you have installed perceval first

$ pip install perceval

You can simply use perceval git module on python with perceval.backends.core.git:

import json
from elasticsearch import Elasticsearch
import logging
from datetime import datetime  # to specify a specific timeframe
import gender_guesser.detector as gender
from perceval.backends.core.git import Git
from dateutil.parser import parse

Perceval has 2 arguments for Git:

URI: the repository or list of repos from GitLab or GitHub you want to analyze
Gitpath: usually something like /tmp/repo-name.git

# It has 2 arguments, uri and gitpath: repo = Git(uri=repository, gitpath='/tmp/'+repository.git)
data_repo = Git(uri='https://github.com/jitsi/jitsi-meet', gitpath='/tmp/jitsi-meet.git')

Moreover, we use Python Elasticsearch client to store data and mapping from python script to our ElasticSearch instance.

# Instance of elasticsearch to put the data into ElasticSearch/Kibana system (it is necessary to have both up on localhost)
    elastic = Elasticsearch()
    # Read mapping json file to ElasticSearch
    mapping_file = open("mapping.json", "r")
    mapping = json.load(mapping_file)
    # Creation of ElasticSearch index using the mapping
    index_name = "repo_jitsi_index_def"
    response = elastic.indices.create(
        index=index_name,
        body=mapping,
        ignore=400  # ignore 400 already exists code
    )

Data transformation and mapping

Perceval gives us raw git data with a lot of information over a period of time (we specify this thanks to datetime python package): Commit, Author, Lines added, Lines removed, Email, Commit Date and more.

However, in order to analyze contributor diversity, we need to create new fields with the data provided. That's why we add organization (based on email domains), gender (using a gender detector python package) and timezone fields (cleaning item field)

You can see how we iterate through the different fields provided by Perceval to get the data in our GitLab repo

Once we generate these new fields, we create our Elasticsearch mapping, transforming item data into a new structure matching the mapping for ElasticSearch

Data visualization

If you are running unity-data.py script with your Elasticsearch and Kibana instance opened, you can go to your Kibana localhost (usually 5601) > management > index patterns and click on create index pattern. You should see the index name given at jitsi-data.py (by default is repo_jitsi_index_def)

We then elaborate a Kibana dashboard with the specific visualizations that fits best with our defined metrics. We called it “impact_dashoard”

At a first glance, we can see an uncommon behavior on Jitsi’s activity since February 2020:

And when comparing 2019 period with 2020, results are also quite interesting:

Image below highlights major changes regarding contributor’s diversity and activity:

We can see that:

Contributor’s Activity increased: there is an increase in commit activity, lines added and removed, authors and organizations involved on Jitsi’s software development

Contributor's Diversity increased: It’s interesting to see that women activity was under 1% last year and since covid-19 started, it increased to 8%. Still a low number, but is a good indicator.

Also, we see some author activity in Asia that didn’t existed before.

Disclaimer: there are “unknown” data that our gender detector couldn’t guess. The main reason is because some Jitsi’s developers have non-occidental characters and our gender detector is optimized for occidental names. We did an exhaustive cleaning for the most relevant names though, reducing our “unknown” data, but we couldn’t erase it completely.

What about Community Risk?

To answer our last question and metric, we work on a query using Jupyter Notebook.

The Bus Factor for developers isn’t something new: It is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel. In other words:

Our Bus Factor approach for our specific goal answers the question Which is the minimum number of contributors needed to make the 50% of git code on Jitsi?

As a result, we get that in 2019, Jitsi only needed 2 developers to make the 50% of git code while in 2020 Jitsi needs at least 9 developers. This fact is indeed good for Jitsi’s project health!

You can see Jupyter Notebook query in our GitLab repo

Future Work

This is just a very introductory analysis to measure covid-19 impact on Jitsi. Of course, we can improve a lot of things. Some future work might be:

Improve author affiliation: We know an existing open source tooling called SortingHat, that can be implemented with Perceval and other GrimoireLab components. SortingHat is an identity affiliation management. Thus, identities corresponding to the same real person can be merged in the same unique identity with a unique uuid.
Filter “unknown” category within gender term: Maybe gender-guesser package is not the best one to use. Do you know other python packages we can use? Leave your comments below :)
Work on new metric for risk assessment, such as the Gini coefficient.

And last but not least: Correlation does not imply causation. We don’t have all the domain knowledge Jitsi maintainers and community specialists might have. A relationship between Covid-19 and contributor's activity might be just coincidence

Remote work: The Open Source Way

Ana Jimenez Santamaria — Fri, 13 Mar 2020 22:40:59 +0000

Working remotely has become mission-critical now more than ever. Since one of the recommendations to stop COVID-19 massive spread is to stay at home, many companies are seeking effective strategies and tools for remote work.

For many people, this situation could be an overwhelming first-time experience, but for many others this is not something new: People behind Open Source programs have been involved in remote collaborative workspaces since decades. How does the Open Source ecosystem manage remote workforce?

Everything starts with culture

People is desperately asking for collaborative tools to stay productive and connected. But is that the best approach?

Think about it, Why giving to someone that has never handle remote work, a bunch of tools without context? In fact, tooling is the last thing you should be thinking of. Open Source has succeed over the past years not just because of tools, but because of culture

Before going nuts with what tools should people use, I suggest to first investigate what strategies are used within open source communities and what can companies learn from them.

This way of thinking can somehow relate to InnerSource term: a methodology which main goal is to establish open source-like culture within organizations. It has a consolidated and active community, called InnerSource Commons

Hands on remote-work: Best practices and suggestions

In my company -Committed Open Source practitioners- working from home has never been a problem- We have been doing it since always!- This might be the reason why, some people from Twitterverse and Instagram asked me about best practices to follow in order to manage remote collaboration and productivity. Then I though that the best way to answer was by sharing my top 6 best practices

Document everything: As in person-meetings are not an option, internal documentation plays an important role to ensure collaboration. This is a really common practice within open source projects, as code is being build collaboratively and remotely, done by people from different time zones and cultures. Some of them are experienced contributors, others are just newbies trying to get into their first open source contribution. in both cases, they need to know how to contribute - as well as pitfalls to avoid, guidelines, etc- and such information needs to be accessible and visible for everyone.
Build a Kanban board: One of the main resources coming from Agile methodologies that can be applied for any situations that requires organizing and scheduling tasks. This is so helpfull that I'm not only use it at work to coordinate tasks with my team, but also with my classmates for my Data Science MS... Even for my personal projects!

Schedule daily meetings to keep track of current work: This will help to keep your team synchronized, know what they are working at and find potential bottle necks... A good idea is to focus on short tasks, that can be done within 1 or 3 days period to track better the activity
Set your own working hours and breaks: set up an alarm and create a daily routine for work. The same way you do at work,dress up -avoid full-day pijama- and have specific clock-in and clock-out hours. This is very helpful to divide work vs spare time, and not fall into an endless working state that at some point will lead into a burnout.

Socialize: working from home doesn't mean you can't have great coffee break conversations with your teammates!. At my company, Every Friday at 2:00pm, we have what we call coffee-chats -. We join the call, talk about our week, and enjoy a nice cup of coffee with friends!

Do Exercise + healthy diet: I'm quite a restless person*, which means I need to remain focus and productive as much as possible if I want to do all the activities I've planned for that day.

*I have full-time work, Data Science MS, Japanese, and boxing classes, while saving time for my family and friends... All in the same week!

Boxing classes helped me a lot when covid-19 situation was not there, but now that gym is not the best place to be, I still doing some cardio at home to relieve stress. Moreover, avoiding added sugar and fast food intake made me feel less tired and more proactive during the day

Tooling

Once you have define a way of collaborative work and clear daily routines, it's time to look at the resources you need to make that possible.

There are lots of things you need to keep an eye on when choosing the best tool for you: Data Privacy, licenses, budget, scalability and more. However, taking into account that I'm not an expert on this field, I'll just enumerate some of my favorite tools I have been using during the past year at work or personal projects, grouped by different scenarios:

Communication Apps

Slack: similar to telegram, but more specialized for remote teams. you can have different channels which makes great to communicate among different working teams, topics or departments

Video chat

Jitsi.meet: No log ins and completely free video conferencing. You know what's best? it's fully encrypted!

Time management

Trello: really useful for those willing to use a Kanban board to manage daily tasks

File hosting services

NextCloud: Actually this is the only one I haven't use it yet, but I've heard a lot of positive feedback about this tool! Despite other alternatives, this solution does NOT store your data.

Podcast/interview recording

Etercast If you're planing to have several guests for an interview, Etercast might be a good solution. No log in, free and very easy to manage

Note: This post by Jesús González-Barahona shows more resources, also applied for distance learning. The best thing is that all of them are entirely free, open source software (FOSS)

Closing thoughts

Not every business is suitable for remote work: My goal here is not to show that everyone can work remotely -cause we can't-, but to give some guidance to those few who potentially can but don't know how to start with and share my thoughts and experience working in remote and collaborative environments.

Thanks for reading until the end. For those experienced people in collaborative and remote environments, I'd love to see any suggestions or tools that you might find useful to share to folks who are facing remote work for their first time.

Feedback is more than welcome :)