DEV Community: Giorgio Lasala

Visualizing MergeStat collected data in Grafana dashboards 📈📊(Part 4)

Giorgio Lasala — Mon, 22 May 2023 17:11:10 +0000

And here we are, at the final part of this journey... When I started less than a month ago, I didn't think I would be able to write all these articles. Thanks to this hackathon, I had the opportunity to study and delve into many topics, from GitHub Actions to Grafana.
In this last part, we will talk about Grafana and how to use it to visualize the data we have collected so far on a dashboard.

Adding GitHub Credentials

First of all, I want to let you know that I made a small modification to the repository indexing preparation script to pass the GitHub token.
In particular, I added the call to this GraphQL mutation after creating the provider:

if(process.env.PAT_TOKEN) {
    await addCredential(idProvider, process.env.PAT_TOKEN);
}

And of course, the configuration within the workflow involves passing the parameter as an environment variable:

- name: Run script
  run: npm start
  env:
    CONFIG_PATH: ../../config.json 
    ENDPOINT: http://localhost:5433/graphql
    PAT_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Regarding the correct value to pass, I had some doubts as well. Initially, I followed the instructions on the MergeStatat website, which suggested passing a GitHub Personal Access Token (PAT), in my scenario associated with my user account as a secret in the workflow. However, I also tried using the GITHUB_TOKEN, and in the end, I found that it works correctly (at least for public repositories), although it has fewer privileges.

Furthermore, compared to the first article, I also modified how the first two parameters are passed, as shown in the code snippet before.

Configure Grafana with Docker-Compose

The goal is to set up a preconfigured instance of Grafana that is ready to use with Codespace. On the MergeStat GitHub repository, there is a docker-compose configuration that instantiates Grafana. Initially, I considered using it, but ultimately opted for a simpler solution without external volumes:

grafana:
    image: grafana/grafana-oss:latest
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    ports:
      - 3000:3000
    depends_on:
      - postgres
    restart: unless-stopped
    volumes:
      - ../grafana:/etc/grafana/provisioning/

In order to simplify the setup, I disabled authentication by directly utilizing environment variables, enabling anonymous access. Additionally, I mounted the ../grafana folder under /etc/grafana/provisioning/, allowing preloading of the datasource and dashboards.

Within the grafana folder, you will find two subfolders:

datasources
dashboards

Under datasources, there is a YAML file with the following configuration, allowing to configure a Postgres datasource and enabling connection to the local database:

apiVersion: 1

datasources:
  - uid: mypostgres
    name: MergestatDb
    type: postgres
    access: proxy
    url: postgres
    user: postgres
    database: mergestat
    withCredentials: false
    isDefault: true
    jsonData:
      sslmode: "disable"
    secureJsonData:
      password: postgres

This file allows you to configure a Postgres datasource, enabling connection to the local database.

For preloading the dashboards, the "dashboards.yaml" file will be used:

apiVersion: 1

providers:
  - name: 'MergeStat'
    type: file
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards
      foldersFromFilesStructure: true

Note that the foldersFromFilesStructure flag will create subfolders under Grafana, mirroring the current file system hierarchy.

Once this is done and Codespace is launched, you can simply connect to the Grafana instance by accessing the address using the Ports functionality of Codespace.

And the dashboards will be ready for immediate use:

GitHub Workflows sample dashboard

Conclusions

Thank you for the link to the MergeStat website, where users can find various pre-made queries and dashboards covering commit statistics to the number of open pull requests.

We hope you enjoyed this article series! If you found it helpful or have any comments or suggestions, we would love to hear from you. Don't forget to leave a comment or give it a feedback ❤️ if you found the content valuable.

Stay tuned for more exciting content in the future!

Configuring GitHub Codespace for repository analysis 🔎📙(Part 3)

Giorgio Lasala — Thu, 18 May 2023 19:59:32 +0000

And here we are, at the final step of configuring our environment for repository analysis. This part, where we create a custom configuration for GitHub Codespaces, has been both enjoyable and a valuable learning experience for me. It's interesting how this particular aspect was the one I was least familiar with, yet it has provided me with the opportunity to learn and grow the most.

What is GitHub Codespaces?

GitHub Codespaces provides developers with a cloud-based development environment, allowing for faster onboarding, coding on any device, and maintaining consistency across environments. Its user interface resembles popular IDEs such as Visual Studio Code, but it operates directly within your web browser. To tailor your project to GitHub Codespaces, you can create custom configuration files that ensure a consistent setup for all project users.

Starting with DevContainers

The first thing we're going to configure is DevContainers feature. What is it?

DevContainers is a feature of Visual Studio Code that allows you to define and configure containerized development environments. With DevContainers, you can specify all the dependencies, tools, and settings required for the development environment within a configuration file, known as devcontainer.json.
When you open Codespace with DevContainers configuration, it automatically starts a Docker container with all the preconfigured necessary resources, providing a consistent and isolated development environment.
This is particularly useful for ensuring that all team members have the same configuration and dependencies, facilitating collaboration and quick project startup.

In our case, I have created a devcontainer.json file inside the .devcontainer folder with the following configuration:

{
  "name": "Mergestat Codespace Project",
  "dockerComposeFile": "docker-compose.yml",
  "workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
  "service": "app",
  "remoteUser": "vscode",

  // Configure tool-specific properties.
  "features": {
    "ghcr.io/devcontainers/features/github-cli:1": {}
  },
  "customizations": {
    "codespaces": {
      "openFiles": [
        "README.md"
      ]
    },
    "vscode": {
      "extensions": [
        "mtxr.sqltools",
        "mtxr.sqltools-driver-pg"
      ],
      "settings": {
        "sqltools.connections": [
          {
            "previewLimit": 50,
            "server": "postgres",
            "port": 5432,
            "driver": "PostgreSQL",
            "name": "mergestat",
            "database": "mergestat",
            "username": "postgres",
            "password": "postgres"
          }
        ]
      }
    }
  },

  // Use 'forwardPorts' to make a list of ports inside the container available locally.
  // This can be used to network with other containers or the host.
  "forwardPorts": [5432],

  "initializeCommand": ".devcontainer/init.sh",
  // Use 'postCreateCommand' to run commands after the container is created.
  "postCreateCommand": ".devcontainer/post-create.sh"
}

Let's analyze the individual sections of the file in detail:

name: Specifies the name of the DevContainer configuration.
dockerFile: Specifies the path to the Dockerfile/DockerCompose used to build the container image.
customizations"/codespaces/openFiles: to open automatically the README file.
customizations/vscode/extensions: Lists the Visual Studio Code extensions that should be installed.
customizations/vscode/settings: Specifies custom Visual Studio Code settings that should be applied when working in the DevContainer. These settings can include editor preferences, formatting options, and more.
forwardPorts: Lists the ports that should be forwarded in Codespace to test and debug your application.
initializeCommand/postCreateCommand: Specifies a command that should be executed when the Codespace is initialized and after the container is created. This is used to prepare the environment as explained in greater detail in the following sections.

Docker-compose configuration

In our docker-compose file, we currently have two containers:

The "postgres" container: This container is responsible for starting the PostgreSQL database. It provides the necessary environment for running and managing the database.
The "app" container: This is the main container, built from the Dockerfile. It installs the PostgreSQL tools and will be used to restore the database.

Additionally, in the .devcontainer configuration, we privately expose port 5432 of the "postgres" container using "forwardPorts". This allows us to access the PostgreSQL database running inside the container from the local machine.

Features & VSCode Customization

With just a few lines of code, we can customize our environment in the following ways:

Installing the GitHub CLI
Installing VSCode extensions for interacting with the database
Initializing the default database connection

Bash Scripts for Codespace Configuration

We have two scripts executed at different times. The first script, init.sh, is executed before initialization and simply creates the folder where the database will be stored. In reality, this step could be skipped, but it helped me understand the Codespace lifecycle better.

The second script post-create.sh is the more important one. It is executed after initialization and performs essential configurations and setup tasks to ensure the environment is fully prepared.

Let's analyze the script:

#!/bin/bash
workflow_name="Run Analysis"

echo "Get info on latest GitHub Actions run..."
run_id=$(gh run list -w "$workflow_name" -L 1 --json databaseId | jq '.[]| .databaseId')

echo "GitHub Actions latest runId: $run_id"

echo "Download artifacts..."
gh run download $run_id -D $(pwd)/pgdata

The first part retrieves the artifacts associated with the latest execution of the "Run Analysis" workflow using the GitHub CLI. The artifact is then deposited into a temporary folder.

echo "Starting database restore..."
touch ~/.pgpass
echo "postgres:5432:mergestat:postgres:postgres" > ~/.pgpass
chmod 0600 ~/.pgpass

pg_restore -h postgres -U postgres -d mergestat --clean -x --no-owner $(pwd)/pgdata/pg-dump/backup_mergestat.dump

To ensure proper execution of the bash scripts, it is essential to grant the execution permissions using the chmod +x command. This command allows the scripts to be run as executable files and can be executed directly inside Codespace and then committed to repository.

Our Codespace will be:

Next Steps

In addition to incorporating more queries into the repository, one of the future developments I have in mind is to automatically provision a Grafana instance for visualizing the data on a dedicated dashboard as described on MergeStat site.

This would allow us to leverage Grafana's powerful visualization capabilities to gain deeper insights and analysis of the collected data.
By setting up an automated process to deploy and configure Grafana alongside our existing workflow, we can streamline the visualization experience and provide an intuitive interface for monitoring and analyzing the repository data.
This enhancement would further enhance the overall workflow and make it easier to extract meaningful insights from the collected information.

I hope to be able to achieve it before the end of the hackathon 😉!!

Collecting repositories stats with GitHub Actions workflow 🚀(Part 2)

Giorgio Lasala — Sun, 14 May 2023 13:13:37 +0000

Mergestat is a powerful tool that allows users to analyze Git repositories and gather important insights related to the commit history but not only it!

It comes in two different modes:

Mergestat Lite
Mergestat (Full)

The main difference between the two is that Mergestat Lite is a fast and lightweight local command line tool that uses SQL queries and SQLite to analyze Git repositories. It doesn't load data into a SQLite database file but fetches data directly from the source (like Git repositories on disk) as queries run. This makes it great for analyzing small repositories quickly, but it might not be the best option for larger ones.

On the other hand, Mergestat Full has a worker process that analyzes the repository and syncs the collected data into a PostgreSQL database. This mode provides a more robust and scalable solution for collecting data from larger repositories and analyze it at a later stage.

Although my initial plan for #GitHubHack23 challenge was to use Mergestat Lite to analyze the repository where the workflow was launched, perform the analysis and create an issue via the GitHub API to summarize all the information collected using the mergestat summarize commits command:

I eventually chose to pursue a more complex solution!
Despite being more challenging, I believed that this approach would be more versatile and capable of fully utilizing the potential of the tools available on GitHub 💪!

In the end, I decided to build a GitHub Action workflow that launch mergestat docker containers to analyze the repository, store the results in a PostgreSQL database and export it as workflow artifact.

Workflow implementation

The workflow itself was not complex and used standard GitHub Actions.

jobs:
  start:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3

      - name: Start containers
        run: docker-compose -f "docker-compose.yaml" up -d
        working-directory: scripts/ci

      - name: Setup node
        uses: actions/setup-node@v3
        with:
          node-version: 16.13.x
          cache: npm
          cache-dependency-path: scripts/ci/package-lock.json

      - name: Install
        run: npm ci
        working-directory: scripts/ci

      - name: Run script
        run: npm start -- ../../config.json http://localhost:5433/graphql
        working-directory: scripts/ci

      - name: Postgres Dump
        run: |
          docker exec -i postgres pg_dump -U postgres -Fc -Z 9 postgres > backup_mergestat.dump

      - uses: actions/upload-artifact@v3
        with:
          name: pg-dump
          path: 'scripts/ci/backup_mergestat.dump'

      - name: Stop containers
        if: always()
        run: docker-compose -f "docker-compose.yaml" down
        working-directory: scripts/ci

The only challenging part was finding a way to instruct Mergestat to index the selected repositories using the command line, as it is normally done via the UI.

Let's take it step by step.

Startup containers and install requirements

The first step is to download the source code and start all the required containers for mergestat using docker compose. To do this, I used the YAML script available on the mergestat website, removed the container for the graphical interface, and, most importantly, simplified the authentication process towards the GraphQL API by modifying the default_role of the Graphile Engine. This way, all API calls, even those not authenticated, will use the admin role, and it will be possible to bypass the authentication call... There might have been other ways, but for the project's purposes, this seemed to be the quickest and safest approach.

I had also started to figure out how to apply GitHub Services Container instead of docker compose to run PostgreSQL and the worker, but from what I've seen it's not (yet) possible to start a service container (in our case, the worker) only when another container (in our case, PostgreSQL) is ready.

After that, I set up the environment for Node.js, installed the necessary packages, and started compiling the TypeScript scripts.

Preparation script to indexing repository

For the writing of this script, I had mainly two options: either to understand the internal details of the database and write the appropriate SQL queries to insert the data, or alternatively, to use the same APIs that are used by the web portal.

In the end, I chose the second option and to speed up the process, I partly analyzed the calls made to the GraphQL endpoint using browser developer tools, and partly studied the code, reusing useful portions such as the schema, queries, and mutations. I also made extensive use of Codespaces to have a ready-to-use environment when needed to spin up all needed tools.

async function main() {
  let configPath = process.argv[2];

  if(!configPath) {
    console.error('ConfigPath is not valid: ' + configPath);
    process.exit(1);
  }
  console.log(configPath);
  const config : ConfigRoot = ConfigReader.read(configPath);

  let endpoint = process.argv[3];
  console.log('Endpoint: ' + endpoint);
  if(!endpoint) {
    endpoint = defaultEndpoint;
  }
  graphQLClient= new GraphQLClient(endpoint);

  await create(config.repositories);

  let syncCompleted = false;
  const total = config.repositories.length;
  do {
    const status = await getReposStatus();
    console.log(status);
    console.log('Wait completing count repositories: ' + total);
    syncCompleted = await checkSyncCompleted(status, total);
    await sleep(5000);
  }
  while(!syncCompleted)
}

The script is quite long, which is why you won't find it all in this article, but it is of course available on the GitHub repository.

I don't write much Typescript code in general (I mainly work on .NET projects), but apart from the initial configuration of the compilation scripts using ts-node, the rest went smoothly and it's just simple query and mutation GraphQL calls... actually, it was a great opportunity for me to deepen my knowledge of GraphQL with a different technology stack.

To make the script configurable, a JSON configuration file is read, such as:

{
    "repositories": [
        {
            "url": "https://github.com/salem84/AspNetCore.VersionInfo",
            "syncs": [ "Git Commits", "Git Files" ]
        },
        {
            "url": "https://github.com/xmasdev-2022/xmaze-api",
            "syncs": [ "Git Files"]
        } 
    ]
}

where there is:

url: repository absolute URL
syncs: an array with the names of the synchronization types to enable on the repository (here you can find other types)

However, I would like to integrate in the next few days the association of a GitHub Personal Access Token (PAT) to enable all possible types of indexing 💡.

Final steps

The last step is to export the database and upload it as an artifact on GitHub. In this case too, there were many ways to perform the dump... from using GitHub Actions available on the Marketplace or using command line and exporting plain scripts or gzip compressed tarballs. I chose to use pg_dump which is directly available inside the PostgreSQL container and export a dump in custom archive compressed mode.

docker exec -i postgres pg_dump -U postgres -Fc -Z 9 postgres > backup_mergestat.dump

where:

-U: username parameter
-Fc: custom archive output
-Z 9: maximum compression

In conclusion, we upload the artifact to GitHub and shut down all the containers.

Don't miss out on next blog post, where we'll explore how to set up Codespace to get all the tools you need for analysis.

Stay tuned!

Boost Your Repository Insights with GitHub Actions 🚀, Codespace 💻 and Mergestat 🔎📙

Giorgio Lasala — Wed, 10 May 2023 19:15:32 +0000

What I built

I have developed a solution that enables easy integration of the Mergestat within GitHub Codespace. With this integration, users can now seamlessly utilize Mergestat to analyze and track GitHub repositories using SQL queries, without needing to install any tools.
Furthermore, by leveraging GitHub Actions workflows, all the needed data can be collected automatically using scheduled workflows, thereby reducing information gathering times.

Category Submission:

Maintainer Must-Haves: Make the lives of Open Source maintainers easier.

Guidance: Developers can use Codespaces to set up an environment to triage issues and review pull requests. Actions can also be used to build tools and resources to help developers create better documentation.

App Link / Link to Source Code

https://github.com/salem84/repo-mergestat-codespace

Permissive License

MIT License

Background (What made you decide to build this particular app? What inspired you?)

My objective in developing these extensions was to create a simple and efficient system for analyzing repositories, which can be easily implemented within one's own GitHub organization. By doing so, users can benefit from quick and accurate insights into their repositories, streamlining their development process and improving overall productivity using open-source tools.

How I built it (How did you utilize GitHub Actions or GitHub Codespaces? Did you learn something new along the way? Pick up a new skill?)

To build this solution, I utilized both GitHub Actions and GitHub Codespaces. Specifically, I created a custom GitHub Actions workflow for data collection, and I configured a GitHub Codespace to execute queries. Along the way, I learned a lot about both tools and discovered new ways to use GitHub Actions in contexts outside of traditional CI/CD pipelines. Moreover, diving into the world of GitHub Codespaces was an eye-opening experience that allowed me to gain a deeper understanding of the platform's capabilities. In my upcoming detailed post, I'll share screenshots and code to describe each step of the process in more detail.

What is MergeStat

MergeStat is a powerful tool that allows developers to track and analyze data in git repositories and related sources (such as the GitHub API).
Using MergeStat, users can gain valuable insights into their codebase and monitor key metrics using SQL queries to extract relevant data from GitHub repositories, and presents the information also in a user-friendly dashboard, making it easy for developers to identify and address issues in their code.

AspNetCore.VersionInfo 1.1.0 is out

Giorgio Lasala — Fri, 10 Mar 2023 14:04:14 +0000

AspNetCore.VersionInfo is a NuGet library that provides a simple and effective way to obtain information about the version of the running ASP.NET Core application. This library can be used for various purposes, such as displaying the version on your web application or the list of referenced assemblies.
This library targets .NET 6 and is easy to install and configure in an existing ASP.NET Web Application.

After installing the NuGet package, it is sufficient to add the VersionInfo middleware to the application's middleware pipeline in the Configure and ConfigureServices method of the Startup object:

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddVersionInfo()
            .With<ClrVersionProvider>()
            .With<AssemblyVersionProvider>();
    }

    public void Configure(IApplicationBuilder app)
    {
        app.UseRouting();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapVersionInfo();
        });
    }
}

Once configured, the library will provide information about the application version through different endpoints; in particular there are 3 endpoints:

/version/json -> will return a JSON object
/version/html -> will return an HTML user-friendly page
/version/badge -> a nice and highly configurable badge image

Using this library has several advantages over manually implementing the application version. Firstly, the library provides a standardized way of obtaining information about the application version with a few lines of code, using built-in providers.

Additionally, the library provides a high level of configurability, allowing for customization of both the collected information and UI visualization (such as badge color and icons).

More information and samples are available on GitHub repository.

I'd love to know what your opinion!

Map your Issues!

Giorgio Lasala — Mon, 07 Sep 2020 22:37:55 +0000

Hi, I am Giorgio Lasala and this is my submission for Github Actions Hackaton.
This is also my first post on Dev.to !

Submission Category

Wacky Wildcards

How it works

Project is composed of three components:

A Github workflow triggered by issue events and powered by free and open-source Github actions
A simple Azure Function to receive event and write to Azure CosmosDb
A PowerBI Report to show finally issues on world map!

Workflow pipeline

Yaml pipeline uses two opensource Github Actions:

octokit/graphql-action: to query about user location info
satak/webrequest-action: to post JSON data to Azure Function

When workflow is triggered by a new issue (or comment), the first step

- uses: octokit/graphql-action@v2.x
  id: get_user_info
  with:
    query: |
      query info($owner:String!) { 
        user(login: $owner) {
          location
        }
      }
    owner: ${{ github.actor }}
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

extracts location from user profile, using GraphQL Github API.

Instead second step

- uses: satak/webrequest-action@master
  env:
    USERLOCATION: ${{ steps.get_user_info.outputs.data }}
    USERNAME: ${{ github.actor }}
    REPOSITORY: ${{ github.repository }}
  with:
      url: 'http://0caad210bf6f.ngrok.io/api/ReceiveIssue'
      method: 'POST'
      headers: '{"x-functions-key": "${{ secrets.FUNCTION_KEY }}"}'
      payload: '{ "location": ${{ env.USERLOCATION }}, "repository": "${{ env.REPOSITORY }}", "username": "${{ env.USERNAME }}" }'

creates a complete JSON payload and send it to Azure Function in order to store it in database.

Azure Function

It is a HTTP Trigger Azure Function that receives event from Github pipeline and writes to CosmosDb using OutputBinding.
Azure function must be invoked using a function key, received by x-functions-key HTTP header.


[FunctionName("ReceiveIssue")]
public static async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Function, "post", Route = null)] HttpRequest req,
    [CosmosDB(
        databaseName: "Github",
        collectionName: "Issues",
        ConnectionStringSetting = "CosmosDBConnection")] IAsyncCollector<Event> eventsOut,
    ILogger log)
{
    log.LogInformation("Received event");

    try
    {
        string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
        dynamic data = JsonConvert.DeserializeObject(requestBody);

        string location = data.location.user.location;
        string repos = data.repository;
        string username = data.username;

        var eventObj = new Event()
        {
            Location = location,
            Username = username,
            EventDate = DateTime.Now,
            Repository = repos
        };

        await eventsOut.AddAsync(eventObj);

        return new OkObjectResult("Event inserted");
    }
    catch(Exception ex)
    {
        log.LogError(ex, "Error insert event");
        return new BadRequestObjectResult("Event not valid");
    }
}

PowerBi Report

This is one of first PowerBi report, so I am not skilled about it and any suggestions are welcome!
There is an issue... on published report, data aren't updated in real time (you can find more details on Microsoft Docs and a trick to schedule hourly data refresh).
However using a DirectQuery supported datasource this problem can be overcome.

At the end, report is published on a public website!

Yaml File and Link to Code

You can check out the code on my GitHub Repository

salem84 / map-your-issues

Map your Issues

This repository contains a Github Action workflow triggered on new issues to create a World map of issues location.

How it works

It is composed of three components:

A Github workflow triggered by issue events and powered by free and open-source Github actions
A simple Azure Function to receive event and write to Azure CosmosDb
A PowerBI Report to show finally issues on world map!

Workflow pipeline

Yaml pipeline uses two opensource Actions:

octokit/graphql-action: to query about user location info using Github GraphQL API
satak/webrequest-action: to post JSON data to Azure Function

View on GitHub

In next days (after I will check some stuff) I will publish also PowerBi report on GitHub repository!

Additional Resources / Info

If you like, please add a ⭐ and especially write an issue, so you can pin 📌 you on map

Nice to have

This is obviously a simple Proof-Of-Concept project developed in some hours, but I think in next days/weeks I will try to improve such as:

⬜ Use DirectQuery Datasource to have near-real time updates (or alternatively develop a complete website using SignalR and Azure Maps or similar)

⬜ Improve Azure-side pipeline to improve scalability (e.g. using Azure Event Hub + data enrichments from other sources)

⬜ Create a shared-world-map of all Github repositories!

⬜ Create IaC script to speed up deployments

⬜ Extract data from issue message, using Text Analysis AI algorithms

✅ Improve Security with Function Authorization Key