DEV Community: Julian

Feedback on Prometheus exporter written in Golang for Magento 2

Julian — Wed, 20 Oct 2021 20:06:32 +0000

I decided to write 📝 a Prometheus exporter (in Golang) for Magento 2. Why? Because I wanted a better understanding of both Prometheus and Golang. Writing an exporter gives me a practical example for both.

"Magento 2 Prometheus Exporter" consists of two parts, the Prometheus exporter itself and a module for Magento 2 which extends upon the REST API to expose metrics where the native API of Magento 2 lacks in performance (such as determining the amount of orders).

I would love (❤️) and appreciate some feedback and constructive criticism on the exporter. Same goes for the Magento 2 module to extend the REST API but, because the personal objective for me is learning more about Prometheus and Golang, feedback on the actual exporter is of much greater value to me.

https://github.com/Serializator/magento2-prometheus-exporter-golang

SonarQube Pull Requests in Bitbucket Cloud

Julian — Sun, 27 Oct 2019 21:52:01 +0000

Please note ❤️

I created the solution I did and wrote what I wrote because there was a problem I was trying to solve within a set of requirements / "boundaries" lets say.

Using this solution requires the usage of SonarQube "analysis mode", which is deprecated since SonarQube 6.6. Once this solution is implemented you can't easily upgrade, without breaking the pull request analysis.

Please use this solution at your own discretion 🔥

I'll start by saying, the solution was way easier than I originally anticipated... I thought I was gonna have to run a Docker container with a microservice of some kind written in Java, Node.JS or Golang and run Sonar Scanner when a pull request was created or updated based on a webhook from Bitbucket.

But... As always, eventually the solution was so much easier, but what's the fun in writing a blog when you don't write about the way you failed miserably before succeeding?

Where it all started, a long... long time ago
From the very moment I met her, SonarQube, I knew I was in love. Knowing that it had a tiny, little, cute container that you could easily run with a single keystroke in the command line, with an embedded H2 database for convenience, and even an embedded instance of Elasticsearch, I couldn't be more intrigued 😍

I looked into some different static analysis tools, such as Code Climate, SonarCloud and Exakat, but they were either priced based on the size of your organization (Code Climate), or your projects (pricing based on LOC for SonarCloud), which might've caused scaling issues in the future. Or they weren't as easy to integrate with the existing tools such as Bitbucket Cloud, or didn't really have a friendly UI for the developer using it in case of Exakat.

Integrating SonarQube with Bitbucket Cloud ☁️
Mibex Software has a wonderful SonarQube plugin, Sonar Bitbucket Cloud Plugin, which analyzes and comments on your pull requests.

I decided to give it a go, I cloned the repository, built it with Maven so that I had a beatiful JAR to rsync to my Droplet on DigitalOcean and installed it to my SonarQube instance.

I installed the Sonar for Bitbucket Cloud plugin through the Bitbucket Marketplace, and expected that it'd have a built-in webhook to inform the SonarQube plugin whenever a pull request was created or updated, ran the analysis and comment on the pull request with its findings.

But, it wasn't that easy. It turns out, that if I had read the documentation fully, from top to bottom, it documents the fact that I had to run Sonar using Maven, or as I later found out the Sonar Scanner, to trigger an analysis on SonarQube.

It was fun for the time being, but... We'll stay friends! ❤️
Sonar for Bitbucket Cloud seemed to really only add the statistics on the overview page of your Bitbucket repository, it didn't really seem to contribute to commenting with found issues on your pull requests.

So... Before running it in automation, lets try it manually? 🔧🔨
It turns out that the Sonar Bitbucket Cloud plugin that I installed on my SonarQube installation also works when I simply run the Sonar Scanner with a few more arguments than I'd normally pass to it.

I hear you thinking, "uhm... but... wasn't that so obvious?!", and I know that now. But in the moment itself I wanted to pull the hairs, that I still have at this age, out of my head because of the frustration.

Originally I read this article (Decorate Bitbucket Cloud Pull Request with SonarQube Server Comments) to get familiar, so I got confused by the fact that the author was using a Maven goal to run the scanner.

When I ran the Sonar Scanner manually the pull request was analyzed and the Sonar Bitbucket Cloud plugin successfully commented on it with the issues that were found, in this case none.



sonar-scanner -Dsonar.projectBaseDir=$(pwd)
    -Dproject.settings=sonar.properties
    -Dsonar.analysis.mode=issues
    -Dsonar.bitbucket.repoSlug=<repoSlug>
    -Dsonar.bitbucket.accountName=<accountName>
    -Dsonar.bitbucket.branchName=<branch>
    -Dsonar.bitbucket.oauthClientKey=<oauth-client-key>
    -Dsonar.bitbucket.oauthClientSecret=<oauth-client-secret>
    -Dsonar.login=<sonar-token>
    -Dsonar.bitbucket.pullRequestId=<pull-request-id>

Should we integrate this with our CI/CD? Of course! 🌹
When I knew that the Sonar Scanner worked, I wanted to integrate it with our CI/CD to automate the analysis of both our integration branch as well as pull requests.

In the article I mentioned earlier, our beloved Jenkins was mentioned as well as some kind of microservice written in Java that was meant to trigger an analysis on SonarQube whenever a pull request was created or updated, based on a Bitbucket webhook.

So... I tried writing a microservice myself using Node.JS and Express.js to trigger an analysis based on a pull request creation or update event, but I quickly found out that Bitbucket has support for so called "Pull Request Pipelines".

I don't like over-engineering, but... It's my first instinct sometimes, and I fear that it's a curse... 💀👀

Using the blessing which are "Pull Request Pipelines" 🙈💘
When I figured out how the pull request pipelines worked, I was (still am) in love. What I ended up with was the pipeline configuration below 😍



pipelines:
  pull-requests:
    '**':
      - step:
          name: SonarQube Analysis
          image: newtmitch/sonar-scanner:4.0.0-alpine
          script:
            - sonar-scanner -Dsonar.projectBaseDir=$(pwd) 
              -Dproject.settings=sonar.properties
              -Dsonar.analysis.mode=issues
              -Dsonar.bitbucket.repoSlug=$BITBUCKET_REPO_SLUG 
              -Dsonar.bitbucket.accountName=$BITBUCKET_REPO_OWNER 
              -Dsonar.bitbucket.branchName=$BITBUCKET_BRANCH 
              -Dsonar.bitbucket.oauthClientKey=$OAUTH_CLIENT_KEY 
              -Dsonar.bitbucket.oauthClientSecret=$OAUTH_CLIENT_SECRET 
              -Dsonar.login=$SONAR_LOGIN 
              -Dsonar.bitbucket.pullRequestId=$BITBUCKET_PR_ID
  branches:
    master:
      - step:
          name: SonarQube Analysis
          image: newtmitch/sonar-scanner:4.0.0-alpine
          script:
            - sonar-scanner -Dsonar.projectBaseDir=$(pwd)
              -Dproject.settings=sonar.properties
              -Dsonar.login=$SONAR_LOGIN

Bitbucket has a bunch of pre-defined environment variables that you can use in these kind of situations. Environment variables that you need to define yourself are:

SONAR_LOGIN which is a SonarQube User Token
OAUTH_CLIENT_KEY and OAUTH_CLIENT_SECRETrequire an OAuth consumer to be configured with read access to the account and write access to pull requests.

For some reason a "Callback URL" is required, otherwise you can't create the OAuth consumer.

Telling SonarQube what project the repository is for! 📃📌
As you might have noticed within the command that I use to run the Sonar Scanner, it has an argument.

Dproject.settings=sonar.properties is the argument I'm talking about. It tells the scanner where to find the properties for this specific project, in this case it's the sonar.properties file at the root of the repository.



sonar.host.url=https://sonarqube.dev
sonar.projectKey=sonarqube-experimentation
sonar.projectName=SonarQube Experimentation

What are my next steps going to be in this journey? 🐒
What I noticed while I was researching, trying stuff out and miserably failing is that I had many individual resources but no way to consistently reproduce them.

What would happen if something went wrong. I corrupt one of my DigitalOcean droplets that was running Docker with SonarQube for example? I couldn't easily spin up another and be certain that it would be exactly the same, I didn't really document the steps I took through the way of research

Because of this I decided to become more familiar with the idea of "Infrastructure as Code", or shortened by the acronym "IaC"!

Terraform kinda fell into my lap at this point 👀💯

Terraform is one of the tools in the HashiStack by HashiCorp. It enables you to write your infrastructure in a declarative configuration, thus "Infrastructure as Code".

I plan to write a different blog about it where I'll reflect onto the stuff I learned, mistakes I made and times I fell face-first into seaming unsolvable problems!

As always, I hope you learned something from my failures (💩), and if not, I'm praying that you at least had fun reading about them 🔥

Alertmanager configuration with PyYAML

Julian — Mon, 09 Sep 2019 17:24:51 +0000

What my nightmares were like trying to use multiple configuration files with Alertmanager 💀👀

I recently started working on monitoring some of the internal tools and services at We Provide. I decided to tinker with the "Prometheus, Grafana and Alertmanager" stack as a first attempt.

Obviously, there we some hiccups... (literally and figuratively) 💩

Originally I started thinking about a tool I could write, something like am-stitcher (ssooo original, not like amtool, at all...), but... I realized that I was over-engineering something that was basically a matter of merging multiple YAML files into one.

Google, here I come, gimme something useful or copy-pastable please! ❤️
I didn't really find anything sufficient for this use-case in terms of literally merging.

Some of the tools I came across did what I wanted them to do, but didn't give me control over in what order to merge them. Some did, but they required me to specify every file individually instead of being able to recursively search a directory.

PyYAML with pyyaml-include 💌 💕
I found an article about !include in YAML, and as naive as I am, I thought it was something that might work in YAML natively... But, of course, it didn't!

Support for !include is added by the pyyaml-include extension, which adds this "directive" to PyYAML.

So... I tried installing PyYAML and the pyyaml-include extension using PIP, hoping that the parsing with PyYAML would work with only the CLI, and it'd automatically use the pyyaml-include extension. But I couldn't find a way to use PyYAML only in the CLI...

I should really stop being so naive... Don't you think so too? 😬😅

What I ended up writing was a small Python script that imports PyYAML and uses pyyaml-include to parse !include "directives".

Don't be fooled, this was a lot of copy-paste in combination with trial and error! 🔥

import sys
import os
import yaml
from yamlinclude import YamlIncludeConstructor

YamlIncludeConstructor.add_to_loader_class(loader_class=yaml.FullLoader, base_dir=os.getcwd())

data = ''

for line in sys.stdin:
    data += line

data = yaml.load(data, Loader=yaml.FullLoader)
yaml.dump(data, sys.stdout, default_flow_style=False)

I decided to use the standard input (sys.stdin) and output (sys.stdout) streams to read and write. I didn't want to bother with using arguments when I didn't even know the Python syntax 😅

python parse-yaml.py < alertmanager.yml > merged.yml

Shall we add a tiny bit of automation?! 🙌

At We Provide we mostly use Bitbucket, for open-source we use GitHub (obviously...), but for most of the stuff we use Bitbucket.

It consists out of three steps, "Parse & Merge", "Validate" and "Deploy". I think those are kinda self-explanatory, don't you think? 😊

pipelines:
  default:
    - step:
        name: Parse & Merge
        image: python:3.5.1
        script:
          - pip install pyyaml pyyaml-include
          - python parse-yaml.py < alertmanager.yml > merged.yml
        artifacts:
          - merged.yml
    - step:
        name: Validate
        image: serializator/amtool
        script:
          - amtool check-config merged.yml
    - step:
        name: Deploy
        image: eeacms/rsync
        script:
          - rsync merged.yml $USER@$HOST:$ALERTMANAGER_YAML
          - curl -L -X POST $HOST$RELOAD_PATH

As you can see, one of the images is serializator/amtool. Originally I tried using the amtool-docker image from Cisco Metacloud (Anthony Rogliano). But the fact that it was built on top of golang:1.9-alpine3.6 caused issues with the usage of /bin/bash instead of /bin/sh in the Bitbucket pipeline.

^ If you know a solution for this, or if I overlooked something in the Bitbucket pipeline that I could've configured, please let me know! ❤️

What I ended up doing was forking the repository and build the image on top of golang:1.13-stretch instead of golang:1.9-alpine3.6.

^ I thought about the increase in size by not using the alpine tag, but it's negligible in my opinion, looking at the use-case.

I'm in love with the "/-/reload" endpoint 🙈🙊
I thought that I'd had to use SSH and use Docker Compose to restart the Alertmanager service, but it luckily didn't end up being that ugly.

Somewhere in 2016 the /-/reload endpoint was introduced by ZhenyangZhao, and I'm glad he made that pull request! I don't know how I'd live with myself knowing that one of my deployments didn't have cURL in it...

So, without much hesitation, I quickly wrote one line more to send a POST request to the /-/reload endpoint, configured some stuff using environment variables, and pushed it to master!

I didn't know that this kinda happiness existed! 😊🔥
I finally laid down on bed, closed my eyes and began dreaming about the many files, folders and !include's I was going to write with a happy smile on my face! 😆

I hope you got something useful out of this, and if not, I hope you at least enjoyed reading it!

I thank you, @yamadashy for this emoji cheatsheet ❤️

First job as a Software Engineer, without a degree...

Julian — Sat, 27 Jul 2019 22:42:08 +0000

Hi! As it says on my profile, my name is Julian, 19 and currently working as a Software Engineer at We Provide.

"Software Engineer", "Full-Stack", "Web", "Computer Scientist?!"
"Software Engineer", honestly, it's only a term I use because I don't think "Web Developer" or "Full-Stack Software Engineer" (or any combination of those) suffice.

^ I changed my mind, "Full Stack Engineer" sounds more right for what I do the most, web development, and maybe it even sounds prettier too! 🙊

I like to tinker with a lot of stuff. I like to learn about algorithms and data structures, that doesn't make me a "Computer Scientist" or anything... I like working on monitoring stuff like Grafana with Prometheus, Logstash for centralized logging, that doesn't make me some kind of DevOps Engineer.

ppssstt, if you know a better job description, tell me!

Where I work 👀
Since I started working at We Provide, which as of this month is over a year ago, I've learned a lot. Definitely not only in a technical aspect, also socially and communicatively (is that a word).

uhm, where is your degree?! 🙄
In my head, writing this feels a bit, egotistical? I mean, the way I imagine you reading this is "wow, look at him, thinking that he is the only one who took this path and got somewhere!". Immediately after, another thought came lurking around, what if I'm writing this to get to know myself a bit better and actually think about what happened the past two years?

I guess this is it, the blog post that's gonna make me ball up in a corner and cry 😜

Somewhere end 2016 🎆
I started my education in "Applicatie Ontwikkeling Niv. 4 (MBO)", which translates to Software Engineering. "Niv. 4 (MBO)" has a better explanation on Wikipedia than I could possibly give you!

Throughout the time I spent at school, I didn't feel like I was really learning anything particularly new. I mostly did the work and helped other students. Though, I'm not saying it wasn't valuable to be there.

I actually felt like I belonged, with people who shared the same interests as I. I can't say how much that helped me, coming from high school, not knowing what I was gonna do in a year, two years, three, and so forth.

I guess, from mid 2017 till end 2017? 🔥💩
An internship... I had to go look for somewhere to do my internship... An internship on its own, love it, learn something in the real world instead of "Let's build a TODO list in C#!". What wasn't lovely about it was the fact that I was, and still am, not good in new environments (especially socially), and with new people who I'd have to be with for 8 hours a day. Scary!

So... I found somewhere to do my internship, obviously had some nerves the night before, and was actually really nervous and anxious throughout that first day. "Just have to get used to it", I thought, nothing weird, everyone gets nervous on their first day, whether it's a little bit or like me.

After a couple of days of thoroughly sweating (it also weren't the coldest days of the year), shivering and throwing up every morning, I decided that it wasn't for me. Because of the situation I was in I started thinking, why? Do I need to do this? Yes, if I want my college degree!

Long story short, I sat down with my mentor and HR at the internship, and talked about what was going on. We also talked about where I saw myself in ... years, or any other reasonable time frame. After I told what I wanted to do and where I wanted to go, HR said something that I don't exactly remember but basically translates to:

I have ..., so I can't do ... anymore, sometimes you have to accept where you are.

We talked for a bit after that, came to the conclusion that the internship wasn't for me, and left. I talked with a couple people at my school who wanted to help me, but it eventually came down to the fact that I didn't see myself going through an internship where I'd shiver all day long, only for a piece of paper 4 years later.

Somewhere mid 2018, I decided to apply for a job! 📄
I finally got of my lazy ass, opened up Google (without getting of my ass, obviously...) and looked for "developer job". That was my first attempt, and I failed miserably! I quickly narrowed that down to "web developer" and found a couple.

What I ended up doing was getting into an (unknowingly) long-term relationship with a recruiter. I took one job interview using this strategy and I thought, nah... I really have to find something myself, I know what I want and I know it when I see it, not by getting a call about some potential employer who needs someone with "my skill set".

I still get calls "how is it going at your job, looking for something?!" 😢

Job interview at We Provide 😄
I eventually found We Provide, a web agency that was looking for a Back-end PHP developer, and didn't have a job description that looked like "HTML, PHP, CSS, MySQL, .NET, MongoDB, Redis, Docker, AWS, Micro-Services" (and all the other terms you could put in there). It just looked like a fun and interesting place to work at, somewhere where I could keep learning and work on myself.

I applied, soon enough got a response, and scheduled a job interview. What I remember most is the fact they thought that I had a degree, because I put "Applicatie Ontwikkeling Niv. 4 (MBO)" on my job application. I forgot that they probably wouldn't look at the years and see "2016-2017" and think "Did he finish that?".

But the thing is, I don't remember that specifically because of it. I remember it because when we talked about it, it wasn't like "it goes downhill from here, he has no degree, sorry, there is the door!", they were, intrigued? I can't come up with a better word for it.

Where I am now... ❤️
Right now, I don't think "ugh, weekend is over, another day at school/work!", I'm happy. I have a lot of fun with the people I work with and most importantly, I learn a lot.

I want to clarify, I don't think that everyone should think "I don't need a degree!", if you can get one, do it. It depends on the situation.

What I want to end with is, when someone doesn't have a degree, don't assume they didn't want it. Don't assume they are lazy or not motivated. I think that when you do something that you like, do it in your spare time and learn because of the fun of it, you can do so much.

I thank you, @yamadashy for this emoji cheatsheet 🙈