DEV Community: schollii

See System Restarts in 3 clicks (after one-time setup)

schollii — Thu, 13 Oct 2022 15:59:31 +0000

Frustrating how it is so non-obvious to find when a Windows system has rebooted! I should just have to click a couple of buttons and have that information easily browsable.

The following setup, once complete, will make this info only a couple of mouse clicks away. On Windows 11: (the first and last steps are slightly different on earlier versions of Windows, but you should not have any trouble figuring it out):

open Event Viewer (which you can find via the Search box or magnifying glass in Windows menu)
expand the Windows Logs, right-click the System item, select Create Custom View
where it says <All Event IDs>, enter 41,1074,6006,6008 and click OK
A Save Filter to Custom View dialog will appear; name the view something like System Restarts and click OK.
(Optional) To group by day (very handy), right-click the new view in Custom Views, then View -> Group by -> Date and Time
On the desktop Task bar, while the Event Viewer window is still open, right-click the Event Viewer icon and select Pin to Taskbar.

Now seeing these restart-related events is a couple of clicks: click the Event Viewer in task bar, expand Custom Views, click System Restarts.

My Event Viewer now looks like this:

Eg, my last restart shows a User32 event type (1074):

for the following reason: Operating System: Service pack (Planned)
Reason Code: 0x80020010
Shutdown Type: restart

Whereas the previous event shows a Kernel-Power event marked Critical:

The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.

Have fun!

Using jq to find the latest master tag of an AWS ECR repository

schollii — Thu, 15 Apr 2021 01:13:01 +0000

Lately I've had to use jq a few times and got to understand the power of its pipe system. Here is an example of how to get the latest docker image tag in a specific docker repo of AWS ECR.

Firstly you have to know that aws ecr describe-images --repository-name $repo_name returns a json object which consists of a list of objects, each object looks like this:

  {
    "registryId": ...,
    "repositoryName": ...,
    "imageDigest": "sha256:...",
    "imageTags": [
      "..."
    ],
    "imageSizeInBytes": ...,
    "imagePushedAt": ...,
    "imageManifestMediaType": "application/...",
    "artifactMediaType": "application/..."
  }

Not all objects have an imageTags.

Further requirements:

I don't want objects that have a non-master-branch tag (they have "dev" in their tag)
I am only interested in the first tag of each object
I want to output the tag in a form that I can consume in a script later, in my case REPO_NAME=tag where REPO_NAME is an environment variable.

Here is the code:

aws ecr describe-images --repository-name $repo_name  | jq '.[] | map(select(has("imageTags"))) | sort_by(.imagePushedAt) | map(select(.imageTags[0] | contains("dev") | not)) | .[-1] | [(.repositoryName | gsub("-";"_") | ascii_upcase), .imageTags[0]] | join("=")'

I explain each line (the following block of code will not run as-is due to the comments; use the above snippet instead):

  repo_name=...
  aws ecr describe-images --repository-name $repo_name | 
    # don't want quotes around the final output so -r:
    jq -r '
    # loop over all objects of the list:
    .[] 
    # select those that have imageTags:
    | map(select(has("imageTags")))
    # filter out those that have "dev" in their tag:
    | map(select(.imageTags[0] | contains("dev") | not))
    # sort remaining ones by push date:
    | sort_by(.imagePushedAt)
    # keep only the most recent one
    | .[-1] 
    # create pair based on fields of the selected object
    | [
         # 1st is env var name for the repo: all upper, and underscores:
         (.repositoryName | gsub("-"; "_") | ascii_upcase), 
         # 2nd is the tag itself:
         .imageTags[0]
      ]
    # format items of the pair into ENV_VAR_NAME=tag:
    | join("=")'

Example output:

MY_REPO_NAME=2021.03.22-9941301

Create and Merge Pull Requests Programmatically in BitBucket

schollii — Mon, 03 Aug 2020 23:05:31 +0000

On this one large project the bitbucket account has a large number of git repos in Atlassian BitBucket Cloud. Sometimes for one JIRA ticket, we need to fix code across a whole bunch of repos, say to change the URL for a server because it was not parametrized, or to adopt a new naming convention. With JetBrains IDE (like IntelliJ, PyCharm, etc), it’s easy to do a bulk commit of all repos for that one change as you can use the same commit message, same referenced JIRA ticket number etc. A couple of button clicks and you’re done that part, piece of cake. Or just use a for loop in bash if you prefer to work in terminal.

But then you have to create a pull request for each repo, ouch!! Each one will likely have the same set of reviewers, same description, etc. I have no patience for that type of repetitive work. And pycharm (as of 2020.1) provides no assistance here.

After a bit of digging and trial and error, I found a convenient way to create pull requests programmatically using bitbucket’s REST API, and after they have all be reviewed and approved, to merge them.

Create one PR in the web UI on one of the repos. Write a useful title, description, select all the reviewers you might need across all repos. Don’t forget to checkmark the "Delete branch on merge" if that is part of your process.
After the PR is created, note its number in the URL (let's call it PR_NUMBER_JUST_CREATED).

From a terminal shell, use this to get the JSON of that PR into a file. Note that your username is the one defined in your Profile Settings, and can be changed. For me it is not the name I use to login with.

BB_URL=https://api.bitbucket.org/2.0/repositories/YOUR_ORG_NAME
PR_NUMBER_JUST_CREATED=<PR_NUMBER_JUST_CREATED>
curl $BB_URL/REPO_NAME/pullrequests/$PR_NUMBER_JUST_CREATED \
    -u USERNAME:PASSWORD | jq . > pr_get.json

Edit the file to remove most elements of it, keeping only the following: description, title, close_source_branch, reviewers, and source
In source element, keep only branch, remove the rest.

Create the remaining PR's programmatically:

for repo in $REPOS; do 
    curl -s $BB_URL/$repo/pullrequests --user USERNAME:PASSWORD \
        --request POST --header 'Content-Type: application/json' \
        --data @pr_get.json \
    | jq -r .links.merge.href
done > pr_merges.txt

Once they are all ready for merging, run this:

for merge in $( cat pr_merges.txt ); do
    curl $merge --user USERNAME:PASSWORD --request POST
done

There's various improvements that could be done of course, say to turn these into a bash command, get password from stdin, etc. This has been good enough so far.

I posted this originally in Create Pull Requests programmatically in BitBucket and Merge Pull Requests programmatically in BitBucket.

HAProxy watch status when no web browser or syslog!

schollii — Mon, 20 Jul 2020 15:20:07 +0000

The other day I got a challenge: I had to 1) do a basic setup of haproxy so one frontend would round-robin on 3 backends (simple task), but 2) verify that it was actually round-robin'ing, without using a web browser (to access the stats page that haproxy can be
configured to serve, eg see “HAProxy Stats Page” on https://www.haproxy.com/blog/introduction-to-haproxy-logging/) or syslog.

I tried curl which technically is not a web browser, but html was horribly formatted and didn't seem to have the backend status up/down etc; and converting to markdown and plain text with pandoc did not help.

This won't be news to those who work with haproxy all the time, but for those starting:

Turns out the haproxy makes status available through a unix socket that can be configured in the global section of /etc/haproxy/haproxy.cfg. On my centOS it defaults to /var/lib/haproxy/stats, whereas on ubuntu it defaults to /run/haproxy/admin.sock.

After a bit of research I found it was sufficient to send the show stat command to the socket, and then process the output. Even better,

watch 'echo "show stat" | sudo nc -U /var/lib/haproxy/stats | cut -d "," -f 1,2,8-10,18 | column -s, -t'

on the load balancer host (LB_HOST) to get the stats. Then in a separate terminal (same or different host), I ran

while true; do curl LB_HOST:HAPROXY_PORT; sleep 1; done

to generate http requests on the load balancer, and in a third terminal I could add/remove backend servers and observe the load-balancer status up/down. Example output of the watched command:

$ echo "show stat" | sudo nc -U /run/haproxy/admin.sock | cut -d "," -f 1,2,8-10,18 | column -s, -t
# pxname       svname    stot  bin    bout   status
test_frontend  FRONTEND  275   21725  67755  OPEN
test_backend   server1   158   12482  39026  UP
test_backend   server2   93    6399   19871  DOWN
test_backend   server3   39    2844   8858   DOWN
test_backend   BACKEND   275   21725  67755  UP

Pretty cool!

You can even have a "prompt" where you can do various things other than stats, like take down one of the backend servers for maintenance, debug certificates/SSL, etc.

Eg with only servers 1 and 2 up (ie 3 is down), I disabled 1 and 2 in haproxy for maintenance, by doing this:

$ sudo nc -U /run/haproxy/admin.sock
prompt
> disable server test_backend/server1
> disable server test_backend/server2

then the watch automatically updated to show this:

# pxname       svname    stot  bin    bout    status
test_frontend  FRONTEND  674   53246  164642  OPEN
test_backend   server1   472   37288  116584  MAINT
test_backend   server2   148   11692  36556   MAINT
test_backend   server3   0     0      0       DOWN
test_backend   BACKEND   674   53246  164642  DOWN

Notice now the whole BACKEND is down too, because of server 3. Start server 3 and observe its state going from DOWN to UP, and then that of BACKEND does same.

Enjoy!

A couple of useful refs:

Convert date formats in Google Sheets

schollii — Sat, 04 Jul 2020 16:12:36 +0000

The other day I downloaded transactions from my bank account and found that the date format had changed from DD/MM/YYYY to MM/DD/YYYY. There is nothing in the CSV file that tells Google Sheets what the format is, so I was seeing 01/09/2019 as "Jan 9, 2019".

I bet TD Bank is not even aware of this, nor the mess this will cause, but anyways, I couldn't find any builtin function in Google Sheets to fix that. Data format did not help either (I guess the formatting says how to render, not how to interpret data).

In the end I resorted to this transformation:

Import data into Google Sheets
Notice the dates are all messed up, and swear profusely that the data provider did this, try a few things and fail, search web, nothing really comes up except (hopefully!) this post!
Insert a column to the right of your date column
Paste the formula =REGEXREPLACE(A1, "(\d+)/(\d+)/(\d+)", "$2/$1/$3") and fill the remainder of the new column with that formula
Select the new column and copy it
Paste the new column over the old, BY VALUE (don't just ctrl-v, instead right-click on the top cell where you want the paste to start, and select Paste... -> Values Only); ignore that the new column now shows re-inverted date format
Remove the new column you created; the old column now contains dates correctly interpreted.

Note: changing locale might have been an option, didn't try, but for me it wasn't really an option anyways, my current locale is exactly the way I want it.

Expand disk in virtualbox Ubuntu 18 guest

schollii — Sat, 04 Jul 2020 15:59:17 +0000

I needed to expand the virtual disk drive of several Ubuntu 18.10 vm guests I had. I looked around the web a bit and found that the posts explaining how to do that involve more steps that is now necessary on Ubuntu 18. So here is what was sufficient on 3 separate VM's:

Resize virtual disk in VirtualBox:
1. Power down the VM
2. Go to Virtual Media Manager (VMM)
3. Select the disk that corresponds to your VM and resize the disk file (this is possible only on powered down VM)
4. Start VM
Resize partition in guest:
1. Install gparted: sudo apt install gparted
2. Start gparted
3. Select partition
4. Select resize
5. Stretch it to desired size
6. Check the details to confirm it did what you want
7. Make OS see the entire partition using pvresize:
```
$ sudo pvresize /dev/sda1
  Physical volume "/dev/sda1" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized
```

Resize the logical volume using lvresize:

$ sudo lvresize -l +100%FREE /dev/mapper/ubuntu--vg-root
  Size of logical volume ubuntu-vg/root changed from 29.03 GiB (7432 extents) to <48.85 GiB (12505 extents).
  Logical volume ubuntu-vg/root successfully resized.

Resize the filesystem:

$ sudo resize2fs /dev/mapper/ubuntu--vg-root
$ df -h
/dev/mapper/ubuntu--vg-root   48G   21G   26G  45% /

Two of the pages that helped a lot:

[How to resize an Ubuntu 18.04 LVM disk][https://vander.host/knowledgebase/operating-systems/how-to-resize-an-ubuntu-18-04-lvm-disk/]
[Ubuntu VM in VirtualBox: How to increase the size of a disk and make small(er) exports for distribution][https://technology.amis.nl/2017/01/30/ubuntu-vm-virtualbox-increase-size-disk-make-smaller-exports-distribution/]