DEV Community: Tomoyuki KOYAMA

cheat sheet for go mod package management

Tomoyuki KOYAMA — Fri, 25 Oct 2024 02:12:00 +0000

This article describes how to use package management commands of go mod.

Project setup

go mod init github.com/YOUR_USER/YOUR_PROJECT

Fixing `go.mod`

This command provides deletion of unused packages and insertion of required packages from go.mod.

go mod tidy

Install packages which have dependency

This command represents installing packages with dependency resolution.

go get .

Install a package

This command represents labstack/echo installation.

go get github.com/labstack/echo/v4@latest

Update packages

This command is for updating the specific package github.com/labstack/echo/v4.

Only update the package to the latest version:

go get github.com/labstack/echo/v4@latest

Update the package with dependency packages to the latest version:

go get -u github.com/labstack/echo/v4@latest

Reference

Download package from GitHub release

Tomoyuki KOYAMA — Mon, 21 Oct 2024 06:38:08 +0000

This article describes the method for package download from GitHub release by CLI.

Prerequisite

GitHub CLI
unzip command

Steps

# Get the package which is named 'my_package'
PKG_NAME=$(gh release list --limit 100 --order desc --json tagName --jq '[ .[] | select(.tagName | startswith("my_package")) ] | .[0].tagName')

# Download the package
gh release download "$PKG_NAME" --pattern '*.zip' --output "${PKG_NAME}.zip"

# Unpack
unzip "${PKG_NAME}.zip"

Use imgp on macOS with Docker

Tomoyuki KOYAMA — Mon, 07 Oct 2024 00:21:50 +0000

Introduction

When I tried to use imgp which is a CLI tool for image processing in Python on macOS (M3), Installation of imgp failed as follows.

clang: warning: argument unused during compilation: '-msse4' [-Wunused-command-line-argument]
src/libImaging/AlphaComposite.c:28:5: error: use of undeclared identifier '__m128i'
    __m128i mm_max_alpha = _mm_set1_epi32(255);
    ^
src/libImaging/AlphaComposite.c:29:5: error: use of undeclared identifier '__m128i'
    __m128i mm_max_alpha2 = _mm_set1_epi32(255 * 255);
    ^
src/libImaging/AlphaComposite.c:30:5: error: use of undeclared identifier '__m128i'
    __m128i mm_zero = _mm_setzero_si128();
    ^
src/libImaging/AlphaComposite.c:31:5: error: use of undeclared identifier '__m128i'
    __m128i mm_half = _mm_set1_epi16(128);
    ^
src/libImaging/AlphaComposite.c:32:5: error: use of undeclared identifier '__m128i'
    __m128i mm_get_lo = _mm_set_epi8(
    ^
src/libImaging/AlphaComposite.c:34:5: error: use of undeclared identifier '__m128i'
    __m128i mm_get_hi = _mm_set_epi8(
    ^
src/libImaging/AlphaComposite.c:137:13: error: use of undeclared identifier '__m128i'
            __m128i mm_dst, mm_dst_lo, mm_dst_hi;
            ^
src/libImaging/AlphaComposite.c:138:13: error: use of undeclared identifier '__m128i'
            __m128i mm_src, mm_src_hi, mm_src_lo;
            ^
src/libImaging/AlphaComposite.c:139:13: error: use of undeclared identifier '__m128i'
            __m128i mm_dst_a, mm_src_a, mm_out_a, mm_blend;
            ^
src/libImaging/AlphaComposite.c:140:13: error: use of undeclared identifier '__m128i'
            __m128i mm_coef1, mm_coef2, mm_out_lo, mm_out_hi;
            ^
src/libImaging/AlphaComposite.c:143:13: error: use of undeclared identifier 'mm_dst'
            mm_dst = _mm_loadu_si128((__m128i *) &dst[x]);
            ^
src/libImaging/AlphaComposite.c:143:22: error: call to undeclared function '_mm_loadu_si128'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
            mm_dst = _mm_loadu_si128((__m128i *) &dst[x]);
                    ^
src/libImaging/AlphaComposite.c:143:48: error: expected expression
            mm_dst = _mm_loadu_si128((__m128i *) &dst[x]);
                                                ^
src/libImaging/AlphaComposite.c:143:39: error: use of undeclared identifier '__m128i'
            mm_dst = _mm_loadu_si128((__m128i *) &dst[x]);
                                    ^
src/libImaging/AlphaComposite.c:145:13: error: use of undeclared identifier 'mm_dst_lo'
            mm_dst_lo = _mm_unpacklo_epi8(mm_dst, mm_zero);
            ^
src/libImaging/AlphaComposite.c:145:25: error: call to undeclared function '_mm_unpacklo_epi8'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
            mm_dst_lo = _mm_unpacklo_epi8(mm_dst, mm_zero);
                        ^
src/libImaging/AlphaComposite.c:145:43: error: use of undeclared identifier 'mm_dst'
            mm_dst_lo = _mm_unpacklo_epi8(mm_dst, mm_zero);
                                        ^
src/libImaging/AlphaComposite.c:145:51: error: use of undeclared identifier 'mm_zero'
            mm_dst_lo = _mm_unpacklo_epi8(mm_dst, mm_zero);
                                                ^
src/libImaging/AlphaComposite.c:147:13: error: use of undeclared identifier 'mm_dst_hi'
            mm_dst_hi = _mm_unpackhi_epi8(mm_dst, mm_zero);
            ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pillow-simd
Failed to build pillow-simd
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pillow-simd)

This error is a known issue as below.

https://github.com/uploadcare/pillow-simd/issues/43

Solution

Building on macOS (M3) is not supported, so that I tried to install imgp using Docker. The installation process is described below.

(1) Start docker container

docker run-v $PWD:/work --rm --platform linux/amd64 -it python:3.12-slim /bin/bash

(2) Install packages for building

apt update
apt install build-essential zlib1g-dev libjpeg-dev

(3) Install imgp

pip install imgp

(4) Verify installed binary

root@cc0fa40c9237:~# imgp
usage: imgp [-h] [-x res] [-o deg] [-a] [-c] [-e] [-f] [-H] [-i] [-k] [-m] [-M res] [-n] [-N] [-O] [-P] [-q N] [-r] [-s byte] [-w] [-d] [PATH ...]

Resize, rotate JPEG and PNG images.

positional arguments:
  PATH                  source file or dir [default: current dir]

options:
  -h, --help            show this help message and exit
  -x res, --res res     output resolution in HxV or percentage
  -o deg, --rotate deg  rotate clockwise by angle (in degrees)
  -a, --adapt           adapt to resolution by orientation [default: off]
  -c, --convert         convert PNG to JPG format [default: off]
  -e, --eraseexif       erase exif metadata [default: off]
  -f, --force           force to exact specified resolution [default: off]
  -H, --hidden          include hidden (dot) files [default: off]
  -i, --includeimgp     re-process _IMGP files. * RISKY: refer to docs
  -k, --keep            skip (honors -c or --pr) images matching specified
                        H or V or --res=100 [default: off]
  -m, --mute            operate silently [default: informative]
  -M res, --minres res  min resolution in HxV or percentage of --res to resize
  -n, --enlarge         enlarge smaller images [default: off]
  -N, --nearest         use nearest neighbour interpolation for PNG [default: antialias]
  -O, --optimize        optimize the output images [default: off]
  -P, --progressive     save JPEG images as progressive [default: off]
  -q N, --quality N     quality factor (N=1-95, JPEG only) [default: 75]
  -r, --recurse         process non-symbolic dirs recursively [default: off]
  -s byte, --size byte  minimum size to process an image [default: 1024]
  -w, --overwrite       overwrite source images [default: off]
  -d, --debug           enable debug logs [default: off]

Version 2.9
Copyright © 2016-2023 Arun Prakash Jana <engineerarun@gmail.com>
License: GPLv3
Webpage: https://github.com/jarun/imgp

Note

$ uname -a
Darwin weizen.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:45:48 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8122 arm64

$ docker -v
Docker version 26.1.1, build 4cf5afa

Cheat sheet for development in Python

Tomoyuki KOYAMA — Sun, 28 Apr 2024 08:12:24 +0000

Snippet

Environemnt Variables

import os

os.getenv("ENV_VAR_NAME", "default_value")

os — Miscellaneous operating system interfaces — Python 3.12.3 documentation

Command Line Argument

import argparse

parser = argparse.ArgumentParser(description='Parse command line options.')
parser.add_argument('integers', metavar='c', type=int, nargs='+',
                    help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
                    const=sum, default=max,
                    help='sum the integers (default: find the max)')

args = parser.parse_args(['--sum', '7', '-1', '42'])
print(args.accumulate(args.integers))

argparse — Parser for command-line options, arguments and sub-commands — Python 3.12.3 documentation

Random number

import random
x = random.randint(0, 100) # 0 <= x <= 100

Logging

from logging import basicConfig, getLogger, DEBUG

# put in entry point file
CHAR_GREEN = '\033[32m'
CHAR_RESET = '\033[0m'
FORMAT = f"{CHAR_GREEN}%(asctime)s %(levelname)s %(name)s {CHAR_RESET}: %(message)s"
basicConfig(level=INFO, format=FORMAT)

# put in all files
logger = getLogger(__name__)

# logging code
logger.info('hello')

Ref: ログ出力のための print と import logging はやめてほしい #Python - Qiita

Packages

Code formatter

Install packages

pip install \
  black \
  autopep8 \
  isort \
  ;

Run installed packages

black .
autopep8 --recursive --in-place --aggressive --aggressive .
isort .

Package Manager

Poetry

Setup project

cd <project_root>
poetry init

Add a package

poetry add <package_name>

Install dependencies using poetry

poetry install

venv

Create venv

python -m venv .venv

Activate venv

source .venv/bin/activate

Install package

pip install <package_name>

pyenv

Check installed version list

pyenv versions

Get version list

pyenv install -l

Install specific version

pyenv install <python_version>

Get global version

pyenv global

Get local version

pyenv local

Create URL shortener by Cloudflare Workers

Tomoyuki KOYAMA — Fri, 28 Jul 2023 11:33:50 +0000

Introduction

In order to use a domain name which was unused, I developed a web service of URL shortener by myself.

Choosing Cloud Providers and its Services

The service candidates for use include container-based service or serverless computing services. URL shortener service has a simple processing logic and doesn't require storing data as in-memory data. Therefore, serverless computing services are chosen.

I have experience for using public cloud services such as AWS, GCP, and Azure. In contrast, I have no experience for using the web services which is provided by Cloudflare. Therefore, this project uses Cloudflare Workers as the service platform.
The requirement for this project is to store URL on databases. Cloudflare provides Cloudflare Workers KV as key-value pair store. These services are provided as a free tier service.

System Architecture

      Clients
         |
         v
+-----------------+     +--------------------+
|Cloudflare Worker| --> |Cloudflare Worker KV|
+-----------------+     +--------------------+

Codes

Following codes are deployed on Cloudflare Workers as a function. Then, you create a key-value store on Cloudflare Workers KV, and this is named as MY_KV.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  if (request.method === "GET") {
    return readRedirect(request)
  } else if (request.method === "POST") {
    return createRedirect(request)
  } else {
    return new Response("Forbidden", { status: 403 })
  }
};

async function readRedirect(request) {
  const url = new URL(request.url);
  const req_path = url.pathname;

  const value = await MY_KV.get(req_path);
  if (value === null) {
    console.log("Not Found", req_path);
    return new Response("Not Found", { status: 404 });
  }

  console.log("Redirect", req_path);
  const statusCode = 301;
  return Response.redirect(value, statusCode);
};

async function createRedirect(request) {
  const url = new URL(request.url);
  const req_path = url.pathname;

  const secret_token = "xxxxx-xxxxxx-xxxxxx-xxxxxx";
  if (req_path !== "/token/" + secret_token) {
    return new Response("Invalid Token", { status: 401 });
  }

  const content_type = request.headers.get("content-type");
  if (!content_type.includes("application/json")) {
    return new Response("Invalid Content Type", { status: 400 });
  }
  const request_body = await request.json();
  const redirect_from_path = request_body["from_path"];  // "/foo"
  const redirect_to_url = request_body["to_url"]; // http://example.com
  await MY_KV.put(redirect_from_path, redirect_to_url);
  return new Response("https://1ch.dev" + redirect_from_path);
};

Usage

iOS Shortcuts

CLI commands

This code is supported for .zshrc.

function tiny() {
    read to_url
    echo $to_url | grep 'http\(s\)\?://' || {
        echo "Invalid URL"
        return 1
    }

    from_path="/cli-$(date +%s)"
    curl -XPOST -H 'Content-Type: application/json' \
      -d "{\"from_path\": \"$from_path\", \"to_url\": \"$to_url\"}" \
      https://1ch-dev.xxxxxxxx.workers.dev/token/xxxxxxxx \
      | pbcopy
}

Usage:

echo "http://example.com/" | tiny

Git failure after updating to macOS Ventura

Tomoyuki KOYAMA — Thu, 02 Feb 2023 05:36:33 +0000

Environment

macOS Ventura
MacBook Air with M1 chip

Problem

I faced the following error after updating macOS Ventura from Monterey.

$ git
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

Solution

One of the solutions for the error is to execute the command as follows.

$ xcode-select --reset

Reference

Git failure after Ventura upgrade : mac

Installation of Embulk on Ubuntu 20.04

Tomoyuki KOYAMA — Fri, 11 Nov 2022 09:04:35 +0000

Environment

Ubuntu 20.04.3 LTS

Instruction

(1) Check Java version

$ java -version

Command 'java' not found, but can be installed with:

sudo apt install default-jre              # version 2:1.11-72, or
sudo apt install openjdk-11-jre-headless  # version 11.0.17+8-1ubuntu2~20.04
sudo apt install openjdk-13-jre-headless  # version 13.0.7+5-0ubuntu1~20.04
sudo apt install openjdk-16-jre-headless  # version 16.0.1+9-1~20.04
sudo apt install openjdk-17-jre-headless  # version 17.0.5+8-2ubuntu1~20.04
sudo apt install openjdk-8-jre-headless   # version 8u352-ga-1~20.04

Not installed java

(2) Install Java 8

EMbulk only operates with Java 8.

sudo apt install openjdk-8-jre-headless

(3) Check Java version

$ java -version
openjdk version "1.8.0_352"
OpenJDK Runtime Environment (build 1.8.0_352-8u352-ga-1~20.04-b08)
OpenJDK 64-Bit Server VM (build 25.352-b08, mixed mode)

(4) Install Embulk

curl --create-dirs -o ~/.embulk/bin/embulk -L "https://dl.embulk.org/embulk-latest.jar"
chmod +x ~/.embulk/bin/embulk
echo 'export PATH="$HOME/.embulk/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

Ref: Embulk

(5) Check embulk command

which embulk

Check IP address on ESP32 via serial console

Tomoyuki KOYAMA — Sun, 09 Oct 2022 08:49:11 +0000

Connect to ESP32 via serial console

screen /dev/cu.usbserial-0001 115200

Check IP address

>>> import network
>>> sta_if = network.WLAN(network.STA_IF)
>>> sta_if.ifconfig()
('192.168.0.14', '255.255.255.0', '192.168.0.1', '192.168.0.1')

The result represents a list that takes as follows: IP address, Subnet mask, Default gateway, DNS server address.

Reference

4. Network basics — MicroPython 1.8.7 documentation

MicroPython setup on ESP32 with macOS (M1 Mac)

Tomoyuki KOYAMA — Sun, 09 Oct 2022 08:18:08 +0000

Hardware devices

ESP32
- ESP32-WROOM-32D
MacBook Air 2020
- OS: macOS 12.0.1
- CPU: Apple M1
- RAM: 16GB
USB Type-C → USB Type-A Adapter
- DST-C02WH

Setup guides

(1) Install esptool

Check pip version



$ pip --version
pip 21.3.1 from /opt/homebrew/lib/python3.9/site-packages/pip (python 3.9)

Install esptool



$ pip install esptool

Check esptool installation



$ which esptool.py
/opt/homebrew/bin/esptool.py

(2) Discover ESP32

esptool automatically tries to connect to the serial port from "v2.4".



$ esptool.py flash_id
esptool.py v3.2
Found 3 serial ports
Serial port /dev/cu.wlan-debug
/dev/cu.wlan-debug failed to connect: [Errno 16] could not open port /dev/cu.wlan-debug: [Errno 16] Resource busy: '/dev/cu.wlan-debug'
Serial port /dev/cu.usbserial-0001
Connecting..............
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting........................
Detecting chip type... ESP32
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 4c:eb:d6:75:49:9c
Uploading stub...
Running stub...
Stub running...
Manufacturer: 5e
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

(3) Connect to ESP32 with serial port

You can connect to ESP32 with serial port.
The destination tty device is displayed on the command result esptool.py flash_id.
For example, the destination tty device is /dev/cu.usbserial-0001 in the above result.
The baudrate is set 115200



screen /dev/cu.usbserial-0001 115200

(4) Erase flash memory data in ESP32



$ esptool.py --chip esp32 --port /dev/cu.usbserial-0001 erase_flash
esptool.py v3.2
Serial port /dev/cu.usbserial-0001
Connecting..........
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 4c:eb:d6:75:49:9c
Uploading stub...
Running stub...
Stub running...
Erasing flash (this may take a while)...
Chip erase completed successfully in 12.8s
Hard resetting via RTS pin...

(5) Write firmware to ESP32

Download firmware and write the firmware to ESP32

https://micropython.org/download/esp32/

Download IDF 3.x that supports BLE and Wi-Fi

Write firmware as below



$ esptool.py --chip esp32 --port /dev/cu.usbserial-0001 --baud 460800 write_flash -z 0x1000 ~/Downloads/esp32-idf3-20210202-v1.14.bin
esptool.py v3.2
Serial port /dev/cu.usbserial-0001
Connecting..............
Chip is ESP32-D0WDQ6 (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 4c:eb:d6:75:49:9c
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Flash will be erased from 0x00001000 to 0x00161fff...
Compressed 1445632 bytes to 925476...
Wrote 1445632 bytes (925476 compressed) at 0x00001000 in 22.8 seconds (effective 507.9 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

(6) Connect to ESP32 with serial connection



$ screen /dev/cu.usbserial-0001 115200

When you want to exit the console, type this keycode: Ctrl + A → Ctrl + K .

Reference

MacでESP32のLチカ（MicroPython/ターミナル版） - Qiita

Fail to send ICMP packets by ping command on Synology NAS

Tomoyuki KOYAMA — Sat, 01 Oct 2022 14:13:18 +0000

When you cannot send ICMP packets by ping command on Synology NAS, you have to set 'sudo' as the command prefix.

https://community.synology.com/enu/forum/2/post/129235

e.g. sudo ping 1.1.1.1

Nginx Ingress returns only a few metrics on Kuberntes.

Tomoyuki KOYAMA — Sat, 01 Oct 2022 14:11:13 +0000

Situation

To visualize http requests on grafana, I installed Nginx Ingress and Prometheus in Kubernetes Cluster (RKE2). The installation steps are based on official guide as follows.

https://kubernetes.github.io/ingress-nginx/

The similar problem is reported on GitHub.

https://github.com/kubernetes/ingress-nginx/issues/5755

The figure shows this situation.

Cause

This problem is caused by not specifying the 'host' parameter in yaml for ingress controller.

The metrics server in nginx ingress only return a few metric about the default host * (generally virtual host). Thus, few metrics display on grafana.

Solution

When you create ingress resource, you set 'host' parameter in yaml. Then, you can check the metrics (such as nginx_ingress_controller_requests) in grafana.

HorizontalPodAutoscaler does not return TARGETS metrics and returns '' on kubectl.

Tomoyuki KOYAMA — Sat, 01 Oct 2022 14:07:07 +0000

I had faced the error: HorizontalPodAutoscaler does not return TARGETS metrics and returns '<unknown>' on kubectl.

Environment

Kubrenetes engine: RKE2 (v1.22.6+rke2r1) x 3 nodes

Manifest: https://github.com/cdsl-research/text-checker/blob/c351a2ebe042560842913ca9501949fc90198252/kubernetes/scaleOut.yml

Solution

First, I tried to run the following command.

kubectl get --raw /apis/metrics.k8s.io/v1beta1"

The command returns an error.

Second, I installed metrics-server to the cluster.

https://github.com/kubernetes-sigs/metrics-server

I retry to run kubectl get --raw /apis/metrics.k8s.io/v1beta1. The command runs successfully.

Third, I fixed kubernetes manifests as summarised below.

Set 'requests' parameter per container in deployment
Fix resource specs for HPA

https://github.com/cdsl-research/text-checker/commit/e13c178eaa4e1ab140462bf658a4e477f87babb1

https://github.com/cdsl-research/text-checker/commit/1bcc0dc21fc5f3fd8a7b42064e582e23cff303ab

Finally, the command 'kubectl get hpa' returns values 'TARGETS' successfully.

DEV Community: Tomoyuki KOYAMA

cheat sheet for go mod package management

Project setup

Fixing go.mod

Install packages which have dependency

Install a package

Update packages

Reference

Download package from GitHub release

Prerequisite

Steps

Use imgp on macOS with Docker

Introduction

Solution

Note

Cheat sheet for development in Python

Snippet

Environemnt Variables

Command Line Argument

Random number

Logging

Packages

Code formatter

Package Manager

Poetry

venv

pyenv

Create URL shortener by Cloudflare Workers

Introduction

Choosing Cloud Providers and its Services

System Architecture

Codes

Usage

iOS Shortcuts

CLI commands

Git failure after updating to macOS Ventura

Environment

Problem

Solution

Reference

Installation of Embulk on Ubuntu 20.04

Environment

Instruction

Check IP address on ESP32 via serial console

MicroPython setup on ESP32 with macOS (M1 Mac)

Hardware devices

Setup guides

(1) Install esptool

(2) Discover ESP32

(3) Connect to ESP32 with serial port

(4) Erase flash memory data in ESP32

(5) Write firmware to ESP32

(6) Connect to ESP32 with serial connection

Reference

Fail to send ICMP packets by ping command on Synology NAS

Nginx Ingress returns only a few metrics on Kuberntes.

Situation

Cause

Solution

HorizontalPodAutoscaler does not return TARGETS metrics and returns '' on kubectl.

Environment

Solution

Fixing `go.mod`