DEV Community: Yaroslav Polyakov

Your own Certificate Authority (CA) in ONE simple command

Yaroslav Polyakov — Mon, 29 Jan 2024 19:56:18 +0000

You can find many instructions how to make certificates with OpenSSL. OpenSSL is very good except... it's too complex.

Showcert is OpenSSL for humans, much simpler to use and it has gencert utility to generate certificates. Very easy.

Generate self-signed cert with simple command

gencert example.com www.example.com

This will make self-signed cert. Easy, isn't it?

Your own CA

Generate CA certificate (1 command):

gencert --ca MyCA

Thats all! Now, you can import generated MyCA.pem file to browsers and generate signed certificates for sites:

gencert --cacert MyCA.pem --cakey MyCA.key example.com

What could be simpler?

Install showcert and play for yourself.

Great design trick I learned in X.509 (critical extension)

Yaroslav Polyakov — Thu, 25 Jan 2024 16:24:17 +0000

Good software, file formats and network protocols are extendable. You may develop new version later, but it should interoperate with older version. Old client software must work with new server and vice versa. Old software must open files created by new version of software and vice versa.

Extensions are good for this, but usually it comes with problem: How should software work, if it finds unknown extension? Often answer is simple - "just ignore it". Skip unknown extension and process file/request. (Same as your web app does if HTTP request has unknown header - it just does not looks at it)

But sometimes this approach will fail. Sometimes simple ignoring extension will not work (especially in security area). And here comes "critical extension". Extension has some identified and simple boolean flag "critical".

If software founds unknown extension it looks for flag. if not critical - ignore extension and go on. If critical - sorry, this software can not process this file/request (and it's very good if old software knows when it can process new data and when it should not try).

https://datatracker.ietf.org/doc/html/rfc5280

Simple decorator to retry after exception

Yaroslav Polyakov — Mon, 18 Dec 2023 19:30:29 +0000

Typical situation: you have unreliable function, for example, doing HTTP requests and getting exception from time to time. Maybe connection went down temporary or 429 Too Many requests problem, or remote service itself is temporary broken.

One simple decorator solves problem:

import functools

def retry(num_times=1):
    def decorator_repeat(func):
        @functools.wraps(func)
        def wrapper_repeat(*args, **kwargs):
            last_exception = None
            for n in range(num_times):
                try:
                    r = func(*args, **kwargs)
                except Exception as e:
                    last_exception = e
                    # handle exception. log it or just pass
                    print(f"Try #{n} got exception {e}, but we will retry")
                else:
                    return r
            raise last_exception
        return wrapper_repeat
    return decorator_repeat

Lets use it with requests:

import requests

@retry(10)
def myhttp():
    r = requests.get('https://httpbin.org/status/200,500,429')
    r.raise_for_status()
    print(r.status_code)

myhttp()

Results:

Try #0 got exception 429 Client Error: TOO MANY REQUESTS for url: https://httpbin.org/status/200,500,429, but we will retry
Try #1 got exception 500 Server Error: INTERNAL SERVER ERROR for url: https://httpbin.org/status/200,500,429, but we will retry
Try #2 got exception 500 Server Error: INTERNAL SERVER ERROR for url: https://httpbin.org/status/200,500,429, but we will retry
200

This way you are immune against temporary problems which could be solved by retry (maybe with time.sleep(10)) but you will still get exception from long-time problems. Change URL in myhttp() to https://httpbin.org/status/500 to see this.

apache2 allows CORS with credentials for any address

Yaroslav Polyakov — Thu, 12 Oct 2023 08:51:00 +0000

You cannot use '*' in Access-Control-Allow-Origin and use Access-Control-Allow-Credentials at same time. And this is actually makes sense, but during development this dirty hack is useful (for apache2):

SetEnvIf Origin "^http(s)?://.*$" REQUEST_ORIGIN=$0
Header always set Access-Control-Allow-Origin %{REQUEST_ORIGIN}e env=REQUEST_ORIGIN
Header always set Access-Control-Allow-Credentials true

How it work in action (I'm using httpie instead of curl):

Example:

$ http -ph POST https://example.com/ Origin:https://google.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://google.com
Connection: Keep-Alive
Content-Encoding: gzip
Content-Length: 1039
Content-Type: text/html
Date: Thu, 12 Oct 2023 08:50:33 GMT
ETag: "9a1-6020521d58f80-gzip"
Keep-Alive: timeout=5, max=100
Last-Modified: Thu, 03 Aug 2023 13:55:26 GMT
Server: Apache/2.4.56 (Debian)
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: disabled

If you want it only for specific Origins:

    SetEnvIf Origin "^https?://(example.com|www.example.com)$" GOODORIGIN=$0
    Header set Access-Control-Allow-Origin %{GOODORIGIN}e env=GOODORIGIN
    Header set Access-Control-Allow-Credentials "true" env=GOODORIGIN
    Header merge Vary Origin

Mix static & client-side rendering on same page with SvelteKit

Yaroslav Polyakov — Wed, 18 Jan 2023 16:08:39 +0000

I love static websites, because they are very fast and very good for SEO, google can 'read' it much better. But these days you need JavaScript almost everywhere.

And I like Svelte as very easy to learn and use lightweight framework.

I've found out, that it's not very simple to mix static rendering and CSR on same page. But possible. I will use time (JavaScript Date() result) as example data, but in production it can be your data, e.g. "BFGoodrich tyre" or "Red apples" or "War and Peace by Leo Tolstoy".

Our example page will print very simple text:

Date: Wed, 18 Jan 2023 15:49:03 GMT
Build: Wed, 18 Jan 2023 15:49:01 GMT

First line (Date:) is content you calculated on client-side, when page is loaded. Second line (Build:) is build-time, something you calculated during npm run build which must be visible to search engine.

Create project

npm create svelte@latest hybrid
cd hybrid
npm install

Enable static adapter

Install it: npm i -D @sveltejs/adapter-static

In svelte.config.js disable adapter-auto and use adapter-static:

import adapter from '@sveltejs/adapter-static';

Now you can generate static website with npm run build and preview it with npm run preview.

src/routes/+page.server.js

export const prerender = true
export function load() {
    return {
        t: new Date().toUTCString()
    }
}

src/routes/+page.svelte:

<script>
    import { browser } from '$app/environment';

    export let data;
    var buildtime = data.t
</script>

Date: {#if browser}{new Date().toUTCString()}{/if}
<br>
Build: {buildtime}

Result

run: npm run build && npm run preview and open http://localhost:4173/ you will see result. If you will reload page in a second, Date will change, but Build will stay same.

Also, if you will open build/index.html you will see there:

...
Date: 
<br>
Build: Wed, 18 Jan 2023 15:58:18 GMT
...

As you see, our statically generated content is clearly visible in page.

Explanation

in +page.server.js we set prerendering=true to generate static HTML site (with JS) as a result. And we return data with current Date(). We must convert it to string here, because client browser may have different timezone and Build: date will be "flashing" (initially it will have date in our timezone, and in fraction of second it will be overwritten). Not nice.

in +page.svelte we use this data. Also we conditionally display new Date() (I used .toUTCString() here just to have both dates to look similar, it's not really needed).

Without {#if} you'd get one date built-in HTML (displayed immediately when page loaded) and other date will overwrite it. Not nice visual effect. With {#if} we suppress first date and display only CSR date over empty page space.

Using this method you can combine benefits for static and client-side rendered content at any SvelteKit page.

Shell oneliner to generate passwords

Yaroslav Polyakov — Thu, 12 Jan 2023 10:38:19 +0000

# 10 character random string from specified alphabet
< /dev/urandom tr -dc '[:graph:]' | head -c10; echo

Bash allows redirections at any part of command, so < file command args above is same as command args < file.

Examples:

KrvJz?TgbO
T[7/9Lg9L9
bp[je2Og*l
93:m_Ss$z5
C}Z=-Nj-mn
mG<85C>!i~
%>7/]AWpMd

If you do not want so many special characters, you may use something like: '[:alnum:]_$#%-'

JCPHawXEvr
orq#0S7cUk
82H7g_cHtY
um-#ACmny4
dYQFqGsX_d

Drop [almost] all outbound mail in postfix for tests

Yaroslav Polyakov — Thu, 12 Jan 2023 10:04:40 +0000

Why to drop mail?

While troubleshooting mailman installation, I've come to a problem: I need to do many tests, but I do not want to disturb recipients of maillist with dozens of test messages. I do not know, if mailman3 has any feature for this kind of testing, but postfix has.

How to drop postfix mail except to whitelisted addresses

First, make sure, you have transport map enabled.

Usually, this line could look like this:

transport_maps = hash:/etc/postfix/transport

Now, create other transport file, my /etc/postfix/blackhole:

mydomain.net :
mymailbox@gmail.com :
* discard:

With this config postfix will deliver to *@mydomain.net and to mymailbox@gmail.com.

run postmap blackhole to generate /etc/postfix/blackhole.db.

add it to postfix transport with comma (if you already has transport map):

transport_maps = hash:/path/to/other/transport/map, hash:/etc/postfix/blackhole

(or as single element, if you had no transport_maps).

Restart postfix: postfix reload. You're ready.

Testing

I sent few mails and here is what I have in logs:

Jan 12 09:53:17 list2 postfix/smtp[62239]: DB97418C0808: to=<mymailbox@gmail.com>, relay=gmail-smtp-in.l.google.com[173.194.76.27]:25, delay=0.41, delays=0.03/0.02/0.14/0.22, dsn=2.0.0, status=sent (250 2.0.0 OK  1673517197 g15-20020a056000118f00b002bdd88acc3dsi514695wrx.503 - gsmtp)

Jan 12 09:53:34 list2 postfix/discard[62251]: 7943118C0808: to=<test@test.com>, relay=none, delay=0.02, delays=0.02/0/0/0, dsn=2.0.0, status=sent (test.com)

First is log line for real delivery (relay= remote mailserver) and second is log line for blackhole delivery, relay=none).

How to get country by IP address

Yaroslav Polyakov — Mon, 09 Jan 2023 11:59:49 +0000

ipdeny.com provides nice free service for this. Download country block for any country you want and use it in your code.

country block is simple CIDR list, e.g. for Afghanistan it starts with:

23.88.192.0/19
43.230.209.0/24
43.250.136.0/22
...

Every C program has main()? (False)

Yaroslav Polyakov — Wed, 04 Jan 2023 22:09:28 +0000

actually, _start is real magic word.

nomain.c:

#include <stdio.h>

extern void _exit(register int);

int _start(){
    printf("Hello world!\n");
    _exit(0);
}

$ gcc -o nomain nomain.c -nostartfiles
$ ./nomain 
Hello world!

apt-file: which deb package to install to get X?

Yaroslav Polyakov — Wed, 04 Jan 2023 09:05:29 +0000

You need to get file X (e.g. hugo), what package to install?

Long time ago, I got used to visit packages.debian.org, scroll down to second search form "Search the contents of packages" and search there.

It's much easier to search with apt-file right from CLI:
Install:

apt install apt-file
apt-file update

Now search:

$ apt-file search hugo
fish-common: /usr/share/fish/completions/hugo.fish
gargoyle-free: /usr/lib/gargoyle/hugo
hugo: /usr/bin/hugo
hugo: /usr/share/bash-completion/completions/hugo
hugo: /usr/share/doc/hugo/changelog.Debian.amd64.gz
...

but if we will search for 'gcc', we will find 170000+ results because apt-find looks for substring match. (and 30k results for 'mc' substring). We should use -x/--regexp search:

$ apt-file -x search bin/gcc$
gcc: /usr/bin/gcc                         
icecc: /usr/lib/icecc/bin/gcc
nvidia-cuda-toolkit: /usr/lib/nvidia-cuda-toolkit/bin/gcc
pentium-builder: /usr/bin/gcc

fail2ban regex works as CLI param, but not as file (Solution)

Yaroslav Polyakov — Sun, 01 Jan 2023 15:29:05 +0000

When writing new fail2ban filter, I see my regexes are working well in CLI test, but not from filter.d file.

Running from CLI (success):

root@liste:/etc/fail2ban/filter.d# fail2ban-regex /var/log/apache2/access_log '<HOST>.*/cgi-bin/mailman/subscribe/.*'

Running tests
=============

Use   failregex line : <HOST>.*/cgi-bin/mailman/subscribe/.*
Use         log file : /var/log/apache2/access_log
Use         encoding : UTF-8


Results
=======

Failregex: 7530 total
|-  #) [# of hits] regular expression
|   1) [7530] <HOST>.*/cgi-bin/mailman/subscribe/.*
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [7796] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-

Lines: 7796 lines, 0 ignored, 7530 matched, 266 missed
[processed in 1.68 sec]

Missed line(s): too many to print.  Use --print-all-missed to print all 266 lines

7.5K matched. Great! now, put this regex into file:

[INCLUDES]
before = common.conf
[Definition]
failregex = <HOST>.*/cgi-bin/mailman/subscribe/.*
ignoreregex =

And.... 0 matched! :-(

root@liste:/etc/fail2ban/filter.d# fail2ban-regex /var/log/apache2/access_log /etc/fail2ban/filter.d/mailman-subscribe.conf 
...
Lines: 7796 lines, 0 ignored, 0 matched, 7796 missed

Solution: just add empty datepattern = to rule.

And now:

root@liste:/etc/fail2ban/filter.d# fail2ban-regex /var/log/apache2/access_log /etc/fail2ban/filter.d/mailman-subscribe.conf 
...
Lines: 7796 lines, 0 ignored, 7530 matched, 266 missed

Svelte-kit (vite) with HMR over NGINX (Solution)

Yaroslav Polyakov — Sun, 25 Dec 2022 16:25:52 +0000

I need to use svelte-kit over nginx reverse proxy (because of third-party cookies which requires HTTPS). If you will just make simple proxy on nginx, pages will load, but HMR will not work. You will get this error message on console:

[vite] failed to connect to websocket.
your current setup:
  (browser) example.com/ <--[HTTP]--> localhost:5173/ (server)
  (browser) example.com:/ <--[WebSocket (failing)]--> localhost:5173/ (server)
Check out your Vite / network configuration and https://vitejs.dev/config/server-options.html#server-hmr .
(anonymous) @ client.ts:48
(anonymous) @ client.ts:99

Problem is you cannot easily proxy HTTP + websocket over one nginx server. But solution is simple:

add server.hmr.clientPort to vite.config.js:

import { sveltekit } from '@sveltejs/kit/vite';

/** @type {import('vite').UserConfig} */
const config = {
    plugins: [sveltekit()],
    server: {
        hmr: {
            clientPort: 5111
        }
    }
};

export default config;

This will make wss to work over different port, so you can use different configuration for HTTP and websockets in nginx:

server {
    listen 443 ssl;
    server_name example.com;

    ssl_certificate     /etc/ssl/private/cert.pem;
    ssl_certificate_key /etc/ssl/private/key.pem;

    location / {
        proxy_pass http://127.0.0.1:5173;
        proxy_set_header Host $host;
    }
}

server {
    listen 5111 ssl;

    ssl_certificate     /etc/ssl/private/cert.pem;
    ssl_certificate_key /etc/ssl/private/key.pem;

    location / {
        proxy_pass http://127.0.0.1:5173;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;

    }
}

Now hot module reloading (HMR) works over nginx!