DEV Community: Santiago Zarate

Extracting postfix journal logs for fun, profit and unbanning

Santiago Zarate — Thu, 02 Nov 2023 00:00:00 +0000

From journal to .json

In order to convert from your journal log to json, so it is easily parseable, jq offers an option that allows you to run a filter, only until the end of the file:

    Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and
    run the filter just once.

This allows you to save the file directly, ready to be processed by your favorite tools, here’s what I used:

journalctl -u postfix.service --since yesterday -g "denied" --output json | jq -s "." > data/log.json

Enter Perl

Now, because I’ve been using Perl for ${deity} knows how long (I wrote my first Perl script at the end of the 90’s), naturally, is my language of choice for quick things where my knowledge of bash isn’t going to cut it:

First I want to load my file, I’m going to rely on Mojo, specifically Mojo::Collection and Mojo::JSON for this as I’m familiar with both, also, if I wan’t to dig a bit into what’s inside my collections, I can always do:

use Mojo::Util qw(dumper);

say dumper $collection->to_array;

But I digress, back to business

The real stuff

This piece of code filters for me, what it reads from a file (I’m doing $_= path(shift); for convenience)

my $file = Mojo::Collection->new(decode_json($_->slurp))->flatten;

// Filter using Mojo::Collection::grep to have a new collection with the data I'm interested in
my $filtered = $file->grep(sub{ 
        $_->{MESSAGE} =~ /denied/
    });

Now that I have the elements on a single array (of course, if I’m looking at a file over a gigabyte, likely I’d look into putting this inside some sort of database, PostgreSQL for instance, has excellent Json support), it’s time to do something with it:

// get anything that looks like a hostname before, and get the ip address
// example: NOQUEUE: reject: RCPT from ns2.pads.ufrj.br[146.164.48.5]: 554 5.7.1 <relaytest@antispam-ufrj.pads.ufrj.br>:
// I want to have ethe IP in a named group so I can later reference it with `$+{'IP'}`
my $regexp = qr{(.*[a-zA-Z0-9-._]+)\[(?<IP>.*)\]>?:.*};

Ideally (and for the future) I might want to filter in a different way, and capture different things, but you get the idea however today, we only want to know which ip addresses were rejected while testing our changes in our postfix’s configuration

$filtered->each(sub{
        if ($_->{'MESSAGE'} =~ /$regexp/ ){
            say "bash ./unban ".$+{'IP'};
        } else {
            warn "CANNOT GET IP: ".$_->{"MESSAGE"};
        }
    });

I have another script that does the unban, but for now I’m ok with copy&pasting :)

The full script is at: https://github.com/foursixnine/stunning-octo-chainsaw/blob/master/postfix-logparser/logparser.pl pull requests are welcome, and maybe in the future I move this to its own thing, but for now, that’s all folks.

Interoperabilidad de Obsidian con Linux, iOS y OSX

Santiago Zarate — Mon, 18 Apr 2022 00:00:00 +0000

Hace tiempo en el grupo de Developers Ve en el mundo, preguntaba si alguien había utilizado Obsidian. Considerándome un power user de Evernote, sé lo que necesito, y en los años que llevo buscando una alternativa que funcione en linux, tenia el problema de que muy pocas realmente funcionan bien o siquiera tienen soporte para linux.

Decidí irme por Obsidian, luego de considerar Notion y otras herramientas principalmente por los plugins, que tiene Zettelkasten como feature, esta pensado con las siguientes otras razones:

Es perfecto para Zettelkasten y para construir un segundo cerebro (Building a Second Brain), y tiene un sistema de tags simple, pero poderoso.
Soporte para Markdown by default.
Los archivos están en mis dispositivos, no en la nube by design.
Puedo elegir alternativas de respaldo, control de versiones y redundancia.
Soporte para Kanban Boards.
Soporte para Day Planners.
Soporte para Natural Language Dates para cosas like: @today, @tomorrow.
La interfaz es exactamente lo que me encanta de Sublime que es lo que había estado utilizando para tomar notas, junto a la aplicación de notas del teléfono y notas via el correo.
Tiene un VIM mode :D.
El roadmap promete.

Revisando varias cosas, y realmente investigando un poco, llegue al siguiente workflow:

Logre construir el workflow que hace lo que necesito:
- Vault en git en mi VPS, en una instancia propia de gitea, la data es mia.
- En Linux:
- El manejo de las distintas Vaults (Bóvedas) seria por git directamente.
  - Nextcloud como mecanismo de backup redundante en el NAS en casa via scsi en un rpi.
  - A mediano, o largo plazo:
  - La solución de NAS podría ser FreeNAS, TrueNAS o Synology
  - VPS como gateway, utilizando vpn con wireguard para mantener todo en una red privada.
- En OSX:
- Seria igual que en linux, por git pero con la diferencia de que las bóvedas estarían alojadas en una carpeta en iCloud.
- La llave ssh con permiso de escritura en el repo seria importada al keystore (ssh-add -K), para que no de problemas a la hora de pedir contraseñas.
- Queda pendiente revisar como hacer con las firmas de los commits con GPG, o maybe usando ssh para firmar commits
- En IOS, los vaults se estarían abriendo via iCloud, dejando por fuera el manejo con git, mientras se agrega el soporte en para ios/mobile en obsidian-git.

En un tiempo revisare este post, y actualizare seguramente a mi nuevo workflow… o hare una vista en retrospectiva de que pudo salir mejor, etc, sin embargo creo que la primera tarea que hare, sera escribir un plugin para poderlo integrar con la creación de posts de este blog, y utilizar el grafo de tags, que por ahora… se ve así:

![[images/blog_tag_cloud.png]]

How to test things with openQA without running your own instance

Santiago Zarate — Fri, 18 Jun 2021 00:00:00 +0000

Wait what?

Yes, there are couple of ways for you, the user, the contributor, the amazing human being who wants to improve the software that is used by millions, to write automated tests and have bots doing all the work for you, once you’ve signed a binding contract with the blood of an unicorn, and have obtained api keys for our public https://openqa.opensuse.org instance.

For now I will leave out the details on how to get those, but will rather point you to the #factory irc channel (or dischord), where you can get in touch with current admins, whom will be able to guide you better in the process.

I have the keys

You should get operator keys and they would look like this (more or less):

[openqa.opensuse.org]
key = 45ABCEB4562ACB04
secret = 4BA0003086C4CB95

Now let’s do this

I will assume that you’re using openSUSE Tumbleweed, instructions are similar for Leap, but if you’re looking for something more esoteric, check thebootstraping guide

Bring up a terminal or your favorite package manager, and install openQA-client, it will pull almost everything you will need

zypper in openQA-client

Once we’re here, we’ve gotta clone the git repo from the tests being ran in openqa.opensuse.org, to do that let’s go to a directory; let’s call it Skynet and create it in our user’s home. My user’s home is /home/foursixnine \o/ and use git to clone the test repository: https://github.com/os-autoinst/os-autoinst-distri-opensuse/.

cd $HOME
mkdir Skynet
cd Skynet
git clone https://github.com/os-autoinst/os-autoinst-distri-opensuse/

Now since we already cloned the test distribution, and also got the openQA client installed, all there is to do is:

1 - Hacking & Commiting 2 - Scheduling a test run

At this point we can do #1 fairly easy, but #2 needs a bit of a push (no pun intended), this is where we will need the API keys that we requested, in the beginning.

We will rely for now, on openqa-clone-custom-git-refspec which reads configuration parameters from “$OPENQA_CONFIG/client.conf”, “~/.config/openqa/client.conf” and “/etc/openqa/client.conf” (you can run openqa-cli –help to get more detailed info on this), for now open up your favorite editor and let’s create the directories and files we’ll need

mkdir ~/.config/openqa
$EDITOR ~/.config/openqa/client.conf

And paste the API keys you already have, you you will be able to post and create jobs on the public instance!

Let’s get hacking

This part is pretty straightforward once you’ve looked at $HOME/Skynet/os-autoinst-distri-opensuse.

For this round, let’s say we want to also test chrome, in incognito mode. By looking at chrome’s help we know that the --incognito is a thing

So let’s go to where all the tests are, edit, commit and push our changes

Remember to set up your fork, however if you want to make your life easier use hub you can find it in the repos too!

cd $HOME/Skynet/os-autoinst-distri-opensuse
vim tests/x11/chrome.pm
git commit -m "Message goes here" tests/x11/chrome.pm
git push $REMOTE
openqa-clone-custom-git-refspec \
 https://github.com/foursixnine/os-autoinst-distri-opensuse/tree/test_incognito_in_chrome \
 https://openqa.opensuse.org/tests/1792294 \
 SCHEDULE=tests/installation/bootloader_start,tests/boot/boot_to_desktop,tests/console/consoletest_setup,tests/x11/chrome \
 BUILD=0 \
 TEST=openQA-WORKSHOP-ALL-CAPS

In the end you will end up with an URL https://openqa.opensuse.org/t1793764, and you will get emails from travis if something is going wrong

How to edit stuff that you've already commited to git? (And squash as a bonus)

Santiago Zarate — Tue, 04 May 2021 00:00:00 +0000

So, you’re in the middle of a review, and have couple of commits but one of the comments is asking you to modify a line that belongs to second to last, or even the first commit in your list, and you’re not willing to do:

git commit -m "Add fixes from the review" $file

Or you simply don’t know, and have no idea what squash or rebase means?, well I won’t explain rebase today, but I will explain rebase

See how I do it, and also how do I screw up!

It all boils down to making sure that you trust git, and hope that things are small enough so that if you lose the stash, you can always rewrite it.

So in the end, for me it was:

git fetch origin
git rebase -i origin/master # if your branch is not clean, git will complain and stop
git stash # because my branch was not clean, and my desired change was already done
git rebase -i origin/master # now, let's do a rebase
# edit desired commits, for this add the edit (or an e) before the commit
# save and quit vim ([esc]+[:][x] or [esc]+[:][w][q], or your editor, if you're using something else
git stash pop # because I already had my change
$HACK # if you get conflicts or if you want to modify more
      # be careful here, if you rewrite history too much
      # you will end up in Back to the Future II
      # Luckly you can do git rebase --abort
git commit --amend $files #(alternatively, git add $files first then git commit --amend
git rebase --continue
git push -f # I think you will need to add remote+branch, git will remind you
# go on with your life

(Note: A squash is gonna put all of the commits together, just make sure that there’s an order:

pick COMMIT1
pick COMMIT2
squash COMMIT3 # (Git will combine this commit, with the one above iir, so COMMIT2+COMMIT3 and git will ask you for a new commit message)

Perl binaries are mismatched - the revenge of the lazy person

Santiago Zarate — Tue, 09 Feb 2021 00:00:00 +0000

The uninvited eldrich terror

You use local::lib, and the pain unfolds: A shell is started, and you find a dreaded:

Cwd.c: loadable library and perl binaries are mismatched (got handshake key 0xdb00080, needed 0xdb80080)

Which means: that the module (Cwd in this case) is not compatible (Because it’s an XS module) with your current version of perl, installed on your system: Likely it was compiled for a previous version, leadin to those binaries mismatching

Don’t panic!

I wrote about this however, I left out how to get to the point where you have already an usable Perl again?

The light

Instead of downloading local::lib from git as I used to do… this time I decided to do it on a much simpler way: use perl-local-lib from my distribution, and let it do the magic, I mean that’s why I run openSUSE Tumbleweed

 $ rm -rf perl5-old || echo "First time eh?"
 $ mv perl5 perl5-old
 $ perl-migrate-modules --from ~/perl5-old/lib/perl5 /usr/bin/perl
 $ perl -MCPAN -Mlocal::lib -e 'CPAN::install(App::cpanminus)'
 $ cpanm App::MigrateModules
 $ perl-migrate-modules --from ~/perl5-old/lib/perl5 /usr/bin/perl

Et voilà, ma chérie!

Cron do not send me empty emails

Santiago Zarate — Tue, 26 Jan 2021 00:00:00 +0000

Just in case, if you’ve ever wondered how to stop cron from sending empty emails, a quick look at man mail will give you the answer you’re looking for (if you know what you’re searching for):

    -E

    If an outgoing message does not contain any text in its first or only message part, do not send it but discard it silently,
    effectively setting the skipemptybody variable at program startup. This is useful for sending messages from scripts started 
    by cron(8).

I got this after visiting couple of forums, and some threads at stack exchange, however this one nailed it

So all you need to do is, fire up that crontab -e and make your script run every five minutes, without fear of the noise

*/5 * * * * /usr/local/bin/only-talk-if-there-are-errors-script |& mail -E -r $(hostname)@opensuse.org -s "[CRON][monitoring] foo bar $(date --iso-8601='minutes')" do-not-spam-me@example.com

Et voilà, ma chérie!

Quick and dirty ipmitool tutorial

Santiago Zarate — Tue, 19 Jan 2021 00:00:00 +0000

Because I always forget

To reboot a SUPERMICRO server:

Just remember that the default user/password might still be ADMIN/ADMIN :)

ipmitool -I lanplus -H $HOST -U USER -P PASSWORD power cycle

To connect to the serial console

ipmitool -I lanplus -H $HOST -U USER -P PASSWORD sol activate

Ext4 filesystem has no space left on device? You liar!

Santiago Zarate — Tue, 15 Sep 2020 00:00:00 +0000

So, you wake up one day, and find that one of your programs, starts to complainig about “No space left on device”:

Next thing (Obviously, duh?) is to see what happened, so you fire up du -h /tmp right?:

$ du -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/zkvm1-root 6.2G 4.6G 1.3G 79% /

Wait, what? there’s space there! How can it be? In all my years of experience (+15!), I’ve never seen such thing!

Gods must be crazy!? or is it a 2020 thing?

$ touch /tmp
touch: cannot touch ‘/tmp/test’: No space left on device

After shamelessly googling/duckducking/searching, I ended up at https://blog.merovius.de/2013/10/20/ext4-mysterious-no-space-left-on.htmlbut alas, that was not my problem, although… perhaps too many files?, let’s check with du -i this time:

$ du -i /tmp
`Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/zkvm1-root 417792 417792 0 100% /

Because I’m super smart ~~I’m not~~ , I now know where my problem is, too many files!, time to start fixing this…

After few minutes of deleting files, moving things around, bind mounting things, I landed with the actual root cause:

Tons of messages waiting in /var/spool/clientmqueue to be processed,I decided to delete some, after all, I don’t care about this system’s mails… so find /var/spool/clientmqueue -type f -delete does the job, and allows me to have tab completion again! YAY!.

However, because deleting files blindly is never a good solution, I ended up in the link from above, the solution was quite simple:

$ systemctl enable --now sendmail

After a while, root user started to receive system mail, and I could delete them afterwards :)

In the end, very simple solution (In my case!) rather than formatting or transfering all the data to a second drive, formatting & playing with inode size and stuff…

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/zkvm1-root 417792 92955 324837 23% /

Et voilà, ma chérie!

This is a very long post, just to say:

ext4 no space left on device can mean: You have no space left, or you don’t have more room to store your files.

Using python virtualenv inside vscode

Santiago Zarate — Sun, 14 Jun 2020 00:00:00 +0000

Quick and dirty

Install python3-virtualenvwrapper (via pip or via package manager)
Export a workon directory: export WORKON_HOME=/home/foursixnine/Projects/python-virtualenv
source virtualenvwrapper

foursixnine@deimos:~/Projects> source virtualenvwrapper    
virtualenvwrapper.user_scripts creating /home/foursixnine/Projects/python-virtualenv/premkproject
...
virtualenvwrapper.user_scripts creating /home/foursixnine/Projects/python-virtualenv/get_env_details

mkvirtualenv newenv

foursixnine@deimos:~/Projects> mkvirtualenv newenv
created virtual environment CPython3.8.3.final.0-64 in 115ms
  creator CPython3Posix(dest=/home/foursixnine/Projects/python-virtualenv/newenv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/home/foursixnine/.local/share/virtualenv/seed-app-data/v1.0.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
virtualenvwrapper.user_scripts creating /home/foursixnine/Projects/python-virtualenv/newenv/bin/predeactivate
...
virtualenvwrapper.user_scripts creating /home/foursixnine/Projects/python-virtualenv/newenv/bin/get_env_details

By this point, you’re already inside newenv:

(newenv) foursixnine@deimos:~/Projects>

You can create multiple virtual environments and switch among them using workon $env so long as you have sourced virtualenvwrapper and your $WORKON_HOME is properly defined.

Real business

Now, if you want to use vscode, remember that you will need to define properly python.PythonPath for your workspace/project (I’m new to this, don’t hang me in a public square ok?), in this case, my env is called linkedinlearningaiml

{
    "python.pythonPath": "/home/foursixnine/Projects/python-virtualenv/linkedinlearningaiml/bin/python"
}

Now your python code will be executed within the context of your virtual environment, so you can get down to serious (or not at all) python development, without screweing up your host or polluting the dependencies and stuff.

PS: Since wanted to be able to run standalone python files, I also needed to change a bit my launch.json (Maybe this is not needed?)

    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Current File",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "cwd": "${fileDirname}"
        }
    ]
}

And off you go, how to use python virtualenv inside vscode

Et voilà, ma chérie!

Looking for exceptions with awk

Santiago Zarate — Thu, 21 May 2020 00:00:00 +0000

Sometimes you just need to search using awk or want to use plain bash to search for an exception in a log file, it’shard to go into google, stack overflow, duck duck go, or any other place to do a search, and find nothing, or at leasta solution that fits your needs.

In my case, I wanted to know where a package was generating a conflict for a friend, and ended up scratching my head,because I didn’t want to write yet another domain specific function to use on the test framework of openQA, and I’mvery stubborn, I ended up with the following solution

journalctl -k | awk 'BEGIN {print "Error - ",NR; group=0}
/ACPI BIOS Error \(bug\)/,/ACPI Error: AE_NOT_FOUND/{ print group"|", 
    $0; if ($0 ~ /ACPI Error: AE_NOT_FOUND,/ ){ print "EOL"; group++ }; 
}'

This is short for:

Define $START_PATTERN as /ACPI BIOS Error $bug$/, and $END_PATTERN as /ACPI Error: AE_NOT_FOUND/
Look for $START_PATTERN
Look for $END_PATTERN
If you find $END_PATTERN add an EOL marker (that is not needed, since the group variable will be incremented)

And there you go: How to search for exceptions in logs, of course it could be more complicated, because you can have nestedcases and whatnot, but for now, this does exactly what I need:

EOL
10| May 20 12:38:36 deimos kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.LPCB.HEC.CHRG], AE_NOT_FOUND (20200110/psargs-330)
10| May 20 12:38:36 deimos kernel: ACPI Error: Aborting method \PNOT due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
10| May 20 12:38:36 deimos kernel: ACPI Error: Aborting method \_SB.AC._PSR due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
10| May 20 12:38:36 deimos kernel: ACPI Error: AE_NOT_FOUND, Error reading AC Adapter state (20200110/ac-115)
EOL
11| May 20 12:39:12 deimos kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.LPCB.HEC.CHRG], AE_NOT_FOUND (20200110/psargs-330)
11| May 20 12:39:12 deimos kernel: ACPI Error: Aborting method \PNOT due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
11| May 20 12:39:12 deimos kernel: ACPI Error: Aborting method \_SB.AC._PSR due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
11| May 20 12:39:12 deimos kernel: ACPI Error: AE_NOT_FOUND, Error reading AC Adapter state (20200110/ac-115)
EOL
12| May 20 13:37:41 deimos kernel: ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.LPCB.HEC.CHRG], AE_NOT_FOUND (20200110/psargs-330)
12| May 20 13:37:41 deimos kernel: ACPI Error: Aborting method \PNOT due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
12| May 20 13:37:41 deimos kernel: ACPI Error: Aborting method \_SB.AC._PSR due to previous error (AE_NOT_FOUND) (20200110/psparse-529)
12| May 20 13:37:41 deimos kernel: ACPI Error: AE_NOT_FOUND, Error reading AC Adapter state (20200110/ac-115)
EOL

So I could later write some code that looks per string, separates the string using the pipe | discards the group id, and adds that record to an arrayof arrays or hashes: [{group: id, errors: [error string .. error string] ]

How to import a gpg key on zypper (openSUSE)

Santiago Zarate — Thu, 07 May 2020 00:00:00 +0000

For some odd reason, I couldn’t find quickly (2 mins in the manpage and some others on DuckDuckGo) a way to import aGPG key.

Just because one of the repos gives this ugly message:

Repository vscode does not define additional 'gpgkey=' URLs.
Warning: File 'repomd.xml' from repository 'vscode' is signed with an unknown key 'EB3E94ADBE1229CF'.

    Note: Signing data enables the recipient to verify that no modifications occurred after the data
    were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
    and in extreme cases even to a system compromise.

    Note: File 'repomd.xml' is the repositories master index file. It ensures the integrity of the
    whole repo.

    Warning: We can't verify that no one meddled with this file, so it might not be trustworthy
    anymore! You should not continue unless you know it's safe.

I know you can use zypper ar --gpg-auto-import-keys, but I didn’t do it when I added it (and I’m too lazy to remove the repo and add it again, just to see if it works)

rpm --import https://packages.microsoft.com/keys/microsoft.asc
zypper ref

Roundcube smtp (250) authentication failed

Santiago Zarate — Fri, 27 Sep 2019 00:00:00 +0000

So, say you find yourself, somehow having the following error in the roundcube logs:

[27-Sep-2019 12:44:48 +0000]: <f930f680> PHP Error: SMTP server does not support authentication (POST /?_task=mail&_unlock=loading1569588324419&_framed=1&_lang=es&_action=send)
[27-Sep-2019 12:44:48 +0000]: <f930f680> SMTP Error: Authentication failure: SMTP server does not support authentication (Code: ) in /roundcube/program/lib/Roundcube/rcube.php on line 1674 (POST /correo/?_task=mail&_unlock=loading 1569588324419&_framed=1&_lang=es&_action=send)

You have tried everything, but still can’t seem to be able to send email from roundcube, you keep getting this annoying “SMTP (250) authentication failed” notification, every time you click “Send”.

Well… Make sure that your server is connecting to the right place. It took me a while to realize that roundcube was trying to connect to localhost, but somehow the authentication mechanism stopped working (it was before upgrading).

Since I don’t really want to debug too much today (it’s friday after all), and because my configuration/use case is over ssl/tls, the solution to the probem was simply:

$config['smtp_server'] = 'tls://services.host.co';

Et voilà, ma chérie!