DEV Community: wasteofserver

Zoom M3 MicTrak file recovery

wasteofserver — Tue, 25 Feb 2025 04:18:31 +0000

This post was originally posted on https://wasteofserver.com/zoom-m3-mictrak-file-recovery/, you will find newer revisions and additional comments there.

🎤 If you're in a hurry, head directly to the Github repository.

While I'm fortunate enough to develop software for state-of-the-art companies, I've always been the "printer guy", a badge I wear with pride.

And maybe that's the reason why, for more than a quarter of a century, people have turned up with storage medium to salvage. I never did it professionally, it’s just something I enjoy and, more often than not, I’m able to help.

The process

My main approach is two-fold:

capture what you can from the dying medium
try to recreate the data

Sometimes, just by adjusting read tolerance, you're able to capture everything from a disk Windows or Mac reject, as the OS times out faster than the dying medium can respond. I've had a disk hooked to a box for 7 months; 100% recovery success!

Sometimes you'll have chunks of data missing, but one of the File Allocation Table / Master File Table / Container superblock / root B-tree are still there and so most data is salvageable with both filenames and tree location.

At the extreme, you’re left with nothing but some of the raw data. All you can do is carve it out - essentially reading every byte from the medium, identifying known headers like JPG (FF D8) or MKV (1A 45 DF A3), and proceed to capture all sequential data until the end of the file. If for any reason, the file is fragmented, carving will obviously fail.

The call from Pierre Zago

Frankie, this hasn't happened before, I'm generally cautious... I don't know what I was thinking - I've accidentally formatted an SD card with audio for an entire session. Worse, I've saved some files onto it!

Pierre is an incredibly talented comedian and an extraordinary actor. While his work is multifaceted, he became widely known for street sketches, some of them absolutely universal. Just check the one below where he simply says "excuse-me".

This sketch’s simplicity and comedic charm create a lighthearted yet universally appealing work of art.

With that simple action, Pierre had joined the club. Everyone messes up. Even Pixar. I gladly took the card. Don't worry, I said, the overwritten contents are gone, but given that it's a microphone formatted card, even without FAT tables we should be able to carve out most of whatever you've recorded as data will probably be sequential.

Little did I know, this would turn out to be one of the most interesting toy projects I’ve worked on in a while. The last time I had this much fun was when I had to rebuild a hardware RAID-0 that contained the masters for an album by 'Os Azeitonas'.

The Echo (echo, echo)...

As expected, the only way to get data back was by carving it out of the image dump. There's a multitude of tools to choose from, Photorec (open source), Recuva (watch out for bundleware), ReclaiMe (paid), etc... though I'm partial to R-Studio (paid). Their results consistently outperform the competition.

In this instance, nonetheless, things wouldn't be so simple. Every software I tried was able to extract the wav files, but there was something rather wrong with the data as they all had this repetition, an echo of sorts.

I'm a bit dull of hearing, so opened them in Audacity to check what was going on. You can clearly see a pattern here:

There's somewhat of a repetition every ~ 0.7 seconds

While the chunks have a pretty similar waveform - there wasn't binary correlation. Besides, there was also something murky, every wav file had what appeared to be 2 consecutive headers.

Mind you, at this point, my knowledge of wav files was that the header is 52 49 46 46. Aside from using a mic, I didn't query Pierre on how he had actually recorded the data. However, when I saw "ZOOM M3" tag in the header, I called the authority on everything sound.

Perfect Pitch, a sort of wizardry

Ed's been on speed dial for the better part of this lifetime. I'm lucky like that. Beyond being an unbelievable composer, just listen to the breathtaking Best Youth, he's also a pitch-perfect, living and breathing, encyclopedia on sound.

Zoom? Yes, yes. I have one. They record both wav and raw. Data is mangled? Ah. Sure. Recover? Twisted files? That's going to be an impossible task and, even if it's not, it'll be easier to just re-record.

And there he had me. It would definitely be easier to just re-record, even Pierre at one point suggested doing a voice-over, but where would the fun be in that?

Zoom M3 MicTrak

The Magic Number

I didn't even appreciate the concept of a raw wave until Ed explained it. Now that I knew the mic saves two simultaneous files, everything fell into place.

It's common for cameras to store both formats, but a microphone is a different beast. There's no way to determine upfront how long the user will record, which wouldn't be an issue if this were a single stream. As it's storing both tracks, there is quite a bit more play involved.

You see, as soon as you hit record, the mic creates two files and continuously flushes the captured data from both onto the card. That, is done in chunks of data. One for the RAW, another for the WAV, repeat.

Since we need to isolate those slices of data in order to untangle it, we must know their exact size. And there lies our magic number!

My first thought was that the chunks might be aligned with exFAT allocation unit size. In this case, 128 KBytes. Let's test it.

The header, clearly states that this is a stereo file (2 channels), recorded at 32 bits per channel, sampled 48k times per second. If you remember from the image above, the chunks repeat at approximately 0.7 seconds.

Let's get a rough approximation on the chunk bytes so we know the ballpark.

1 second of data = 2 channels * 32 bits * 48000 samples
1 second of data = 384000 bytes
0.7 seconds ~ 268800 bytes

We're looking into chunks of around 268 KBytes.

And just like that, the idea that data may be chunked to exFAT AUS of 128 Kbytes is instantly disproved.

The next obvious step would be to travel upwards on base 2. Given that 4096 is a good balance for buffers, let's evaluate from there:

4096 * 32 = 131072 (falls short by about 1/2)
4096 * 64 = 262144 (is in the ballpark of what we're expecting)

262144/384000 ~ 0.682 seconds of data

0.0682 seconds matched our estimate of 0.7 seconds so perfectly that I immediately knew 262144 was the constant we were after.

The reconstruction

Conceptually, the problem was solved. Now I just had to build the tool. For that it would be necessary to:

Untangle the files directly from the image dump. Since pieces recovered by other carving software would be half the expected size (the data chunk contains two files, so it’s actually double the size reported).
Learn how to create a RIFF header.
Create a RIFF header with a BEXT chunk to make the recovered files compatible with the "M3 ZOOM Edit & Play" software.

⌨️ All the code is on GitHub !

And I'm pretty sure you'll feel more at home there.

However, for the sake of Google indexing, I’m leaving the methods here that create both the RIFF and BEXT headers, something I couldn’t find, which unfortunately made the process take longer than I’d like to admit.

public class RiffFile {

    /**
     * Creates a RIFF header with BEXT and fmt chunks
     *
     * @param sampleRate the sample rate of the audio (8000Hz, 44100Hz, 48000Hz, etc) times per second the audio is sampled
     * @param bitsPerSample the bits per sample (8bits, 16bits, 32bits, etc)
     * @param channels the number of channels (1 mono, 2 stereo, etc)
     * @param audioDataSize the size of the audio data in bytes
     * @return the RIFF header
     * @throws IOException if an I/O error occurs
     */
    public static byte[] createRiffHeader(int sampleRate, short bitsPerSample, short channels, int audioDataSize) throws IOException {
        // calculate the byte rate, block align and file size
        int byteRate = sampleRate * channels * bitsPerSample / 8;
        short blockAlign = (short) (bitsPerSample * channels / 8);

        // stream that will carry the new RIFF file
        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        DataOutputStream out = new DataOutputStream(byteArrayOutputStream);

        // riff header
        out.writeBytes("RIFF");
        out.writeInt(Integer.reverseBytes(0));
        out.writeBytes("WAVE"); // 9-12 Format always WAVE

        // bext chunk
        writeBextChunk(out);

        // fmt chunk
        out.writeBytes("fmt "); // 13-16 chunkID is "fmt " with trailing whitespace
        out.writeInt(Integer.reverseBytes(16)); // 17-20 size of this chunk, is 16 byts
        out.writeShort(Short.reverseBytes((short) 3)); // 21-22 (2 bytes) audioFormat (1 PCM integer, 3 IEEE 754 float)
        out.writeShort(Short.reverseBytes(channels)); // 23-24 (2 bytes) numChannels (1 mono, 2 stereo, 4, etc)
        out.writeInt(Integer.reverseBytes(sampleRate)); // 25-28 (4 bytes) sampleRate (8000, 44100, 48000, etc)
        out.writeInt(Integer.reverseBytes(byteRate)); // 29-32 (4 bytes) byteRate (sampleRate * numChannels * bitsPerSample/8)
        out.writeShort(Short.reverseBytes(blockAlign)); // 33-34 (2 bytes) blockAlign (numChannels * bitsPerSample/8)
        out.writeShort(Short.reverseBytes(bitsPerSample)); // 35-36 (2 bytes) bitsPerSample (8bits, 16bits, 32bits, etc)

        // data chunk
        out.writeBytes("data"); // 37-40 chunkID ID is "data"
        out.writeInt(Integer.reverseBytes(audioDataSize)); // 41-44 size of this chunk varies
        out.close();

        // write the full size of the file on the 4-8 bytes
        byte[] outArr = byteArrayOutputStream.toByteArray();
        int size = outArr.length - 8;
        ByteBuffer.wrap(outArr, 4, 4).order(ByteOrder.LITTLE_ENDIAN).putInt(size);
        return outArr;
    }

    private static void writeBextChunk(DataOutputStream out) throws IOException {
        // bext chunk
        out.writeBytes("bext");
        out.writeInt(Integer.reverseBytes(256 + 32 + 32 + 10 + 8 + 8 + 8 + 2 + 180 + 4 + 4 + 4 + 4 + 4 + 180)); // bext chunk size (fixed size for BWF)

        // description 256 bytes
        writeToArray(out, 256, ""); // 256 bytes description
        writeToArray(out, 32, "ZOOM M3"); // 32 bytes originator
        writeToArray(out, 32, ""); // 32 bytes originator reference
        writeToArray(out, 10, "2023-10-01"); // 10 bytes origination date
        writeToArray(out, 8, "12:00:00"); // 8 bytes origination time
        writeToArray(out, 8, "12:00:00"); // 8 bytes time reference

        out.writeLong(Long.reverseBytes(0L)); // 8 bytes time reference
        out.writeShort(Short.reverseBytes((short) 0)); // 2 bytes version
        out.write(new byte[180]); // 180 bytes UMID
        out.writeFloat(0.0f); // 4 bytes loudness value
        out.writeFloat(0.0f); // 4 bytes loudness range
        out.writeFloat(0.0f); // 4 bytes max true peak level
        out.writeFloat(0.0f); // 4 bytes max momentary loudness
        out.writeFloat(0.0f); // 4 bytes max short term loudness

        // zoom m3 needs this bit to allow file to be read from "zoom m3 edit & play"
        writeToArray(out, 180, "A=PCM,F=48000,W=32,M=stereo,T=M3;VERSION=1.00;MSRAW=ON ;");
    }

}

As you may see, there wasn't much effort put into the BEXT chunk; I simply created it to ensure "Zoom M3 Edit & Play" would be compatible.

The Tool

I hope you had an interesting read. I tried to lift the curtain on the thought process while keeping this engaging. The actual code is hopefully self-explanatory, and you will find it here:

https://github.com/wasteofserver/zoom_m3_mic_wav_data_recover

The challenge wasn't just about recovering lost recordings - it was about understanding why traditional tools failed and developing a method that worked.

While it might have been easier to re-record, the thrill of solving the puzzle made the effort worthwhile. In the end we got a custom-built solution that successfully restores Zoom M3 MicTrak recordings.

If you ever find yourself in a similar situation, hopefully, this breakdown helps you out. And if not, well, at least you got to enjoy a little adventure into the world of data recovery.

This post was originally posted on https://wasteofserver.com/zoom-m3-mictrak-file-recovery/, you will find newer revisions and additional comments there.

Permanently delete unwanted emails from Gmail. Out of sight, out of mind!

wasteofserver — Fri, 21 Feb 2025 05:07:59 +0000

This post was originally posted on https://wasteofserver.com/permanently-delete-unwanted-emails-from-gmail-out-of-sight-out-of-mind/, you will find newer revisions and additional comments there.

This post is a bit different from my usual content. While it includes a snippet of code, it's designed for a less technical audience, thus the flood of screenshots.

I received an atypical cry for help. A couple is splitting and one of the partners is spamming the other with abusive emails. The receiver created a Gmail filter to trash the messages, but Gmail trash retention policy is 30 days and, in moments of weakness, can't resist clicking in the trash and reading them, which understandably exacerbates the issue.

Given the violent nature of the emails, I didn’t want to risk deleting potential evidence so my suggestion was to create a process that would:

Auto-forward the emails from that sender to a lawyer
Delete those emails and purge them from trash, bypassing Gmail's 30-day retention policy

Out of sight, out of mind can be the best policy.

Create a forward address

On this particular case, the forward address will be the lawyer. Go to Gmail settings, select "Forwarding and POP/IMAP" and click on "Add a forwarding address".

Remember to keep forwarding DISABLED, we only want to forward emails from a single sender

After adding a forward address, Gmail will send a confirmation to that email - your lawyer - asking for permission. As soon as that's granted, you may proceed to the next step. Just remember to keep Forwarding disabled!

Create a new Filter

Here we will be specifying that all emails that come from someone@example.com will be forwarded to laywer@example.com and then sent to the trash.

Go to "Filters and Blocked Addresses" and then click on "Create a new filter".

The form for adding filters will open. You want all messages from that specific address to be filtered, so just add "someone@example.com" to the filter and choose "Create filter".

You can be picky with filters, but here we want ALL emails from sender to match

Now you'll have to select exactly what you want the filter to do. On our specific case, we'll forward the email to the lawyer, so tick that box. We also want to delete the email, so also tick that one.

And just like that, emails from this sender are forwarded and trashed!

This solves half of the issue and is the most straightforward part of the process. The tricky bit comes next, how to instantly purge those emails from the trash so that we're not tempted to read them. Google Apps Script to the rescue!

Create a Google Apps Script

Google Drive offers a feature that allows you to host and run scripts. While most developers are familiar with this, power users may not be aware of it. For the task at hand, this feature is absolutely perfect.

Head into https://script.google.com/, follow the authentication procedures, if needed, and then click on "New Project".

Hurray for your first script!

You're now on your project screen. You'll need to interact with Gmail so let's add that service. Click on the big + next to "Services", look for Gmail API and add it.

Your script needs the Gmail API to access your emails

Now, replace the myFunction with this piece of code. Remember, YOU MUST CHANGE someone@example.com to the actual address you want to remove from trash!

function deleteMailsFromTrash() {
  var gmailSearchString = `in:trash from:someone@example.com`

  var threads = GmailApp.search(gmailSearchString);
  const n = threads.length;
  if (n <= 0) {
    Logger.log("No threads matching search string \"%s\"", gmailSearchString);
    return
  } else {
    Logger.log("%s threads matching action **%s**", n, gmailSearchString);
  }

  for (var i = 0; i < threads.length; i++) {
    var thread = threads[i];
    Logger.log(`\t Thread# ${i} [ID: ${thread.getId()}]: [message : ${thread.getFirstMessageSubject()}] deleted`);
    Gmail.Users.Threads.remove('me', thread.getId());
  }
}

This script will look in the trash for emails from someone@example.com and delete them

Your screen should now resemble this. Go ahead and rename "Untitled project" to something more meaningful, like "Purge Specific Mails from Trash". Also change myFunction to deleteMailsFromTrash and then hit Run.

You'll be asked to give permissions to access your Google account.

Now you'll get an error! Google hasn't verified this app. While the developer hasn't verified the app, you shouldn't use. In this particular case, you are the developer! That's why I didn't release this solution as a pre-made script. It's safer to have the code running on your side.

Click on that "Go to Purge Specific Mails from Trash (unsafe)" link and then continue. On your Apps Script window, you'll see the first execution of your script.

On my case, since I don't have email from someone@example.com on the "trash" the program simply prints a "No threads matching search string". On your specific case, you may see that a couple of emails have been deleted! Well done.

Now everything is working, but we still need to set up a trigger to run the script automatically, ensuring unwanted emails are deleted in a timely manner.

Configure a trigger!

You'll want a time driven trigger. Something that runs on a schedule making sure that the emails that were placed in the trash by the filter you created above are purged before you can get to them.

Click on the clock on the left sidebar, then on the big blue button on the bottom right that says "Add Trigger" and configure the filter as per the image below.

And there you have it! Process is automated.

I’ve set this script to run every 5 minutes, but you can adjust the interval to as low as 1 minute if you prefer. Tweak as needed. Setting a longer interval is simply a way to considerate of Google's infrastructure.

Now, to make sure things are working as expected, on the left sidebar, click on executions. You'll see a table with all the times the script has run. As you've just implemented it, there should probably be two to three runs. One manual Type: Editor from when you manually executed it and then a couple more from the time-driven trigger labeled with Type: Time-Driven.

Peace.

As challenging as it may be to navigate these situations, it’s important to remember that protecting your health is a priority.

While technology can help us minimize harmful distractions, healing takes time and self-compassion. Stay strong, take care of yourself, and don’t hesitate to seek support.

You deserve peace and healing through this process.

I don’t know if you’ve tried rucking before, but it’s something that liberates my mind.

It’s simply walking while carrying weight. Originally a military exercise, rucking is gaining popularity for its benefits to physical health, stability, and mental well-being. Give it a try!

Stop 404 prying bots with HAProxy

wasteofserver — Mon, 17 Feb 2025 06:09:16 +0000

This post was originally posted on https://wasteofserver.com/stop-404-prying-bots-with-haproxy/, you will find newer revisions and additional comments there.

Your logs are filled with 404 hits on /.DS_Store, /backup.sql, /.vscode/sftp.json and a multitude of other URLs. While these requests are mostly harmless, unless of course, your server actually does have something to offer at those locations, you should placate the bots.

Why?

Hitting a server is a resource intensive task and, given that those bots have an extensive list of different URLs, there's no caching mechanism that can help you. Besides, stopping bots is always a safety measure.

We've previously used HAProxy to mitigate attacks on Wordpress login page, the idea is to extend that approach to also cover 404 errors.

Bots will try their best to create havoc in your server

I've taken inspiration from Sasa Tekovic, namely on not blocking actual search engine crawlers and allowing 404 on static resources to prevent actual missing resources - an error on your part - from not blocking legitimate users.

Before implementing, it's always good to spin up a local test environment. Let's start HAProxy and Apache using Docker. We do need an actual backend server to give us those 404.

version : '3'

services:
    haproxy:
        image: haproxy:3.1.3-alpine
        ports:
            - "8100:80"
        volumes:
            - "./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg"
        networks:
            - webnet
    apache:
        image: httpd:latest
        container_name: apache1
        ports:
            - "8080:80"
        volumes:
            - ./html:/usr/local/apache2/htdocs/
        networks:
            - webnet

networks:
    webnet:

Then, simply run docker-compose up, and you can access localhost:8100 in your browser.

The haproxy.cfg file is pretty much self-explanatory:

global
    log stdout format raw daemon debug

defaults
    log global
    mode http

frontend main
    bind *:80

    acl static_file path_end .css .js .jpg .jpeg .gif .ico .png .bmp .webp .csv .ttf .woff .svg .svgz
    acl excluded_user_agent hdr_reg(user-agent) -i (yahoo|yandex|kagi|(google|bing)bot)

    # tracks IPs but exclude hits on static files and search engine crawlers
    http-request track-sc0 src table mock_404_tracking if !static_file !excluded_user_agent
    # increment gpc0 if response code was 404
    http-response sc-inc-gpc0(0) if { status 404 }
    # checks if the 404 error rate limit was exceeded
    http-request deny deny_status 403 content-type text/html lf-string "404 abuse" if { sc0_gpc0_rate(mock_404_tracking) ge 5 }

    # whatever backend you're using
    use_backend apache_servers

backend apache_servers
    server apache1 apache1:80 maxconn 32

# mock backend to hold a stick table
backend mock_404_tracking
    stick-table type ip size 100k expire 10m store gpc0,gpc0_rate(1m)

If you get more than 5 hits on 404 requests in a single minute, bot will be banned for 10 minutes.

As it stands, this setup effectively rate-limits bots generating excessive 404s. However, we also want to integrate it with our previous example, where we used HAProxy to block attacks on WordPress.

global
    log stdout format raw daemon debug

defaults
    log global
    mode http

frontend main
    bind *:80

    # We may, or may not, be running this with Cloudflare acting as a CDN.
    # If Cloudflare is in front of our servers, user/bot IP will be in 
    # 'CF-Connecting-IP', otherwise user IP with be in 'src'. So we make
    # sure to set a variable 'txn.actual_ip' that has the IP, no matter what
    http-request set-var(txn.actual_ip) hdr_ip(CF-Connecting-IP) if { hdr(CF-Connecting-IP) -m found }
    http-request set-var(txn.actual_ip) src if !{ hdr(CF-Connecting-IP) -m found }

    # gets the actual IP on logs
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr IP:%{+Q}[var(txn.actual_ip)]"

    # common static files where we may get 404 errors and also common search engine
    # crawlers that we don't want blocked
    acl static_file path_end .css .js .jpg .jpeg .gif .ico .png .bmp .webp .csv .ttf .woff .svg .svgz
    acl excluded_user_agent hdr_reg(user-agent) -i (yahoo|yandex|kagi|google|bing)

    # paths where we will rate limit users to prevent Wordpress abuse
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

    # 404 abuse blocker
    # track IPs but exclude hits on static files and search engine crawlers
    # increment gpc0 counter if response status was 404 and deny if rate exceeded
    http-request track-sc0 var(txn.actual_ip) table mock_404_track if !static_file !excluded_user_agent
    http-response sc-inc-gpc0(0) if { status 404 }
    http-request deny deny_status 403 content-type text/html lf-string "404 abuse" if { sc0_gpc0_rate(mock_404_track) ge 5 }

    # wordpress abuse blocker
    # track IPs if the request hits one of the monitored paths with a POST request
    # increment gpc1 counter if path was hit and deny if rate exceeded
    http-request track-sc1 var(txn.actual_ip) table mock_wplogin_track if is_wp_login is_post
    http-request sc-inc-gpc1(1) if is_wp_login is_post 
    http-request deny deny_status 403 content-type text/html lf-string "login abuse" if { sc1_gpc1_rate(mock_wplogin_track) ge 5 }

    # your backend, here using apache for demonstration purposes
    use_backend apache_servers

backend apache_servers
    server apache1 apache1:80 maxconn 32

# mock backends for storing sticky tables
backend mock_404_track
    stick-table type ip size 100k expire 10m store gpc0,gpc0_rate(1m)
backend mock_wplogin_track
    stick-table type ip size 100k expire 10m store gpc1,gpc1_rate(1m)

Running with two stick tables, and stopping both threats.

And there you have it. HAProxy once again used for much more than as a simple reverse proxy. It's a little Swiss Knife!

This headlamp has been a game-changer when working on repairs.

I had one, but when it broke, hesitated to replace it, resorting to the phone’s flashlight. And sure, it works - but once you experience the convenience of having both hands free again - there’s no going back. If you need reliable, hands-free lighting, this is a must-have!

How to mitigate attacks on WordPress when running under a full CDN like Cloudflare

wasteofserver — Thu, 19 Sep 2024 17:04:48 +0000

This post was originally posted on wasteofserver.com, you may find newer revisions and additional comments there.

How to mitigate attacks on WordPress when running under a full CDN like Cloudflare

The most effective way to prevent bots from spamming your server is to drop them at the firewall. This is generally achieved using tools like Denyhosts or fail2ban, which monitor your logs, identify suspicious activity, and block the offending IP addresses before they cause harm.

Denyhosts works at the application level by adding entries to /etc/hosts.deny, whereas fail2ban operates at the firewall level using iptables, which makes it far more efficient.

However, on resource-constrained machines, fail2ban can still be taxing. A few years ago, we shared a demo of a lightweight log parser called banbylog, tailored specifically for our needs (SSH and WordPress activity monitoring) at a much lower resource cost. If that sounds like a good fit, feel free to check it out, but keep in mind that it’s not production-ready!

While the consensus is to parse logs and block hostile IPs at the firewall, it will not work under Cloudflare umbrella!

Ideally the attack would stop here, but it won't work with a full CDN like Cloudflare

Why can't we flag offending IPs with Cloudflare?

Because the IPs that are "attacking" your server are not the actual offending IPs, but Clouflare machines that are proxying the request to your server. Take a look at the diagram below:

Cloudflare acts as a middleman between your server and the users. The only IP addresses visible to your firewall are from Cloudflare, not from the original user.

In fact, you should explicitly whitelist Cloudflare IP range. If you happen to block an IP that belongs to Cloudflare, legitimate users will see your site as down.

😱

While this article's focus is on securing servers while working under a Content Distribution Network, Cloudflare offers a variety of tools - free and paid - you may leverage to achieve these same results.

If the IP we see doesn't belong to the user making the request, what can we look for?

Every request Cloudflare sends to your server has an attached header that carries the user original IP under CF-Connecting-IP. And that's what we can leverage to get them! Unfortunately, reading http headers is too upstream for iptables, thus HAProxy to the rescue!

While iptables operates at Layer 4, HAProxy can operate at OSI Layer 7.

💡

Remember the OSI model:

Layer 7 - Application; protocols (HTTP, ...)  
Layer 6 - Presentation; character encoding (ASCII, UTF8, ...)  
Layer 5 - Session; stick client to server  
Layer 4 - Transport; protocols (TCP, UDP, ...)  
Layer 3 - Network; routing protocols (IP)  
Layer 2 - Data Link; physical to network (ARP, Ethernet)  
Layer 1 - Physical; cabling, Wi-Fi

The easiest way to test HAProxy configurations is to boot a Docker instance of HAProxy:

docker run --name haproxy -p 888:80 -v ${pwd}:/usr/local/etc/haproxy --sysctl net.ipv4.ip_unprivileged_port_start=0 haproxy:2.3

Start HAProxy listening on host port 888

The above assumes you're using Powershell and are in the directory that contains haproxy.cfg. Change ${pwd} accordingly if not.

docker kill -s HUP haproxy

This command forces HAProxy restart, which is useful to iterate different configs

Below is a straightforward haproxy.cfg that will put HAProxy listening on port 80, log to stdout so you can get an instant glimpse on what's going on, and also print the CF-Connecting-IP header.

global
    # output logs straight to stdout
    log stdout format raw daemon debug

defaults
    # put HAProxy in Level 7 mode allowing to inspect http protocol
    log global
    mode http

frontend main
    bind *:80

    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    # use backend ok, which always returns a 200, for debug
    use_backend ok

backend ok
    http-request return status 200 content-type "text/plain" lf-string "ok"

This should be enough to test HAProxy

Let's make a test request to HAProxy.

I'm partial to Bruno, a portable and offline alternative to Postman. Download it, add the CF-Connecting-IP header and make a POST request to http://localhost:888/wp-login.php to test if everything's working.

Nothing to install, a POST request in under a minute!

If things went as planned, your HAProxy should have printed this:

172.17.0.1 main ok/<NOSRV> -1/-1/0 78 LR POST /wp-login.php HTTP/1.1 200 -1 CF-IP:"222.222.222.222"

Now that we have the offending IP (222.222.222.222), we can block it!

On our particular case, bots are hitting wp-login.php, xmlrpc.php and xmrlpc.php (last one is a typo, but we've had more than 100k hits in the last 24h!). We also know that they're flooding the server with POST requests trying to brute-force passwords.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

Add this to end of the frontend main block

We now have a couple of options:

a) Now that we have the offending IPs in the log, we could change banbylog to write them to a file, and have HAProxy deny those requests. While this would work, HAProxy would need to be constantly reloaded.

b) Or we may simply leverage HAProxy stick tables and do a rate-limiting on offending requests.

While "a" would allow us to ban the offending IP for an indefinite amount of time, "b" has less moving pieces.

frontend main
    # requests that will be monitored and blocked if abused
    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST

    # table than can store 100k IPs, entries expire after 1 minute
    stick-table type ip size 100k expire 1m store http_req_rate(1m)

    # we'll track (save to table) the original IP only if the request hits
    # one of the monitored paths with a POST request
    http-request track-sc0 hdr(CF-Connecting-IP) if is_wp_login is_post

    # we now query the stick-table and if the IP has made more than 
    # 5 requests of the offending type in the last minute, 
    # current request is denied
    http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

HAProxy has multiple deny options, tarpit, silent drop, reject or shadowban. A tarpit deny would be something like this:

# make request hang for 20 seconds before replying back with a 403
timeout tarpit 20s
http-request tarpit deny_status 403 if is_wp_login is_post { sc_http_req_rate(0) gt 2 }

Notice that this puts extra stress on both HAProxy and iptables as they must keep the connection open

Just for reference, here the full, simplified, HAProxy config file that blocks requests if they hit one of the monitored URL's with more than 5 hits in less than a minute:

global
    log stdout format raw daemon debug

defaults
    log global
    mode http

frontend main
    bind *:80

    # capture original IP from header and put it in variable txn.cf_conn_ip
    http-request set-var(txn.cf_conn_ip) hdr(CF-Connecting-IP)

    # get the original IP on logs (just to check if things are working)
    log-format "%ci\ %hr\ %ft\ %b/%s\ %Tw/%Tc/%Tt\ %B\ %ts\ %r\ %ST\ %Tr CF-IP:%{+Q}[var(txn.cf_conn_ip)]"

    acl is_wp_login path_end -i /wp-login.php /xmlrpc.php /xmrlpc.php
    acl is_post method POST
    stick-table type ip size 100k expire 1m store http_req_rate(1m)
    http-request track-sc0 var(txn.cf_conn_ip) if is_wp_login is_post
    http-request deny if is_wp_login is_post { sc_http_req_rate(0) gt 5 }

    # obviously change this to whatever backend you're using
    use_backend ok

backend ok
    http-request return status 200 content-type "text/plain" lf-string "ok"

And there you have it, using your proxy as a gatekeeper!

The CPU toll of a wp-login.php brute-force attack and then HAProxy acting as a bouncer.

Re-Captcha, obviously!

While this article focus on the server side of things, the first thing you should obviously do, is to foolproof forms with Re-Captcha, which you can easily do with a plugin such as reCaptcha by BestWebSoft.

EDIT: A user queried: «I have some WordPress installations under Cloudflare and some exposing the server directly. Can it work on both?»

You can. It's as simple as doing something like this:

# if CF-Connecting-IP is not set, put src IP in txn.client_ip
http-request set-var(txn.client_ip) hdr(CF-Connecting-IP)
http-request set-var(txn.client_ip) src if !{ var(txn.client_ip) -m found }

But beware, if some sites are under Cloudflare and some aren't, a spammer can forge CF-Connecting-IP to trick the simple if/else above. You must first check if src belongs to Cloudflare before extracting txn.client_ip from the header.

This week, I assisted a friend in upgrading to professional TP-Link access points. I'm a strong advocate for devices that excel at a single task, and these EAP610 access points do just that. Highly recommended!

Premature optimization, where software thrives unless you kill it first - a tale of Java GC

wasteofserver — Sat, 30 Mar 2024 17:45:49 +0000

This post was originally posted on wasteofserver.com, you may find newer revisions and additional comments there.

Before going heads on into Java and the ways to tackle interference, either from the garbage collector or from context switching, let's first glance over the fundamentals of writing code for your future self.

Premature optimization is the root of all evil.

You've heard it before; premature optimization is the root of all evil. Well, sometimes. When writing software, I'm a firm believer of being:

1) as descriptive as possible ; you should try to narrate intentions as if you were writing a story.

2) as optimal as possible ; which means that you should know the fundamentals of the language and apply them accordingly.

As descriptive as possible

Your code should speak intention, and a lot of it pertains to the way you name methods and variables.

int[10] array1; // bad
int[10] numItems; // better
int[10] backPackItems; // great

Just by the variable name, you can already infer functionality

While numItems is abstract, backPackItems tells you a lot about expected behaviour.

Or say you have this method:

List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return new ArrayList<>(0);
    }
    // (...)
    return listOfVisitedCountries;
}

As far as code goes, this looks more or less ok.

Can we do better? We definitely can!

List<Countries> visitedCountries() {
    if(noCountryVisitedYet)
        return Collections.emptyList();
    }
    // (...)
    return listOfVisitedCountries;
}

Reading Collections.emptyList() is much more descriptive than new ArrayList<>(0);

Imagine you're reading the above code for the first time and stumble on the guard clause that checks if the user has actually visited countries. Also imagine this is buried in a lengthy class, reading Collections.emptyList() is definitely more descriptive than new ArrayList<>(0), you're also making sure it's immutable making sure client code can't modify it.

As optimal as possible

Know your language and use it accordingly. If you need a double there's no need to wrap it in a Double object. The same goes to using a List if all you actually need is an Array.

Know that you should concatenate Strings using StringBuilder or StringBuffer if you're sharing state between threads.

// don't do this
String votesByCounty = "";
for (County county : counties) {
    votesByCounty += county.toString();
}

// do this instead
StringBuilder votesByCounty = new StringBuilder();
for (County county : counties) {
    votesByCounty.append(county.toString());
}

Know how to index your database. Anticipate bottlenecks and cache accordingly. All the above are optimizations. They are the kind of optimizations that you should be aware and implement as first citizens.

How do you kill it first?

I'll never forget about a hack I read a couple of years ago. Truth be said, the author backtracked quickly, but it goes to show how a lot of evil can spur from good intention.

// do not do this, ever!
int i = 0;
while (i<10000000) {
    // business logic

    if (i % 3000 == 0) { //prevent long gc
        try {
            Thread.sleep(0);
        } catch (Ignored e) { }
    }
}

A garbage collector hack from hell!

You can read more on why and how the above code works in the original article and, while the exploit is definitely interesting, this is one of those things you should never ever do.

Works by side effects, Thread.sleep(0) has no purpose in this block
Works by exploiting a deficiency of code downstream
For anyone inheriting this code, it's obscure and magical

Only start forging something a bit more involved if, after writing with all the default optimizations the language provides , you've hit a bottleneck. But steer away from concoctions as the above.

An interpretation of Java's future Garbage Collector "imagined" by Microsoft Copilot

How to tackle that Garbage Collector?

If after all's done, the Garbage Collector is still the piece that's offering resistance, these are some of the things you may try:

If your service is so latency sensitive that you can't allow for GC, run with "Epsilon GC" and avoid GC altogether. -XX:+UnlockExperimentalVMOptions -XX:+UseEpsilonGC This will obviously grow your memory until you get an OOM exception, so either it's a short-lived scenario or your program is optimized not to create objects
If your service is somewhat latency sensitive, but the allowed tolerance permits some leeway , run GC1 and feed it something like -XX:MaxGCPauseTimeMillis=100 (default is 250ms)
If the issue spurs from external libraries , say one of them calls System.gc() or Runtime.getRuntime().gc() which are stop-the-world garbage collectors, you can override offending behaviour by running with -XX:+DisableExplicitGC
If you're running on a JVM above 11, do try the Z Garbage Collector (ZGC), performance improvements are monumental! -XX:+UnlockExperimentalVMOptions -XX:+UseZGC. You may also want to check this JDK 21 GC benchmark.

Version Start	Version End	Default GC
Java 1	Java 4	Serial Garbage Collector
Java 5	Java 8	Parallel Garbage Collector
Java 9	ongoing	G1 Garbage Collector

Note 1: since Java 15, ZGC is production ready, but you still have to explicitly activate it with -XX:+UseZGC.

Note 2: The VM considers machines as server-class if the VM detects more than two processors and a heap size larger or equal to 1792 MB. If not server-class, it will default to the Serial GC.

In essence, opt for GC tuning when it's clear that the application's performance constraints are directly tied to garbage collection behavior and you have the necessary expertise to make informed adjustments. Otherwise, trust the JVM's default settings and focus on optimizing application-level code.

u/shiphe - you'll want to read the full comment

Other relevant libraries you may want to explore:

Java Microbenchmark Harness (JMH)

If you're optimizing out of feeling without any real benchmarking, you're doing yourself a disservice. JMH is the de facto Java library to test your algorithms' performance. Use it.

Java-Thread-Affinity

Pinning a process to a specific core may improve cache hits. It will depend on the underlying hardware and how your routine is dealing with data. Nonetheless, this library makes it so easy to implement that, if a CPU intensive method is dragging you, you'll want to test it.

LMAX Disruptor

This is one of those libraries that, even if you don't need, you'll want to study. The idea is to allow for ultra low latency concurrency. But the way it's implemented, from mechanical sympathy to the ring buffer, brings a lot of new concepts. I still remember when I first discovered it, seven years ago, pulling an all-nighter to digest it.

Netflix jvmquake

The premiss of jvmquake is that when things go sideways with the JVM, you want it to die and not hang. A couple of years ago, I was running simulations on an HTCondor cluster that was on tight memory constraints and sometimes jobs would get stuck due to "out of memory" errors. This library forces the JVM to die, allowing you to deal with the actual error. On this specific case, HTCondor would auto re-schedule the job.

Final thoughts

The code that made me write this post? I've written way worse. I still do. The best we can hope for is to continuously mess up less.

I'm expecting to be disgruntled looking at my own code a few years down the road.

And that's a good sign.

Given the nature of this post, I found it appropriate to promote a product I've had for quite some time, a home shredder!

This is the 3rd different model/brand I've had and can definitely attest to Amazon Basics sturdiness.

While I do prefer signed and encrypted PDFs, there are a lot of financial institutions that still share data via paper. I shred those. Then I shred some other trivial stuff just to make it safe by obscurity.

In all honesty, I find it soothing.

Edits & Thank You:

to FlorianSchaetz for catching the mutability error in visitedCountries() and the detailed explanation.
to u/brunocborges and u/BikingSquirrel for explaining that on lower end machines, you get Serial GC
to u/shiphe for taking the time to better explain when you should mess with the GC and when you shouldn't
to u/tomwhoiscontrary to put me in the right track regarding what should be considered standard practice
to u/BikingSquirrel (again) for providing the link to JDK 21 garbage collectors benchmark

How to calculate P&L

wasteofserver — Thu, 07 Mar 2024 06:29:30 +0000

This post was originally posted on wasteofserver.com, you may find newer revisions and additional comments there.

Calculating profits and losses from a mathematical perspective is trivial, just subtract all things sold to all things bought. The remainder is your profit. That's pretty much how mom-and-pop retail works.

In future markets tough, people may get confused as you'll have open positions, which in retail would translate to inventory, that may not have been bought yet (if you're short selling).

Say you're trading in Mini S&P Futures from CME.

The contract lot size is 50 units and it's currently trading at around 5140. Let's see how that works out in a simple BUY/SELL trade:

// final value is num_contracts * lot_size * price
BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
                           +100 USD (PNL)

Let's add a third order to the mix to see how it works:

BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
SELL 2 ESH4 @ 5142.00 > 514_200 USD

If you think it through, you have an open position of -2 ES contracts that you'll eventually have to close. To calculate the instant P&L you must assume current market price.

BUY  2 ESH4 @ 5140.00 > 514_000 USD
SELL 2 ESH4 @ 5141.00 > 514_100 USD
SELL 2 ESH4 @ 5142.00 > 514_200 USD
// current market price is 5145.00
// you have -2 ES contracts to close
BUY  2 ESH4 @ 5145.00 > 514_500 USD
                           -200 USD (PNL)

Simple arithmetic. Buy orders are treated as negative values, sell orders are treated as positive values. That's really all you need to think about.

If you're not flat, you'll have a position that you must pretend to close (at current market value) to calculate your current P&L.

BUY  3 ESH4 @ 5100 = -3 * 50 * 5100
BUY  2 ESH4 @ 5150 = -2 * 50 * 5150
BUY  1 ESH4 @ 5200 = -1 * 50 * 5200
SELL 4 ESH4 @ 5300 = +4 * 50 * 5300
// contracts open (-3) - 2 - 1 + 4 = -2
                   = -2 * 50 * current_market_price

So now that we've covered the basis, you really only need to iterate all the orders, count buys as negative, sells as positive and, in the end, calculate whatever is open at current market price.

What about performance?

It's pretty evident that given a large amount of orders, doing this calculation for every single tick will become troublesome very fast. But as you've probably guessed by now, the system only needs to do that calculation on fills.

As soon as an order gets filled, you calculate the P&L for all filled positions and cache it. You also cache the number of contracts open. Then you just need to do a single calculation on price change:

pnl_realized + contracts_open * lot_size * current_market_price

If your system is not doing any sort of risk control, you can even push that calculation to the user GUI.

Commission fees?

Once you get the mechanics, they are very simple to add. Even if your current broker offers a flat fee, you'll want to delegate such calculation to its own class, as it will make code more extensible to future changes. I'm thinking about quantity rebates, tiers, etc.

I've recently been gifted this Amazon Basics phone holder, and I'm converted.

I use it to browse online newspappers during breakfast and it also serves as a recipe holder in the kitchen. It has proven its worth daily. Highly recommended!

SSH, Mosh and tmux

wasteofserver — Mon, 30 Oct 2023 18:00:22 +0000

This post was originally posted on wasteofserver.com, you may find newer revisions and additional comments there.

In one way or another, this post has been written thousands of times across the web. I'm rewriting it for two reasons:

Command reference
mosh under Cygwinerror.

Mosh (mobile shell)

Mosh uses UDP to create an unbreakable ssh session. If you're on the road with flaky connections, it's a lifesaver.

On Windows, you can use mosh either in WSL or Cygwin. If using via Cygwin, you may stumble on the error «Did not find remote IP address (is SSH ProxyCommand disabled?)».

$ ./mosh frankie@192.168.5.2
CreateProcessW failed error:2
posix_spawnp: No such file or directory
./mosh: Did not find remote IP address (is SSH ProxyCommand disabled?).

To fix it, it's a simple as passing a few extra parameters:

$ ./mosh --predict=always --experimental-remote-ip=remote frankie@192.168.5.2

--predict=controls use of speculative local echo

--experimental-remote-ip= used to discover IP address mosh connects to

Notice that if you're having errors connecting, you should check if you're actually allowed to UDP into the server. For that, use netcat.

nc -u <host> <port>

check connection to mosh server (generally port 60001)

tmux (terminal multiplexer)

tmux puts your terminal connection into a proper session. Say you're working on a remote server from your home. You can disconnect that session. Go to work and resume the session exactly as it was.

On top of that, it allows you to easily split the screen.

tmux new - to start

tmux new -s session-name - to name the session

tmux -t session-name - to attach to the named session

During the session, your go-to command is Ctrl-b.

Ctrl+b % - splits the screen vertically

Ctrl+b " - splits the screen horizontally

Ctrl+b x - closes current session

Enjoy your super-powered ssh connection!

I've never had one of these iPhone cables fail.

Even after being chewed up by Roomba. It's funny how Amazon does a better iPhone cable than Apple, but there you have it. Can't recommend them enough.