DEV Community: Tomasz Wegrzanowski

Break XOR Cipher with Z3

Tomasz Wegrzanowski — Sun, 01 Jan 2023 03:38:57 +0000

In previous post, I showed how to break Caesar cipher with Z3.

This wasn't really all that exciting as Caesar cipher has only 26 possibilities, so it's triival to break with brute force, and you can even just display all possible answers and choose the right one manually.

So let's do something slightly more complicated - a 4 characetr XOR cipher. It has 4.3 billion possible keys, so solving it by brute force would be a hassle.

Writing solvers for XOR cipher is still fairly basic CTF exercise, but at least we're one step into not completely trivial territory.

The solver can handle XOR ciphers of arbitrary length as long as it's known.

The Challenge

The following plaintext from Wikipedia:

Cryptography, or cryptology is the practice and study of techniques for secure communication in the presence of adversarial behavior. More generally, cryptography is about constructing and analyzing protocols that prevent third parties or the public from reading private messages. Modern cryptography exists at the intersection of the disciplines of mathematics, computer science, information security, electrical engineering, digital signal processing, physics, and others. Core concepts related to information security (data confidentiality, data integrity, authentication, and non-repudiation) are also central to cryptography. Practical applications of cryptography include electronic commerce, chip-based payment cards, digital currencies, computer passwords, and military communications.

Random encryption

This program picks a random key, and runs XOR cipher on provided plaintext:

#!/usr/bin/env ruby

key_len = 4
key = key_len.times.map{ rand(256) }
text = STDIN.read.bytes
encrypted = (0...text.size).map{|i| text[i] ^ key[i % key_len]}.pack("C*")
print encrypted

The output is unprintable binary so I'm not including it here.

How to break Caesar Cipher with Z3

Z3 rules explaining XOR cipher are easier than for Caesar cipher, we just need to use 8-bit bit vectors instead of integers.

We add as a hard assumption that all characters are in printable range, either 0x0A (newline), or a printable ASCII character (0x20-0x7E).

So all we need is some rule how to decide which decoding is correct. And for that I'm just counting two most common characters in English text - space and lowercase "e". For longer keys, shorter texts, or something more complicated than English text we could include more rules, but this is enough.

Solver

#!/usr/bin/env ruby

require "z3"

class XorSolver
  def initialize(text)
    @key_len = 4
    @key = (0...@key_len).map{|i| Z3.Bitvec("k#{i}", 8) }
    @ct = text.bytes
  end

  def call
    @solver = Z3::Optimize.new
    # Variables for each plaintext letter
    @pt = (0...@ct.size).map{|i| Z3.Bitvec("pt#{i}", 8) }

    # This is how XOR cipher works
    (0...@pt.size).each do |i|
      @solver.assert @pt[i] == @ct[i] ^ @key[i % @key_len]
    end

    # All characters are newline (0x0A) or printable character (0x20-0x7E)
    @pt.each do |pti|
      @solver.assert Z3.Or(pti == 0x0A, pti.unsigned_ge(0x20))
      @solver.assert pti.unsigned_le(0x7E)
    end

    # Get as many space, and lowercase "e" as possible
    @pt.each do |pti|
      @solver.assert_soft pti == 0x20
      @solver.assert_soft pti == 'e'.ord
    end

    if @solver.satisfiable?
      model = @solver.model
      puts @pt.map{|pti| model[pti].to_s.to_i.chr}.join
    else
      puts "No solution found"
    end
  end
end

text = open(ARGV[0] || "encrypted.txt").read.b
XorSolver.new(text).call

Is it practical?

This is actually a reasonable way to write simple solvers for CTFs. They're not as performant as hand made solvers, but it's a lot more straightforward to write a Z3 solver than a hand made one. If you know encryption algorithm exactly, it's weak, and you have partial knowledge of the plaintext - situation that's essentially limited to easy CTFs - Z3 can be of some use.

Code

All code for this post is available on GitHub.

Break Caesar Cipher with Z3

Tomasz Wegrzanowski — Sat, 31 Dec 2022 18:03:12 +0000

Z3 is usually given a list of hard constraints, and told to solve them, but it can do quite a few more things.

Let's use it unconventionally and break Caesar cipher.

The Challenge

The following plaintext from Wikipedia:

In cryptography, a Caesar cipher, also known as Caesar's cipher, the shift cipher, Caesar's code or Caesar shift, is one of the simplest and most widely known encryption techniques. It is a type of substitution cipher in which each letter in the plaintext is replaced by a letter some fixed number of positions down the alphabet. For example, with a left shift of three, D would be replaced by A, E would become B, and so on. The method is named after Julius Caesar, who used it in his private correspondence.

I replaced digit 3 by word "three", as Caesar cipher doesn't handle punctuation or digits - understandably as Romans used letters for digits.

It also explains what we're doing.

Random encryption

This program picks a random key, and runs Caesar cipher on it:

#!/usr/bin/env ruby

key = rand(0..25)
text = STDIN.read.downcase.gsub(/[^a-z]/, "")

puts text.bytes.map{|c| (((c-'a'.ord)+key) % 26 + 'a'.ord).chr}.join

I got the following output:

pujyfwavnyhwofhjhlzhyjpwolyhszvruvduhzjhlzhyzjpwolyaolzopmajpwolyjhlzhyzjvklvyjhlzhyzopmapzvulvmaolzptwslzahuktvzadpklsfruvdulujyfwapvualjoupxblzpapzhafwlvmzbizapabapvujpwolypudopjolhjoslaalypuaolwshpualeapzylwshjlkifhslaalyzvtlmpelkubtilyvmwvzpapvuzkvduaolhswohilamvylehtwsldpaohslmazopmavmaoyllkdvbskilylwshjlkifhldvbskiljvtlihukzvvuaoltlaovkpzuhtlkhmalyqbspbzjhlzhydovbzlkpapuopzwypchaljvyylzwvuklujl

How to break Caesar Cipher old fashioned way

It's actually really easy to do it by brute force, as we only have 26 possible keys. So there are 26 possible decryptions, and you can check which one is most likely by simple methods like checking most frequent letters, common words in output and so on.

How to break Caesar Cipher with Z3

The problem for a constraint solver like is that technically all 26 possible keys are "valid", so if we just tell it what the ciphertext is, and how Caesar's cipher works, it would give us one of 26 possible valid decodings, but which one?

Fortunately in addition to default "hard constraints" Z3 also supports "soft constraints". We can list a lot of them, and Z3 will try to give us a solution that fulfills as many as possible, in addition to every hard constraint. We could also give them weights, put them in different groups, and so on, but for now we'll just use the simplest solution. And our only constraint is that each letter is "E". So Z3 will try to give us solution with as many "E"s as possible.

Solver

#!/usr/bin/env ruby

require "z3"

class CaesarSolver
  def initialize(text)
    @key = Z3.Int("key")
    @ct = text.bytes.map{|c| to_number(c)}
  end

  def to_letter(i)
    (i + 'a'.ord).chr
  end

  def to_number(c)
    c.ord - 'a'.ord
  end

  def call
    @solver = Z3::Optimize.new
    # Variables for each plaintext letter
    @pt = (0...@ct.size).map{|i| Z3.Int("pt#{i}")}

    # Possible keys
    @solver.assert @key >= 0
    @solver.assert @key <= 25

    # This is how Caesar cipher works
    @pt.zip(@ct).each do |p,c|
      # modulo is a bit awkward
      @solver.assert p >= 0
      @solver.assert p <= 25
      @solver.assert Z3.Or(p == @key + c, p == @key + c - 26)
    end

    # Tell Z3 that there's probably a lot of common letters like E
    @pt.each do |p|
      @solver.assert_soft p == to_number('e')
    end

    if @solver.satisfiable?
      model = @solver.model
      puts @pt.map{|p| to_letter(model[p].to_i)}.join
    else
      puts "No solution found"
    end
  end
end

text = open(ARGV[0] || "encrypted.txt").read.chomp
CaesarSolver.new(text).call

Is it practical?

This should break a lot of encrypted messages, unless they're very short, very unusual, or not in English.

We could make it more powerful, by telling it about other common letters like T, A, O, I, N etc, with somewhat lower weights. We could even include common letter combinations etc.

It is quite slow, and brute forcing 26 solution, and counting ones with most Es in a lot faster. The difference is that Z3 can deal with situations where brute force isn't possible.

In this case would could probably improve Z3 performance by using bitvectors instead of integers, as then would have native modulo operations.

A good example of a cipher that would likely work quite well in Z3 but might be difficult to break by brute force would be a XOR cipher with short key. Of course for all such common ciphers there already exist fast tools to break them, so it's mostly useful if you're trying to solve a CTF challenge with a novel but simple cipher.

Code

All code for this post is available on GitHub.

Getting Started with Web Speech Synthesis API and Svelte

Tomasz Wegrzanowski — Sun, 25 Dec 2022 18:07:52 +0000

Browser get new APIs all the time, one of such APIs is Web Speech Synthesis. Let's explore it with Svelte.

Getting Started with the API

The entry point of the API is speechSynthesis object. To get list of available voices we can do:

console.log(speechSynthesis.getVoices())

Which on my Chrome on OSX laptop returns a list of 67 voices.

So let's just create a new page, and put that in JavaScript, and... it returns an empty array. What just happened?

Unfortunately the Web Speech Synthesis API is terribly designed. The list of voices is populated asynchronously, which is fair enough, but instead of just returning a promise or some event like onSpeechSynthesisReady it will just happily return an empty array if you call it too early.

There is onVoicesChanged event. Nothing in the spec says even implies that it will only trigger once during page load, and I think a browser could work the other way around (have voice list pre-populated, so no event triggers), but it seems to work fine in Chrome.

For any production use it would likely needa lot more robust code and some cross-browser cross-OS testing. Arguably a timeout loop of checking it every 16ms until it's non-empty or some max timeout elapsed might even be more robust, but we're just exploring the API here.

Display list of available voices

To get started we can wrap this event in a promise, and use Svelte to await on the promise.

<script>
  let voicesPromise = new Promise((resolve) => {
    speechSynthesis.addEventListener("voiceschanged", ev => {
      resolve(speechSynthesis.getVoices())
    })
  })
</script>

<div>Available Voices:</div>
{#await voicesPromise then voices}
  <ul>
    {#each voices as voice}
      <li>{voice.name} - {voice.lang}</li>
    {/each}
  </ul>
{/await}

<style>
:global(body) {
  margin: 0;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}
</style>

Say something

Now it's just a matter of adding some radio boxes for voice selection, text input for text to say, a button to run it, and it works.

<script>
  let loading = true
  let voices = []
  speechSynthesis.addEventListener("voiceschanged", ev => {
    loading = false
    voices = speechSynthesis.getVoices()
  })
  let text = "Hello, world!"
  let voiceIndex = 0
  $: voice = voices[voiceIndex]

  function sayIt() {
    let u = new SpeechSynthesisUtterance(text)
    u.voice = voice
    speechSynthesis.speak(u)
  }
</script>

<div>
  <label>Text to say:
    <input bind:value={text} />
  </label>
  <button on:click={sayIt}>Say it</button>
</div>

{#if loading}
  <div>Please wait for voices to load</div>
{:else}
  <div>Available Voices:</div>
  <ul>
    {#each voices as v, i}
      <li>
        <label>
          <input type="radio" bind:group={voiceIndex} value={i}>
          {v.name} - {v.lang}
        </label>
      </li>
    {/each}
  </ul>
{/if}

<style>
:global(body) {
  margin: 0;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}
</style>

Some notes:

I changed from promise to loading flag as I want the array of voices in the script as well as in rendered mode, and it's slightly easier this way
We need to create new SpeechSynthesisUtterance(text) instead of just doing more obvious voice.say("some text"). SpeechSynthesisUtterance object has some additional properties like speed and pitch, so you can use it for speed reading, or for UwU voice etc.
the API has some additional event for when the speech starts, ends etc. so you might consider listening to the events to know if the browser is speaking right now if you need some visual feedback as well

Code

You can try it here.

All code for this post is available on GitHub.

Getting Started with ClickHouse

Tomasz Wegrzanowski — Sat, 24 Dec 2022 17:49:14 +0000

ClickHouse is a column-oriented database system with SQL-like interface. Its main selling point is high performance when handling enormous datasets in near real time. The sacrifice you'll need to make is very limited support for updates, deletes, or transactions. If your dataset is enormous, and you mostly read, aggregate, and insert, it could work for you.

Which arguably makes it more suitable as an secondary database, not as primary one.

Run ClickHouse with Docker

You can use ClickHouse with ClickHouse Cloud, and that's what most documentation refers to, but for some practice, you can run it locally as well.

The easiest way is to use Docker. To run a local server execute this command:

$ docker run -d --name clickhouse-server clickhouse/clickhouse-server

Then we can connect to it:

$ docker exec -it clickhouse-server clickhouse-client

If you need multiple servers, just change clickhouse-server accordingly.

Let's run some queries

d7ba07d71412 :) select 2 + 2

SELECT 2 + 2

Query id: 1565da74-dc33-4d1f-84ae-d5241b052b5c

┌─plus(2, 2)─┐
│          4 │
└────────────┘

1 row in set. Elapsed: 0.001 sec.

d7ba07d71412 :) select concat('Hello', ', ', 'world!')

SELECT concat('Hello', ', ', 'world!')

Query id: 090687b1-3a8f-4300-9368-579a87c348c0

┌─concat('Hello', ', ', 'world!')─┐
│ Hello, world!                   │
└─────────────────────────────────┘

1 row in set. Elapsed: 0.001 sec.

FizzBuzz

We don't have any data yet, but we can select from system.numbers - that's a basically infinite virtual table of all numbers starting from 0.

d7ba07d71412 :) select if(number % 15 = 0, 'FizzBuzz', if(number % 5 = 0, 'Buzz', if(number % 3 = 0, 'Fizz', toString(number)))) as FizzBuzz from system.numbers where number >= 1 limit 100

SELECT if((number % 15) = 0, 'FizzBuzz', if((number % 5) = 0, 'Buzz', if((number % 3) = 0, 'Fizz', toString(number)))) AS FizzBuzz
FROM system.numbers
WHERE number >= 1
LIMIT 100

Query id: 8badf288-95b5-4121-b4ad-18730fbfafc1

┌─FizzBuzz─┐
│ 1        │
│ 2        │
│ Fizz     │
│ 4        │
│ Buzz     │
│ Fizz     │
│ 7        │
│ 8        │
│ Fizz     │
│ Buzz     │
│ 11       │
│ Fizz     │
│ 13       │
│ 14       │
│ FizzBuzz │
│ 16       │
│ 17       │
│ Fizz     │
│ 19       │
│ Buzz     │
│ Fizz     │
│ 22       │
│ 23       │
│ Fizz     │
│ Buzz     │
│ 26       │
│ Fizz     │
│ 28       │
│ 29       │
│ FizzBuzz │
│ 31       │
│ 32       │
│ Fizz     │
│ 34       │
│ Buzz     │
│ Fizz     │
│ 37       │
│ 38       │
│ Fizz     │
│ Buzz     │
│ 41       │
│ Fizz     │
│ 43       │
│ 44       │
│ FizzBuzz │
│ 46       │
│ 47       │
│ Fizz     │
│ 49       │
│ Buzz     │
│ Fizz     │
│ 52       │
│ 53       │
│ Fizz     │
│ Buzz     │
│ 56       │
│ Fizz     │
│ 58       │
│ 59       │
│ FizzBuzz │
│ 61       │
│ 62       │
│ Fizz     │
│ 64       │
│ Buzz     │
│ Fizz     │
│ 67       │
│ 68       │
│ Fizz     │
│ Buzz     │
│ 71       │
│ Fizz     │
│ 73       │
│ 74       │
│ FizzBuzz │
│ 76       │
│ 77       │
│ Fizz     │
│ 79       │
│ Buzz     │
│ Fizz     │
│ 82       │
│ 83       │
│ Fizz     │
│ Buzz     │
│ 86       │
│ Fizz     │
│ 88       │
│ 89       │
│ FizzBuzz │
│ 91       │
│ 92       │
│ Fizz     │
│ 94       │
│ Buzz     │
│ Fizz     │
│ 97       │
│ 98       │
│ Fizz     │
│ Buzz     │
└──────────┘

100 rows in set. Elapsed: 0.002 sec. Processed 65.41 thousand rows, 523.27 KB (31.21 million rows/s., 249.67 MB/s.)

d7ba07d71412 :)

Same notes:

even though ClickHouse isn't relational database, even surprsingly complex SQL just works
system.numbers is implicitly ordered, so system.numbers where number >= 1 limit 100 will return numbers from 1 to 100
we need explicit typecast in if(number % 3 = 0, 'Fizz', toString(number)), SQL databases tend to do a lot of automatic type conversions, ClickHouse less so

Connect from Python

We won't be coding from REPL, so let's write some code. There's no official Ruby driver, but there's a Python one. And some unofficial Ruby drivers.

$ pip3 install clickhouse-connect

We also need to expose our Docker container's Clickhouse port. First let's remove the old one (we can adjust it on running container, but it's more complex and we don't have any data there anyway):

$ docker stop clickhouse-server
$ docker rm clickhouse-server

And start it with port 8123 mapped:

$ docker run -p 8123:8123 -d --name clickhouse-server clickhouse/clickhouse-server

Python Clickhouse FizzBuzz

Now we can write some Python code to talk to our database:

#!/usr/bin/env python3

import clickhouse_connect

client = clickhouse_connect.get_client()
query = """
  select
    if(number % 15 = 0, 'FizzBuzz',
    if(number % 5 = 0, 'Buzz',
    if(number % 3 = 0, 'Fizz',
    toString(number))))
    as FizzBuzz
  from system.numbers
  where number >= 1
  limit 100
  """

print(client.command(query))

The result of client.command(query) is a multiline string, so it's mostly useful for debugging.

There are other methods that return more obvious data structures like list of tuples, but also Pandas dataframe and so on.

What's next

So we got ClickHouse running. In a future post, we'll make it work with some real data.

Code

All code for this post is available on GitHub.

PyScript: Second Impressions

Tomasz Wegrzanowski — Sat, 24 Dec 2022 14:32:52 +0000

PyScript lets you run Python directly in the browser.

I tried PyScript once before, but back then it didn't even run. Fortunately it got over this issue, and now it runs.

Installation

There's probably some way to get it through npm or pip, but PyScript is intended to be used with just plain JS and CSS files.
As it's still pre-1.0, and API can change anytime, let's get local copy of these two files:

Hello, World!

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>Hello World</title>
  <link rel="stylesheet" href="./pyscript.css" />
  <script defer src="./pyscript.js"></script>
  <style>
    body {
      margin: 0;
    }
  </style>
</head>
<body>
  <py-script>
    print("Hello World")
  </py-script>
</body>
</html>

You can view it here.

FizzBuzz

Of course we can do a FizzBuzz:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>FizzBuzz</title>
  <link rel="stylesheet" href="./pyscript.css" />
  <script defer src="./pyscript.js"></script>
  <style>
    body {
      margin: 0;
    }
  </style>
</head>
<body>
  <py-script>
    for i in range(1, 101):
      if i % 15 == 0:
        print("FizzBuzz")
      elif i % 3 == 0:
        print("Fizz")
      elif i % 5 == 0:
        print("Buzz")
      else:
        print(i)
  </py-script>
</body>
</html>

You can view it here.

Prompt Input

So we got it running in the browser, but it did nothing exciting.

Interaction is extremely limited. For an obvious approach, we could use input function:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>Prompt Input</title>
  <link rel="stylesheet" href="./pyscript.css" />
  <script defer src="./pyscript.js"></script>
  <style>
    body {
      margin: 0;
    }
  </style>
</head>
<body>
  <py-script>
    name = input("What is your name? ")
    print(f"Hello, {name}")
  </py-script>
</body>
</html>

But it will use browser prompt without showing the prompt question, and with the page being totally blank! You only see the prompt after you answered it. So that's pretty much useless.

You can view it here.

Interacting with DOM

PyScript has extremely minimal set of DOM bindings.

You can select DOM elements with pyscript.Element, and you can read and write from them, but there are no event bindings, so it's not possible to have some button that calls your Python code when clicked, or to have Python track when some form input field changes. Even worse, there's no way to create elements, either by DOM APIs or even through innerHtml. You can clone elements, so you could write a template somewhere and copy that, but that would be far more painful than even IE4 coding.

This is not even close to being minimally usable API, which would need at least .addEventListener and at least one of .innerHtml= or .createElement.

There are also no HTTP requests either.

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>DOM</title>
  <link rel="stylesheet" href="./pyscript.css" />
  <script defer src="./pyscript.js"></script>
  <style>
    body {
      margin: 0;
    }
    div {
      white-space: pre-wrap;
    }
  </style>
</head>
<body>
  <div id="output"></div>
  <py-script>
    from pyscript import Element
    def fizzbuzz(i):
      if i % 15 == 0:
        return "FizzBuzz"
      elif i % 3 == 0:
        return "Fizz"
      elif i % 5 == 0:
        return "Buzz"
      else:
        return str(i)
    list = Element("output")
    list.write("\n".join([fizzbuzz(i) for i in range(1, 101)]) )
  </py-script>
</body>
</html>

You can view it here.

Should you use PyScript?

The answer changed slightly in the sense that you can at least run it, but it's not even a legitimate proof-of-concept.

It's sort of impressive that you can get Python code running in the browser, sort of, but lack of any way to interact with the page makes it unusuable even for a demo.

Still, maybe it will turn into a real thing at some point.

Code

All code for this post is available on GitHub.

Upgrading old Imba 1 apps to run on modern Node

Tomasz Wegrzanowski — Sun, 18 Dec 2022 13:16:03 +0000

Node has the least stable ecosystem of all the commonly used languages. With Ruby or Python or frontend JavaScript, you can use take a project that's a few years old, and it will generally run just fine. Frontend JavaScript is one of the most stable languages.

With Node, it never happens. Every new Node version breaks pretty much every old package, and all you'll see is a wall of node-gyp errors.

This happened to me as well. Back when I was playing with various frameworks, I discovered Imba 1, and liked it quite a bit. So I built a lot of small apps, with Imba and SCSS. And of course SCSS won't work on latest Node. There was also a hell of webpack plugin dependencies involved, as Imba 1 and SCSS needed some webpack plugins, and I can't just upgrade that to latest version either.

GitHub's dependabot also keeps bothering me to upgrade dependencies to fix "security issues". This sounds like a good idea, except it has about >99% false positive rate overall, but the upgrades will absolutely break your apps.

While I was at it, I also did a Cypress upgrade, but that's an easy process of running cypress open and using builtin migration tool.

Dependencies

Before:

  "scripts": {
    "dev": "webpack-dev-server --mode=development --content-base dist/",
    "build": "webpack --mode=production",
    "watch": "webpack --watch --progress --colors --display-modules",
    "predeploy": "pnpm run build",
    "deploy": "gh-pages -d dist"
  },
  "dependencies": {
    "css-loader": "^1.0.1",
    "cypress": "^3.4.1",
    "express": "^4.16.2",
    "extract-text-webpack-plugin": "^4.0.0-beta.0",
    "gh-pages": "^2.0.1",
    "imba": "1.5.2",
    "normalize-scss": "^7.0.1",
    "style-loader": "^0.23.1",
    "webpack-dev-server": "^3.7.1"
    "node-sass": "^4.12.0",
    "sass-loader": "^7.1.0",
    "webpack": "^4.26.0",
    "webpack-cli": "^3.1.2"
  }

After:

  "scripts": {
    "dev": "webpack-dev-server --mode=development --static dist/",
    "build": "webpack --mode=production",
    "watch": "webpack --watch --progress --colors --display-modules",
    "predeploy": "npm run build",
    "deploy": "gh-pages -d dist"
  },
  "dependencies": {
    "css-loader": "^5.2.7",
    "cypress": "^12.1.0",
    "gh-pages": "^4.0.0",
    "imba": "1.5.2",
    "mini-css-extract-plugin": "^1.6.2",
    "node-sass": "^8.0.0",
    "normalize-scss": "^7.0.1",
    "sass-loader": "^10.4.1",
    "webpack": "^4.46.0",
    "webpack-cli": "^4.10.0",
    "webpack-dev-server": "^4.11.1"
  }

It took some serious trial and error. The important thing is that I needed to stay on Webpack 4 for Imba 1 to work, so for everything I picked the most recent version that still worked with Webpack 4. I fully expect it to stop working eventually, when Node 30 breaks Webpack 4, and all that code becomes unrunnable, while people can still run frontend JavaScript from the 90s without any issues whatsoever. But at least it fixes it for now.

`webpack.config.js`

Before:

let ExtractTextPlugin = require('extract-text-webpack-plugin');

module.exports = {
  module: {
    rules: [
      {
        test: /\.imba$/,
        loader: 'imba/loader',
      },
      {
        test: /\.scss$/,
        use: ExtractTextPlugin.extract({
          fallback: 'style-loader',
          use: ['css-loader', 'sass-loader']
        })
      }
    ],
  },
  resolve: {
    extensions: [".imba", ".js", ".json", ".scss"]
  },
  entry: ["./src/app.imba", "./src/app.scss"],
  output: {  path: __dirname + '/dist', filename: "app.js" },
  plugins: [
    new ExtractTextPlugin('app.css')
  ]
}

After:

let MiniCssExtractPlugin = require('mini-css-extract-plugin');

module.exports = {
  module: {
    rules: [
      {
        test: /\.imba$/,
        loader: 'imba/loader',
      },
      {
        test: /\.scss$/,
        use: [
          MiniCssExtractPlugin.loader,
          "css-loader",
          "sass-loader",
        ]
      }
    ],
  },
  resolve: {
    extensions: [".imba", ".js", ".json", ".scss"]
  },
  entry: ["./src/app.imba", "./src/app.scss"],
  output: {  path: __dirname + '/dist', filename: "app.js" },
  plugins: [
    new MiniCssExtractPlugin({filename: 'app.css'})
  ]
}

The plugin package changed, and so did its options.

Code

For now I upgraded imba-rotn package. The code is available here. It's also available online here.

Now I have about 30 more Imba 1 packages that all need similar fix.

How to access Kaggle data from command line

Tomasz Wegrzanowski — Sun, 20 Nov 2022 17:46:48 +0000

The war isn't going anywhere for now, so every couple of days I have to do the following steps to update Russian losses tracker:

download zip from Kaggle
run update_csv script
optionally, verify that data looks right with git diff, as occasionally there's a typo which makes losses go backwards (it happened only a few times, and it always gets corrected in the next update)
delete archive.zip

The annoying part is that Kaggle requires me to be logged in in a browser to download data, so I can't just replace that step with a curl request.

So let's try to improve this flow a bit.

Get Kaggle API token

You need to create an account on Kaggle.

Then go to your account settings by clicking on top right icon, and selecting Account (https://www.kaggle.com/<name>/account).

There's "Create New API Token" button, which will create new account token, and download it as kaggle.json. Create ~/.kaggle folder, and save that file to ~/.kaggle/kaggle.json.

Kaggle will complain if you don't secure the file so run this: chmod 0600 ~/.kaggle/kaggle.json

Install Kaggle CLI tools

If you have Python3 installed, you just need to do pip3 install kaggle

Download dataset

User name and ID of the data set are in the URL, so to download https://www.kaggle.com/datasets/piterfm/2022-ukraine-russian-war you need to run:

$ kaggle datasets download piterfm/2022-ukraine-russian-war

It will save it as 2022-ukraine-russian-war.zip. There are extra options like where you want to download it, or unzipping it etc.

Full process

Now I can automate the whole process:

$ kaggle datasets download piterfm/2022-ukraine-russian-war
$ ./update_csv 2022-ukraine-russian-war.zip
$ trash 2022-ukraine-russian-war.zip
$ git add -u
$ git ci -m 'Data Update'
$ git push

And since it's just a series of commands, I can even make it run automatically every day, without any intervention.

I could also add some kind of data checks to the process, so if there's anything weird like numbers going backwards, it would stop the update and wait for the next day. But overall, I'm happy with how it all ended up.

Reading from Zip archives in Ruby

Tomasz Wegrzanowski — Sat, 05 Nov 2022 12:09:01 +0000

The war isn't going anywhere for now, so every couple of days I have to do the following steps to update Russian losses tracker:

download zip from Kaggle
unzip it with unall utility
run update_csv script
verify that data looks right with git diff, as occasionally there's a typo which makes losses go backwards (always corrected on next update)
delete archive folder

The part that I'd like to get rid of is unzipping. We know exactly the path inside the zip, so why bother?

Old script

The script is very straightforward, the methods updated_equipment and updated_personnel will need replacing to get data from a zip.

#!/usr/bin/env ruby

require "pathname"

class UpdateCSV
  def initialize(archive_path)
    @archive_path = Pathname(archive_path)
  end

  def updated_equipment
    @updated_equipment ||= (@archive_path + "russia_losses_equipment.csv").read
  end

  def updated_personnel
    @updated_personnel ||= (@archive_path + "russia_losses_personnel.csv").read
  end

  def csv_files
    @csv_files ||= `git ls`.lines.map(&:chomp).grep(/\.csv\z/)
  end

  def call
    csv_files.each do |path|
      case path
      when /russia_losses_equipment/
        Pathname(path).write(updated_equipment)
      when /russia_losses_personnel/
        Pathname(path).write(updated_personnel)
      else
        puts "Unknown CSV file: #{path}"
      end
    end
  end
end

unless ARGV[0]
  STDERR.puts "Usage: #{$0} path_to_updated_archive"
  exit 1
end

UpdateCSV.new(ARGV[0]).call

Gem `rubyzip` vs `zip`

There's a bit of a problem with Zip for Ruby, as there's two gems - rubyzip and zip. rubyzip is the correct one, zip is an obsolete fork that somehow got a better name.

To use either of them, you need to do require "zip" - which will use whichever one is installed. This is a leftover mess from early days of RubyGems, and such things don't really happen anymore. Just don't gem install zip, and you'll be good.

Abstract archive handling

There's going to be some shared code between updated_equipment and updated_personenel, so let's move it into a new method read_file:

  def updated_equipment
    @updated_equipment ||= read_file("russia_losses_equipment.csv")
  end

  def updated_personnel
    @updated_personnel ||= read_file("russia_losses_personnel.csv")
  end

Read file from either directory or archive

And now the read_file method:

  def read_file(path)
    if @archive_path.directory?
      (@archive_path + path).read
    else
      Zip::File.open(@archive_path).read(path)
    end
  end

There are a lot more things we could be doing with zips, but just reading is all we need.

And that's it! It saves me one of the steps.

Full code

Here's the full script:

#!/usr/bin/env ruby

require "pathname"
require "zip"

class UpdateCSV
  def initialize(archive_path)
    @archive_path = Pathname(archive_path)
  end

  def read_file(path)
    if @archive_path.directory?
      (@archive_path + path).read
    else
      Zip::File.open(@archive_path).read(path)
    end
  end

  def updated_equipment
    @updated_equipment ||= read_file("russia_losses_equipment.csv")
  end

  def updated_personnel
    @updated_personnel ||= read_file("russia_losses_personnel.csv")
  end

  def csv_files
    @csv_files ||= `git ls`.lines.map(&:chomp).grep(/\.csv\z/)
  end

  def call
    csv_files.each do |path|
      case path
      when /russia_losses_equipment/
        Pathname(path).write(updated_equipment)
      when /russia_losses_personnel/
        Pathname(path).write(updated_personnel)
      else
        puts "Unknown CSV file: #{path}"
      end
    end
  end
end

unless ARGV[0]
  STDERR.puts "Usage: #{$0} path_to_updated_archive"
  exit 1
end

UpdateCSV.new(ARGV[0]).call

Open Source Adventures: Episode 82: Series Retrospective

Tomasz Wegrzanowski — Fri, 04 Nov 2022 18:42:26 +0000

It's time to finish the series.

Previously I did Electron Adventures and 100 Languages Speedrun, which were both highly focused 100-episode 1-episode-per-day series.

I have a few more ideas for big series, but the time wasn't right for that, and I wanted to write some disconnected posts instead. So I labelled them "Open Source Adventures" even though without any focus, and without any schedule, it doesn't make sense to treat it as a "series". Having label like "Episode 76" on what's essentially a disconnected post is only confusing. And it included a few short series of posts, which should get their own miniseries instead (possibly without "Episode XX" numbering).

So I'm hereby closing the series. The content was good, but the organization really didn't make much sense.

I might do another megaseries at some point, depending on time availability. But more likely I'll just post standalone posts and short series of a few posts for now.

Open Source Adventures: Episode 81: Exploring Raku Regular Expression API

Tomasz Wegrzanowski — Wed, 14 Sep 2022 15:08:30 +0000

In previous three episodes I explored regular expression APIs of Ruby, Crystal, and Python, so let's finish this by doing the same exercise in Raku.

The problem is the same - there's multiple date formats, and we want to extract information from whichever one matches.

I'm doing it with just 3 regular expressions, but in real world there could be hundreds. Doing it naively with a list of regular expressions would require massive code duplication, and a lot of calls to regular expression engine, which is generally dramatically slower than just matching a|b|c|... once.

The Problem

#!/usr/bin/env raku

use JSON::Fast;

for qw[2015-05-25 2016/06/26 27/07/2017] {
  say to-json(parse_date($_), :!pretty)
}

And expected output is:

[2015,5,25]
[2016,6,26]
[2017,7,27]

Solution 1

sub parse_date($s) {
  if $s ~~ /(\d\d\d\d)\-(\d\d)\-(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif $s ~~ /(\d\d\d\d)\/(\d\d)\/(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif $s ~~ /(\d\d)\/(\d\d)\/(\d\d\d\d)/ {
    [+$2, +$1, +$0]
  }
}

The first way is to just list every possible regular expression sequentially.

In case you're not familiar with Raku, here's a few minor things to notice:

match groups are numbered from $0 not from $1
these variables aren't String type like in other languages, they're Match
we can't to-json a Match object without some kind of conversion, either to string or ot number, so returning [$0, $1, $2] would just crash
+ converts to a number
match operator is ~~, not =~
\d is Unicode digit, not just 0 to 9

There's also a bigger thing to consider - in other languages, unknown punctuation like / or - can be used in regular expression literally. In retrospect this was a mistake, as it prevents adding new regular expression syntax without breaking backwards compatibility. Raku forces escaping every such punctuation, even currently unused, so at some point in the future it can make - or / have some meaning.

Anyway, just like solution 1 in other languages, this suffers from both problems - code duplication, and poor performance due to sequential match.

Solution 2

sub parse_date($_) {
  if /(\d\d\d\d)\-(\d\d)\-(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif /(\d\d\d\d)\/(\d\d)\/(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif /(\d\d)\/(\d\d)\/(\d\d\d\d)/ {
    [+$2, +$1, +$0]
  }
}

Just like in Perl, we don't need to use ~~, if we use regular expression in boolean context it will automatically match $_.

Solution 3

sub parse_date($_) {
  if /(\d\d\d\d)\-(\d\d)\-(\d\d)/ or /(\d\d\d\d)\/(\d\d)\/(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif /(\d\d)\/(\d\d)\/(\d\d\d\d)/ {
    [+$2, +$1, +$0]
  }
}

We can reduce code duplication if groups are in the same order.

Solution 4

sub parse_date($_) {
  if /(\d\d\d\d)\-(\d\d)\-(\d\d) | (\d\d\d\d)\/(\d\d)\/(\d\d)/ {
    [+$0, +$1, +$2]
  } elsif /(\d\d)\/(\d\d)\/(\d\d\d\d)/ {
    [+$2, +$1, +$0]
  }
}

What's going on here? In Raku $0, $1, $2 don't mean the Nth match group in the expression, it means Nth match group that actually matched (at least for | alternatives, the full story is more complicated)!

This is great as we can put as many expressions as we want without any nonsense like +($0 or $3 or $6).

On the other hand, this means we have no way to do alternative if they're in a different order.

Solution 5

sub parse_date($_) {
  if /(\d\d\d\d)\-(\d\d)\-(\d\d) |
      (\d\d\d\d)\/(\d\d)\/(\d\d) |
      (\d\d)\/(\d\d)\/(\d\d\d\d)/ {
    [+$0, +$1, +$2]
  }
}

This doesn't actually work. The block can't tell which branch matched, so it doesn't know if the groups are in YMD or DMY order.

Solution 6

sub parse_date($_) {
  if /
    $<y>=(\d\d\d\d) \- $<m>=(\d\d) \- $<d>=(\d\d) |
    $<y>=(\d\d\d\d) \/ $<m>=(\d\d) \/ $<d>=(\d\d) |
    $<d>=(\d\d) \/ $<m>=(\d\d) \/ $<y>=(\d\d\d\d)
    / {
    [+$<y>, +$<m>, +$<d>]
  }
}

We can solve this with named captures, and it works great.

Story so far

The syntax and API are both very different from traditional regular expressions, but in the end we got everything we needed.

All the code is on GitHub.

Coming next

I was planning to write another post on how we could improve regular expression APIs, but it turns out the APIs we explored (except Python's disappointingly limited one) actually have features that cover most of what I wanted to say, even if they're rarely used in the real world yet.

So the next episode will be about something completely different.

Open Source Adventures: Episode 80: Exploring Python Regular Expression API

Tomasz Wegrzanowski — Mon, 12 Sep 2022 23:27:35 +0000

In previous two episodes I explored regular expression APIs of Ruby, and Crystal, so let's do the same exercise in Python.

The problem is the same - there's multiple date formats, and we want to extract information from whichever one matches.

The Problem

Here's the test case:

#!/usr/bin/env python3

import re

def parse_date(s):
  ...

for date in ["2015-05-25", "2016/06/26", "27/07/2017"]:
  print(parse_date(date))

And expected output:

[2015, 5, 25]
[2016, 6, 26]
[2017, 7, 27]

Solution 1

def parse_date(s):
  m = re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)', s)
  if m:
    return [int(m[1]), int(m[2]), int(m[3])]
  else:
    m = re.search(r'(\d\d\d\d)/(\d\d)/(\d\d)', s)
    if m:
      return [int(m[1]), int(m[2]), int(m[3])]
    else:
      m = re.search('(\d\d)/(\d\d)/(\d\d\d\d)', s)
      if m:
        return [int(m[3]), int(m[2]), int(m[1])]

The naive solution in Python is just awful. Python doesn't have any kind of $~, $1, etc. so we need to assign to a match variable. This isn't a big deal, most languages don't have $~, so we'd normally be able to do a chain of if m = re.search(...) followed by elif m = re.search(...) etc.

Python however, is not like most languages, and doesn't treat assignment as expression, so if m = re.search(...) is not allowed. So porting code from other languages to Python just results in truly disgusting code with ever increasing indentation. It already looks bad with 3, just imagine 100 (even with 2 space indentation, I'm not doing 4 spaces on the blog, as it looks unreadable on narrow screens).

Solution 2

def parse_date(s):
  if m := re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)', s):
    return [int(m[1]), int(m[2]), int(m[3])]
  elif m := re.search(r'(\d\d\d\d)/(\d\d)/(\d\d)', s):
    return [int(m[1]), int(m[2]), int(m[3])]
  elif m := re.search('(\d\d)/(\d\d)/(\d\d\d\d)', s):
    return [int(m[3]), int(m[2]), int(m[1])]

Python 3.8 finally added a feature which all other languages had, but instead of making = assignment an expression, it introduced a new and very limited :=.

We still have considerable code duplication here, but it's much better than the pre-3.8 solution.

Solution 3

def parse_date(s):
  m = re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)', s)
  if m:
    return [int(m[1]), int(m[2]), int(m[3])]
  m = re.search(r'(\d\d\d\d)/(\d\d)/(\d\d)', s)
  if m:
    return [int(m[1]), int(m[2]), int(m[3])]
  m = re.search('(\d\d)/(\d\d)/(\d\d\d\d)', s)
  if m:
    return [int(m[3]), int(m[2]), int(m[1])]

Because we're returning from every branch anyway, we don't actually need to do else. I think this is actually a more realistic pre-3.8 solution, but it only works because we extracted the logic to top level of a function or method, it's not always possible.

Solution 4

def parse_date(s):
  if m := re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)', s) or re.search(r'(\d\d\d\d)/(\d\d)/(\d\d)', s):
    return [int(m[1]), int(m[2]), int(m[3])]
  elif m := re.search('(\d\d)/(\d\d)/(\d\d\d\d)', s):
    return [int(m[3]), int(m[2]), int(m[1])]

We can reduce code duplication by doing m := rx1 or rx2.

Solution 5:

def parse_date(s):
  if m := re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)|(\d\d\d\d)/(\d\d)/(\d\d)', s):
    return [int(m[1] or m[4]), int(m[2] or m[5]), int(m[3] or m[6])]
  elif m := re.search('(\d\d)/(\d\d)/(\d\d\d\d)', s):
    return [int(m[3]), int(m[2]), int(m[1])]

We can reduce number of calls to the regular expression engine, and extract whichever group matches with m[num] or m[num].

There's a Python specific gotcha here. In Python empty string in falsey, so (m[1] or m[4]) will evaluate to m[4] not only when m[1] is None, but also when m[1] matches an empty string "". So you can only use this technique if you're sure no group is ever empty.

I'm not even sure if Python has any a or b that only evaluates b if a is specifically None.

Solution 6:

def parse_date(s):
  if m := re.search(r'(\d\d\d\d)-(\d\d)-(\d\d)|(\d\d\d\d)/(\d\d)/(\d\d)|(\d\d)/(\d\d)/(\d\d\d\d)', s):
    return [
      int(m[1] or m[4] or m[9]),
      int(m[2] or m[5] or m[8]),
      int(m[3] or m[6] or m[7]),
    ]

This same technique, with the same gotcha, can be used even if different patterns have different order. Of course it really doesn't scale, and even this is too many indexes.

Solution 7

def parse_date(s):
  if m := re.search(r'''(?x)
    (\d\d\d\d)-(\d\d)-(\d\d) |
    (\d\d\d\d)/(\d\d)/(\d\d) |
    (\d\d)/(\d\d)/(\d\d\d\d)
    ''', s):
    return [
      int(m[1] or m[4] or m[9]),
      int(m[2] or m[5] or m[8]),
      int(m[3] or m[6] or m[7]),
    ]

Python equivalent of //x is (?x) which must be placed immediately at the start of the regular expression, without any preceding whitespace.

Solution 8

def parse_date(s):
  if m := (
      re.search(r'(?P<year>\d\d\d\d)-(?P<month>\d\d)-(?P<day>\d\d)', s) or
      re.search(r'(?P<year>\d\d\d\d)/(?P<month>\d\d)/(?P<day>\d\d)', s) or
      re.search(r'(?P<day>\d\d)/(?P<month>\d\d)/(?P<year>\d\d\d\d)', s)
    ):
    return [int(m["year"]), int(m["month"]), int(m["day"])]

Python has a slightly different syntax for named groups, (?P<name>...) compared with Ruby's (?<name>...).

Solution 9

def parse_date(s):
  if m := re.search(r'''(?x)
    (?P<year>\d\d\d\d)-(?P<month>\d\d)-(?P<day>\d\d) |
    (?P<year>\d\d\d\d)/(?P<month>\d\d)/(?P<day>\d\d) |
    (?P<day>\d\d)/(?P<month>\d\d)/(?P<year>\d\d\d\d)
    ''', s):
    return [int(m["year"]), int(m["month"]), int(m["day"])]

And we can finally reach the solution we wanted - symbolic names at appropriate positions. One regular expression engine call, no code duplication, perfection.

Except it doesn't work in Python at all, as Python requires every group name to be unique.

As far as I know, there's no way to express it in Python as clearly as we did with Ruby/Crystal.

Solution 10

def parse_date(s):
  if m := re.search(r'''(?x)
    (?P<year1>\d\d\d\d)-(?P<month1>\d\d)-(?P<day1>\d\d) |
    (?P<year2>\d\d\d\d)/(?P<month2>\d\d)/(?P<day2>\d\d) |
    (?P<day3>\d\d)/(?P<month3>\d\d)/(?P<year3>\d\d\d\d)
    ''', s):
    return [
      int(m["year1"] or m["year2"] or m["year3"]),
      int(m["month1"] or m["month2"] or m["month3"]),
      int(m["day1"] or m["day2"] or m["day3"]),
    ]

But before we give up, let's try one more thing. Every group name is unique. This looks extremely verbose - and runs into the gotcha that any group matching empty substring would crash it - but it's probably more readable and scalable than using very high numbers.

Story so far

Python definitely disappointed a bit here. There are some solutions that are not too awful, but there's no way to fully avoid code duplication (like solution 9 would if it worked), and all the solutions using or chains only work if none of the match group can be empty, limitations Ruby and Crystal didn't have.

All the code is on GitHub.

Coming next

In the next episode we'll see how other languages handle this problem.

Open Source Adventures: Episode 79: Exploring Crystal Regular Expression API

Tomasz Wegrzanowski — Sun, 11 Sep 2022 17:37:36 +0000

In the previous episode we've taken a look at Ruby Regular Expression API. I want to try a few more languages, and the most obvious one to start with is Crystal.

A lot of solutions work exactly like in Ruby, but some of the differences are interesting.

Test case

Crystal doesn't have %W, which is one of my favorite Ruby features, but in this case its non-interpolating and much less awesome relative %w will do.

Here's the test case:

%w[
  2015-05-25
  2016/06/26
  27/07/2017
].each do |s|
  p parse_date(s)
end

And the expected result:

[2015, 5, 25]
[2016, 6, 26]
[2017, 7, 27]

Solution 1

def parse_date(s)
  case s
  when %r[(\d\d\d\d)-(\d\d)-(\d\d)]
    [$1.to_i, $2.to_i, $3.to_i]
  when %r[(\d\d\d\d)/(\d\d)/(\d\d)]
    [$1.to_i, $2.to_i, $3.to_i]
  when %r[(\d\d)/(\d\d)/(\d\d\d\d)]
    [$3.to_i, $2.to_i, $1.to_i]
  end
end

The most straightforward solution works exactly as it did in Ruby with no changes.

Solution 2

#!/usr/bin/env crystal

def parse_date(s)
  case s
  when %r[(\d\d\d\d)-(\d\d)-(\d\d)], %r[(\d\d\d\d)/(\d\d)/(\d\d)]
    [$1.to_i, $2.to_i, $3.to_i]
  when %r[(\d\d)/(\d\d)/(\d\d\d\d)]
    [$3.to_i, $2.to_i, $1.to_i]
  end
end

Grouping when options works just like in Ruby.

Solution 3

def parse_date(s)
  case s
  when %r[(\d\d\d\d)([/-])(\d\d)\2(\d\d)]
    [$1.to_i, $3.to_i, $4.to_i]
  when %r[(\d\d)/(\d\d)/(\d\d\d\d)]
    [$3.to_i, $2.to_i, $1.to_i]
  end
end

Back-references also work just like in Ruby.

Solution 4

Now this does not work:

def parse_date(s)
  case s
  when %r[(\d\d\d\d)-(\d\d)-(\d\d)|(\d\d\d\d)/(\d\d)/(\d\d)]
    [($1 || $4).to_i, ($2 || $5).to_i, ($3 || $6).to_i]
  when %r[(\d\d)/(\d\d)/(\d\d\d\d)]
    [$3.to_i, $2.to_i, $1.to_i]
  end
end

The reason is that in Ruby $1 can be either a String or a nil. In Crystal $1 is a String, so if it didn't match, it's an error to access it.

Crystal also has nilable equivalents $1?, $2? etc. Notice that to make the whole expression not nilable, we don't put ? on the last one:

def parse_date(s)
  case s
  when %r[(\d\d\d\d)-(\d\d)-(\d\d)|(\d\d\d\d)/(\d\d)/(\d\d)]
    [($1? || $4).to_i, ($2? || $5).to_i, ($3? || $6).to_i]
  when %r[(\d\d)/(\d\d)/(\d\d\d\d)]
    [$3.to_i, $2.to_i, $1.to_i]
  end
end

Solution 5

def parse_date(s)
  case s
  when %r[(\d\d\d\d)-(\d\d)-(\d\d)|(\d\d\d\d)/(\d\d)/(\d\d)|(\d\d)/(\d\d)/(\d\d\d\d)]
    [($1? || $4? || $9).to_i, ($2? || $5? || $8).to_i, ($3? || $6? || $7).to_i]
  end
end

Knowing what we know, we can use the same trick, rewriting ($1 || $4 || $9) into ($1? || $4? || $9) and so on.

Solution 6

def parse_date(s)
  case s
  when
    %r[(?<year>\d\d\d\d)-(?<month>\d\d)-(?<day>\d\d)],
    %r[(?<year>\d\d\d\d)/(?<month>\d\d)/(?<day>\d\d)],
    %r[(?<day>\d\d)/(?<month>\d\d)/(?<year>\d\d\d\d)]
    [$~["year"].to_i, $~["month"].to_i, $~["day"].to_i]
  end
end

Using named captures works identically to the Ruby version.

Solution 7

def parse_date(s)
  case s
  when %r[(?<year>\d\d\d\d)-(?<month>\d\d)-(?<day>\d\d)|(?<year>\d\d\d\d)/(?<month>\d\d)/(?<day>\d\d)|(?<day>\d\d)/(?<month>\d\d)/(?<year>\d\d\d\d)]
    [$~["year"].to_i, $~["month"].to_i, $~["day"].to_i]
  end
end

Having capture groups with the same name works just like in Ruby without changes.

Solution 8

def parse_date(s)
  case s
  when %r[
      (?<year>\d\d\d\d)-(?<month>\d\d)-(?<day>\d\d) |
      (?<year>\d\d\d\d)/(?<month>\d\d)/(?<day>\d\d) |
      (?<day>\d\d)/(?<month>\d\d)/(?<year>\d\d\d\d)
    ]x
    [$~["year"].to_i, $~["month"].to_i, $~["day"].to_i]
  end
end

And so does //x flag - everything just works.

Solution 9

def parse_date(s)
  if %r[
      (?<year>\d\d\d\d)-(?<month>\d\d)-(?<day>\d\d) |
      (?<year>\d\d\d\d)/(?<month>\d\d)/(?<day>\d\d) |
      (?<day>\d\d)/(?<month>\d\d)/(?<year>\d\d\d\d)
    ]x =~ s
    [year.to_i, month.to_i, day.to_i]
  end
end

This on the other hand is completely unsupported - the only side effect of regular expression match is overriding $~ variable ($1 is just alias for $[1]? etc.). Regular expression match can't override other local variables.

I'm not really comfortable with this Ruby feature, so it's not surprising it didn't find its way here.

Story so far

Everything just worked with no or minimal changes. This is my usual experience with Crystal. Things just work most of the time.

All the code is on GitHub.

Coming next

In the next episode we'll see how other languages handle this problem.

DEV Community: Tomasz Wegrzanowski

Break XOR Cipher with Z3

The Challenge

Random encryption

How to break Caesar Cipher with Z3

Solver

Is it practical?

Code

Break Caesar Cipher with Z3

The Challenge

Random encryption

How to break Caesar Cipher old fashioned way

How to break Caesar Cipher with Z3

Solver

Is it practical?

Code

Getting Started with Web Speech Synthesis API and Svelte

Getting Started with the API

Display list of available voices

Say something

Code

Getting Started with ClickHouse

Run ClickHouse with Docker

Let's run some queries

FizzBuzz

Connect from Python

Python Clickhouse FizzBuzz

What's next

Code

PyScript: Second Impressions

Installation

Hello, World!

FizzBuzz

Prompt Input

Interacting with DOM

Should you use PyScript?

Code

Upgrading old Imba 1 apps to run on modern Node

Dependencies

webpack.config.js

Code

How to access Kaggle data from command line

Get Kaggle API token

Install Kaggle CLI tools

Download dataset

Full process

Reading from Zip archives in Ruby

Old script

Gem rubyzip vs zip

Abstract archive handling

Read file from either directory or archive

Full code

Open Source Adventures: Episode 82: Series Retrospective

Open Source Adventures: Episode 81: Exploring Raku Regular Expression API

The Problem

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5

Solution 6

Story so far

Coming next

Open Source Adventures: Episode 80: Exploring Python Regular Expression API

The Problem

Solution 1

Solution 2

Solution 3

Solution 4

Solution 5:

Solution 6:

Solution 7

Solution 8

Solution 9

Solution 10

Story so far

Coming next

Open Source Adventures: Episode 79: Exploring Crystal Regular Expression API

Test case

Solution 1

`webpack.config.js`

Gem `rubyzip` vs `zip`