DEV Community: Lucas Quaresma

How to deal with regex? An extremely concise and definitive AI workflow.

Lucas Quaresma — Wed, 09 Apr 2025 16:19:35 +0000

Motivation

Some time ago I had to deal with some complex regex patterns to process huge amounts of data via Apache Spark (using the pyspark API). In this process I've found a pretty simple, and yet very effective way to deal with the construction and understanding of regex patterns.

I'm not a regex expert, and I don't intend to become one. Deal with regex is not something pleasant to me, neither something would like to be doing. However, sometimes it's necessary and this article is intended to show my workflow, partially powered by AI models.

If you're in a similar situation, this article is for you. If you're a seasoned developer who knows regex pretty well and can immediately understand upon reading, this article is not for you.

On LLMs and correctness

LLMs are purely probabilistic models, having NO GUARANTEE on the pseudo information they can provide. A helpful way to understand this behavior is to think of them as some type of extremely efficient and lossy compression. To some degree is possible to trust that the information they yield is ruffly in the ballpark of the correctness, but you should never assume it actually is! Of course, this is true provided that such model been trained on this kind o information.

This shortcoming can be much improved upon when using techniques that explicitly include actual relevant information on the context window of the model alongside your prompt. This can be achieved with RAGs, for instance, but this will not be discussed here.

Workflow

The process basically consists in two steps:

Using IA (LLMs) for getting a some understanding of the patterns.
Using some deterministic tool to assert that the patterns you deal with actually do what the IA assumes/intends them to do.

Understanding some regex pattern that already exists

In this case we use the following example: "[^0-9a-zA-Z_\-]+"

Let's assume you have stumbled in this pattern on some code and wants to understand what it does. First you can use some IA model to get an overview explanation.

We proceed with local LLMs, using DeepSeek R1 (14B parameters), (excluding thinking) in this article, but you can compare the results with Mistral Nemo (12B parameters) on the this link.

Model results in a chat:

Prompt: What the regex [^0-9a-zA-Z_\-]+ does?

The regular expression [^0-9a-zA-Z_\-] is designed to match any single character that is NOT one of the following:

Digits: 0 through 9

Letters:

Lowercase: a through z

Uppercase: A through Z

Commonly used in variable names.

Included as a literal character.

This regex is useful for identifying characters that are outside the allowed set of digits, letters, underscores, and hyphens. For instance, it would match special characters like @, #, $, etc., but not any digit, letter, underscore, or hyphen.

Example Usage:

In a string "user@name123", the '@' symbol would be matched by this regex.

In "Hello_World-123", none of the characters would be matched because they all fall within the allowed set.

Using the results above, its possible to grasp a pretty good comprehension of how the regex pattern works and how it could be used. However, we arrive at one of biggest problem with LLMs: they DO NOT yield deterministic information.

As cited in the session On LLMs and correctness, it's not possible to have any trust what so ever in this information. Hence, the need to verify each and every piece of the result. This can be done in several manners, such as testing locally with examples etc, but here we proceed using a static analysis with regex101, a free online tool tailored for this purpose.

Putting the pattern in the tool, and using the python standard alongside with option /gm (global and multi line) we see:

With all of this, we can confirm that the regex actually does what the model explained it does, and we can proceed to alter it or manage it however necessary.

Writing some new regex pattern to perform some function

Using the same kind of process, we can use the same IA models to generate regex patterns that perform some kind of function. For instance, let's say we need a pattern that recognizes white-spaces at the end of lines to be removed.

Model results in a chat:

Prompt: Make a regex pattern that matches any white-spaces at the end of any line in the text.

To create a regex pattern that matches any whitespace characters at the end of any line, you can use:

Pattern:
\s+$

This pattern works as follows:

\s+ : Matches one or more whitespace characters (spaces, tabs, newlines, etc.).

$ : Asserts the position at the end of a line.

Options:

You may want to add the m (multiline) option so that the $ anchor matches the end of each line rather than just the end of the string.

Example in regex with multiline mode enabled:
\s+$
This will match any line that consists entirely of whitespace characters.

Now we can verify the actual function of the pattern using regex101:

Now, at first glance, the pattern seems to match everything we need, but we can be sure testing it. In doing so, we'll use the following multi line string:

test with no trailing spaces

test with one trailing space 
test with tow trailing spaces  

   test with leading and trailing spaces     
     test with some leading spaces
   test with one leading space
   test with leading and trailing spaces

And now we replace the pattern matches with an emoji for easy visualization using regex101:

As can be seen above, the pattern the model suggested to us has an unfortunate side-effect: it removes blank lines from the input string. Since in the present hypothetical case this is not intended to happen, we proceed improving it with the model.

Continuation of the model results in a chat:

Prompt: The pattern is removing blank lines, how can i make it match only the characters before the end of the line?

To modify the regex to only match whitespaces at the end of a line when there are non-whitespace characters before them (ignoring blank lines), you can use:

Pattern:
^\S+\s+$

This pattern matches:

^: Start of the line

\S+: One or more non-whitespace characters

\s+: One or more whitespace characters at the end of the line

$: End of the line

Explanation:

This regex ensures that there is at least one non-whitespace character before any trailing whitespace.

Blank lines (which only contain whitespace) will not be matched because they lack the \S+ part.

Example:
In a text like:
Hello World    <-- Matched
   (contains only spaces)
Another line    <-- Matched
Another line    <-- Not matched (blank line)
The regex ^\S+\s+$ will match lines 1 and 3 but not the blank line.

Now, checking the functionality and testing the new pattern:

And indeed we can assert that the regex does exactly what is intended to do and that such behavior is guaranteed via tests.

Conclusion

Even as a firm pessimist on the majority of the use cases for IA and LLMs, still is not possible for me to ignore some of the actual good use cases for it. I believe that this post showcases some responsible and productive uses of such technology.

AI is here to stay, not as the solve-all-problems hype that it is today, but with some actual good usage cases. I hope this post has given you, the reader, some insight and maybe inspiration to find more of them.

P.S. Tests are your friends, take care of them, treat them well, and they will do the same for you. And always remember, that the Crowdstrike Incident was in part due to regex errors according to themselfs. Be careful!

How to start using nix?

Lucas Quaresma — Fri, 08 Nov 2024 21:49:44 +0000

If you're a Linux user, it's possible you could have heard some fuss about Nix.

But what is it? How to use it? Where could you start?

This post intends to present some introductory answers to ease the burden of your path coming into the best side of the force.

What is nix?

Nix is tree things, all at once, all at the same time. They are:

A programing language (therefore referred as nixlang).
A package and dependency manager (that includes other nice tools).
An operational system, in the form of Linux distribution, called NixOS.

The relationship among these parts, on a pretty high level, can be explained by:

1 is used as a means to work with 2 and 3. For instance, performing shell commands on 2, and writing .nix config files on 3.
2 is used in 3 as the manager for all packages and dependencies.

Declarative builds

The most interesting feature of all the nix ecosystem is being completely based on declarative builds and configurations.

We'll not go any further in details here, but think of packages that never change, builds that are completely stable, and reproducibility everywhere you go. No matter what happens, if your config worked somewhere once, it will always work anywhere else.

Where to start?

After this brief introduction you might be wondering where to start. If you should try something like NixOS on a virtual machine, or fully dive into it and substitute the OS on your work machine. The answer is for sure: no.

I used Linux myself for about ten years, even though, when switched to NixOS with no experience on Nix (the package manager) or nixlang, it was a big headache. The logic of nix and declarative systems is not similar to other Linux distributions. Therefore, directly switching distros to NixOS with no previous experience on the ecosystem would most probably result on giving up before any meaningfull benefits. However, if you're really stubborn like me, and have a bit of brio, it's possible. Once again, I do not recommend it.

Brio illustrative reference (Brazilian meme of the highest quality):

Other things to avoid are all the experimental features, like flakes and nix develop, or non-declarative alternatives, like nix-env. Do not use experimental features!!! But if you really want to learn more, and in spite of my advice if you do use them, for the love of god don't complain about it for the community, were not responsible for you skill issues.

What you should start learning is the the basics, just the basics. First, take a look at the docs for the nixlang, it's not great, but they will give you an overall idea of how things work and some capabilities to work with .nix files when needed.

Second, you should be able to understand how to use nix-shell and it's features. In my opinion it's one of the simplest, yet most powerful things in the whole ecosystem. With it you can create purely declarative, completely isolated and ephemeral shells, that can be used anywhere, on any Linux distro, and give you real power. Along with it, it's possible to integrate them with bash, use them stand alone with simple terminal commands, or based on <shell_name>.nix config files.

Third, you should be able to understand and configure nix package channels, understand how generations and unstable branches work and how to mix and match them to your liking. I recommend start doing this inside nix-shell and any other nix environments. Use some packages from stable, others from unstable, look for things that are available in one of them and not the others etc.

Forth, go easy! What you should really be looking forward is to try nix on the small things. In that regard, it's much like bash, it will not solve most of your problems, but will for sure make your life hell of a lot easier. Look for small things like test environments, quick setups, the disposable spaces where you want to use some new thing and be sure it will not hurt your system configs etc.

When will you be ready to use NixOS?

This really depends. For me, changing distros was kind of a necessity since I was in distro hopping hell due to Nvidia/TensorFlow fucking ecosystem for CUDA and machine learning. It saved me, and thank god I have no longer this kind of problem.

However, sometimes on NixOS you could have other problems. You are not me, and your problems and tradeoffs can be different than mine. If you already know the basics, and think the tradeoffs could be worth, go for it.

One thing I can say for certain is, if you have a single stable system, you'll always have at least that version working everywhere. That is more peace of mind that you could ever find elsewhere.

Concluding remarks

For me, using the whole nix ecosystem has been a wonderful experience. It solved so many problems I had and didn't think there could be a solution. It also helped me to make elaborate things work in the most simple and elegant way. If this kind of thing is what you're looking for, nix could be for you.

Lastly, good luck! I hope this post can give at least some idea or inspiration on where to start.