DEV Community: Vincent Ge

How to detect and fix flaky tests in Pytest

Vincent Ge — Tue, 11 Mar 2025 17:57:32 +0000

What are flaky tests?

Flaky tests produce different results each time you run them. This means you can rerun the same CI job on the same commit 10 times and see inconsistent results. These flaky tests are the tests that fail and block your PRs when you fix a typo or bump a dependency, even though your changes couldn’t have possibly broken a test. They’re the tests that make you habitually hit the rerun job button when tests fail.

Why do flaky tests occur?

Flaky tests are most common in integration tests and e2e (end-to-end) tests. These tests tend to be complex and can be flaky for a multitude of reasons.

Flaky test code (bad)
Flaky application code (really bad)
Flaky infrastructure (pain 🙃)

This flakiness is inevitable to some degree. You write complex e2e and integration tests to test your complex system. These more complex tests give you a higher degree of confidence about your complex system. This is why the best teams in the world write e2e tests. It’s a conscious tradeoff of complexity, flakiness, and confidence where you write more complex and potentially more flaky tests for better confidence that tests validate the complete system.

Test Type	Complexity	Confidence	Flakiness
Unit tests	Low	Low	High
Integration tests	Medium	Medium	Medium
End-to-End tests	High	High	Low

We can break down the causes of flaky tests into a few common categories. Let me give you some examples.

Machine learning and data science

Python has a special and common use case for training and deploying machine learning models. The problem with applications of this nature is that they're probabilistic in nature. If you're testing training code for a model or regression on a data set, the convergence of a model given fixed parameters is non-deterministic unless you're willing to let the model train for an impractical number of epochs.

What this means is that flakiness in machine learning and some data science applications is almost inevitable. The best you can do in these cases is to tweak the assertion criteria so the tests are less flaky. We'll also discuss how you can do this in a later section.

Concurrency and Test Order Dependencies

When tests are run concurrently and in random order, tests that are otherwise not flaky can become flaky due to isolation problems.Consider the following scenario where test A updates a row in the table and test B deletes the row. When executed alone, these tests may produce deterministic results. When executed concurrently, they can become flaky if test B deletes the row updated by test A before test A’s assertion completes.

Consider another scenario when test A and test B are executed in different order, they could affect each other’s results. If test B deletes the row updated by test A before test A runs, the test could fail. The timing in which these tests are executed affects their results.

External resources

If you’re to be truly confident in the results of your tests, you need tests that cover parts of your entire system. This sometimes means external resources that aren’t set up and taken down with your test run.

This commonly means running tests against a dev or staging environment, against external databases and APIs, or interacting with a third-party service. These systems’ states are inconsistent from run to run. Whether it’s a QA engineer who unknowingly updates a row that your e2e tests depend on or it’s an external API timing out because of slow network performance, these resources are unreliable. Unreliable external resources create flaky tests.

Lack of isolation

Your tests should ideally be isolated. This means they’re independent of each other and they’re independent from run to run. (Good) Tests read, create, update, and delete artifacts in the form of database entries, files uploaded, users created, and background job scheduled. This is great for write tests that are realistic and inspire confidence, but these artifacts need to be properly cleaned up so they don’t impact the results of future runs.

To have truly independent tests, you need to be running them on a fresh setup each time. But you can’t reasonably create an entire VM or clusters of VMs to run each test, which is how you’d achieve perfect isolation test to test. You sometimes can’t even run each CI job against a fresh setup. If you tried, you’d either be paying buckets for CI resources or you’re waiting for hours to await resources on a busy day.

The alternative is to have setup and tear down between tests, test suites, or CI jobs. The problem here, is that they’re difficult to get perfect. You might forget some teardown logic here and there, some bits of orphaned code might sprinkle random artifacts, and jobs that don’t run to completion due to crashes or cancellations will also leave behind a mess. The build-up of these unexpected artifacts will cause flakiness.

Inconsistent runtimes

Each CI machine will perform differently. This could be due to differences in CPU performance, disk performance, RAM available, and network load. This could cause some seriously annoying to debug problems, usually involving out-of-memory issues or tests timing out because a particular CI machine is slower.

These are extra annoying to fix because they’re impossible to reproduce locally. After all, your development machine is probably several times more powerful than any CI machine. These problems are also more common if your tests involve physical devices, such as phones and watches for native development or GPUs for AI and simulations.

Buggy production code

Sometimes, the root of flakiness isn’t in your test code but in your production code. This means you have actual bugs to deal with. You might have queries or APIs that occasionally fail in production. You might have discovered unhandled edge cases. You may have stumbled upon problems with your infrastructure if you test on a development or staging environment.

This is probably the most concerning type of flaky test, but also the hardest to fix. You want to fix these problems since this inconsistent behavior could affect paying customers. Fixing these problems is often difficult because they might result from multiple moving parts in a larger system. We'll discuss how to best approach these difficult fixes in a later section.

Why should I be concerned?

There’s good reason to be concerned about flaky tests. Flaky tests are rarely the result of bad test code alone. They could indicate flakiness in your infrastructure and production code. For many teams, this is the single biggest reason for investigating and fixing every flaky test.

You might also notice the impact of flaky tests on the reliability of your CI. You rely on CI as guardrails so you can ship code quickly without worrying about introducing regressions or breaking existing features. Flaky tests can both ruin your trust in CI results and seriously slow down your team. If your CI jobs fail 20% of the time, you have to choose between rerunning your tests or ignoring flaky failures which forces you to choose between slowing your team down or poisoning your test culture.

"Flaky tests are poison. You can have the best tests in the world, the most sophisticated testing that finds the craziest bugs no one would think of, but if developers can’t trust the results, they can’t trust the results 99% of the time, it doesn’t matter."
Roy Williams, Meta

Dealing with Flaky Tests

Flaky tests are inevitable if you write e2e tests at scale. Your software is complex, so you need some complex tests to provide confidence and validate your changes. If your tests are complex, you will have some flakiness in your tests. It’s a matter of economics; you can make your tests complex and reliable, but that will require many engineering hours that you may be unable to justify. Instead, you should focus on reducing the impact of flaky tests and tackle high-impact flaky tests efficiently.

To effectively reduce the impact of flaky tests, you should do the following:

Avoid flaky tests by learning common anti-patterns
Detect flaky tests with automated tools and communicate them with your team
Quarantine known flaky tests to mitigate it’s impact on the team
Fix high impact flaky tests for the bang for your buck

Let’s walk through each of these steps in more detail.

Avoiding Flaky Tests in Pytest

Avoid overly strict assertions

One of the weird quirks of Python floating point arithmetic is that numbers that logically should be equal, are sometimes not. Take the following example:

>>> 0.1 + 0.2 == 0.3
False

>>> 0.2 + 0.2 == 0.4
True

This is due to the way floating point numbers are represented in memory, certain numbers like 0.3 are stored as a number very close to the original value (0.30000000000000004), but not the exact same. This problem also exists in other languages like JavaScript and C++, but is much more likely to cause problems in common applications of Python like machine learning and data science.

Pytest provides a great way to combat floating point errors with the approx() method. You can use it on floating point assertions and even numpy arrays:

from pytest import approx
>>> 0.1 + 0.2 == approx(0.3)
True

>>> np.array([0.1, 0.2]) + np.array([0.2, 0.4]) == approx(np.array([0.3, 0.6])) 
True

You can also assign relative and absolute tolerances for these approximations, which you can read more about in the
Pytest docs on approx().

Another reason to avoid overly strict assertions is for machine learning applications. The convergence of some algorithms, even over a consistent data set, is non-deterministic. You will have to tweak these assertions to be a good approximation of "good enough" to avoid excessive flakiness.

Avoid hard awaits

Asynchronous programming in Python is complicated due to the limitations of the Python interpreter, but it hasn’t stopped Python developers from trying. The problem with the many libraries that do introduce multi-threading, multi-processing, and event-loop based asyncio is that they’re more awkward to test in Python. We’re gonna take asyncio as an example here, but the same principles apply to other implementations of async or multi-threaded programming in Python.

In Pytest, if you try to test an async method using asyncio naively, the test will fail because Pytest doesn’t support async operations.

from example import async_method

async def test():
    res = await async_method(0)
    ## This will return false because 
    assert res == 42

The best solution is to use a library like pytest-asyncio to properly support async operations:

@pytest.mark.asyncio
async def test():
   res = await method(0)
   assert res == 42

Properly awaiting for async operations will usually result in a lower chance of flakiness in Python.

What often causes flakiness when dealing with async operations is naively waiting for some time. This approach is intuitive but will be flaky because async operations don’t return in a deterministic amount of time. You can try stretching your sleep time longer and longer, which might become more reliable, but your test suites will take ages to run.

Naively waiting will almost always cause some degree of flakiness, so if you see code snippets like this in your test suite, fix them immediately:

from example import async_method
import time

async def test():
    res = await async_method(0)
    time.sleep(2)
    ## This will be flaky
    assert res === 42

You should also avoid mocking the async code for testing purposes. Mocking async introduces unnecessary complexity and reduces your confidence level in your test by making it less realistic.

Control your test environment

If your test environment changes from run to run, your tests are much more likely to be flaky. Your Pytest e2e and integration tests will likely depend on your entire system's state to produce consistent results. If you test on a persistent test environment or a shared dev/staging environment, that environment might differ each time the test is run.

This is particularly problematic if you're using a library like pytest-xdist to distribute your tests across multiple CPUs. Testing in parallel and in random order introduces opportunities for both inconsistent system states to cause flakiness and race conditions where concurrently running tests interfere with each other.

If you're using a persistent testing environment, you must ensure that artifacts created by your tests, such as new database tables and rows, files created in storage, and new background jobs started, are properly cleaned up. Leftover data from run to run might affect the results of future test runs.

Similarly, you should be careful when testing a development or staging environment. These environments might have other developers who create and destroy data as part of their development process. This constantly changing environment can cause flakiness if other developers accidentally update or delete tables and files used during testing, create resources with unique IDs that collide with a test, or use up the environment's compute resources causing the test to timeout.

The easiest way to avoid these potential problems is to use a fresh environment whenever possible when you run your tests. A transient environment set up and destroyed at the end of each test run will help ensure that each run's environment is consistent and no artifacts are left behind. If that's impossible, pay close attention to how you design your tests' setup and takedown to remove all dependencies. Each test should set up and take down its own resources.

Limit External Resources

While it's important to test all parts of your app fully, it's important to limit the number of tests that involve external resources, like external APIs and third-party integrations. These external resources may go down, they might change, and you may hit rate limits. They're all potential sources of flaky failures. You should aim to cover each external resource in a few tests, but avoid involving them in your tests excessively.

Mocking is a decent strategy for testing specific behaviors, but it should not be used as a replacement for end-to-end testing. You can't entirely avoid external resources, but you should be mindful of which test suites involve these resources. Have some dedicated tests that involve external resources and mock them for other tests.

Automatic Reruns

If you only have a few flaky tests, rerunning tests on failure to clear flakiness can be effective. Libraries like pytest-rerunfailures let you configure reruns easily, which reruns each test on the first failure to see if it's just flaky.

This approach does have its own tradeoffs. It will cost more CI resources and CI time, which depending on the size of your test suite can be a negligible or profound problem. If you have hundreds of thousands of tests, and a few thousand of them are flaky, this will be unbelievably expensive. You will also still catch flaky tests slipping through the cracks occasionally by pure chance.

A much bigger problem is reruns just cover up the problem. When you introduce new flaky tests, you may not even notice. This is bad because some flaky tests indicate real problems with your code and the silent build-up of flaky tests can cause scaling problems within your CI system.

Detecting Flaky Tests in Pytest

If you only have a few flaky tests, a great place to start is by having a central place to report them, such as a simple spreadsheet. To identify flaky tests, you can use commit SHAs and test retries. A commit SHA is a unique identifier for a specific commit in your version control system, like Git. If a test passes for one commit SHA but fails for the same SHA later, it's likely flaky since nothing has changed.

Dropbox open-sourced a solution that helps you find flaky tests called pytest-flakefinder. This allows you to automatically rerun each test multiple times to find flaky tests. By default, the tool will run each test 50 times on the same commit and validate if the results are consistent. On the same commit, the code should be identical from run to run. If a test is not producing consistent results, it is considered flaky.

Running tests multiple times to find flaky tests is a great way to audit your existing repo or new suites of tests. The problem with using a tool like this is that it relies on a "periodic cleanup" type workflow, instead of monitoring continuously. Relying on finding time to periodically clean up tech debt rarely works. Other higher-priority tasks, typically anything but testing, will displace this workflow and you'll end up with a pileup of flaky tests again.

Continuously monitoring flaky tests also gives the benefit of catching flaky tests due to changes in external resources, tool rot, and other sources of flakiness that happen due to drift and not code change. These problems are often easier to fix when caught early. Continuous monitoring also tells you if your fixes actually worked or not, since most flaky tests aren't fixed completely on the first attempt but will see reduced flakiness.

Some form of continuous monitoring is especially important for Python applications that test ML or data science-related code. As we mentioned before, many ML and data science applications are inherently non-deterministic, and the best fix for these flaky tests is to just tweak the assertion parameters so they're acceptably flaky.

Collecting test results in CI and comparing results on the same commit is a great starting point for detecting flaky tests continuously and automatically. There are some nuances to consider for the different branches on which the tests are run. You should assume that the test should not normally fail on the main branch (or any other stable branch) but can be expected to fail more frequently on your PR and feature branches. How much you factor different branches' flakiness signals will depend on your team's circumstances.

If you're looking for a tool for this, Trunk Flaky Tests automatically detects and tracks flaky tests for you. Trunk also aggregates and displays the detected flaky tests, their past results, relevant stack traces, and flaky failure summaries on a single dashboard.

For example, you'll get a GitHub comment on each PR, calling out if a flaky test caused the CI jobs to fail.

Quarantining Flaky Tests in Pytest

What do you do with flaky tests after you detect them? In an ideal world, you'd fix them immediately. In reality, they're usually sent to the back of your backlog. You have project deadlines to meet and products to deliver, all of which deliver more business value than fixing a flaky test. What's most likely is that you'll always have some known but unfixed flaky tests in your repo, so the goal is to reduce their impact before you can fix them.

We've written in a past blog that flaky tests are harmful for most teams because they block PRs and reduce trust in tests. So once you know a test to be flaky, it's important to stop it from producing noise in CI and blocking PRs. We recommend you do this by quarantining these tests.

Quarantining is the process of continuing to run a flaky test in CI without allowing failures to block PR merge. We recommend this method over disabling or deleting tests because disabled tests are usually forgotten and swept under the rug. We want our tests to produce less noise, not 0 noise.

It's important to note that studies have shown that initial attempted fixes for flaky tests usually don't succeed. You need to have a historical record to know if the fix reduced the flake rate or completely fixed the flaky test. This is another reason why we believe you should keep running tests quarantined.

To quarantine tests in Pytest, you can mark them as quarantined and run them in a separate CI job that doesn't block your PRs.

@pytest.mark.quarantined
def flaky_test():
    # Generates a random float between 0 and 1
    random_value = random.random()
    assert random_value <= 0.6

You can then run them with pytest -v -m "quarantined" and exclude them with pytest -v -m "not quarantined".

Some limitations here:

This requires a code change to quarantine instead of happening dynamically at runtime.
Doesn't integrate with automated systems to detect flaky tests and quarantine them.
They produce very little noise and are easily forgotten.

A better way to approach this is to quarantine at runtime, which means to quarantine failures without updating the code. As tests are labeled flaky or return to a healthy status, they should be quarantined and unquarantined automatically. This is especially important for large repos with many contributors. A possible approach to accomplish this is to host a record of known flaky tests, run all tests, and then check if all failures are due to known flaky tests. If all tests are from known flaky tests, override the exit code of the CI job to pass.

Again, this is really useful for ML and data science applications, where models become more or less flaky when updated. You might just want to quarantine certain tests that are flaky because the models don't converge in a short enough amount of time. Since they still run, you can track their failure rate to make sure they're not degrading and inspect the failure logs to catch actual errors.

If you're looking for a way to quarantine flaky tests at runtime, Trunk can help you here. Trunk will check if failed tests are known to be flaky and unblock your PRs if all failures can be quarantined.

Learn more about Trunk's Quarantining.

Fixing Flaky Tests

We've covered some common anti-patterns earlier to help you avoid flaky tests, but if your flaky test is due to a more complex cause, how you approach a fix will vary heavily. We can't show you how to fix every way your tests flake; it can be very complex. Instead, let's cover prioritizing which tests to fix and reproducing flaky tests.

When deciding on which tests to fix first, we first need a way to rank them by their impact. What we discovered to be a good measure of impact when we worked with our partners is that flaky tests blocking the most PRs should be fixed first. Ultimately, we want to eliminate flaky tests because they block PRs from being merged when they fail in CI. You can do this by tracking the number of times a known flaky test fails in CI on PR branches, either manually for smaller projects or automatically with a tool.

This also helps you justify the engineering effort put towards fixing flaky tests, because with tech-debt, justifying for time invested is often a bigger blocker than the fix itself. When you reduce the number of blocked PRs, you save expensive engineering hours. You can further extrapolate the number of engineering hours saved by factoring in context-switching costs, which some studies show to be ~23 minutes per context switch for knowledge workers.

This approach is especially true for flakiness in ML and data science apps, where a complete fix is impractical. For these applications, a fix usually just finds the high-impact flaky tests and reduces how strict the assertions are.

After finding the high-impact tests to fix first, you can use a library like pytest-replay to help reproduce tests locally. The difficult-to-reproduce tests often reveal real problems with your infrastructure or code. Pytest-replay lets you replay failures from replay records saved in CI, which can be a real-time save.

If you're looking for a straightforward way to report flaky tests for any language or framework, see their impact on your team, find the highest impact tests to fix, and track past failures and stack traces, you can try Trunk Flaky Tests.

Learn more about Trunk Flaky Tests dashboards.

Need help?

Flaky tests take a combination of well-written tests, taking advantage of your test framework's capabilities, and good tooling to eliminate. If you write e2e tests with Python and Pytest and face flaky tests, know that you're not alone. You don't need to invent your own tools. Trunk can give you the tools needed to tackle flaky tests:

Autodetect the flaky tests in your build system
See them in a dashboard across all your repos
Quarantine tests manually or automatically
Get detailed stats to target the root cause of the problem
Get reports weekly, nightly, or instantly sent right to email and Slack
Intelligently file tickets to the right engineer

Try Trunk

What We Learned From Analyzing 20.2 Million CI Jobs In Trunk Flaky Tests - Part 2

Vincent Ge — Wed, 27 Nov 2024 15:01:17 +0000

This is part two of a blog series about building Trunk Flaky Tests. Part one covered what we’ve learned about the problem space surrounding flaky tests, like how they impact teams, why they’re prevalent, and why they’re so hard to fix. This part focuses on the technical challenges of building a product to detect, quarantine, and eliminate flaky tests.

If you’re curious about what we’ve built, you can find more in the Trunk Flaky Tests Public Beta Announcement blog.

Building a Solution

So far, we’ve only discussed the pain points we’ve discovered while working with our beta customers. To recap, flaky tests are prevalent if you write end-to-end tests, but teams underestimate how many of their tests flake and how much time is wasted dealing with Flaky Tests. Fixing flaky tests can cost more engineering resources than ignoring them, but letting flaky tests accumulate will become debilitating. Finding practical ways to deal with flaky tests is the challenge at hand. Let’s talk about some of the technical challenges we’ve faced in building a solution to this problem.

The Scale of Flaky Tests

In Uber’s Go monorepo, approximately 1,000 out of 600,000 tests were consistently detected as flaky. Dropbox reported running over 35,000 builds and millions of automated tests daily, with flaky tests causing significant operational overhead. At Google, out of 4.2 million tests run on their CI system, about 63,000 (1.5%) had a flaky run over a week. More strikingly, approximately 16% of all their tests exhibit flakiness. Over our private beta, we’ve also identified 2.4K tests that we labeled flaky.

The Challenges That Come With Scale

Production test suites can be huge. They’re run hundreds of times daily on hundreds of PRs, making it challenging to process all that test data to detect flaky tests. Over our private beta, we processed 20.2 million uploads. This means we received 20.2 million uploads of test results from different CI workflows. Each upload can contain the results of one or many test suites worth of results.

Even with a limited number of private beta customers, Flaky Tests processes 40 times more data than all of Trunk’s other product offerings combined. We see thousands of times more data for specific types of tables in our database. The volume of data is a challenge in its own right and difficult to query. Creating summary metrics for each test and overall summary metrics across thousands of runs of tens of thousands of tests is very time-consuming.

Single uploads could contain thousands of tests from a CI job. This data needs to be processed, transformed, and labeled to be displayed on various parts of the Flaky Test dashboard. For example, individual test runs need to have their failure reason summarized by an LLM to be displayed in the test details tab. They also need to be labeled with the correct status, such as if they’re healthy, broken, flaky, and if they’ve been rerun on the same PR.

The practical implication is that we can’t naively query and process the data at the time of request because it would take minutes, which is unacceptable. The metrics need to be precomputed in a timely manner. No one wants to wait more than a few minutes for a summary of a PR.

It’s also not trivial to precompute and cache metrics. Different metrics take a different amount of time to compute. If one metric updates faster than another, you’ll see conflicting numbers on different pieces of UI, which is unacceptable. It turns out that extracting useful metrics from a bunch of test runs and providing them quickly enough to be useful is already a challenge.

Displaying Tests Results

Another interesting challenge is displaying the thousands of tests, their results, their health status, whether they have an attached ticket/issue, and whether they’ve been quarantined. A lot of information needs to be displayed differently if you’re a team lead keeping track of overall test health or a contributor debugging flaky tests on a PR. After revisions with our beta partners, we landed on three key views for different workflows.

The first screen is the overview dashboard, which shows the overall health of your tests across many CI jobs. You’ll see how many tests are flaky and broken, and you’ll see time-series data on the trends in your repo. This view focuses on helping you understand the overall impact of flaky tests on your repo and finding the highest-impact tests to tackle next.

When you click on a single flaky test, it opens another screen that focuses on helping you debug an individual test. You’ll see a report of its results over the last seven days and the unique ways that the test has failed. You can see when the test started failing and on which branches. This helps you understand what might have caused the test to become flaky and how it fails when it’s flaky, and ultimately helps you debug the test.

On this same screen, you can click through the stack trace of different test runs with a similar failure reason to help debug flaky tests.

The third screen is a PR Test Summary, which shows an aggregated report for all test results from a single PR. This screen slices the data differently so engineers working on a single PR can quickly scan through failures and verify the results. This screen shows if a test failure is a known flaky test, a known broken test, if it’s quarantined, or if this PR introduces the failure. This helps engineers quickly identify the failures they need to address.

These screens don’t come from the requirements or feedback of a single customer, everyone has a different set of information that they intuitively believe to be important. Through the beta program, by pooling common feedback from different customers, we think these three different screens manage to condense the massive amount of information needed to understand flaky tests into a digestible format. It fits a variety of use cases and scales well even if you’ve got tens of thousands of tests.

Data Quality

There isn’t a single shared standard output format for test results. While most test runners produce JUnit XML, that is still a loose standard. You can't produce high-quality metrics if you don’t get high-quality data uploads.

When we ingested data from our customers, we saw random test names, automatic reruns reported as duplicates or overriding the original results, no test file paths, and all kinds of weird data forms. Handling and sanitizing all of this data is challenging. It felt like a game of constant whack-a-mole: We figure out a way to handle one type of data anomaly and out comes another.

Detecting the inconsistencies and problems in a customer’s uploaded data, prompting them with the proper follow-up steps, and sanitizing the rest is challenging and time-consuming. While we can justify the cost of handling new logic for a new language or test runner, we can’t imagine the pain of building a similar in-house solution, especially if you’ve got a diverse testing stack with legacy code.

Detection Rules

What is a flaky test? It’s a debatable and nuanced topic that begins to feel more like a philosophical question than an engineering one when scrutinized. A test can be non-deterministic in a variety of ways:

How frequently does it have to fail?
Is a timeout failure the same as a DB connection failed failure?
Should a failure on a PR treated the same as on the main branch?
Should the window for detecting a test’s health be N number of runs or a specific time window?
How do we adjust detection for repos that see 10 test runs vs 1000 test runs?
How do we determine if a test is healthy again after a fix?
What about edge cases of tests that fail more than they pass? What about tests that fail once every 1000 runs?
What if tests never run on main or another protected branch?

There are many signals to consider when detecting Flaky Tests; the answers here aren’t black and white. While detection can never be perfect and never needs to be perfect, it’s possible to hit a maximum for the vast majority of projects and fine-tune for the outliers. Even during a limited private beta, having access to many diverse technology stacks and millions of test runs gives Trunk Flaky Tests a unique advantage in refining our algorithm continuously.

The private beta customers have already helped us find a number of edge cases that have helped us detect flaky tests more accurately. Our partnership with partners during our private beta and our new public beta program will hopefully allow us to further improve this process.

Quarantining Tests

Quarantining lets you isolate failures for known flaky tests so they don't fail your CI jobs while continuing to run them. Quarantine looks for known flaky tests and determines which tests should be isolated at runtime so you can avoid code changes usually required to disable flaky tests. This is better than disabling tests because it doesn’t require commenting out tests, which will likely never be revisited. After all, disabled tests produce no noise. It also automatically allows for “un-quarantining” tests if we detect that it’s healthy after a fix. The bottom line is that you’re less likely to lose tests when quarantined.

Quarantining isn’t as challenging to implement as it is tedious. To implement quarantining, you first need a detection engine for flaky tests. Then, you need to serve that information through some API that a script can fetch in CI jobs to see if a test is known to be flaky. That script needs to run and collect the results of tests (recall the non-standard output formats), compare these results with the API response, and then override the exit code if all failures are quarantined. It must handle overrides for specific tests that should always be quarantined or never quarantined. And that script must work on whatever platform and environment you choose to run your CI jobs on. Finally, you must communicate what’s quarantined in each CI job and track quarantined test results. None of this is very sophisticated, but getting all the domino pieces to fall into place nicely is hard.

End-to-End UX

How do you manage flaky tests end-to-end? As discussed in Part 1 of this blog, we don’t see a consensus on a workflow online or among our beta partners. So, if a convention doesn’t exist, we must propose one ourselves. Here’s what we build with the help of our closed-beta customers.

What’s Next?

We don’t think flaky tests are a realistic or practical problem for any tooling team to tackle in-house. The problem is nuanced, and there are no conventions for dealing with it effectively. Our private beta customers allowed us to facilitate much-needed discourse around this problem. By having diverse teams and CI setups to test our solutions and pool feedback and ideas, we can help everyone reduce the cost of trial and error when dealing with flaky tests.

We’re opening Trunk Flaky Tests for public beta so you can help us build a more effective solution for all. The sooner we stop suffering in isolation, the sooner we’ll end flaky tests.

Join the Trunk Flaky Tests Public Beta

What We Learned From Analyzing 20.2 Million CI Jobs In Trunk Flaky Tests - Part 1

Vincent Ge — Mon, 18 Nov 2024 15:58:59 +0000

If you encounter problems with CI stability, it’s probably due to flaky tests. This has proven to be a recurring theme in our conversations with our CI Analytics and Merge Queue users. So, continuing with our goal to eliminate problems that engineers hate, we embarked on a journey to help developers eliminate flaky tests.

Flaky tests are ubiquitous in teams with automated integration and end-to-end tests, where developers suffer in silence. They’re the biggest challenge to CI stability and a developer pain we’re looking to solve. Even the best teams like Google, Meta, Spotify, and Uber waste countless developer hours due to flaky tests, and wasted engineering hours are expensive.

It’s easy to dismiss flaky tests as a “skill issue” and blame bad test code. But the deeper we dug into test flakiness, the more it looked like a dark, confusing rabbit hole. With our closed-beta partners, we analyzed test results uploaded from 20.2 million CI jobs. We learned that addressing flaky tests in practice requires nuance, you can't just fix all of them because it's impractical. You also can't ignore them because they poison your trust in your test results and slow you down.

While building Flaky Tests, we’ve learned and suffered, and we can’t wait to share our discoveries with you. Both technical and non-technical. If any of what you read resonates with you or you’re also down the rabbit hole with us looking for a better way to tackle flaky tests, we’d love your feedback in the Trunk Flaky Tests Public Beta.

What’s in Part 1?

This blog is divided into two parts. Part one (what you’re reading) focuses on the problem space. It discusses what we learned about flaky tests, how they impact teams, why they’re prevalent, and why they’re so hard to fix. Part two covers the challenges of building a tool for flaky tests.

You can read part 2 here.

You Will Always Have Some Flaky Tests

From our research, we observed a phenomenon in which organizations that start writing more tests and more realistic tests see a sharp increase in the flakiness of their tests.

Roy Williams from Meta described the problem as that with a growing number of tests, they hit a “knee” or “hockey stick growth” with flaky tests detected in their code base. Jeff Listfield at Google described in one blog that “over the course of a week, 0.5% of our small tests were flaky, 1.6% of our medium tests were flaky, and 14% of our large tests were flaky.” When you consider that Google has over 4.2 million tests, this is a huge number of flaky tests.

A significant contributor to this phenomenon is that your tests become more complex when you write more realistic tests, like integration tests and end-to-end tests. In an end-to-end test, there can be hundreds of moving parts. Your tests can be flaky for various reasons, like running out of RAM, a service container not spinning up in time, or misconfigured DNS.

Testing is about confidence, and you won't be confident if you don’t have any tests that span your whole system. Dylan Frankland, a lead engineer here at Trunk, puts it like this:

“You can try to make your tests completely un-flakable. But you will have very low confidence in your code. You can mock out every piece and never have a flake, but at the end of the day, then, you're testing mocks, not real code.”

With complexity, writing tests that never flake is practically inevitable.

Suffering in Silence

Software engineers often suffer silently. Tenaciousness, independence, and resourcefulness are often considered standout qualities of great software engineers. However, when tackling flaky tests, these qualities work against them by masking the problem.

When an engineer encounters a flaky test, they might debug it for 30 minutes, decide that this can’t be related to their change, and rerun the test. This sometimes fixes the problem. Since the problem is not critical, they sweep it under the rug, and no one else learns about it. This lets flaky tests accumulate until rerunning no longer clears failures.

If you’re like Uber, who in their Go Monorepo reports 1000+ flaky tests, you’ll rarely see PRs without flaky failures. Even if each test only has a 0.1% flake rate, the chances of having at least 1 flaky failure are 1 - 0.999^1000 or ~63%. That means 63% of PRs will need at least 1 rerun to clear the flaky failure and distract an engineer to debug the failures. If you let the innocent-looking flaky tests accumulate silently, when you finally notice it, it'll already be a monster that paralyzes your PR velocity.

We made a calculator to help you better understand how flaky tests could impact your team, and we recommend that you experiment with the numbers yourself.

We had this problem at Trunk, too. Federico, one of our TLMs, noted after adopting dogfooding flaky tests: "We knew we had flaky tests because we were experiencing failures, but the tool revealed the full extent of the issue." It wasn’t until he saw the numbers in the Flaky Tests dashboard that he realized just how many PRs are being blocked by flaky tests.

Our beta partners also shared this sentiment. They frequently underestimated the scale of their flaky test problems and were surprised by the data they saw.

What’s The Impact?

When we interviewed our closed beta partners on the value of eliminating flaky tests, the top response was the number of engineering hours saved. What we intuitively thought would matter the most was that the CI cost saved from not rerunning flaky tests didn’t even come close. We think flaky tests impact engineering velocity in three main ways: time lost from blocked PRs, time lost context switching, and time lost managing flaky tests.

Time lost from flaky tests blocking PRs is easy to understand. If your CI jobs aren’t passing, you can’t merge a PR. Most repositories have branch protections that prevent PRs from being merged unless all checks/tests are passing. If your tests are very flaky, trying to merge perfectly good code still feels like a dice roll. Combined with long-running tests, blocked PRs become very painful.

In the study Cost of Interrupted, research pointed to ~23 minutes of productivity lost per context switch. I wouldn't take this at face value, but if you observe your coworkers in an office, you might find this number close.

You might see a series of events like this:

10:00:00 - INFO - Bob submits a pull request (PR).
10:15:00 - ERROR - CI job fails on Bob's PR.
10:20:00 - INFO - Bob reruns the tests after identifying the failure is unrelated to his changes.
10:25:00 - INFO - Bob takes a break for coffee and begins a new task.
10:45:00 - INFO - Bob remembers to check the test results for his PR.
10:46:00 - SUCCESS - Bob merges his PR.
11:15:00 - ALERT - Bob is paged due to a flaky test failure in the main branch.

I'm exaggerating, but if your tests are flaky and your team is large, those little distractions add up fast.

Looping back, we also mentioned a third way flaky tests impact engineering velocity, which is time lost managing flaky tests. This is all the time lost from replying to email threads, Slack messages, GitHub, and comments about which tests are flaky. It’s time lost when creating tickets, maintaining a list of known flaky tests, and updating the list when the test is fixed. Team leads who usually have the most insight into a team’s test health and are responsible for ensuring code standards usually feel this the most. They’re the ones that get pinged, and they manage the tickets.

The Importance of Communication

Making it easier to triage flaky tests and communicate them with your team is essential. A lot of the time wasted on Flaky Tests is in the interruptions and duplicated work debugging flaky tests. Each time a flaky failure pops up on your CI jobs, an engineer context switches to debug the test. If the fix is non-trivial and a rerun clears the failure, they’ll give up and move on.

The problem is that they don’t tell anyone else about this, and the same debugging process is repeated for every engineer who comes across this flaky test. As mentioned before, each context switch also takes time to recover from. If your tests take 20 minutes to run, you’ll be distracted by a notification 20 minutes into the next task you picked up after submitting your PR. That’s right when you enter deep focus.

Offloading these spontaneous disruptions and turning them into planned work reduces context switching. Communicating who owns which flaky test and if the ticket is being worked on can prevent wasted time from duplicated efforts to debug the same flaky test.

Fixing Every Flaky Test Isn’t The Solution

Flaky tests can be caused by flaky test code, flaky infrastructure, or flaky production code. This last part is why many teams intuitively aim to fix all flaky tests.

Although we’ve seen some of our beta customers adopt on-call rotations to swat flaky tests as they appear, most of our other beta customers decided against fixing all their flaky tests. Most teams opt against fixing all their flaky tests because it’s impractical.

Dropbox tried to offload the burden to an engineering rotation to fix broken tests, and they noted, “The operational load of that rotation became too high, and we distributed the responsibility to multiple teams, which all felt the burden of managing build health.” At some point, the cure became worse than the disease.

Addressing flaky tests is intended to speed up development and reduce pain. This means fewer blocked PRs, fewer developer hours wasted debugging flaky failures, and fewer wasted CI resources. If this is the goal, then not every flaky test is worth fixing. You can’t spend more engineering hours fixing flaky tests than the time lost when ignoring them; the investment has to be economical.

Some flaky failures appear once every few weeks and disappear again, while others might block every other CI job. Fixing the flaky tests that block most developers first and reducing the felt impact of other flaky tests yields the most bang for the buck. Trunk helps you make more cost-effective decisions by letting you sort tests by PRs impacted.

For the difficult-to-fix and lower-impact flaky tests, you still want to reduce their impact even if fixing them is impractical. The sweet spot here seems to be quarantining flaky tests so they don’t block PRs. Quarantining lets you isolate test results from flaky tests at run time. These quarantined tests still run and report results but don’t block PRs. This also ensures that no tests are disabled because disabled tests are rarely revisited and disabled. They just rot, and you lose test coverage. When you fix a flaky test, this continued tracking also clarifies if that fix works.

Another important observation is that we sometimes see flaky tests because of the underlying infrastructure, which is often outside the teams’ control. For tests like that, there’s very little you can do to fix the test. These could be special tests that span many services, require specific hardware, or rely on external APIs. A “spillway” if you will, where failures due to reasons outside a team’s practical control get handled are useful in their own right.

What's more, is that studies have shown that "fixes" submitted by developers to fix flaky tests often don't work anyway, which is more of a reason to have a proper monitoring solution for flaky tests. You need to be able to see that a fix actually worked on your flaky test before marking it healthy again, otherwise, it just ruins your developers' trust in your tests.

Cleaning Under The Bed

One of the main concerns about quarantining tests is that they will be forgotten after the friction is gone. Quarantining fixes the issue of the “boy who cried wolf,” where teams ignored their CI results because the tests were so flaky they lost trust in tests. Quarantine can potentially do the opposite, where flaky tests cause too little friction, and people still ignore them.

To prevent this, we must ensure good reporting and insight to prevent erosion in test coverage over time. We’re currently trying a few ideas. One sees time series data on the number of quarantined test runs over time. You want to see this number go down, not up.

Another is periodic reports. These should surface overall trends in test health, like if your tests are becoming flakier, point to the new flaky tests, and show if the top flaky tests have changed.

What’s In The Next Part

To recap, flaky tests are prevalent if you write end-to-end tests, but teams underestimate how many of their tests flake and how much time is wasted dealing with Flaky Tests. Flaky Tests are annoying enough to have a substantial impact on productivity, but the fix for many teams is more expensive than to keep rerunning tests. The challenge, then, is to create a workflow that reduces the impact of flaky tests without making the cure worse than the disease.

Read part two for more about the challenges we faced building a solution to detect, quarantine, and eliminate flaky tests.

Try Trunk Flaky Tests

We don’t think flaky tests are a realistic or practical problem to tackle in-house. For this exact reason, we’re opening Trunk Flaky Tests for public beta so you can help us build a more effective solution for all. The sooner we stop suffering in isolation and begin pooling our feedback into a single solution, the sooner we’ll end flaky tests.

Join the Trunk Flaky Tests Public Beta and read part two of this blog here.

Convincing non-technical leadership to invest in DevEx.

Vincent Ge — Mon, 26 Aug 2024 17:51:35 +0000

Working as a DevRel for a DevTools company, I sometimes find myself advocating not just for our users/our company, I often do advocacy strategy for our users toward their leadership because they need to convince their org on why they're buying in.

It's hard if their non-technical leadership cares about only MRR and ARR numbers (which is most of them).

Wait, some teams don't run tests?

Whether we're on Reddit, Tech Twitter, or DailyDev, we share (or pretend to share) an understanding of striving for cleanly written, well-tested, and well-maintained code bases.

The reality, however, is that many engineering orgs struggle to do the basics.

I've heard something similar to this several times:

We don't need to keep tests from failing in main. Some individuals on my team don't even believe in writing tests. I don't think they'll buy into a merge queue.

Or something like:

Our non-technical leadership doesn't like it when we spend time working on code or tools that don't impact revenue. Linting and formatting falls into this category.

Or even:

100% test coverage is not necessary, some parts of the code is so simple, I don't need to test it to know if it works.

These people will often make it to engineering leadership in an org for being pragmatic. EW...

But "I" know better!

Now we know better. We know that full test coverage helps us make changes faster and more confidently because we have guardrails that prevent us from breaking other people's code. We know that untested code is code that doesn't work. We know we need linted and formatted code so everyone can focus on problem-solving instead of arguing about how many tabs to use. We know that linters can help catch anti-patterns that can impact security and performance.

I know that you know better. How do you ensure that leadership understands why these things are important?

And no, saying that they're incompetent and engineers should lead your org isn't a valid answer, most of the time.

How to sell to non-engineers

Our concerns about DevEx, code quality, testing, and well-architected code are all valid. Where it goes wrong is how we sell this to leadership.

The problem is that we're too abstract in how we sell it.

We often focus on saying this is the standard, this is better practice, or this is what other top engineers do. This is abstract. Engineers take these "standard practices" as truth because someone else (and maybe even you!) has felt the pain hundreds of times and learned to avoid it. Your non-technical leadership and even some technical leaders who haven't wet their toes in years will not relate easily.

Instead, this is what you need to focus on. This is the story you need to tell.

Focus on impact

Poor testing has a cost. If you change code and there's insufficient testing to ensure you won't break someone else's code, it will resurface unexpectedly during development/code review or in production.

If your code base has poor coverage, this has probably already happened. Collect these anecdotes, or create them by doing some mocks. Create some innocent-looking code change that will have an unexpected impact to use as an example.

Same with code quality and linters/formatters. If you can't find stories, commit a change with tabs instead of spaces, 4 wide tabs instead of 2 wide, add your line breaks in the wrong places, argue with your coworkers for a few hours, and save the email exchange or Slack thread.

Quantify the impact

How much does it cost to break the main? How much does it cost to argue about code quality? How much does it cost to take down your production application?

Remember how much you and your teammates' time cost? What we often fail to illustrate is how fucking expensive it is for a dev team to argue about where the { should go or how an innocent-looking change is breaking tests in main.

While your leadership won't understand pain, they understand developer_hours == $$$. Now you have your anecdotes, make a chart of how much it costs to manually retest things every code review, or argue about code format.

For example:

Every PR has an 80% chance of containing a formatting issue; 4 engineers argue for an hour 20% of the time.
Every PR has a 20% chance of breaking something unexpectedly; 50% of the time, it's caught, but it costs 2 engineers 4 hours on average to find the issue. The other 50% of the time it takes down prod for an hour.
We put in 10 PRs per week.
Engineers make $75 an hour.

Now do the math... Actually I feel lazy, here's what ChatGPT says.

Here's Claude:

Wow they disagree, but regardless, that's a big number, probably easily 10-20x the cost of a tool, and in a year, enough money to justify hiring an intern to do nothing but address these issues.

Restricting scope

Now, pitch your fix. How much time, what tools, how long, and what's the ROI. Focus on 1 high-impact issue that has a clear scope. Focus on how long you think it will take and what you get back. You're looking to achieve ~10X ROI in a year for a pitch that they can't resist (and it's not hard).

Time to practice

This stuff takes practice and is often the differentiator between staff engineer/tech leads and your run-of-the-mill dev. You should try these tricks if you're not yet a staff engineer/tech lead. It shows initiative and has 0 risk for you (because it's not your job anyway).

A word from our sponsor

Trunk makes several dev tools that you can use to refine and practice your pitch:

If you'd like to get some advice on how to write your pitch, chat with us on Slack.

Regex, the good bits.

Vincent Ge — Tue, 25 Jun 2024 04:03:54 +0000

There are two types of developers: those who fear regex because they don't understand it and those who abuse regex to flex on their millennial teammates.

The purpose of this blog is to get you somewhere in between. Know the bits that will be super useful without being dangerous.

Wait, regex can be dangerous?

Regex can do some spectacular things. You can write entire programs in regex. But just because you can, doesn't mean you should. Imagine a giant regex pattern uses all the powerful bits of regex, like recursive patterns, conditional patterns, look ahead and look behinds, and introducing side-effects with a replace.

I mean look at this:

(function(a,b){if(/(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge |maemo|midp|mmp|mobile.+firefox|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows ce|xda|xiino/i.test(a)||/1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r |s )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1 u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp( i|ip)|hs\-c|ht(c(\-| |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac( |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt( |\/)|klon|kpt |kwc\-|kyo(c|k)|le(no|xi)|lg( g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-| |o|v)|zz)|mt(50|p1|v )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-| )|webc|whit|wi(g |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-/i.test(a.substr(0,4)))window.location=b})(navigator.userAgent||navigator.vendor||window.opera,'http://detectmobilebrowser.com/mobile');

This was a somewhat commonly used pattern to detect mobile browsers at some point.

My problem with regex is this:

Regex syntax is extremely concise, which means extreme mental-overhead to read and write.
Regex has lots of exceptions. It's grammar and rules are inconsistent at best.
Really hard to split into multiple lines and not endup with a huge cursed string.
Updating regex to accept new behavior, reuse bits of the logic is hard.
When it gets large enough, everyone's afraid to touch it.

You will end up with impossible to read and maintain code if you go too far with regex.

On to the good bits

At its core, regex is a powerful way to search and match text based on rules, and extract information into variables. It can be used to manipulate string, but I'm going to avoid this. Most people do not expect regex to have side effects.

Stuff like pulling out a html tag with certain classnames, formatting phone numbers, and log parsing are great examples of good places to use regex.

Basic patterns

Take this example of using regex in JS.

const words = [
    "Hello world",
    "This is a short! message that says \"Hello world\"",
    "I love regular expressions"
];

// this is the pattern
const re = new RegExp("ello");

// the pattern can be used to "test" for matches
console.log(words.filter(value => re.test(value)))

The regular expression pattern ello used in re.test() will match any string containing the pattern as a substring. This is the simplest type of pattern.

It will match the following lines:

[ 'Hello world', 'This is a short! message that says "Hello world"' ]

These patterns are by default case sensitive. You can define case in-sensitive with the option new RegExp("ello", "i");.

Start and end of text

Regular expressions has "meta characters" that define logical rules for your pattern to match against.

The character ^ means beginning of text and $ means end of string.

For example:

// This will match "Hello, world!" but not "Message: Hello, world!".
const re = new RegExp("^Hello");

// This will match "Hello, world" but not "Hello, world!"
const re = new RegExp("world$");

Match one of variations

Sometimes you want to match variations of a similar pattern. In the most basic cases, variations of words like fine, pine, and line. In these cases, you can define a group of options in brackets like this: [].

const words = [
    "fine",
    "pine",
    "line!"
];

// This will match all the words above.
const re = new RegExp("[fpl]ine");

console.log(words.filter(value => re.test(value)))

You can use ranges of ascii characters in these any-of groups like this [a-zA-Z0-9].

const words = [
    "1ine",
    "Pine",
    "zine!"
];
// still match all the words.
const re = new RegExp("[a-zA-Z0-9]ine");

console.log(words.filter(value => re.test(value)))

An alternative approach is to use | which represents a logical or to match alternatives.

const words = [
    "color",
    "colour",
];
const re = new RegExp("color|colour");

console.log(words.filter(value => re.test(value)))

Wildcard

Sometimes you don't want to specify options, you want to match every variation imaginable. We can use the . character to specify a wildcard.

const words = [
    "%ine",
    "}ine",
    "`ine!"
];
const re = new RegExp(".ine");

console.log(words.filter(value => re.test(value)))

Repeating patterns

Sometimes we want to match a character repeatedly. For example, matching every variation of yeet, like yeeeeeeeeeet or yeeeeeeeeeeeeeeeeeeeeeeeet.

We can use + or *. * matches the preceding element zero or more times, + Matches the preceding element one or more times.

const words = [
    "yeet",
    "yeeet",
    "yeeeeeeeeet"
];
// matches [ 'yeeet', 'yeeeeeeeeet' ]
const re1 = new RegExp("yeee+t");

// matches [ 'yeet', 'yeeet', 'yeeeeeeeeet' ]
const re2 = new RegExp("yeee*t");

console.log(words.filter(value => re1.test(value)))
console.log(words.filter(value => re2.test(value)))

Use + if you want to match the character at least once, use * to mean it's optional but try to match many times if possible.

A interesting side effect of this is that they can be combined with the wildcard .. Try .* and .+ in your patterns, but becareful, .* will match literally anything which can be very error prone.

Another useful piece of syntax is {} which specifies the number of times a character or part of a pattern is repeated. For example:

const words = [
    "100",
    "1011",
    "222222"
];
// matches [ '1011' ]
const re = new RegExp("^[0-9]{4}$");

console.log(words.filter(value => re.test(value)))

Useful macros

There are some metacharacters that behave kinda like macros. These metacharacters are fundamental in constructing regex patterns to match specific text patterns in strings.

Metacharacter	Description	Example Match
`\d`	Digit (0-9)	`4`, `9`, `0`
`\D`	Non-digit	`a`, `Z`, `%`
`\w`	Word character	`a`, `A`, `1`, `_`
`\W`	Non-word character	`!`, `@`, `#`
`\s`	Whitespace	, `\t`, `\n`
`\S`	Non-whitespace	`a`, `1`, `%`
`\b`	Word boundary	`\bword\b`, `\b123\b`
`\B`	Non-word boundary	`\Bword\B`, `\B123\B`

\d, \w, and \s are pretty self-explanatory. What I wanna focus on is the \b and \B meta characters. These are extremely useful when parsing prose, because they respect natural word boudaries. For example in "hello, boss", hello is a independent word, but it's followed by a , which means if you naively matched the pattern \shello\s, the word will be missed. Similarly, matching for hello naively will also match words like phelloplastics.

const words = [
    "hello, boss",
    "galvanized square steel.",
    "dave has saved up a looooonnnggg time for his new prison-esque house."
];
// matches words ["hello, boss"]
const re1 = new RegExp("\\bhello\\b");

// matches in words ["dave has saved up a looooonnnggg t...]
const re2 = new RegExp("\\Bsq\\B");

console.log(words.filter(value => re1.test(value)))
console.log(words.filter(value => re2.test(value)))

Extracting values

Capture groups in regex let you extract values with a patter. You define capture groups with (<subpattern>) and everything matched by the pattern enclosed in () is returned.

For example, parsing an email:

let email = "example.user123@example.com";
let r = /^([\w\.-]+)@([\w\.-]+)\.([a-zA-Z]{2,6})$/;
let match = r.exec(email);

if (!match)
  throw Error("invalid email")

// the capture group is returned as an array of matches.
let username = match[1] ?? '';
let domain = match[2] ?? '';
let tld = match[3] ?? '';

console.log("Username:", username);
console.log("Domain:", domain);
console.log("Top-Level Domain (TLD):", tld);

What's even cooler is that capture groups can be named for more readable patterns and they can be used to match multiple times. For example, here we can extract multiple emails:

let email = "example.user123@example.com example.user123@example.com example.user123@example.com";
let pattern = /(?<username>[\w\.-]+)@(?<domain>[\w\.-]+)\.(?<tld>[a-zA-Z]{2,6})/;
let match = pattern.exec(email);

if (!match)
  throw Error("invalid email")

// loop over all matches
for (group in match.groups) {
  let username = match.groups.username;
  let domain = match.groups.domain;
  let tld = match.groups.tld;

  console.log("Username:", username);
  console.log("Domain:", domain);
  console.log("Top-Level Domain (TLD):", tld);
}

Wait, you missed this cool thing!

If you already know a lot of regex, great! I know everyone has their favorite little tricks with regex. The point of this post is to remove the fear many developers feel when they see regex in code. I think the subset of regex introduced in this post give you more than enough to be powerful and literate in regex, but not enough to become abusive.

Of couse, if you feel like there's something cool others should know that I missed, leave it in the comments!

Cool stuff you can do

I've seen engineers do crazy things with grep. At a old job where we wrote realtime operating systems (ancient 30 year-old code bases), I saw entire chunks of our build process written in sed and awk which relies heavily on regex.

Other cool things you can do is write your own linters. I work at Trunk, and we make a thing called Trunk Check where you can write grep linters in less than a minute if you know your regex patterns.

Play around and share the cool/crazy stuff you make with the internet.

Find me on socials

You're parsing URLs wrong.

Vincent Ge — Sun, 26 May 2024 00:00:51 +0000

There are somethings that you should never build by yourself. Not because it's difficult, but because it's time consuming and filled with gotchas.

One of these things is URL parsing.

Try implementing your own URL parsing

Raise your hand if you've done this before ✋

const baseUrl = "https://example.com";
const endpoint = "api";
const resourceId = "123";

const fullUrl = `${baseUrl}/${endpoint}/${resourceId}`;
console.log(fullUrl); // Output: https://example.com/api/123

But you're not a barbarian, maybe you abstract this into a function:

function joinUrlParts(...parts) {
    return parts.map(part => part.trim()).join('/');
}

// Example usage:
const baseUrl = "https://example.com";
const endpoint = "api";
const resourceId = "123";

const fullUrl = joinUrlParts(baseUrl, endpoint, resourceId);
console.log(fullUrl); // Output: https://example.com/api/123

Well, that's certainly better, but this example can break with a single stray /:

// Example usage that breaks:
const baseUrl = "https://example.com";
const endpoint = "api/";
const resourceId = "123";

const fullUrl = joinUrlParts(baseUrl, endpoint, resourceId);
console.log(fullUrl); // Output: https://example.com/api//123

So maybe you tried again, you add sanitation to your input.

function joinUrlParts(parts) {
    // Trim leading and trailing slashes from each part
    const sanitizedParts = parts.map(part => part.trim().replace(/^\/+|\/+$/g, ''));
    // Join the sanitized parts with slashes
    return sanitizedParts.join('/');
}

This looks bullet proof? Right? Not that many edge cases!

Well, yes, but actually no.

You still have to support some interesting use cases:

What about joining https://example.com/cats and ../dogs/corgie.
This doesn't understand https://example.com/dogs/corgie#origins.
This doesn't escape characters http://www.example.com/d%C3%A9monstration.html
This doesn't accept query parameters https://some.site/?id=123
This doesn't let you parse URLs.

The list goes on. You CAN implement what

But why does this matter?

If you write modern JavaScript, you know that the URL object exists from JS Web APIs.

But JavaScript didn't always have a good way to construct and parse URLs built in. The URL object was first included in the ECMAScript 2015 specs.

There are still lots of older videos and blogs that parse URLs with all kinds of fancy magic like JavaScript - How to Get URL Parameters - Parse URL Variables
and some fun workarounds like The lazy man's URL parsing.

When I wrote my first few JavaScript projects in 2016, I parsed and build URLs with all kinds giant loops and regex. The solutions were hard to read at best, and extremely buggy at worse.

Then I used Node.js path.join() Method in my later projects. Until this week, I had assumed this was still the way to go. I even tried importing browserify's path implementation for the browser.

The modern JS way to URL handling

If you've been sleeping under a rock like me, here's a quick overview of the URL object from Web APIs.

// Create a new URL object
const url = new URL('https://example.com/path?param1=value1&param2=value2#section');

// Parse the URL
console.log('Host:', url.host); // Host: example.com
console.log('Path:', url.pathname); // Path: /path
console.log('Search Params:', url.searchParams.toString()); // Search Params: param1=value1&param2=value2
console.log('Hash:', url.hash); // Hash: #section

// Update pars of the URL
url.protocol = 'https';
url.host = 'new.example.com';
url.searchParams.set('param3', 'value3');
url.hash = '#updated-section';

// Recreate the URL
const rebuiltUrl = new URL(url.href);

// Update the URL object directly
rebuiltUrl.searchParams.set('param2', 'updatedValue');

// Print the updated URL
console.log('Rebuilt URL:', rebuiltUrl.href); // Rebuilt URL: https://new.example.com/path?param1=value1&param2=updatedValue&param3=value3#updated-section

Browser compatibility

All remotely recent, modern browsers will support the URL library. Some methods may not be fully implemented, but the basic usage remains consistent. Find the full browser compatibility table here.

If you must support ancient browsers, the core-js project provides a polyfill for older browsers.

Bottom line

Please use native Web APIs. Avoid building one off utility classes for common actions. JavaScript is constantly changing, I wrote my first lines of JS in 2016 and I'm constantly finding myself leaning into old, outdated information.

If you have a moment, take a look through the Web API docs from MDN. I guarantee you will find something new that solves a problem you needed to build your own solution for in the past.

A fun aside

I still hate how the JavaScript Date object's interface is so lacking.

Try finding a date 5 days in the past, or comparing if two events happened 3 days apart.

The fact that moment.js or day.js needs to exist in 2024 bothers me a lot.

More fun stuff

Come chat with me in these places:

In technical writing, the devil is NOT in the details

Vincent Ge — Mon, 13 May 2024 22:36:48 +0000

I've been interviewing technical writers. What stands out about amazing candidates is that they spend surprisingly little time on the details. This surprises me and had me reflect that I look for, or rather, what makes a good technical writer.

I think the assumption is that a good technical writer is made in the details. You want detailed, complete, and accurate documentation. This is what a lot of candidates focuses on, how they approach technical writing to make sure they understand the complete product use cases and accurately regurgitate that information on a page.

If this is a candidate's focus, to me, that's a giant red flag 🚩

As a technical writer, I don't think our value is in the ability to accurately recount the technical demos given to us by engineers. It's in structuring and filtering information to make it more accessible.

Take a look at the R manual, for example:

Complete? *Yes! *

Detailed? Yes!

Accurate? Definitely!

Would I, or anyone learning R read this thing? Well, this part is debatable.

It's a long running joke among technical writers that if your documentation requires a "how to read" section, you've done it wrong.

Technical writing serves a clear purpose, making information accessible. Getting accurate information is the easy part, any engineer can do a better job than you, because you didn't build the feature.

As a technical writer, your job is to abstract, structure, and enhance information to make it easier to access.

Once more for emphasis:

Your job is to abstract, structure, and enhance information to make it easier to access.

The devil is not in the details. The information is not novel. What's novel is the presentation and structure that lets a developer find what they need in 5 minutes, not 50.

Here's what I look for in a technical writer

Start with the journey

Good docs flow from most general information to least, commonly needed workflows to power moves, and low resolution to high resolution.

Give me only the information I need at a given stage in my learning process. Don't overwhelm me. Keep the examples simple, even if I'm missing details.

Quick wins

Your docs serve 0 purpose unless someone reads them. To encourage them, like a benevolent teacher, you need to give them confidence that you're leading them somewhere.

Quick wins encourage the readers to keep exploring. Attention span and patience are finite, you gotta fight for it.

Information hierarchy

Most important information first. I need to scan the start of a page to know if the page contains the information I'm looking for. I need to scan the start of a paragraph to know if the details are relevant.

Take this paragraph for example:

Security is important in development. That's why you should take care to protect secrets. Secrets should be safely stored as a environment variable in a vault. You can find vaults under **settings** > **security** > **vault**. Don't share this with anyone!

Now the same information:

Store secrets as environment variables in vaults by navigating to **settings** > **security** > **vault**. Your secrets should never be shared. You must ensure data privacy, sharing secrets can compromise security during development.

Same information, one is easier to read than others.

Know the reader

Some products expect expert readers. Some products expect novice readers.

Take a look at the getting started guide for PlanetScale and getting started guide for Supabase,

One focuses on high-level information to help senior devs make decisions on the tech stack, one focuses on quick wins and short examples to help you get your foot in the door.

Accessible links (bonus)

One thing I've started to look for is accessible links. How many times have you seen an embedded link like this.

Your links should be complete and have full context for it to be more accessible. While some can tell what a link does from context, those using accessibility tools might find it annoying to figure out where a link leads.

Do this ✅:

You can learn more about database indexing in our documentation.

Not this ❌:

You can learn more about database indexing here.

Bottom line

I care about accuracy in docs. If this is all I had to care about, I'd just make everyone read API references.

Fact of the matter is, I need to help a developer go from 0 to 1, but also experts to scale from 1K to 1M. You do this by abstracting and structuring information, so focus on your ability to do this on your next interview.

How I use Appwrite Databases with Pinia to build my own habit tracker

Vincent Ge — Mon, 15 Apr 2024 02:25:44 +0000

Some context

I'm building a project with Vue and Appwrite called Sisyphus. Much like Sisyphus, we have many tasks that are a real uphill grind everyday. And like Sisyphus, we repeatedly push that figurative boulder up the hill each day, anyway.

I built Sisyphus to help track my own habits with an UI like a GitHub commit history.

Here's a GitHub commit history for reference.

My inspiration

I used Appwrite to implement authentication and database APIs for my app. The SDK API is very similar to a REST API, which gives you lots of flexibility.

Inspired by a video by my colleague @dennisivy11 I decided to share how I use Appwrite Databases, too.

Do this when writing database queries with Appwrite - YouTube

How to write less, and cleaner code when working with an Appwrite database.Instructor: https://twitter.com/dennisivy11 / https://www.linkedin.com/in/dennis-i...

youtube.com

Here's how you can use Appwrite Databases with Vue 3, Pinia, and TypeScript to create elegant data stores that can be consumed in your components.

The Databases API

Here's what CRUD operations look like with Appwrite's SDKs.

// create a document
await 
databases.createDocument(
    'sisyphys', 
    'boulders'
    ID.unique, 
    {foo: 'bar'}
)
// read some documents
await databases.listDocuments(
    'sisyphys', 
    'boulders'
);
// update a document
await databases.updateDocument(
    'sisyphys', 
    'boulders'
    'drink-water'
    {food: 'baz'}
);
// delete a document
await databases.deleteDocument(
    'sisyphys', 
    'boulders'
    'drink-water'
);

Simple, RESTful, but not useful for building a UI. I need these operations to play nice with a store like Pinia to manage state and share data across components.

My data

I also have two collections in Appwrite with the following structure:

export type Pushes = Models.Document & {
  date: string;
  distance: number;
  boulderId: string;
}

export type Boulder = Models.Document & {
  name: string;
  description: string;
  distance: number;
}

Pushes tracks each time I pushed a boulder forward, it describes how far I pushed, on which day, and which goal/boulder.

Boulder tracks the goals I've created, their name, a short description, and the goal of how far I'd ideally push that boulder forward in a day in distance.

Looking back at the UI, the data needs to be consumed at these levels:

My boulder container component, which creates a boulder card for each goal I set.
Each boulder card, which needs to render a history of my progress on that goal.
Each form in my app, like when I create new goals or push a boulder and move my goal forward.

Creating a generic store for Appwrite collections

All Appwrite collections share these commonalities:

Belong to a database and has it's own ID
Data extends Models.Document, containing information like $id, permissions, $createdAt, $updatedAt, etc.
Has create, list, get, update, delete.

This means code related to these commonalities are going to be the same for each store I need for each Appwrite Databases Collection.

So, here's how I implement a base collection for all my Appwrite Collections.

Generic interface

// This is my base store
import { databases, ID, Query, type Models } from '@/lib/appwrite'
import { defineStore } from 'pinia'

export function defineCollection<Type>(
  name: string,
  database: string,
  collection: string,
) {
  return defineStore({
    id: name,
    state: (): => {
      return {
        documents: [],
      }
    },
    getters: {
       // we'll cover these later
    },
    actions: {
       // we'll cover these later
    },
  })
}

My collection store is a factory that returns a define store based on the name of the collection, the database id, and the collection id.

Notice the use of the generic Type, we pass this in from each collection store that composes/extends this base Pinia store with the structure of the collection.

For example:

// This is how I extend my store with a type
export type Boulder = Models.Document & {
  name: string;
  description: string;
  distance: number;
}

const COLLECTION = 'boulders';

const collectionStore = defineCollection<Boulder>(
  'collection-' + COLLECTION,
  import.meta.env.VITE_DATABASES_ID,
  COLLECTION
)

Here I'm extending the base collection store with the type <Boulder> so I get helpful type hints when I consume the store.

The state of the store is a list of documents that extends this type.

Getters and actions

Pinia implements getters and actions to help you interact with the app. Here's the generic getters and actions used by all my collection stores.

Here are my getters:

// ... skipped code in my base collection store
    getters: {
      count(state) {
        return state.documents.length
      }
    },

I don't need a lot of fancy getters in my base store, just something to return a count.

Here are my CRUD actions:

// ... skipped code in my base collection store
    actions: {
      async get(id: string) {
        return databases.getDocument(
          database,
          collection,
          id
        )
      },
      async list(queries = [] as string[]) {
        return databases.listDocuments(
          database,
          collection,
          queries
        )
      },
      async create(data: any) {
        return databases.createDocument(
          database,
          collection,
          ID.unique(),
          data
        )
      },
      async update(id: string, data: any) {
        return databases.updateDocument(
          database,
          collection,
          id,
          data
        )
      },
    // ... more actions

They just implement a simpler version of the existing SDK methods. Let's me reduce my method calls from this:

databases.createDocument(
    'sisyphys', 
    'boulders'
    ID.unique, 
    {foo: 'bar'}
)

To this:

collection.create({foo: 'bar'})

Why use many lines when few lines do trick?

Some more actions I implement are .load() and .all():

    // ... previous actions
      async load(queries = [], batchSize = 50) {
        try {
          this.documents = await this.all(queries, batchSize)
        }
        catch (error) {
          this.documents = []
        }
      },
      async all(queries = [] as string[], batchSize = 50) {
        let documents = [] as Type[];
        let after = null;

        let response: any = await this.list(
          [
            ...queries,
            Query.limit(batchSize),
          ]
        );

        while (response.documents.length > 0) {
          documents = [...documents, ...response.documents];
          after = response.documents[response.documents.length - 1].$id;
          response = await this.list(
            [
              ...queries,
              Query.limit(batchSize),
              ...(after ? [Query.cursorAfter(after)] : [])
            ]);
        }

        return documents;
      }

I call collection.load() to initialize or update the state of my store and collection.all() lets me responsibly paginate through all my data. These are utility methods I need in all my collection stores.

Of course, if you need methods like collection.page() or collection.nextPage() for a paginated store, you can extend the base store with more actions.

Sisyphus is quite simple, so we load all the data at once.

All these methods make the store generic enough that I will use all these methods in every collection I interface with and provide a more elegant interface to consume data as a store.

Extending my generic store

Remember how I mentioned I extend all my stores? Here's my boulders store as an example.

// My boulders store
export type Boulder = Models.Document & {
  name: string;
  description: string;
  distance: number;
}

const COLLECTION = 'boulders';

const collectionStore = defineCollection<Boulder>(
  'collection-' + COLLECTION,
  import.meta.env.VITE_DATABASES_ID,
  COLLECTION
)

export const useBoulders = defineStore(COLLECTION, () => {
  const parent = collectionStore();

  const boulders = computed(() => {
    return parent.documents;
  })
  const boulder = computed(() => (id: string) => {
    return parent.documents.find((boulder) => boulder.$id === id);
  });

  async function load() {
    return await parent.load();
  }
  async function add(boulder: Boulder) {
    await parent.create(boulder);
  }

  return { boulders, boulder, load, add };
});

I define the data structure expected to be returned by the store and call defineCollection<Boulder> to create this store.

Then, I use Pinia's composition API to extend it with a few methods:

I instantiate the generic collectionStore as the parent store.
I then add a getter called boulders that just returns a computed with the value of parent.documents. This ensures I maintain reactivity, which means when parent.documents changes, my Vue 3 components are notified.
Here's some ✨ magic ✨ , boulder() is a getter that takes in a boulder ID, searches for it, and returns it. Because it's a computed value, it is also reactive.
I then wrapped the parents load and create methods to use less generic names that make more sense when correlated with the verbs I use in UI.

Here's another example with my Pushes collection:

import { type Models } from '@/lib/appwrite'
import { defineStore } from 'pinia'
import { defineCollection } from './collection';
import { computed } from 'vue';
import { getToday } from '@/lib/date';
import type { Boulder } from './boulders';

export type Pushes = Models.Document & {
  date: string;
  distance: number;
  boulderId: string;
}

const COLLECTION = 'pushes';

const collectionStore = defineCollection<Pushes>(
  'collection-' + COLLECTION,
  import.meta.env.VITE_DATABASES_ID,
  COLLECTION
)

export const usePushes = defineStore(COLLECTION, () => {
  const parent = collectionStore();
  const pushes = computed(() => (id: string) => {
    return parent.documents.filter((push) => push.boulderId === id) ?? {
      date: '',
      distance: 0,
      boulderId: '',
    };
  });

  async function load() {
    await parent.load([], 500);
  }
  async function push(distance: number, boulderId: number) {
    await parent.create({
      date: getToday().toISOString(),
      distance: distance,
      boulderId: boulderId,
    } as unknown as Pushes);
    load();
  }

  return { pushes, load, push };
});

While I load all pushes at once, each boulder card needs to get filtered results from the store. I again use computed to make sure the filtered value is reactive.

Consuming these stores

Important note about consuming stores like this is that you will need to convert them to refs so they retain reactivity in your UI.

For this, we can destructure them like this:

<script setup lang='ts'>
// ... skipped imports
import { useBoulders } from '@/stores/boulders';
const { boulder } = storeToRefs(useBoulders());
</script>

<template>
          <Boulder v-for="boulder in boulders" :boulderId="boulder.$id"/>
</template>

This would render a list of boulders and dynamically render new ones when the boulders document changes.

Other fun things to think about

When needed, you can extend these stores with Realtime, so that they can listen to updates updates in a collection made by other users in realtime. This would be great for chat apps, etc.

If you're interested in seeing realtime or pagination added to one of these stores, let me know in the comments.

If you haven't tried Appwrite, make sure you give it a spin. It's a open source backend that packs authentication, databases, storage, serverless functions, and all kinds of utilities in a neat API. Appwrite can be self-hosted, or you can use Appwrite Cloud starting with a generous free plan.

Cheers~

Hackathons: A Comedy of Errors, Sleep Deprivation, and Unforgettable Memories

Vincent Ge — Wed, 24 May 2023 14:22:11 +0000

Have you tried explaining a Hackathon to someone who’s not a developer or a student?

No really, thinking back to my first Hackathon, Hack the North 2017 in the University of Waterloo, I’m not sure how I’d explain it to family and friends.

I was with 1000+ sweaty strangers with their laptops, stuck in the same building on campus, coding straight for 36+ hours without sleep, sustained solely by Soylent and caffeine chocolates. Then we present the abomination of a project we dreamed up to be judged by a panel of professionals to be our first impressions. The best part, we did it voluntarily and thought it was fun.

Hackers at Junction 2015

Hysterical and borderline insane. This is probably how someone outside the development community would see us, if we didn’t provide anymore context.

So what are hackathons really like and why do thousands of developers still flock to them, every year?

Well, here’s the typical experience of every first-time hacker.

🤩 The pre-Hackathon frenzy

It all starts with that fateful moment when you stumble upon a hackathon announcement online. Your eyes light up, your heart races, and you begin fantasizing about the possibilities.

“What if I win a prize? What if my skills are recognized by a sponsor and I receive a job offer? What if I finally get the perfect idea for my startup, and this is my chance to impress investors,” you thought, scrolling through the list of companies sponsoring the hackathon.

🕵️‍♂️ The great team hunt

Hackathons are a bit like a speed dating show for techies. You're searching for the perfect match to complement your skills and bring your ideas to life. So, you mingle, make small talk, and subtly show off your coding prowess.

Finally, you assemble a team that's a blend of mad scientists, digital wizards, and that one person who takes a nap 12 hours into the hackathon, never to be seen again until 5 minutes before the submission deadline.

Many team hunts start on Slack or Discord, where attendees mingle before the event begins.

💡 Abundant optimism, romanticism, and naivety

Now comes the brainstorming session. A whirlwind of caffeine and adrenaline fueled excitement and an avalanche of ideas. It's a creative explosion of clever solutions to solve the world's most pressing problems.

With each idea, your team becomes more confident and more excited. You begin to actually believe in your ideas and your team’s skill to pull it off. You start wondering why no one has thought of these ideas before, and if they’ll make you the next Elon Musk.

🌙 The Sleepless Sprint

With your collective genius focused on a single goal, you embark on a coding marathon that feels like a high-stakes heist movie. As the clock ticks down, your team deftly navigates a labyrinth of code, design, and countless energy drinks.

Caffeine chocolates like this are frequently handed out to hackers during MLH hackathons.

While the star programmer of the team pumps out mountains of code, you and friend help your other teammate to setup Git, debug NPM, and complain about his choice to bring a Windows laptop to a hackathon.

The sevens stages of compromise

You look up, 5 hours left on the clock, reality starts to sink in, and your grand vision crumbles like a cookie under the weight of deadlines. And you begin wondering if your missing teammate, on a hunt for swag and snacks, has fallen asleep on a toilet.

In a matter of minutes, you blitz through the stages of grief. Shock, denial, anger, bargaining, depressing, finally accepting your fate, you begin to discuss alternate plans with your team.

You cut features, make sacrifices, fake data, and pick out your favorite powerpoint template. In the end, wrap your Frankenstein's monster of a project in an elegant looking frontend interface, and hide it all behind a confident, charismatic presentation.

🎤 The Moment of Truth

With frayed nerves and weary smiles, you stand before the judges to present your project. The room is filled with the collective groans of sleep-deprived brains and the faint smell of coffee.

As you share your team's story and make up answers to questions you have no response, you can't help but feel a sense of pride in your sleep-deprived, caffeine-driven accomplishment.

Winning team at TC Hackathon at Disrupt Berlin.

That sense of pride quickly vanishes as “that one team” team walks in with a fully functioning electro-magnetic scanner that detects cancer and solves global warming at the same time. Hardware and algorithm, developed in under 36 hours.

👋 The Bittersweet Goodbye

As the hackathon comes to a close, you part ways with your newfound friends, bonded by shared memories of chaos, laughter, and caffeine-induced hallucinations. You may not have changed the world, but you've emerged as a more resilient, adaptable, and slightly twitchy version of yourself.

You also start considering an alternate career path. There’s no way you’re competing with people from “that one team,” who completed a project you couldn’t finish in 36 months, and somehow had better hair, too.

Nevertheless, you made it, and it’s a moment you’ll never forget.

💫 The Appwrite Hashnode Hackathon

In person hackathons are stressful. But the Appwrite Hashnode hackathon isn’t. You can sign up today to participate from the comfort of your home, over the internet.

No sweaty gymnasiums, fighting over ethernet ports, sleepless nights, and wondering if the room smells like farts or the pile of takeout someone spilled in the corner.

Join today with your team, for a chance to win up to $5000 USD and limited edition Appwrite swag!

Introducing DateTime Support and ISO 8601 Dates

Vincent Ge — Wed, 14 Sep 2022 13:14:03 +0000

In Appwrite 1.0, we’re introducing a lot of new features. Among them is one of our most frequently requested features, DateTime support. Appwrite Databases now provides a dedicated DateTime attribute for storing date and time information. All Appwrite endpoints will also now return timestamps that are in DateTime format following ISO 8601 specification.

First time hearing about Appwrite? Appwrite is an open-source backend-as-a-service that abstracts all the complexity involved in building a modern application by providing you with a set of REST APIs for your core backend needs. Appwrite handles user authentication and authorization, real-time databases, cloud functions, webhooks, and much more!

Common DateTime Representations

There are two main representations of storing date and time in computing, Unix Time and DateTime.

Unix Time, also known as Epoch time or Posix time, stores time since Unix Epoch on January 1, 1970 at midnight UTC. Unix Time is expressed and stored as a 32 bit integer.

DateTime stores time as a string. The format of the string is human readable, expressing year, month, date, hour, seconds, and milliseconds. For example, 2022-08-17T00:00:00, which is the Midnight of August 17th, 2022 in UTC.

The advantage of using Unix Time is that it’s stored as an integer. This makes Unix Time easier to store, query, and sort. A 32 bit integer takes up less space than a DateTime string and is less computationally intensive to do comparisons and arithmetic with. Unix Time is also independent of timezones, making it perfect for representing time on the web.

The downside to Unix Time is that it’s not easily human readable and limited in date range. Unix Time cannot store time before 1970 or after 2038, which is very limiting.

DateTime following ISO 8601 is highly readable by a human and the default storage format for JSON. The potential downside of DateTime is that it can be timezone ambiguous and somewhat expensive to store and sort.

Why DateTime?

Supporting DateTime attribute is one of the most frequently requested features we receive. The upsides of being highly readable by humans and being more flexible is highly valued to many Appwrite developers.

We also decided to use DateTime for Appwrite’s response objects because it’s the default format provided by JavaScript’s Date() object. This helps us stay consistent.

To remove timezone ambiguity, Appwrite will handle and store all DateTime in UTC timezone. Unix Timestamps are still supported by Appwrite's Databases Service through 32 bit integers.

Examples

Most languages will support both DateTime following ISO 8601 and Unix Timestamps in their system library. I’ll take JavaScript as an example to show you how to manipulate DateTime strings.

To create a datetime string of the current time in UTC:

// Gets current date and time in UTC
var date = new Date(Date.now());

// Print current date and time in UTC
console.log(date.toISOString())

Just as easily, you can parse a UTC DateTime string.

// A DateTime string in UTC, converted to date object
const date = new Date('2022-08-24T20:41:21.909Z');
console.log(date.toISOString());

Notice that the dateTime string will have a Z at the end to denote UTC. When Javascript parses this string, it will automatically convert it to your machine’s timezone.

Conclusions

One of the biggest benefits of building open source software in full publicity is the real-time feedback we receive from the community. It allows us to hear more voices and iterate faster.

This new feature was the result of many suggestions and discussions on GitHub. I wanted to shout out those who pitched this idea to us time and time again:

Your feedback is as valuable as code contributions and we encourage readers to pitch your thoughts to us.

Appwrite Loves Open Source: Why I Choose To Support Kdenlive

Vincent Ge — Wed, 03 Aug 2022 20:58:10 +0000

To me, the open-source community is about coming together and building cool stuff that empowers others. This shouldn't be limited to building developer oriented tools like TensorFlow and Jest. Open-source can also empower those outside the developer community by removing barriers to accessing software tools.

Open-source is at the heart ❤️ of everything we do at Appwrite, and we want to enable and foster the open-source community that helped us grow to over 16K stars ⭐ on GitHub. At Appwrite, we recognize how difficult it can be to build open-source software. Sustaining a project and a team takes enormous commitment, and getting adequate funding is difficult. This is especially true to open-source projects that don't face developers, making it harder to attract contributors. That's why Appwrite allows each engineer to pick an open-source project to sponsor.

When Appwrite offered to sponsor an open-source project on my behalf, I jumped at this opportunity to support Kdenlive.

Equal Representation in Media

Everyone deserves a voice. Creating videos to raise awareness to problems, tell stories, or share cool ideas is an important avenue to be heard in today's world. The problem is that while videos are effective, they are also expensive to make. Even with increasingly cheap access to relatively high quality cameras and microphones, video editing software keeps getting more expensive.

With Adobe Premiere Pro priced $52.99 USD/month and Final Cut Pro priced at $299.99 USD, their high price shuts out many individuals and communities from having a voice. This can lead to many important stories and ideas to be underrepresented, even in the age of the internet. Young creators, or those with less funds to create videos (like myself when I was younger), can't express their ideas. Entire segments of the population might be shutout of YouTube, TikTok, and other media platforms as a result. These segments of the population are often most in need of a voice.

What is Kdenlive?

Kdenlive is a fully featured video editor and provides a multitrack workflow similar to other mainstream software like Adobe Premiere. Kdenlive comes packaged with enough video/audio effects, transitions, fonts, configurable hot-keys, and supported input/output formats that allows a user to develop a workflow just as effective and efficient as Adobe Premiere. To rival the industry is surprisingly difficult to achieve with creative software, especially for an open-source project, because of the amount proprietary knowledge required to replicate the vast set of features.

Why I choose Kdenlive

What I love about Kdenlive is their regularity and transparency in communication. They are fully transparent about their road maps, efforts, community, and how they spend donation money. Each year, Kdenlive creates a report detailing everything that they do in a year, and their plans moving forward.

Here are some upcoming features I'm excited about. The Kdenlive team is working to implement nested timelines like in Premiere Pro. This allows you to work on small timelines for different parts of a production, then nest them within a larger timeline to apply affects to them together, reducing repetitive work. Another feature is “embedded effect”, which allows you to apply effects to all clips at once, instead of one by one (like in Premiere :P). They're also working to increase responsiveness and UX of the software.

Like Appwrite, Kdenlive aims to provide an open-source alternative to existing expensive, close-sourced software. Kdenlive is also not a carbon copy of Premiere Pro, they try to fill a similar niche, but have their own opinions and philosophy behind development. What's not to love?

Learn More About Appwrite

Appwrite is an open-source Backend-as-a-Service (BaaS), packaged as a set of Docker micro-services, to give developers of any background the tools necessary to build modern apps quickly and securely.

Check out Appwrite as the backend for your next web, Flutter, or server-side application. Here are some handy links for more information:

How Do I Index My Appwrite Collection?

Vincent Ge — Tue, 28 Jun 2022 18:49:11 +0000

The Appwrite Hackathon on DEV has just concluded and one of the frequent pain points among new Appwrite developers is with adding indexes to collection. When attempting to query a collection or sort the documents returned, Appwrite developers frequently faced errors related to missing or incorrect indexes. This post will cover what indexes are, why we need them, and how to use them correctly in Appwrite.

What’s an Index?

An index helps you search a collection of documents more efficiently. Appwrite requires an index for all sort and query type operations to ensure your application runs efficiently.

Imagine a collection as a catalog of different movies, where each movie has a unique and random ID and the catalog is sorted by ID. If you know the movie’s ID, finding the movie is easy. If you wanted to look up a film by a particular title, you might have to look through the entire catalog to find the one movie. That’s a lot of work! If the catalog had an index of sorted movie titles and their corresponding ID, on the other hand, you’ll have a much easier time finding movies.

An index in a database works in a very similar way, only that they’re implemented using data structures such as B-Trees. The performance of an index lookup is O(log(n)), while a full table scan is of O(n). That's a huge difference!

Indexes found in Appwrite rely on the implementation found in the underlying database. As of today, Appwrite only supports MariaDB as a database adaptor, so the behavior of indexes and querying indexes should be very similar to that of MariaDB.

When do I need an index?

For performance reasons, Appwrite requires an index for every query and sort operation. If you need to query multiple attributes in the same query, you’ll need an index with multiple attributes. Appwrite supports three types of indexes.

Key: Plain indexes that allow you to perform sorts and queries.
Unique: Indexes that don’t allow duplicates, useful for indexing unique identifiers.
Fulltext: Indexes that allow you to query string attributes that may not match the query parameter exactly. You can use this field type to look for keywords in movie descriptions, for example.

When an index is created, an equivalent index is created in the underlying database (MariaDB) to make efficient queries and sorting possible.

Using Indexes in Appwrite

Before you can consume an index, you’ll need to first create them. I’ll be using a collection of Disney movies as example to show you how to create different types of indexes and how to query them. The collection configuration and code referenced can be found in this Gist

The collection belongs to a database called entertainment has the following attributes:

We will create indexes to allow querying for movies by title, querying unique movies by title and year, as well as searching movie descriptions using keywords.

Creating Indexes

To search for movies by title, we need to create an index with the name title and type Key. This allow us to query for all movies that share the same title. The match is exact.

To find unique movies and filter out remakes or duplicate titles, we can create a Unique index named title-and-year. Notice how this index includes both title and year attributes. This is necessary to query both attributes at the same time.

To search for keywords in a movie’s description, we can create a Fulltext index named description.

Querying Using Indexes

To find every version of “The Lion King”, the query will look like this:

// Point sdk to entertainment database
const databases = new sdk.Databases(client, 'entertainment');

// Query for the movie "The Lion King"
databases.listDocuments('movies', [sdk.Query.equal('title', 'The Lion King')]),

This will returned the following:

{
   "total":2,
   "documents":[
      {
         "year":2019,
         "title":"The Lion King",
         "description":"The animated remake of Lion King",
         "$id":"62a793e62d7c105ee92f",
         "$read":[

         ],
         "$write":[

         ],
         "$collection":"movies"
      },
      {
         "year":1994,
         "title":"The Lion King",
         "description":"The original cartoon Lion King from Disney.",
         "$id":"62a793e631cce5281683",
         "$read":[

         ],
         "$write":[

         ],
         "$collection":"movies"
      }
   ]
}

To find only the 1950 edition of Cinderella, the query will look like this:

databases.listDocuments('movies', [sdk.Query.equal('title', 'Cinderella'), sdk.Query.equal('year', 1950)]),

This will returned the following:

""{
   "total":1,
   "documents":[
      {
         "year":1950,
         "title":"Cinderella",
         "description":"With a wicked stepmother (Wilfred Jackson) and two jealous stepsisters (Homer Brightman, Harry Reeves) who keep her enslaved and in rags, Cinderella (Clyde Geronimi) stands no chance of attending the royal ball.",
         "$id":"62a793e6886c41162589",
         "$read":[

         ],
         "$write":[

         ],
         "$collection":"movies"
      }
   ]
}

Finally, to search for all remakes, we can search descriptions for the keyword remake:

databases.listDocuments('movies', [sdk.Query.search('description', 'remake')])

This yields the following results:

{
   "total":2,
   "documents":[
      {
         "year":2019,
         "title":"The Lion King",
         "description":"The animated remake of Lion King",
         "$id":"62a793e62d7c105ee92f""$read":[

         ],
         "$write":[

         ],
         "$collection":"movies"
      },
      {
         "year":2015,
         "title":"Cinderella",
         "description":"The 2015 remake of the original animated Cinderella.",
         "$id":"62a793e688f4ff05f620",
         "$read":[

         ],
         "$write":[

         ],
         "$collection":"movies"
      }
   ]
}

Et voilà! That’s how you create and query indexes with Appwrite!

Final Remarks

Indexes come at a cost. They require additional data structures which costs processing to create and maintain on writes, as well as more storage space. It’s a trade off to get faster reads for slightly slower writes speeds and more storage space. As a developer, you should be conscious of which attributes you need to query and only create necessary indexes.