DEV Community: AppSignal

Find and Fix N+1 Queries in Django Using AppSignal

Nik Tomazic — Wed, 18 Dec 2024 13:48:22 +0000

In this article, you'll learn about N+1 queries, how to detect them with AppSignal, and how to fix them to speed up your Django apps significantly.

We'll start with the theoretical aspects and then move on to practical examples. The practical examples will mirror scenarios you might encounter in a production environment.

Let's get started!

What Are N+1 Queries?

The N+1 query problem is a prevalent performance issue in web applications that interact with a database. These queries can cause significant bottlenecks, which intensify as your database grows.

The problem occurs when you retrieve a collection of objects and then access the related objects for each item in the collection. For instance, fetching a list of books requires a single query (1 query), but accessing the author for each book triggers an additional query for every item (N queries).

N+1 problems can also occur when creating or updating data in a database. For example, iterating through a loop to create or update objects individually, rather than using methods like bulk_create() or bulk_update(), can result in excessive queries.

N+1 queries are highly inefficient because executing numerous small queries is significantly slower and more resource-intensive than consolidating operations into fewer, larger queries.

Django's default QuerySet behavior can inadvertently lead to N+1 issues, especially if you're unaware of how QuerySets work. Querysets in Django are lazy, meaning no database queries are executed until the QuerySet is evaluated.

Prerequisites

Ensure you have:

Python 3.9+ and Git installed on your local machine
An AppSignal-supported operating system
An AppSignal account

Note: The source code for this project can be found in the appsignal-django-n-plus-one GitHub repository.

Project Setup

We'll work with a book management web app. The web app is built to demonstrate the N+1 query problem and how to resolve it.

Start by cloning the base branch of the GitHub repo:

$ git clone git@github.com:duplxey/appsignal-django-n-plus-one.git \
    --single-branch --branch base && cd appsignal-django-n-plus-one

Next, create and activate a virtual environment:

$ python3 -m venv venv && source venv/bin/activate

Install the requirements:

(venv)$ pip install -r requirements.txt

Migrate and populate the database:

(venv)$ python manage.py migrate
(venv)$ python manage.py populate_db

Lastly, start the development server:

(venv)$ python manage.py runserver

Open your favorite web browser and navigate to http://localhost:8000/books. The web app should return a JSON list of 500 books from the database.

The Django admin site is accessible at http://localhost:8000/admin. The admin credentials are:

user: username
pass: password

Install AppSignal for Django

To install AppSignal on your Django project, follow the official docs:

Ensure everything works by restarting the development server:

(venv)$ python manage.py runserver

Your app should automatically send a demo error to AppSignal. From this point forward, all your errors will be sent to AppSignal. Additionally, AppSignal will monitor your app's performance and detect any issues.

Web App Logic

The prerequisite to fixing N+1 queries is understanding your app's database schema. Pay close attention to your models' relationships: they can help you pinpoint potential N+1 problems.

Models

The web app has two models — Author and Book — which share a one-to-many (1:M) relationship. This means each book is associated with a single author, while an author can be linked to multiple books.

Both models have a to_dict() method for serializing model instances to JSON. On top of that, the Book model uses deep serialization (serializing the book as well as the book's author).

The models are defined in books/models.py:

# books/models.py
class Author(models.Model):
    first_name = models.CharField(max_length=64)
    last_name = models.CharField(max_length=64)
    birth_date = models.DateField()

    def full_name(self):
        return f"{self.first_name} {self.last_name}"

    def to_dict(self):
        return {
            "id": self.id,
            "first_name": self.first_name,
            "last_name": self.last_name,
            "birth_date": self.birth_date,
        }

    def __str__(self):
        return f"{self.first_name} {self.last_name}"


class Book(models.Model):
    title = models.CharField(max_length=128)
    author = models.ForeignKey(
        to=Author,
        related_name="books",
        on_delete=models.CASCADE,
    )
    summary = models.TextField(max_length=512, blank=True, null=True)
    isbn = models.CharField(max_length=13, unique=True, help_text="ISBN-13")
    published_at = models.DateField()

    def to_dict(self):
        return {
            "id": self.id,
            "title": self.title,
            "author": self.author.to_dict(),
            "summary": self.summary,
            "isbn": self.isbn,
            "published_at": self.published_at,
        }

    def __str__(self):
        return f"{self.author}: {self.title}"

They are then registered for the Django admin site in books/admin.py, like so:

# books/admin.py
class BookInline(admin.TabularInline):
    model = Book
    extra = 0


class AuthorAdmin(admin.ModelAdmin):
    list_display = ["full_name", "birth_date"]
    inlines = [BookInline]


class BookAdmin(admin.ModelAdmin):
    list_display = ["title", "author", "published_at"]


admin.site.register(Author, AuthorAdmin)
admin.site.register(Book, BookAdmin)

Notice that AuthorAdmin uses BookInline to display the author's books within the author's admin page.

Views

The web app provides the following endpoints:

/books/ returns the list of books
/books/<book_id>/ returns a specific book
/books/by-authors/ returns a list of books grouped by authors
/books/authors/ returns the list of authors
/books/authors/<author_id>/ returns a specific author

The links above are clickable if you have the development web server running.

And they're defined in books/views.py like so:

# books/views.py
def book_list_view(request):
    books = Book.objects.all()
    return JsonResponse(
        {
            "count": books.count(),
            "results": [book.to_dict() for book in books],
        }
    )


def book_details_view(request, book_id):
    try:
        book = Book.objects.get(id=book_id)
        return JsonResponse(book.to_dict())
    except Book.DoesNotExist:
        return JsonResponse({"error": "Book not found"}, status=404)


def book_by_author_list_view(request):
    try:
        authors = Author.objects.all()
        return JsonResponse(
            {
                "count": authors.count(),
                "results": [
                    {
                        "author": author.to_dict(),
                        "books": [book.to_dict() for book in author.books.all()],
                    }
                    for author in authors
                ],
            }
        )
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)


def author_list_view(request):
    authors = Author.objects.all()
    return JsonResponse(
        {
            "count": authors.count(),
            "results": [author.to_dict() for author in authors],
        }
    )


def author_details_view(request, author_id):
    try:
        author = Author.objects.get(id=author_id)
        return JsonResponse(author.to_dict())
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)

Great, you now know how the web app works!

In the next section, we'll benchmark our app to detect N+1 queries with AppSignal and then modify the code to eliminate them.

Detect N+1 Queries in Your Django App with AppSignal

Detecting performance issues with AppSignal is easy. All you have to do is use/test the app as you normally would (for example, perform end-user testing by visiting all the endpoints and validating the responses).

When an endpoint is hit, AppSignal will create a performance report for it and group all related visits together. Each visit will be recorded as a sample in the endpoint's report.

Detect N+1 Queries in Views

Firstly, visit all your app's endpoints to generate the performance reports:

Next, let's use the AppSignal dashboard to analyze slow endpoints.

Example 1: One-To-One Relationship (`select_related()`)

Navigate to your AppSignal app and select Performance > Issue list on the sidebar. Then click Mean to sort the issues by descending mean response time.

Click on the slowest endpoint (books/) to view its details.

Looking at the latest sample, we can see that this endpoint returns a response in 1090 milliseconds. The group breakdown shows that SQLite takes 651 milliseconds while Django takes 439.

This indicates a problem because an endpoint as simple as this shouldn't take as long.

To get more details on what happened, select Samples in the sidebar and then the latest sample.

Scroll down to the Event Timeline to see what SQL queries got executed.

Hovering over the query.sql text displays the actual SQL query.

More than 1000 queries were executed:

SELECT * FROM "books_book";
SELECT * FROM "books_author" WHERE "books_author"."id" = 3; -- 3= 1st book's author id
SELECT * FROM "books_author" WHERE "books_author"."id" = 6; -- 6= 2nd book's author id
...
SELECT * FROM "books_author" WHERE "books_author"."id" = n; -- n= n-th book's author id

These are a clear sign of N+1 queries. The first query fetched a book (1), and each subsequent query fetched the book's author's details (N).

To fix it, navigate to books/views.py and modify book_list_view() like so:

# books/views.py
def book_list_view(request):
    books = Book.objects.all().select_related("author")  # modified
    return JsonResponse(
        {
            "count": books.count(),
            "results": [book.to_dict() for book in books],
        }
    )

By utilizing Django's select_related() method, we select the additional related object data (i.e., author) in the initial query. The ORM will now leverage a SQL join, and the final query will look something like this:

SELECT * FROM "books_book"
    INNER JOIN "books_author" ON ("books_book"."author_id" = "books_author"."id")

Wait for the development server to restart and retest the affected endpoint.

After benchmarking again, the response time goes from 1090 to 45, and the number of queries lowers from 1024 to 2. This is a 24x and 512x improvement, respectively.

Example 2: Many-To-One Relationship (`prefetch_related()`)

Next, let's look at the second slowest endpoint (books/by-authors/).

Use the dashboard as we did in the previous step to inspect the endpoint's SQL queries. You'll notice a similar but less severe N+1 pattern with this endpoint.

This endpoint's performance is less severe because Django is smart enough to cache the frequently executed SQL queries, i.e., repeatedly fetching the author of a book. Check out the official docs to learn more about Django caching.

Let's utilize prefetch_related() in books/views.py to speed up the endpoint:

# books/views.py
def book_by_author_list_view(request):
    try:
        authors = Author.objects.all().prefetch_related("books")  # modified
        return JsonResponse(
            {
                "count": authors.count(),
                "results": [
                    {
                        "author": author.to_dict(),
                        "books": [book.to_dict() for book in author.books.all()],
                    }
                    for author in authors
                ],
            }
        )
    except Author.DoesNotExist:
        return JsonResponse({"error": "Author not found"}, status=404)

In the previous section, we used the select_related() method to handle a one-to-one relationship (each book has a single author). However, in this case, we're handling a one-to-many relationship (an author can have multiple books), so we must use prefetch_related().

The difference between these two methods is that select_related() works on the SQL level, while prefetch_related() optimizes on the Python level. The latter method can also be used for many-to-many relationships.

For more information, check out Django's official docs on prefetch_related().

After benchmarking, the response time goes from 90 to 44 milliseconds, and the number of queries lowers from 32 to 4.

Detect N+1 Queries in Django Admin

Discovering N+1 queries in the Django admin site works similarly.

First, log in to your admin site and generate performance reports (for example, create a few authors or books, update, and delete them).

Next, navigate to your AppSignal app dashboard, this time filtering the issues by admin:

In my case, the two slowest endpoints are:

/admin/login
/admin/books/author/<object_id>

We can't do much about /admin/login, since it's entirely handled by Django, so let's focus on the second slowest endpoint. Inspecting it will reveal an N+1 query problem. The author is fetched separately for each book.

To fix this, override get_queryset() in BookInline to fetch author details in the initial query:

# books/admin.py
class BookInline(admin.TabularInline):
    model = Book
    extra = 0

    def get_queryset(self, request):
        queryset = super().get_queryset(request)
        return queryset.select_related("author")

Benchmark once again and verify that the number of queries has decreased.

Wrapping Up

In this post, we've discussed detecting and fixing N+1 queries in Django using AppSignal.

Leveraging what you've learned here can help you significantly speed up your Django web apps.

The two most essential methods to keep in mind are select_related() and prefetch_related(). The first is used for one-to-one relationships, and the second is for one-to-many and many-to-many relationships.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

Rack for Ruby: Socket Hijacking

Ayush Newatia — Wed, 04 Dec 2024 14:00:00 +0000

In the first part of this series, we set up a basic Rack app, learned how to process a request and send a response.

In this post, we'll take over connections from Rack and hold persistent connections to enable pathways such as WebSockets.

First, though, let's look at how an HTTP connection actually works.

HTTP Connections

As this diagram shows, a TCP socket is opened, and a request is sent to a server. The server responds and closes the connection. All communication is in plain text.

Using a technique called socket hijacking, we can take control of a socket from Rack when a request comes in. Rack offers two techniques for socket hijacking:

Partial hijack: Rack sends the HTTP response headers and hands over the connection to the application.
Full hijack: Rack simply hands over the connection to the client without writing anything to the socket.

Partial Hijacking

This is how you do a partial hijack:

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain", "rack.hijack" => body }, []]
  end
end

rack.hijack is a Rack header, set in the same Hash as the HTTP response headers. Rack will look for such headers and process them as per the specification, instead of writing them to the HTTP response.

Run the above app and curl to it. You'll see that it writes the time at one-second intervals.

$ curl -i localhost:9292

Full Hijacking

This is how you'd do a full hijack:

class App
  def call(env)
    headers = [
      "HTTP/1.1 200 OK",
      "Content-Type: text/plain"
    ]

    stream = env["rack.hijack"].call
    stream.write(headers.map { |header| header + "\r\n" }.join)
    stream.write("\r\n")
    stream.flush

    begin
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [-1, {}, []]
  end
end

In this case, we call the proc passed to us using the rack.hijack key, instead of setting one ourselves in the response. This gives us complete control over the socket. At the end, we return an array with the status -1 only because Rack expects an array to be returned. The contents of this array are ignored since we've taken over the socket.

This is a bad practice, rife with gotchas and weird behavior. Don't do it. Samuel Williams, who is a maintainer of Rack, recommends against it as well.

Streaming Bodies in Rack for Ruby

While full hijacking is a terrible idea, partial hijacking is a useful tool. But it still feels hacky, so Rack 3 formally adopted that approach into the spec by introducing the concept of streaming bodies.

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Here we provide a block as the response body rather than an array. Rack keeps the connection open until the block finishes executing.

There's a huge gotcha here when using Puma. Puma is a multi-threaded server that assigns a thread to each incoming request. We're taking over the socket from Rack, but we're still tying up a Puma thread as long as the connection is open.

Puma concurrency can be configured, but threads are limited, and tying one up for long periods is not a good idea. Let's see this in action first.

$ bundle exec puma -w 1 -t 1:1

In two separate terminal windows, run the following command at the same time:

$ curl localhost:9292

One request is immediately served, but the other is held until the first one completes. This is because we started Puma with a single worker and single thread, meaning it can only serve a single request at a time.

We can get around this by creating our own thread.

class App
  def call(env)
    body = proc do |stream|
      Thread.new do
        5.times do
          stream.write "#{Time.now}\n\n"
          sleep 1
        end
      ensure
        stream.close
      end
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Now if you try the above experiment again, you'll see both curl requests are served concurrently because they don't tie up a Puma thread.

Once again, I must warn against this approach, unless you know what you're doing. These demonstrations are largely academic, as systems programming is a deep and complex topic.

Falcon Web Server

Since the threading problem is specific to the Puma web server, let's look at another option: Falcon. This is a new, highly concurrent Rack-compliant web server built on the async gem. It uses Ruby Fibers instead of Threads, which are cheaper to create and have much lower overhead.

The async gem hooks into all Ruby I/O and other waiting operations, such as sleep, and uses these to switch between different Fibers (ensuring a program is never held up doing nothing).

Revert your app to the previous version where we're not spawning a new thread:

class App
  def call(env)
    body = proc do |stream|
      5.times do
        stream.write "#{Time.now}\n\n"
        sleep 1
      end
    ensure
      stream.close
    end

    [200, { "content-type" => "text/plain" }, body]
  end
end

Then remove Puma and install Falcon.

$ bundle remove puma
$ bundle add falcon

Run the Falcon server. We need to explicitly bind it because it only serves https traffic by default.

$ bundle exec falcon serve -n 1 -b http://localhost:9292

The server only uses a single thread, which you can confirm with the command below. You'll need to grab your specific pid from Falcon's logs.

$ top -pid <pid> -stats pid,th

The thread count printed by the above command will be 2 because the MRI uses a thread internally.

Try the earlier experiment again and run two curl requests simultaneously.

$ curl localhost:9292

You'll see they're both served at the same time, thanks to Ruby Fibers!

Falcon is relatively new. Ruby Fibers were only introduced in Ruby 3.0. Since Falcon is Rack-compliant, it can be used with Rails too, but the docs recommend using it with v7.1 or newer only. As such, it's a bit risky to use Falcon in production but it's a very exciting development in the Ruby world, in my opinion. I can't wait to see its progress in the next few years.

We've now learned how to create persistent connections in Rack and how to run them without blocking other requests, but the use cases so far have been academic and contrived. In the next and final part of this series, we'll examine how we can use this technique in a practical way.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Avoiding False Positives in Node.js Tests

Greg Gorlen — Wed, 04 Dec 2024 11:30:34 +0000

When running tests, it's a great feeling to see dozens of green check marks indicating that a test suite is passing. It's especially gratifying after tackling a tricky bug or slogging through a tough feature. But those passing tests may be giving you a false sense of security.

Often, bugs lurk in passing tests, undermining trust in the test suite and your application. Such tests can cause more harm than good, giving you a hearty pat on the back while hiding broken functionality. It can take months to weed out a false positive test, and that might be only after a customer complaint.

This post examines several common false positive patterns that can crop up in test suites. Taken out of context, the examples may appear obvious, but they find their way into complex, real-world tests all the same.

Prerequisites

I will assume readers are familiar with ES6 JavaScript syntax and have written at least a handful of tests in NodeJS.

Although I don't intend to be technology-specific, inevitably some of the pitfalls are specific to quirks in the JavaScript ecosystem (particularly its UI testing libraries). I expect readers will be unfamiliar with some (or most) of the libraries featured in the post, but I hope the underlying principles will be relevant.

This post is current as of Node 22.11.1, jest 29.7.0, mocha 10.3.0, chai 5.1.0, chai-http 4.3.0, supertest 6.3.3, eslint 8.57.0, @playwright/test 1.42.1, and @testing-library/react 13.4.0.

Let's begin.

Common False Positive Test Patterns in Node.js

First, we'll look at some common patterns when it comes to false positive test results.

Using `equal()` Rather Than `strictEqual()`

As a warmup, check out this Mocha test with Chai assertions:

import { assert } from "chai";

describe("strict equality", () => {
  test("1 equals '1'", () => {
    assert.equal(1, "1");
  });

  test("0 equals false", () => {
    assert.equal(0, false);
  });

  test("null equals undefined", () => {
    assert.equal(null, undefined);
  });
});

Here's the output:

$ npx mocha strict-equality-bad.test.js


  strict equality
    ✔ 1 equals '1'
    ✔ 0 equals false
    ✔ null equals undefined


  3 passing (2ms)

This classically illustrates a surprising JS type coercion, courtesy of the loose equality operator ==. Replacing assert.equal() with assert.strictEqual() (akin to ===) gives the more desirable result — these values being unequal (some output omitted for brevity):

$ npx mocha strict-equality-good.test.js


  strict equality
    1) 1 equals '1'
    2) 0 equals false
    3) null equals undefined


  0 passing (3ms)
  3 failing

  1) strict equality
       1 equals '1':
     AssertionError: expected 1 to equal '1'
  2) strict equality
       0 equals false:
     AssertionError: expected +0 to equal false
  3) strict equality
       null equals undefined:
     AssertionError: expected null to equal undefined

The following assertions also fail, as expected:

// Jest:
expect(1).toBe("1");
expect(1).toEqual("1");

// Chai:
expect(1).to.eq("1");
expect(1).to.equal("1");

The theme here (which will be a recurring theme in this post) is that matchers don't always behave as expected.

These mistakes can be detected by experimenting with assertions to trigger failures. For example, consider the assertion:

assert.equal(add(-3, 3)).to.be(0);

Try plugging in temporary values like .to.be(-1), .to.be(undefined), and .to.be(false) to ensure the assertion fails. If they don't, the assertion is too weak and might be hiding a bug. Failures are good!

Test-driven development can help to avoid overly-permissive assertions. Tests that start out by failing before passing are more likely to function as intended than ones that pass from the outset, written against already implemented code.

Using Overly General Assertions

The next example looks at the dangers of broad matchers such as .toBeTruthy() and .toBeFalsy(), which are present in most assertion libraries. The following test has a serious bug, but passes:

describe("parser", () => {
  it("should parse `text` without throwing", () => {
    const parser = new Parser(text);
    expect(parser.parse).toBeTruthy();
  });
});

The author wants to assert that the return value of the parse() function is truthy; for example, that it returns an object. But the test author forgot to call the function, so the test is only asserting that the parse property exists!

Using .toBeDefined() and .toBeInstanceOf(Object) would fail in the same way. Variables and properties are typically defined and are often objects, so these assertions are too weak. Frequent use of loose assertions can indicate a code smell in the test or in the application under test. Functions should return values with predictable types, and assertions should be strict and specific to these types.

General assertions tend to read poorly and emit vague failure messages. Rewrite assertions like expect(meaningOfLife() === 42).toBe(true) to expect(meaningOfLife()).toBe(42).

See this Playwright question on Stack Overflow for an example of a false positive .toBeTruthy() assertion in the wild.

Using Shallow Equality for Deep Comparisons

Similarly, false positives can occur when comparing objects:

import { assert, expect } from "chai";

describe("deep equality", () => {
  test("[assert] two objects with same contents not equal", () => {
    assert.notStrictEqual({ foo: 42 }, { foo: 42 }); // incorrect, passes
    assert.notDeepEqual({ foo: 42 }, { foo: 42 }); // correct, fails
  });

  test("[expect] two objects with same contents not equal", () => {
    expect({ foo: 42 }).not.to.eq({ foo: 42 }); // incorrect, passes
    expect({ foo: 42 }).not.to.deep.eq({ foo: 42 }); // correct, fails
  });
});

The assertions labeled "incorrect" test identity rather than deep value equality.

Note that .not is a contributing factor to the problem. Avoiding .not whenever possible can improve readability, as it often does in boolean logic in application code. Positive assertions tend to match a narrower set of values and are therefore more meaningful.

Misunderstanding Assertion Behavior

I've discussed some of the gotchas when using overly broad assertions, but fine-grained assertions can hide subtle, surprising behavior as well.

Example: Playwright

Setting aside unit testing for a moment, the browser automation library Playwright offers a Jest-style matcher called .toBeHidden(). Playwright's documentation describes this matcher as follows (emphasis mine):

Ensures that Locator either does not resolve to any DOM node, or resolves to a non-visible one.

The first part of this sentence is surprising. .toBeHidden() is a broader assertion than its name suggests. If an application never renders an element, .toBeHidden() still passes, even if the author's intention was to ensure the element exists, but in a hidden state.

Example: React Testing Library

React Testing Library's getBy queries implicitly assert by throwing an error when an element isn't located, and are often used without expect.

However, it can be easy to forget that queryBy variants return null rather than throwing, and can't be used as implicit assertions like their getBy complements.

The lesson is to read the documentation for your assertions carefully. Names can be misleading. Always force assertions to fail to make sure they're working as advertised and intended.

Forgetting to Call a Matcher

Here's an embarrassing mistake I made migrating a Mocha/chai-http test to Jest/supertest.

expect(response).to.be.json functions as expected in Mocha, but becomes a no-op in Jest when naively updated to expect(response).toBe.json. The property .json is undefined, and even if it were defined, it'd be a no-op without parentheses to call the function. The same mistake can appear in assertions like expect(someValue).toBeTrue, which looks sensible from a linguistic perspective, but is do-nothing code.

Jest offers expect.assertions(number), which holds you accountable to calling a certain number of assertions in a test. This adds a bit more protection against forgetting to call matchers and asynchronous assertions that run after the test ends. Linting can also identify do-nothing expressions and unused variables.

Misusing Mocks

Writing mocks can be time-consuming so you can be tempted to cut corners. Poorly written test mocks can hide bugs in application code. Consider the following Sequelize database query:

const user = await User.findOne({ where: { id } });

If User.findOne was mocked to ignore its argument and unconditionally return {id: 1, name: "Alan"}, the test would still pass, even if the function call argument was incorrect:

import User from "./user";

jest.mock("./user", () => ({ findOne: jest.fn() }));

describe("User", () => {
  it("should create and retrieve a user", async () => {
    User.findOne.mockResolvedValueOnce({
      id: 1,
      name: "Alan",
      createdAt: new Date(),
      updatedAt: new Date(),
    });

    // a bad call made by the code under test
    const user = await User.findOne({ nonsenseArg: 42 });

    // passes, even though the call to findOne is broken!
    expect(user).toEqual(
      expect.objectContaining({
        id: 1,
        name: "Alan",
        createdAt: expect.any(Date),
        updatedAt: expect.any(Date),
      })
    );
  });
});

A solution is to assert that a particular argument was passed to the User.findOne() mock using expect(User.findOne).toHaveBeenCalledWith({where: 1}). Although this helps to avoid a false positive, it can make refactoring and writing tests frustrating.

Similarly, in the early days of React, the Enzyme testing library supported invasive mocks of implementation details, like the setState() hook. Such mocks can easily be misused to strongarm the suite into 100% coverage, failing to test the component's actual behavior by injecting values that cause the component to behave differently than it would at run time.

Using Regex Matching Incorrectly

Testing libraries like Playwright, React Testing Library, and Jest offer many matchers which accept regexes. It's easy to forget that /Hello./g matches "Hello world" and "Hello!" because . is a regex special character, not a period, and the regex doesn't have anchors. Perhaps /^Hello\.$/ was intended. Using a regex when a plain string match would suffice is a testing code smell.

Even when using plain strings, inadvertently using substring matches when exact matches were intended can cause tests to pass when they shouldn't. Luckily, modern UI testing frameworks like Playwright and React Testing Library offer strict assertions and queries by default, failing with a clear error if multiple elements match an overly broad query.

Copy-Paste Errors

Testing mistakes often come down to silly copy-paste errors and typos. These are particularly common in test suites with long chains of similar assertions.

For example, when testing a form with a few buttons, a tester might create the following suite:

import { expect, test } from "@playwright/test";

// simple page for demonstration
const html = `<!DOCTYPE html><html><body>
<form>
  <input type="submit" value="Submit">
  <input type="reset" value="Reset">
  <button type="button">Cancel</button>
</form>
</body></html>`;

test.describe("form", () => {
  test.beforeEach(({ page }) => page.setContent(html));

  test("submit button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Submit" })).toBeVisible();
  });

  test("cancel button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Cancel" })).toBeVisible();
  });

  test("reset button exists", async ({ page }) => {
    await expect(page.getByRole("button", { name: "Submit" })).toBeVisible();
  });
});

Here, the author forgot to adjust the content of the third test block after copy-pasting the first block twice. The test name was updated, so the output misleadingly makes it seem like the 'reset' button is being tested. Removing the <input type="reset" value="Reset"> element from the page doesn't cause the test case to fail as it should.

In the era of GPT-generated testing code, a hallucination like this is easy to overlook.

On more complex tests that trigger JS code, a coverage tool can help catch these mistakes. Static analysis tools available as IDE plugins can also detect duplicate code.

Using Incorrect Properties in Configuration Objects

Configuration objects are a readable way to pass arguments to a function, making up for the fact that JS doesn't support named arguments. Modern UI testing libraries use them liberally.

However, if you're using plain JS rather than TypeScript, it's easy to make a call to a function with a typo or incorrect property name in the configuration object:

screen.getByRole("heading", { value: "Hello" }); // incorrect!

This React Testing Library query should use the name: key rather than value:. JS will silently ignore the argument, possibly retrieving an unexpected element or failing to implicitly assert the header's user-visible text.

This scenario can also creep up in global configuration files and strings in general. In addition to type checking and linting, the usual strategies for false positive avoidance described throughout this post apply.

Misusing Snapshot Tests

Snapshot tests let you diff a live UI component (for example, a React or Vue component) against a version-controlled, stringified snapshot of the component. If the strings match, the test passes.

Be careful, though: snapshot testing can lead to false positives if the reference snapshot is updated to match a broken component. Jest makes it effortless to update all failing snapshots in one command, even when some shouldn't be updated. Hastily updating snapshots without careful examination can bake false positives into tests.

Unlike traditional assertions, it's not as apparent from glancing at a large snapshot whether it's accurate or not, allowing bad snapshots to be handwaved through a code review. In some cases, the content of external snapshot files may not be examined at all.

Having a strong code review culture that examines tests as critically and thoroughly as application code is another tool to mitigate false positives.

Wrapping Up

After reading this article, I hope you'll walk away with a more critical eye on your test suites. This is far from an exhaustive list, so be on the lookout for other pitfalls, especially when you're onboarding a codebase or using an unfamiliar testing library.

The testing errors we've examined are a subclass of logic errors in general applications. Working cautiously to guard against and surface silent errors helps ensure the code you're working with stays correct.

Happy debugging!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

Streamlined Contract Testing in Node.js: A Simple and Achievable Approach

Ashley Davis — Wed, 27 Nov 2024 16:01:50 +0000

Do you want the benefits of contract testing with much less effort? Are you convinced of the benefits of contract testing but think it’s just too difficult to roll out across your organization?

You might worry that implementing Pact in your organization requires challenging changes to culture and process.

In this article, I’ll show you a drastically simplified approach to contract testing that a single developer can bring online. You'll get many of the benefits of contract testing for much less work. You'll build a platform that will help convince your team of how much contract testing can help prevent problems going out to production.

If you've ever tried to read the Pact documentation, you might think contract testing is a complicated team-based affair that requires new infrastructure. Let me show you that it doesn’t have to be this way.

Pre-requisites

The example code for this article is available on GitHub.

You can clone the code repository using Git or download the zip file and unpack it.

You’ll need Node.js installed to run the example code. The code repository contains three working examples that we’ll go through in turn. Follow along and try out each example for yourself.

Testing a REST API with a Simplified Contract

For our first example of simplified contract testing, let’s run some tests against an existing REST API. We’ll use the JSON Placeholder REST API.

JSON Placeholder is a fake REST API that includes endpoints for creating, updating, reading, and deleting “blog posts”. In Figure 1, you can see the JSON response for getting all blog posts. Load this URL in your web browser to see it for yourself:

Figure 1: The JSON response from JSON Placeholder getting all blog posts, viewed in Chrome.

We'll use the Jest testing framework to run our contract tests. The Axios code library will make HTTP requests to the JSON Placeholder REST API, then we’ll check that the responses conform to a contract defined in a JSON schema. The setup for our first example is illustrated in Figure 2:

Figure 2: Using Jest to make HTTP requests and checking the responses against a JSON schema.

To try out the contract tests, you’ll need a local copy of the code repository and Node.js installed. Change into the rest-api subdirectory for the first example project:

cd simplified-contract-testing/rest-api

Then install project dependencies:

npm install

Now run the tests:

npm test

After a few moments, you should see the output from a handful of successful tests.

Now let’s see how the code works. Listing 1 shows a pared-down version of the YAML test spec that makes a HTTP request to the /post endpoint and checks the response against the GetPostsResponse schema. The code repo has an expanded version that contains tests for multiple HTTP endpoints.

schema:
  definitions:
    # Schema for a blog post:
    Post:
      title: Represents a blog post.
      type: object
      required:
        - userId
        - id
        - title
        - body
      properties:
        userId:
          type: number
        id:
          type: number
        title:
          type: string
        body:
          type: string
    # Schema for the REST API response:
    GetPostsResponse:
      title: GET /post response
      type: array
      items:
        # Reference to the Post schema.
        $ref: "#/schema/definitions/Post"
# List of tests specs.
specs:
  # Tests the REST API that gets the list of blog posts.
  - title: Gets all blog posts
    description: Gets all blog posts from the REST API.
    method: get
    url: /posts
    expected:
      status: 200
      headers:
        Content-Type: application/json; charset=utf-8
      body:
        # Reference to the schema for the REST API response.
        $ref: "#/schema/definitions/GetPostsResponse"

Listing 1: A Test Spec That Makes an HTTP Request to the `/posts` Endpoint

How do we convert the test spec to a series of automated tests? For that, we'll use Jest’s test.each function. We can use a very cool technique to generate automated tests from our data.

In this case, we generate tests from our test spec loaded from the YAML file. You can see in Listing 2 that the code required to achieve this is minimal and not complicated. Listing 2 is slightly abbreviated, but not by much if you compare it to the complete code file.

const axios = require("axios");
const yaml = require("yaml");
const fs = require("fs");
const { resolveRefs } = require("./lib/resolver");

const { matchers } = require("jest-json-schema");
expect.extend(matchers);

// The REST API we are testing:
const baseURL = `https://jsonplaceholder.typicode.com`;

describe("Contract tests", () => {
  // Loads the test spec:
  const testSpec = resolveRefs(
    yaml.parse(fs.readFileSync(`${__dirname}/test-spec.yaml`, "utf-8"))
  );

  // Generates a test for each contract test in the spec:
  test.each(testSpec.specs)(`$title`, async (spec) => {
    // Makes the HTTP request:
    const response = await axios({
      method: spec.method,
      url: spec.url,
      baseURL,
      data: spec.body,
      validateStatus: () => true, // All status codes are ok.
    });

    // Matches headers:
    if (spec.expected.headers) {
      for ([headerName, expectedValue] of Object.entries(
        spec.expected.headers
      )) {
        const actualValue = response.headers[headerName.toLowerCase()];
        expect(actualValue).toEqual(expectedValue);
      }
    }

    // Matches response body against the expected schema:
    if (spec.expected.body) {
      expect(response.data).toMatchSchema(spec.expected.body);
    }
  });
});

Listing 2: Generating a Series of Jest Tests from Our Data

We have already seen how simple it can be to make data-driven contract tests against an existing REST API. Now let’s run our contract tests against a more realistic REST API backed by a database.

Mocking Your Database for Fast Contract Tests

To try out this second example for yourself, change into the mocked-mongodb subdirectory, install dependencies, and then run the tests:

cd simplified-contract-testing/mocked-mongodb
npm install
npm test

The problem with testing against real infrastructure (like a real database) is that it makes our automated tests run very slowly. We can speed this up massively by mocking our database. By that, I mean replacing the real database with a fake version we can load up with pre-canned data fixtures to run our automated tests against. So while the real version of the service is configured to use a real database, the version we load for automated testing is configured to use the fake database instead. If you are interested, you can see the full code for the REST API in the code repo.

You can see how this looks in Figure 3. Now we are running a REST API locally, and we have it talking to a mock database where we can control the data fixtures on a test-by-test basis:

Figure 3: Mocking the database and controlling data fixtures on a test-by-test basis.

It turns out to be incredibly easy to mock whole modules when using Jest. Any file we place in the __mocks__ subdirectory can replace a real code module while automated tests run. So we place the file mongodb.js in the __mocks__ subdirectory, and Jest automatically replaces require(“mongodb”) with our mock version of the code. You can see our mock version of MongoDB in Listing 3. Note how it exports the function __setData__, which we can call from our tests to control the data fixture loaded for each test.

//
// A mock version of the MongoDB library.
//

let data = {}; // Data fixtures are plugged in here.

// --snip--

class MongoCollection {
  constructor(collectionName) {
    this.collectionName = collectionName;
  }

  async findOne({ _id }) {
    const collectionData = data[this.collectionName] || [];
    return collectionData.find((el) => el._id.__value === _id.__value);
  }

  find() {
    // A mock version of Mongodb's find function.
    return {
      async toArray() {
        // Returns the data fixture:
        return data[this.collectionName] || [];
      },
    };
  }

  async insertOne() {
    return {
      insertedId: new ObjectId("newly-inserted"),
    };
  }
}

class MongoDatabase {
  collection(collectionName) {
    return new MongoCollection(collectionName);
  }
}

class MongoClient {
  async connect() {}

  db() {
    return new MongoDatabase();
  }
}

// Allows automated tests to set data fixtures.
function __setData__(_data) {
  data = _data;
}

module.exports = {
  MongoClient,
  ObjectId,
  __setData__,
};

Listing 3: The Mock Version of MongoDB

Now that we have a mock database and the ability to load data fixtures, we must give our test spec the ability to control which data fixture is loaded for each test. You can see a snippet of the updated test spec in Listing 3. Note the new fixture field that, in this case, specifies that this test should load the data fixture named many-posts.

- title: Gets all blog posts
    description: Gets all blog posts from the REST API.
    # Specifies the data fixture to load before running the test:
    fixture: many-posts
    method: get
    url: /posts
    expected:
        status: 200
        headers:
            Content-Type: application/json; charset=utf-8
        body:
            $ref: "#/schema/definitions/GetPostsResponse"

Listing 4: A Test That Loads the Data Fixture `many-posts`

What we are missing now is the code that loads the data fixture for each test, which you can see in Listing 5 — or check out the full version in the code repo. We are now running a local REST API so you can see how we start the web server before each test. Also note the call to loadFixture at the start of each test, which loads the data
fixture requested by the test.

// --snip--

describe("Contract tests", () => {
  // Loads the test spec:
  const testSpec = resolveRefs(
    yaml.parse(fs.readFileSync(`${__dirname}/test-spec.yaml`, "utf-8"))
  );

  // --snip--

  beforeEach(async () => {
    // Starts the web server on a random port before each test:
    await startServer();
  });

  afterEach(async () => {
    await closeServer();
  });

  test.each(testSpec.specs)(`$title`, async (spec) => {
    if (spec.fixture) {
      // Loads a named database fixture into the mock MongoDB for each test:
      loadFixture(spec.fixture);
    } else {
      clearFixture();
    }

    // Makes the HTTP request:
    const response = await axios({
      method: spec.method,
      url: spec.url,
      baseURL,
      data: spec.body,
      validateStatus: () => true,
    });

    // Matches status:
    if (spec.expected.status) {
      expect(response.status).toEqual(spec.expected.status);
    }

    // Matches headers:
    if (spec.expected.headers) {
      for ([headerName, expectedValue] of Object.entries(
        spec.expected.headers
      )) {
        const actualValue = response.headers[headerName.toLowerCase()];
        expect(actualValue).toEqual(expectedValue);
      }
    }

    // Matches response body against the expected schema:
    if (spec.expected.body) {
      expect(response.data).toMatchSchema(spec.expected.body);
    }
  });
});

Listing 5: Starting the Web Server and Loading Data Fixtures Before Each Test

Running contract tests against a REST API is pretty useful, but we must often also work with services that communicate via asynchronous message queues. For this third and final example, we will extend our REST API so that it can send and receive messages through a RabbitMQ instance. Try out the tests for yourself:

cd simplified-contract-testing/mocked-rabbit
npm install
npm test

Mocking Your Message Queue for Fast, Asynchronous Contract Tests

Please note that we are mocking RabbitMQ in this example, but in principle, this technique can work for any asynchronous messaging system (such as Kafka or SQS).

Let's mock the module amqplib (essentially RabbitMQ), so that we can:

Directly invoke asynchronous message handlers.
Retrieve asynchronous published messages.

You can see how this looks in Figure 4. Our Jest tests are now acting through our mock version of RabbitMQ to interact with our REST API:

Figure 4: Using a mock version of RabbitMQ to interact with our REST API during automated testing.

Our mock version of AMQPlib, which we use to replace RabbitMQ, is similar to our mock version of MongoDB. We add the file amqplib.js to the __mocks__ subdirectory and Jest automatically replaces require(“amqplib”) with the mock version you see in Listing 6. Published messages are saved in the __published__ object, and the mock AMQPlib also tracks message handlers in the __consume__ object.

const crypto = require("crypto");

// Allows the contract tests to check messages that were published to the message queue:
const __published__ = {};

// Allows the contract tests to invoke message handlers:
const __consume__ = {};

const __queue_bindings__ = {};

const mockChannel = {
  async assertExchange() {},

  async assertQueue() {
    return {
      queue: crypto.randomBytes(5).toString("hex"),
    };
  },

  async bindQueue(queueName, exchangeName) {
    __queue_bindings__[exchangeName] = queueName;
  },

  // Publishes a message to a mock queue.
  async publish(exchange, routingKey, content) {
    __published__[exchange] = JSON.parse(content.toString());
  },

  // Binds an event handler to a message queue.
  async consume(queue, handler) {
    __consume__[queue] = handler;
  },

  ack() {},
};

const mockConnection = {
  async createChannel() {
    return mockChannel;
  },
};

async function connect() {
  return mockConnection;
}

module.exports = {
  connect,
  __published__,
  __consume__,
  __queue_bindings__,
};

Listing 6: The Mock Version of AMQPlib Used to Replace RabbitMQ

With RabbitMQ mocked, our Jest tests can now invoke asynchronous message handlers directly in our code and then check any asynchronous responses that are subsequently published. Now that we can trigger HTTP requests or asynchronous messages from our tests, we need to update our test spec to specify the required type for each test. You can see an example in Listing 7. Note how the type field specifies rabbit, and the exchange field selects the RabbitMQ “exchange” we are using to trigger the asynchronous message handler we are trying to test.

At this point, we are not just checking “immediate responses” from HTTP requests, we now must also be able to check for asynchronous messages and compare them against the expected JSON schema. Note the asyncResponse section of Listing 7 and how it defines the schema for a message that is expected to be published on the payment-processed exchange.

- title: Processes a payment on new user
    description: Processes a payment on adding a new user.
    fixture: many-posts
    # The "trigger" that invokes the code to be tested:
    type: rabbit
    exchange: new-user
    body:
        userId: 1
        name: John Doe
    expected:
        # Expectations for the asynchronous response:
        asyncResponse:
            type: rabbit
            exchange: payment-processed
            body:
                $ref: "#/schema/definitions/UserPaymentProcessedMessage"

Listing 7: A Test Triggers an Async Message Handler

We have extended our contract testing system to be even more flexible:

We can trigger HTTP requests or asynchronous message handlers.
We can match HTTP responses and asynchronous responses against a schema.

This means we can test the following combinations:

Trigger an HTTP request and check that its immediate response matches expectations.
Trigger an HTTP request and check that it results in publishing an asynchronous message that matches expectations.
Trigger an async message handler and check that it results in publishing an asynchronous message that matches expectations.

You can find examples of all these combinations in the expanded test spec in the code repo.

The code that generates our contract testing suite is getting more complex, as you can see in Listing 8. Still, it’s not bad considering the number of tests and the different testing combinations that we can achieve with this relatively small piece of code. See the full code that generates the contract tests in the code repo.

// --snip--

// The "http" trigger.
async function http(spec) {
  return await axios({
    method: spec.method,
    url: spec.url,
    baseURL,
    data: spec.body,
    validateStatus: () => true,
  });
}

// The "rabbit" trigger.
async function rabbit(spec) {
  const queue = amqplib.__queue_bindings__[spec.exchange];
  if (!queue) {
    throw new Error(`No queue is bound for exchange '${spec.exchange}'`);
  }

  // Uses the mock "amqplib" to invoke the message handler:
  const consumeHandler = amqplib.__consume__[queue];
  if (!consumeHandler) {
    throw new Error(
      `No consumer found for queue '${queue}' bound to exchange '${spec.exchange}'`
    );
  }

  consumeHandler({ content: Buffer.from(JSON.stringify(spec.body)) });
}

const triggers = {
  http,
  rabbit,
};

test.each(testSpec.specs)(`$title`, async (spec) => {
  // --snip--

  // Triggers the code to be tested.
  const trigger = triggers[spec.type];
  const immediateResponse = await trigger(spec);

  // Matches the immediate response against the expected schema.
  if (spec.expected.immediateResponse) {
    if (spec.expected.immediateResponse.status) {
      expect(immediateResponse.status).toEqual(
        spec.expected.immediateResponse.status
      );
    }

    if (spec.expected.immediateResponse.headers) {
      for ([headerName, expectedValue] of Object.entries(
        spec.expected.immediateResponse.headers
      )) {
        const actualValue = immediateResponse.headers[headerName.toLowerCase()];
        expect(actualValue).toEqual(expectedValue);
      }
    }

    if (spec.expected.immediateResponse.body) {
      expect(immediateResponse.data).toMatchSchema(
        spec.expected.immediateResponse.body
      );
    }
  }

  // Matches the async response against the expected schema.
  if (spec.expected.asyncResponse) {
    const expectedResponse = spec.expected.asyncResponse;
    if (!expectedResponse.exchange) {
      throw new Error(
        `An exchange is required for async responses on spec '${spec.title}'`
      );
    }
    const actualAsyncResponsePayload =
      amqplib.__published__[expectedResponse.exchange];
    if (!actualAsyncResponsePayload) {
      throw new Error(
        `Expected async response to be published on exchange '${expectedResponse.exchange}'.`
      );
    }

    if (expectedResponse.body) {
      expect(actualAsyncResponsePayload).toMatchSchema(expectedResponse.body);
    }
  }
});

Listing 8: Our Contract Testing Code Now Handles HTTP and Asynchronous Requests and Responses

Our adventure in contract testing isn’t complete until we fully automate our tests.

Adding Contract Tests to Your CI/CD Pipeline

We can automate our tests in any CI/CD provider, as long as we can clone our code, install Node.js, and run the following commands in our project:

npm ci
npm test

Given that our external dependencies are mocked (MongoDB and RabbitMQ in these examples), that’s all that we need to get our simplified contract tests running in our CI/CD pipeline.

Listing 9 shows a CI pipeline for GitHub Actions. You can see the full, multi-project version in the code repo.

name: Automated contract tests

on:
  push:
    branches: ["main"]
  pull_request:
    branches: ["main"]

jobs:
  contract-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v3 # Installs Node.js.
        with:
          node-version: 20

      - run: npm ci # Installs project dependencies.
      - run: npm test # Runs the automated contract tests.

Listing 9: The GitHub Actions Workflow Runs Simplified Contract Tests for a Project

Back in the beginning of this article, I showed how we can run contract tests against an existing REST API. In the same way, we can run our contract tests against any REST API or service that we can access.

Because we control the code making the HTTP request, we can augment the code to include authentication details for the production deployment of our REST API and then run the contract tests against it. If you are contract testing against a production service via asynchronous messaging, then you'll need access (e.g., via an SSH tunnel or Kubernetes port forwarding) to your message queue.

Obviously, we must be very careful if we run contract tests against a production system, especially when tests can make changes to that system. In the full version of my example, you may have noticed that one of the tests makes an HTTP POST request to create a blog post in the REST API. Tests like this are probably best run against a QA or staging deployment, but if you really do want to run them against production, you might want a test account to easily flush out after running tests.

Bringing It Together: How I Applied Contract Testing

In my team, we used the simplified approach to contract testing covered in this post without making the team adopt any arduous processes. We integrated it into our usual automated testing process (in this example, running npm test in our CI pipeline), and we didn’t need any new infrastructure (like a contract broker) to bring all this together.

Mocking, while not specifically related to contract testing, proved essential in making it convenient and fast to run these tests. Contract testing is more on the level of integration testing in terms of how much code it can cover for the least amount of effort.

But because we mock certain external services (like the database and message queue), our contract tests can be much closer to unit tests in terms of performance. This makes it easy and fast to run our contract tests locally for frequent feedback while we make code changes to a REST API or microservice.

Wrapping Up

In this article, we've shown you that contract testing in Node.js can be as simple as defining a JSON schema and then using your favorite testing framework to check responses from REST APIs and asynchronous messaging.

Using this simplified approach to contract testing, you can get started very quickly and gain practical results that will help convince your team that contract testing is worthwhile.

Happy testing!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to Track Errors in Oban for Elixir Using AppSignal

Aestimo K. — Tue, 26 Nov 2024 11:14:49 +0000

When developing an Elixir app, you'll often need to handle tasks in a way that does not interrupt the normal user request-response cycle.

Tasks like sending emails are great examples of jobs that should be delegated to a capable background job processing service. In the Elixir ecosystem, Oban is one such background job processing library.

In this article, we'll learn what Oban is, how it works, and how to instrument it using AppSignal.

Prerequisites

To follow along with this tutorial, I recommend you have the following:

An AppSignal account - you can sign up for a free trial.
An Elixir app. In my case, I am using a Phoenix LiveView app with a landing page where a user can subscribe to an email newsletter with a couple of background jobs to schedule and send out emails. You can go ahead and fork the app.
The AppSignal package added and configured for your app.
Elixir, Phoenix, and PostgreSQL installed on your local development environment.
Some basic experience of working with Elixir and Phoenix.

And with that, we can get started.

Introducing Oban

Oban is a stable and high-performing background job processing library for Elixir applications. Unlike other job processing systems that might require their own infrastructure to run background jobs, such as RabbitMQ and Sidekiq (which needs a key-value store like Redis), Oban uses your app's existing PostgreSQL or SQLite database.

With Oban, you can define, enqueue, and process jobs in the background. Another important thing to note is that Oban jobs are persistent, which means that even if something unexpected happens in the server environment, jobs will be retried for you.

Installing and Configuring Oban in Elixir

First, add Oban to your app's mix.exs like so:

# mix.exs

defmodule EmailSubscriptionApp.MixProject do
  ...

  defp deps do
    [
      ...
      {:oban, "~> 2.17"}
    ]
  end

  ...
end

Then run mix deps.get in the terminal to install the package and its dependencies.

Before moving on, it's important to cover the database your app is using since it will determine how you configure Oban.

If you've already configured PostgreSQL for your app, the Postgres package is likely already installed, and you'll not have to add it manually. However, if your app uses SQLite, you'll have to add the EctoSQLite3 package to mix.exs and configure it accordingly. In this article, the featured app uses PostgreSQL for the database, and Oban is installed to run on top of that.

Next, generate a migration file that will create all the database tables Oban needs to run and keep tabs on job queues.

mix ecto.gen.migration add_oban_jobs_table

Then modify the generated file to look like this:

# priv/repo/migrations/timestamp_add_oban_jobs_table.exs

defmodule EmailSubscriptionApp.Repo.Migrations.AddObanJobsTable do
  use Ecto.Migration

  def up do
    Oban.Migration.up(version: 12)
  end

  def down do
    Oban.Migration.down(version: 1)
  end
end

Run the migration with mix ecto.migrate.

With that done, you next need to configure Oban to run with the Postgres engine by adding the following lines to config.exs:

# config/config.exs

config :email_subscription_app, Oban,
  engine: Oban.Engines.Basic,
  queues: [default: 10],
  repo: EmailSubscriptionApp.Repo

It's also recommended that you prevent Oban from running jobs in testing mode by adding the lines below to text.exs:

# config/text.exs

config :email_subscription_app, Oban, testing: :inline

Finally, we'll need to add Oban to the app's list of supervised children since they run as isolated supervision trees. To do this, add Oban to the app supervisor like so:

defmodule EmailSubscriptionApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      ...
      EmailSubscriptionApp.Repo,
      {Oban, Application.fetch_env!(:email_subscription_app, Oban)},
      ...
  end
  ...
end

Then, to finish up the installation and configuration, open up a new interactive Elixir shell with iex -S mix. Run the command Oban.config() to return an output similar to the one below:

iex(1)> Oban.config()
%Oban.Config{
  dispatch_cooldown: 5,
  engine: Oban.Engines.Basic,
  get_dynamic_repo: nil,
  insert_trigger: true,
  log: false,
  name: Oban,
  node: "...",
  notifier: {Oban.Notifiers.Postgres, []},
  peer: {Oban.Peers.Postgres, []},
  plugins: [],
  prefix: "public",
  queues: [default: [limit: 10]],
  repo: EmailSubscriptionApp.Repo,
  shutdown_grace_period: 15000,
  stage_interval: 1000,
  testing: :disabled
}

Before moving on to defining a few jobs and instrumenting them using AppSignal, let's touch on how instrumentation actually works.

Automatic Instrumentation of Oban Using AppSignal

When you added the AppSignal package to your app, Oban instrumentation was automatically added. The AppSignal package provides instrumentation for Oban workers and will also collect metrics on how background jobs are performing.

With that in mind, let's write a few jobs and see how they appear on the AppSignal dashboard.

Defining the First Background Job

Our practice Phoenix application is all about receiving a user's email in an input on the homepage, then sending that user a series of emails (starting with a welcome email that is sent immediately when the user subscribes, a follow-up email sent ten minutes later, and then a final email sent thirty minutes later).

Let's start by defining the background job that will trigger the first email.

# lib/email_subscription_app/workers/welcome_email_worker.ex

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default

  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()

    :ok
  end
end

Let's dig into this example code a bit to understand how Oban workers work. This information will prove helpful when troubleshooting job errors.

You have the module definition — in this case, the WelcomeEmailWorker — then one or more job-performing functions, usually called perform/1.

The perform/1 function receives an Oban.Job struct containing the specific job's arguments which, in our case, is the email key.

Even though we haven't shown it in this example code, Oban also allows for job scheduling using the schedule_at and schedule_in options.

And like we mentioned before, AppSignal automatically instruments Oban and displays some default dashboards, as you can see in the examples below.

Here are our Oban workers listed on AppSignal:

And details of a worker's instrumentation:

Now we've installed Oban, defined a simple background worker job, instrumented Oban with AppSignal, and visualized the output. Let's turn to when things go wrong with Oban and how we can use AppSignal to help us track errors.

Oban Errors

Although Elixir systems are well-known for their fault tolerance, errors and bugs can affect them, resulting in a poor user experience if not addressed. In this section, we'll look at some common errors that could affect your Oban workers, as well as how you can instrument errors with AppSignal and see them on your dashboard.

Many different errors could affect an Oban worker, including:

Database connection errors - Since Oban depends on accessing a PostgreSQL or SQLite 3 database, any connection issues between your app and its database will definitely affect any workers you have defined.
External service failures - For example, if there is an issue that affects emails being sent through a third-party email service provider in an email subscription app, the background jobs we've defined could fail to send emails, which would result in Oban job errors. Further, if you call third-party API services from your app, any rate-limiting efforts by the API service provider could result in connected Oban jobs failing to complete and resulting in errors.
Job timeout errors - These errors occur if a job's actual execution time exceeds the configured time.
Job queue congestion errors - These happen when there are too many jobs in the queue, which could result in delays in job execution.

Instrumenting Oban Errors Using AppSignal for Elixir

One of the major benefits of AppSignal is that most common errors are automatically instrumented for you. For example, see the code snippet below:

defmodule EmailSubscriptionApp.Workers.WelcomeEmailWorker do
  use Oban.Worker, queue: :default
  alias EmailSubscriptionApp.Emails.{EmailSender, Mailer}

  @impl Oban.Worker
  def perform(%Oban.Job{args: %{"email" => email}}) do
    email
    |> EmailSender.welcome_email()
    |> Mailer.deliver()
    |> case do
      {:ok, _} -> :ok
      {:error, reason} ->
        raise "Mailer delivery error: #{inspect(reason)}"
  end
end

If an error occurs within the perform/1 function, AppSignal will catch it and give you a dashboard view with details of the error, as you can see in the screenshot below:

Before wrapping up this article, let's take a look at the error details AppSignal shows you in the dashboard:

Error message - The actual error message associated with the raised error.
Backtrace - The backtrace gives you fine-grained context on what happens before an error occurs in your app.
Parameters - AppSignal also shows you any parameters involved when the error occurred.
Deploy - Finally, you also get information on whether the error occurred during a deployment or not.

And that's that!

Wrapping Up

In this article, we looked at the background job processing package Oban. We learned how to install it, configure background workers, simulate an error, and instrument errors using AppSignal. I recommend using this information as a foundation for using Oban and AppSignal in your Elixir apps.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Optimize Database Performance in Ruby on Rails and ActiveRecord

Daniel — Wed, 20 Nov 2024 13:00:00 +0000

In Rails, we're more likely to use SQL databases than other frameworks. Unlike NoSQL databases, which can be scaled horizontally with relative ease, SQL databases like PostgreSQL or MySQL are much less amenable to easy scaling.

As a result, our database usually becomes the primary bottleneck as our business grows. Although SQL databases are very efficient, as our growing customer base puts an increasing load on our servers, we begin scaling our instance counts, workers, etc. But we can't just make copies of our database for each new server we spin up. This makes optimizing database performance critical for any serious Rails project.

In this post, we'll explore strategies for optimizing performance to minimize the load on our database. We'll start with some more basic topics like eager loading and the N+1 query problem, database indexing, select, pluck, and immediate loading. Then, we'll move on to more involved topics like performance profiling, scaling via techniques like database sharding, and background jobs using read replicas.

Let's get going!

Getting ActiveRecord N+1s Out of the Way

No discussion of database performance involving ActiveRecord (or any ORM) is complete without addressing the infamous N+1 problem. While most readers are likely familiar with it, it's still worth revisiting, as N+1 queries can (and do) creep into our projects over time, especially as our codebase evolves and we add or update features.

The N+1 problem generally occurs when iterating through a (potentially large) group of retrieved records. Because we haven't loaded their association/s, each loop means an additional query for each associated model we access inside the loop. This can quickly spiral out of control, resulting in a staggering number of queries and causing serious consequences such as crashed pages, an exhausted database connection pool, and memory running out, ultimately grinding our site to a halt.

There are a few useful tools at your disposal to help identify and resolve N+1 problems. The first is simply looking at your server output; generally, this works pretty well, as N+1s are easy to spot. For a more assisted approach, the Bullet gem is a popular tool that automatically detects N+1s in applications and suggests ways to fix them. Another, arguably better option is prosopite, a less well-known option that generally provides better results with fewer false positives (and false negatives).

Eager loading should be used with care, however; while you're probably safe loading has_one associations, has_many can sometimes be dangerous: what happens when we've got a query of multiple joined, many-to-many tables, and eager load one or more of the many-to-many associations?

We can end up loading way too many records into memory. This can crash our app just as surely as a bad N+1. If you're at this point, and both the N+1 and eager loading approaches are causing you problems, it may be time to reevaluate the query itself. Maybe you're trying to load associations you just want to COUNT (which is something you might be able to build into your query), tighten your pagination limits, or consider offloading the query to a background job if possible (which we'll cover).

Database Performance in Rails: Considerations and Optimization

Sometimes, breaking queries down into a couple of steps can both improve performance and simplify a query. Ever find yourself trying to get a subset of records to complete a mind-bending query? In certain cases, just finding the IDs of the subset you want and then feeding them into a second, less complex query can help reduce the need for complex joins, subqueries, etc.

Have you ever fallen victim to Rails' (usually beneficial) default lazy-loading and evaluation, and wished you could load something immediately rather than the first time it's used? Sometimes we need not only the results of a query, but another aspect of it too, like the number of records we've retrieved. Our seemingly harmless code looks like this:

@users = User.where("email ILIKE ?", search)
@total_users = @users.size

Normally, this is fine, but what if we need to count users before we do anything with them (maybe we're displaying the number of users returned in our live search at the top of the page, followed by the users themselves)? We might add COUNT to our query, but then we need to retrieve the records again.

We can get around this by explicitly calling load on our query — for example:

@users = User.where("email ILIKE ?", search).load
@total_users = @users.size

This simple change can save us a query. With our users already loaded, our display of @total_users above the user list will no longer trigger an expensive COUNT followed by another query to get users.

In the other direction, we have load_async with Rails 7. This allows us to asynchronously query the database with multiple requests at once, rather than being locked to one synchronous query at a time. This can be a total game-changer if you're in a situation that can benefit from it. Be careful though, as this (probably unsurprisingly) uses Ruby threads under the hood, so comes with all the same issues you'd expect from threads. That's not to mention the fact that it (potentially) opens our database up to a greatly increased number of connections, and can exhaust our connection pool if we aren't careful.

A large part of database performance in Rails comes down to how you utilize ActiveRecord. While AR is an invaluable tool, it can also lead to performance issues if not used with care. As you're probably aware, it loads every column of every table involved in a query (SELECT *) by default, regardless of how much of that data we actually use. It's pretty easy to forget this though, especially as our queries evolve, and we can end up loading a lot of unnecessary data. It's important to consider whether or not you're really using all of the columns you're loading, or if a select (or even a pluck) statement might be a better fit for your use case.

You're probably familiar with eager loading in Rails, but one thing that's often overlooked is that Rails makes it easy to introduce optional eager-loading (and WHERE clauses, and SELECTs). If you've ever had slightly different requirements for two very similar queries, you might feel like you're left with two options: write two queries, or write covering both use cases, each retrieving more than you need. This might happen in a before_action, where you typically want the same thing for each action in a particular controller. But one controller might benefit from eager loading, more careful column selection, or a search results page which is just filtering the usual index based on some criteria.

There's another way, though:

class User
  def search_user_posts(before_date: nil, optional_selects: [], optional_eager_loads: [], exclude_inactive: false, search: false)
    optional_before_date_constraint = ["created_at < ?", before_date] if before_date
    optional_joins = {user_posts: [posts: :comments]} if search

    self.posts
        .includes(optional_eager_loads)
        .joins(:shared_joins)
        .joins(optional_joins)
        .select(:columns_shared_by_queries)
        .select(optional_selects)
        .where("your shared WHERE constraints")
        .where(optional_before_date_constraint)
  end
end

Rails is smart enough to know not to execute anything here if there's nothing passed in; where(nil) will behave as though it doesn't exist, as will includes([]), etc. This can really clean up your code and help you tailor performance to your needs.

Finally, rethinking your JOINs and WHEREs can sometimes be beneficial. Are you querying a very large dataset where using a JOIN could significantly reduce the initial result set? Try it out, and don't be afraid to run #explain on your query. It can help you determine if switching to a JOIN might work better and identify any missing indexes on critical queries.

Database Indexing

One of our most important (and nearly universally necessary) tools for database performance is indexing. As you're probably aware, indexes are special data structures (typically B-trees) that a database uses to quickly find records, improving retrieval times from O(n) to O(log(n)). However, it's important to add indexes carefully, as indexing columns unnecessarily can actually harm performance.

The trick is to identify the columns and associations that your application needs to query frequently, and then create indexes on those. This allows the database to quickly locate the relevant data without having to scan the entire table.

You can also use partial indexing. Partial indexes allow you to index a subset of a table's rows based on a specified condition. This is particularly useful when your queries frequently target only a specific portion of the data.

For example, maybe your app has a core of frequent visitors, the majority of whom have their own accounts. They make up the lion's share of your traffic, but represent a small minority of your total users. You could define a partial index on the users table like this:

class AddPartialIndexToUsersOnGuest < ActiveRecord::Migration[7.1]
  def change
    add_index :users, :guest, where: "(guest = false)" # the `where: condition` here makes this a partial index.
  end
end

Here, we've specifically targeted rows where guest is false. This optimizes the performance of any query searching for non-guest users by reducing the total rows covered by the index.

It's important to revisit your indexing profile and database schema periodically. The specific indexing needs of your application may change over time as the user base, codebase, and feature set evolve. Doing so often means you end up reviewing your indexes during comfortable periods where you have time to analyze and think about your database, rather than in a panic when an influx of traffic crashes your app.

All of this brings us to our next topic: performance profiling.

Performance Profiling for Your Ruby on Rails App

So, we've identified some major categories of performance optimization; what now? Do we go through our entire app from top to bottom, optimizing line by line? Probably not.

It's pretty unlikely that you have evenly distributed traffic across your whole site. It's even less likely that you can afford to go through your entire codebase; premature optimization may not actually be the root of all evil, but you also don't have time for it. So how do we know what to focus on?

This is where profiling tools come in. While your local server output can provide valuable performance insights into response times, memory use, and query performance, you are only one user. Your local database likely doesn't reflect the scale or patterns of your production database. Valuable as it is for spotting potential performance issues, your local environment doesn't provide the context you need to make decisions about what genuinely needs your attention.

Application Performance Monitoring tools (APMs) like AppSignal for Ruby are essential for monitoring and maintaining our site's performance. They provide comprehensive insights not just at the database layer, but across the rest of the stack, allowing us to pinpoint specific areas (pages, endpoints, and even specific queries) that are causing headaches. With a good APM tool, we can track the performance of any request over various timescales, monitor background jobs, spot memory leaks, set up custom error and performance alerts, and more.

APMs provide visualization tools that make it much easier to spot and understand complex performance trends, leading to quicker, more informed decisions. They help prioritize optimization efforts by highlighting not only the slowest queries and most resource-intensive parts of an application, but also the endpoints that consume the most total time (total request count * average response time). This means we can focus on what's actually important. Remember that page we were worried about that takes a couple of seconds to load? Turns out, its total impact on our site amounts to practically nothing. Our products page though, clocking in at a reasonable 150ms? That's taking up 15% of our total server time.

To find out more about measuring database performance using AppSignal, check out our post Monitor the Performance of Your Ruby on Rails Application Using AppSignal.

Background Jobs

It isn't always an option, but running queries in background jobs is a powerful tool for improving performance.

You might be asking yourself how this could be, given that we only have one database; what difference does it make if we're merely calling on it from another thread or process? But there are a few main ways background jobs can come to our database's rescue.

First, just by nature of the fact that our background jobs run asynchronously, we're pretty free to run relatively resource-intensive tasks — like setting up bulk inserts (or upserts), for example (note: neither validations nor callbacks are triggered for bulk operations) — and potentially distributing large numbers of queries across periods of time (perhaps especially for write operations), possibly with the use of find_each.

You're likely used to using a job to process a CSV or spreadsheet; those tasks tend to lend themselves to asynchronous processing as well as bulk insertions. If you're doing this on your server's main thread, you're likely doing something wrong. But sometimes other, less obvious things we're currently doing on our main thread can be moved to a background job to improve performance on the main thread and reduce load on the database.

Consider a scenario where your site allows people to make potentially large purchases of relatively inexpensive items. A school, for example, might order tens of thousands of pencils at once, and maybe your system creates a PurchaseItem for every single item purchased. Instead of creating or updating thousands of rows one at a time, we can set up a bulk insert in a background job, create all the individual hashes inside a loop, and use insert_all to get the job done — taking advantage of our background worker to set up our bulk inserts, and then writing to the database once, instead of locking it with thousands of writes.

Read Replicas

Background jobs are great, but eventually, even if we're spacing out those large, numerous database queries, we hit a point where we're asking too much of our database. We're probably okay in terms of write performance (we tend to read from our database far more than we write to it, after all), but we're constantly bombarding it with queries. Our site's response times are steadily rising, and user experience is suffering.

At this point, the next logical step may be to introduce read replicas to offload work from the primary database. Read replicas allow you to distribute the read query load across one or more replicas of your database. This can improve performance across the board, as it greatly reduces the number of reads (and thus the total I/O) of your primary database, leaving it less swamped by requests (most of which are likely reads in the first place).

Ideally, large database operations are performed in background jobs reading from replicas, setting up a bulk insert or upsert, and then making one atomic write to your main database. This means you can potentially raise not only the read performance of your site, but can also boost write performance to some degree, as your primary database is consistently freer to accept write requests.

While read replicas are not painless to set up, they can provide the extra performance our site needs.

Database Sharding At A High Level

Remember how we said SQL databases don't scale horizontally? Technically, they can scale horizontally through database sharding. Sharding involves splitting up your database into multiple databases, which are then distributed across multiple servers (and often across geographical locations).

There are several approaches for doing this, including:

Geographical sharding (based on user locations).
Vertical sharding (splitting into different tables or groups of tables based on access patterns, with each shard storing a different subset of the database’s tables).
Functional sharding (splitting based on business function; like separating out billing from user content).

And more. Each approach has its use cases, and some naturally lend themselves better to some businesses than others.

As you've doubtlessly realized by now, sharding is generally difficult, and depending on your business, sometimes not even possible. It adds complexity, developmental and maintenance overhead to our apps, and potentially a lot of it. The good news is that if you don't have the resources to cover this, you probably don't need it in the first place. The bad news is that even if you do, it still complicates things and means increased costs (presumably balanced by the potential benefits).

But if you've done all you can with query optimization, read replicas, and caching, sharding might just be the next necessary step — and one that can greatly increase your I/O capability.

Words of Caution and Final Words: Database Optimization in Ruby on Rails

We need to be conscientious in how we approach our optimizations. It's possible to over-index (or index the wrong things, which hurts write performance). In the same way that N+1s can get out of control if we're not careful, so can eager loading, and it's not always immediately obvious when we're developing locally (hence the importance of at least approximating your real-world site locally).

Keep an eye on things via an APM, as traffic and user patterns can change and new ones can emerge, not just over time but potentially week to week and season to season, depending on your business.

There's no one-size-fits-all approach, so we have to be mindful of how we tailor our optimizations to our own site. Remember not to waste too much time trying to optimize things until you have an idea of the impact they'll have on your site.

That doesn't mean you shouldn't try to write generally efficient code, just that there's generally no reason to agonize over tweaking things to be absolutely optimal or wasting time on micro-optimizations. Try to keep the time-to-benefit ratio in mind, and how what you're working on will likely fit into your existing site as it is now, and as it evolves.

If you're interested in diving deeper into aspects of database optimization like sharding, you can learn more from the Rails guides. Also, check out Paweł Urbanek's great article on load_async.

Happy optimizing!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Advanced Open edX Monitoring with AppSignal for Python

AMIR TADRISI — Wed, 20 Nov 2024 09:53:11 +0000

In the first part of this series, we explored how AppSignal can significantly enhance the robustness of Open edX platforms. We saw the challenges that Open edX faces as it scales and how AppSignal's features — including real-time performance monitoring and automated error tracking — provide essential tools for DevOps teams. Our walkthrough covered the initial setup and integration of AppSignal with Open edX, highlighting the immediate benefits of this powerful observability framework.

In this second post, we'll dive deeper into the advanced monitoring capabilities that AppSignal offers. This includes streaming logs from Open edX to AppSignal, monitoring background workers with Celery, and tracking Redis queries. We will demonstrate how these features can be leveraged to address specific operational challenges, ensuring that our learning platform remains fail-safe under varying circumstances.

By the end of this article, you will know how to utilize AppSignal to its full potential in maintaining and improving the performance and reliability of your Open edX platform.

Streaming Logs to AppSignal

One of AppSignal's strongest features is centralized log management.

Commonly at Open edX, the support team reports an issue with the site, and an engineer can SSH into the server right away to check for Nginx, Mongo, MySQL, and Open edX Application logs.

A centralized storage place that houses logs without the need for you to SSH into the server is a really powerful feature. We can also set up notifications based on an issue's severity.

Now let's see how we can stream our logs from Open edX to AppSignal.

Create a Source

Under the Logging section, click on Manage sources and create a new source, with HTTP as the platform and JSON as the format. After creating the source, AppSignal provides an endpoint and API KEY that we can POST our logs to.

To have more control over log transmission, we can write a simple Python script that reads logs from our local Open edX, pre-processes them, and moves the important ones to AppSignal. For example, I wrote the following script to move only ERROR logs to AppSignal (skipping INFO and WARNING logs):

import requests
import json
from datetime import datetime
import logging

# Setup logging configuration
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# File to keep track of the last processed line
log_pointer_file = '/root/.local/share/tutor/data/lms/logs/processed.log'
log_file = '/root/.local/share/tutor/data/lms/logs/all.log'

# APpSignal API KEY
api_key = "MY-API-KEY"  # Replace with your actual API key
# URL to post the logs
url = f'https://appsignal-endpoint.net/logs?api_key={api_key}'

def read_last_processed():
    try:
        with open(log_pointer_file, 'r') as file:
            content = file.read().strip()
            last_processed = int(content) if content else 0
            logging.info(f"Last processed line number read: {last_processed}")
            return last_processed
    except (FileNotFoundError, ValueError) as e:
        logging.error(f"Could not read from log pointer file: {e}")
        return 0

def update_last_processed(line_number):
    try:
        with open(log_pointer_file, 'w') as file:
            file.write(str(line_number))
            logging.info(f"Updated last processed to line number: {line_number}")
    except Exception as e:
        logging.error(f"Could not update log pointer file: {e}")

def parse_log_line(line):
    if 'ERROR' in line:
        parts = line.split('ERROR', 1)
        timestamp = parts[0].strip()
        message_parts = parts[1].strip().split(' - ', 1)
        message = message_parts[1] if len(message_parts) > 1 else ''
        attributes_part = message_parts[0].strip('[]').split('] [')
        # Flatten attributes into a dictionary with string keys and values
        attributes = {}
        for attr in attributes_part:
            key_value = attr.split(None, 1)
            if len(key_value) == 2:
                key, value = key_value
                key = key.rstrip(']:').replace(' ', '_').replace('.', '_')  # Replace spaces and dots in keys
                if len(key) <= 50:
                    attributes[key] = value
        # Format the timestamp
        formatted_timestamp = datetime.strptime(timestamp, '%Y-%m-%d %H:%M:%S,%f').isoformat()[:-3] + 'Z'
        # Add the message and attributes to the log structure
        json_data = {
            "timestamp": formatted_timestamp,
            "group": "openedx",
            "severity": "error",
            "hostname": "tutor",
            "message": message,
        }
        json_data.update(attributes)  # Add the attributes directly to the json_data dictionary
        return json_data

def post_logs(json_data):
    headers = {'Content-Type': 'application/json'}
    response = requests.post(url, json=json_data, headers=headers)
    logging.info(f"Posted log to server; HTTP status code: {response.status_code}, Response: {response.content}")
    return response.status_code

def process_logs():
    last_processed = read_last_processed()
    with open(log_file, 'r') as file:
        for i, line in enumerate(file, 1):
            if i > last_processed:
                json_data = parse_log_line(line)
                if json_data:
                    response_code = post_logs(json_data)
                    if response_code == 200:
                        update_last_processed(i)
                    else:
                        logging.warning(f"Failed to post log, HTTP status code: {response_code}")

if __name__ == '__main__':
    logging.info("Starting log processing script.")
    process_logs()
    logging.info("Finished log processing.")

Here's how the script works:

Log File Management: Tutor saves all of the logs in the /root/.local/share/tutor/data/lms/logs/all.log file. This file contains MySQL, LMS, CMS, Caddy, Celery, and other services. The script uses a pointer /root/.local/share/tutor/data/lms/logs/processed.log file that tracks the last processed line. This ensures that each log is processed only once.
Error Filtering: As mentioned, we only send ERROR logs to AppSignal.
Data Parsing and Formatting: Each error log is parsed to extract key pieces of information, such as the timestamp and error message. The script formats this data into a JSON structure suitable for transmission.
Log Transmission: The formatted log data is sent to AppSignal using an HTTP POST request.

Important: Please make sure you don't send any personally identifiable information to the endpoint.

Now run this script and it should move ERROR logs to AppSignal:

You can also create a new trigger to notify you as soon as a specific event like ERROR happens:

Monitor Celery and Redis using AppSignal

Celery (a distributed task queue) is a vital component of Open edX, responsible for managing background tasks such as grading, certificate generation, and bulk email dispatch. Redis often acts as the broker for Celery, managing task queues. Both systems are essential for asynchronous processing and can become bottlenecks during periods of high usage. Monitoring these services with AppSignal provides valuable insights into task execution and queue health, helping you preemptively address potential issues. Let's see how we can monitor Celery and Redis.

First, install the necessary packages. Add the following to the OPENEDX_EXTRA_PIP_REQUIREMENTS variable in the .local/share/tutor/config.yml file:

- opentelemetry-instrumentation-celery==0.45b0
- opentelemetry-instrumentation-redis==0.45b0

It should look like the following:

OPENEDX_EXTRA_PIP_REQUIREMENTS:
  - appsignal==1.3.0
  - opentelemetry-instrumentation-django==0.45b0
  - opentelemetry-instrumentation-celery==0.45b0
  - opentelemetry-instrumentation-redis==0.45b0

As you can see, we are installing opentelemetry packages for Celery and Redis.

Now, we can instrument Celery with worker_process_init to report its metrics to AppSignal.

Heading back to our dashboard in AppSignal, we should see Celery and Redis reports in the Performance section, with background as the namespace.

For Redis queries, you can click on Slow queries:

Practical Monitoring: Enhancing Open edX with AppSignal

In this section, we'll revisit the initial issues outlined in part one of this series and apply practical AppSignal monitoring solutions to ensure our Open edX platform stays robust and reliable. Here’s a breakdown.

Site Performance Improvement

Let's begin by assessing overall site performance. In the Performance section, under the Issue list, we can see key metrics for all visited URLs:

Response Time: Directly reflects user experience by measuring the time taken to process and respond to requests. Factors influencing this include database queries and middleware operations.
Throughput: Indicates the number of requests handled within a given timeframe.
Mean Response Time: Provides an average response time across all requests to a specific endpoint. Any mean response time over 1 second is a potential concern and highlights areas that need optimization.
90th Percentile Response Time: For example, a 90th percentile response time of 7 ms for GET store/ suggests that 90% of requests complete in 7 ms or less.

Now let's order all the actions based on the mean. Any item higher than 1 second should be considered a red flag:

As we see, Celery tasks to rescore and reset student attempts, LMS requests to show course content, and some APIs are taking more than 1 second. Also, we should note that this is only for one active user. If we have more concurrent users, this response time will go up. Our first solution is to add more resources to the server (CPU and memory) and do another performance test.

After identifying actions with mean response times exceeding 1 second, consider performance optimization strategies such as:

Minimizing JavaScript execution
Using CDNs for static content
Implementing caching techniques.

Server Resource Monitoring

We talked about anomaly detection and host monitoring in the previous article. Let's add triggers for the following items:

CPU usage
Disk usage
Memory usage
Network traffic
Error rate

Custom Metrics

Two really important metrics for our platform are our number of active users and enrollments. Let's see how we can measure these metrics using AppSignal.

First, add increment_counter to common/djangoapps/student/views/management.py and openedx/core/djangoapps/user_authn/views/login.py to track and increment the number of logins and enrollments when there is a new event.

Now let's log in to Open edX and enroll in a course. Next, let's head to our dashboard in AppSignal. Click on Add dashboard, then Create dashboard, and give it a name and description.

Click on Add graph, enter Active Users as the title, select Add Metric and use login_count:

Your dashboard should look like the following:

You can follow the same steps to add a graph for enrollments using an enrollment_count metric.

Ensuring Consistent Styling

To make sure our site's styling stays consistent, let's add a new uptime check for static/tailwind/css/lms-main-v1.css and get notified when a URL is broken:

Email Delivery and Error Handling

In the Error section of the dashboard, we can view all errors, set up notifications for them, and work on fixes as soon as possible to prevent users from being negatively impacted.

Background Job Efficiency for Grading

In the Monitor Celery and Redis section of this article, we saw how to instrument Celery and Redis using AppSignal. Let's follow the same steps to enable AppSignal so we can see graded tasks. In the lms/djangoapps/grades/tasks.py file, add the following lines:

We should now see a couple of items to grade under Performance -> Issue list.

As you can see, recalculate_subsection_grade_v3 (our main grading Celery task) takes 212 milliseconds. For regrading, lms.djangoapps.instructor_task.tasks.reset_problem_attempts and lms.djangoapps.instructor_task.tasks.rescore_problem take 1.77 seconds.

Wrapping Up

In this two-part series, we integrated AppSignal with Open edX to fortify its monitoring capabilities. We started with the basics — setting up and understanding the fundamental offerings of AppSignal, including error tracking and performance monitoring.

In this article, we tackled how to efficiently stream logs from various Open edX services to AppSignal, ensuring all relevant information was centralized and readily accessible. We also monitored crucial asynchronous tasks handled by Celery and Redis.

Finally, we addressed some real-world challenges, such as slow site responses, resource bottlenecks during high enrollment periods, and unexpected issues like broken styling.

By now, you should have a comprehensive understanding of how to leverage AppSignal to not just monitor, but also significantly improve, the performance and reliability of your Open edX platform.

If you have any questions about Open edX or need further assistance, feel free to visit cubite.io or reach out to me directly at amir@cubite.io.

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

The Basics of Rack for Ruby

Ayush Newatia — Wed, 13 Nov 2024 14:06:53 +0000

Rack is the foundation for every popular Ruby web framework in existence. It standardizes an interface between a Ruby application and a web server. This mechanism allows us to pair any Rack-compliant web server (such as Puma, Unicorn, or Falcon) with any Rack-compliant web framework (like Rails, Sinatra, Roda, or Hanami).

Separating the concerns like this is immensely powerful and provides a lot of flexibility. It does, however, also come with limitations.

Rack 2 operated on the assumption that every request must provide a response and close the connection. It made no facility for persistent connections to enable pathways like WebSockets.

Developers had to make use of a hacky escape hatch to take over connections from Rack to implement WebSockets or similar persistent connections.

This all changed with Rack 3. But first, let's backtrack and take a closer look at Rack itself.

A Barebones Rack App

A basic Rack app looks like this:

class App
  def call(env)
    [200, { "Content-Type" => "text/plain" }, ["Hello World"]]
  end
end

run App.new

env is a hash containing request-specific information such as HTTP headers. When a request is made, the call method is called, and we return an array representing the response.

The first element is the HTTP response code, in this case 200. The second element is a hash containing any Rack and HTTP response headers we wish to send. Finally, the last element is an array of strings representing the response body.

Let's organize this app into a folder and run it.

$ mkdir rack-demo
$ cd rack-demo
$ bundle init
$ bundle add rack rackup
$ touch app.rb
$ touch config.ru

Fill in app.rb with the following:

class App
  def call(env)
    [200, { "content-type" => "text/plain" }, ["Hello World"]]
  end
end

And config.ru with:

require_relative "app"

run App.new

We can run this app using the default WEBrick server by running:

$ bundle exec rackup

The server will run on port 9292. We can verify this with a curl command.

$ curl localhost:9292
Hello World

That's got the basic app running!

Changing Web Servers

WEBrick is a development-only server, so let's swap it out for Puma:

$ bundle add puma

Now try running rackup again. You'll see it has automatically detected Puma in the bundle and started that instead of WEBrick!

$ bundle exec rackup
Puma starting in single mode...
* Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
*  Min threads: 0
*  Max threads: 5
*  Environment: development
*          PID: 45877
* Listening on http://127.0.0.1:9292
* Listening on http://[::1]:9292
Use Ctrl-C to stop

I recommend starting Puma directly instead of using rackup, as that allows us to pass configuration arguments should we want to. The -w 4 below starts Puma using 4 workers, meaning 4 instances of Puma are started up simultaneously.

$ bundle exec puma -w 4
[45968] Puma starting in cluster mode...
[45968] * Puma version: 6.4.2 (ruby 3.2.2-p53) ("The Eagle of Durango")
[45968] *  Min threads: 0
[45968] *  Max threads: 5
[45968] *  Environment: development
[45968] *   Master PID: 45968
[45968] *      Workers: 4
[45968] *     Restarts: (✔) hot (✔) phased
[45968] * Listening on http://0.0.0.0:9292
[45968] Use Ctrl-C to stop
[45968] - Worker 0 (PID: 45981) booted in 0.0s, phase: 0
[45968] - Worker 1 (PID: 45982) booted in 0.0s, phase: 0
[45968] - Worker 2 (PID: 45983) booted in 0.0s, phase: 0
[45968] - Worker 3 (PID: 45984) booted in 0.0s, phase: 0

This basic app demonstrates the Rack interface. An incoming HTTP request is parsed into the env hash and provided to the application. The application processes the request and supplies an array as the response that the server formats and sends to the client.

Rack Compliance In Frameworks

Every compliant web framework follows the Rack spec under the hood and provides an access point to go down to this level.

In Rails, we can send a Rack response in a controller:

class HomeController
  def index
    self.response = [200, {}, ["I'm Home!"]]
  end
end

Similarly, in Roda:

route do |r|
  r.on "home" do
    r.halt [200, {}, ["I'm Home!"]]
  end
end

Every Rack-compliant framework will have a slightly different syntax for accomplishing this, but since they're all sending Rack responses under the hood, they will have an API for you to access that response.

You can find the full, relatively accessible Rack specification on GitHub.

Wrapping Up

As this demo shows, Rack operates under the assumption that a request comes in, is processed by a web application, and a response is sent back. Throwing persistent connections into the mix totally breaks this model, yet Rack-compliant frameworks like Rails implement WebSockets.

In the next post, part two of a three-part series, we'll cover how to take over connections from Rack so we can hold persistent connections to enable pathways such as WebSockets.

Until then, happy coding!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Managing Distributed State with GenServers in Phoenix and Elixir

Pulkit Goyal — Tue, 12 Nov 2024 14:11:08 +0000

Phoenix and Elixir are designed at their core to build real-time, fault-tolerant applications. With its elegant syntax and the robustness of the Erlang VM, Elixir is an ideal candidate for tackling the challenges of distributed state management.

This two-part series will guide Phoenix/Elixir developers through the intricacies of working with Phoenix in a distributed setup.

The focus of this first post will be on GenServers — the key components behind state management in distributed systems. We will explore how GenServers can be leveraged to maintain state across nodes in a Phoenix application, ensuring data consistency and fault tolerance. We’ll break down the complexities of GenServers and distributed state management into manageable and understandable segments.

Overview of Our Phoenix Application

Let's focus on distributed state management in the context of a CRUD API. Imagine you have a popular API service built with Phoenix that handles a large number of requests from various clients. To prevent abuse and ensure fair usage, you want to implement a rate limiter that restricts the number of requests a client can make within a specific timeframe, such as 100 requests per minute.

You can implement this simply using the "token bucket" strategy. In this strategy, a bucket is maintained for each client that is filled with tokens at a constant rate. Each API call consumes a single token from the bucket and is blocked if there are no tokens available.

In the next section, we will set up a token bucket for a single-node scenario using a GenServer.

A Single-Node Rate Limiter Implementation Using GenServer for Elixir

GenServers are a cornerstone within the Elixir ecosystem for managing state and encapsulating functionality. They are the go-to structure for maintaining state within an application because they can hold state throughout their lifecycle and be interacted with asynchronously.

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  # tokens per second
  @rate 1
  # maximum tokens in the bucket
  @bucket_size 10

  def start_link([]), do: GenServer.start_link(__MODULE__, nil, name: __MODULE__)

  @impl true
  def init(nil), do: {:ok, %{buckets: %{}}}

  def allow?(client_id) do
    GenServer.call(__MODULE__, {:allow?, client_id})
  end

  @impl true
  def handle_call({:allow?, client_id}, _from, state) do
    current_time = System.monotonic_time(:second)
    {reply, updated_bucket} =
      state.buckets
      |> get_or_init_bucket(client_id)
      |> update_bucket(current_time)
      |> maybe_consume_token(current_time)

    state = put_in(state, [:buckets, client_id], updated_bucket)
    {:reply, reply, state}
  end

  defp maybe_consume_token(%{tokens: tokens} = bucket, current_time)
       when tokens >= 1 do
    updated_bucket = %{bucket | tokens: tokens - 1, last_checked: current_time}
    {true, updated_bucket}
  end

  defp maybe_consume_token(bucket, _current_time), do: {false, bucket}

  defp get_or_init_bucket(buckets, client_id) do
    Map.get(buckets, client_id, %{
      tokens: @bucket_size,
      last_checked: current_time
    })
  end

  defp update_bucket(bucket, current_time) do
    # Add tokens based on the last time the bucket was checked
    time_passed = current_time - bucket.last_checked
    new_tokens = time_passed * @rate
    updated_tokens = min(bucket.tokens + new_tokens, @bucket_size)
    %{bucket | tokens: updated_tokens, last_checked: current_time}
  end
end

This works well in a single-node setup because the GenServer holds the token state for the entire application across web requests. We can add it to the application's supervision tree to start the API rate limiter automatically:

defmodule MyApp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      # Other Children...
      MyApp.TokenBucketRateLimiter
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # ...
end

The first time allow? is called with a new client ID, it initializes the bucket to 10 tokens, and then each allow? with the same client ID consumes one token per request. Eventually, the eleventh request in the same second is rejected because there are no more tokens. If the client tries again after a while, the requests are allowed again.

iex> 1..11 |> Enum.map(fn _i -> MyApp.TokenBucketRateLimiter.allow?("some client id") end)
[true, true, true, true, true, true, true, true, true, true, false]

iex> Process.sleep(1000)
:ok
iex> MyApp.TokenBucketRateLimiter.allow?("some client id")
true

But as the app scales and we deploy it across multiple nodes (e.g., in a Kubernetes cluster or across different regions), we face the challenge of coordinating rate limiting across all nodes.

Each node needs to be aware of the request counts from other nodes to enforce the rate limit consistently. Without a distributed solution, each node would only track requests it directly handles, leading to inconsistent rate limiting.

You might think that maintaining request limits in memory is not feasible at a scale that requires multiple nodes. But since each client's state is just a single integer for the token size and a single integer for the timestamp, this doesn't require a large amount of memory (2 * 64 bytes per client). Even if there are 100k concurrent clients, less than 15MB of in-memory state is required.

Understanding GenServers in a Distributed Elixir Environment

Distributed systems pose unique challenges, particularly when it comes to state management. Network partitions, node failures, and latency are just a few issues that can disrupt state consistency across nodes. Ensuring that all nodes have the correct state or can recover it when things go wrong is critical for a system's reliability.

GenServers can be distributed across nodes in a cluster, allowing them to share state and handle requests from different parts of a system. By leveraging the built-in distribution mechanisms of the BEAM VM, GenServers can communicate across nodes with the same ease as within a single node.

In the next section, we will use some strategies to support rate limiting across multiple nodes in the cluster.

Implementing Distributed GenServers Using a Single Global Process

The simplest strategy to support a rate limiter across a whole cluster is to start a single global process that manages the limits.
This is simple to do using Erlang's :global name registration.

In the single node setup above, we register the GenServer under the name MyApp.TokenBucketRateLimiter (__MODULE__ contains the name of the current module). This registers the process locally on the node. It is also possible to register it globally using a {:global, name} tuple when calling GenServer.start_link/3.
This registers the process under the given name globally across the whole cluster. Note that since we update the process name in start_link, we also have to update it when using any call, cast, and friends.

Let's update the code to do this:

defmodule MyApp.TokenBucketRateLimiter do
  def start_link([]) do
    GenServer.start_link(__MODULE__, nil, name: {:global, __MODULE__})
  end

  def allow?(client_id) do
    GenServer.call({:global, __MODULE__}, {:allow?, client_id})
  end
end

This enforces that only a single process with the global name MyApp.TokenBucketRateLimiter can exist across the cluster. However, this is insufficient, as it will prevent all other application nodes from joining our cluster because the supervision tree cannot start all its processes. Let's try it out by starting a few nodes in the cluster.

The first node starts fine:

   ➜ iex --name node1@127.0.0.1 --cookie asdf -S mix
   iex(node1@127.0.0.1)1>

But as soon as we start a second node and connect it to the cluster, the application crashes:

   ➜ iex --name node2@127.0.0.1 --cookie asdf -S mix
   iex(node2@127.0.0.1)1> Node.connect(:"node1@127.0.0.1")
   [notice] Application my_app exited: shutdown

We need to make sure that the start_link function of the rate limiter doesn't generate any errors when a process already exists on another node. To do this, we can check and modify the start_link function to return :ignore when the process already exists:

def start_link([]) do
  tuple = {:global, __MODULE__}

  case GenServer.whereis(tuple) do
    nil -> GenServer.start_link(__MODULE__, nil, name: tuple)
    _pid -> :ignore
  end
end

Once that's done, we can start multiple nodes in the cluster, and they will all ping a central rate limiter instance to allow or deny incoming requests. The central rate limiter will be the GenServer process that happens to start first, i.e., the first node in the cluster to come up.

While this works, there are several issues with this approach:

The node is a single point of failure now. If the node goes down, the rate limiter will be unreachable until another node starts it. Even worse, this is not handled automatically, so we might be left with no rate limiter process unless a new node is started.
The state is on a single node. This means that if the rate limiter process is terminated/restarted (either due to a programming error or node failure), the state is lost.
All nodes in the cluster have to make a call to the single node on every API call.

Using Multiple Processes Across the Cluster

In order to provide better fault tolerance when managing state across distributed nodes, we can use a combination of techniques, including:

State partitioning: Dividing the state into partitions so that each GenServer instance handles a subset of the data.
Replication: Replicating state across nodes to provide redundancy and increase fault tolerance.
Conflict resolution: Implementing strategies to handle state conflicts, such as using version vectors or CRDTs (Conflict-free Replicated Data Types).

Let's see how we can use CRDTs to provide super-fast data access alongside eventual consistency and data replication guarantees.

We will use the Elixir library DeltaCrdt to manage shared state across multiple nodes in the cluster.

Note: In addition to DeltaCrdt, other libraries like Horde and Swarm can help you to coordinate processes and state across several nodes in the cluster.

Let's add it to mix.exs to get started:

defmodule MyApp.MixProject do
  defp deps do
    [
      # ...
      {:delta_crdt, "~> 0.6.3"}
    ]
  end
end

DeltaCrdt provides a simple-to-use API for managing a map of data that is guaranteed to be conflict-free (across distributed usage) and replicated across multiple nodes. For conflict resolution, it provides AWLWWMap (that stands for 'Add-Wins Last-Write-Wins Map) so that in the case of a conflict, the last write always wins.

The API is simple — you start multiple DeltaCrdt processes (which can be on the same node or multiple nodes) and create a cluster with them using set_neighbours/2:

{:ok, crdt1} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
{:ok, crdt2} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap)
DeltaCrdt.set_neighbours(crdt1, [crdt2])
DeltaCrdt.read(crdt1)
%{}

Once neighbours are set, adding any data to one CRDT automatically syncs with the second one.

DeltaCrdt.put(crdt1, "foo", "bar")
DeltaCrdt.read(crdt2)
%{"foo" => "bar"}

Starting A One Rate Limiter Per Node

With this out of the way, we can update our implementation of the rate limiter to use DeltaCrdt to sync states. First, we need to make sure all rate limiter processes are discoverable globally, but with unique names. One way to do that is to use the node name when naming the process:

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  def start_link([]), do: GenServer.start_link(__MODULE__, nil, name: global_tuple())

  defp global_tuple(node \\ Node.self()), do: {:global, to_string(node) <> to_string(__MODULE__)}
end

Syncing GenServer State With CRDT

Now that all nodes start their own copy of the rate limiter, we need to sync the state. Let's do that by starting a DeltaCrdt process from the GenServer and using the CRDT to make decisions about the rate limits:

defmodule MyApp.TokenBucketRateLimiter do
  use GenServer

  @impl true
  def init(nil) do
    {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 100)

    {:ok, %{crdt: crdt}}
  end

  def allow?(client_id), do: GenServer.call(global_tuple(), {:allow?, client_id})

  @impl true
  def handle_call({:allow?, client_id}, _from, state) do
    current_time = System.os_time(:second)

    {reply, updated_bucket} =
      state.crdt
      |> get_or_init_bucket(client_id, current_time)
      |> update_bucket(current_time)
      |> maybe_consume_token(current_time)

    # Update the CRDT with the new bucket state
    DeltaCrdt.put(state.crdt, client_id, updated_bucket)

    {:reply, reply, state}
  end

  defp get_or_init_bucket(crdt, client_id, current_time) do
    case DeltaCrdt.get(crdt, client_id) do
      nil -> default_bucket(current_time)
      bucket -> bucket
    end
  end

  defp default_bucket(time), do: %{tokens: @bucket_size, last_checked: time}
end

The major changes here from the single-node approach are:

Instead of Map.get from the local GenServer state, we use DeltaCrdt.get to get the key from the CRDT instead. This fetches the data from the local copy of the CRDT which is very fast to access and guarantees eventual consistency.
Similar to the above, we use DeltaCrdt.put to update the shared state instead of the local GenServer state when updating the tokens count. This is again a fast operation that updates the local state — but with a guarantee that the changes will be eventually synced with all processes in the cluster.

Clustering the GenServers

While we are almost there, there is one last step left. The CRDTs are currently local and do not sync with other nodes in the GenServer. We need to use the set_neighbours/2 API from DeltaCRDT to set up a sync.

Let's create a new function named update_neighbours inside the GenServer that does this:

defp update_neighbours(crdt) do
  neighbours =
    Node.list()
    |> Enum.map(fn node ->
      node
      |> global_tuple()
      |> GenServer.whereis()
      |> :sys.get_state()
      |> Map.get(:crdt)
    end)

  DeltaCrdt.set_neighbours(crdt, neighbours)
end

Now all we need to do is call this function when the GenServer starts (from init or a handle_continue callback).

The above piece of code takes care of updating CRDT neighbors when a new process starts. But set_neighbours is a one-sided operation (i.e., it sets up puts from the CRDT to its neighbors, but not the other way around). To ensure servers that have already started also update their neighbors, we must call update_neighbours on all cluster processes as soon as any service starts.

Let's add a new refresh_peer_neighbours function to do that (which should then be called after the initialization of the GenServer).

defp refresh_peer_neighbours() do
  Node.list()
  |> Enum.each(fn node ->
    node
    |> global_tuple()
    |> GenServer.whereis()
    |> Process.send(pid, :refresh_neighbours, [])
  end)
end

@impl true
def handle_info(:refresh_neighbours, state) do
  update_neighbours(state.crdt)
  {:noreply, state}
end

Handling Cluster Changes

With all this in place, there is still one final piece of the puzzle. What happens when nodes are added or removed from the cluster? To update the CRDT neighborhood on cluster-level changes, we can use :net_kernel.monitor_nodes(true) from Erlang. This monitors the nodes and refreshes the neighbors on any nodeup or nodedown events:

@impl true
def init(nil) do
  {:ok, crdt} = DeltaCrdt.start_link(DeltaCrdt.AWLWWMap, sync_interval: 100)

  # Monitor for addition of new nodes
  :net_kernel.monitor_nodes(true)

  {:ok, %{crdt: crdt}, {:continue, nil}}
end

@impl true
def handle_info({:nodeup, _node}, state) do
  Process.send_after(self(), :refresh_neighbours, 1_000)
  {:noreply, state}
end

def handle_info({:nodedown, _node}, state) do
  Process.send_after(self(), :refresh_neighbours, 1_000)
  {:noreply, state}
end

This finally brings us to the end of our implementation. There are some edge cases to consider:

Avoiding deadlocks when refreshing neighbours (since we need to fetch the PID of the CRDTs from all nodes).
Handling cases where the RateLimiter does not exist on a node (e.g., when it is being restarted).

Here is the full implementation. Let's take it for a spin:

# Node 1
➜ iex --name node1@127.0.0.1 --cookie asdf -S mix
iex(node1@127.0.0.1)1>

# Node2
➜ iex --name node2@127.0.0.1 --cookie asdf -S mix
iex(node2@127.0.0.1)1> Node.connect(:"node1@127.0.0.1")
true
iex(node2@127.0.0.1)2> MyApp.TokenBucketRateLimiter.allow?("some client id")
true
iex(node2@127.0.0.1)3> MyApp.TokenBucketRateLimiter.crdt() |> DeltaCrdt.get("some client id")
%{tokens: 9, last_checked: 1725358705}

# Back on Node1
iex(node1@127.0.0.1)1> MyApp.TokenBucketRateLimiter.crdt() |> DeltaCrdt.get("some client id")
%{tokens: 9, last_checked: 1725358705}

And that's it for part one of this series!

Wrapping Up

In this first part of our Distributed Phoenix series, we've taken an in-depth look at how GenServers and Delta CRDTs can be leveraged to manage distributed state in a Phoenix application. We've implemented a token bucket rate limiter across multiple nodes, ensuring consistency and fault tolerance even in a distributed setup.

This foundational knowledge is crucial for building scalable, resilient systems that can handle high traffic and maintain data integrity across multiple servers.

In part two, we'll dive deeper into advanced techniques for optimizing distributed systems, including strategies for deploying and scaling distributed applications.

Stay tuned for more insights and practical examples that will help you master the art of distributed applications with Phoenix and Elixir.

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

Find and Fix N+1 Queries Using AppSignal for a Phoenix App in Elixir

Sapan Diwakar — Tue, 05 Nov 2024 10:00:45 +0000

N+1 queries are a frequent issue in complex applications built with Elixir and Phoenix. These queries can silently degrade application performance, often going unnoticed until they've compounded into a significant problem.

They can substantially increase web page load times, as each additional query adds overhead to a database, consuming more time and resources. That's why it's crucial to detect and resolve N+1 queries to optimize production systems. It's not just about maintaining a seamless user experience; it's also about ensuring you maintain your Elixir application's scalability and efficiency.

Using AppSignal, you can identify N+1 queries in Phoenix applications, alongside a comprehensive suite of tools to monitor, diagnose, and rectify performance bottlenecks in Elixir projects.

Let's explore how N+1 queries happen, their impact on performance, and some strategies to detect and fix them in your Elixir application.

Understanding And Fixing N+1 Queries in Phoenix for Elixir

N+1 queries occur when an application retrieves a single database record, followed by an additional query for each related record.
In Phoenix applications, N+1 queries often emerge in the context of complex relationships and Ecto associations.

Even small mistakes in Ecto queries can lead to cascading N+1 issues, resulting in multiple round trips to a database, each fetching only a small amount of data.

Some common examples include:

Using Repo.all without a proper preload clause (this will not fetch associated records in the same query).
In GraphQL, when using Absinthe with Elixir, N+1 query issues can arise when resolving fields that require data from associated records.

Check out Avoiding N+1 Queries for Absinthe for an overview of this topic.

A Simple Example Using Ecto

For instance, consider a blog application where each post has many comments. A naive Ecto query might retrieve all posts and then run a separate query for each post's comments. This results in one query for the posts (N) and one query for each post's comments (+1), which is highly inefficient.

# Fetching posts without preloading comments
posts = Repo.all(Post)

# This will cause N+1 queries, as for each post, comments are fetched separately
posts |> Enum.each(fn post ->
  comments = Repo.all(from c in Comment, where: c.post_id == ^post.id)
  IO.inspect(comments)
end)

The above might sound like a contrived example — experienced developers are quite unlikely to do this. But it is meant to serve as a simplified scenario of how N+1 queries can creep into systems: any query inside a loop is a suspect. N+1 queries are usually hidden in plain sight, for example, behind a context function like Blog.list_comments(post_id: post.id) inside the loop.

Next, let's discuss a complex scenario.

Complex Situations Leading to N+1 Queries

Imagine a blog application displaying a feed of all recent posts and the count of comments (or likes) for each post.

A sample implementation could look like this:

# Load posts from subscribed authors
posts = Repo.all(Post)

# Load comments count per post
for post <- posts do
  comments_count = Repo.one(from c in Comment, where: c.post_id == ^post.id, select: count(c.id))
  render("post.html", post: post, comments_count: comments_count)
end

Here, the comments_count query causes an N+1 problem because we fetch each individual post's count. One might argue that we can just add comments to the preloads and be done with it. While that solves the N+1 query, it loads too much unnecessary data (we don't need all the comments yet, only the total number of comments) and might actually hurt loading times.

A better solution is to join the tables and use select to load the comments count within the posts query:

posts_with_counts =
  from(p in Post,
    left_join: c in Comment, on: c.post_id == p.id,
    group_by: [p.id],
    select: %{p | comments_count: count(c.id)}
  )
  |> Repo.all()

for post <- posts_with_counts do
  render("post.html", post: post, comments_count: post.comments_count)
end

To avoid and detect N+1 queries, developers must be vigilant when writing Ecto queries and templates.
Using preload correctly to fetch all necessary records in a single database query is essential.

Detecting N+1 Queries By Their Impact

Understanding the data access patterns of your application can help you anticipate and prevent N+1 issues before they arise.
One way to identify N+1 queries is to observe and analyze common bottlenecks in your app, including:

Increased Load Times: Each query takes time to execute, and when hundreds or thousands of them run sequentially, the cumulative effect leads to noticeable delays in content rendering.
Database Strain: Databases are designed to handle multiple queries efficiently, but an influx of unnecessary queries can strain the system, leading to slower response times for all users.
Server Resource Drain: A server's resources are finite, and handling a multitude of queries consumes more CPU and memory, potentially affecting other operations and leading to resource exhaustion.
Scalability Issues: As a user base grows, inefficient N+1 queries can prevent an application from scaling smoothly, requiring more hardware resources to handle the same workload.
Cost Implications: Increased server load and database usage can lead to higher operational costs, especially in cloud-based environments where resources are metered.

However, it is difficult to spot small differences in performance and then pinpoint the parts of your application that have caused such degradation.

The next sections will show how AppSignal can help you detect and fix N+1 queries, ensuring your application runs smoothly and efficiently.

Detecting N+1 Queries in Phoenix for Elixir Using AppSignal

Before detecting N+1 queries, we need to set up AppSignal in your Phoenix application. You can sign up for a free 30-day trial of AppSignal.

AppSignal provides an Elixir package that integrates seamlessly with Phoenix applications. The AppSignal for Elixir setup process involves adding the Elixir package to your project's dependencies and configuring it with your AppSignal Push API key. Once installed, AppSignal starts monitoring your application's performance, including database query patterns.

Analyzing Elixir Data with AppSignal

AppSignal's instrumentation for Phoenix and Ecto is designed to provide detailed insights into your application's database interactions. It automatically tracks database query times, helping you spot inefficiencies. With AppSignal's instrumentation in place, detecting N+1 queries becomes a matter of analyzing the collected data.

Use Slow Queries to Find N+1 Queries

AppSignal's Slow Queries dashboard breaks down web requests, showing the slowest queries and potential bottlenecks. A series of similar queries within a single web request is a strong indicator of an N+1 query problem.

For example, if you see repeated queries fetching comments for different posts while a blog page is being rendered, you've likely encountered an N+1 issue. AppSignal provides context, including specific database calls, making it easier to pinpoint the exact location in your code where the issue originated.

We can see that the first query had almost 95% impact on all our queries executed through Ecto.
In a real-world app, this number might be lower, as there can be hundreds of unique queries. However, any outlier that's highlighted here is a great candidate for analysis.

💡 Note: The "impact" of a query on an application is based on its usage compared to other queries.

Another metric to consider here is the high throughput of 720.
Throughput is the total number of queries that were executed in a specified time window.

Let's check out the full details of the query:

SELECT c0."id", c0."body", c0."post_id", c0."user_id", c0."inserted_at", c0."updated_at", c0."post_id" FROM "comments" AS c0 WHERE (c0."post_id" = $1) ORDER BY c0."post_id"

Since the query is targeting a single post, it is quite likely an N+1 query.

Trace N+1 Queries with the Event Timeline

AppSignal's Event Timeline is also particularly useful for finding and tracing N+1 queries.
It visualizes a sequence of database calls during a web request, allowing you to identify the hallmark successive database queries that characterize N+1 issues.

Here's an example event timeline for a page that has an N+1 query issue:

We can see that many quick queries are being performed.
Hovering over query.ecto also shows the exact query that was performed to help us pinpoint which query is the culprit.

Another good indication of a possible issue is when the ecto time of an event is abnormally high.
For example, in this case, we can see that Ecto was the major time-hog:

Addressing Detected N+1 Queries

Once you've identified an N+1 query with AppSignal, the next step is to address it.
This typically involves optimizing your Ecto queries to preload associated records.

AppSignal's detailed metrics allow you to compare performance before and after the changes, giving you concrete feedback on the impact of your optimizations.

And that's it!

Wrapping Up

Whether you're a seasoned developer or new to Phoenix, understanding the nuances of N+1 queries and leveraging AppSignal's capabilities will empower you to build faster, more reliable applications.

Remember, the key to maintaining a performant application is not just fixing issues as they arise, but continuously monitoring and improving your codebase with the help of powerful tools like AppSignal.

Happy coding!

P.S. If you'd like to read Elixir Alchemy posts as soon as they get off the press, subscribe to our Elixir Alchemy newsletter and never miss a single post!

What's New in Ruby on Rails 8

Damilola Olatunji — Thu, 31 Oct 2024 12:08:59 +0000

The first Rails 8 beta has officially been released, bringing an exciting set of features, bug fixes, and improvements. This version builds on the foundation of Rails 7.2, while introducing new features and optimizations to make Rails development even more productive and enjoyable.

Key highlights include an integration with Kamal 2 for hassle-free deployments, the introduction of Propshaft as the new default asset pipeline, and extensive ActiveRecord enhancements. Rails 8 also brings several SQLite integration upgrades that make it a viable option for production use.

Let's dive in and explore everything that Rails 8 has to offer!

Effortless Deployments with Kamal 2 and Thruster

Rails 8 makes deploying your applications simple with Kamal 2 and Thruster.

Kamal 2 reduces the need for reliance on managed cloud services and Platform as a Service (PaaS) platforms by enabling quick and easy deployment to cloud VMs, bare metal servers, or VPS environments in just minutes.

With a single command (kamal setup), you can set up a production-ready Rails environment on a standard Linux box, making deployment both easy and cost-effective.

Kamal 2 also integrates with Thruster, a custom proxy built specifically for Rails that enables zero-downtime deployments, HTTP/2 support, automated SSL with Let's Encrypt, Gzip compression, and easy hosting of multiple apps on a single server — all without complex setup.

With Kamal 2 and Thruster, Rails 8 makes deploying apps easier than ever. And if you prefer a different deployment setup, you can opt out using the --skip-kamal flag to maintain your existing workflows.

Leaner Rails Deployments with Solid Adapters

One of the big improvements in Rails 8 is simpler deployments by reducing the number of additional services needed to implement common web application requirements.

Traditionally, if you required features like job queues, caching, and pub/sub messaging, you'd use a combination of a database like PostgreSQL with Redis for auxiliary functions.

With Rails 8, you can handle all these with just SQLite, thanks to three new database-backed adapters: Solid Cable, Solid Cache, and Solid Queue.

Solid Cable is Rails' new default Action Cable adapter in production and means you can drop the common dependency on Redis. It acts as the pub/sub server, relaying messages between the app and connected clients using fast polling through SQLite. Despite polling, Solid Cable's performance is comparable to Redis in most situations.
Solid Cache replaces the need for Redis by using disk storage instead of RAM for caching. This approach allows for much larger, more cost-effective caches that persist longer and handle more requests without sacrificing performance. It also supports encrypted storage and retention policies to meet privacy requirements.
Solid Queue replaces Redis for Active Job background processing, using the FOR UPDATE SKIP LOCKED mechanism for efficient job handling (compatible with PostgreSQL, MySQL, or SQLite). It includes essential features like concurrency control, retries, and recurring jobs, and has proven itself at HEY, where it now manages 20 million jobs per day.

These three adapters are designed around a simple idea: modern SSDs and NVMe drives are fast enough to handle many tasks that previously required in-memory solutions. By tapping into these speedy drives, Rails cuts out the need for separate RAM-based tools like Redis.

SQLite is Ready for Production

Rails 8 takes SQLite from a lightweight development tool to a reliable choice for production use, thanks to extensive work on the SQLite adapter and Ruby driver.

With the introduction of the solid adapters discussed above, SQLite now has the capability to power Action Cable, Rails.cache, and Active Job effectively, expanding its role beyond just prototyping or testing environments.

Here are some key improvements to the SQLite integration in Rails 8:

Full-text search and virtual tables are now supported using create_virtual_table.
The adapter now allows bulk insert fixtures for enhanced data seeding performance.
Transactions default to IMMEDIATE mode to improve concurrency.
Enhanced error handling by translating SQLite3::BusyException into ActiveRecord::StatementTimeout.

A New Era for the Asset Pipeline with Propshaft

Rails 8 also introduces Propshaft as the new asset pipeline default, replacing the long-standing Sprockets system. Sprockets served Rails developers well for over a decade, but it was designed in a different era — before the explosion of JavaScript build tools and modern browser improvements.

Propshaft reflects a simpler, modern approach to managing assets, built around the core needs of today's developers. Its purpose is straightforward: to provide a clear path for assets and apply digest stamps for caching.

Unlike Sprockets, which took on numerous additional tasks, Propshaft focuses only on what's essential, fitting naturally with the new Rails philosophy of keeping asset pipelines lean (while complex JavaScript handling is left to specialized tools like Esbuild or Vite).

Built-In Authentication Made Simple

Rails has been building the key components of authentication over the years, from has_secure_password in Rails 5 to normalizes, generates_token_for and authenticate_by in Rails 7.1.

With Rails 8, all these components come together to give you a straightforward starting point for building a secure, session-based authentication system.

By running a single command, you can set up all the essentials for an authentication system with database-backed sessions and password reset functionality:

bin/rails generate authentication

This command generates key files, including models, controllers, mailers, and views:

app/models/current.rb
app/models/user.rb
app/models/session.rb
app/controllers/sessions_controller.rb
app/controllers/passwords_controller.rb
app/mailers/passwords_mailer.rb
app/views/sessions/new.html.erb
app/views/passwords/new.html.erb
app/views/passwords/edit.html.erb
app/views/passwords_mailer/reset.html.erb
app/views/passwords_mailer/reset.text.erb
db/migrate/xxxxxxx_create_users.rb
db/migrate/xxxxxxx_create_sessions.rb
test/mailers/previews/passwords_mailer_preview.rb

This effectively puts you on the fast track to secure, production-ready authentication. All that's left is to integrate a user sign-up flow customized to your application's needs.

New Script Folder and Generator

Rails 8 introduces a new script folder dedicated to holding one-off or general-purpose scripts, such as data migrations, cleanup tasks, or other utility operations. This addition helps organize these scripts neatly, keeping them separate from your main application logic.

To make script creation easier, a new script generator is available. You can generate scripts with a simple command:

bin/rails generate script my_script

These commands create the corresponding script files, which you can then execute with:

bundle exec ruby script/my_script.rb

This streamlined approach keeps your application organized and makes handling custom scripts more convenient and maintainable.

A Slew of Active Record Improvements

Active Record has also seen major enhancements in Rails 8 to improve
performance, simplify migrations, improve troubleshooting, and provide better support for complex database use cases.

Below are some of the key changes introduced in this latest version:

Rails 8 now distinguishes between float4 and float8 in PostgreSQL.
drop_table now supports dropping multiple tables at once.
Support for PostgreSQL table inheritance and native partitioning has been added.
Bulk inserts of fixtures are now supported to improve data seeding performance.
Migrating a fresh database now starts by loading the database schema before running the migrations.
create_schema and drop_schema operations are now reversible.
Rails 8 now requires MySQL 5.6.4 or later due to advancements like datetime with precision.
Query log tags are enabled by default in development environments to trace SQL statements back to the application code and identify which database is in use.

Wrapping Up

Rails 8 introduces a range of impactful updates, from easier deployments with Kamal and a modern asset pipeline to significant ActiveRecord enhancements and improved production capabilities for SQLite.

These advancements not only boost developer productivity but also align with modern best practices, allowing you to focus on building your application instead of dealing with infrastructure complexities.

For a detailed list of all the new features, optimizations, and changes, check out the official Rails 8 release notes.

If you want to get involved with contributing to Rails, visit the
Rails GitHub repository to explore open issues and review the contribution guidelines.

Thanks for reading!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

How to Use Lambda Functions in Python

Federico Trotta — Wed, 30 Oct 2024 16:03:10 +0000

Lambda functions in Python are a powerful way to create small, anonymous functions on the fly. These functions are typically used for short, simple operations where the overhead of a full function definition would be unnecessary.

While traditional functions are defined using the def keyword, Lambda functions are defined using the lambda keyword and are directly integrated into lines of code. In particular, they are often used as arguments for built-in functions. They enable developers to write clean and readable code by eliminating the need for temporary function definitions.

In this article, we'll cover what Lambda functions do and their syntax. We'll also provide some examples and best practices for using them, and discuss their pros and cons.

Prerequisites

Lambda functions have been a part of Python since version 2.0, so you'll need:

Minimum Python version: 2.0.
Recommended Python version: 3.10 or later.

In this tutorial, we'll see how to use Lambda functions with the library Pandas: a fast, powerful, flexible, and easy-to-use open-source data analysis and manipulation library. If you don't have it installed, run the following:

pip install pandas

Syntax and Basics of Lambda Functions for Python

First, let's define the syntax developers must use to create Lambda functions.

A Lambda function is defined using the lambda keyword, followed by one or more arguments and an expression:

lambda arguments: expression

Let's imagine we want to create a Lambda function that adds up two numbers:

add = lambda x, y: x + y

Run the following:

result = add(3, 5)
print(result)

This results in:

We've created an anonymous function that takes two arguments, x and y. Unlike traditional functions, Lambda functions don't have a name: that's why we say they are "anonymous."

Also, we don't use the return statement, as we do in regular Python functions. So we can use the Lambda function at will: it can be printed (as we did in this case), stored in a variable, etc.

Now let's see some common use cases for Lambda functions.

Common Use Cases for Lambda Functions

Lambda functions are particularly used in situations where we need a temporarily simple function. In particular, they are commonly used as arguments for higher-order functions.

Let's see some practical examples.

Using Lambda Functions with the `map()` Function

map() is a built-in function that applies a given function to each item of an iterable and returns a map object with the results.

For example, let's say we want to calculate the square roots of each number in a list. We could use a Lambda function like so:

# Define the list of numbers
numbers = [1, 2, 3, 4]

# Calculate square values and print results
squared = list(map(lambda x: x ** 2, numbers))
print(squared)

This results in:

[1, 4, 9, 16]

We now have a list containing the square roots of the initial numbers.

As we can see, this greatly simplifies processes to use functions on the fly that don't need to be reused later.

Using Lambda Functions with the `filter()` Function

Now, suppose we have a list of numbers and want to filter even numbers.

We can use a Lambda function as follows:

# Create a list of numbers
numbers = [1, 2, 3, 4]

# Filter for even numbers and print results
even = list(filter(lambda x: x % 2 == 0, numbers))
print(even)

This results in:

[2,4]

Using Lambda Functions with the `sorted()` Function

The sorted() function in Python returns a new sorted list from the elements of any iterable. Using Lambda functions, we can apply specific filtering criteria to these lists.

For example, suppose we have a list of points in two dimensions: (x,y). We want to create a list that orders the y values incrementally.

We can do it like so:

# Creates a list of points
points = [(1, 2), (3, 1), (5, -1)]

# Sort the points and print
points_sorted = sorted(points, key=lambda point: point[1])
print(points_sorted)

And we get:

[(5, -1), (3, 1), (1, 2)]

Using Lambda Functions in List Comprehensions

Given their conciseness, Lambda functions can be embedded in list comprehensions for on-the-fly computations.

Suppose we have a list of numbers. We want to:

Iterate over the whole list
Calculate and print double the initial values.

Here's how we can do that:

# Create a list of numbers
numbers = [1, 2, 3, 4]

# Calculate and print the double of each one
squared = [(lambda x: x ** 2)(x) for x in numbers]
print(squared)

And we obtain:

[1, 4, 9, 16]

Advantages of Using Lambda Functions

Given the examples we've explored, let's run through some advantages of using Lambda functions:

Conciseness and readability where the logic is simple: Lambda functions allow for concise code, reducing the need for standard function definitions. This improves readability in cases where function logic is simple.
Enhanced functional programming capabilities: Lambda functions align well with functional programming principles, enabling functional constructs in Python code. In particular, they facilitate the use of higher-order functions and the application of functions as first-class objects.
When and why to prefer Lambda functions: Lambda functions are particularly advantageous when defining short, "throwaway" functions that don't need to be reused elsewhere in code. So they are ideal for inline use, such as arguments to higher-order functions.

Limitations and Drawbacks

Let's briefly discuss some limitations and drawbacks of Lambda functions in Python:

Readability challenges in complex expressions: While Lambda functions are concise, they can become difficult to read and understand when used for complex expressions. This can lead to code that is harder to maintain and debug.
Limitations in error handling and debugging: As Lambda functions can only contain a single expression, they can't include statements, like the try-except block for error handling. This limitation makes them unsuitable for complex operations that require these features.
Restricted functionality: Since Lambda functions can only contain a single expression, they are less versatile than standard functions. This by-design restriction limits their use to simple operations and transformations.

Best Practices for Using Lambda Functions

Now that we've considered some pros and cons, let's define some best practices for using Lambda functions effectively:

Keep them simple: To maintain readability and simplicity, Lambda functions should be kept short and limited to straightforward operations. Functions with complex logic should be refactored into standard functions.
Avoid overuse: While Lambda functions are convenient for numerous situations, overusing them can lead to code that is difficult to read and maintain. Use them judiciously and opt for standard functions when clarity is fundamental.
Combine Lambda functions with other Python features: As we've seen, Lambda functions can be effectively combined with other Python features, such as list comprehensions and higher-order functions. This can result in more expressive and concise code when used appropriately.

Advanced Techniques with Lambda Functions

In certain cases, more advanced Lambda function techniques can be of help.

Let's see some examples.

Nested Lambda Functions

Lambda functions can be nested for complex operations.

This technique is useful in scenarios where you need to have multiple small transformations in a sequence.

For example, suppose you want to create a function that calculates the square root of a number and then adds 1. Here's how you can use Lambda functions to do so:

# Create a nested lambda function
nested_lambda = lambda x: (lambda y: y ** 2)(x) + 1

# Print the result for the value 3
print(nested_lambda(3))

You get:

Integration with Python Libraries for Advanced Functionality

Many Python libraries leverage Lambda functions to simplify complex data processing tasks.

For example, Lambda functions can be used with Pandas and NumPy to simplify data manipulation and transformation.

Suppose we have a data frame with two columns. We want to create another column that is the sum of the other two. In this case, we can use Lambda functions as follows:

# Create the columns' data
data = {'A': [1, 2, 3], 'B': [4, 5, 6]}

# Create data frame
df = pd.DataFrame(data)

# Create row C as A+B and print the dataframe
df['C'] = df.apply(lambda row: row['A'] + row['B'], axis=1)
print(df)

And we get:

That's it for our whistle-stop tour of Lambda functions in Python!

Wrapping Up

In this article, we've seen how to use Lambda functions in Python, explored their pros and cons, some best practices, and touched on a couple of advanced use cases.

Happy coding!

P.S. If you'd like to read Python posts as soon as they get off the press, subscribe to our Python Wizardry newsletter and never miss a single post!

DEV Community: AppSignal

Find and Fix N+1 Queries in Django Using AppSignal

What Are N+1 Queries?

Prerequisites

Project Setup

Install AppSignal for Django

Web App Logic

Models

Views

Detect N+1 Queries in Your Django App with AppSignal

Detect N+1 Queries in Views

Example 1: One-To-One Relationship (select_related())

Example 2: Many-To-One Relationship (prefetch_related())

Detect N+1 Queries in Django Admin

Wrapping Up

Rack for Ruby: Socket Hijacking

HTTP Connections

Partial Hijacking

Full Hijacking

Streaming Bodies in Rack for Ruby

Falcon Web Server

Avoiding False Positives in Node.js Tests

Prerequisites

Common False Positive Test Patterns in Node.js

Using equal() Rather Than strictEqual()

Using Overly General Assertions

Using Shallow Equality for Deep Comparisons

Misunderstanding Assertion Behavior

Example: Playwright

Example: React Testing Library

Forgetting to Call a Matcher

Misusing Mocks

Using Regex Matching Incorrectly

Copy-Paste Errors

Using Incorrect Properties in Configuration Objects

Misusing Snapshot Tests

Wrapping Up

Streamlined Contract Testing in Node.js: A Simple and Achievable Approach

Pre-requisites

Testing a REST API with a Simplified Contract

Listing 1: A Test Spec That Makes an HTTP Request to the /posts Endpoint

Listing 2: Generating a Series of Jest Tests from Our Data

Mocking Your Database for Fast Contract Tests

Listing 3: The Mock Version of MongoDB

Listing 4: A Test That Loads the Data Fixture many-posts

Listing 5: Starting the Web Server and Loading Data Fixtures Before Each Test

Mocking Your Message Queue for Fast, Asynchronous Contract Tests

Listing 6: The Mock Version of AMQPlib Used to Replace RabbitMQ

Listing 7: A Test Triggers an Async Message Handler

Listing 8: Our Contract Testing Code Now Handles HTTP and Asynchronous Requests and Responses

Adding Contract Tests to Your CI/CD Pipeline

Listing 9: The GitHub Actions Workflow Runs Simplified Contract Tests for a Project

Bringing It Together: How I Applied Contract Testing

Wrapping Up

How to Track Errors in Oban for Elixir Using AppSignal

Prerequisites

Introducing Oban

Installing and Configuring Oban in Elixir

Automatic Instrumentation of Oban Using AppSignal

Defining the First Background Job

Oban Errors

Instrumenting Oban Errors Using AppSignal for Elixir

Wrapping Up

Optimize Database Performance in Ruby on Rails and ActiveRecord

Getting ActiveRecord N+1s Out of the Way

Database Performance in Rails: Considerations and Optimization

Database Indexing

Performance Profiling for Your Ruby on Rails App

Background Jobs

Read Replicas

Database Sharding At A High Level

Words of Caution and Final Words: Database Optimization in Ruby on Rails

Advanced Open edX Monitoring with AppSignal for Python

Streaming Logs to AppSignal

Create a Source

Monitor Celery and Redis using AppSignal

Practical Monitoring: Enhancing Open edX with AppSignal

Site Performance Improvement

Server Resource Monitoring

Custom Metrics

Example 1: One-To-One Relationship (`select_related()`)

Example 2: Many-To-One Relationship (`prefetch_related()`)

Using `equal()` Rather Than `strictEqual()`

Listing 1: A Test Spec That Makes an HTTP Request to the `/posts` Endpoint

Listing 4: A Test That Loads the Data Fixture `many-posts`

Using Lambda Functions with the `map()` Function

Using Lambda Functions with the `filter()` Function

Using Lambda Functions with the `sorted()` Function