DEV Community: Ryan

Cleaning up queries with dot notation

Ryan — Wed, 16 Jun 2021 01:30:57 +0000

🔗 This article was originally posted, with additional formatting, on my personal blog at https://ryanc.co/blog/cleaning-up-queries-using-dot-notation

One of the many conventions found throughout the Laravel framework is dot notation. Dot notation is the use of dots . in strings to represent depth. In many instances, it allows for writing cleaner and more clear code. Take for example it's use case of accessing nested elements in an array using the Arr helper

Arr::get($array, 'user.profile.billing_details.address');

This not only looks nicer but it also takes care of handling undefined index errors and this method specifically has the added benefit of providing a default value as a third parameter. Use cases similar to this can be found in most Arr methods, most Collection methods, when validating arrays, and in most array_* helper methods (for those of you on legacy versions of Laravel) to name a few.

Eloquent & dot notation

Support for dot notation can can also be found throughout Laravel when interacting with Eloquent relations.

whereHas()

For example, say you want to query a User model for all records that have an associated Invoice record which has an associated Payment record. While this can be done a handful of ways, one way would be to use the whereHas() Eloquent Builder method, like so

User::whereHas('invoices', function ($invoices) {
    $invoices->whereHas('payment');
})->get();

Awesome! Task achieved. But consider the following alternative using dot notation

User::whereHas('invoices.payment')->get();

While the contrived example may only save you a few lines/characters, that is still characters saved, at no expense to the readability of the code — if anything, it's improved. This usage of dot notation can be used to clean up complex nested relationships that would otherwise result in deep chains of single-depth whereHas() methods. Most importantly, the resulting query is exactly the same between the two examples:

select * from `users` where exists (select * from `invoices` where `users`.`id` = `invoices`.`user_id` and exists (select * from `payments` where `invoices`.`payment_id` = `payments`.`id` and `payments`.`deleted_at` is null) and `invoices`.`deleted_at` is null) and `users`.`deleted_at` is null

As an added bonus, you still have the flexibility of breaking away from dot notation at any level of depth and passing a closure to the second parameter of whereHas() to add query constraints and even continue with dot notation as part of your constraints. Still, the resulting query will be exactly the same as it's single-depth, chained whereHas() counterpart.

with()

One usage of dot notation that I find more powerful than in whereHas() methods is in with() methods to eager load related models. Though, I often find myself questioning its inner-workings.

Using our previous example, say you want to not only fetch users with paid invoices, but you also want to fetch them with their related Invoice and Payment records eagerly loaded. One way to achieve this would be using single-depth, chained with() methods.

As you've probably guessed, you can use dot notation for that too! So I'll save you the time and get to the interesting part: say you only want to load these related models but only with select attributes to limit memory usage. Using the with() method without dot notation would look something like this

User::with('invoices', function ($invoices) {
    $invoices->select('id', 'amount', 'payment_id')
        ->with('payment', function ($payment) {
            $payment->select('id', 'paid_at');
        });
})->get();

Sprinkle on some dot notation magic and...

User::with('invoices:id,amount,payment_id', 'invoices.payment:id,paid_at')->get();

A clean, considerably easier to parse one-line equivalent. Clearly, this is more than dot notation alone and, as you can see, this additional functionality does not rely on dot notation. But, when paired, this duo can clean up otherwise long, deeply nested blocks of code.

ℹ️ One thing to be mindful of when writing queries that select specific columns in general is to be sure to select id and foreign key columns relating to any models you wish to load.

But... what's actually going on here? Is performance impacted, for each level of nesting are we re-querying the parent models? The most important thing to note is that both approaches result in the exact same query. Meaning there are no redundant queries being made or any other performance implications one way or the other.

select * from `users` where `users`.`deleted_at` is null;

select `id`, `amount`, `payment_id` from `invoices` where `invoices`.`user_id` in (?) and `invoices`.`deleted_at` is null;

select `id`, `paid_at` from `payments` where `payments`.`id` in (1) and `payments`.`deleted_at` is null;

Closing

Safe to say, where it makes sense to, dot notation is a safe and viable way to clean up some of your Laravel queries without running the risk of any performance implications as it does not effect the resulting query in any way. Hopefully this has saved you at least a few minutes of tinkering trying to answer these questions on your own.

🎉 Like the content? Feel free to let me know on Twitter where I share all of my articles, packages, and occasional opinions (mostly technical).

Understanding Laravel's SerializesModels trait

Ryan — Sat, 01 May 2021 17:51:26 +0000

🔗 This article was originally posted, with additional formatting, on my personal blog at https://ryanc.co/blog/understanding-laravels-serializesmodels

Background

When dispatching an object onto the queue, behind the scenes Laravel is recursively serializing the object and all of its properties into a string representation that is then written to the queue. There it awaits a queue worker to retrieve it from the queue and unserialize it back into a PHP object (Phew!).

Problem

When complicated objects are serialized, their string representations can be atrociously long, taking up unnecessary resources both on the queue and application servers.

Solution

Because of this, Laravel offers a trait called SerializesModels which, when added to an object, finds any properties of type Model or Eloquent\Collection during serialization and replaces them with a plain-old-PHP-object (POPO) known as a ModelIdentifier. These identifier objects represent the original properties Model type and ID, or IDs in the case of an Eloquent\Collection, with a much smaller string representation when serialized. When these objects are unserialized, the ModelIdentifiers are then replaced with the Model or Eloquent\Collection of Models that they temporarily represented.

ℹ️ Curious to know how the SerializesModels trait is "replacing" these properties at runtime? Before jumping into the source code, you may want to read the PHP docs for a quick primer on what the reflection API offers. For a more detailed explanation including examples of how Laravel uses reflection check out this article.

Gotcha!

🗣 Because of the SerializesModels trait that the job is using, Eloquent models and their loaded relationships will be gracefully serialized and unserialized when the job is processing.

While this quote from the docs sounds promising, it can be misleading. Here is an example of how a Model would be represented when serialized.

As you can see, the relation is serialized as records, which is the name of a one-to-many relation between User and Record as it exists on the User model. However, even though we're only selecting the id and user_id columns and have a limit of 3 records to be returned with the User model, there's no mention of which records, which properties, or how many in the serialized representation.

Let's see what the query log looks like when we unserialize this object.

Wow! We're selecting all Record models, in their entirety, associated to the User! As you can probably imagine, this can cause unforeseen issues for the unexpecting artisan.

Workarounds

Hope is not lost! There are workarounds without having to sacrifice resources on the queue, application, or database servers.

Unload unnecessary relations

If you don't need the loaded relations to be re-loaded, you can simply call withoutRelations() on your Model before it's serialized.

As you can see, there are no longer any relations that will be loaded when the User model is unserialized.

Make necessary relations their own property

If the loaded relation is going to be required after the Model is unserialized you can store the relation (which is an Eloquent\Collection) as its own property on the object being serialized.

As you can see, the serialized representation of the Eloquent\Collection specifies what the Model type is and which IDs need to be retrieved from the database.

Much better! Now that our Model and Eloquent\Collection of Records have been hydrated, we can even set the relation back on the Model by using the setRelation method.

Recap

While it's recommended to use the SerializesModels trait on any and all objects that will be queued (or otherwise serialized) it is crucial to be at least aware of its potential pitfalls and shortcomings as well as how to avoid running into them, if not understand how it all works under the hood.

🎉 Like the content? Feel free to let me know on Twitter where I share all of my articles, packages, and occasional opinions (mostly technical).

Simple Scaling with Redis Lists

Ryan — Thu, 29 Apr 2021 20:29:06 +0000

This article was originally posted, with additional formatting, on my personal blog at https://ryanc.co/blog/simple-scaling

Keeping It Simple

Although simple and powerful, it's no secret that relational databases have their limitations. So what happens when simple doesn't cut it, and your relational database falls short? You might try throwing more servers at it, reaching for replication, or load balancing which would assist in scaling read-intensive operations. But how would you go about scaling write-intensive operations?

Take for example some of the problems I've faced working with clients in the Adtech space:

Tracking impressions across an ad network
Monitoring user behaviors and actions
Tracking traffic to and from affiliate networks
Capturing spikes of traffic resulting in periodic loads 10x higher than normal

All of these have required high data ingestion rates that couldn't be achieved the same way that you would remedy read-intensive bottlenecks. Rather than rearchitecting from the ground up, in each of these instances, I was able to keep it simple and lean in on tools that were already familiar and readily available.

Redis is an [...] in-memory data structure store, used as a database, cache, and message broker.

Although commonly used for caching the output of CPU intensive or time consuming tasks, Redis' use cases extend far beyond that. Redis supports various data types including hashes, lists, sets/sorted sets, and of course strings. Each offering the same speed and scalability that you've come to expect from the blazing fast, in-memory, highly scalable data store.

We're going to look at an example of how to use Redis Lists for fast data ingestion to then process and persist the data to a relational database later on.

Redis Lists are simply lists of strings, sorted by insertion order.

Lists support a maximum length of over 4 billion elements and, as long as you're accessing those elements near the head and tail of the list, maintain constant time insertion and deletion. You can think of Lists as a feature-limited extension to your programming language's array or list data type.

Because of this, Redis Lists are great for fast data ingestion. We can use Lists as a "data channel" to push data into in real-time and then consume, process, and ingest the data into our relational database in more performant batches.

All of the below pseudo code samples are in PHP and make references to components of the Laravel framework. However, the same methodologies can be applied to your language/framework of choice.

Filename: AdSpotController.php
Description: Push data from an incoming request onto the end of the list as requests come in.

public function __invoke(Request $request) 
{
    Redis::rpush('impressions', json_encode([
        'spot_id' => $request->spot_id,
        // more data points...
        'timestamp' => time(),
    ]));

    // ...
}

Filename: ImpressionListConsumer.php

Description: Pop data off the end of the list in chunks of 1,000 and process it before storing it in the relational database. Scheduled to run at a fixed interval (i.e. every 10 minutes).

public function handle()
{
    while ($chunk = Redis::rpop('impressions', 1000) {
        DB::transaction(function () use ($chunk) {
            collect($chunk)
                ->map(fn($row) => json_decode($row, true))
                ->transform(...) // do some data processing...
                ->mapToGroups(...) // maybe summarize high polarity data
                ->each(function ($group) {
                    Impression::createMany($group);
                });
        });
    }
}

Closing

While these examples aren't exactly functional, they do accurately depict how simple it is to get started using Redis Lists for fast data ingestion as a means to scaling your relational database driven application without introducing any unnecessary complexity.

If there's one thing to take away from this article, aside from how powerful Redis is, the next time you're facing a scaling problem or technical limitation due to the constraints of your stack, consider exploring what is already available to you rather than introducing any additional complexity.

But wait...

If you’re curious about how far you can push this, I have easily scaled apps to tens of thousands of writes/second with a single 4gb Redis instance. Redis’ enterprise offering claims to be able to support up to 200m writes/second should you need that.

Like the content? Feel free to follow and interact with me on Twitter where I share all of my articles, packages, and occasional opinions (mostly technical).

The power of static in PHP

Ryan — Mon, 23 Sep 2019 18:42:08 +0000

The power of static

Although most have probably seen the static keyword used in PHP before, have you ever wondered how many different ways it can be used? Or possibly how many ways these different uses can provide value to your code? Let's dive into uses and examples and find out.

Method signature

Arguably the most common usage of the static keyword is the static method. While static methods can be referred to using the object operator (->) it is recommended to use the scope resolution operator (::) as the alternative is deprecated and may be removed in the future. The scope resolution operator allows you to call static methods directly on the class rather than an instance of the class. This also results in the $this keyword being unavailable in the body of static methods.

Static methods can be used to implement the Factory Method Pattern which acts as a factory for producing new instances of the containing class each time the method is called.

In this example we have created a factory method fromArray which instantiates a User object, assigns the instance values from the array and returns the instance.

class User
{
    public static function fromArray($attributes)
    {
        $user = new User();
        $user->name = $attributes['first'] . ' ' . $attributes['last'];
        $user->password = password_hash($attributes['password'], PASSWORD_BCRYPT);

        return $user;
    }
}

$user = User::fromArray([
    'first' => 'ryan',
    'last' => 'cco',
    'password' => 'password'
]);

Alternatively, this logic can be extracted to a standalone class which is also known as the Static Factory Pattern.

class UserFactory
{
    public static function build($attributes)
    {
        //
    }
}

$user = UserFactory::build($attributes);

Properties

Unlike regular properties, changing the value of a static property during the execution of your program will affect all instances of the containing class. Even those which have yet to be instantiated. In this way, static properties can be thought of as "mutable class constants". Static properties are only able to be referenced using the scope resolution operator.

Due to the nature of static properties, they can be used to implement the Singleton Pattern. The Singleton Pattern maintains a single instance of a given class throughout the execution of your program.

In this example, the initial call to Queue::getInstance() will create and assign an instance of Queue to Queue::$instance and return it. Every subsequent call will return the same instance of Queue previously assigned to Queue::$instance.

class Queue
{
    private static $instance;

    public static function getInstance()
    {
        if (static::$instance === null) {
            static::$instance = new Queue();
        }

        return static::$instance;
    }
}

Variables

When in context of a function, static variables maintain their value even once the program has moved out of scope of the containing function. When in the context of a class method, static variables have additional behavior similar to static properties in that changes to their values reflect across all instances of the containing class. Although similar in behavior to static properties, static variables are only accessible within a function or method body.

Static variables are often used as part of an optimization technique known as Memoization. Memoization aims to speed up an otherwise expensive operation by caching the results and saving them for a later call with the same parameters.

In this example, we are creating a hash which is unique to the parameters provided and use it to uniquely identify the call as $key. If the value of $key is not found as an index on $cache, we perform the preg_replace and store its output on the $key index of $cache. Every subsequent call to replace with the same parameters will bypass the call to preg_replace and return the value from a previous call.

function replace($pattern, $replacement, $subject)
{
    static $cache = [];

    $key = md5(serialize(func_get_args()));

    if (! isset($cache[$key])) {
        $cache[$key] = preg_replace(
            $pattern, $replacement, $subject
        );
    }

    return $cache[$key];
}

Anonymous functions

Like methods, when an anonymous function is defined in the context of a class they are not bound to the containing class and do not have the $this keyword available.

Static anonymous functions do not necessarily have any unique uses over regular anonymous functions. It is worth noting that in context of a class, static anonymous functions yield micro performance improvements and may be suggested in any instances which you do not need the containing class to be bound to the anonymous function.

class PostsCollectionFilter
{
    public function unpublished($posts)
    {
        return $posts->filter(static function ($post) {
            return ! $post->published;
        });
    }
}

Late Static Binding (LSB)

Late static binding demonstrates another usage of the static keyword: in the context of inheritance. In this context static refers to the class being called rather than that which the method was defined (which would be referred to with self or __CLASS__).

class A
{
    public function saySelfValue()
    {
        echo self::class;
    }

    public function sayClassValue()
    {
        echo __CLASS__;
    }

    public function sayStaticValue()
    {
        echo static::class;
    }
}

class B extends A
{

}

$a = new A();
$a->saySelfValue(); // 'A'
$a->sayStaticValue(); // 'A'
$a->sayClassValue(); // 'A'

$b = new B();
$b->saySelfValue(); // 'A'
$b->sayStaticValue(); // 'B'
$b->sayClassValue(); // 'A'

Closing

We've covered a few examples, some more contrived than others, of the different uses of the static keyword in PHP. Although contrived, the use cases are all very real and the value that they can deliver to your code can be hugely impactful. At the very least, familiarizing yourself with these uses and techniques to have available in your toolbox will make you that much stronger of a developer.

If you have some more real-world examples, possibly from open source projects, that could provide insights for other readers please share links in the comments!

Originally posted on my personal site at ryanc.co/blog/the-power-of-static

Faster Eloquent Chunking

Ryan — Wed, 11 Sep 2019 21:51:35 +0000

As your applications scale, processing large amounts of database records with Laravel Eloquent can become increasingly difficult. Resulting in out of memory exceptions and overall slowing down your application. Why is that?

When fetching results from the database you are in turn pulling that data into memory. Take this snippet of code for instance

Post::all()->each(function ($post) {
    // ...
});

Which results in the following query, loading all records in the posts table into memory

select * from posts;

Typically for tables with a small number of records, this is absolutely acceptable. However, as you accumulate tens of thousands of posts you will eventually begin hitting memory resource constraints of your webserver.

Chunking

A common approach in Laravel is to use Eloquent's (via BuildsQuery) chunk() method which fetches a fixed amount of records breaking a larger set into more consumable chunks.

Post::chunk(1000, function ($post) {
    // ...
});

While this might seem fine there are both improvements and gotchas to be aware of.

First, imagine the following scenario: you are fetching Post records from the database to update an attribute that you are also using in a where clause

Post::where('published_at', '<', now())->chunk(1000, function ($post) {
    $post->update('published_at', now());
});

Although contrived, this exemplifies a very real problem where such a query would result in an endless loop as the published_at attribute will always be less than now() at the time of the queries next execution (assuming accuracy to the second as with MySQL's timestamp column type or similar).

Second, there is the matter of the queries performance and its impact on the database server. The above code would result in a query similar to the following

select * from posts order by posts.id asc limit 1000 offset 9000

MySQL is unable to go directly to the offset due to deleted records and additional query constraints and, therefore, this query has to effectively select the first 10,000 records to return only the last 1,000 selected. As you can imagine this will not scale well into many-thousands or even millions of rows. This will cause your database server to use unnecessary resources slowing down all other queries from your application.

Chunking... but better!

In order to both prevent the unforeseen gotcha and improve database server performance we can use Eloquent's chunkById method

Post::where('published_at', '<', now())->chunkById(1000, function ($post) {
    $post->update('published_at', now());
});

The above code snippet will result in a query similar to the following

select * from posts where published_at < '2019-09-11 12:00:00' and id > 9000 order by id asc limit 1000

Why is this approach considered "better"?

a) it allows MySQL to completely skip the first 9000 (assumingly sequential) records
b) we will no longer be re-selecting records which we have already updated due to the id constraint in our where clause

Bonus - How?! 🤔

Diving into the chunkyById method of the BuildsQueries trait we see that the id of the last record fetched (remember, we're ordering by id in ascending order) is stored and used as a parameter in the next query to be run.

Laravel Eloquent Model UUID Route Keys

Ryan — Mon, 09 Sep 2019 19:33:22 +0000

Without arguing the reason for the packages conception, auto-incrementing keys in URLs are often discussed as an anti-pattern, security flaw, and generally ugly (see here, here, here for a few).

Laravel UUID Models provides a trait that, when applied to a Laravel Eloquent Model, seamlessly handles allowing it to be routed using UUID's (rather than auto-incrementing ID's).

Additionally, there are no further code changes required for generating the time ordered UUID, validation, type-casting, or editing any existing routes or links*. It's as simple as installing the package, placing the trait on the eloquent models you would like to use UUID route keys, and adding a migration to store the UUID field.

Any suggestions or changes are always encouraged and welcomed on the repository.

* All routes will work assuming they're generated with the route() helper.