DEV Community: Morgan Aubert

Digging into Marten query sets

Morgan Aubert — Fri, 17 Nov 2023 01:42:27 +0000

The Marten web framework incorporates an intuitive object-relational mapper (ORM), streamlining the development process for users who may not possess extensive knowledge of databases and SQL. To facilitate interactions with collections of model records, Marten introduces a dynamic mechanism called "Query sets". A query set represents a collection of model records in the database. It can have filters, be paginated, sliced, etc. In this guide, we will cover how to leverage such querying capabilities.

Quick setup

Before delving into our initial examples of query sets, let's first set up a small project that will serve as the foundation for this guide. Assuming that the Marten CLI is properly installed, we can use the marten new command to do so:

marten new project qset-showcase
cd qset-showcase && shards install

Next, let's define a very simple User model by creating a src/models/user.cr file and copying the following content in it:

class User < Marten::Model
  field :id, :big_int, primary_key: true, auto: true

  field :username, :string, max_size: 128
  field :email, :email
  field :first_name, :string, max_size: 128, blank: true, null: true
  field :last_name, :string, max_size: 128, blank: true, null: true

  with_timestamp_fields
end

This provides us a good opportunity to appreciate Marten's declarative approach to model definition. In Marten, models are articulated as subclasses of the Marten::Model base class, explicitly outlining their "fields" through the use of a #field macro.

Marten’s vision when it comes to models is that everything needed to understand a model should be defined within the model itself. In this light, Marten's models drive tables (and not the reverse way around) and migrations are generated automatically from model definitions. While migrations can still be manually written using a convenient DSL if needed, the core idea revolves around the simplicity of defining a model and its fields to ensure that the framework takes care of the associated table automatically.

To generate the migration for the User model that we defined above, we can leverage the marten genmigrations command:

marten genmigrations

This should produce the following output:

Generating migrations for app 'main':
  › Creating [src/migrations/202311151921341_create_main_user_table.cr]... DONE
      ○ Create main_user table

If we inspect the generated file, we should see the following content:

# Generated by Marten 0.3.3 on 2023-11-15 19:21:34 -05:00

class Migration::Main::V202311151921341 < Marten::Migration
  def plan
    create_table :main_user do
      column :id, :big_int, primary_key: true, auto: true
      column :username, :string, max_size: 128
      column :email, :string, max_size: 254
      column :first_name, :string, max_size: 128, null: true
      column :last_name, :string, max_size: 128, null: true
      column :created_at, :date_time
      column :updated_at, :date_time
    end
  end
end

This migration essentially takes care of creating the database table corresponding to the User model we defined previously. Should any adjustments be necessary for this model in the future, the marten genmigrations command can be executed as many times as needed. When doing so, Marten will generate new migration files for each change and will ensure that the generated migrations are interdependent so that they are applied in the right order.

We can now apply our automatically-generated migration by running the marten migrate command:

marten migrate

Which should output something along those lines:

Running migrations:
  › Applying main_202311151921341_create_main_user_table... DONE

And voilà! Our database is ready.

Before delving into query sets, let's finally open a Crystal playbook and let's create a few records. To do so, let's run the following command:

crystal play

We can now head to http://localhost:8080 and paste the following Crystal snippet into the live editor:

require "./src/project"
Marten.setup

User.create!(username: "johndoe", email: "john.doe@example.com")
User.create!(username: "mary", email: "mary01@example.com", first_name: "Mary", last_name: "Williams")

This essentially requires our Marten project and creates two User records. We can keep the Crystal playground open as we will be using it for running and testing queries in the following sections.

What are query sets?

A query set represents a specific query specified for a given model. It can involve filters, be paginated, etc. Unless a specific "write" operation is performed on such query sets, they will usually be mapped to a standard SELECT statement where filters are converted to WHERE clauses.

Query sets are lazily evaluated: defining a query set will usually not involve any database operations. Additionally, most methods provided by query sets also return new query set objects. Query sets are only translated to SQL queries hitting the underlying database when records need to be extracted or manipulated by the considered codebase.

For example:

qset = User.filter(first_name: "John") # the query set is not evaluated
qset = qset.filter(last_name: "Doe") # the query set is not evaluated
puts qset # the query set is evaluated

Basic queries

Let's start with some basic query set examples.

Querying all records

Retrieving all the records of a specific model can be achieved through the use of the #all query set method:

User.all

Filtering specific records

Filtering records can be achieved by leveraging the #filter query set method. For example:

User.filter(username: "johndoe")

Will return all User records whose username is johndoe.

It is possible to filter records by applying multiple filters. To do so, either multiple arguments can be specified to #filter or calls to #filter can be chained. For example, the following queries are equivalent:

User.filter(username: "johndoe").filter(first_name: "John")
User.filter(username: "johndoe", first_name: "John")

By default, filters involving multiple parameters like in the above examples always produce SQL queries whose parameters are "AND"ed together.

Retrieving single objects

Retrieving a specific record is achieved through the use of the #get! query set method. For example:

User.get!(username: "johndoe")
User.get!(id: 1)

As you can see, any kind of filters can be specified to this method, but only one record must be returned. If the record is not found, an exception is raised by this method (Marten::DB::Errors::RecordNotFound). If multiple records are found, another kind of exception is raised too (Marten::DB::Errors::MultipleRecordsFound).

It is also possible to use the #get method, which will simply return nil if the record is not found.

Retrieving the first or last record

The #first and #last query set methods can be used to retrieve the first or last record for a given query set. For example:

User.first
User.last

User.filter(username: "johndoe").first
User.filter(username: "johndoe").last

Ordering records

The #order query set method can be used to specify in which order records should be queried. Multiple fields can be specified in order to define the final ordering.

For example:

qset = User.all
qset.order("-created_at", "username")

In the above example, records would be ordered by descending creation date (because of the - prefix), and then by username (ascending).

Advanced queries

Now that we covered some of the fundamental query set operations, let's delve into more advanced examples.

Filter predicates

Filter predicates define the type of filtering that is done at the SQL level. They map to WHERE clauses in the produced SQL queries.

For example:

User.filter(username__icontains: "john")

Will translate to a SQL query like the following one (using PostgreSQL's syntax):

SELECT * FROM main_user WHERE username LIKE UPPER("john")

By default, unless explicitly specified, the exact predicate (exact match) is used for all filters. As such, the following two query sets are equivalent:

User.filter(username: "johndoe")
User.filter(username__exact: "johndoe")

Note the double underscores notation used in the above examples: __ is used to separate the filtered field name (here, username) from the type of filtering that is intended to be performed on the corresponding column.

Other predicates can be used, such as iexact (case-insensitive match), contains (containment test), startswith (starts-with test), etc:

User.filter(username__startswith: "john")
User.filter(username__contains: "do")

Complex filters with `q` expressions

As mentioned previously, field predicates expressed as keyword arguments to the #filter method will use an AND operator in the produced WHERE clauses. In order to produce conditions using other operators, it is necessary to use "q expressions".

In this light, most of the methods we mentioned previously like #filter or #get can receive a block allowing to define complex conditions. Inside of this block, a #q method can be used in order to define conditions nodes that can be combined together using the following logical operators: & (logical "AND"), | (logical "OR"), and - (logical negation).

For example, the following snippet will return all the User records whose username starts with "john" or "mary":

User.filter { q(username__startswith: "john") | q(username__startswith: "mary") }

Using this approach, it is possible to produce complex conditions by combining q expressions with the &, |, and - operators. Parentheses can also be used to group statements:

User.filter { 
  (q(first_name: "John") |  q(first_name: "Mary")) & 
  -q(last_name: "Doe")
}

Paginating results

Marten query sets provide a built-in pagination mechanism that you can leverage in order to easily iterate over records that are split across several pages of data. In this light, each query set has the capability to generate "paginator" objects that are initialized with a given page size and that can be used to request specific pages.

For example:

query_set = User.all

paginator = query_set.paginator(10)
paginator.page_size   # => 10
paginator.pages_count # => 6

# Retrieve the first page and iterate over the underlying records
page = paginator.page(1)
page.each { |user| puts user }

page.number               # 1
page.previous_page?       # => false
page.previous_page_number # => nil
page.next_page?           # => true
page.next_page_number     # => 2

As you can see, each page object also gives access to some useful metadata (sibling page numbers, etc) in order to build pagination features.

Performing raw SQL queries

Marten also lets you get fully instantiated records retrieved from raw SQL queries. This can be achieved by leveraging the #raw query set method.

For example, the following snippet would allow iterating over all the User model records:

User.raw("select * from main_user").each do |user|
  # Do something with `user` record
end

It is worth mentioning that is is possible to "inject" parameters into raw SQL queries. Both positional and named arguments are supported. Positional parameters must be specified using the ? syntax while named parameters must be specified using the :param format.

For example:

# Using positional arguments:
User.raw(
  "SELECT * FROM main_user WHERE first_name = ? and created_at > ?", 
  "John", 
  "2023-11-16"
)

# Using named arguments:
User.raw(
  "SELECT * FROM main_user WHERE first_name = :first_name and created_at > :created_at", 
  first_name: "John", 
  created_at: "2023-11-16"
)

Conclusion

In this article, we delved into Marten's user-friendly object-relational mapper (ORM) and its dynamic "Query sets" mechanism, designed to simplify database interactions in web application projects. We covered both fundamental and advanced query operations, demonstrating Marten's adaptability for building efficient, database-driven applications. It's important to note that our coverage only scratched the surface of Marten's capabilities, as we did not explore more complex scenarios involving relationships, joins, and other advanced query set functionalities.

Background job processing with Marten and Mosquito

Morgan Aubert — Sat, 08 Apr 2023 21:19:33 +0000

Marten is a web framework written in Crystal that makes building web applications easy and enjoyable. Mosquito is a background task runner for Crystal that uses Redis and that makes it easy to schedule and run tasks asynchronously.

Combining Marten with Mosquito can be a powerful way to build web applications with background tasks that run asynchronously. In this blog post, we'll show you how to get started with using Marten and Mosquito together.

Installation

Assuming that you already have a working Marten project that was initialized with the built-in authentication at hand, you can start by adding Mosquito to your shard.yml file:

dependencies:
  mosquito:
    github: mosquito-cr/mosquito

Then run shards install to install the new dependencies.

We can now ensure that Mosquito is properly required by our project. To do so, let's update the content of the src/project.cr file as follows:

# Third party requirements.
require "marten"
require "mosquito"
# other requirements...

# Project requirements.
require "./jobs/**"

# other requirements...

You'll notice that we require both mosquito and the files of a jobs folder, where we'll define the actual jobs.

Now we can create a new config/initializers/mosquito.cr initializer file where the actual Mosquito configuration will live. We won't cover all the Mosquito configuration options in this article, but this initializer could simply set the right Redis URL to use for now and could look something like this:

Mosquito.configure do |settings|
  settings.redis_url = (ENV["REDIS_URL"]? || "redis://localhost:6379")
end

Finally, we need to create a new file where we will (i) setup Marten itself and (ii) start the Mosquito runner. In this light, let's create a new src/worker.cr file with the following content:

require "./project"

Marten.setup

Mosquito::Runner.start

Defining jobs

For the purpose of this article, we will be defining jobs under a src/jobs folder. If your Marten project is split into multiple applications, you can also decide to create a jobs folder inside each of these applications.

So let's create a src/jobs folder for now and let's define a send_welcome_email_job.cr file in it with the following content:

class SendWelcomeEmailJob < Mosquito::QueuedJob
  params(user_id : Int64)

  def perform
    user = Auth::User.get!(id: user_id)
    user.send_welcome_email
    user.update!(welcome_email_sent: true)
  end
end

This example job simply retrieves a User model record using a user_id job parameter, sends an email by calling a hypothetical #send_welcome_email method, and updates the considered record by setting a hypothetical welcome_email_sent field to true.

Enqueuing jobs

Enqueuing a job is as simple as initializing an instance of a job class with the specified parameters and calling the #enqueue method. The job above was all about sending a welcome email to newly created users. As such, this job could be enqueued from a callback that we could define in the Auth::User model. For example:

module Auth
  class User < MartenAuth::User
    after_commit :trigger_welcome_email_sending, on: :create

    private def trigger_welcome_email_sending
      SendWelcomeEmailJob.new(user_id: id!.to_i64).enqueue
    end
  end
end

In this example, we are triggering the job from a model callback but it should be noted that you can enqueue jobs from pretty much anywhere in your Marten project codebase (eg. handlers, schemas, etc).

Running the Mosquito worker

As you might have guessed, the Mosquito worker runs in a dedicated process. This means that we have to explicitly start the runner we defined earlier by compiling and executing the src/worker.cr file. While in development, we can simply leverage the crystal run command to do so:

crystal run src/worker.cr

A few tips regarding deployments

At deployment time, it's likely that you already compile two binaries: your project's server (src/server.cr) and your project's management CLI (manage.cr). In addition to these, you will now need to ensure that your worker is compiled as well. This can be accomplished by relying on the crystal build command:

crystal build src/worker.cr -o bin/worker --release

You could also add a target to your shard.yml file and even trigger the execution of your compiled worker from a Procfile file depending on your deployment strategy. You can learn more about deploying Marten in the dedicated documentation.

Conclusion

In this blog post, we've gone through the steps of setting up Mosquito with Marten, including installation, configuration, and usage. Using Mosquito with the Marten web framework can help you build fast and scalable web applications that leverage background job processing capabilities.

The vision behind the Marten web framework

Morgan Aubert — Mon, 12 Dec 2022 02:11:29 +0000

Over the course of my career, I used to work extensively with two web frameworks: Django and Ruby on Rails. These two frameworks are regularly compared, and I can't even imagine the number of articles or discussions that touched on this subject over the years. The bottom line is that both frameworks have their strengths: in my opinion, Django shines with its design decisions and the general assumptions it makes about things like models, migrations, templates, routing, etc. Rails shines with its awesome DSLs (like callbacks, or validations) and the overall productivity it enables.

A few years ago, I was trying Sorbet for the first time and I was realizing how frustrating the developer experience provided by this tool was: horrible syntax with sig extensions (and duplicated method definitions), magic comments, separate RBI files generated from the codebase, ... Don't get me wrong: Sorbet certainly fulfills its mission of bringing typing to the Ruby ecosystem, and enabling most of the benefits that come with typed languages. But unfortunately, this is to the detriment of the developer experience because all of this makes Ruby lose its beauty and slickness.

Developer experience matters. And well, while I was reading more on Sorbet, I stumbled upon a language called Crystal. The timing couldn't have been better: Crystal - heavily inspired by Ruby's syntax - provided exactly the kind of typing syntax that I would've loved to rely on in a "typed" Ruby: elegant, intuitive, and expressive.

On top of that, Crystal featured a bunch of other very interesting characteristics:

A powerful compilation and the ability to catch typing errors at compile time
Meta-programming capabilities with macros
Very good performances (in a lot of benchmarks, Crystal regularly beats other compiled languages like Go or Elixir)

I literally fell in love with the language and started playing with it regularly. Soon I started playing with the existing options in terms of high-level web frameworks (mainly Amber and Lucky). I thought it would be fun to create a web framework that enabled design decisions similar to Django's while also leveraging Crystal's syntax and performances to enable an enjoyable and productive developer experience. That's how I started working on the Marten web framework.

What is Marten?

Marten is a Crystal Web framework that enables pragmatic development and rapid prototyping. It provides a consistent and extensible set of tools that developers can leverage to build web applications without reinventing the wheel.

Principles

Marten's development has been guided by a few key principles. As it stands, Marten focuses on the following aspects:

Simple and easy to use: Marten tries to ensure that everything it enables is as simple as possible and that the syntax provided for dealing with the framework's components remains obvious and easy to remember (and certainly not complex or obscure). The framework makes it as easy as possible to leverage its capabilities and perform CRUD operations.
Full-featured : Marten adheres to the "batteries included" philosophy. Out of the box, it provides the tools and features that are commonly required by web applications: ORM, migrations, translations, templating engine, sessions, and (soon) emailing and authentication.
Extensible: Marten gives developers the ability to contribute extra functionalities to the framework easily. Things like custom model field implementations, new route parameter types, session stores, etc... can all be registered to the framework easily.
DB-Neutral: The framework's ORM is usable with multiple database backends (including MySQL, PostgreSQL, and SQLite).
App-oriented: Marten allows separating projects into a set of logical "apps", which helps improve code organization and makes it easy for multiple developers to work on different components. Each app can contribute specific abstractions and features to a project like models and migrations, templates, HTTP handlers and routes, etc. These apps can also be extracted in Crystal shards in order to contribute features and behaviours to other Marten projects. The goal behind this capability is to allow the creation of a powerful apps ecosystem over time and to encourage "reusability" and "pluggability"
Backend-oriented: The framework is intentionally very "backend-oriented" because the idea is to not make too many assumptions regarding how the frontend code and assets should be structured, packaged or bundled together. The framework can't account for all the ways assets can be packaged and/or bundled together and does not advocate for specific solutions in this area. Some projects might require a webpack strategy to bundle assets, some might require a fingerprinting step on top of that, and others might need something entirely different. How these toolchains are configured or set up is left to the discretion of web application developers, and the framework simply makes it easy to reference these assets and collect them at deploy time to upload them to their final destination

What does it look like?

As mentioned previously, Marten provides support for many key components in order to help developers build web applications in a productive way. The most important ones certainly are models , templates , and handlers.

Models let you define what data can be persisted in your application:

class Article < Marten::Model
  field :id, :big_int, primary_key: true, auto: true
  field :title, :string, max_size: 128
  field :content, :text
  field :author, :many_to_one, to: User
end

Marten’s vision when it comes to models is that everything needed to understand a model should be defined within the model itself. In this light, Marten’s models drive tables (and not the reverse way around) and migrations are generated automatically from model definitions. Migrations can still be written manually using a convenient DSL if necessary, but the idea is that defining a model and its fields is all that is necessary in order to have a corresponding table taken care of automatically by the framework. This is a very convenient mechanism: most Crystal web frameworks require that you write the migrations corresponding to your model definitions manually; being able to rely on an automatic mechanism that does this for you by default is certainly a plus.

As highlighted in the above snippet, models explicitly define "fields" and relationships through the use of a unique and simple field macro. These fields can contribute database columns to the model table and they can be queried through the use of an automatically-generated database access API. One interesting aspect of model fields is that they operate at a higher level than columns: they can actually contribute features or validations to your models and they don't have to correspond to primitive types only. For example, an email model field could ensure that only email address values are allowed for a specific field while also contributing a string column at the database level.

Templates let you define your presentation logic:

{% extend "base.html" %}
{% block content %}
<ul>
  {% for article in articles %}
    <li>{{ article.title }}</li>
  {% endfor %}
</ul>
{% endblock content %}

Templates provide a convenient way of defining your presentation logic and writing contents (such as HTML) that are rendered dynamically. These renderings can involve model records or any other variables needed for the templates' requirements.

Marten's templates provide a Jinja-like syntax where you can use dynamic variables as well as some programming constructs and leverage relatively common patterns such as template tags or filters. This syntax is relatively common now and has been used in a lot of projects such as Django, Liquid, or Nunjucks. The advantage of relying on a dedicated templating engine is that templates don't assume Crystal knowledge. As such, the framework acknowledges the fact that such templates may be written by developers of various backgrounds that don't necessarily work on the backend of the project.

Moreover, these templates are parsed and rendered at runtime. As such, they are not tied to the compilation process of the project's Crystal binary. This makes them convenient to edit (because they don't require a compilation), which is a plus for projects involving lots of templates. Obviously, the use of these templates (although recommended by the framework) is optional and ECR (Crystal's compiled template language) can technically be used instead.

Handlers let you process HTTP requests:

class ArticleListHandler < Marten::Handler
  def dispatch
    render "articles/list.html", { articles: Article.all }
  end
end

Handlers are responsible for processing HTTP requests and for returning HTTP responses. They are the equivalent of a function that would take a request in and that would produce a response out. In the process they can do possibly anything: loading records from the database, rendering HTML templates, producing JSON payloads, ...

Routing in Marten only cares about URL paths. As such the framework does not route based on HTTP methods and it is the responsibility of the mapped handlers to implement the right logic based on the incoming request's verb. For example, a handler might display an HTML page containing a form when processing a GET request, and it might validate possible form data when handling a POST request.

Marten recognizes the existence of common or frequently encountered handler patterns, like the form example I just mentioned above. As such, the framework includes a set of generic handlers that can be leveraged to perform common tasks. These tasks are frequently encountered when working on web applications. For example: displaying a list of records extracted from the database, or deleting a record. Generic handlers take care of these common patterns so that developers - again - don't end up reimplementing the wheel.

Obviously, this only scratches the surface of what's possible with the Marten web framework. You can have a look at the documentation in order to learn more about other components or features such as routing (allowing to map your handlers to URL paths), schemas (allowing to validate request data), file management, internationalization, management commands, security helpers, assets management,...

Get started with Marten

Here are a few ideas on how you can get started with Marten:

Check out the Marten GitHub repository
Check out the official documentation
- The installation guide will help you install Crystal and the Marten CLI
- The introduction tutorial will help you discover the main features of the framework by creating a simple web application
Ask for help or chat with the community in our Discord

DEV Community: Morgan Aubert

Digging into Marten query sets

Quick setup

What are query sets?

Basic queries

Querying all records

Filtering specific records

Retrieving single objects

Retrieving the first or last record

Ordering records

Advanced queries

Filter predicates

Complex filters with q expressions

Paginating results

Performing raw SQL queries

Conclusion

Background job processing with Marten and Mosquito

Installation

Defining jobs

Enqueuing jobs

Running the Mosquito worker

A few tips regarding deployments

Conclusion

The vision behind the Marten web framework

What is Marten?

Principles

What does it look like?

Get started with Marten

Complex filters with `q` expressions