DEV Community: Arunanshu Biswas

Unlocking Conditional Component Rendering with Vue.js Slots

Arunanshu Biswas — Mon, 22 May 2023 08:07:00 +0000

Introduction

Vue.js is a powerful framework that allows developers to build interactive and flexible user interfaces. One of its notable features is the ability to work with slots, which enable the creation of reusable components. In this guide, we will explore how to leverage Vue.js conditional slots to create dynamic layouts that adapt based on the parent's usage of specific slots.

Prerequisites

Before we dive into conditional slots, make sure you have a basic understanding of Vue.js and its component-based architecture. Familiarity with HTML and JavaScript will also be beneficial. We will use Vue.js 3 here. Some knowledge of TypeScript is recommended, but not mandatory.

Creating the Slotted Component

To begin, let's create a slotted component that will serve as the container for our conditional slots. In this example, we'll call it Card. Here's a basic template for the component:

<!-- ./components/Card.vue -->
<template>
  <div class="card">
    <div v-if="$slots.annotated" class="annotation">
      <slot name="annotated" />
    </div>
    <slot />
  </div>
</template>

<style>
/* some basic styling */
.card {
  padding: 20px;
  border: 1px solid #ccc;
  border-radius: 5px;
  margin: 10px;
}

.annotation {
  background-color: lightblue;
  padding: 10px;
  border-radius: 5px;
  margin-bottom: 10px;
}
</style>

Explanation

We define the parent component with the class .card as the container. It has a border and padding to provide a basic visual structure.
Using v-if="$slots.annotated", we check if the parent component uses the annotated slot. If it does, the annotation div will be rendered.
Inside the annotation div, we use <slot name="annotated" /> to define the location where the content of the annotated slot will be inserted.
The .annotation class applies a background color and padding to visually distinguish the annotated content.

Understanding the `$slots` variable

The $slots variable is automatically available within a Vue.js component and represents an object that holds the component's slots. The $slots object has properties corresponding to each slot's name, allowing you to conditionally render content based on the existence of a specific slot.

To learn more about slots in Vue.js, you can visit the following links:

Vue.js Documentation: Slots: This document provides detailed information about slots in Vue.js, including default slots, named slots, and scoped slots.
Vue.js Documentation: $slots: Here, you can find the documentation specifically for the $slots variable in Vue.js, which explains its usage and provides examples.

Using the Slotted Component

Now that we have our slotted component defined, let's see how we can utilize the conditional slot. We will use the component within our App.vue file:

<!-- App.vue -->
<script setup lang="ts">
import Card from './components/Card.vue';
</script>

<template>
  <Card>
    Without Annotation.<br />
    Note that no extra styling was applied.
  </Card>
  <Card>
    <template #annotated>With Annotation</template>
    But this is without annotation again.
  </Card>
</template>

In the <script setup> section, we import the Card component. Note that we utilize the Composition API's <script setup>, which you can read more about here.
Inside the <template> section, we use the Card component by placing it within the template. We have two instances of the Card component.
Without Annotation: The first instance of the Card component is used without an annotated slot. Note that no extra styling is applied.
With Annotation: The second instance of the Card component includes the annotated slot. We define the content for the slot using <template #annotated>. Note the styling applied.

The Results

Conclusion

In this tutorial, we have explored the concept of conditional slots in Vue.js and learned how to utilize them effectively. Conditional slots provide a powerful mechanism for dynamic component rendering and styling, allowing the parent component to selectively render specific content within the child component based on given conditions. By leveraging this feature, you can create more flexible and reusable components in your Vue.js applications.

Remember, conditional slots are just one of the many powerful features Vue.js offers. It's always a good idea to explore the official Vue.js documentation and experiment with different features to further enhance your understanding and proficiency in building robust web applications.

The Async Side of SQLModel Relationships — Part 2

Arunanshu Biswas — Wed, 01 Feb 2023 13:00:55 +0000

So far, we've looked at how to use foreign keys and relationship attributes to create database relationships. We've also seen how doing an implicit SQL I/O throws errors. Next, we are going to cover the various ways in which we can fix the errors.

TL;DR

Eager Loading Approach

Add sa_relationship_kwargs={"lazy": LAZY_STRATEGY} to Relationship(...).

Values for LAZY_STRATEGY have been described in SQLAlchemy's docs.

Lazy Loading Approach

Generate queries using select(User).where(User.id == 1).options(LOADER_API(User.pets)).

Values for LOADER_API have been described in SQLAlchemy's docs on Relationship Loader API.

Goal

Our goal remains the same. We want to create a pet management app.

In part 3 of this blog series, we discussed how to create relationships between SQLModel models.

Setting Up the Environment

We use Poetry for dependency management.

Our project environment looks like this:

sqlmodel-alembic-async/
├── sqlmodel_alembic_async/
│   ├── __init__.py
│   ├── databases.py
│   └── models.py
├── poetry.lock
└── pyproject.toml

At the time of writing, the dependencies are:

# pyproject.toml
[tool.poetry.dependencies]
python = "^3.8"
sqlmodel = "^0.0.8"
alembic = "^1.9.2"
aiosqlite = "^0.18.0" # you can use some other async database engine

Make sure you activate the Poetry shell using poetry shell.

Recap

>>> # ...
>>> user_someone.pets
INFO sqlalchemy.engine.Engine SELECT pet.id AS pet_id, pet.name AS pet_name, pet.user_id AS pet_user_id 
FROM pet 
WHERE ? = pet.user_id
INFO sqlalchemy.engine.Engine [generated in 0.00017s] (3,)

MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)

According to the log message, SQLAlchemy is successfully constructing a query to retrieve pets with a foreign key called user_id equal to 3. However, on attempting to execute the query, an error occurs, indicating a failure in initiating a Greenlet coroutine, which is used for concurrent programming, most likely due to an implicit I/O operation not being performed in an async context.

Lazy or Not

To solve the problem, we can take two approaches:

Load everything in one go. Also called eager loading.
Load as required. Lazy loading, but with async support.

Load Everything in One Go

This is how the strategy works:

You query the database for an instance of a User using the ORM.

>>> db_user = await db.get(User, 1) # this means: select user with id = 1
INFO sqlalchemy.engine.Engine BEGIN (implicit)
INFO sqlalchemy.engine.Engine SELECT user.name AS user_name, user.id AS user_id 
FROM user 
WHERE user.id = ?
INFO sqlalchemy.engine.Engine [generated in 0.00021s] (1,)

(the logs continue below)

The ORM issues an additional query to fetch the pets owned by the User.

INFO sqlalchemy.engine.Engine SELECT pet.user_id AS pet_user_id, pet.id AS pet_id, pet.name AS pet_name 
FROM pet 
WHERE pet.user_id IN (?)
INFO sqlalchemy.engine.Engine [generated in 0.00042s] (1,)
>>> db_user.pets
[...]

Different types of loading-related strategies in SQLAlchemy's ORM include lazy loading, select-in loading, joined loading, raise loading, subquery loading, write-only loading, and dynamic loading.

The default style is lazy loading, which loads a related reference on a single object at a time.
The select-in loading method loads all members of a linked collection/scalar references at the same time.
Joined loading loads rows that are related in the same result set.
Raise loading raises an exception if an unwanted lazy load occurs.
Subquery loading uses a subquery to load all members of a related collection/scalar reference.
Write-only loading only allows modification to a collection and requires a SELECT statement to query the collection.
Dynamic loading generates a Query object for the collection, however, it is less effective for managing big collections and is being phased out in favour of write-only loading.

The settings are fully explained in SQLAlchemy's documentation.

Also, the loading strategy you decide to use depends heavily on the use case. Each of them has its benefits and tradeoff. Start by looking into N+1 selects problem faced by an ORM.

We will stick with selectin loading strategy for our application.

# sqlmodel_alembic_async/models.py
# ...

class User(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

    pets: List["Pet"] = Relationship(
        back_populates="user",
+       sa_relationship_kwargs={"lazy": "selectin"},
    )

class Pet(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

    user_id: int | None = Field(default=None, foreign_key="user.id")
    user: Optional[User] = Relationship(
        back_populates="pets",
+       sa_relationship_kwargs={"lazy": "selectin"},
    )

And now you can query the database as usual.

>>> from sqlmodel import select
>>> stmt = select(Pet).where(Pet.id == 1)
>>> pet = (await db.exec(stmt)).one()
>>> pet
Pet(user_id=1, name='phroge', id=1, user=User(id=1, name='chonky', pets=[Pet(user_id=1, name='phroge', id=1, user=User(id=1, name='chonky', pets=[...]))]))
>>> pet.user
User(id=1, name='chonky', pets=[Pet(user_id=1, name='phroge', id=1, user=User(id=1, name='chonky', pets=[...]))])

Benefits

Very simple to implement.

Drawbacks

If the related objects are big, then loading them unnecessarily would waste CPU cycles.
It has been described as a code smell.

Load as Required

In this strategy, we will use the loader strategies described above within the query itself.

>>> from sqlmodel import select
>>> from sqlalchemy.orm import selectinload
>>>
>>> stmt = select(User).where(User.id == 5).options(selectinload(User.pets))
>>>
>>> db_user = (await db.exec(stmt)).one()
>>>
>>> db_user
User(id=5, name='waby', pets=[Pet(user_id=5, name='bb', id=4, user=User(id=5, name='waby', pets=[...]))])

Let's break down the above code.

>>> stmt = select(User).where(User.id == 5).options(selectinload(User.pets))

Here, we ask the database to select a User whose ID is equal to 1. Additionally, issue an additional query using selectinload strategy that loads the related objects called pets that is accessed via User.pets attribute.

Where did the selectinload() come from? It is a part of SQLAlchemy's Relationship Loader API.

>>> db_user = (await db.exec(stmt)).one()

Then we execute the statement and pick one item from the result.

And that's the "Load as Required" strategy.

Benefits

Truly lazy strategy: You load what you need.

Drawbacks

Complex to work with

A "Lazy" Tip

Should you choose to implement the "Load as Required" approach, it is possible to configure SQLAlchemy to raise an exception in the event of an unwanted implicit I/O operation being triggered by attribute access.

In the code below, we use the lazy="raise" strategy. Raise loading raises an exception if an unwanted lazy load occurs.

# sqlmodel_alembic_async/models.py
# ...

class User(UserBase, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

    pets: List["Pet"] = Relationship(
        back_populates="user",
+       sa_relationship_kwargs={"lazy": "raise"},
    )

class Pet(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

    user_id: int | None = Field(default=None, foreign_key="user.id")
    user: Optional[User] = Relationship(
        back_populates="pets",
+       sa_relationship_kwargs={"lazy": "raise"},
    )

What's next?

Now that we've learnt how to take responsibility for our relationships 😌, we will learn how to integrate it into a working FastAPI application.

Image Source

Photo by Stephen Meyers on Unsplash

The Async Side of SQLModel Relationships — Part 1

Arunanshu Biswas — Tue, 31 Jan 2023 17:51:06 +0000

So far, we've covered how to perform database operations asynchronously. Next, we are going to create relationships between models and discover a related pitfall.

TL;DR

Create the foreign key in a model using sqlmodel.Field.
Migrate the database using alembic revision --autogenerate -m "migration message" and alembic upgrade head.
Add relationship attributes using sqlmodel.Relationship(back_populates="column").

Since step 3 has a trap that needs careful attention, you might wish to read the article.

Goal

Our goal remains the same. We want to create a pet management app.

In part 2 of this blog series, we discussed how to perform database operations asynchronously.

Setting Up the Environment

We use Poetry for dependency management.

Our project environment looks like this:

sqlmodel-alembic-async/
├── sqlmodel_alembic_async/
│   ├── __init__.py
│   ├── databases.py
│   └── models.py
├── poetry.lock
└── pyproject.toml

At the time of writing, the dependencies are:

# pyproject.toml
[tool.poetry.dependencies]
python = "^3.8"
sqlmodel = "^0.0.8"
alembic = "^1.9.2"
aiosqlite = "^0.18.0" # you can use some other async database engine

Make sure you activate the Poetry shell using poetry shell.

Creating Relationships with SQLModel

Add a Foreign Key

So, the problem is this: one user can own multiple pets. Or in other words: one pet is owned by one user.

We can see that it's a one-to-many relationship. Understand this with a diagram:

              is owned by
1 Pet ────────────────────────────► 1 User


                     ┌───────► Pet
                     │
           owns      │
1 User ──────────────┼───────► Pet  (multiple Pets)
                     │
                     │
                     └───────► Pet

In the database, we use a foreign key to represent our one-to-many relationship.

# sqlmodel_alembic_async/models.py
from typing import Optional
from sqlmodel import Field, SQLModel

metadata = SQLModel.metadata


class User(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str


class Pet(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str
+   user_id: Optional[int] = Field(default=None, foreign_key="user.id")

Create Revision and Migrate

Create a Revision

This step is pretty easy.

alembic revision --autogenerate -m "add Pet.user_id foreign key"
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.autogenerate.compare] Detected added column 'pet.user_id'
# INFO  [alembic.autogenerate.compare] Detected added foreign key (user_id)(id) on table pet
#  Generating ./sqlmodel-alembic-async/migrations/versions/<somehash>_add_pet_user_id_foreign_key.py ...  done

Inspect the Revision

# migrations/versions/<somehash>_add_pet_user_id_foreign_key.py
"""add Pet.user_id foreign key

Revision ID: <some hash>
Revises: <some hash>
Create Date: <some date>
"""
from alembic import op
import sqlalchemy as sa
import sqlmodel.sql.sqltypes


# revision identifiers, used by Alembic.
revision = '<some hash>'
down_revision = '<some hash>'
branch_labels = None
depends_on = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    with op.batch_alter_table('pet', schema=None) as batch_op:
        batch_op.add_column(sa.Column('user_id', sa.Integer(), nullable=True))
        batch_op.create_foreign_key(batch_op.f('fk_pet_user_id_user'), 'user', ['user_id'], ['id'])

    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    with op.batch_alter_table('pet', schema=None) as batch_op:
        batch_op.drop_constraint(batch_op.f('fk_pet_user_id_user'), type_='foreignkey')
        batch_op.drop_column('user_id')

    # ### end Alembic commands ###

Examine the following lines:

        # ...
        batch_op.create_foreign_key(batch_op.f('fk_pet_user_id_user'), 'user', ['user_id'], ['id'])
        # ...

This name fk_pet_user_id_user is generated using the naming convention we created in part 1 of this blog series.

Migrate the Database

alembic upgrade head
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.runtime.migration] Running upgrade <some hash> -> <some hash>, add Pet.user_id foreign key

Playing with Database Models

Create a Session

Before we begin, we must create a session and bind it with an engine. In part 2 of this blog series, we saw how to create an async session.

>>> from sqlmodel.ext.asyncio.session import AsyncSession
>>> from sqlmodel_alembic_async.databases import engine
>>>
>>> session = AsyncSession(engine)

We are now ready to experiment with our database models.

The Experiment

First, we import the database models.

>>> # ...
>>> from sqlmodel_alembic_async.models import User, Pet

Let's create a User called "chonky". She happens to own a pet called "phroge".

>>> # ...
>>> user_chonky = User(name="chonky")
>>> pet_frog = Pet(name="phroge")

To link the user to the pet, we would need the value of the id of user_chonky. But our user does not have an id yet. So, we need to commit it to the database.

>>> session.add(user_chonky)
>>> await session.commit(user_chonky)
INFO sqlalchemy.engine.Engine BEGIN (implicit)
INFO sqlalchemy.engine.Engine INSERT INTO user (name) VALUES (?)
INFO sqlalchemy.engine.Engine [generated in 0.00016s] ('chonky',)
INFO sqlalchemy.engine.Engine COMMIT

And then we refresh the user_chonky object to update its attributes.

>>> await session.refresh(user_chonky)
INFO sqlalchemy.engine.Engine BEGIN (implicit)
INFO sqlalchemy.engine.Engine SELECT user.name, user.id 
FROM user 
WHERE user.id = ?
INFO sqlalchemy.engine.Engine [generated in 0.00018s] (1,)
>>>
>>> print(user_chonky)
User(id=2, name='chonky')

And boom! We now have a value of the id!

Where is the INFO sqlalchemy... log coming from?

SQLAlchemy is logging its operations since we passed echo=True to create_async_engine. This has been described in part 2 of the blog series.

>>> pet_frog = user_chonky.id
>>> await session.commit()
>>> # ... log messages skipped ...
>>> await session.refresh(pet_frog)
>>> # ... log messages skipped ...
>>> print(pet_frog)
Pet(id=1, name='phroge', user_id=2)

And remember to close the database session once you're done.

>>> await session.close()

And tada! We successfully created a relationship between two objects using SQLModel.

However, using the id to create relationships seems cumbersome. It would be nice if we could somehow interact todo

Relationship Attributes

Using relationship attributes, we can directly interact with the users and their pets using familiar python attributes.

todo

# sqlmodel_alembic_async/models.py
# ...
+from typing import List, Optional

class User(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

+   pets: List["Pet"] = Relationship(back_populates="user")


class Pet(SQLModel, table=True):
    id: int | None = Field(default=None, primary_key=True)
    name: str

    user_id: int | None = Field(default=None, foreign_key="user.id")
+   user: Optional[User] = Relationship(back_populates="pets")

The List["Pet"] and Optional[User] are type hints. It is assumed that you know what type hints are. SQLModel's docs go into detail on using type hints and relationship attributes.

Using Models with Relationship Attributes

Start by importing the required classes and instances.

>>> from sqlmodel_alembic_async.models import User, Pet
>>> from sqlmodel_alembic_async.databases import engine
>>> from sqlmodel.ext.asyncio.session import AsyncSession
>>>
>>> session = AsyncSession(engine)

We create a user and its pet.

>>> # ...
>>> user_someone = User(name="someone")
>>> pet_something = Pet(name="something")

Now we tell pet_something that its "owner" is user_someone.

>>> # ...
>>> pet_something.user = user_someone

Let's see what the pet_something and user_someone look like.

>>> # ...
>>> pet_something
Pet(id=None, name='something', user_id=None, user=User(name='someone', id=None, pets=[Pet(id=None, name='something', user_id=None, user=User(name='someone', id=None, pets=[...]))]))
>>>
>>> user_someone
User(name='someone', id=None, pets=[Pet(id=None, name='something', user_id=None, user=User(name='someone', id=None, pets=[...]))])

What's going on? Why the [...]? And is that a recursive object?

Yes! It is indeed a recursive object. It shows that SQLModel understands that there is a relationship (one-to-many, to be precise) between the objects.

Understand it like this:

              is owned by
1 Pet ────────────────────────────► 1 User


                     ┌───────► Pet
                     │
           owns      │
1 User ──────────────┼───────► Pet  (multiple Pets)
                     │
                     │
                     └───────► Pet

Now let's add it to the database, commit it and then refresh the objects.

>>> # ...
>>> session.add_all((user_someone, pet_something))
>>> await session.commit()
... # log message skipped
>>> await session.refresh(user_someone)
... # log message skipped
>>> await session.refresh(pet_something)
... # log message skipped

Let's try to inspect user_someone.

>>> # ...
>>> user_someone.pets
INFO sqlalchemy.engine.Engine SELECT pet.id AS pet_id, pet.name AS pet_name, pet.user_id AS pet_user_id 
FROM pet 
WHERE ? = pet.user_id
INFO sqlalchemy.engine.Engine [generated in 0.00017s] (3,)

MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)

Oh no! It fails with some obscure MissingGreenlet error. What is going on!?

Analysing the Attribute Access Error

Let's inspect the log message and the error message.

>>> # ...
>>> user_someone.pets
INFO sqlalchemy.engine.Engine SELECT pet.id AS pet_id, pet.name AS pet_name, pet.user_id AS pet_user_id 
FROM pet 
WHERE ? = pet.user_id
INFO sqlalchemy.engine.Engine [generated in 0.00017s] (3,)

In the log message, it is evident that SQLAlchemy makes a query to the database. It is trying to fetch the pets that have the foreign key user_id equal to 3. That means queries are being built without any failure. So far, so good.

MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)

But this error says that some "IO operation was carried out when it was not expected." Also, it seems to use Greenlet. Looks like the error happens when SQLAlchemy tries to execute the query.

The documentation for Greenlet describes itself as a library for concurrent programming. Taking our error into account, we may conclude that SQLAlchemy attempted but failed to start a Greenlet coroutine. This shows that the error occurs when SQLAlchemy attempts to do an implicit I/O operation while not in an "async context."

What's next?

Now that we've learnt how to create relationships between models and how performing an SQL query outside an "async context" leads to errors, we will look at some ways to fix the errors.

Or you can say, we will improve our relationship with Async SQL operations. 😏

Image Source

Photo by fabio on Unsplash

Async Database Operations with SQLModel

Arunanshu Biswas — Mon, 30 Jan 2023 14:30:03 +0000

So far, we've covered how to manage database migrations with SQLModel and Alembic. Next, we are going to perform database operations asynchronously.

TL;DR

Install an async-powered database engine (like, aiosqlite, asyncpg etc.)
```
poetry add aiosqlite
```

Set up a database URL that includes the name of the async database engine.

>>> DATABASE_URL = "sqlite+aiosqlite:///path/to/database.db"
>>> # Note the use of aiosqlite

Create a database engine with sqlalchemy.ext.asyncio.create_async_engine().

>>> from sqlalchemy.ext.asyncio import create_async_engine
>>> engine = create_async_engine(DATABASE_URL)

Use the engine to create a database session using sqlmodel.ext.asyncio.session.AsyncSession.

>>> from sqlmodel.ext.asyncio.session import AsyncSession
>>> async with AsyncSession(engine) as session:
...     # perform database operations

Use the session to perform database operations.

>>> async with AsyncSession(engine) as session:
...     db_user = models.User(name="chonk")
...     session.add(db_user)
...     await session.commit()
...     await session.refresh(db_user)
>>> print(db_user.id)
2

Goal

Our goal remains the same. We want to create a basic pet management app.

In part 1 of this blog series, we discussed how to manage database migrations with SQLModel and Alembic. We created two models, User and Pet.

Setting Up the Environment

We use Poetry for dependency management.

Our project environment looks like this:

sqlmodel-alembic-async/
├── sqlmodel_alembic_async/
│   ├── __init__.py
│   └── models.py
├── poetry.lock
└── pyproject.toml

At the time of writing, the dependencies are:

# pyproject.toml
[tool.poetry.dependencies]
python = "^3.8"
sqlmodel = "^0.0.8"
alembic = "^1.9.2"
aiosqlite = "^0.18.0"

Make sure you activate the Poetry shell using poetry shell.

Installing an Async DB Engine

In this tutorial, we use SQLite as the database with aiosqlite as the database engine. Install it using:

poetry add aiosqlite

If you're using a different database, you may utilise alternative engines like asyncpgfor Postgres and asyncmy for MySQL or MariaDB, among others.

Create Async Engine

# sqlmodel_alembic_async/databases.py
from sqlalchemy.ext.asyncio import create_async_engine
from sqlmodel.ext.asyncio.session import AsyncSession

DATABASE_URL = "sqlite+aiosqlite:///./data.db"

engine = create_async_engine(DATABASE_URL, echo=True)

A better solution would be to derive the DATABASE_URL from the environment. However, that is out of the scope of this tutorial.

Create Async Session

NOTE: In this step, and the following steps we are going to use REPL. However, Python's default REPL does NOT have support for bare awaitstatements.

It is recommended that you use IPython.

We use the engine created in the previous step and bind it to the AsyncSession.

>>> from sqlmodel.ext.asyncio.session import AsyncSession
>>> from sqlmodel_alembic_async.databases import engine
>>>
>>> session = AsyncSession(engine)

Performing Database Operations

Now, we just have to create the model objects. We created the model objects in part 1.

>>> # ...
>>> from sqlmodel_alembic_async.models import User, Pet
>>> user_chonky = User(name="chonky")
>>> pet_frog = Pet(name="phroge")

We add our object to the session and commit to the database using session.commit().

>>> # ...
>>> # Use the session to perform database operations
>>> session.add_all((user_chonky, pet_tiger))
>>> await session.commit()
INFO sqlalchemy.engine.Engine BEGIN (implicit)
INFO sqlalchemy.engine.Engine INSERT INTO user (name) VALUES (?)
INFO sqlalchemy.engine.Engine [generated in 0.00023s] ('chonky',)
INFO sqlalchemy.engine.Engine INSERT INTO pet (name) VALUES (?)
INFO sqlalchemy.engine.Engine [generated in 0.00018s] ('phroge',)
INFO sqlalchemy.engine.Engine COMMIT

Where is the INFO sqlalchemy... log coming from?

SQLAlchemy is logging its operations since we passed echo=True to create_async_engine.

Once committed, the objects must be refreshed with session.refresh().

>>> # ...
>>> await session.refresh(user_chonky)
INFO sqlalchemy.engine.Engine BEGIN (implicit)
INFO sqlalchemy.engine.Engine SELECT user.name, user.id
FROM user 
WHERE user.id = ?
INFO sqlalchemy.engine.Engine [generated in 0.00016s] (3,)
>>>
>>> await session.refresh(pet_tiger)
INFO sqlalchemy.engine.Engine SELECT pet.id, pet.name 
FROM pet 
WHERE pet.id = ?
INFO sqlalchemy.engine.Engine [generated in 0.00018s] (2,)

Let's inspect our model objects:

>>> # ...
>>> print(user_chonky)
User(id=3, name="chonky")
>>> print(pet_frog)
Pet(id=2, name="phroge")

And once you're done with it, you can close the session.

>>> # ...
>>> await session.close()
INFO sqlalchemy.engine.Engine ROLLBACK

You can also use the session object as an async context manager like this:

>>> async with AsyncSession(engine) as session:
...     db_user = models.User(name="chonk")
...     session.add(db_user)
...     await session.commit()
...     await session.refresh(db_user)
...     # perform some other operation

What's next?

Now that we've learnt how to use database operations asynchronously, we'll look at how to use SQLModel to construct relationships between models.

Using SQLModel with Alembic

Arunanshu Biswas — Sun, 29 Jan 2023 20:36:35 +0000

TL;DR

Install Alembic and SQLModel.
Create models with SQLModel.SQLModel base class.
Add naming conventions to SQLModel.metadata using SQLModel.metadata.naming_convention.
Initialize Alembic with alembic init migrations.
Add your database URL to the alembic.ini configuration file.
In migrations/env.py, set target_metadata to SQLModel.metadata. This should be done after the Model classes have been imported.
Add the render_as_batch=True parameter to context.configure(...) in the function run_migrations_online of the migrations/env.py file.
Add the user_module_prefix="sqlmodel.sql.sqltypes." parameter to context.configure(...) in the function run_migrations_online of the migrations/env.py file.
Add import sqlmodel.sql.sqltypes in migrations/script.py.mako.
Now make changes in your data models and run alembic revision --autogenerate -m "some change message".
Upgrade the database using alembic upgrade head.

The Goal

We want to create a basic Pet manager app.

Setting Up the Environment

This step is optional.

We use Poetry for package management.

Our project environment looks like this:

sqlmodel-alembic/
├── sqlmodel_alembic/
│   ├── __init__.py
│   └── models.py
├── poetry.lock
└── pyproject.toml

At the time of writing, the dependencies are:

# pyproject.toml
[tool.poetry.dependencies]
python = "^3.8"
sqlmodel = "^0.0.8"
alembic = "^1.9.2"

Installing Alembic and SQLModel

Install alembic and SQLModel with your preferred package manager. I prefer Poetry.

poetry add alembic SQLModel

Make sure you activate the Poetry shell using poetry shell before executing any Alembic command.

Creating Models Using SQLModel

We create two entities: User and Pet.

# sqlmodel_alembic/models.py
from typing import Optional
from sqlmodel import Field, SQLModel

metadata = SQLModel.metadata


class User(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str


class Pet(SQLModel, table=True):
    id: Optional[int] = Field(default=None, primary_key=True)
    name: str

SQL Constraint Naming Convention

This step is optional but recommended.

Consider utilising a naming convention when adding SQL constraints to help prevent migration errors (especially with SQLite).

# sqlmodel_alembic/models.py
# ...
NAMING_CONVENTION = {
    "ix": "ix_%(column_0_label)s",
    "uq": "uq_%(table_name)s_%(column_0_name)s",
    "ck": "ck_%(table_name)s_%(constraint_name)s",
    "fk": "fk_%(table_name)s_%(column_0_name)s_%(referred_table_name)s",
    "pk": "pk_%(table_name)s",
}

metadata = SQLModel.metadata
metadata.naming_convention = NAMING_CONVENTION
# ...

Alembic's documentation explains the importance of naming conventions.

Initialize Alembic Migrations Directory

alembic init migrations
# Creating directory ./sqlmodel-alembic/migrations ...  done
# Creating directory ./sqlmodel-alembic/migrations/versions ...  done
# Generating ./sqlmodel-alembic/migrations/README ...  done
# File ./sqlmodel-alembic/alembic.ini already exists, skipping
# Generating ./sqlmodel-alembic/migrations/env.py ...  done
# Generating ./sqlmodel-alembic/migrations/script.py.mako ...  done
#Please edit configuration/connection/logging settings in './sqlmodel-alembic/alembic.ini' before proceeding.

This generates alembic.ini and migrations directory. After executing the command, our project directory looks like this:

sqlmodel-alembic/
├── migrations/
│   ├── versions/
│   ├── env.py
│   ├── README
│   └── script.py.mako
├── sqlmodel_alembic/
│   ├── __init__.py
│   └── models.py
├── alembic.ini
├── poetry.lock
└── pyproject.toml

Add Database URL to `alembic.ini`

# alembic.ini
[alembic]
# ... some other configuration above 👆
sqlalchemy.url = sqlite:///./data.db
# ... some other configuration below 👇

A better way to do this is by getting the database URL from the environment. However, that is out of the scope of this tutorial.

Set target metadata

# migrations/env.py
# ...
import sqlmodel_alembic.models
target_metadata = sqlmodel_alembic.models.metadata
# ...

Note that we use the metadata variable from the sqlmodel_alembic.models module itself. Why? We want metadata to store information about our models during runtime.

Configure `env.py`

Batch Rendering Mode

Miguel Grinberg's blog outlines the importance of batch rendering and why it is NOT a magic bullet.

With SQLite, the ALTER TABLE command only supports adding or renaming columns. Any other change to columns or constraints is going to be rejected with an error.

We instruct Alembic to perform batch operations by configuring run_migrations_online() like this:

# migrations/env.py
# ...
def run_migrations_online() -> None:
    # ...
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=target_metadata,
+           render_as_batch=True,
        )
# ...

You can also include render_as_batch=True in run_migrations_offline, however, this has not been tested.

What is the difference between offline and online migration?

Taken from Alembic's documentation:

Generation of migrations as SQL scripts, instead of running them against the database - is also referred to as offline mode.

SQLModel Custom Datatypes

SQLModel uses its custom datatypes and we need to instruct Alembic to include them in migration scripts.

For that, we add the use_module_prefix argument to context.configure() in run_migrations_online().

# ...
def run_migrations_online() -> None:
    # ...
    with connectable.connect() as connection:
        context.configure(
            connection=connection,
            target_metadata=target_metadata,
            render_as_batch=True,
+           user_module_prefix="sqlmodel.sql.sqltypes.",
        )
# ...

You can also include user_module_prefix="sqlmodel.sql.sqltypes." in run_migrations_offline, however, this has not been tested.

For more information, refer to Alembic's documentation on custom datatypes.

Adding SQLModel Import to `script.py.mako`

Finally, we include the actual SQLModel datatypes import in migrations/script.py.mako.

# ...
from alembic import op
import sqlalchemy as sa
+import sqlmodel.sql.sqltypes
${imports if imports else ""}
# ...

For more information, refer to Alembic's documentation on custom datatypes.

Create the Initial Migration

alembic revision --autogenerate -m "initial"
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.autogenerate.compare] Detected added table 'pet'
# INFO  [alembic.autogenerate.compare] Detected added table 'user'
#  Generating ./sqlmodel-alembic/migrations/versions/<somehash>_initial.py ...  done

The generated migration script should look approximately like this:

# migrations/versions/<somehash>_initial.py
"""initial

Revision ID: <somehash>
Revises: 
Create Date: 2023-01-29 19:39:24.556269

"""
from alembic import op
import sqlalchemy as sa
import sqlmodel.sql.sqltypes


# revision identifiers, used by Alembic.
revision = '<somehash>'
down_revision = None
branch_labels = None
depends_on = None


def upgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.create_table('pet',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('name', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
    sa.PrimaryKeyConstraint('id')
    )
    op.create_table('user',
    sa.Column('id', sa.Integer(), nullable=False),
    sa.Column('name', sqlmodel.sql.sqltypes.AutoString(), nullable=False),
    sa.PrimaryKeyConstraint('id')
    )
    # ### end Alembic commands ###


def downgrade() -> None:
    # ### commands auto generated by Alembic - please adjust! ###
    op.drop_table('user')
    op.drop_table('pet')
    # ### end Alembic commands ###

Migrate our Database

alembic upgrade head
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.runtime.migration] Running upgrade  -> <somehash>, initial

And done! You are now harnessing the power of Alembic and SQLModel to manage your database.

What's next?

Next, we are going to use SQLModel with Asyncio.

Solve cuInit: CUDA_ERROR_UNKNOWN: unkown error

Arunanshu Biswas — Sun, 26 Jun 2022 12:33:01 +0000

Steps

Enable the following services in systemd:
- nvidia-suspend.service
- nvidia-resume.service
- nvidia-hibernate.service
Enable via sudo systemctl enable <service name>.
Create a file named /etc/modprobe.d/nvidia-suspend.conf.

Add the following line to the file:

options nvidia NVreg_PreserveVideoMemoryAllocations=1

Reboot

This shall prevent the cuInit: CUDA_ERROR_UNKNOWN: unknown error.

If that does not work

Execute the following:

sudo modprobe -r nvidia_uvm # or: sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm

Making a proper init.lua (For real this time!)

Arunanshu Biswas — Thu, 27 Jan 2022 19:52:39 +0000

TL;DR

Just the config? Click here.

Aim

Our aim is to create a single file configuration for Neovim. There are too many tutorials on nEsTeD cOnFiGs. Real men use a single file to configure an entire editor 😤.

Flat is better than nested.
― The Zen of Python

Setting up `Packer`

Downloading

First, we are going to install Packer on our system.

local install_path = vim.fn.stdpath('data') .. '/site/pack/packer/start/packer.nvim'

if vim.fn.empty(vim.fn.glob(install_path)) > 0 then
  vim.fn.execute(
    '!git clone https://github.com/wbthomason/packer.nvim ' .. install_path
  )
end

There are three things involved here:

Specifying where to install Packer.
Checking if the installation directory is already empty.
Using Git to download Packer.

Building Packer's Lazy Loader

Next, we tell Neovim to call PackerCompile every time the buffer has been written to.

vim.cmd [[
augroup Packer
  autocmd!
  autocmd BufWritePost init.lua PackerCompile
augroup end
]]

What PackerCompile does is not essential for us to know at this stage.

Installing Plugins

We need two things for installing packages. The packer.use() function, and the packer.startup() function.

Note that the packer.startup() function takes a callable as an argument.

use = require("packer").use
require("packer").startup(function()
  -- Package manager
  use 'wbthomason/packer.nvim'

  -- our plugins go here!
end)

Now we can start filling the body of the callable.

Adding Packages

For starters, we need two things. One, something to set up Neovim's native LSP Client and two, something to download actual Language Servers.

LSP Client and Installer

You don't need to understand what LSP is. Just know that it allows you to use IDE like features like autocompletion, error checking and code suggestions.

use = require("packer").use
require("packer").startup(function()
  -- ...
  -- LSP Client
  use 'neovim/nvim-lspconfig'

  -- Language Server installer
  use {
    'williamboman/nvim-lsp-installer',
    requires = 'neovim/nvim-lspconfig',
  }

  -- BONUS: Customizations over LSP
  -- Show VSCode-esque pictograms
  use 'onsails/lspkind-nvim'
  -- show various elements of LSP as UI
  use {'tami5/lspsaga.nvim', requires = {'neovim/nvim-lspconfig'}}
  -- ...
end)

But note that LSP alone does not provide the cool features like code completion, hints etc. We need a separate autocomplete engine plugin for that.

Autocompletion Engine

For an autocompletion engine, we are going to use nvim-cmp. As a bonus, we will also configure it with extra plugins like snippet expansion, Vim command autocompletion, buffer completion etc.

-- ...
require("packer").setup(function()
  -- ...
  -- Autocompletion plugin
  use {
    'hrsh7th/nvim-cmp',
    requires = {
      'hrsh7th/cmp-nvim-lsp',
      'hrsh7th/cmp-buffer',
      'hrsh7th/cmp-path',
      'hrsh7th/cmp-cmdline',
    }
  }

  -- snippets
  use {
    'hrsh7th/cmp-vsnip', requires = {
      'hrsh7th/vim-vsnip',
      'rafamadriz/friendly-snippets',
    }
  }
  -- ...
end)

Auto bracket pairing

Let neovim close the brackets for you.

-- ...
require("packer").setup(function()
  -- ...
  -- bracket autocompletion
  use 'vim-scripts/auto-pairs-gentle'
  -- ...
end)
-- ...

Statusline

Nothing better than a fast and feature-packed statusline.

-- ...
require("packer").setup(function()
  -- ...
  -- Fancier statusline
  use {
    'nvim-lualine/lualine.nvim',
    requires = {
      'kyazdani42/nvim-web-devicons',
      'arkav/lualine-lsp-progress',
    },
  }
  -- ...
end)
-- ...

Incremental Parsing

We will also use an incremental parsing library for Neovim.

-- ...
require("packer").setup(function()
  -- ...
  -- Fast incremental parsing library
  use 'nvim-treesitter/nvim-treesitter'
  -- ...
end)
-- ...

And a Treesitter powered colorscheme ✨

-- ...
require("packer").setup(function()
  -- ...
  -- Beautiful colorscheme
  use 'navarasu/onedark.nvim'
end)
-- ...

Configuring Essential Neovim Options

Not much is here to describe. They will later prove to be useful once you start editing stuff.

-- ...
-- Do not show current vim mode since it is already shown by Lualine
vim.o.showmode = false

-- enable autowriteall
vim.o.autowriteall = true

-- Show the line numbers
vim.wo.number = true

-- Show chars at the end of line
vim.opt.list = true

-- Enable break indent
vim.o.breakindent = true

--Case insensitive searching UNLESS /C or capital in search
vim.o.ignorecase = true
vim.o.smartcase = true

-- Decrease update time
vim.o.updatetime = 250

-- Shows signs by Autocompletion plugin
vim.wo.signcolumn = 'yes'

-- Enable termguicolors. Very essential if you want 24-bit RGB color in TUI.
vim.o.termguicolors = true

-- Set completeopt to have a better completion experience
vim.o.completeopt = 'menu,menuone'

--Remap for dealing with word wrap
vim.api.nvim_set_keymap('n', 'k', "v:count == 0 ? 'gk' : 'k'", { noremap = true, expr = true, silent = true })
vim.api.nvim_set_keymap('n', 'j', "v:count == 0 ? 'gj' : 'j'", { noremap = true, expr = true, silent = true })

-- Highlight on yank
vim.cmd [[
augroup YankHighlight
autocmd!
autocmd TextYankPost * silent! lua vim.highlight.on_yank()
augroup end
]]
-- ...

Configuring Plugins

Now comes the big part. We will configure each plugin and set it up properly.

LSP Client and Installer

First, we import the required plugins. Then, we list the Language Servers that we need. Feel free to add or remove items from the list!

-- ...
require('lspconfig')
local lsp_installer = require("nvim-lsp-installer")

-- The required servers
local servers = {
  "bashls",
  "pyright",
  "rust_analyzer",
  "sumneko_lua",
  "html",
  "clangd",
  "vimls",
  "emmet_ls",
}
-- ...

Next we tell nvim-lsp-installer to autoinstall the required servers.

-- ...
for _, name in pairs(servers) do
  local server_is_found, server = lsp_installer.get_server(name)
  if server_is_found and not server:is_installed() then
    print("Installing " .. name)
    server:install()
  end
end
-- ...

Sweet! Now we can configure the essential keymaps. The on_attach() function given below hosts the keymaps and options that will be enabled only in the presence of an active Language Server.

Almost all of the keymaps are compatible with coc.nvim. So for those of you who are migrating from coc.nvim, you won't have to change anything!

Here, we are going to use our beloved lspsaga for fancy UI over LSP!

-- ...
local on_attach = function(_, bufnr)
  -- Create some shortcut functions.
  -- NOTE: The `vim` variable is supplied by Neovim.
  local function buf_set_keymap(...)
    vim.api.nvim_buf_set_keymap(bufnr, ...)
  end
  local function buf_set_option(...)
    vim.api.nvim_buf_set_option(bufnr, ...)
  end

  -- Enable completion triggered by <c-x><c-o>
  buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

  local opts = {noremap=true, silent=true}

  -- ======================= The Keymaps =========================
  -- jump to definition
  buf_set_keymap('n', 'gd', '<cmd>lua vim.lsp.buf.definition()<CR>', opts)

  -- Format buffer
  buf_set_keymap('n', '<F3>', '<cmd>lua vim.lsp.buf.formatting()<CR>', opts)

  -- Jump LSP diagnostics
  -- NOTE: Currently, there is a bug in lspsaga.diagnostic module. Thus we use
  --       Vim commands to move through diagnostics.
  buf_set_keymap('n', '[g', ':Lspsaga diagnostic_jump_prev<CR>', opts)
  buf_set_keymap('n', ']g', ':Lspsaga diagnostic_jump_next<CR>', opts)

  -- Rename symbol
  buf_set_keymap('n', '<leader>rn', "<cmd>lua require('lspsaga.rename').rename()<CR>", opts)

  -- Find references
  buf_set_keymap('n', 'gr', '<cmd>lua require("lspsaga.provider").lsp_finder()<CR>', opts)

  -- Doc popup scrolling
  buf_set_keymap('n', 'K', "<cmd>lua require('lspsaga.hover').render_hover_doc()<CR>", opts)
  buf_set_keymap('n', '<C-f>', "<cmd>lua require('lspsaga.action').smart_scroll_with_saga(1)<CR>", opts)
  buf_set_keymap('n', '<C-b>', "<cmd>lua require('lspsaga.action').smart_scroll_with_saga(-1)<CR>", opts)

  -- codeaction
  buf_set_keymap('n', '<leader>ac', "<cmd>lua require('lspsaga.codeaction').code_action()<CR>", opts)
  buf_set_keymap('v', '<leader>a', ":<C-U>lua require('lspsaga.codeaction').range_code_action()<CR>", opts)

  -- Floating terminal
  -- NOTE: Use `vim.cmd` since `buf_set_keymap` is not working with `tnoremap...`
  vim.cmd [[
  nnoremap <silent> <A-d> <cmd>lua require('lspsaga.floaterm').open_float_terminal()<CR>
  tnoremap <silent> <A-d> <C-\><C-n>:lua require('lspsaga.floaterm').close_float_terminal()<CR>
  ]]
end
-- ...

Now that we have configured our keymaps, we are going to add some server specific options. Here a default is provided. Don't worry if it looks intimidating. You will soon find out how it works.

-- ...
local server_specific_opts = {
  sumneko_lua = function(opts)
    opts.settings = {
      Lua = {
        -- NOTE: This is required for expansion of lua function signatures!
        completion = {callSnippet = "Replace"},
        diagnostics = {
          globals = {'vim'},
        },
      },
    }
  end,

  html = function(opts)
    opts.filetypes = {"html", "htmldjango"}
  end,
}
-- ...

Now we instruct nvim-lsp-installer to supply the on_attach, extra flags and server specific options to the Language Server once it is ready.

We will also use the feature rich nvim-cmp plugin for its extra capabilities.

-- ...
-- `nvim-cmp` comes with additional capabilities, alongside the ones
-- provided by Neovim!
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities = require('cmp_nvim_lsp').update_capabilities(capabilities)

lsp_installer.on_server_ready(function(server)
  -- the keymaps, flags and capabilities that will be sent to the server as
  -- options.
  local opts = {
    on_attach = on_attach,
    flags = {debounce_text_changes = 150},
    capabilities = capabilities,
  }

  -- If the current surver's name matches with the ones specified in the
  -- `server_specific_opts`, set the options.
  if server_specific_opts[server.name] then
    server_specific_opts[server.name](opts)
  end

  -- And set up the server with our configuration!
  server:setup(opts)
end)
-- ...

Cool! Now that we have set up the server and configured it properly, we should set the autocompletion plugin too!

Auotcompletion Plugin

Now it's time to set up the autocompletion plugin. We will start by importing the plugin and defining some helper functions.

-- ...
local lspkind = require('lspkind')
local cmp = require("cmp")

local has_words_before = function()
  local line, col = unpack(vim.api.nvim_win_get_cursor(0))
  return col ~= 0 and vim.api.nvim_buf_get_lines(0, line - 1, line, true)[1]:sub(col, col):match("%s") == nil
end

local feedkey = function(key, mode)
  vim.api.nvim_feedkeys(vim.api.nvim_replace_termcodes(key, true, true, true), mode, true)
end
-- ...

Next, we will define some beautiful glyphs. These glyphs will then be shown by lspkind.

-- ...
local cmp_kinds = {
  Text = "",
  Method = "",
  Function = "",
  Constructor = "",
  Field = "ﰠ",
  Variable = "",
  Class = "ﴯ",
  Interface = "",
  Module = "",
  Property = "ﰠ",
  Unit = "塞",
  Value = "",
  Enum = "",
  Keyword = "",
  Snippet = "",
  Color = "",
  File = "",
  Reference = "",
  Folder = "",
  EnumMember = "",
  Constant = "",
  Struct = "פּ",
  Event = "",
  Operator = "",
  TypeParameter = "",
}
-- ...

Next we set up formatting (and by extension, lspkind), snippet support, completion keymaps and completion sources.

This is where all our configuration for nvim-cmp will go:

-- ...
cmp.setup({
  -- Add your configuration here!
})
-- ...

First, the snippet support. Note that this is mandatory.

-- ...
cmp.setup({
  snippet = {
    expand = function(args)
      vim.fn["vsnip#anonymous"](args.body)
    end,
  },
  -- ...
})
-- ...

Next, the formatting.

-- ...
cmp.setup({
  -- ...
  formatting = {
    format = lspkind.cmp_format({
      with_text = true,
      preset = 'codicons',
      symbol_map = cmp_kinds, -- The glyphs will be used by `lspkind`
      menu = ({
        buffer = "[Buffer]",
        nvim_lsp = "[LSP]",
        luasnip = "[LuaSnip]",
        nvim_lua = "[Lua]",
        latex_symbols = "[Latex]",
      }),
    }),
  },
  -- ...
})
-- ...

Third, the custom keymaps for autocompletion suggestions. Feel free to change the keymaps as required. Note that the Tab key has been set up in such a way that you can use it to browse through the completion suggestions.

-- ...
cmp.setup({
  -- ...
  mapping = {
    ['<C-p>'] = cmp.mapping.select_prev_item(),
    ['<C-n>'] = cmp.mapping.select_next_item(),
    ['<C-d>'] = cmp.mapping.scroll_docs(-4),
    ['<C-f>'] = cmp.mapping.scroll_docs(4),
    ['<C-Space>'] = cmp.mapping.complete(),
    ['<C-e>'] = cmp.mapping.close(),
    ['<CR>'] = cmp.mapping.confirm {
      behavior = cmp.ConfirmBehavior.Replace,
      select = true,
    },

    -- Use Tab and Shift-Tab to browse through the suggestions.
    ["<Tab>"] = cmp.mapping(function(fallback)
      if cmp.visible() then
        cmp.select_next_item()
      elseif vim.fn["vsnip#available"](1) == 1 then
        feedkey("<Plug>(vsnip-expand-or-jump)", "")
      elseif has_words_before() then
        cmp.complete()
      else
        fallback()
      end
    end, { "i", "s" }),

    ["<S-Tab>"] = cmp.mapping(function()
      if cmp.visible() then
        cmp.select_prev_item()
      elseif vim.fn["vsnip#jumpable"](-1) == 1 then
        feedkey("<Plug>(vsnip-jump-prev)", "")
      end
    end, { "i", "s" }),
  },
  -- ...
})
-- ...

And finally, we specify the completion sources.

-- ...
cmp.setup({
  -- ...
  sources = {
    { name = 'nvim_lsp' },
    { name = 'vsnip' },
    { name = 'buffer' },
  },
})
-- ...

And finally, we will also configure autocompletion for Vim commands and Vim search.

-- ...
-- Use buffer source for `/`
cmp.setup.cmdline('/', {
  sources = {
    { name = 'buffer' }
  }
})

-- Use cmdline & path source for ':'
cmp.setup.cmdline(':', {
  sources = cmp.config.sources({
    { name = 'path' }
  }, {
    { name = 'cmdline' }
  })
})
-- ...

Wow! That was a loooot of work for a simple autocompletion plugin's configuration. But, as we will see, our hard work will pay off!

Next, we configure the remaining plugins.

Auto Bracket Pairing

Save yourself some keystrokes!

-- ...
vim.g.AutoPairs = {
  ['(']=')',
  ['[']=']',
  ['{']='}',
  ["'"]="'",
  ['"']='"',
  ['`']='`',
  ['<']='>',
}
-- ...

Colorscheme ✨

Bless your eyes.

-- ...
require("onedark").setup({
  style = "darker",
})
require('onedark').load()
-- ...

Status bar

A status bar that can change its colour automatically based on the colorscheme? Yes please!

-- ...
require('lualine').setup({
  sections = {
    lualine_c = {
      {'filename', path = 1},
      'lsp_progress',
    },
  },
})
-- ...

Treesitter

We will set up treesitter for a select number of languages. Feel free to add or remove items from the list.

-- ...
require('nvim-treesitter.configs').setup {
  ensure_installed = {"python", "rust", "c", "cpp", "bash", "go", "html"},
  highlight = {
    enable = true, -- false will disable the whole extension
  },
}
-- ...

LSPSaga

And finally, we set up some configs for LSPSaga.

-- ...
require("lspsaga").init_lsp_saga({
  finder_action_keys = {
    open = '<CR>',
    quit = {'q', '<esc>'},
  },
  code_action_keys = {
    quit = {'q', '<esc>'},
  },
  rename_action_keys = {
    quit = '<esc>',
  },
})

What to do next

Make sure you run :PackerInstall once you run Neovim after configuring the init.lua script. It will download all the configured plugins. And as a word of advice, run :PackerUpdate from time to time.

The Entire Config

If you read through all of the above, then sit back and pat yourself on the back!

-- Packer installation
local install_path = vim.fn.stdpath('data') .. '/site/pack/packer/start/packer.nvim'

if vim.fn.empty(vim.fn.glob(install_path)) > 0 then
  vim.fn.execute(
    '!git clone https://github.com/wbthomason/packer.nvim ' .. install_path
  )
end

vim.cmd [[
augroup Packer
  autocmd!
  autocmd BufWritePost init.lua PackerCompile
augroup end
]]

-- Plugin installation
use = require("packer").use
require("packer").startup(function()
  -- Package manager
  use 'wbthomason/packer.nvim'

  -- LSP Client
  use 'neovim/nvim-lspconfig'

  -- Language Server installer
  use {
    'williamboman/nvim-lsp-installer',
    requires = 'neovim/nvim-lspconfig',
  }

  -- BONUS: Customizations over LSP
  -- Show VSCode-esque pictograms
  use 'onsails/lspkind-nvim'
  -- show various elements of LSP as UI
  use {'tami5/lspsaga.nvim', requires = {'neovim/nvim-lspconfig'}}

  -- Autocompletion plugin
  use {
    'hrsh7th/nvim-cmp',
    requires = {
      'hrsh7th/cmp-nvim-lsp',
      'hrsh7th/cmp-buffer',
      'hrsh7th/cmp-path',
      'hrsh7th/cmp-cmdline',
    }
  }

  -- snippets
  use {
    'hrsh7th/cmp-vsnip', requires = {
      'hrsh7th/vim-vsnip',
      'rafamadriz/friendly-snippets',
    }
  }

  -- bracket autocompletion
  use 'vim-scripts/auto-pairs-gentle'

  -- Fancier statusline
  use {
    'nvim-lualine/lualine.nvim',
    requires = {
      'kyazdani42/nvim-web-devicons',
      'arkav/lualine-lsp-progress',
    },
  }

  -- Fast incremental parsing library
  use 'nvim-treesitter/nvim-treesitter'

  -- Beautiful colorscheme
  use 'navarasu/onedark.nvim'
end)

-- Neovim configuration
-- Do not show current vim mode since it is already shown by Lualine
vim.o.showmode = false

-- enable autowriteall
vim.o.autowriteall = true

-- Show the line numbers
vim.wo.number = true

-- Show chars at the end of line
vim.opt.list = true

-- Enable break indent
vim.o.breakindent = true

--Case insensitive searching UNLESS /C or capital in search
vim.o.ignorecase = true
vim.o.smartcase = true

-- Decrease update time
vim.o.updatetime = 250

-- Shows signs by Autocompletion plugin
vim.wo.signcolumn = 'yes'

-- Enable termguicolors. Very essential if you want 24-bit RGB color in TUI.
vim.o.termguicolors = true

-- Set completeopt to have a better completion experience
vim.o.completeopt = 'menu,menuone'

--Remap for dealing with word wrap
vim.api.nvim_set_keymap('n', 'k', "v:count == 0 ? 'gk' : 'k'", { noremap = true, expr = true, silent = true })
vim.api.nvim_set_keymap('n', 'j', "v:count == 0 ? 'gj' : 'j'", { noremap = true, expr = true, silent = true })

-- Highlight on yank
vim.cmd [[
augroup YankHighlight
autocmd!
autocmd TextYankPost * silent! lua vim.highlight.on_yank()
augroup end
]]

-- Plugin configuration
-- LSP and LS Installer
require('lspconfig')
local lsp_installer = require("nvim-lsp-installer")

-- The required servers
local servers = {
  "bashls",
  "pyright",
  "rust_analyzer",
  "sumneko_lua",
  "html",
  "clangd",
  "vimls",
  "emmet_ls",
}

for _, name in pairs(servers) do
  local server_is_found, server = lsp_installer.get_server(name)
  if server_is_found and not server:is_installed() then
    print("Installing " .. name)
    server:install()
  end
end

local on_attach = function(_, bufnr)
  -- Create some shortcut functions.
  -- NOTE: The `vim` variable is supplied by Neovim.
  local function buf_set_keymap(...)
    vim.api.nvim_buf_set_keymap(bufnr, ...)
  end
  local function buf_set_option(...)
    vim.api.nvim_buf_set_option(bufnr, ...)
  end

  -- Enable completion triggered by <c-x><c-o>
  buf_set_option('omnifunc', 'v:lua.vim.lsp.omnifunc')

  local opts = {noremap=true, silent=true}

  -- ======================= The Keymaps =========================
  -- jump to definition
  buf_set_keymap('n', 'gd', '<cmd>lua vim.lsp.buf.definition()<CR>', opts)

  -- Format buffer
  buf_set_keymap('n', '<F3>', '<cmd>lua vim.lsp.buf.formatting()<CR>', opts)

  -- Jump LSP diagnostics
  -- NOTE: Currently, there is a bug in lspsaga.diagnostic module. Thus we use
  --       Vim commands to move through diagnostics.
  buf_set_keymap('n', '[g', ':Lspsaga diagnostic_jump_prev<CR>', opts)
  buf_set_keymap('n', ']g', ':Lspsaga diagnostic_jump_next<CR>', opts)

  -- Rename symbol
  buf_set_keymap('n', '<leader>rn', "<cmd>lua require('lspsaga.rename').rename()<CR>", opts)

  -- Find references
  buf_set_keymap('n', 'gr', '<cmd>lua require("lspsaga.provider").lsp_finder()<CR>', opts)

  -- Doc popup scrolling
  buf_set_keymap('n', 'K', "<cmd>lua require('lspsaga.hover').render_hover_doc()<CR>", opts)
  buf_set_keymap('n', '<C-f>', "<cmd>lua require('lspsaga.action').smart_scroll_with_saga(1)<CR>", opts)
  buf_set_keymap('n', '<C-b>', "<cmd>lua require('lspsaga.action').smart_scroll_with_saga(-1)<CR>", opts)

  -- codeaction
  buf_set_keymap('n', '<leader>ac', "<cmd>lua require('lspsaga.codeaction').code_action()<CR>", opts)
  buf_set_keymap('v', '<leader>a', ":<C-U>lua require('lspsaga.codeaction').range_code_action()<CR>", opts)

  -- Floating terminal
  -- NOTE: Use `vim.cmd` since `buf_set_keymap` is not working with `tnoremap...`
  vim.cmd [[
  nnoremap <silent> <A-d> <cmd>lua require('lspsaga.floaterm').open_float_terminal()<CR>
  tnoremap <silent> <A-d> <C-\><C-n>:lua require('lspsaga.floaterm').close_float_terminal()<CR>
  ]]
end

local server_specific_opts = {
  sumneko_lua = function(opts)
    opts.settings = {
      Lua = {
        -- NOTE: This is required for expansion of lua function signatures!
        completion = {callSnippet = "Replace"},
        diagnostics = {
          globals = {'vim'},
        },
      },
    }
  end,

  html = function(opts)
    opts.filetypes = {"html", "htmldjango"}
  end,
}

-- `nvim-cmp` comes with additional capabilities, alongside the ones
-- provided by Neovim!
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities = require('cmp_nvim_lsp').update_capabilities(capabilities)

lsp_installer.on_server_ready(function(server)
  -- the keymaps, flags and capabilities that will be sent to the server as
  -- options.
  local opts = {
    on_attach = on_attach,
    flags = {debounce_text_changes = 150},
    capabilities = capabilities,
  }

  -- If the current surver's name matches with the ones specified in the
  -- `server_specific_opts`, set the options.
  if server_specific_opts[server.name] then
    server_specific_opts[server.name](opts)
  end

  -- And set up the server with our configuration!
  server:setup(opts)
end)

-- nvim-cmp
local lspkind = require('lspkind')
local cmp = require("cmp")

local has_words_before = function()
  local line, col = unpack(vim.api.nvim_win_get_cursor(0))
  return col ~= 0 and vim.api.nvim_buf_get_lines(0, line - 1, line, true)[1]:sub(col, col):match("%s") == nil
end

local feedkey = function(key, mode)
  vim.api.nvim_feedkeys(vim.api.nvim_replace_termcodes(key, true, true, true), mode, true)
end

local cmp_kinds = {
  Text = "",
  Method = "",
  Function = "",
  Constructor = "",
  Field = "ﰠ",
  Variable = "",
  Class = "ﴯ",
  Interface = "",
  Module = "",
  Property = "ﰠ",
  Unit = "塞",
  Value = "",
  Enum = "",
  Keyword = "",
  Snippet = "",
  Color = "",
  File = "",
  Reference = "",
  Folder = "",
  EnumMember = "",
  Constant = "",
  Struct = "פּ",
  Event = "",
  Operator = "",
  TypeParameter = "",
}

cmp.setup({
  snippet = {
    expand = function(args)
      vim.fn["vsnip#anonymous"](args.body)
    end,
  },

  formatting = {
    format = lspkind.cmp_format({
      with_text = true,
      preset = 'codicons',
      symbol_map = cmp_kinds, -- The glyphs will be used by `lspkind`
      menu = ({
        buffer = "[Buffer]",
        nvim_lsp = "[LSP]",
        luasnip = "[LuaSnip]",
        nvim_lua = "[Lua]",
        latex_symbols = "[Latex]",
      }),
    }),
  },

  mapping = {
    ['<C-p>'] = cmp.mapping.select_prev_item(),
    ['<C-n>'] = cmp.mapping.select_next_item(),
    ['<C-d>'] = cmp.mapping.scroll_docs(-4),
    ['<C-f>'] = cmp.mapping.scroll_docs(4),
    ['<C-Space>'] = cmp.mapping.complete(),
    ['<C-e>'] = cmp.mapping.close(),
    ['<CR>'] = cmp.mapping.confirm {
      behavior = cmp.ConfirmBehavior.Replace,
      select = true,
    },

    -- Use Tab and Shift-Tab to browse through the suggestions.
    ["<Tab>"] = cmp.mapping(function(fallback)
      if cmp.visible() then
        cmp.select_next_item()
      elseif vim.fn["vsnip#available"](1) == 1 then
        feedkey("<Plug>(vsnip-expand-or-jump)", "")
      elseif has_words_before() then
        cmp.complete()
      else
        fallback()
      end
    end, { "i", "s" }),

    ["<S-Tab>"] = cmp.mapping(function()
      if cmp.visible() then
        cmp.select_prev_item()
      elseif vim.fn["vsnip#jumpable"](-1) == 1 then
        feedkey("<Plug>(vsnip-jump-prev)", "")
      end
    end, { "i", "s" }),
  },

  sources = {
    { name = 'nvim_lsp' },
    { name = 'vsnip' },
    { name = 'buffer' },
  },
})

-- Use buffer source for `/`
cmp.setup.cmdline('/', {
  sources = {
    { name = 'buffer' }
  }
})

-- Use cmdline & path source for ':'
cmp.setup.cmdline(':', {
  sources = cmp.config.sources({
    { name = 'path' }
  }, {
    { name = 'cmdline' }
  })
})

-- autopairs
vim.g.AutoPairs = {
  ['(']=')',
  ['[']=']',
  ['{']='}',
  ["'"]="'",
  ['"']='"',
  ['`']='`',
  ['<']='>',
}

-- onedark
require("onedark").setup({
  style = "darker",
})
require('onedark').load()

-- lualine
require('lualine').setup({
  sections = {
    lualine_c = {
      {'filename', path = 1},
      'lsp_progress',
    },
  },
})

-- treesitter
require('nvim-treesitter.configs').setup {
  ensure_installed = {"python", "rust", "c", "cpp", "bash", "go", "html"},
  highlight = {
    enable = true, -- false will disable the whole extension
  },
}

-- lspsaga
require("lspsaga").init_lsp_saga({
  finder_action_keys = {
    open = '<CR>',
    quit = {'q', '<esc>'},
  },
  code_action_keys = {
    quit = {'q', '<esc>'},
  },
  rename_action_keys = {
    quit = '<esc>',
  },
})

Conclusion

I hope this article helps you in configuring your Neovim to your heart's desire.

Questions? Comments? Concerns? Please put them down below and I'd be happy to help you.

Cover Image Source: Photo by luis gomes from Pexels 😃

Self Referential Structs in Rust (Part 2)

Arunanshu Biswas — Fri, 26 Nov 2021 19:04:50 +0000

Before You Read

I strongly advise you to read Part 1 if you haven't read it yet. It goes into the basics of self referential structs, how a basic solution can be achieved (given below) and the flaws in the design.

This article aims to fix the flaws and create a robust system.

Click here to go to Part 1.

TL;DR

Create a wrapper type over the wrapped type and store the wrapped type as a raw pointer. Here our wrapper type is MeWrapper(*mut Me) over Me.
Implement comparison functions (ie, Eq, Ord, PartialOrd) manually and Deref traits for the wrapper type (here, MeWrapper).
Make sure that the comparison functions compare the actual wrapped types.
Store MeWrapper instead of *mut Me in the holder (here Holder).
Implement Drop properly by building the MeWrapper again and using it to remove the MeWrapper from Holder.
Use derivative crate to "Ignore" the comparison of my_holder in Me by Ord using #[derivative(Ord = "ignore")].

Click here to jump to solution.

Recap

Let us see what we have done till now.

use std::{
    cell::RefCell, collections::BTreeSet, marker::PhantomPinned, pin::Pin,
    rc::Rc,
};
type RcCell<T> = Rc<RefCell<T>>;

#[derive(Debug)]
struct Holder {
    set_of_me: BTreeSet<*mut Me>,
}

impl Holder {
    fn new() -> RcCell<Self> {
        Rc::new(RefCell::new(Self {
            // this is initially empty
            set_of_me: Default::default(),
        }))
    }

    /// Mutate every value of `Me`.
    /// Note how a pinned value is reconstructed.
    fn mutate_value_of_me(&self, val: i32) {
        self.set_of_me.iter().for_each(|a| {
            let a = unsafe { Pin::new_unchecked(&mut **a) };
            a.mutate_me(val);
        })
    }
}

#[derive(Debug)]
struct Me {
    name: String,
    mutate_by_holder: i32,
    my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

impl Me {
    /// Accept a name
    pub fn new(
        holder: RcCell<Holder>,
        name: impl Into<String>,
    ) -> Pin<Box<Self>> {
        let mut this = Box::pin(Self {
            name: name.into(),
            mutate_by_holder: 0,
            my_holder: holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut _ = unsafe { this.as_mut().get_unchecked_mut() };
        this.my_holder.borrow_mut().set_of_me.insert(this_ptr);

        this
    }

    /// Allows you to mutate a value within me.
    /// Run this from `Holder` to see what happens.
    fn mutate_me(self: Pin<&mut Self>, val: i32) {
        let this = unsafe { self.get_unchecked_mut() };
        this.mutate_by_holder += val;
    }
}

impl Drop for Me {
    fn drop(&mut self) {
        println!("Dropping {:#?}", self);
        let this = &(self as *mut _);
        self.my_holder.borrow_mut().set_of_me.remove(this);
    }
}

/// A test function to play with `Holder`s.
fn make_ref_of_holder(holder: RcCell<Holder>) {
    let holder = Rc::clone(&holder);
    println!("Making a ref of {:?}", holder);
    println!("No. of refs = {}", Rc::strong_count(&holder));
}

pub fn main() {
    let holder = Holder::new();

    // be explicit about `Rc`'s cloning
    let a = Me::new(Rc::clone(&holder), "a");
    let b = Me::new(Rc::clone(&holder), "b");

    holder.borrow().mutate_value_of_me(455);
    make_ref_of_holder(Rc::clone(&holder));
}

Discovery of the Flaw

As we saw in Part 1, insert the following line in the main()
function above:

// ...
pub fn main() {
    let holder = Holder::new();

    // be explicit about `Rc`'s cloning
    let a = Me::new(Rc::clone(&holder), "a");
    let b = Me::new(Rc::clone(&holder), "b");
+   let c = Me::new(Rc::clone(&holder), "b");

    holder.borrow().mutate_value_of_me(455);
    make_ref_of_holder(Rc::clone(&holder));
}

We might think, "Oh since we are storing our Me objects in a set, c will not be added."

But notice that the definition of set_of_me is BTreeSet<*mut Me>. Which means, the set stores unique pointers to Me, not unique Mes itself! And that's why c gets stored in holder.

Understanding the Problem

So, we need write code in a way that it compares instances of Me instead of comparing *mut Me.

Now, notice that *mut Me is basically a pointer to an instance of Me. So, we can write a wrapper for *mut Me that allows us to dereference and access the underlying Me.

At last, we need to implement some traits so that the wrapper can be used as a placeholder to compare the underlying Mes.

Building a Solution

The Wrapper

Now, we need a wrapper. Let's call it MeWrapper, and since it has only one non-zero-sized field, we will use #[repr(transparent)].

#[repr(transparent)]
struct MeWrapper(*mut Me);

Read more about #[repr(transparent)] here, as it is beyond the scope of this article.

The Dereferencing Abilities

Since we want to be able to access the underlying Me without any hassle of using unsafe, we are going to implement Deref.

impl Deref for MeWrapper {
    type Target = Me;
    fn deref(&self) -> &Self::Target {
        unsafe { return &*self.0 }
    }
}

The Comparison Functions

Now, it's time to write the comparison functions. They are fairly simple to understand. We simply take advantage of Deref trait and access the underlying Me. Then we compare the underlying objects using their implementations of the same traits.

impl Eq for MeWrapper {}

impl PartialEq for MeWrapper {
    fn eq(&self, other: &Self) -> bool {
        let this = &**self;
        let other = &**other;
        this == other
    }
}

impl Ord for MeWrapper {
    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
        let this = &**self;
        let other = &**other;
        this.cmp(&other)
    }
}

impl PartialOrd for MeWrapper {
    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
        let this = &**self;
        let other = &**other;
        this.partial_cmp(&other)
    }
}

Note that we still haven't implemented the traits on Holder and Me. To make this wrapper work, we need to implement them. Don't worry though, we will discuss about that when the necessity comes.

`Holder` holds our Wrappers

This part is simple. We simply replace *mut Me with MeWrapper.

#[derive(Debug)]
struct Holder {
-    set_of_me: BTreeSet<*mut Me>,
+    set_of_me: BTreeSet<MeWrapper>,
}

Hence, it'd look like:

#[derive(Debug)]
struct Holder {
     set_of_me: BTreeSet<MeWrapper>,
}

Easy Conversion of `*mut Me` to `MeWrapper`

We simply take advantage of From trait and implement for *mut Me.

impl From<*mut Me> for MeWrapper {
    fn from(value: *mut Me) -> Self {
        Self(value)
    }
}

This automatically gives us the ability to convert *mut Me to MeWrapper using .into().

`Drop`ping Wrappers Properly

Now we just need to convert this into a MeWrapper and remove it from the set. Not that we take advantage of From trait by calling the .into() function.

impl Drop for Me {
    fn drop(&mut self) {
        println!("Dropping {:#?}", self);
-       let this = &(self as *mut _);
+       let this = (self as *mut Self).into();
-       self.my_holder.borrow_mut().set_of_me.remove(this);
+       self.my_holder.borrow_mut().set_of_me.remove(&this);
    }
}

Hence it'd look like:

impl Drop for Me {
    fn drop(&mut self) {
        println!("Dropping {:#?}", self);
        let this = (self as *mut Self).into();
        self.my_holder.borrow_mut().set_of_me.remove(&this);
    }
}

Inserting Wrappers into Holder Properly

Just like Drop trait, we will update Me::new(...) to insert the wrappers properly.

    pub fn new(
        holder: RcCell<Holder>,
        name: impl Into<String>,
    ) -> Pin<Box<Self>> {
        let mut this = Box::pin(Self {
            name: name.into(),
            mutate_by_holder: 0,
            my_holder: holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut _ = unsafe { this.as_mut().get_unchecked_mut() };
-       this.my_holder.borrow_mut().set_of_me.insert(this_ptr);
+       this.my_holder.borrow_mut().set_of_me.insert(this_ptr.into());

        this
    }

Hence it’d look like:

pub fn new(
    holder: RcCell<Holder>,
    name: impl Into<String>,
) -> Pin<Box<Self>> {
    let mut this = Box::pin(Self {
        name: name.into(),
        mutate_by_holder: 0,
        my_holder: holder,
        _pinned: PhantomPinned,
    });

    let this_ptr: *mut _ = unsafe { this.as_mut().get_unchecked_mut() };
    this.my_holder.borrow_mut().set_of_me.insert(this_ptr.into());

    this
}

More on Comparing Objects

This is where the fun and complication begins. So buckle up and grab a cup of coffee.

Let's say we did add comparison traits to the Me structs via #[derive(...)].

-#[derive(Debug)]
+#[derive(Debug, PartialOrd, Ord, PartialEq, Eq)]
struct Me {
    name: String,
    mutate_by_holder: i32,
    my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

And thus, we also add the same to the Holder struct.

-#[derive(Debug)]
+#[derive(Debug, PartialOrd, Ord, PartialEq, Eq)]
struct Holder {
    set_of_me: BTreeSet<MeWrapper>,
}

Now go on and execute the program and see what happens. You'll face something like:

thread 'main' panicked at 'already mutably borrowed: BorrowError'

So, why did that happen?

`RefCell` and Borrows

Now, to understand that, we have to understand what BorrowError is. The BorrowError is basically the feature of RefCell, where Rust's borrow checking is performed dynamically.

Now normally, Rust complains about borrow issues when we either:

When multiple mutable references are created.
Create an immutable reference when a mutable reference of the same object is being used.

In RefCell, we create mutable references with .borrow_mut() and mutable
references with .borrow().

Keeping this previous statement in mind, we have to find where one of the two rules is being violated.

Search for `BorrowError`'s source

Look at the source code, especially Holder's initialization. We create a Rc<RefCell<Holder>>. That means, the issue must be with or related to Holder.

Now look for the part where we create a mutable reference of Holder (via .borrow_mut()).

let this_ptr: *mut Self = unsafe { this.as_mut().get_unchecked_mut() };
this.my_holder.borrow_mut().set_of_me.insert(this_ptr.into());

Here we are creating a mutable reference to Holder and inserting a MeWrapper. Now let's check for any violations of the two rules stated above.

Nope. We are creating only one mutable reference via .borrow_mut() and using to insert values in set_of_me.
This is where things get tricky. Read on, and make sure you have your seatbelts on!

Relation Between Comparing Objects and Borrowing

Now let us focus on this part of the code:

set_of_me.insert(this_ptr.into());
//        ^^^^^^^^^^^^^^^^^^^^^^

What are we doing here? We are inserting a MeWrapper in a BTreeSet. And how does a BTreeSet check for uniqueness of a value? By comparing with other values.

So now we have atleast some idea of where the error originates from. It is definitely related to some sort of comparison function(s).

Now, look for the comparison functions we've implemented for MeWrapper. They are eq(), cmp() and partial_cmp().

`PartialEq`

PartialEq is a simple one:

this == other

It just checks for equality of two values. Here, they are instances of references to Me. So, this is not the culprit.

`Ord` and `PartialOrd`

Try an experiment. Put dbg!(""); in both cmp() and partial_cmp(), just before the functions are executed and result is returned.

impl Ord for MeWrapper {
    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
        let this = &**self;
        let other = &**other;
+       dbg!("did it fail in cmp?");
        this.cmp(&other)
    }
}

impl PartialOrd for MeWrapper {
    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
        let this = &**self;
        let other = &**other;
+       dbg!("did it fail in partial_cmp?");
        this.partial_cmp(&other)
    }
}

Execute it and you'll notice that cmp() is executed just before the error happens! So, what is going on?

`Ord`, Borrowing and `BorrowError`

First, let us look at the signature of insert() function of BTreeSet:

pub fn insert(&mut self, value: T) -> bool
where
    T: Ord,
{
    // ...
}

Here, we see that .insert() expects a value that implements Ord. Thus, we were correct about Ord's involvement in the error.

Now the question arises, why does the error happen?

During comparison via .cmp(), every value in a given struct is compared with the corresponding values of the other struct. And RefCell<T> is no exception. Now what's interesting is what actually happens during a .cmp() call for RefCell<T>.

If we look at the source of RefCell::cmp() , we will find that it borrows the holder inside it immutably.

fn cmp(&self, other: &RefCell<T>) -> Ordering {
    self.borrow().cmp(&*other.borrow())
}

But simultaneously we are also borrowing Holder mutably in this line:

this.my_holder.borrow_mut().set_of_me.insert(this_ptr.into());

And the official docs says this about Ord, PartialOrd and PartialEq traits:

Panics if the value in either RefCell is currently borrowed.

But this error is not unique to Ord. We get this error from Ord only because of its involvement in .insert() (look at .insert()'s signature again).

Now that we know where the error is coming from and its reason, we need to somehow disable the comparison of Holder values. You might ask, "Isn't it daunting, to write custom trait implementations just to prevent comparison of a
particular field? Only if we had something to automatically disable the comparison of unreqired fields." Yes, it is daunting, and a total waste of time. And thankfully, we have a pretty nice solution.

To `#[derive]` or not to `#[derive]`

Meet derivative! It is an amazing crate that automates derivation of several traits and performs the heavy-lifting of conditional derivation for us!

Let us focus on our issue. We need to disable the comparison of holder in Me by Ord trait.

Start by bringing derivative crate into scope using extern. And by the way, don't forget to include it in Cargo.toml's dependencies! Then, put a #[macro_use] above that statement to actually use the macro.

#[macro_use]
extern crate derivative;

Now, we need to disable Ord for my_holder field in Me struct. Or in derivative's words, we need to ignore the field my_holder. As we shall see, it is extremely easy!

-#[derive(Debug)]
+#[derive(Debug, Derivative)]
+#[derivative(PartialEq, Eq, PartialOrd, Ord)]
struct Me {
    name: String,
    mutate_by_holder: i32,
+   #[derivative(Ord = "ignore")]
    my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

We follow these simple steps:

Derive Derivative via #[derive(...)].
Use #[derivative(...)] on struct to derive the required traits.
"Ignore" a particular field by specifying #[derivative(<trait name> = "ignore")].

And boom! That's all it takes to solve the issue!

Speaking of derivative, we can also take advantage of its custom Debug trait derivation for MeWrapper!

#[repr(transparent)]
#[derive(Derivative)]
#[derivative(Debug = "transparent")]
struct MeWrapper(*mut Me);

The Final Solution

And this is the solution! Run it, play with it, break it!

#[macro_use]
extern crate derivative;

use std::{
    cell::RefCell, collections::BTreeSet, marker::PhantomPinned, pin::Pin,
    rc::Rc,
};
type RcCell<T> = Rc<RefCell<T>>;

#[derive(Debug, PartialEq, Eq, PartialOrd, Ord)]
struct Holder {
    set_of_me: BTreeSet<MeWrapper>,
}

impl Holder {
    fn new() -> RcCell<Self> {
        Rc::new(RefCell::new(Self {
            // this is initially empty
            set_of_me: Default::default(),
        }))
    }

    /// Mutate every value of `Me`.
    /// Note how a pinned value is reconstructed.
    fn mutate_value_of_me(&self, val: i32) {
        self.set_of_me.iter().for_each(|a| {
            let a = unsafe { Pin::new_unchecked(&mut *a.0) };
            a.mutate_me(val);
        })
    }
}

#[derive(Debug, Derivative)]
#[derivative(PartialEq, Eq, PartialOrd, Ord)]
struct Me {
    name: String,
    mutate_by_holder: i32,
    #[derivative(Ord = "ignore")]
    my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

impl Me {
    /// Accept a name
    pub fn new(
        holder: RcCell<Holder>,
        name: impl Into<String>,
    ) -> Pin<Box<Self>> {
        let mut this = Box::pin(Self {
            name: name.into(),
            mutate_by_holder: 0,
            my_holder: holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut Self = unsafe { this.as_mut().get_unchecked_mut() };
        this.my_holder
            .borrow_mut()
            .set_of_me
            .insert(this_ptr.into());

        this
    }

    /// Allows you to mutate a value within me.
    /// Run this from `Holder` to see what happens.
    fn mutate_me(self: Pin<&mut Self>, val: i32) {
        let this = unsafe { self.get_unchecked_mut() };
        this.mutate_by_holder += val;
    }
}

#[repr(transparent)]
#[derive(Derivative)]
#[derivative(Debug = "transparent")]
struct MeWrapper(*mut Me);

impl From<*mut Me> for MeWrapper {
    fn from(value: *mut Me) -> Self {
        Self(value)
    }
}

impl std::ops::Deref for MeWrapper {
    type Target = Me;
    fn deref(&self) -> &Self::Target {
        unsafe { return &*self.0 }
    }
}

impl Eq for MeWrapper {}

impl PartialEq for MeWrapper {
    fn eq(&self, other: &Self) -> bool {
        let this = &**self;
        let other = &**other;
        this == other
    }
}

impl Ord for MeWrapper {
    fn cmp(&self, other: &Self) -> std::cmp::Ordering {
        let this = &**self;
        let other = &**other;
        this.cmp(&other)
    }
}

impl PartialOrd for MeWrapper {
    fn partial_cmp(&self, other: &Self) -> Option<std::cmp::Ordering> {
        let this = &**self;
        let other = &**other;
        this.partial_cmp(&other)
    }
}

impl Drop for Me {
    fn drop(&mut self) {
        println!("Dropping {:#?}", self);
        let this = (self as *mut Self).into();
        self.my_holder.borrow_mut().set_of_me.remove(&this);
    }
}

/// A test function to play with `Holder`s.
fn make_ref_of_holder(holder: RcCell<Holder>) {
    let holder = Rc::clone(&holder);
    println!("Making a ref of {:?}", holder);
    println!("No. of refs = {}", Rc::strong_count(&holder));
}

pub fn main() {
    let holder = Holder::new();

    // be explicit about `Rc`'s cloning
    let a = Me::new(Rc::clone(&holder), "a");
    let b = Me::new(Rc::clone(&holder), "b");
    let c = Me::new(Rc::clone(&holder), "b");

    holder.borrow().mutate_value_of_me(455);
    make_ref_of_holder(Rc::clone(&holder));
}

Conclusion

I hope this article helps you in understanding Rust's Self Referential Structs a bit better.

Questions? Comments? Concerns? Please put them down below and I'd be happy to help you.

Cover Image Source: Manjaro's /usr/share/backgrounds folder 😃

Self Referential Structs in Rust (Part 1)

Arunanshu Biswas — Thu, 04 Nov 2021 06:56:34 +0000

TL;DR

Use Pin<Box<T>> for the held value (here, Me).
Mark Me as unpinnable by setting a field as std::marker::PhantomPinned.

Eg, _pinned: std::marker::PhantomPinned.
Use Rc<RefCell<H>> for holder, (here, Holder) and store Me as raw
pointers.

Eg, me: BTreeSet<*const Me>.
Store Holder within Me using Rc::clone(...).
Store Me within Holder by converting it into a raw pointer. It is completely upto you if you want to store a mutable raw pointer or a const raw pointer.
Implement Drop trait such that it removes all references to Me from Holder once it is dropped.

Click here to jump to solution.

Introduction

There are mainly two ways to represent self-referential struct in Rust.

Reference within the struct:

struct RefWithinMe {
    value: String,

    pointer_to_value: &'? str,
}

These can easily be solved using ouroboros.

#[macro_use]
extern crate ouroboros;

#[self_referential]
struct RefWithinMe {
    value: String,

    #[borrows(value)]
    pointer_to_value: &'this str,
}

We won't be discussing about these.

Reference outside the struct (A referential cycle):

use std::collections::BTreeSet;

#[derive(Ord, PartialEq, Eq, PartialOrd)]
struct Me<'a> {
    /// The my_holder that I hold.
    my_holder: &'a Holder<'a>,
}

/// The my_holder that holds `Me`.
#[derive(Ord, PartialEq, Eq, PartialOrd)]
struct Holder<'a> {
    set_of_me: BTreeSet<&'a Me<'a>>,
}

impl Me<'a> {
    fn new(my_holder: &'a mut Holder<'a>) -> Self {
        let this = Self {
            my_holder,
        };
        // Insert myself here.
        my_holder.set_of_me.insert(&this);

        // but the value is being moved 🤔
        this
    }
}

We are going to discuss about this.

Problem Description

Given that you have two structs, a holder (Holder, say) and a struct to hold something (Me, say), how will you establish a circular reference?

Experimentation

Try to run the above code and see what happens.

You get an error where Rust complaints about:

error[E0502]: cannot borrow `my_holder.set_of_me` as mutable because it is also borrowed as immutable
   |
15 | impl<'a> Me<'a> {
   |      -- lifetime `'a` defined here
...
18 |            my_holder,
   |            ---------
   |            |
   |            immutable borrow occurs here
   |            requires that `*my_holder` is borrowed for `'a`
...
21 |        my_holder.set_of_me.insert(&this);
   |        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ mutable borrow occurs here

error[E0597]: `this` does not live long enough
   |
15 | impl<'a> Me<'a> {
   |      -- lifetime `'a` defined here
...
21 |        my_holder.set_of_me.insert(&this);
   |        ---------------------------^^^^^-
   |        |                          |
   |        |                          borrowed value does not live long enough
   |        argument requires that `this` is borrowed for `'a`
...
25 |    }
   |    - `this` dropped here while still borrowed

error[E0505]: cannot move out of `this` because it is borrowed
   |
15 | impl<'a> Me<'a> {
   |      -- lifetime `'a` defined here
...
21 |        my_holder.set_of_me.insert(&this);
   |        ---------------------------------
   |        |                          |
   |        |                          borrow of `this` occurs here
   |        argument requires that `this` is borrowed for `'a`
...
24 |        this
   |        ^^^^ move out of `this` occurs here

error: aborting due to 3 previous errors

Aaaand, we hit a lifetime error!

I'll assume that you have a basic knowledge about Rust's lifetimes. If not, don't worry! I'll try to keep this article as simple as possible. But if you're interested, You can read my post on Rust Lifetimes.

Understanding the Error

So, here we see that Rust complaints us about this variable not "living long enough".

Pay attention to the message, "borrowed value does not live long enough". Rust wants us to understand that if this is moved, then my_holder.set_of_me will have a reference to a nonexistent object. And it's never a good idea to dereference a reference (or, a pointer) to something that does not exist. *SEGFAULT intensifies 😖*

In short, Rust prevents the creation of dangling pointer.

But, this prevents us from creating a self-referential struct. Worry not, because we can pull up some tricks to make the Rust compiler "think" that there are no lifetime violations. However, it requires you to go down to the unsafe level.

But first, let us talk about moves and why they are a big deal.

Understanding Moves in Rust

A "move" in a function happens when you return a value:

fn foo() -> String {
    let value = "ABC".to_owned();
    value  // move occurs
}

Now, for ordinary objects, "moving" is fine. But since objects (and their references) created within the function have the lifetime of the function itself, Rust disallows it (thankfully!).

That also means, we need to find a way to tell Rust that "our move will keep the reference so please don't drop it!".

Raw Pointers of Rust

Let's jump right to the point. Raw pointers are your typical C-like pointers that are not bound to Rust's borrow checker.

But Rust's pointer suffer from a fatal flaw. They are not safe to use when two variables are swapped.

Let me demonstrate it. In fact, the following is adapted from The Rust Language Tutorial for Async. Follow this link to know more.

#[derive(Debug)]
struct Test {
    value: String,
    pointer_to_value: *const String,
}

impl Test {
    fn new(txt: &str) -> Self {
        let mut this = Test {
            value: String::from(txt),
            // In `C`, you'd call this initialize to `NULL`.
            pointer_to_value: std::ptr::null(),
        };
        this.pointer_to_value = &this.value;
        this
    }

    fn get_value(&self) -> &str {
        &self.value
    }

    fn get_pointer_to_value(&self) -> &String {
        unsafe { &*(self.pointer_to_value) }
    }
}

Here we have a simple struct called Test. pointer_to_value points to value. Two methods (get_value and get_pointer_to_value) have been provided as getters.

Now, let's say we create two instances of Test and use our getters to inspect the contents.

fn main() {
    let mut test1 = Test::new("test1");
    let mut test2 = Test::new("test2");

    println!(
        "test1: value: {}, ptr to value: {}",
        test1.get_value(),
        test1.get_pointer_to_value()
    );
    println!(
        "test2: value: {}, ptr to value: {}",
        test2.get_value(),
        test2.get_pointer_to_value()
    );
}

Executing the code gives:

test1: value: test1, ptr to value: test1
test2: value: test2, ptr to value: test2

Nothing out of the ordinary right? You get the contents of value and pointer_to_value and print it. But the real danger happens when we swap the variables.

fn main() {
    // ...
    println!("Swapping...");
    std::mem::swap(&mut test1, &mut test2);

    println!(
        "test1: value: {}, ptr to value: {}",
        test1.get_value(),
        test1.get_pointer_to_value()
    );
    println!(
        "test2: value: {}, ptr to value: {}",
        test2.get_value(),
        test2.get_pointer_to_value()
    );
}

You might expect this to print:

Swapping...
test1: value: test2, ptr to value: test2
test2: value: test1, ptr to value: test1

But to our surprise, this prints:

Swapping...
test1: value: test2, ptr to value: test1
test2: value: test1, ptr to value: test2

Unfolding the Mystery of Raw Pointers

So why did that happen? The answer is quite simple. "A pointer follows its pointee".

When we swapped the variables, the contents got swapped too. But that does not mean the pointers will change their pointee. This image demonstrates the point:

But, this is not good as this might potentially lead to situations like "double free" and "incorrect access".

We need a way to tell Rust that our variables should not be moved. Or in other words, we need something that will encapsulate our objects and make them immovable.

`Pin<T>` the objects

Meet Pin<T>! They "pin" objects to its location in memory².

Now, you can either pin to the stack or pin to the heap. For this case, pinning to the heap is the simplest and most viable option.

In Rust, we know that we can put a variable to the heap by using Box<T>, Rc<T> or Arc<T>. For this case, Box<T> is sufficient.

To our convenience, Box<T> provides a method called Box::pin(...). We can use this to pin our object to encapsulate and prevent it from incorrect moves.

So how do we use Pin<T>? Let's modify our code to find out.

use std::pin::Pin;

#[derive(Debug)]
struct Test {
    // same as before
}

impl Test {
    fn new(txt: &str) -> Pin<Box<Self>> {  // return Pin<Box<Self>>
        let mut this = Box::pin(Test {
            value: String::from(txt),
            // In `C`, you'd call this initialize to `NULL`.
            pointer_to_value: std::ptr::null(),
        });
        this.as_mut().pointer_to_value = &this.value;
        this
    }

    fn get_value(self: Pin<&Self>) -> &str {  // modify `self` parameter
        &self.get_ref().value
    }

    // modify `self` parameter
    fn get_pointer_to_value(self: Pin<&Self>) -> &String {
        unsafe { &*(self.pointer_to_value) }
    }
}

Note that whenever you use Pin<T>, the self argument is modified to accept a Pin<T> type only.

Running the main function as before:

fn main() {
    let mut test1 = Test::new("test1");
    let mut test2 = Test::new("test2");

    println!(
        "test1: value: {}, ptr to value: {}",
        test1.as_ref().get_value(),
        test1.as_ref().get_pointer_to_value(),
    );
    println!(
        "test2: value: {}, ptr to value: {}",
        test2.as_ref().get_value(),
        test2.as_ref().get_pointer_to_value(),
    );

    println!("Swapping...");
    std::mem::swap(&mut test1, &mut test2);

    println!(
        "test1: value: {}, ptr to value: {}",
        test1.as_ref().get_value(),
        test1.as_ref().get_pointer_to_value()
    );
    println!(
        "test2: value: {}, ptr to value: {}",
        test2.as_ref().get_value(),
        test2.as_ref().get_pointer_to_value()
    );
}

This prints:

test1: value: test1, ptr to value: test1
test2: value: test2, ptr to value: test2
Swapping...
test1: value: test2, ptr to value: test2
test2: value: test1, ptr to value: test1

As expected!

Note that we can still get a mutable reference to our enclosed Test by using as_mut().get_mut(). This cannot be good!

`PhantomPinned` to the Rescue

By using std::marker::PhantomPinned, we tell Rust that our object cannot be unpinned (Rust calls it !Unpin. Read as "Not Unpinnable").

Now, any calls to get_mut() will fail.

So, how do you use it?

modify your struct to hold that extra field.

use std::{
+   marker::PhantomPinned,
    pin::Pin,
};

#[derive(Debug)]
struct Test {
    value: String,
    pointer_to_value: *const String,
+   _pinned: PhantomPinned,
}

Initialize PhantomPinned in new()

impl Test {
    fn new(txt: &str) -> Pin<Box<Self>> {
        let mut this = Box::pin(Test {
            value: String::from(txt),
            // In `C`, you'd call this initialize to `NULL`.
            pointer_to_value: std::ptr::null(),
+           _pinned: PhantomPinned,
        });
        this.as_mut().pointer_to_value = &this.value;
        this
    }

// ...
}

Try to run it, and you'll face with yet another complaint from Rust! This time, it will say: "cannot assign to data in a dereference of Pin<&mut Test>". So, what does that mean?

It means: in the presence of PhantomPinned, you cannot modify, or mutate through a dereference. This is a side effect of PhantomPinned.

To overcome this problem, we have to "force our way" into the enclosed object. That means, we will use unsafe.

impl Test {
    fn new(txt: &str) -> Pin<Box<Self>> {
        let mut this = Box::pin(Test {
            value: String::from(txt),
            // In `C`, you'd call this initialize to `NULL`.
            pointer_to_value: std::ptr::null(),
            _pinned: PhantomPinned,
        });
+       unsafe {
+           this.as_mut().get_unchecked_mut().pointer_to_value = &this.value;
+       }
        this
    }
// ...
}

Aaaand that's it! We have constructed a self-referential struct using Pin.

But why did we go through all this? This doesn't even solve our original problem?!

We went through this in order to understand the mechanism of Rust's Pin and its associated intricacies.

Now that we have understood how pinning works, let us go back to our original problem.

Solving the Original Problem

Let us wrap Me with Pin<T>.

use std::{
   marker::PhantomPinned,
   pin::Pin,
}

#[derive(Debug)]
struct Me<'a> {
    my_holder: &'a Holder,
    _pinned: PhantomPinned,
}

impl Me {
    pub fn new(my_holder: &'a mut Holder) -> Pin<Box<Self>> {
        let this = Box::pin(Self {
            my_holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut _ = unsafe { val.as_mut().get_unchecked_mut() };
        my_holder.set_of_me.insert(this_ptr);

        val
    }
}

But we have a small problem with this scheme. But first, let us check what Holder looks like:

#[derive(Debug)]
struct Holder {
    set_of_me: BTreeSet<*mut Me>,
}

Holder holds several instances of Me. Or in other words, Me inserts itself in Holder, and it can insert at any time at any place. But if we keep using references for Holder (ie, &'a mut Holder), we'll soon realize that Rust does not allow us to go beyond what Borrow Checker permits.

We need to find a way to share references without letting Rust know that it is a reference. And it should be destroyed once its job is done.

`Rc<T>` and `RefCell<T>`

Rc<T> is the reference counted version of Box<T>, and RefCell<T> allows you to mutate a seemingly immutable object. It is useful mostly in these contexts. So how do we use it?

First, we create a type alias for shorthand!

type RcCell<T> = Rc<RefCell<T>>;

Next, we modify Holder to return a RcCell<Holder>.

use std::{
   cell::RefCell,
   rc::Rc,
}

#[derive(Debug)]
struct Holder {
    set_of_me: BTreeSet<*mut Me>,
}

impl Holder {
    fn new() -> RcCell<Self> {
        Rc::new(RefCell::new(Self {
            set_of_me: Default::default(),
        }))
    }
}

Let's discuss a bit about RefCell<T>.

Why `RefCell<T>`?

Let's see what happens when we try to modify anything in an Rc<T>:

fn main() {
    let mut x = Rc::new(String::from("EEE"));
    x.push('e');
}

The last line will complain: "cannot borrow data in an Rc as mutable". That means, Rc<T> acts as an immutable enclosure around the object.

Now let's see what happens when we use RefCell<T>:

fn main() {
    let mut x = Rc::new(RefCell::new(String::from("EEE")));
    x.borrow_mut().push('e');
}

RefCell<T> adds a new method called borrow_mut() that allows you to receive a mutable reference to the underlying object. And the added benefit of using RefCell<T> is that it strictly follows Rust's borrowing rules. This ensures nothing funny or obscure happens with the object.

Putting the Pieces Together

Let us tell Me to accept a RcCell<Holder>.

#[derive(Debug)]
struct Me {
+   my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

impl Me {
    pub fn new(
+       my_holder: RcCell<Holder>,
    ) -> Pin<Box<Self>> {
        let mut this = Box::pin(Self {
            my_holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut _ = unsafe { this.as_mut().get_unchecked_mut() };
+       this.my_holder.borrow_mut().set_of_me.insert(this_ptr);

        this
    }
}

But before you jump and rejoice, think about what happens in the end? What happens to the references in Holder's set_of_me after an instance of Me has been destroyed?

Right! They are still there as DANGLING POINTERS!

We need a way to remove Me from Holder's set_of_me when it will be destroyed. Fortunately, we have something called Drop trait in Rust.

impl Drop for Me {
    fn drop(&mut self) {
        let this = &(self as *mut _);
        self.my_holder.borrow_mut().set_of_me.remove(this);
    }
}

And that's it.

Why not `Weak<T>` for storing `Me`?

Now you could ask, why go through all the pain of raw pointers and Pins when I could've used Weak<T>.

Yes you could've, but Weak<T> does not implement Ord, PartialOrd, PartialEq etc. This will limit usability to a great degree.

Similarly, you cannot use Rc<T> for storing Me because Rc<T> will not help you in resolving the circular reference and will create a memory leak.

Solution

Here is the entire solution. Run it and experiment with it to understand how it works.

use std::{
    cell::RefCell, collections::BTreeSet, marker::PhantomPinned, pin::Pin,
    rc::Rc,
};
type RcCell<T> = Rc<RefCell<T>>;

#[derive(Debug)]
struct Holder {
    set_of_me: BTreeSet<*mut Me>,
}

impl Holder {
    fn new() -> RcCell<Self> {
        Rc::new(RefCell::new(Self {
            // this is initially empty
            set_of_me: Default::default(),
        }))
    }

    /// Mutate every value of `Me`.
    /// Note how a pinned value is reconstructed.
    fn mutate_value_of_me(&self, val: i32) {
        self.set_of_me.iter().for_each(|a| {
            let a = unsafe { Pin::new_unchecked(&mut **a) };
            a.mutate_me(val);
        })
    }
}

#[derive(Debug)]
struct Me {
    name: String,
    mutate_by_holder: i32,
    my_holder: RcCell<Holder>,
    _pinned: PhantomPinned,
}

impl Me {
    /// Accept a name
    pub fn new(
        holder: RcCell<Holder>,
        name: impl Into<String>,
    ) -> Pin<Box<Self>> {
        let mut this = Box::pin(Self {
            name: name.into(),
            mutate_by_holder: 0,
            my_holder: holder,
            _pinned: PhantomPinned,
        });

        let this_ptr: *mut _ = unsafe { this.as_mut().get_unchecked_mut() };
        this.my_holder.borrow_mut().set_of_me.insert(this_ptr);

        this
    }

    /// Allows you to mutate a value within me.
    /// Run this from `Holder` to see what happens.
    fn mutate_me(self: Pin<&mut Self>, val: i32) {
        let this = unsafe { self.get_unchecked_mut() };
        this.mutate_by_holder += val;
    }
}

impl Drop for Me {
    fn drop(&mut self) {
        println!("Dropping {:#?}", self);
        let this = &(self as *mut _);
        self.my_holder.borrow_mut().set_of_me.remove(this);
    }
}

/// A test function to play with `Holder`s.
fn make_ref_of_holder(holder: RcCell<Holder>) {
    let holder = Rc::clone(&holder);
    println!("Making a ref of {:?}", holder);
    println!("No. of refs = {}", Rc::strong_count(&holder));
}

pub fn main() {
    let holder = Holder::new();

    // be explicit about `Rc`'s cloning
    let a = Me::new(Rc::clone(&holder), "a");
    let b = Me::new(Rc::clone(&holder), "b");

    holder.borrow().mutate_value_of_me(455);
    make_ref_of_holder(Rc::clone(&holder));
}

If you look at the code carefully and the code that was shown before, you'd notice that we are suffering from one big issue: Management of duplicate objects.

Duplicate Objects

To see what the problem is, add the following line to the code given in the solution.

// ...
pub fn main() {
    let holder = Holder::new();

    // be explicit about `Rc`'s cloning
    let a = Me::new(Rc::clone(&holder), "a");
    let b = Me::new(Rc::clone(&holder), "b");
+   let c = Me::new(Rc::clone(&holder), "b");

    holder.borrow().mutate_value_of_me(455);
    make_ref_of_holder(Rc::clone(&holder));
}

We might think, "Oh since we are storing our Me objects in a set, c will not be added."

But notice that the definition of set_of_me is BTreeSet<*mut Me>. Which means, the set stores unique pointers to Me, not unique Mes itself! And that's why c gets stored in holder.

So what's the solution?

One way to solve this issue is to create a wrapper type over *mut Me that will allow dereferencing, and thus comparison of objects.

We will discuss about that in Part 2.

Conclusion

I hope this article helps you in understanding Rust's Self Referential Structs a bit better.

Questions? Comments? Concerns? Please put them down below and I'd be happy to help you.

Cover Image Source: Manjaro's /usr/share/backgrounds folder 😃

Image taken from Rust Language's Async Tutorial Book ↩
Taken from https://doc.rust-lang.org/std/pin/index.html ↩

Managing Git credentials the easy way

Arunanshu Biswas — Sat, 31 Jul 2021 07:31:02 +0000

Introduction

Credential management in Git should not be difficult to use or configure, but (unfortunately) looking at the existing tools, it is difficult to even think about it, let alone configure them without errors.

NOT. ANY. MORE!

Presenting git-credential-netconf, an easy yet powerful way to manage your Git credentials by harnessing the power of GnuPG!

Getting Started

The following guide assumes you have a GPG key.

0. Installing `git-credential-netconf`

pip install git-credential-netconf

Visit the project page.

1. Create a `.netconf` file in your home directory

touch ~/.netconf

2. Start by filling in your username and password

[conf]
login = yourname
password = very-secret-password

The .netconf file uses configuration file format.

Learn more about .netconf file.

3. Encrypt your `.netconf` file with `gpg`

gpg --recipient yourname@example.com --output ~/.netconf.gpg \
    --encrypt --sign ~/.netconf

And remove your original .netconf for security:

shred -u ~/.netconf

4. Tell Git to use it

⚠️ Make sure that ~/.local/bin is in $PATH

git config --global credential.helper 'netconf --debug'

5. Now use git without hassle!

git push

This will prompt git-credential-netconf to decrypt the ~/.netconf.gpg file using GPG and fetch the username and password, among other values.

You'll be asked for the password to GPG private key when decrypting.

Conclusion

I hope this article helps you in managing your precious credentials a bit better.

Questions? Comments? Concerns? Please put them down below and I'd be happy to help you.

Image Source: Manjaro's /usr/share/backgrounds folder 😃

Demystifying Mutability and References in Rust

Arunanshu Biswas — Mon, 31 May 2021 12:37:21 +0000

The Un-Sandwiching rule

All references and reference related problems follow the "un-sandwiching"
rule.

The "un-sandwiching" rule states that:

Given a value, you cannot use a mutable reference between an immutable reference's declaration "zone" and immutable reference's usage "zone". Also, you cannot use mutable or immutable reference between a mutable's declaration zone and mutable's usage zone.

Note:

"Un-Sandwiching rule" is just a made up name that I created to better explain the quirks of immutable and mutable references. Don't use this name in forums!

What does it mean?

This rule is a mashed and simplified form of:

At any given time, you can have either one mutable reference or any number of immutable references. References must always be valid.

(I agree that the "un-sandwiching" rule looks more lengthy, but it will help
you to simplify things.)

Explaination of the rule

The rule can be broken down into two parts.

You cannot use a mutable reference while you are between the immutable reference's declaration and usage zone. However, you may have unlimited immutable references of the same value.
You cannot have any reference (immutable or mutable), if you are between a mutable reference's declaration and usage zone.

A demonstration

fn main() {
    let mut a = String::from("hello world");

    let immut_ref = &a; // <- declaration zone
    /*


       This is the "un-sandwiching" zone. You can have unlimited immutable
       references to `a`, directly and indirectly (more on that below), but
       *NOT A SINGLE MUTABLE REFERENCE IS ALLOWED*.


    */
    println!("{}", immut_ref); // <- usage zone
}

What is allowed?

You may have these types of references between that zone (not an exhaustive list):

    let b = &a; // (direct) *immutable* reference to `a`

    let c = &b; // (indirect) *immutable* reference to `b` (as `b` is immutable too)

    let d = &(*b); // (indirect) *immutable* reference to `a` via `b`

What is not allowed?

These (or any variations of these) are illegal:

    let b_mut = &mut a; // You can see why. It clearly violates the "un-sandwiching"
                        // rule as we are creating a mutable reference between the
                        // declaration and usage zone.

    a.push_str("breeze"); // Cannot do this because `a` is an "immutable" now.
                          // More on this below.

Let us look at the a.push_str("breeze") statement at a more granular level.

The signature of push_str is:

fn push_str(&mut self, string: &str) {
    // ...
}

Can you see what's going on here?

In the signature, we have a &mut self, which basically means &mut a (since self is a). But going back to the "un-sandwiching" rule, we see that we cannot have a mutable reference in the declaration-usage zone.

What do I mean by a becomes "immutable"?

I say a becomes "immutable" because you cannot mutate a as long as you are in the "un-sandwiching" zone. You can only have multiple immutable references.

What if we have multiple immutable references?

Let's say we have:

fn main() {
    let mut a = String::from("great breeze");

    let b = &a;
    // use `b`
    // use some more
    // ...

    let c = &a;
    // use `c`
    // use `b`
    // ...

    let d = &c;
    println!("{}", b); // last usage of `b`.
    // use `d`
    // use `c`

    do_something(d); // last usage of `d`
    // use `c`
    // ...

    do_something_else(c); // last usage of `c`
}

Whew! That's a lot of jumbled references! Let's see this through our declaration-usage lens.

b
│
│
│      c
│      │
│      │
▼      │      d
~b     │      │
       │      │
       │      │
       │      ▼
       │      ~d
       │
       │
       ▼
       ~c

The starting points are declarations, and the ~s are the last usage of the references. Note that we don't care if a reference is a direct or an indirect one (eg. d is an indirect reference).

From this diagram, we can clearly see that the "un-sandwiching" zone ranges from the first immutable reference b, to the last usage of immutable reference c. Between b and ~c, you cannot have any mutable reference.

And what about moves?

    let mut a = String::from("some");
    let c = &a;

    let other = a; // is this possible?

    println!("{}", c);

This is very simple to answer.

If the object cannot be copied, then you cannot perform a move within the zone.

Your homework

Deduce whether the following code is possible or not:

fn main() {
    let mut a = String::from("mutable");
    let b = &mut a; // a *mutable* reference

    // ...

    let c = &a; // 1. Is this possible?

    // ...

    println!("{}", b); // last usage
}

And what about this?

fn main() {
    let mut a = String::from("mutable");
    let b = &mut a; // a *mutable* reference

    // ...

    let d = &mut a; // 2. Is this possible?

    // ...

    println!("{}", b); // last usage
}

Explaination

let c = &a is not possible.

By the "un-sandwiching" rule we cannot have a immutable reference while we are within a mutable reference's "un-sandwiching" zone.

let d = &mut a is not possible.

You can see why. We are trying to create a mutable reference of the value while we are within the mutable reference's "un-sandwiching" zone. This is not allowed.

Conclusion

I hope this article helps you in understanding Rust a bit better.

Questions? Comments? Concerns? Please put them down below and I'd be happy to help you.

Image Source: Manjaro's /usr/share/backgrounds folder 😃

DEV Community: Arunanshu Biswas

Unlocking Conditional Component Rendering with Vue.js Slots

Introduction

Prerequisites

Creating the Slotted Component

Explanation

Understanding the $slots variable

Using the Slotted Component

The Results

Conclusion

The Async Side of SQLModel Relationships — Part 2

TL;DR

Eager Loading Approach

Lazy Loading Approach

Goal

Setting Up the Environment

Recap

Lazy or Not

Load Everything in One Go

Benefits

Drawbacks

Load as Required

Benefits

Drawbacks

A "Lazy" Tip

What's next?

Image Source

The Async Side of SQLModel Relationships — Part 1

TL;DR

Goal

Setting Up the Environment

Creating Relationships with SQLModel

Add a Foreign Key

Create Revision and Migrate

Create a Revision

Inspect the Revision

Migrate the Database

Playing with Database Models

Create a Session

The Experiment

Relationship Attributes

Using Models with Relationship Attributes

Analysing the Attribute Access Error

What's next?

Image Source

Async Database Operations with SQLModel

TL;DR

Goal

Setting Up the Environment

Installing an Async DB Engine

Create Async Engine

Create Async Session

Performing Database Operations

What's next?

Using SQLModel with Alembic

TL;DR

The Goal

Setting Up the Environment

Installing Alembic and SQLModel

Creating Models Using SQLModel

SQL Constraint Naming Convention

Initialize Alembic Migrations Directory

Add Database URL to alembic.ini

Set target metadata

Configure env.py

Batch Rendering Mode

SQLModel Custom Datatypes

Adding SQLModel Import to script.py.mako

Create the Initial Migration

Migrate our Database

What's next?

Solve cuInit: CUDA_ERROR_UNKNOWN: unkown error

Steps

If that does not work

Making a proper init.lua (For real this time!)

TL;DR

Aim

Setting up Packer

Downloading

Building Packer's Lazy Loader

Understanding the `$slots` variable

Add Database URL to `alembic.ini`

Configure `env.py`

Adding SQLModel Import to `script.py.mako`

Setting up `Packer`

`Holder` holds our Wrappers

Easy Conversion of `*mut Me` to `MeWrapper`

`Drop`ping Wrappers Properly

`RefCell` and Borrows

Search for `BorrowError`'s source

`PartialEq`

`Ord` and `PartialOrd`

`Ord`, Borrowing and `BorrowError`

To `#[derive]` or not to `#[derive]`

`Pin<T>` the objects

`PhantomPinned` to the Rescue

`Rc<T>` and `RefCell<T>`

Why `RefCell<T>`?

Why not `Weak<T>` for storing `Me`?

0. Installing `git-credential-netconf`

1. Create a `.netconf` file in your home directory

3. Encrypt your `.netconf` file with `gpg`