DEV Community: Bernd Stübinger

How to Do MongoDB Indexes

Bernd Stübinger — Fri, 01 Aug 2025 08:24:30 +0000

You just finished building an awesome feature that involves some new database queries. You have added unit tests, you carefully validated it in your test environment, and you deployed it to production, waiting for cheers and happy feedback.

Except that... your database suddenly implodes, because one of your new queries is extremely slow, taking down the whole database cluster (and your application) with it as well.

Sound familiar? Here's how we avoid it.

What Happened?

But first of all: What was the issue, actually? You tested everything, even in a real environment, right? Right, but not with a real data volume.

In this case, your query was using a collection scan because it was not able to use an index. And while this worked locally and in your test environment, data volume on production was orders of magnitude larger, resulting in significantly higher load on the database and exponentially slower response times.

Surely, you could duplicate data from production to your test environment. But not only does that come with a certain price tag, you would also have to consider data anonymization and data access, as test environments are usually less restricted - but still must not leak production data.

So what's the better way?

Requirements

To follow our approach, you need to maintain your MongoDB indexes in your application code. This feels like a natural choice when you are already maintaining your queries and data model there, and helps to avoid other problems as well. But more on that later.

Let's assume you have a simple DataMongoRepository to manage a collection of Data documents (yes, naming things is one of the two challenges in software engineering):

class Data(val id: Long, val name: String)

class DataMongoRepository(mongoClient: MongoClient) {
    private val database = mongoClient.getDatabase("database")
    private val collection = database.getCollection<Data>("datas")

    suspend fun insert(data: Data) {
        collection.insertOne(data)
    }

    suspend fun fetchById(id: Long): Data? =
        collection.find(Filters.eq(Data::id.name, id)).firstOrNull()
}

In that case, a typical test could look like this:

class DataMongoRepositoryTest {
    private val mongoContainer = MongoDbTestContainer().also { it.start() }
    private val mongoClient = MongoClient.create(
        mongoContainer.getReplicaSetUrl("database")
    )
    private val repository = DataMongoRepository(mongoClient)

    @Test
    fun `test simple insert`() {
        runBlocking {
            // Given
            val data = Data(1, "data1")
            repository.insert(data)

            // When
            val fetched = repository.fetchById(data.id)

            // Then
            assertEquals(data, fetched)
        }
    }
}

It starts an actual MongoDB via testcontainers, inserts some test data and then fetches it to see if it is still the same.

Using an Index

Now, to ensure that this query works with higher loads, let's add an index in our DataMongoRepository:

class DataMongoRepository(mongoClient: MongoClient) {
    ...

    init {
        runBlocking {
            collection.createIndex(Indexes.ascending(Data::id.name))
        }
    }

   ...
}

If we run the test again, it still succeeds (as expected, of course).

But what if we forgot to add this index? Can we somehow make this more robust?
Yes, we can!

Ensuring Index Usage

The magic lies in MongoDB's notablescan parameter. When enabled, it prevents queries from running collection scans, and returns an error instead.

The parameter can be configured using the setParameter command on the admin database:

mongoClient
    .getDatabase("admin")
    .runCommand(
        Document.parse("{ setParameter: 1, notablescan: 1 }")
    )

In our test setup (here simplified as @BeforeTest) we enable this as default for all tests, which allows us to quickly catch new queries that would try to do a collection scan, and react accordingly:

class DataMongoRepositoryTest {
    ...

    @BeforeTest
    fun setUp() {
        runBlocking {
            mongoClient.getDatabase("admin").runCommand(
                Document.parse("{ setParameter: 1, notablescan: 1 }")
            )
        }
    }

    ...
}

Let's see how this works by adding a query by name:

class DataMongoRepository(mongoClient: MongoClient) {
    ...

    suspend fun fetchAllByName(name: String): List<Data> =
        collection.find(Filters.eq(Data::name.name, name)).toList()
}

And a corresponding test:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test fetch by name`() {
        runBlocking {
            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

Running the new test fails with an exception:

com.mongodb.MongoQueryException: Command failed with error 291 (NoQueryExecutionPlans)

Nice, isn't it?

To make that test work, we could of course add an index for the name field. But that's not always desired, so let's see how we can disable validation for a particular test:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test fetch by name`() {
        runBlocking {
            // Allow table scan
            mongoClient.getDatabase("admin").runCommand(
                Document.parse("{ setParameter: 1, notablescan: 0 }")
            )

            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

Et voilà.

By using a Kotlin extension function we can make this even more concise:

class DataMongoRepositoryTest {
    ...

    // Runs the given [block] with `notablescan` disabled
    private fun allowTableScan(block: suspend CoroutineScope.() -> Unit) = runBlocking {
        mongoClient.getDatabase("admin").let {
            it.runCommand(Document.parse("{ setParameter: 1, notablescan: 0 }"))
            block()
            it.runCommand(Document.parse("{ setParameter: 1, notablescan: 1 }"))
        }
    }

    @Test
    fun `test fetch by name`() {
        allowTableScan {
            // Given
            val data1 = Data(1, "data")
            val data2 = Data(2, "data")
            repository.insert(data1)
            repository.insert(data2)

            // When
            val fetched = repository.fetchAllByName("data")

            // Then
            assertEquals(listOf(data1, data2), fetched)
        }
    }
}

This can be useful for e.g. complex search queries with multiple parameters, where you either cannot or don't want to create an index for every possible combination of fields. Indexes are not for free, after all - they consume memory and need to be maintained when fields are updated, incurring additional overhead.

Unique Constraint Validation

As mentioned, indexes are not only created for performance reasons. Let's get back to our example: Isn't an id actually supposed to be unique?

The following test runs without error, so obviously there is no guarantee:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test duplicate insert`() {
        runBlocking {
            val data1 = Data(1, "data1")
            repository.insert(data)
            repository.insert(data)
        }
    }
}

In fact, we now have two Data documents in our collection with id=1:

[
  Document{{_id=688b83c7bc39ec12e9b2a15a, id=1, name=data1}},
  Document{{_id=688b83c7bc39ec12e9b2a15b, id=1, name=data1}}
]

This is especially dangerous because fetchById will simply ignore all except the first document, so it may take a while to actually notice that issue.

Let's fix the problem by making our index unique using IndexOptions().unique(true):

class DataMongoRepository(mongoClient: MongoClient) {
    private val database = mongoClient.getDatabase("database")
    private val collection = database.getCollection<Data>("datas")

    init {
        runBlocking {
            collection.createIndex(
                Indexes.ascending("id"), IndexOptions().unique(false)
            )
        }
    }

    ...
}

Now the test succeeds and we can validate the expected error code:

class DataMongoRepositoryTest {
    ...

    @Test
    fun `test duplicate insert`() {
        runBlocking {
            // Given
            val data = Data(1, "data1")
            repository.insert(data)

            // When/Then
            val exception = assertFailsWith<MongoWriteException> {
                repository.insert(data)
            }
            assertEquals(ErrorCategory.DUPLICATE_KEY, ErrorCategory.fromErrorCode(exception.code))
        }
    }
}

Summary

We now have an easy way of validating database indexes, and an automated reminder that breaks our build if we accidentally add a new query without corresponding index.

Naturally, this cannot ensure that all queries use an optimal index. But while choosing the optimal index for a given query can have significant impact on performance, it is not always trivial, and you've already come a long way when you can at least guarantee that a query uses any index in order to avoid worst case scenarios.

And sometimes, all it needs is a little reminder, so there is really no reason to not include index validation in your build - especially, if it is such a trivial implementation.

get to know us 👉 https://mms.tech 👈

An Elegant Way to Solve Multi-Tenancy

Bernd Stübinger — Tue, 30 Jan 2024 16:14:46 +0000

In this article, I will show you an elegant way to introduce multi-tenancy in an already established code base for a Kotlin/Koin stack. And while the examples are specific to this stack, the general idea should be applicable to any dependency injection framework that supports qualifiers.

History

Our product initially started below the radar to support the migration of MediaMarktSaturn's webshop to the new platform. Back then, we had to replace an existing legacy promotion system by the beginning of the Black Friday season, which meant that we already had a fixed (business) scope and a fixed deadline.

Naturally, we had to cut short on some of the more technical aspects of our code base. One of that aspects was multi-tenancy, in our case different countries and sales lines - Media Markt Germany would be a different tenant than Saturn Germany or Media Markt Austria.

Of course, we knew that we had to deal with multiple tenants at some point but since the whole migration targeted mediamarkt.de only, we could easily postpone that topic until after the first go-live. We already included the relevant parts in our data model and in our API, though, so that we wouldn't have to perform a data migration and consumers of our system could keep their existing integrations.

For our implementation we used the much easier "ignorance" strategy, i.e. we didn't deal with tenants anywhere - and in the (few) places where we needed to work with a tenant, we simply hardcoded it to Media Markt Germany.

Initial Setup

We are using a pure Kotlin reactive stack with Ktor as web framework and Koin for dependency injection and application configuration. Our product manages coupons and calculates discounts for customers, i.e assesses their baskets with respect to pre-configured promotion rules.

In its essence, our setup looked like this:

val serviceModule = module {
    single { OutletRepository() }
    single { CalculationService() }
    single { PromotionRepository() }
    single { CouponRepository() }
}

You can think of a module as a space to collect all your Koin-managed components, similar to "beans" in Spring or CDI. single declares a singleton component that will be instantiated only once and re-used for every injection point.

In our routing layer we receive basket assessment requests and forward them to our main service, the CalculationService:

routing {
    val calculationService: CalculationService = get()

    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        call.respond(calculationService.calculateBasket(request))
    }
}

The CalculationService then fetches active promotions from the PromotionRepository and applies coupons using a CouponRepository. Naturally, this setup has different requirements to multi-tenancy: While coupons and active promotions differ between country and sales line, the core calculation logic will always stay the same and also outlet master data will be identical across all tenants. Coupons are identified by their code and thus even need to be separated on persistence level, so that multiple tenants can use identical codes.

Inside our CalculationService we use get to let Koin eagerly resolve and inject the managed instance(s) of other services:

class CalculationService {
    private val promotionRepository: PromotionRepository = get()
    private val couponRepository: CouponRepository = get()
    private val outletRepository: OutletRepository = get()
    ...
}

From Simple...

When we eventually added support for multi-tenancy, we already had an established code base, so we were facing a few challenges:

We couldn't afford (and didn't want) to rebuild everything from scratch
We had lots of functions that required to know which tenant they were working with
We also had lots of tests that didn't know anything about tenants yet

So we were looking for a solution that would allow us to keep as much of our current implementation as possible while at the same time didn't force us to introduce changes everywhere. We decided to separate all data on database level already so that our tenants could work independently from each other - I've already mentioned coupon codes above, but this can be seen as a good practice in general.

The simple approach was to determine a tenant from the request, and then add a tenant parameter to each subsequent function call. Of course, that wasn't exactly... elegant.

We had some places that experimented with the CoroutineContext to transfer tenant information but there were still a lot of places (including tests) that needed to be adapted, and back then we were just starting to understand how coroutines work. Also, we had to rely on the presence of a tenant during runtime, and actually wanted to have a bit more compile-time safety.

...to Elegant

So instead, we opted for "tenant-aware" services. We divided our existing service implementations into those that would provide common functionality and those that needed a tenant. For the latter we introduced a TenantAware interface with a single property:

interface TenantAware {
    val tenant: Tenant
}

All services that needed a tenant would now implement this interface and receive a tenant in their constructor. So our module configuration changed to:

val serviceModule = module {
    single { OutletRepository() }
    SupportedTenant.values().forEach { tenant ->
        single(named(tenant.id)) { CalculationService(tenant) }
        single(named(tenant.id)) { PromotionRepository(tenant) }
        single(named(tenant.id)) { CouponRepository(tenant) }
    }
}

This way, we had e.g. an instance of PromotionRepository for MediaMarkt Germany and one for Saturn Germany, and within that instance we had reliable access to the tenant property whenever we needed it - for example, when referring to other tenant-aware services that would now resolve the tenant-specific instance from Koin:

class CalculationService(override val tenant: Tenant) : TenantAware {
    private val promotionRepository: PromotionRepository = get(named(tenant.id))
    private val couponRepository: CouponRepository = get(named(tenant.id))
    private val outletRepository: OutletRepository = get()
    ...
}

This allowed us to limit all new tenant functionality to our routing layer, and we didn't have to change anything in our business code because every tenant-aware instance would only ever talk to other instances for the same tenant:

routing {
    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        val calculationService = get<CalculationService>(named(request.tenant.id))
        call.respond(calculationService.calculateBasket(request))
    }
}

And by adding a tenant suffix to our database collection (we are using MongoDB) we could easily achieve data separation in our persistence as well:

class PromotionRepository(override val tenant: Tenant) : TenantAware {
    val collectionName = "promotions.${tenant.id}"
    ...
}

In our tests we only had to adapt our injection points and didn't need to touch any of the actual test methods - voila! If needed, we could easily test multiple tenants by simply injecting different instances of a service.

Extension Functions

As last polish, we used Kotlin's extension functions to simplify our module definition and dependency resolution:

val Tenant.qualifier
    get() = named(id)

inline fun <reified T> Module.singleByTenant(noinline block: Scope.(Tenant) -> T) =
    SupportedTenant.values().map { tenant ->
        single(tenant.qualifier) { block(tenant) }
    }

inline fun <reified T : Any> get(tenant: Tenant) = get<T>(tenant.qualifier)

val serviceModule = module {
    single { OutletRepository() }
    singleByTenant { CalculationService(it) }
    singleByTenant { PromotionRepository(it) }
    singleByTenant { CouponRepository(it) }
}

class CalculationService(override val tenant: Tenant) : TenantAware {
    private val promotionRepository: PromotionRepository = get(tenant)
    private val couponRepository: CouponRepository = get(tenant)
    private val outletRepository: OutletRepository = get()
    ...
}

routing {
    get("basket-assessments") {
        val request = call.receive<CalculationRequest>()
        val calculationService = get<CalculationService>(request.tenant)
        call.respond(calculationService.calculateBasket(request))
    }
}

During the initial design we spent some days tweaking our multi-tenancy implementation but since then we've had no issues and are still using it without major changes.

And with our meanwhile grown understanding of coroutines, we have also been able to use our TenantAware interface for elegant tenant-aware authorization:

suspend inline fun <T> TenantAware.requireClaim(
    claim: Claim,
    block: () -> T
): T {
    // Check if current user has required `claim` for current `tenant`
    ...
}

class CouponService(override val tenant: Tenant) : TenantAware {
    fun createCoupon() = requireClaim(Claim.COUPON_CREATE) {
        ...
    }
}

Tell me what you think and especially, if you see even more potential for improving!

get to know us 👉 https://mms.tech 👈