Sergey Nikolaev

Posted on Jan 8 • Originally published at manticoresearch.com

Manticore Load Emulator

TL;DR

Manticore Load Emulator is an open-source benchmarking tool that helps you validate and optimize your Manticore Search deployments. Whether you're planning a production deployment, debugging performance issues, or fine-tuning your configuration, this tool provides comprehensive testing capabilities with real-time monitoring and detailed analytics. Built for both simplicity and power, it supports everything from basic query testing to complex multi-process workload simulation.

Introduction

Here's a fun fact: when people ask if Manticore can handle their workload, we take it as a challenge. And when performance matters as much as it does to us, you need to be prepared. Enter Manticore Load Emulator — an open-source tool designed to put Manticore (and your hardware) through its paces.
Whether you're wondering if Manticore can handle your unique setup or just looking to squeeze every drop of performance out of it, this tool has you covered.
You can grab it on GitHub or by installing as a package from our repository (Manticore version > 6.3.8).

Why We Built It

We get asked a lot: "Can Manticore handle my workload?" Well, the short answer is usually "Yes." But instead of making you take our word for it, we built a tool so you can test it yourself. Manticore Load Emulator is all about transparency and empowerment—plus, it's a lot of fun to push your hardware to its limits.

Use Cases

1. Performance Validation for New Deployments

Before going live with Manticore Search, use the Load Emulator to simulate your expected workload. This ensures your infrastructure is properly scaled and configured for peak performance.

2. Debugging and Troubleshooting

Identify bottlenecks or performance degradation by running controlled tests. For example, analyze how specific query patterns or large datasets impact query latency and throughput.

3. Optimization of Existing Setups

Fine-tune configurations such as batch sizes, thread counts, or query caching to achieve optimal performance. Use the hyperparameter testing feature to automate comparisons.

4. Evaluating Infrastructure Changes

Planning to upgrade your hardware, adjust database sharding, or move to a different cloud provider? The Load Emulator allows you to benchmark and compare performance across different setups.

5. Stress Testing for Scalability

Simulate extreme workloads to evaluate how well your deployment scales under heavy traffic. This is especially useful for preparing for events like product launches or seasonal spikes.

Key Features

Here's why Manticore Load Emulator is a standout tool for benchmarking and testing:

1. SQL-Powered Simplicity

The tool leverages Manticore's SQL support, making it easy to emulate a wide variety of workloads. Writing your load or query scenarios is as simple as writing SQL commands.

2. High Concurrency Support

Simulate real-world, high-load scenarios using multiple threads and processes. If your server can handle it, this tool will push it to the brink.

3. Custom Query Generation

Generate dynamic queries with flexible patterns, from random text to precise ranges of integers, floats, or arrays:

value Exact value to use
<increment> Auto-incrementing value starting from 1
<increment/1000> Auto-incrementing value starting from 1000
<string/3/10> Random string, length between 3 and 10
<text/20/100> Random text with 20 to 100 words
<text/{/path/to/file}/10/100> Random text using words from file, 10 to 100 words
<int/1/100> Random integer between 1 and 100
<float/1/1000> Random float between 1 and 1000
<boolean> Random true or false
<array/2/10/100/1000> Array of 2-10 elements, values 100-1000
<array_float/256/512/0/1> Array of 256-512 random floats, values between 0 and 1

4. Batch Loading

Efficiently load millions of records at once with configurable batch sizes.

5. Real-Time Monitoring & Analytics

Track your test performance with comprehensive metrics including:

Progress and throughput (QPS)
Detailed latency percentiles
Performance bottlenecks identification
Real-time status updates ### 6. Flexible Configuration Command-line arguments let you tweak every aspect of your workload. Need to simulate multiple types of queries? Use the --together option to run different workloads in parallel. ### 7. Hyperparameter Testing Compare thread counts or batch sizes in a single run with comma-separated values. For example:

--threads=1,2,4,8 --batch-size=100,1000,10000

8. Quiet Mode for Analysis

In --quiet mode, the tool outputs semicolon-separated results that make it easy to visualize by copying directly into Google Sheets or Excel.

9. Multi-Process Support

Need to test sharded setups or multiple workloads simultaneously? This powerful flag allows you to run multiple test configurations in parallel within the same test session. Instead of running tests sequentially, which can be time-consuming, --together combines your test runs into a single, efficient execution.
For example, you can:

Test multiple shards of your database simultaneously
Run the same test suite against different database versions
Validate your application across various configuration settings
Compare performance metrics between different setups Simply append --together to your test command, and specify multiple test configurations. The tool will handle the parallel execution while keeping the results organized and easily comparable.

10. Graceful Shutdown

If you need to stop mid-test (or accidentally hit Ctrl+C), the tool ensures a clean termination without leaving things messy.

11. Two Modes of Latency Tracking

Histogram-Based: Memory-efficient and approximate for broad trends
Precise: For exact latency percentiles, because sometimes precision is key

Getting Started

Here's a quick example to get you started:

Example 1: Simple Insert

Insert 1 million integers in batches of 10K:

manticore-load \
  --drop \
  --init="create table t(a int)" \
  --load="insert into t values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --total=1000000

Example 2: Simulating Concurrency

7-8% CPU load and 160K docs per second seems too low for your workload? Let's load it up by increasing the thread count:

manticore-load \
  --drop \
  --init="create table t(a int)" \
  --load="insert into t values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --threads=16 \
  --total=10000000

OK, now we got 665K at 25% CPU load. Looks like we can squeeze more from the instance, but it requires writing to multiple tables at once. Let's do that:

Example 3: Sharded Workload

Test a sharded environment by loading data into multiple tables simultaneously:

manticore-load --quiet \
  --drop \
  --init="create table t(a int)" \
  --load="insert into t values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --threads=8 \
  --total=5000000 \
  --together \
  --drop \
  --init="create table t2(a int)" \
  --load="insert into t2 values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --threads=8 \
  --total=5000000 \
  --together \
  --drop \
  --init="create table t3(a int)" \
  --load="insert into t3 values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --threads=8 \
  --total=5000000 \
  --together \
  --drop \
  --init="create table t4(a int)" \
  --load="insert into t4 values(0, <int/1/1000000>)" \
  --batch-size=10000 \
  --threads=8 \
  --total=5000000

Too much data and you don't need the progress? By adding --quiet flag, the tool will output only the final results:

Example 4: Combining Writes and Reads

Previous examples were focused on loading data. What if you want to test write/read performance? Let's also use a more realistic workload than just integers.

manticore-load \
  --total=1000000 \
  --batch-size=10000 \
  --drop \
  --init="create table products(name text, price float, stock int)" \
  --load="insert into products(name, price, stock) values('<text/3/10>', <float/1/100>, <int/1/100>)" \
  --together \
  --total=10000 \
  --load="select * from products where match('<text/2/5>') and price < 10"

We can see how the read performance degrades as more data is loaded.

Example 5: Hyperparameter Testing

Let's test how different thread counts affect the performance:

manticore-load \
  --total=10000 \
  --quiet \
  --threads=1,4,8,16,32,64 \
  --load="select * from products where match('<text/2/5>') and price < 10"

We can see that with the current schema and 1 million docs in the table the throughput can be up to 9858 selects per second at 4.3ms average latency with 6.5ms p95 latency.
Isn't it great you can make such conclusions in just a couple of minutes?

Conclusion

Manticore Load Emulator is more than just a benchmarking tool—it's your partner in ensuring optimal performance for your search deployments. Whether you're running a small proof-of-concept or scaling to handle millions of queries, this tool provides the insights you need to make informed decisions about your infrastructure and configuration.
Remember: the best performance testing is iterative. Start small, gather data, make adjustments, and test again. With Manticore Load Emulator, you have all the tools you need to build and validate a high-performance search solution.

DEV Community