DEV Community: Sanskriti Harmukh

Building Serverless APIs with Kotlin Ktor and Coroutines

Sanskriti Harmukh — Thu, 22 Oct 2026 23:24:17 +0000

Modern serverless platforms demand fast startup times, low resource overhead, and efficient concurrency. Kotlin, with its modern syntax and coroutine-based concurrency, and Ktor, its lightweight asynchronous web framework, provide an ideal combination for building high-performance serverless APIs. This guide demonstrates how to build a Product Catalog CRUD API using Kotlin and Ktor, structure a coroutine-driven application, handle concurrent requests efficiently, and package it for deployment across cloud providers in fully serverless or containerized environments. By the end, you'll have a working CRUD API, hardened for external calls, packaged as a portable JAR ready for serverless or container deployment.

Prerequisites

Before you begin:

Have access to an Ubuntu 24.04 based server as a non-root user with sudo privileges.
Install JDK 17+ (OpenJDK recommended).
Use the project's Gradle wrapper (./gradlew). Installing Gradle system-wide is optional.

Setting Up Your Development Environment

1. Verify Java and Gradle:

$ java -version
$ ./gradlew -v || echo "Gradle wrapper will be generated by Ktor project"

2. Generate a new project at start.ktor.io with:

Group: io.demo
Artifact: product-api
Package: io.demo.productcatalog
Dependencies: Server Core, Netty, Routing, Content Negotiation (kotlinx-serialization)

3. Unpack and open the project:

$ unzip product-api.zip -d product-api
$ cd product-api

4. Run the template to confirm the toolchain:

$ ./gradlew run

In another terminal:

$ curl -s http://localhost:8080/ || true

5. Ensure JSON serialization is enabled (add if missing):

// build.gradle.kts
dependencies {
   implementation("io.ktor:ktor-server-core-jvm")
   implementation("io.ktor:ktor-server-netty-jvm")
   implementation("io.ktor:ktor-server-content-negotiation-jvm")
   implementation("io.ktor:ktor-serialization-kotlinx-json-jvm")
   implementation("io.ktor:ktor-server-config-yaml-jvm")
   implementation("io.ktor:ktor-server-call-logging-jvm")
   implementation("io.ktor:ktor-server-compression-jvm")
   implementation("io.ktor:ktor-server-caching-headers-jvm")
   implementation("io.ktor:ktor-server-request-timeout-jvm")
   implementation("io.ktor:ktor-server-metrics-micrometer-jvm")
   implementation("ch.qos.logback:logback-classic:1.5.6")
}

6. Wire the plugin in the application:

// src/main/kotlin/io/demo/productcatalog/Application.kt
install(ContentNegotiation) { json() }

Understanding Ktor and Coroutines for Serverless

Ktor uses non-blocking I/O and Kotlin coroutines to handle high concurrency with low memory overhead, ideal for short-lived, autoscaled serverless workloads.

Event loop + suspending handlers: Handlers suspend on I/O instead of blocking threads, so a few threads serve many requests.
Coroutines vs. threads: Coroutines are lightweight; you can run thousands without the context-switch cost of OS threads.
Structured concurrency: Scope your launches (e.g., call or application scope) to avoid leaks across requests.
Backpressure: Keep endpoints non-blocking; offload CPU-heavy or blocking work to Dispatchers.Default/custom pools.
Cold starts: Trim dependencies and init work; defer heavy setup until first use.

Building Your First Serverless API with Ktor

Build a Product Catalog CRUD API to demonstrate routing, JSON serialization, and coroutine-friendly data access.

1. Create the project structure:

src/
└── main/
    ├── kotlin/io/demo/productcatalog/
    │   ├── Application.kt
    │   ├── Routing.kt
    │   ├── model/Product.kt
    │   └── repository/ProductRepository.kt
    └── resources/
        ├── application.yaml
        └── logback.xml

2. Create the model file at src/main/kotlin/io/demo/productcatalog/model/Product.kt:

@Serializable
data class Product(
   val id: String = UUID.randomUUID().toString(),
   val name: String,
   val price: Double,
   val stock: Int
)

@Serializable
data class ProductCreate(
   val name: String,
   val price: Double,
   val stock: Int
)

3. Create the in-memory repository file (simulated async) at src/main/kotlin/io/demo/productcatalog/repository/ProductRepository.kt:

object ProductRepository {
    private val products = mutableListOf<Product>()

    init {
        products += Product(name = "Laptop", price = 1200.0, stock = 20)
        products += Product(name = "Smartphone", price = 800.0, stock = 10)
        products += Product(name = "Headphones", price = 150.0, stock = 30)
    }

    suspend fun all(): List<Product> {
        delay(50)
        return products.toList()
    }

    suspend fun find(id: String): Product? {
        delay(50)
        return products.find { it.id == id }
    }

    suspend fun add(product: Product): Product {
        delay(50)
        products.add(product)
        return product
    }

    suspend fun update(id: String, product: Product): Product? {
        delay(50)
        val idx = products.indexOfFirst { it.id == id }
        return if (idx != -1) product.copy(id = id).also { products[idx] = it } else null
    }

    suspend fun delete(id: String): Boolean {
        delay(50)
        return products.removeIf { it.id == id }
    }
}

4. Create the routing configuration file at src/main/kotlin/io/demo/productcatalog/Routing.kt:

fun Application.configureRouting() {
    routing {
        get("/") { call.respond(mapOf("message" to "Product Catalog API")) }
        get("/ping") { call.respond(mapOf("message" to "pong")) }

        get("/products") { call.respond(ProductRepository.all()) }

        get("/products/{id}") {
            val id = call.parameters["id"] ?: return@get call.respond(HttpStatusCode.BadRequest)
            ProductRepository.find(id)?.let { call.respond(it) }
                ?: call.respond(HttpStatusCode.NotFound)
        }

        post("/products") {
            val dto = call.receive<ProductCreate>()
            if (dto.name.isBlank() || dto.price < 0 || dto.stock < 0) {
                return@post call.respond(HttpStatusCode.BadRequest, mapOf("error" to "Invalid product: name must be non-blank, price and stock ≥ 0"))
            }
            val saved = ProductRepository.add(Product(name = dto.name, price = dto.price, stock = dto.stock))
            call.respond(HttpStatusCode.Created, saved)
        }

        put("/products/{id}") {
            val id = call.parameters["id"] ?: return@put call.respond(HttpStatusCode.BadRequest)
            val dto = call.receive<ProductCreate>()
            if (dto.name.isBlank() || dto.price < 0 || dto.stock < 0) {
                return@put call.respond(HttpStatusCode.BadRequest, mapOf("error" to "Invalid product: name must be non-blank, price and stock ≥ 0"))
            }
            val updated = ProductRepository.update(id, Product(name = dto.name, price = dto.price, stock = dto.stock))
            updated?.let { call.respond(it) } ?: call.respond(HttpStatusCode.NotFound)
        }

        delete("/products/{id}") {
           val id = call.parameters["id"] ?: return@delete call.respond(HttpStatusCode.BadRequest)
            if (ProductRepository.delete(id)) call.respond(HttpStatusCode.NoContent)
            else call.respond(HttpStatusCode.NotFound)
        }
    }
}

5. Edit the application file with the following module configuration in src/main/kotlin/io/demo/productcatalog/Application.kt:

fun Application.module() {
    install(ContentNegotiation) { json() }
    configureRouting()
}

This enables JSON serialization and registers the routing configuration.

6. Run the application using Gradle:

$ ./gradlew run

Output:

> Task :run
2025-10-13 15:19:55.869 [main] INFO  Application - Autoreload is disabled because the development mode is off.
2025-10-13 15:19:56.298 [main] INFO  Application - Application started in 0.774 seconds.
2025-10-13 15:19:56.629 [DefaultDispatcher-worker-1] INFO  Application - Responding at http://0.0.0.0:8080
<==========---> 83% EXECUTING [19m 10s]
> :run

7. Verify the base routes:

$ curl http://localhost:8080/
$ curl http://localhost:8080/ping

Output:

{"message":"Product Catalog API"}
{"message":"pong"}

Handling Concurrency and Performance

Ktor serves each request on a lightweight coroutine, so a handful of threads can handle heavy concurrency. Tune the app to stay non-blocking, predictable, and fast.

Keep handlers non-blocking. Use withContext(Dispatchers.IO) for blocking work (file I/O, JDBC) so you don't stall the event loop.
```
suspend fun readFile(path: Path): String =
    withContext(Dispatchers.IO) { Files.readString(path) }
```

Use structured concurrency in routes. Launch child coroutines in a scope and await() them together. If one fails, Ktor cancels siblings and the request.

get("/report") {
    coroutineScope {
        val a = async { service.fetchA() }   // suspend fun
        val b = async { service.fetchB() }
        call.respond(mapOf("a" to a.await(), "b" to b.await()))
    }
}

Right-size dispatchers (only when needed). Let Ktor manage event loops. If you must isolate heavy I/O/CPU work, bound the pool and close it on shutdown.
```
val ioPool = Executors.newFixedThreadPool(8).asCoroutineDispatcher()
environment.monitor.subscribe(ApplicationStopped) { ioPool.close() }
```
Remove artificial delays in production. Replace delay() in repositories with real async I/O (database, HTTP client).

Add low-cost server features. Compression, caching headers, and request timeouts improve latency and protect the app.

fun Application.module() {
    install(ContentNegotiation) { json() }
    install(Compression) { gzip() }
    install(CachingHeaders) {
        options { CachingOptions(CacheControl.MaxAge(maxAgeSeconds = 60)) }
    }
    install(RequestTimeout) { requestTimeoutMillis = 5_000 }
    configureRouting()
}

Integrating External Services

Connect to databases and APIs with suspendable, time-bounded calls. Add retries sparingly, log context, and protect upstreams.

Ktor HTTP Client (timeouts, retries, JSON)

Configure a resilient Ktor HTTP client with JSON, timeouts, and limited retries.

val http = HttpClient(CIO) {
    expectSuccess = true

    install(ContentNegotiation) { json() }

    install(HttpTimeout) {
        requestTimeoutMillis = 2_000
        connectTimeoutMillis = 1_000
        socketTimeoutMillis  = 2_000
    }

    // Ktor's retry feature (Ktor 2.x)
    install(HttpRequestRetry) {
        retryOnServerErrors(maxRetries = 2)
        exponentialDelay(base = 200, maxDelayMs = 1_000)
        // Optionally retry on specific exceptions:
        retryIf { _, cause -> cause is HttpRequestTimeoutException }
        retryOnExceptionIf { _, cause -> cause is java.io.IOException }
    }

    defaultRequest {
        header("Accept", "application/json")
    }
}

Add client dependencies to your build file.

// build.gradle.kts
dependencies {
    implementation("io.ktor:ktor-client-core-jvm")
    implementation("io.ktor:ktor-client-cio-jvm")
    implementation("io.ktor:ktor-client-content-negotiation-jvm")
    implementation("io.ktor:ktor-serialization-kotlinx-json-jvm")
    implementation("io.ktor:ktor-client-http-timeout-jvm")
    implementation("io.ktor:ktor-client-retry-jvm")
    testImplementation("io.ktor:ktor-client-mock-jvm")
}

Close the shared client on shutdown to prevent resource leaks.

environment.monitor.subscribe(ApplicationStopped) { http.close() }

Inject the client and base URL for testability. Example with a repository and a MockEngine-based test setup:

class PriceRepository(
    private val client: HttpClient,
    private val baseUrl: String
) {
    suspend fun fetchPrices(): List<Price> = client.get("$baseUrl/prices").body()
}

// wiring (e.g., in Application.module)
val pricesBaseUrl: String = environment.config.propertyOrNull("app.services.prices.baseUrl")?.getString()
    ?: System.getenv("PRICES_BASE_URL")
    ?: "https://api.example.com"
val priceRepository = PriceRepository(http, pricesBaseUrl)

// test setup
val mockClient = HttpClient(MockEngine) {
    engine {
        addHandler { request ->
            if (request.url.encodedPath.endsWith("/prices")) {
                respond(
                    content = "[{\"sku\":\"A\",\"value\":10.0}]",
                    headers = headersOf("Content-Type" to listOf("application/json"))
                )
            } else error("Unhandled ${'$'}{request.url}")
        }
    }
    install(ContentNegotiation) { json() }
}
val mockRepo = PriceRepository(mockClient, "https://mock.local")

Call the client from a repository; keep functions suspend, add per-call timeouts if a dependency is flaky.

@Serializable data class Price(val sku: String, val value: Double)

object PriceService {
    suspend fun fetchPrices(): List<Price> =
        http.get("https://api.example.com/prices").body()
}

Per-call timeout override:

suspend fun fetchPricesFast(): List<Price> =
    http.get("https://api.example.com/prices") {
        timeout { requestTimeoutMillis = 1_000 }
    }.body()

Minimal "circuit breaker" guard (lightweight)

If an upstream often times out, short-circuit for a brief window. Keep it simple without extra libs.

class SimpleBreaker(
    private val threshold: Int = 5,
    private val openMillis: Long = 5_000
) {
    @Volatile private var failures = 0
    @Volatile private var openedAt = 0L

    fun allow(): Boolean =
        if (System.currentTimeMillis() - openedAt > openMillis) true else failures < threshold

    fun recordSuccess() { failures = 0; openedAt = 0 }
    fun recordFailure() {
        failures++
        if (failures >= threshold) openedAt = System.currentTimeMillis()
    }
}

val breaker = SimpleBreaker()

suspend fun guardedPrices(): List<Price> {
    if (!breaker.allow()) return emptyList() // fallback
    return try {
        val data = PriceService.fetchPrices()
        breaker.recordSuccess()
        data
    } catch (e: Exception) {
        breaker.recordFailure()
        emptyList() // degrade gracefully
    }
}

Database integration tips (JDBC/JPA/R2DBC)

JDBC/JPA: treat calls as blocking; wrap in withContext(Dispatchers.IO). Keep pools small but sufficient.
R2DBC / async drivers: prefer truly non-blocking drivers when available.
Set short query timeouts and fail fast on connection errors.

suspend fun findProductById(id: String): Product? =
    withContext(Dispatchers.IO) {
        dataSource.connection.use { conn ->
            conn.prepareStatement("SELECT ... WHERE id = ?").use { ps ->
                ps.queryTimeout = 2 // seconds
                ps.setString(1, id)
                ps.executeQuery().use { rs -> /* map to Product */ null }
            }
        }
    }

Observability (logs, metrics, traces)

Log request IDs: propagate a X-Request-ID header.
Emit timing metrics (Micrometer) around integrations.
Sample traces on external calls (OpenTelemetry) for latency hotspots.

install(CallLogging) {
    mdc("requestId") { call.request.headers["X-Request-ID"] ?: UUID.randomUUID().toString() }
}

Hardening checklist

Time-box every external call (client + per-call timeouts).
Add bounded retries with exponential backoff (no infinite retries).
Use fallback/cached responses for non-critical paths.
Validate and sanitize all inbound/outbound payloads.
Keep secrets in environment variables or a secret store; never hard-code.

Deploying and Monitoring Across Cloud Providers

Ship a portable artifact, run it the same way everywhere, and expose health/metrics for hands-off ops.

Package a single, portable JAR

Serverless/container platforms don't resolve Gradle deps, so build a fat JAR.

1. Add the following content in libs.versions.toml:

[plugins]
shadow = { id = "com.github.johnrengelman.shadow", version = "8.1.1" }

2. Add the following content in build.gradle.kts:

plugins {
    alias(libs.plugins.kotlin.jvm)
    alias(libs.plugins.kotlin.plugin.serialization)
    alias(libs.plugins.ktor)
    alias(libs.plugins.shadow)
}

application {
    mainClass.set("io.demo.productcatalog.ApplicationKt")
}

tasks {
    shadowJar {
        archiveBaseName.set("product-api")
        archiveClassifier.set("")
        archiveVersion.set("")
    }
}

3. Build and smoke-test locally:

$ ./gradlew shadowJar
$ java -jar build/libs/product-api-all.jar

Make the app cloud-friendly (port, health, graceful shutdown)

1. Configure the application port and external services in resources/application.yaml:

ktor:
  deployment:
    port: ${PORT:8080}   # use env var PORT if provided
  application:
    modules:
      - io.demo.productcatalog.ApplicationKt.module
app:
  services:
    prices:
      baseUrl: ${PRICES_BASE_URL:https://api.example.com}

This configuration allows the application to bind to a dynamic port provided by the cloud platform and defines an external service base URL through environment variables.

2. Add lightweight ops features:

fun Application.module() {
    install(ContentNegotiation) { json() }
    install(CallLogging)
    install(Compression) { gzip() }
    // optional timeouts to avoid hung requests
    install(RequestTimeout) { requestTimeoutMillis = 5_000 }

    routing {
        get("/ready") { call.respond(mapOf("status" to "ok")) }   // readiness
        get("/live") { call.respond(mapOf("status" to "ok")) }    // liveness
    }
    configureRouting()
}

These changes make the application suitable for cloud platforms by supporting dynamic port binding, health probes for orchestration systems, request logging, response compression, and graceful request timeouts.

Observability: logs, metrics, traces (portable setup)

Structured logs:

install(CallLogging) // keep message rates low, include request ids if available

Prometheus metrics:

// build.gradle.kts: implementation("io.micrometer:micrometer-registry-prometheus:<version>")
val registry = io.micrometer.prometheus.PrometheusMeterRegistry(io.micrometer.prometheus.PrometheusConfig.DEFAULT)
install(io.ktor.server.metrics.micrometer.MicrometerMetrics) { this.registry = registry }

routing { get("/metrics") { call.respondText(registry.scrape()) } }

Advanced Tips for Optimizing Ktor in Serverless Environments

Keep dependencies lean. Every jar adds cold-start time.
Avoid blocking the event loop. Wrap unavoidable blocking in a bounded IO dispatcher.
Warm paths on deploy. Optionally pre-load serializers/config.
Cache smartly. Memoize hot, static reads with short TTLs.
Benchmark routinely. Use wrk/k6 to tune timeouts, thread pools, and memory.

Next Steps

Add authentication/authorization (e.g., JWT) in front of the /products routes.
Swap the in-memory ProductRepository for an async-friendly database driver.
Wire up the Prometheus metrics endpoint and an OpenTelemetry exporter for production tracing.
Load-test the packaged JAR with wrk or k6 to validate timeout and retry settings under concurrency.

For the full guide with additional tips, visit the original article on Vultr Docs.

Using Rsync to Transfer Files in Linux

Sanskriti Harmukh — Thu, 22 Oct 2026 23:23:25 +0000

Rsync is a file transfer tool for transferring and synchronizing files across systems. It's commonly used for backups, file mirroring, and updating files between local directories or remote servers. Rsync reduces data transfer by copying only the differences between the source and destination files, and its speed, flexibility, and extensive options make it a popular choice for system administrators and developers. This guide installs Rsync and walks through using it to perform local and remote file transfers from your workstation. By the end, you'll have transferred and excluded files locally, and synchronized a directory to a remote machine over SSH.

Install Rsync

Rsync is available in the official repositories on most Linux distributions and is often pre-installed.

1. Install Rsync on Ubuntu and Debian using the APT package manager:

$ sudo apt install rsync

2. Install Rsync on FreeBSD using the PKG package manager:

$ sudo pkg install rsync

3. Install Rsync on Arch Linux using the Pacman package manager:

$ sudo pacman -S rsync

4. Install Rsync on Fedora, AlmaLinux, Rocky Linux, and VzLinux using the DNF package manager:

$ sudo dnf install rsync

5. Install Rsync on CentOS using the YUM package manager:

$ sudo yum install rsync

Rsync Command Syntax

Rsync uses the following command syntax.

rsync [OPTION] SRC DEST

Within the command:

[OPTION]: Represents the options or flags passed to the Rsync command.
SRC: The source files and directories you want to synchronize.
- Use USER@HOST:SRC when transferring files remotely over SSH.
DEST: The destination where the files or directories will be synchronized.
- Use USER@HOST:DEST when transferring files remotely over SSH.

Rsync Command Options

The following are the most common Rsync command options:

Option	Long Form	Short Form	Description
Help	`--help`	`-h`	Show Rsync command help
Verbosity	`--verbose`	`-v`	Increase verbosity
	`--quiet`	`-q`	Suppress non-error messages
File Selection	`--archive`	`-a`	Archive mode
	`--recursive`	`-r`	Recurse into directories
	`--relative`	`-R`	Use relative path names
Updates	`--update`	`-u`	Skip files that are newer on the receiver
	`--inplace`		Update destination files in-place
Backups	`--backup`	`-b`	Make backups (see `--suffix` & `--backup-dir`)
	`--backup-dir=DIR`		Make backups into hierarchy based in DIR
Symlinks	`--links`	`-l`	Copy symlinks as symlinks
	`--copy-links`	`-L`	Transform symlink into referent file/dir
Permissions	`--perms`	`-p`	Preserve permissions
Timestamps	`--times`	`-t`	Preserve modification times
Deletion	`--delete`		Delete extraneous files from dest dirs
	`--delete-excluded`		Also delete excluded files from dest dirs
Simulation	`--dry-run`	`-n`	Perform a trial run with no changes made
Remote Shell	`--rsh=COMMAND`	`-e`	Specify the remote shell to use
Size Control	`--max-size=SIZE`		Don't transfer any file larger than `SIZE`
	`--min-size=SIZE`		Don't transfer any file smaller than `SIZE`
Partial Transfer	`--partial`		Keep incomplete transferred files
File Filtering	`--exclude=PATTERN`		Exclude files matching PATTERN
	`--include=PATTERN`		Don't exclude files matching PATTERN
Output Format	`--progress`		Show progress during transfer
	`-P`		Same as `--partial --progress`
	`--human-readable`	`-h`	Output numbers in a human-readable format
Compression	`--compress`	`-z`	Compress file data during the transfer
IP Version	`--ipv4`	`-4`	Prefer IPv4
	`--ipv6`	`-6`	Prefer IPv6

Transfer Files and Directories Using Rsync

Set up a test environment before running the Rsync commands below.

1. Create a directory named rsync-test to work with:

$ mkdir rsync-test

2. Navigate into the newly created directory:

$ cd rsync-test

3. Create three sample files:

$ touch file1 file2 file3

Transfer Local Files

1. Copy file1, file2, and file3 to a new directory named dest. In Rsync, the last argument always represents the destination. Ensure the destination path is correctly specified to prevent accidental overwrites:

$ rsync file1 file2 file3 dest

file1 file2 file3: Source files to copy.
dest: Destination directory where the files will be copied.

2. Verify that the files were copied successfully:

$ ls dest/

Your output should be similar to the one below.

./dest/file1
./dest/file2
./dest/file3

Transfer Local Directories

Rsync does not copy files inside a directory by default. Use the -a flag to enable recursive copying, which enables archive mode, preserving timestamps, permissions, group, owner, and symbolic links.

1. Copy the dest directory you created earlier:

$ rsync -a dest/ dest-copy

-a: Enables archive mode to preserve file attributes while copying recursively.
dest/: The source directory to copy (trailing / means copy contents, not the directory itself. If you omit the trailing /, Rsync copies the entire directory instead of just its contents).
dest-copy: The destination directory.

2. Verify the copy operation:

$ ls dest-copy/

Your output should be similar to the one below. A dest-copy directory is created with the same content as the dest directory.

./dest-copy/file1
./dest-copy/file2
./dest-copy/file3

Exclude Files or Directories Using Rsync

To exclude specific files or directories from being copied, use the Rsync --exclude flag.

1. Copy the contents of the current directory, excluding file2 and all directories that start with dest:

$ rsync -a --exclude 'file2' --exclude 'dest*' . output

-a: Enable archive mode.
--exclude 'file2': Exclude files and directories named file2.
--exclude 'dest*': Exclude files and directories that start with dest.
.: Specify the current directory as the source.
output: The destination directory.

2. Verify the copy operation:

$ ls output/

Your output should be similar to the one below. Notice that the output directory contains only file1 and file3.

./output/file1
./output/file3

Synchronize Data to a Remote Machine Using Rsync

1. Synchronize files and directories to a remote machine:

$ rsync -hazvP --delete --stats . user@192.0.2.1:synced_from_local

-h: Output numbers in human-readable format.
-a: Enables archive mode for preserving metadata.
-z: Compresses data during transfer.
-v: Enables verbose output.
-P: Displays progress and resumes partial transfers.
--delete: Removes files from the destination that no longer exist in the source.
--stats: Provides a detailed summary of the transfer.
.: Specify the current directory as the source.
user@192.0.2.1:synced_from_local: Specify the remote machine as the destination. This consists of three parts:
- user: The username to access the remote machine.
- 192.0.2.1: The IP address of the remote machine.
- synced_from_local: The destination path.

2. Log in to the remote machine using SSH:

$ ssh user@192.0.2.1

3. Verify that the files have been synced:

$ ls synced_from_local/

Output:

synced_from_local/dest-copy/file1
synced_from_local/dest-copy/file2
synced_from_local/dest-copy/file3
synced_from_local/dest/file1
synced_from_local/dest/file2
synced_from_local/dest/file3
synced_from_local/file1
synced_from_local/file2
synced_from_local/file3
synced_from_local/output/file1
synced_from_local/output/file3

Warning on Using the `--delete` Rsync Option

Be cautious when using --delete, as it permanently removes files from the destination if they don't exist in the source. Always perform a dry run first:

$ rsync -a --delete --dry-run SRC DEST

Next Steps

Schedule recurring Rsync jobs with cron to keep a backup directory continuously in sync.
Combine --exclude with a .rsync-filter file to manage more complex exclusion rules.
Use --dry-run as a standard habit before any sync that includes --delete.
Run man rsync for the full list of options and remote-shell configurations.

For the full guide with additional tips, visit the original article on Vultr Docs.

Using Curl

Sanskriti Harmukh — Thu, 22 Oct 2026 23:22:54 +0000

Curl is a command-line utility and underlying library (libcurl) used to transfer data across a wide range of network protocols, including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, LDAP, MQTT, and more. It supports features like proxy support, user authentication, cookies, file uploads, and SSL connections, making it a go-to tool for developers, system administrators, and anyone working with web APIs or networked services. This guide walks through using curl to perform HTTP requests from your terminal, including downloading files, setting custom headers, working with different HTTP methods, handling redirects, sending JSON payloads, authenticating with servers, and troubleshooting with verbose mode. By the end, you'll be comfortable using curl to test APIs and automate network operations from the command line.

The Short Answer Version

If you just want a quick reference for common curl commands, here's a compact list of essentials:

# Basic GET request
$ curl https://example.com

# Save output to file
$ curl -o page.html https://example.com

# Follow redirects
$ curl -L http://example.com

# Send POST form data
$ curl -d "name=John&age=30" -X POST https://example.com

# Send JSON payload
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://example.com

# Add custom header
$ curl -H "Authorization: Bearer <token>" https://api.example.com

# Basic auth
$ curl -u user:pass https://example.com

Using Curl

This section expands on the quick reference above and explains basic curl operations in more detail, including file downloads, protocol handling, and request formatting.

The basic syntax of the curl command is:

$ curl [options] <URL>

`curl` Command Options

Option	Description
`-o <file>`	Save output to the specified file
`-O`	Save with the same name as the remote file
`-C -`	Resume an interrupted download
`-v`	Enable verbose mode (request/response debugging)
`-s`	Silent mode, hides progress meter
`-L`	Follow redirects
`-I`	Fetch only response headers
`-X <METHOD>`	Set HTTP method (`GET`, `POST`, `DELETE`, etc.)
`-d <data>`	Send POST form data
`-H "<header>"`	Add custom headers (e.g., `Content-Type`, `Authorization`)
`--user-agent`	Override the default User-Agent string
`-u user:pass`	Send basic authentication credentials
`-F`	Upload file using multipart form
`--fail`	Exit with error on 4xx/5xx responses, suppresses body output
`-w`	Output custom variables like time taken (`%{time_total}`)

Detailed Usage

The following sections explain how each curl command works in more detail, with examples and syntax breakdowns.

Download a Webpage

1. Download a webpage and print its source code to the terminal:

$ curl <URL>

Example:

$ curl https://www.example.com/index.html

Use FTP Protocol

1. Retrieve a file using FTP:

$ curl ftp://<host>/<path>

Example:

$ curl ftp://ftp.example.com/public/readme.txt

Save Output to a File

1. Save the downloaded output to a local file instead of printing it to the terminal:

$ curl -o <filename> <URL>
$ curl -O <URL>

-o: Save output with a custom filename.
-O: Save output using the remote filename.

Example:

$ curl -o localcopy.html https://example.com/index.html
$ curl -O https://example.com/index.html

Resume an Interrupted Download

1. Resume an interrupted download from where it left off instead of restarting from scratch:

$ curl -C -O <URL>

-C -: Resume download.
-O: Save using remote file name.

Example:

$ curl -C -O https://example.com/bigfile.zip

Enable Verbose Output

1. Enable verbose output to debug request/response communication:

$ curl -v <URL>

Example:

$ curl -v https://example.com

Making HTTP Requests

Curl provides a wide range of options to interact with web services, including inspecting headers, following redirects, changing HTTP methods, and customizing request headers.

Get Only Response Headers

1. Fetch only the response headers of a URL:

$ curl -I <URL>

Example:

$ curl -I https://example.com

Follow Redirects

1. Enable following of HTTP 3xx redirects:

$ curl -L <URL>

Example:

$ curl -L http://example.com

Change the HTTP Method

1. Set a custom HTTP method using -X:

$ curl -X [METHOD] <URL>

Example:

$ curl -X DELETE https://example.com/resource/123

Add Custom Headers

1. Use the -H flag to add one or more HTTP headers:

$ curl -H "Header-Name: value" <URL>

Example:

$ curl -H "User-Agent: Mozilla/5.0" -H "Cache-Control: no-cache" https://example.com

Set a Custom User-Agent

1. Use --user-agent to define the request's User-Agent string:

$ curl --user-agent "[Agent String]" <URL>

Example:

$ curl --user-agent "Mozilla/5.0 (Macintosh)" https://example.com

Send Form or JSON Data (POST)

Use curl to send form-encoded or JSON data in HTTP POST requests. This is essential for interacting with REST APIs or submitting data to a server.

1. Send form-encoded or JSON data:

$ curl -d "<key1=value1&key2=value2>" -X POST <URL>
$ curl -X POST -H "Content-Type: application/json" -d '<JSON>' <URL>

Example:

$ curl -d "name=John&age=30" -X POST https://example.com
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://example.com

Basic Authentication

When accessing resources protected by HTTP Basic Auth, use the -u option to include your credentials directly in the request.

1. Send basic auth credentials:

$ curl -u <username>:<password> <URL>

Example:

$ curl -u admin:secret https://example.com

Advanced Usage and Tips

This section covers advanced curl patterns commonly used in scripts, automation, and API integrations.

Exit on HTTP Errors (`--fail`)

1. Make curl exit silently with a non-zero code when HTTP errors like 404 or 500 occur:

$ curl --fail <URL>

--fail: Exit silently on HTTP errors (4xx, 5xx).

Example:

$ curl --fail https://example.com/missing-file

Use this in scripts to halt execution when a URL is unreachable or responds with an error.

Measure HTTP Request Time (`-w`)

1. Print the total time taken to complete a request:

$ curl -w "%{time_total}\n" -o /dev/null -s <URL>

-w: Custom output after completion.
%{time_total}: Total transfer time.
-o /dev/null: Discard body.
-s: Silent mode to suppress progress.

Example:

$ curl -w "%{time_total}\n" -o /dev/null -s https://example.com

This helps measure API response time or network latency.

Upload Files (`-F`)

1. Upload files using a multipart form submission:

$ curl -F "key=@filename.ext" <URL>

-F: Send form field with file.
@filename.ext: Reads the file from local disk.

Example:

$ curl -F "file=@report.pdf" https://example.com/upload

Use -F for uploading files as part of a form. Use --data-binary if you want to send raw files.

Authenticate with Bearer Token

1. Use bearer tokens to access secured APIs (e.g., OAuth2 access tokens):

$ curl -H "Authorization: Bearer <your_token>" <URL>

Example:

$ curl -H "Authorization: Bearer abc123" https://api.example.com/data

This is required when accessing private endpoints on services like GitHub, Stripe, or cloud APIs.

Quiet or Debug Output (`-s`, `-v`)

1. Suppress the progress bar and errors for scripts:

$ curl -s <URL>

2. Show full request and response details for debugging:

$ curl -v <URL>

Example:

$ curl -s https://example.com

$ curl -v https://example.com

Use -s for scripts and logging. Use -v when debugging failures or testing headers.

Next Steps

Script a health-check that uses curl --fail and -w "%{time_total}" to alert on slow or failing endpoints.
Combine -H "Authorization: Bearer <token>" with -X POST to scaffold quick API client tests.
Use -F to exercise file-upload endpoints before writing a full client library.
Review the official curl documentation for protocol-specific features and scripting patterns.

For the full guide with additional tips, visit the original article on Vultr Docs.

Using the Linux IP Command to Manage Server Network Interfaces

Sanskriti Harmukh — Thu, 22 Oct 2026 23:22:30 +0000

The ip command is a built-in Linux network management utility that offers a wide range of features and tools for managing network interfaces. It's an improved utility that combines the functionality of legacy commands such as ifconfig, arp, and route into a single tool for managing, monitoring, and troubleshooting your server's network interfaces. This guide covers using the ip command to manage ip objects such as links, addresses, routes, and neighbors, including interface configuration, static routing, enabling/disabling interfaces, and MTU management. By the end, you'll be able to inspect, configure, and troubleshoot your server's network interfaces using the ip command.

Prerequisites

Before you begin, you need to:

Deploy an Ubuntu 24.04 server instance to test the ip command functionalities.
Attach the server to a VPC network.
Access the server using SSH as a non-root user with sudo privileges.

How to Use the IP Command

The ip command uses the following syntax to manage network information and interfaces on your server.

$ ip [OPTIONS] OBJECT COMMAND [ARGUMENTS]

Within the above command:

OPTIONS: Sets the ip command operation mode.
OBJECT: Specifies the type of network object to view or manage. For example, link, address, route.
COMMAND: Defines the action to perform on the specified object. For example, add, del, show.
ARGUMENTS: Sets additional options to execute with the command.

View Network Interface Information

The ip command displays network interface information at varying levels of detail depending on the object and options used.

1. View information about all network interfaces available on the server:

$ ip link show

Your output should be similar to the one below.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 56:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

Based on the above output, the command displays the total number of available interfaces, the MTU value, state, and MAC address for each interface. Active interfaces use the UP state while inactive interfaces use the DOWN state. Interfaces with a DOWN state cannot send or receive network traffic on the server unless enabled.

2. Display network interfaces in an active and running state:

$ ip link show up

Output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 56:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

3. Use the -br option to display only important information about the network interfaces:

$ ip -br link show

Output:

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> 
enp1s0           UP             56:00:04:ef:40:23 <BROADCAST,MULTICAST,UP,LOWER_UP> 
enp8s0           UP             5a:00:04:ef:40:23 <BROADCAST,MULTICAST,UP,LOWER_UP>

4. Use the addr option, or its a shorthand, to view full information including IP addresses on all network interfaces:

$ ip a

Your output should look like the one below.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 56:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    inet 192.0.2.100/23 brd 192.0.2.255 scope global dynamic enp1s0
       valid_lft 78163sec preferred_lft 78163sec
    inet6 2a05:f480:2c00:1ef7:5400:4ff:feef:4023/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 2591891sec preferred_lft 604691sec
    inet6 fe80::5400:4ff:feef:4023/64 scope link 
       valid_lft forever preferred_lft forever
3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    inet 10.50.112.3/20 brd 10.50.127.255 scope global enp8s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5800:4ff:feef:4023/64 scope link 
       valid_lft forever preferred_lft forever

5. View specific information about a specific network interface. For example, view information about the VPC interface, enp8s0:

$ ip addr show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    inet 10.50.112.3/20 brd 10.50.127.255 scope global enp8s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5800:4ff:feef:4023/64 scope link 
       valid_lft forever preferred_lft forever

6. View detailed statistics about a specific network interface. For example, enp8s0:

$ ip -s link show enp8s0

Your output should be similar to the one below.

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast
      25426631   61808      0       0       0       0
    TX:  bytes packets errors dropped carrier collsns
       4351567   40677      0       0       0       0

Manage Interface IP Addresses Using the IP Command

IP addresses can be assigned to network interfaces using the Dynamic Host Configuration Protocol (DHCP) or static addressing to enable network connections on the interface. DHCP is the default address assignment protocol on most servers, but static IP addressing may be required when working with multiple addresses or new interfaces on your server.

1. Add a new IP address on your network interface. Replace enp8s0 with your desired interface name:

$ sudo ip addr add 192.168.1.10/24 dev enp8s0

The above command adds a new private network address 192.168.1.10 to the enp8s0 interface. Depending on your addressing scheme, the interface can run with multiple IP addresses that offer different functions or services on your server.

2. View the interface information to verify that the new IP address is available:

$ ip addr show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    inet 10.50.112.3/20 brd 10.50.127.255 scope global enp8s0
       valid_lft forever preferred_lft forever
    inet 192.168.1.10/24 scope global enp8s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5800:4ff:feef:4023/64 scope link 
       valid_lft forever preferred_lft forever

When using multiple IP addresses, you may need to configure routing to enable specific network connections to use the interface IP address.

3. Use the del option to remove IP addresses on network interfaces. For example, remove the 192.168.1.10/24 address on the enp8s0 interface:

$ sudo ip addr del 192.168.1.10/24 dev enp8s0

4. View the interface address information to verify that the address is removed:

$ ip addr show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff
    inet 10.50.112.3/20 brd 10.50.127.255 scope global enp8s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5800:4ff:feef:4023/64 scope link 
       valid_lft forever preferred_lft forever

Configure Static Routing Using the IP Command

Static routing directs traffic to remote networks through a specific network interface or IP address as the next hop to reach the network. Network requests to unmatched destinations are directed through the default route via the main server interface, while requests to unknown routes may fail due to missing route information.

1. View the default IP routing table:

$ ip route show

Your output should be similar to the one below.

default via 192.0.2.100 dev enp1s0 proto dhcp src 192.0.2.100 metric 100 
10.50.112.0/20 dev enp8s0 proto kernel scope link src 10.50.112.3 
192.0.2.0/23 dev enp1s0 proto kernel scope link src 192.0.2.100 
169.254.169.254 via 192.0.2.100 dev enp1s0 proto dhcp src 192.0.2.100 metric 100

Based on the above output, direct network addresses use the dev option while non-linked addresses use the via option to direct network traffic using the specified address as the next hop. The default route handles all unknown routes on the server unless specified in the routing table.

2. Add a new static route to a remote network such as 10.0.0.0/16 accessible through a server on the VPC network at 10.50.112.4 as the next hop:

$ sudo ip route add 10.0.0.0/16 via 10.50.112.4

3. View the routing table again to verify that the new route is available:

$ ip route show

Output:

default via 192.0.2.100 dev enp1s0 proto dhcp src 192.0.2.100 metric 100 
10.0.0.0/16 via 10.50.112.4 dev enp8s0 
10.50.112.0/20 dev enp8s0 proto kernel scope link src 10.50.112.3 
192.0.2.0/23 dev enp1s0 proto kernel scope link src 192.0.2.100 
169.254.169.254 via 192.0.2.100 dev enp1s0 proto dhcp src 192.0.2.100 metric 100

Enable or Disable Network Interfaces

Disabling a network interface stops it from sending or receiving traffic without removing its configuration, which is useful for maintenance or troubleshooting.

1. Use the ip link set command to disable the enp8s0 interface:

$ sudo ip link set dev enp8s0 down

2. View the network interface status and verify that the state is DOWN:

$ ip link show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST> mtu 1450 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

3. Enable the network interface:

$ sudo ip link set dev enp8s0 up

4. View the interface information to verify the new status:

$ ip link show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

Managing the Network Interface MTU (Maximum Transmission Unit)

MTU is the maximum size of a packet in bytes that can be transmitted over a network interface without fragmentation. The default MTU value on most network interfaces is 1500. Depending on your needs, you may modify the value to allow the transfer of large or small data packets on private networks such as a VPC network.

1. View your target interface MTU value. For example, the VPC interface enp8s0:

$ ip link show enp8s0

2. Verify the MTU value in your output, similar to the one below:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

3. Set a new MTU value on the interface. For example, 3000, to transfer large data packets and improve efficiency with other servers in the VPC network:

$ sudo ip link set dev enp8s0 mtu 3000

4. View the interface information again and verify the new MTU value:

$ ip link show enp8s0

Output:

3: enp8s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 3000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 5a:00:04:ef:40:23 brd ff:ff:ff:ff:ff:ff

Next Steps

Use ip neigh to inspect and manage the ARP/neighbor table on your server.
Persist static routes and IP addresses in your distribution's network configuration so they survive a reboot.
Combine ip -br output with a monitoring script to alert on interfaces that drop to a DOWN state.
Run man ip for the full list of ip objects and commands beyond what's covered here.

For the full guide with additional tips, visit the original article on Vultr Docs.

Scanning for Open Ports and Vulnerabilities with Nmap

Sanskriti Harmukh — Thu, 22 Oct 2026 23:21:07 +0000

Nmap (Network Mapper) is an open-source network discovery tool that identifies hosts, services, and potential security issues. It's an essential tool for system administrators and developers to map a server's network, letting you see which ports are open, what services are running, and what operating system a target system is using. This guide covers key security tasks with Nmap, including scanning a host for open ports, identifying services running on those ports, and detecting vulnerabilities using the Nmap Scripting Engine (NSE), plus how to interpret the output and take basic remediation steps. By the end, you'll have run multiple Nmap scans, detected a real misconfiguration, and fixed it.

Prerequisites

Before you begin, you need access to a Linux-based server as a non-root user with sudo privileges.

Install Nmap

1. Install nmap according to your operating system.

On Debian/Ubuntu-based systems:

$ sudo apt update && sudo apt install -y nmap

On Rocky Linux and other RHEL-based systems:

$ sudo dnf install -y nmap

2. Verify the installation:

$ nmap --version

You should see output similar to the following:

Nmap version 7.94SVN ( https://nmap.org )
Platform: x86_64-pc-linux-gnu
Compiled with: liblua-5.4.6 openssl-3.0.13 libssh2-1.11.0 libz-1.3 libpcre2-10.42 libpcap-1.10.4 nmap-libdnet-1.12 ipv6
Compiled without:
Available nsock engines: epoll poll select

The `nmap` Command

The nmap command uses the following syntax:

nmap [SCAN TYPES...] [OPTIONS] TARGET

[SCAN TYPES...]: Defines how Nmap scans. This group controls if and how Nmap probes the target for open ports.
[OPTIONS]: One or more optional flags and parameters that modify Nmap's behavior. They control the output's scope, speed, and verbosity.
TARGET: The hosts or networks to be scanned. You can provide a single IP, an IP range, a hostname, or a subnet.

Scan Types

Option	Description
`-sS`	TCP SYN scan (default, root)
`-sT`	TCP connect scan
`-sU`	UDP scan
`-sn`	Host discovery only (no port scan)
`-sV`	Detect service versions
`-O`	Detect operating system
`-A`	Aggressive scan (enables -sV, -O, -sC)

Command Options

Option	Flag	Description
Port Specification	`-p`	Specify which ports to scan, like `-p 22,80,443`.
Fast Scan	`-F`	Scans only the 100 most common ports instead of the default 1000.
Verbose Output	`-v`	Provides detailed output.

Perform a Basic Port Scan

A basic scan checks the 1,000 most common ports. This guide uses localhost as the target — replace it with an IP address or hostname you have explicit permission to scan.

1. Run a basic scan:

$ sudo nmap localhost

Sample output:

Starting Nmap 7.94SVN ( https://nmap.org ) at 2025-10-03 21:49 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000011s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 997 closed tcp ports (reset)
PORT     STATE SERVICE
22/tcp   open  ssh
3001/tcp open  nessus
5432/tcp open  postgresql

Nmap done: 1 IP address (1 host up) scanned in 0.07 seconds

Understanding the output:

Not shown: 997 closed tcp ports: These ports are closed or filtered.
PORT: Port number and protocol.
STATE: Whether the port is open, closed, or filtered.
SERVICE: The service Nmap detected running on the port (when identifiable).

In this example, open ports include 22 (SSH), 3001 (reported as nessus), and 5432 (PostgreSQL).

Scan for Service Versions and Operating Systems

Nmap can reveal detailed service information on open ports, including version numbers, which helps detect outdated or vulnerable services.

1. Use the -sV flag to probe open ports for running services:

$ sudo nmap -sV localhost

Sample output:

Starting Nmap 7.94SVN ( https://nmap.org ) at 2025-10-03 21:56 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000011s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 997 closed tcp ports (reset)
PORT     STATE SERVICE    VERSION
22/tcp   open  ssh        OpenSSH 9.6p1 Ubuntu 3ubuntu13.13 (Ubuntu Linux; protocol 2.0)
3001/tcp open  nessus?
5432/tcp open  postgresql PostgreSQL DB 9.6.0 or later

This scan shows specific versions for detected services, such as OpenSSH 9.6p1 and PostgreSQL 9.6.0 or later. The question mark after Nessus indicates that Nmap is unsure, and the service identification may require further verification.

Scan for Vulnerabilities Using the Nmap Scripting Engine (NSE)

The Nmap Scripting Engine (NSE) allows Nmap to use scripts to automate various networking tasks, including vulnerability scanning. The vuln script category tests services for known vulnerabilities. This section installs vsftpd, an FTP server, intentionally creates a vulnerability, and then detects and mitigates it using Nmap's scripting engine.

1. Install vsftpd.

On Ubuntu/Debian-based systems:

$ sudo apt install -y vsftpd

On Rocky Linux/RHEL-based systems:

$ sudo dnf install -y vsftpd

2. Enable anonymous login for vsftpd to create a vulnerability. Edit the vsftpd configuration file:

$ sudo nano /etc/vsftpd.conf

Find the line anonymous_enable=NO and change the value from NO to YES. Save and close the file.

3. Restart vsftpd:

$ sudo systemctl restart vsftpd

Now that you've intentionally created this issue, you can use a specific Nmap script to detect it. The ftp-anon script is part of the NSE and automatically tests whether an FTP server allows anonymous login.

4. Run a vulnerability scan using ftp-anon on port 21, the port used by vsftpd:

$ sudo nmap -p 21 --script ftp-anon localhost

Sample output:

Starting Nmap 7.94SVN ( https://nmap.org ) at 2025-08-22 11:04 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00010s latency).
Other addresses for localhost (not scanned): ::1

PORT   STATE SERVICE
21/tcp open  ftp
|_ftp-anon: Anonymous FTP login allowed (FTP code 230)

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

The scan detected an issue: Anonymous FTP login allowed (FTP code 230).

Steps to Mitigate

1. Edit the configuration file:

$ sudo nano /etc/vsftpd.conf

Find the line anonymous_enable=YES and change the value from YES to NO. Save and exit the file.

2. Restart the service to apply the changes:

$ sudo systemctl restart vsftpd

3. Run the Nmap scan on port 21 again:

$ sudo nmap -p 21 --script ftp-anon localhost

Output:

Starting Nmap 7.94SVN ( https://nmap.org ) at 2025-08-22 11:19 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00010s latency).
Other addresses for localhost (not scanned): ::1

PORT   STATE SERVICE
21/tcp open  ftp

Nmap done: 1 IP address (1 host up) scanned in 2.53 seconds

Next Steps

Automate recurring Nmap scans with cron to catch newly opened ports or services.
Explore the full vuln NSE script category to check for known CVEs beyond anonymous FTP.
Pair Nmap output with a firewall tool like UFW or firewalld to close unneeded ports.
Review the Nmap official documentation for more scan types and NSE scripts.

For the full guide with additional tips, visit the original article on Vultr Docs.

Setting Up Firewall Policies using Firewalld

Sanskriti Harmukh — Thu, 22 Oct 2026 23:20:24 +0000

Firewalld is a dynamic firewall application that provides a flexible way of setting up network filtering policies on a server by extending the kernel-based iptables functionality. It provides flexible configuration options, including zone-based firewall management policies that represent a specific level of trust depending on your server's network connections, to protect the system from unauthorized access. This guide walks through setting up firewall policies using Firewalld and configuring sample rules to filter network requests on a Linux server. By the end, you'll have Firewalld installed, zones and rules configured, and panic mode ready as a last-resort lockdown.

Prerequisites

Before you begin:

A Rocky Linux (or other RHEL-based) server to use as the workstation.
Access to the server using SSH as a non-root user with sudo privileges.
Update the server:
```
$ sudo dnf update
```

1. Install Firewalld

Firewalld is available as a default system package on most Red Hat-based server distributions such as CentOS, Rocky Linux, and AlmaLinux. When unavailable, follow the steps below to install and enable Firewalld to set up firewall filtering rules on the server.

1. Install Firewalld:

$ sudo dnf install firewalld

2. Enable the Firewalld application to start at boot time:

$ sudo systemctl enable firewalld

3. Start Firewalld:

$ sudo systemctl start firewalld

4. Verify that the Firewalld service is active and running:

$ sudo firewall-cmd --state

5. Save all runtime changes to the permanent Firewalld configuration:

$ sudo firewall-cmd --runtime-to-permanent

2. Set Up Firewalld Zones

Firewalld zones represent the level of trust for network connections associated with a specific network interface. Each zone includes a different security level and policies that govern the traffic flow on the associated interface. By default, Firewalld runs with the following zones:

public: Applies to public-facing interfaces connected to external networks such as the Internet.
home: Applies to network interfaces associated with home networks or small office environments.
work: Works with corporate or organization network interfaces with better security over the home zone.
internal: Works with private interfaces connected to networks such as a VPC.
external: Applies to external untrusted networks. It's similar to the public zone but applies a higher security level by blocking incoming connection requests while accepting outgoing network connections.
dmz: Works with network interfaces connected to demilitarized zones partly connected to the Internet and filtered by a firewall.
block: Blocks all incoming and outgoing network traffic. It's useful when troubleshooting server network interfaces or networking errors.

1. View all available Firewalld zones:

$ sudo firewall-cmd --get-zones

Output:

block dmz drop external home internal nm-shared public trusted work

2. View the default Firewalld zone:

$ sudo firewall-cmd --get-default-zone

Output:

public

3. View the active Firewalld zone and the associated server network interfaces:

$ sudo firewall-cmd --get-active-zones

Output:

public
  interfaces: enp1s0

4. View the Firewalld zone assigned to a specific network interface. For example, view the enp8s0 network interface zone:

$ sudo firewall-cmd --get-zone-of-interface=enp8s0

5. Change an interface from one zone to another. For example, change the enp1s0 interface to the external zone:

$ sudo firewall-cmd --zone=home --change-interface=eth0

Output:

success

Switching interfaces between zones sets different firewall rules and policies on the network interface for troubleshooting or testing purposes on the server.

6. View the active zones again to verify that two Firewalld zones are active due to the new interface assignment:

$ sudo firewall-cmd --get-active-zones

Output:

home
  interfaces: enp8s0
public
  interfaces: enp1s0

7. View all firewall rules associated with the active zone. By default, public:

$ sudo firewall-cmd --list-all

Output:

public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0 enp8s0
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

8. Change the active zone depending on your server networking environment. For example, change the Firewalld zone to external:

$ sudo firewall-cmd --change-zone=public

3. Set Up Firewall Policies using Firewalld

Firewalld accepts network ports, system services, and service names across multiple zones depending on your server interfaces. Every Firewalld zone, for example public, applies the set policies and rules to the associated interfaces. Follow the steps below to set up firewall policies using Firewalld on the server.

To delete Firewalld rules, replace the --add option with --remove to delete the rule. For example, delete the FTP port 21 policy in the public zone:

$ sudo firewall-cmd --zone=public --permanent --remove-port=22/tcp

Allow Network Ports

1. Add a new Firewalld rule that allows the SSH TCP port 22 through the public zone:

$ sudo firewall-cmd --zone=public --permanent --add-port=22/tcp

The above rule allows SSH access to the server using port 22 through the public zone. The rule is also permanently saved in the firewall table using the --permanent option.

2. Allow another service port, 80, through the firewall:

$ sudo firewall-cmd --zone=public --permanent --add-port=80/tcp

3. View the active Firewalld zone ports to verify the new rules:

$ sudo firewall-cmd --zone=public --list-ports

Output:

22/tcp 80/tcp

4. Allow a range of ports through the public Firewalld zone. For example, port 4990 to 4999 depending on your internal hosts:

$ sudo firewall-cmd --zone=public --permanent --add-port=4990-4999/udp

Allow System Services

1. View the list of available Firewalld services on your server:

$ sudo firewall-cmd --get-services

Output:

RH-Satellite-6 RH-Satellite-6-capsule afp amanda-client amanda-k5-client amqp amqps apcupsd audit ausweisapp2 bacula bacula-client bb bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc bittorrent-lsd ceph ceph-mon cfengine checkmk-agent cockpit collectd condor-collector cratedb ctdb dhcp dhcpv6 dhcpv6-client distcc dns dns-over-tls docker-registry docker-swarm dropbox-lansync elasticsearch etcd-client etcd-server finger foreman foreman-proxy freeipa-4 freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp galera ganglia-client ganglia-master git gpsd grafana gre high-availability http http3 https ident imap imaps ipfs ipp ipp-client ipsec irc ircs iscsi-target isns jenkins kadmin kdeconnect kerberos kibana klogin kpasswd kprop kshell kube-api kube-apiserver kube-control-plane kube-control-plane-secure kube-controller-manager kube-controller-manager-secure kube-nodeport-services kube-scheduler kube-scheduler-secure kube-worker kubelet kubelet-readonly kubelet-worker ldap ldaps libvirt libvirt-tls lightning-network llmnr llmnr-tcp llmnr-udp managesieve matrix mdns memcache minidlna mongodb mosh mountd mqtt mqtt-tls ms-wbt mssql murmur mysql nbd netbios-ns netdata-dashboard nfs nfs3 nmea-0183 nrpe ntp nut openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole plex pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy prometheus prometheus-node-exporter proxy-dhcp ps3netsrv ptp pulseaudio puppetmaster quassel radius rdp redis redis-sentinel rpc-bind rquotad rsh rsyncd rtsp salt-master samba samba-client samba-dc sane sip sips slp smtp smtp-submission smtps snmp snmptls snmptls-trap snmptrap spideroak-lansync spotify-sync squid ssdp ssh steam-streaming svdrp svn syncthing syncthing-gui synergy syslog syslog-tls telnet tentacle tftp tile38 tinc tor-socks transmission-client upnp-client vdsm vnc-server wbem-http wbem-https wireguard ws-discovery ws-discovery-client ws-discovery-tcp ws-discovery-udp wsman wsmans xdmcp xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent zabbix-server zerotier

2. Allow a service such as HTTPS for access through the public zone table:

$ sudo firewall-cmd --zone=public --add-service=https --permanent

3. Allow another service, such as FTP, through the public zone:

$ sudo firewall-cmd --zone=public --add-service=ftp --permanent

4. Reload Firewalld to apply the new service changes:

$ sudo firewall-cmd --reload

Output:

cockpit dhcpv6-client ftp http https ssh

Enable Forwarding and IP Address Rules

1. Allow incoming traffic to the SSH port 22 from your public server IP address 192.0.2.100:

$ sudo firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="192.0.2.100" port protocol="tcp" port="80" accept'

2. Forward traffic from one port to another. For example, forward traffic from the internal port 8080 to the destination port 8900 through all IP addresses:

$ sudo firewall-cmd --zone=public --add-forward-port=port=80:proto=tcp:toport=8080:toaddr=0.0.0.0 --permanent

The above rule forwards all incoming requests on port 8080 to 8900 to securely expose a specific resource on different ports.

3. Reload Firewalld to save the zone changes:

$ sudo firewall-cmd --reload

4. Create Custom Firewalld Zones

1. Create a new Firewalld zone with a custom name, such as example-zone:

$ sudo firewall-cmd --permanent --new-zone=example-zone

2. Reload Firewalld to save changes:

$ sudo firewall-cmd --reload

3. View the Firewalld zones and verify that the new custom zone is available:

$ sudo firewall-cmd --permanent --get-zones

Output:

block dmz drop example-zone external home internal nm-shared public trusted work

5. Firewalld Panic Mode

Firewalld panic mode blocks all incoming and outgoing network connections on the server to protect the system in case of security threats. When enabled, the server blocks all connections, including your active SSH session. Follow the steps below to enable panic mode and troubleshoot the system in case of external threats or network failures.

1. View the Firewalld panic mode status:

$ sudo firewall-cmd --query-panic

Output:

no

2. Enable panic mode to block all network connections on the server:

$ sudo firewall-cmd --panic-on

3. View the Firewalld panic mode status and verify that it's active:

$ sudo firewall-cmd --query-panic

Output:

yes

In case Firewalld blocks your SSH session, use your cloud provider's web console to start a new, out-of-band terminal session to the server.

Next Steps

Audit your zone assignments with firewall-cmd --get-active-zones before opening any new ports in production.
Replace broad rich rules with tighter source-address scoping once you've confirmed which hosts actually need access.
Script firewall-cmd --panic-off into your incident runbook so panic mode doesn't leave the server unreachable longer than intended.
Explore /etc/firewalld directly if you need to version-control your zone and service definitions.

For the full guide with additional tips, visit the original article on Vultr Docs.

Building Disaggregated Inference with NVIDIA Dynamo

Sanskriti Harmukh — Thu, 22 Oct 2026 23:18:08 +0000

NVIDIA Dynamo is an open-source inference framework for deploying large-scale generative AI models across multi-node, multi-GPU environments. Disaggregated serving architecture separates prompt processing (prefill) and token generation (decode) phases across different GPU workers, enabling independent scaling and optimization of each phase based on their distinct computational characteristics. This guide covers building disaggregated inference deployments with NVIDIA Dynamo using AIConfigurator for optimal configuration discovery, deploying on Kubernetes with RDMA, and validating performance. Disaggregated architecture can achieve up to 1.7x throughput improvement over aggregated serving for large models (32B+) with long input contexts; for smaller models, AIConfigurator automatically determines when aggregated serving is more efficient. By the end, you'll have used AIConfigurator to find an optimal serving configuration, deployed it on a GPU Kubernetes cluster with RDMA-accelerated KV cache transfer, and validated throughput with AIPerf.

Prerequisites

Before you begin, ensure you have one of the following environments:

For Kubernetes Deployment:

A Kubernetes cluster with NVIDIA GPU nodes and the RDMA device plugin installed.
kubectl configured to access your cluster.
An RDMA-capable network (InfiniBand or RoCE).
The NVIDIA Dynamo platform installed on your cluster, including the Dynamo Operator and GPU resource management configured.

For Standalone Instance Deployment:

A GPU-enabled server with NVIDIA GPUs and the NVIDIA Container Toolkit configured.
2+ GPUs for disaggregated serving.
Docker Engine and Docker Compose.
A non-root user with sudo privileges.

Common Requirements:

Python 3.10+ for running AIConfigurator.
A Hugging Face account with an access token for gated models.

Understanding Disaggregated Serving Architecture

Disaggregated serving splits LLM inference into two specialized phases, each optimized for different computational characteristics.

Prefill Phase vs Decode Phase

Phase	Characteristics	Optimization Focus
Prefill	Processes entire input prompt in parallel	Memory-bound, benefits from high-bandwidth GPUs, uses low batch sizes (1-4)
Decode	Generates tokens sequentially, one at a time	Compute-bound, benefits from high concurrency, uses high batch sizes (256-1024)

Architecture Components

Prefill Workers:

Process incoming prompts and generate KV cache.
Run with low batch sizes for optimal TTFT.
Can use different tensor parallelism (TP) than decode workers.
Transfer KV cache to decode workers via NIXL.

Decode Workers:

Receive KV cache from prefill workers.
Generate output tokens sequentially.
Handle high concurrency with large batch sizes.
Optimized for sustained token throughput.

Request Routing:

Frontend routes requests to available prefill workers.
PrefillRouter selects prefill workers based on KV cache affinity.
After prefill, decode workers receive KV cache and continue generation.
NIXL enables GPU-to-GPU KV cache transfer with RDMA.

When to Use Disaggregated Serving

Use Disaggregated Serving When:

Model size is large (32B+ parameters) — smaller models have lower KV cache transfer overhead that doesn't justify phase separation.
Input sequence length (ISL) significantly exceeds output sequence length (OSL) — typically ISL > 4000 tokens.
Workload requires independent scaling of prefill and decode capacity.
RDMA-capable network infrastructure is available (critical requirement).
Maximizing per-GPU throughput is the primary objective.

Use Aggregated Serving When:

Model size is small (<10B parameters) — KV cache transfer overhead outweighs benefits.
Balanced ISL/OSL ratios (between 2:1 and 10:1).
Simpler deployment with lower operational complexity is preferred.
RDMA infrastructure is not available.
Strict latency SLAs require minimal inter-worker communication overhead.

Performance Benefits (for Large Models):

Up to 1.7x higher throughput compared to aggregated serving for large models with long-context workloads.
Better GPU utilization through phase-specific optimization.
Independent scaling of prefill and decode resources based on workload patterns.

Note: RDMA is required. Disaggregated serving requires RDMA for KV cache transfer. Without RDMA, performance degrades by 40x (TTFT increases from ~400ms to 10+ seconds) as the system falls back to TCP transfer, creating a severe bottleneck.

1. Install and Configure AIConfigurator

AIConfigurator is a performance optimization tool that automatically determines the optimal configuration for deploying LLMs with Dynamo by evaluating both aggregated and disaggregated architectures.

1. Create a Python 3 virtual environment:

$ python3 -m venv aiconfigurator_env

2. Activate the virtual environment:

$ source aiconfigurator_env/bin/activate

3. Install AIConfigurator using pip:

$ pip3 install aiconfigurator

4. Verify the installation:

$ aiconfigurator version

The output displays the installed version.

5. View available parameters and help documentation:

$ aiconfigurator cli default --help

Understanding AIConfigurator Parameters

AIConfigurator requires the following key parameters:

Parameter	Description	Example
`--hf_id`	HuggingFace model ID or local path	`nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1`
`--system`	GPU system type	`h200_sxm`, `h100_sxm`, `a100_sxm`
`--total_gpus`	Number of GPUs available for deployment	`4`
`--isl` / `--osl`	Input/Output sequence lengths in tokens	`4000` / `500`
`--ttft` / `--tpot`	SLA targets - Time To First Token (ms) and Time Per Output Token (ms)	`600` / `16.67`
`--backend`	Inference backend	`vllm`, `trtllm`, `sglang`
`--backend_version`	Backend version	`0.12.0` (for vLLM)
`--save_dir`	Directory to save generated configs	`./results_vllm`

How AIConfigurator Works:

Evaluates all possible configurations (tensor/pipeline parallelism combinations).
Simulates performance for both aggregated and disaggregated architectures.
Filters configurations that meet SLA targets (TTFT, TPOT).
Ranks results by per-GPU throughput (tokens/s/gpu).
Automatically selects the best architecture (agg or disagg) based on actual performance.
Generates deployment manifests for top-ranked configurations.

AIConfigurator considers factors like model size, KV cache transfer overhead, and GPU utilization to determine whether disaggregated serving will outperform aggregated for your specific workload.

2. Find the Optimal Disaggregated Configuration

Use AIConfigurator to discover the best configuration for your model and hardware.

1. Run AIConfigurator with your model and system specifications:

$ aiconfigurator cli default \
  --hf_id nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \
  --system h200_sxm \
  --total_gpus 4 \
  --isl 4000 \
  --osl 500 \
  --ttft 600 \
  --tpot 16.67 \
  --backend vllm \
  --backend_version 0.12.0 \
  --save_dir ./results_vllm

The command evaluates all possible configurations for 4x H200 GPUs with the specified model and SLA targets, generating deployment manifests in the ./results_vllm directory.

2. Review the AIConfigurator output summary:

********************************************************************************
*                     Dynamo aiconfigurator Final Results                      *
********************************************************************************
  ----------------------------------------------------------------------------
  Input Configuration & SLA Target:
    Model: nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 (is_moe: False)
    Total GPUs: 4
    Best Experiment Chosen: agg at 3785.20 tokens/s/gpu (disagg 0.81x better)
  ----------------------------------------------------------------------------
  Overall Best Configuration:
    - Best Throughput: 15,140.82 tokens/s
    - Per-GPU Throughput: 3785.20 tokens/s/gpu
    - Per-User Throughput: 61.41 tokens/s/user
    - TTFT: 395.88ms
    - TPOT: 16.28ms
    - Request Latency: 8521.26ms
  ----------------------------------------------------------------------------

The summary shows that for this 4B model, aggregated serving performs better than disaggregated (3785.20 vs 3067.51 tokens/s/gpu). This is expected for smaller models where the overhead of KV cache transfer outweighs the benefits of phase separation. Disaggregated serving becomes advantageous with larger models (32B+) and longer context lengths.

3. Review the detailed configuration comparison:

agg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | request_latency | concurrency  | total_gpus (used) | replicas | parallel | bs |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+
|  1   |   3785.20    |     61.41     | 395.88 |     8521.26     | 256 (=64x4)  |     4 (4=4x1)     |    4     |  tp1pp1  | 64 |
|  2   |   3036.90    |     65.06     | 331.10 |     8000.51     | 192 (=96x2)  |     4 (4=2x2)     |    2     |  tp2pp1  | 96 |
|  3   |   2305.76    |     65.36     | 249.55 |     7883.93     | 144 (=144x1) |     4 (4=1x4)     |    1     |  tp4pp1  | 144|
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+----------+----+

disagg Top Configurations: (Sorted by tokens/s/gpu)
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+
| Rank | tokens/s/gpu | tokens/s/user |  TTFT  | request_latency | concurrency  | total_gpus (used) | replicas | (p)parallel | (d)parallel |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+
|  1   |   3067.51    |     63.99     | 111.33 |     7909.20     | 208 (=104x2) |     4 (4=2x2)     |    2     |   tp1pp1    |  tp1pp1     |
|  2   |   3011.39    |     62.82     | 111.33 |     8054.41     | 208 (=208x1) |     4 (4=1x4)     |    1     |   tp1pp1    |  tp2pp1     |
+------+--------------+---------------+--------+-----------------+--------------+-------------------+----------+-------------+-------------+

Understanding the Output:

tokens/s/gpu: Overall throughput efficiency — higher is better (agg: 3785.20 vs disagg: 3067.51 = agg is 23% faster).
tokens/s/user: Per-request generation speed (inverse of TPOT).
TTFT: Predicted time to first token (395.88ms for agg, 111.33ms for disagg prefill).
concurrency: Total concurrent requests (e.g., 256 (=64x4) = batch size 64 x 4 replicas).
parallel: Tensor/pipeline parallelism (tp=tensor parallel, pp=pipeline parallel).
Result: For this 4B model, aggregated serving is more efficient due to lower overhead.

Note: model size matters. Disaggregated serving becomes advantageous with larger models (32B+) where the benefits of phase-specific optimization outweigh KV cache transfer overhead. For smaller models like this 4B example, aggregated serving is more efficient.

4. Examine the generated directory structure:

$ tree results_vllm/

results_vllm/
└── nvidia/
    └── Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_472853/
        ├── agg/
        │   ├── best_config_topn.csv
        │   ├── config.yaml
        │   ├── pareto.csv
        │   ├── top1/
        │   │   ├── generator_config.yaml
        │   │   ├── k8s_deploy.yaml
        │   │   └── run_0.sh
        │   ├── top2/
        │   │   └── ...
        │   └── top3/
        │       └── ...
        ├── disagg/
        │   ├── best_config_topn.csv
        │   ├── config.yaml
        │   ├── pareto.csv
        │   ├── top1/
        │   │   ├── generator_config.yaml
        │   │   ├── k8s_deploy.yaml
        │   │   └── run_0.sh
        │   └── top2/
        │       └── ...
        └── pareto_frontier.png

The results are organized by model name and experiment parameters. While aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes and larger model scenarios.

3. Deploy on Kubernetes

Note: while AIConfigurator determined that aggregated serving is optimal for this 4B model, the following sections demonstrate disaggregated deployment for learning purposes. Use disaggregated architecture when deploying larger models (32B+) or when AIConfigurator recommends it for your specific workload.

Kubernetes deployment provides the infrastructure for disaggregated serving with RDMA-accelerated KV cache transfer.

Install the NVIDIA Dynamo Platform

Before deploying disaggregated workloads, install the NVIDIA Dynamo platform (the Dynamo Operator) on your Kubernetes cluster, configure GPU resource management, set up the DynamoGraphDeployment CRD, and verify the cluster is ready for Dynamo workloads.

Prepare the Kubernetes Environment

1. Set your Kubernetes namespace as an environment variable. Replace YOUR_NAMESPACE with the namespace where you want to deploy Dynamo workloads:

$ export LLM_NAMESPACE=YOUR_NAMESPACE

2. Create the namespace:

$ kubectl create namespace $LLM_NAMESPACE

3. Export your Hugging Face token. Replace YOUR_HF_TOKEN with your Hugging Face access token:

$ export HF_TOKEN=YOUR_HF_TOKEN

4. Create a Hugging Face token secret for accessing gated models:

$ kubectl create secret generic hf-token-secret \
  --namespace $LLM_NAMESPACE \
  --from-literal=HF_TOKEN="$HF_TOKEN"

5. Create a persistent volume claim for caching model weights:

$ cat << EOF > model-cache-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
  namespace: $LLM_NAMESPACE
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
EOF

Note: this PVC uses ReadWriteMany (RWX) access mode, which allows multiple pods to mount the volume simultaneously for shared model caching. Ensure your cluster has a StorageClass that supports RWX access mode (such as NFS, CephFS, or cloud-provider RWX storage). Verify available StorageClasses:
$ kubectl get storageclass

6. Apply the PVC configuration:

$ kubectl apply -f model-cache-pvc.yaml

7. Verify the RDMA device plugin is running:

$ kubectl get pods -n kube-system | grep rdma

The output should show RDMA device plugin pods running on GPU nodes. If not installed, follow the Mellanox RDMA device plugin installation guide.

8. Verify RDMA resources are available on your nodes. Replace NODE-NAME with your GPU node name:

$ kubectl get node NODE-NAME -o json | jq '.status.allocatable'

The output should show rdma/hca_shared_devices resources:

{
......
  "rdma/hca_shared_devices": "1k",
......
}

The rdma/hca_shared_devices: "1k" entry confirms RDMA resources are available. If this resource is missing, verify the RDMA device plugin DaemonSet configuration and node labels.

Deploy Using Generated Manifests

1. Review the generated disaggregated deployment manifest:

$ cat results_vllm/nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1_isl4000_osl500_ttft600_tpot16_*/disagg/top1/k8s_deploy.yaml

The file contains a DynamoGraphDeployment custom resource with frontend, prefill workers, and decode workers configured according to AIConfigurator's recommendations.

2. Create the complete deployment manifest with RDMA configuration:

$ cat << EOF > dynamo-disagg-deployment.yaml
apiVersion: nvidia.com/v1alpha1
kind: DynamoGraphDeployment
metadata:
  name: dynamo-disagg
  namespace: $LLM_NAMESPACE
spec:
  backendFramework: vllm
  pvcs:
    - name: model-cache
      create: false
  services:
    Frontend:
      componentType: frontend
      replicas: 1
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          imagePullPolicy: IfNotPresent

    VLLMPrefillWorker:
      envFromSecret: hf-token-secret
      componentType: worker
      subComponentType: prefill
      replicas: 2
      resources:
        limits:
          gpu: "1"
      sharedMemory:
        size: 16Gi
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
        - name: POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: UCX_TLS
          value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
        - name: UCX_RNDV_SCHEME
          value: "get_zcopy"
        - name: UCX_RNDV_THRESH
          value: "0"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          workingDir: /workspace
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          resources:
            limits:
              rdma/hca_shared_devices: "1"
            requests:
              rdma/hca_shared_devices: "1"
          command: ["python3", "-m", "dynamo.vllm"]
          args:
            - --model
            - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1"
            - "--tensor-parallel-size"
            - "1"
            - "--kv-cache-dtype"
            - "fp8"
            - "--max-num-seqs"
            - "1"
            - --trust-remote-code
            - --is-prefill-worker

    VLLMDecodeWorker:
      envFromSecret: hf-token-secret
      componentType: worker
      subComponentType: decode
      replicas: 2
      resources:
        limits:
          gpu: "1"
      sharedMemory:
        size: 16Gi
      volumeMounts:
        - name: model-cache
          mountPoint: /opt/models
      envs:
        - name: HF_HOME
          value: /opt/models
        - name: POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: UCX_TLS
          value: "rc_x,rc,dc_x,dc,cuda_copy,cuda_ipc"
        - name: UCX_RNDV_SCHEME
          value: "get_zcopy"
        - name: UCX_RNDV_THRESH
          value: "0"
      extraPodSpec:
        mainContainer:
          image: nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13
          workingDir: /workspace
          imagePullPolicy: IfNotPresent
          securityContext:
            capabilities:
              add: ["IPC_LOCK"]
          resources:
            limits:
              rdma/hca_shared_devices: "1"
            requests:
              rdma/hca_shared_devices: "1"
          command: ["python3", "-m", "dynamo.vllm"]
          args:
            - --model
            - "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1"
            - "--tensor-parallel-size"
            - "1"
            - "--kv-cache-dtype"
            - "fp8"
            - "--max-num-seqs"
            - "1024"
            - --trust-remote-code
            - --is-decode-worker
EOF

Critical RDMA Configuration:

rdma/hca_shared_devices resources: Request RDMA resources matching tensor parallelism size (1 for both prefill and decode with TP1).
IPC_LOCK capability: Required for RDMA memory registration.
POD_UID environment variable: Retrieves pod UID from metadata for worker identification.
UCX_TLS environment variable: Enables RDMA transports (rc_x, dc_x for InfiniBand).
UCX_RNDV_SCHEME=get_zcopy: Enables zero-copy RDMA transfers for maximum performance.
UCX_RNDV_THRESH=0: Forces RDMA usage for all message sizes.
--trust-remote-code flag: Allows execution of custom model code from Hugging Face.
CUDA 13 runtime: Uses nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 for GPU compatibility.

3. Deploy the configuration to Kubernetes:

$ kubectl apply -f dynamo-disagg-deployment.yaml

4. Monitor the deployment progress:

$ kubectl get pods -n $LLM_NAMESPACE -w

Wait for all pods to reach Running status. This may take several minutes as models are downloaded and loaded.

Verify the Kubernetes Deployment

1. Check pod status and resource allocation:

$ kubectl get pods -n $LLM_NAMESPACE

The output shows all pods running successfully:

NAME                                      READY   STATUS    RESTARTS   AGE
dynamo-disagg-0-frontend-77bn5            1/1     Running   0          2m42s
dynamo-disagg-0-vllmdecodeworker-gzgbw    1/1     Running   0          2m42s
dynamo-disagg-0-vllmdecodeworker-kn8xz    1/1     Running   0          2m42s
dynamo-disagg-0-vllmprefillworker-2b7hm   1/1     Running   0          2m42s
dynamo-disagg-0-vllmprefillworker-7p4mq   1/1     Running   0          2m42s

2. Verify RDMA is active by checking worker logs. Replace PREFILL-WORKER-POD with your prefill worker pod name from the previous step:

$ kubectl logs -n $LLM_NAMESPACE PREFILL-WORKER-POD | grep -i "UCX\|NIXL"

The output shows NIXL initialization logs:

2026-03-03T22:52:23.152441Z  INFO nixl_connector: NIXL is available
2026-03-03T22:52:23.154049Z  INFO factory.create_connector: Creating v1 connector with name: NixlConnector and engine_id: e0dc9d5a-80b5-4b06-aeb9-05958f83d6ea
(Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO    _api.py:363 Backend UCX was instantiated
(Worker pid=1241) 2026-03-03 22:52:25 NIXL INFO    _api.py:253 Initialized NIXL agent: 93612abb-847f-435f-bd56-73bfc8d5f3d6
2026-03-03T22:52:25.487423Z  INFO nixl_connector.register_kv_caches: Registering KV_Caches. use_mla: False, kv_buffer_device: cuda, use_host_buffer: False

The key indicators confirm RDMA is working:

"NIXL is available": NIXL connector initialized successfully.
"Backend UCX was instantiated": UCX backend (RDMA transport) is active.
"Initialized NIXL agent": NIXL agent ready for KV cache transfer.
"Registering KV_Caches": KV cache buffers registered for GPU-to-GPU transfer.

3. Test inference through the frontend service:

$ kubectl port-forward -n $LLM_NAMESPACE service/dynamo-disagg-frontend 8000:8000

In a separate terminal, send a test request:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [{"role": "user", "content": "Explain the importance of tensor parallelism in LLM inference."}],
    "max_tokens": 100
  }'

The output displays the model's response in JSON format.

4. Monitor worker metrics to verify disaggregated flow:

$ curl -s localhost:8000/metrics | grep dynamo_frontend_requests_total

The output displays the metrics showing successful request processing:

# HELP dynamo_frontend_requests_total Total number of LLM requests processed
# TYPE dynamo_frontend_requests_total counter
dynamo_frontend_requests_total{endpoint="chat_completions",model="nvidia/llama-3.1-nemotron-nano-4b-v1.1",request_type="unary",status="success"} 1

4. Deploy on Standalone Instances

For standalone instance deployments (non-Kubernetes), refer to the backend-specific deployment guides that cover disaggregated serving:

Deploy vLLM Disaggregated Serving: 2 prefill workers + 2 decode workers with NIXL-based KV cache transfer.
Deploy TensorRT-LLM Disaggregated Serving: TRT-LLM disaggregated configuration with KV cache optimization.
Deploy SGLang Disaggregated Serving: SGLang disaggregated setup with bootstrap_info transfer.

These guides provide complete deployment scripts with environment variable configuration, container runtime setup with GPU allocation, NIXL configuration for inter-worker KV cache transfer, and verification/troubleshooting steps.

5. Validate Performance with AIPerf

After deployment, validate AIConfigurator's predictions against actual performance using AIPerf.

1. Install the AIPerf benchmarking tool:

$ pip3 install aiperf

2. Derive AIPerf parameters from the AIConfigurator output. Values used for testing:

--concurrency 30 (AIConfigurator recommends 208 for max throughput)
--num-requests 800 (Recommended: 8320 = concurrency x 40 for statistical stability)
--isl 4000 --osl 500 (Match your AIConfigurator inputs)
--extra-inputs "ignore_eos:true" (Ensures exact OSL tokens generated)

3. Run the AIPerf benchmark against the deployment:

$ aiperf profile \
  -m nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1 \
  --endpoint-type chat \
  -u http://localhost:8000 \
  --isl 4000 --isl-stddev 0 \
  --osl 500 --osl-stddev 0 \
  --num-requests 800 \
  --concurrency 30 \
  --streaming \
  --extra-inputs "ignore_eos:true" \
  --num-warmup-requests 40 \
  --ui-type simple

Note: the command above uses --concurrency 30 and --num-requests 800 for early success and faster testing. For best throughput validation, use the recommended values: --concurrency 208 and --num-requests 8320 (concurrency x 40 for statistical stability).

The benchmark runs 800 requests with 30 concurrent connections. The output displays performance metrics:

                                         NVIDIA AIPerf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┓
┃                               Metric ┃      avg ┃      min ┃      max ┃      p99 ┃      p90 ┃      p50 ┃      std ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━┩
│             Time to First Token (ms) │   496.77 │    55.53 │ 8,411.38 │ 6,972.51 │    64.59 │    59.78 │ 1,606.66 │
│            Time to Second Token (ms) │     0.24 │     0.00 │     4.72 │     3.55 │     0.00 │     0.00 │     0.81 │
│      Time to First Output Token (ms) │   496.77 │    55.53 │ 8,411.38 │ 6,972.51 │    64.59 │    59.78 │ 1,606.66 │
│                 Request Latency (ms) │ 2,367.45 │ 1,505.01 │ 9,807.65 │ 8,869.04 │ 1,996.11 │ 1,972.56 │ 1,528.95 │
│             Inter Token Latency (ms) │     3.75 │     2.74 │     3.90 │     3.89 │     3.85 │     3.83 │     0.25 │
│     Output Token Throughput Per User │   268.03 │   256.55 │   365.49 │   357.48 │   279.88 │   260.84 │    22.84 │
│                    (tokens/sec/user) │          │          │          │          │          │          │          │
│      Output Sequence Length (tokens) │   499.53 │   487.00 │   510.00 │   502.00 │   500.10 │   500.00 │     1.30 │
│       Input Sequence Length (tokens) │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │ 4,000.00 │     0.00 │
│ Output Token Throughput (tokens/sec) │ 6,232.31 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
│    Request Throughput (requests/sec) │    12.48 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
│             Request Count (requests) │   800.00 │      N/A │      N/A │      N/A │      N/A │      N/A │      N/A │
└──────────────────────────────────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

Benchmark Duration: 64.12 sec

4. Compare actual vs predicted performance:

Metric	AIConfigurator Prediction (2 GPUs)	AIPerf Actual (2 GPUs)	Status
TTFT (ms)	111.33	59.78 (p50) / 496.77 (avg)	p50 close to prediction
ITL/TPOT (ms)	15.62 (1000/63.99)	3.75	4x better than predicted
Throughput (tokens/s)	6,135.02 (3,067.51 x 2)	6,232.31	101.6% - exceeds prediction
Per-GPU Throughput	3,067.51 tokens/s/gpu	3,116 tokens/s/gpu	Matches prediction

The per-GPU throughput (6,232.31 / 2 GPUs = 3,116 tokens/s/gpu) closely matches AIConfigurator's prediction of 3,067.51 tokens/s/gpu. The total throughput (6,232.31 tokens/s) actually exceeds the prediction for 2 GPUs (6,135.02 tokens/s). This benchmark was run with 2 GPUs (1 prefill + 1 decode worker); scaling to 4 GPUs (2 prefill + 2 decode workers as shown in the deployment manifest) would double the throughput to approximately 12,464 tokens/s. The lower throughput compared to maximum capacity is due to using concurrency 30 instead of the recommended 208. The Inter Token Latency (3.75ms) significantly outperforms predictions, demonstrating efficient token generation.

Note: run benchmarks inside the Kubernetes cluster (using a Job pod) to eliminate network latency. External benchmarking adds latency that affects TTFT measurements but provides realistic end-to-end performance metrics.

Configuration Best Practices

GPU Allocation Strategies

Prefill Workers:

Allocate fewer GPUs with higher memory bandwidth (memory-bound workload).
Use lower tensor parallelism (TP1 or TP2) to minimize communication overhead.
Deploy multiple replicas for higher prefill throughput.

Decode Workers:

Allocate more GPUs for parallel token generation (compute-bound workload).
Use higher tensor parallelism (TP4 or TP8) to distribute computation.
Single replica with high batch size often performs best.

Example Configuration (4 GPUs):

2x Prefill workers with TP1 (2 GPUs total, 1 GPU each).
2x Decode workers with TP1 (2 GPUs total, 1 GPU each).
Achieves balanced prefill and decode capacity with minimal tensor parallelism overhead.

Tensor Parallelism Considerations

Worker Type	Recommended TP	Reasoning
Prefill	TP1 or TP2	Minimizes all-reduce communication overhead, prefill is memory-bound
Decode	TP4 or TP8	Distributes compute load, decode is compute-bound

Batch Size Tuning

Prefill Workers:

Use low batch size (1-4) for optimal TTFT.
Higher batch sizes increase memory pressure and TTFT latency.
Prefill throughput comes from multiple workers, not high batch size.

Decode Workers:

Use high batch size (256-1024) for optimal throughput.
Decode operations are compute-bound and benefit from high concurrency.
Adjust based on available GPU memory and model KV cache requirements.

RDMA Requirements

Network Requirements:

InfiniBand (preferred) or RoCE network with RDMA support.
Sufficient bandwidth: 200 Gbps+ recommended for optimal performance.
Low latency: <10μs for RDMA operations.

Verification:

Check worker logs for "NIXL is available" and "Backend UCX was instantiated" messages.
Monitor RDMA device utilization: rdma resource show.
Measure RDMA bandwidth: ib_write_bw or rdma_bw tools.

Next Steps

Re-run AIConfigurator against a 32B+ model to see disaggregated serving's throughput advantage materialize.
Scale prefill and decode replicas independently based on observed ISL/OSL ratios in production traffic.
Run AIPerf with the full recommended concurrency (208) and request count (8320) for a production-grade validation.
Review the backend-specific deployment guides (vLLM, TensorRT-LLM, SGLang) if you need a standalone, non-Kubernetes setup.

For the full guide with additional tips, visit the original article on Vultr Docs.

Creating MCP Server Using FastMCP in Python

Sanskriti Harmukh — Thu, 22 Oct 2026 23:17:48 +0000

The Model Context Protocol (MCP) is an emerging standard for structuring and managing contextual information provided to Large Language Models (LLMs), enabling them to operate more intelligently and consistently across sessions, tools, and multi-agent systems. Rather than relying solely on raw prompts, MCP introduces a structured format that includes user identity, task goals, memory state, available tools, and environmental data — allowing models to behave more like persistent, context-aware agents. This guide walks through running a sample MCP server built with FastMCP, backed by an SQLite database, and querying it using natural language through an MCP client. By the end, you'll have a working FastMCP server that an MCP client can query against a local SQLite database using plain language.

1. Get the Server Code Running

The code for the MCP server and the sample SQLite database is available in this GitHub repository. To set up the server locally, follow the steps below in your system terminal.

1. Clone the GitHub repository:

$ git clone https://github.com/vultr-marketing/code-samples.git

2. Navigate into the mcp_sqlite directory:

$ cd code-samples/mcp/mcp_sqlite

3. Create a .env file:

$ nano .env

4. Add the following variables to the file, then save and close it:

DB_PATH=sample.db
MCP_PORT=8080
READ_ONLY=true

DB_PATH: Path to your SQLite database file.
MCP_PORT: Port used by the MCP server (not used in stdio mode, but good to define).
READ_ONLY: Set to true to block insert/update/delete queries for safety.

5. Install dependencies:

$ pip3 install -r requirements.txt

2. Understand the Sample MCP Server

The server script launches a Model Context Protocol (MCP) server using FastMCP and connects it to a local SQLite database. It enables structured interaction with the database using natural language, typically via an LLM interface.

Key Features

Environment-Based Configuration: Uses .env for settings like DB_PATH, MCP_PORT, and READ_ONLY.
Async Database Access: Handles SQLite operations asynchronously with aiosqlite.
Sample DB Initialization: Creates sample tables if the database doesn't exist.
Query Validation: Prevents dangerous SQL operations, especially when READ_ONLY is enabled.
Structured Logging: Provides detailed startup and runtime logs for debugging.

Transport Layer: stdio

The server uses the stdio transport, allowing it to communicate over standard input and output. This is especially useful when integrating with desktop agents like Claude Desktop, enabling lightweight inter-process communication without a web server.

Available Tools in the Sample MCP Server

Tools are structured, callable functions that a language model can use to interact with external systems like databases, APIs, or services. Each tool has a defined input schema and a predictable output format, enabling the model to perform specific operations safely and effectively — like querying data, listing tables, or modifying records.

Tools extend the capabilities of LLMs beyond simple text generation, allowing them to reason over structured data, fetch real-time information, or take actions based on user intent.

Tool Name	Description
`execute_query`	Executes a validated SQL `SELECT` query and returns the result set.
`list_tables`	Lists all user-defined tables in the SQLite database.
`describe_table`	Returns the schema (columns, types, constraints) of a specified table.
`count_rows`	Counts and returns the number of rows in a specified table.
`insert_sample_data`	Inserts predefined sample data into tables (only if not read-only).

3. Use an MCP Client to Interact

Download the Claude Desktop application — it automatically detects MCP servers defined in its configuration file and makes them accessible for use within any conversation.

1. Start the Claude Desktop application.

2. Locate the settings option in the application.

3. Select Developer settings.

4. Click Edit Config.

5. Add the following configuration snippet to your Claude Desktop config file:

{
  "mcpServers": {
    "sqlite": {
      "command": "<path to uv executable>",
      "args": [
        "--directory",
        "<path to directory containing server.py>",
        "run",
        "server.py"
      ]
    }
  }
}

6. Once the new configuration is in place, restart the Claude Desktop application.

7. To confirm the server is running, go to the Developer settings in Claude Desktop and check if the status next to the JSON file shows as running.

8. Make sure the MCP server tools are available to the application.

Ask Questions and Perform Operations

You can run interactive queries by chatting naturally. Using the available tools, the LLM translates your requests into SQL queries, executes them on the sample database, and responds in plain language. The LLM also displays the database queries it is executing and the tools it is using to retrieve the data.

Note: the data in the sample database is for demonstration purposes only and may differ from actual data.

Next Steps

Point DB_PATH at a real SQLite database and extend the tool list beyond execute_query, list_tables, describe_table, and count_rows.
Swap the stdio transport for an HTTP-based transport if you need the server reachable from a remote MCP client.
Add authentication and request logging before exposing an MCP server with READ_ONLY=false beyond your local machine.
Try connecting the same server to other MCP-compatible clients, not just Claude Desktop.

For the full guide with additional tips, visit the original article on Vultr Docs.

Building a Docker Image

Sanskriti Harmukh — Thu, 22 Oct 2026 23:17:22 +0000

Docker is an open-source containerization platform that enables developers to package applications and all their dependencies into lightweight, portable containers that run consistently across different environments. Docker images are the blueprint for containers — they package your application code, dependencies, environment variables, and runtime configurations into a portable, reproducible format. This guide explains how to build Docker images efficiently and create an optimized Dockerfile. By the end, you'll have a tested, size-optimized Docker image built with docker buildx and ready to push to a registry.

The Short Answer

If you're already familiar with Docker and just need a quick reference, use the following commands:

# Build a Docker image from a Dockerfile in the current directory
$ docker buildx build -t <image-name>:<tag> .

# Run a container from the image and map a port
$ docker container run -d -p <host-port>:<container-port> <image-name>:<tag>

# Tag the image for a container registry
$ docker image tag <image-name>:<tag> <registry>/<image-name>:<tag>

# Push your image to a registry (e.g., Docker Hub or another container registry)
$ docker image push <registry>/<image-name>:<tag>

Continue reading for detailed explanations and examples of each command.

1. Create a Dockerfile

To build a Docker image, start by creating a Dockerfile. This file defines the environment, dependencies, and commands needed to build your application into a container image. Follow the steps below to create a Dockerfile for a Node.js web application.

1. Create a file named Dockerfile in your project's root directory:

$ vim Dockerfile

2. Add the following content to the file, then save and exit:

# Use Node.js as the base image
FROM node:22-alpine

# Set the working directory
WORKDIR /app

# Copy dependency files
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy application source code
COPY . .

# Expose application port
EXPOSE 3000

# Define the startup command
CMD ["node", "app.js"]

Breakdown of the Dockerfile

FROM node:22-alpine: Defines node:22-alpine as the base image.
WORKDIR /app: Sets /app as the working directory inside the container.
COPY package*.json ./: Copies dependency definition files to the working directory.
RUN npm install: Installs project dependencies using npm.
COPY . .: Copies the entire project directory into the container.
EXPOSE 3000: Declares port 3000 as the container's exposed port.
CMD ["node", "app.js"]: Defines the startup command to run the application.

The above Dockerfile follows several best practices:

Uses a specific version of the base image.
Copies dependency files first to take advantage of Docker's build cache.
Uses a minimal Alpine-based image to reduce image size.
Defines the default command to run the application.

2. Build the Docker Image

With your Dockerfile ready, use the docker buildx build command to create your image. Docker Buildx enables advanced build features such as multi-platform support, build caching, and pushing to registries.

1. Basic build with a tag:

$ docker buildx build -t my-app:latest .

2. Build with multiple tags and build arguments:

$ docker buildx build -t my-app:1.0 -t my-app:latest --build-arg ENV=production .

3. For a multi-platform build and push to a registry, first create and bootstrap a builder (one-time setup):

$ docker buildx create \
  --name container-builder \
  --driver docker-container \
  --bootstrap --use

4. After the container-builder is running, run the multi-platform build:

$ docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t my-app:latest \
  --push .

Command Breakdown

Explanation of key options used with docker buildx build:

docker buildx build: Uses Docker BuildKit's extended build command for advanced features like multi-platform builds and better caching.
-t my-app:latest: Tags the image with a name (my-app) and version (latest).
-t my-app:1.0: Adds another tag (1.0) to the image.
--build-arg ENV=production: Passes build-time variable ENV with the value production into the Dockerfile.
--platform linux/amd64,linux/arm64: Builds images for multiple CPU architectures (for example, x86_64 and ARM64).
--push: Pushes the built image and manifest list to the specified container registry.
--no-cache: Forces the build to skip caching and build each layer from scratch.
--pull: Always pulls the latest version of the base image instead of using the cached one.
-f Dockerfile.prod: Specifies a custom Dockerfile (Dockerfile.prod) instead of the default Dockerfile.
.: Sets the build context to the present directory.

Example Scenario

Follow this example to build and tag a Docker image for a Node.js application.

1. Clone an example Docker Node.js application repository:

$ git clone https://github.com/docker/docker-nodejs-sample

2. Navigate to the cloned project directory:

$ cd docker-nodejs-sample

3. Create a Dockerfile inside the project root directory:

$ vim Dockerfile

4. Add the following content to the file:

FROM node:22-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
USER node
EXPOSE 3000
CMD ["node", "src/index.js"]

5. Build the Docker image:

$ docker buildx build -t my-app:latest .

6. View the newly built Docker image:

$ docker image ls

The above command lists all the Docker images available on your system.

How the Docker Build Works:

Docker processes the Dockerfile in a layered fashion.
Starts with the base image (node:22-alpine in this case).
Creates a temporary container for each instruction (for example, COPY, RUN, CMD).
Executes the instruction inside the container and commits the result as a new image layer.
Removes the temporary container.
Repeats the process for each instruction in the Dockerfile.
Tags the final image with the names and versions you specified.

3. Test the Docker Image

After building your Docker image, test it by running a container and verifying that your application behaves as intended.

1. Run a container from your image:

$ docker container run -d -p 3000:3000 --name check-list-app my-app:latest

In the above command:

docker container run: Starts a container from an image.
-d: Runs the container in detached mode (in the background).
-p 3000:3000: Maps port 3000 of the host to port 3000 of the container.
--name check-list-app: Assigns a custom name to the container.
my-app:latest: Specifies the image and tag to use.

2. Verify the container is running:

$ docker container ls

This command lists all running containers. After confirming your container is listed and running, you can access your application to ensure it behaves as expected.

3. Check the application logs for debugging purposes:

$ docker container logs check-list-app

This command displays the container's logs, allowing you to verify application startup messages or troubleshoot any issues.

4. Stop and remove the container:

$ docker container stop check-list-app
$ docker container rm check-list-app

The above commands stop the running container and then remove it entirely from your system.

4. Optimize Docker Image Size

Optimizing Docker image size leads to faster deployments, lower storage costs, and improved security. Follow these best practices to create smaller, more efficient Docker images.

Use Multi-Stage Builds

Multi-stage builds let you create lean final images by separating build and runtime environments. Each FROM starts a new build stage, allowing you to copy only necessary artifacts into the final image.

# Build stage
FROM golang:1.17 AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

# Final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /app/app .
CMD ["./app"]

The final image includes only the compiled binary and runtime dependencies, excluding source code and build tools.

Choose Smaller Base Images

Using smaller base images helps reduce Docker image size, improve security, and speed up build and deployment times. Choose a base image that includes only the essential components needed to run your application.

# Builder stage: install dependencies
FROM python:3.9-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt --target /app/packages

# Final image: minimal runtime using distroless
FROM gcr.io/distroless/python3
COPY --from=builder /app/packages /packages
ENV PYTHONPATH=/packages
COPY app.py /
CMD ["app.py"]

Best practices:

Use alpine or slim based images as a minimal base for building dependencies.
Use distroless for a secure, production-ready runtime image.
Consider scratch images for statically compiled binaries in languages like Go or Rust.
Apply multi-stage builds to separate build and runtime environments.
Exclude package managers and shells from the final image.
Set environment variables like PYTHONPATH to locate dependencies at runtime.

Reduce Layer Count and Image Size

Each instruction in a Dockerfile creates a new layer. Fewer, optimized layers reduce image size and speed up builds.

Example of an inefficient Dockerfile (creates many layers):

# Bad practice - many layers
FROM ubuntu:20.04
RUN apt-get update
RUN apt-get install -y python3
RUN apt-get install -y python3-pip
RUN pip install flask

In the above Dockerfile:

Each RUN instruction creates a new layer.
No cleanup of APT cache or temporary files.
Slower builds and larger images.

Example of an optimized Dockerfile (fewer layers and smaller image):

# Better practice - fewer layers
FROM ubuntu:20.04
RUN apt-get update && \
 apt-get install -y python3 python3-pip && \
 pip install flask && \
 apt-get clean && \
 rm -rf /var/lib/apt/lists/*

In the above Dockerfile:

Related commands are combined using && to reduce layers.
Package caches and temporary files are cleaned up during installation.

Use a `.dockerignore` File

Exclude unnecessary files and directories from the Docker build context to reduce image size and speed up build times.

node_modules
npm-debug.log
.git
.gitignore
Dockerfile
README.md
.env
*.pyc
__pycache__

In the above .dockerignore file:

Prevents unneeded files from being copied into the image.
Keeps the image clean and lightweight.
Speeds up build time by minimizing the context sent to the Docker daemon.

Leverage Docker Build Cache

Docker caches image layers to avoid rebuilding unchanged steps. Structure your Dockerfile to take full advantage of this.

Best practices:

Place less frequently changing instructions at the top of your Dockerfile.
Copy dependency files first, install packages, then copy the application source last.

Example of an efficient caching structure:

FROM node:16-alpine

WORKDIR /app

# Copy dependency files first
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy the source code last
COPY . .

CMD ["node", "app.js"]

In the above Dockerfile:

Dependency layers are reused unless package*.json changes.
Source code changes won't invalidate the cached dependency layers.
Results in faster builds and smaller diffs in CI/CD pipelines.

Next Steps

Wire docker buildx build --push into a CI/CD pipeline so images rebuild and publish automatically on every commit.
Scan your images for vulnerabilities with a tool like Trivy or Docker Scout before pushing to production.
Add a multi-stage build to any existing Dockerfile that still ships build tools in the final image.
Set up docker buildx multi-platform builds if you need to support both x86_64 and ARM64 deployments.

For the full guide with additional tips, visit the original article on Vultr Docs.

Building a Llama.cpp Container Image

Sanskriti Harmukh — Thu, 22 Oct 2026 23:16:25 +0000

Llama.cpp is a high-performance inference platform designed for Large Language Models (LLMs) like Llama, Falcon, and Mistral. It provides a streamlined development environment compatible with both CPU and GPU systems. This guide explains how to build Llama.cpp into a Docker container image for CPU and GPU systems and push the finished image to a container registry. By the end, you'll have CPU and GPU container images of Llama.cpp built, tagged, and pushed to a container registry.

Prerequisites

A server with Docker installed (a GPU-enabled server with NVIDIA drivers and the NVIDIA Container Toolkit for the GPU build).
Access to the server using SSH.
Start the Docker service.
```
$ sudo systemctl start docker
```
Add the non-root user to the Docker group. For example, linuxuser.
```
$ sudo usermod -aG docker linuxuser
```
Switch to the user:
```
$ su - linuxuser
```

1. Set Up the Project Directory

1. Create a new directory to store your Llama.cpp project files:

$ mkdir llama-project

2. Switch to the directory:

$ cd llama-project

3. Clone the Llama.cpp project repository using Git:

$ git clone https://github.com/ggerganov/llama.cpp/

4. List files and verify that a new llama.cpp directory is available:

$ ls

Output:

llama.cpp

5. Switch to the llama.cpp project directory:

$ cd llama.cpp

6. List all hidden files and verify that a new .devops directory is available:

$ ls -a

Output:

..                 cmake           convert.py                     flake.lock     ggml-common.h        ggml-metal.h      ggml-quantsh     .git                         llama.cpp        
CMakeLists.txt  .devops

The .devops directory contains the following Dockerfile resources:

main.Dockerfile: Contains the build context for CPU systems.
main-cuda.Dockerfile: Contains the build context for GPU systems.

Use these resources in the next sections to build a CPU or GPU system container image.

2. Build a Llama.cpp Container Image for CPU Systems

Follow the steps below to build a new Llama.cpp container image using main.Dockerfile, which contains the required build context and installs all necessary dependency packages for CPU-based systems.

1. Copy the main.Dockerfile from the .devops directory to the main llama.cpp project directory:

$ cp .devops/main.Dockerfile .

2. Build a new container image using main.Dockerfile with all files in the project directory. Replace llama-image with your desired image name:

$ docker build -f main.Dockerfile -t llama-image .

3. View all Docker images on the server and verify that a new llama-image is available:

$ docker images

Output:

REPOSITORY                                  TAG       IMAGE ID       CREATED              SIZE
llama-image                                 latest    f4fcb571a2f7   About a minute ago   80.1MB

3. Build a Llama.cpp Container Image for GPU Systems

The Llama.cpp main-cuda.Dockerfile resource contains the build context for NVIDIA GPU systems that run the latest CUDA driver packages. Follow the steps below to build a Llama container image compatible with GPU systems.

1. Copy main-cuda.Dockerfile to the Llama.cpp project directory:

$ cp .devops/main-cuda.Dockerfile .

2. Build a new container image llama-gpu-image using the main-cuda.Dockerfile with all files in the working project directory:

$ docker build -f main-cuda.Dockerfile -t llama-gpu-image .

3. View all Docker images on the server and verify that the new llama-gpu-image is available:

$ docker images

Output:

REPOSITORY                                  TAG       IMAGE ID       CREATED          SIZE
llama-gpu-image                             latest    40421f1469cf   2 minutes ago    2.08GB

To run the Llama.cpp GPU container image, verify that your target host includes the minimum or higher CUDA version referenced by the ARG CUDA_VERSION= directive within main-cuda.Dockerfile.

4. Verify the target CUDA version:

$ cat main-cuda.Dockerfile | grep CUDA_VERSION=

Output:

ARG CUDA_VERSION=12.3.1

4. Push the Llama.cpp Container Image to a Container Registry

1. Open your container registry's management panel and copy the registry URL, username, and API key (or password) to use when accessing the registry.

2. Switch to your server terminal session and log in to your container registry. Replace your-registry-url, exampleregistry, exampleuser, and registry-password with your actual registry details:

$ docker login https://your-registry-url/exampleregistry -u exampleuser -p registry-password

3. Tag the Llama.cpp container image with your desired registry tag. For example, your-registry-url/exampleregistry/llama-gpu-image:

$ docker tag llama-gpu-image your-registry-url/exampleregistry/llama-gpu-image

4. View all Docker images on the server and verify that the new tagged image is available:

$ docker images

Output:

REPOSITORY                                           TAG       IMAGE ID       CREATED          SIZE
your-registry-url/exampleregistry/llama-gpu-image    latest    40421f1469cf   3 minutes ago    2.08GB
llama-gpu-image                                      latest    40421f1469cf   3 minutes ago    2.08GB

5. Push the tagged image to your container registry repository:

$ docker push your-registry-url/exampleregistry/llama-gpu-image

6. Open your container registry's management panel and check the repositories list to verify that the new repository is available.

Next Steps

Automate the CPU and GPU builds in a CI/CD pipeline so new images are rebuilt whenever the Llama.cpp repository updates.
Pull the GPU image onto a multi-GPU host and benchmark inference throughput across different quantization levels.
Pin the CUDA_VERSION in main-cuda.Dockerfile to match the driver version installed on your target hosts.
Explore Llama.cpp's server mode to expose an OpenAI-compatible API from the container.

For the full guide with additional tips, visit the original article on Vultr Docs.

Configuring Smart Routing in NVIDIA Dynamo

Sanskriti Harmukh — Wed, 14 Oct 2026 21:07:15 +0000

NVIDIA Dynamo is an open-source, high-throughput, low-latency inference framework designed to deploy large-scale generative AI and reasoning models across multi-node, multi-GPU environments. Its KV Router intelligently directs inference requests by evaluating computational costs across workers, considering both prefill and decode phases. By maximizing KV cache reuse and minimizing redundant computation, smart routing reduces time-to-first-token (TTFT) by up to 40% and increases overall throughput by 25-30% compared to round-robin load balancing. This guide covers infrastructure setup, router enablement for both aggregated and disaggregated serving patterns, parameter tuning, and monitoring for production deployments. By the end, you'll have KV-aware smart routing configured and verified across both serving patterns, with the parameters tuned for your workload.

Prerequisites

Before you begin, ensure you:

Have access to a GPU-enabled server with NVIDIA GPUs installed and the NVIDIA Container Toolkit configured, using a non-root user with sudo privileges.
- 2 GPUs for aggregated serving with smart routing.
- 4 GPUs for disaggregated serving with smart routing.
Install Docker Engine and Docker Compose.
Create a Hugging Face account and generate an access token for gated models like Llama.
Understand basic LLM inference concepts (prefill vs decode phases, KV cache).

Understanding KV-Aware Routing

Traditional load balancing strategies like round-robin or random routing distribute requests evenly across workers without considering their cached state. KV-aware routing optimizes performance by directing requests to workers that already have relevant KV cache blocks, reducing redundant computation during the prefill phase.

How KV Cache Routing Works

The KV Router makes routing decisions based on a cost function that balances two factors.

1. Prefill Cost — the router calculates prefill cost by determining how many tokens need processing from scratch. When a worker has cached KV blocks matching the request's input prefix, those blocks can be reused, reducing prefill computation:

Prefill blocks = (Total input tokens - Cached tokens) / Block size

2. Decode Cost — the router tracks active blocks on each worker to estimate decode capacity. Workers with fewer active sequences have more capacity for new requests.

3. Combined Cost Formula:

cost = overlap_score_weight × prefill_blocks + decode_blocks

Where:

overlap_score_weight balances TTFT optimization (higher values) vs load distribution (lower values).
Lower cost indicates a better routing choice.
Default overlap_score_weight is 1.0.

Worker Selection Process

The router evaluates all available workers and selects the one with the lowest cost. Consider this example with 3 workers and overlap_score_weight = 1.0:

Worker	Cached Blocks	Prefill Blocks	Decode Blocks	Total Cost	Selected
Worker 1	2	8	10	18 (1.0 × 8 + 10)	No
Worker 2	5	5	5	10 (1.0 × 5 + 5)	Yes
Worker 3	8	2	9	11 (1.0 × 2 + 9)	No

Worker 2 provides the best balance of cache reuse and available capacity, minimizing both prefill computation and decode load.

Performance Benefits

KV-aware routing provides measurable improvements over basic load balancing:

TTFT Reduction: 30-40% lower time-to-first-token by maximizing cache hits.
Throughput Increase: 25-30% higher tokens/second by reducing redundant prefill computation.
Better GPU Utilization: More balanced load distribution across workers.
Cache Efficiency: Higher cache hit rates through intelligent request placement.

These benefits are most significant for workloads with long input prompts (>2000 tokens) where prefill cost dominates, similar or repetitive input prefixes across requests, and multiple inference workers with independent KV caches.

Clone the Dynamo Repository

NVIDIA Dynamo provides deployment scripts, launch utilities, and runtime modules for inference workloads.

1. Clone the repository:

$ git clone https://github.com/ai-dynamo/dynamo.git

2. Navigate to the repository directory:

$ cd dynamo

3. Switch to the latest stable release:

$ git checkout release/0.9.0

Visit the Dynamo releases page to find the latest stable version.

Verify CUDA Version and Pull the Container Image

The vLLM container requires a CUDA version match between the host driver and container runtime to prevent GPU kernel incompatibilities.

1. Check the installed CUDA version:

$ nvidia-smi

The output displays the CUDA version in the top-right corner of the table:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
....

2. Pull the vLLM container image from NGC, matching the image tag to your system's CUDA version:

For CUDA 13.x:

$ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13

For CUDA 12.x:

$ docker pull nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0

Visit the NVIDIA NGC Catalog to view all available image tags and CUDA versions.

3. (Optional) Build the container from source instead of pulling the pre-built image:

$ ./container/build.sh --framework VLLM

This creates an image named dynamo:latest-vllm. If you use this locally built image, replace nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 with dynamo:latest-vllm in all subsequent commands.

Configure Hugging Face Cache Permissions

The container runs as UID 1000 and requires write access to the Hugging Face cache directory for model downloads. Incorrect permissions prevent the container from accessing cached model weights, causing worker initialization failures.

1. Create the cache directory if it does not exist:

$ mkdir -p container/.cache/huggingface

2. Set ownership to the container user (UID 1000):

$ sudo chown -R 1000:1000 container/.cache/huggingface

3. Set appropriate permissions:

$ sudo chmod -R 775 container/.cache/huggingface

Start Infrastructure Services

Dynamo's KV Router requires etcd for worker discovery and NATS for KV cache event propagation. The etcd service maintains a registry of active workers and their capabilities, while NATS handles real-time KV block events (creation and eviction) that let the router track cached state across workers.

1. Start the infrastructure services using Docker Compose:

$ docker compose -f deploy/docker-compose.yml up -d

2. Verify the services are running:

$ docker compose -f deploy/docker-compose.yml ps

The output displays the running etcd and NATS containers:

NAME                   IMAGE                      COMMAND                  SERVICE       CREATED          STATUS          PORTS
deploy-etcd-server-1   bitnamilegacy/etcd:3.6.1   "/opt/bitnami/script…"   etcd-server   18 seconds ago   Up 18 seconds   0.0.0.0:2379-2380->2379-2380/tcp, [::]:2379-2380->2379-2380/tcp
deploy-nats-server-1   nats:2.11.4                "/nats-server -c /et…"   nats-server   18 seconds ago   Up 18 seconds   0.0.0.0:4222->4222/tcp, [::]:4222->4222/tcp, 0.0.0.0:6222->6222/tcp, [::]:6222->6222/tcp, 0.0.0.0:8222->8222/tcp, [::]:8222->8222/tcp

KV Router Configuration Parameters

The KV Router uses environment variables to control routing behavior, cache tracking, and worker selection logic. These parameters determine how the router balances Time To First Token (TTFT) optimization through cache reuse against Inter-Token Latency (ITL) through load distribution. All parameters can be set via environment variables or command-line arguments on the Dynamo frontend.

Core Router Parameters

Variable	Description	Default	Recommended	Notes
`DYN_ROUTER_MODE`	Enables KV cache-aware routing strategy	`round_robin`	`kv`	Set to `kv` to enable cache-aware routing. Other modes: `round_robin`, `random`
`DYN_ROUTER_TEMPERATURE`	Controls routing randomness via softmax sampling	`0.0`	`0.1`	`0.0` = deterministic (always lowest cost). Higher values add randomness to prevent saturation
`DYN_KV_OVERLAP_SCORE_WEIGHT`	Weight for prefix cache overlaps in cost calculation	`1.0`	`1.5`	Formula: `cost = weight × prefill_blocks + decode_blocks`. Higher values prioritize TTFT over ITL
`DYN_KV_CACHE_BLOCK_SIZE`	KV cache block size in tokens (must match backend)	Backend-specific	`16`	Must match vLLM's block size. Used for prefill block calculation
`DYN_KV_EVENTS`	Enables real-time KV event tracking from workers	`true`	`true`	When disabled, router predicts cache state with TTL-based expiration

Worker Capacity Thresholds

These parameters define when a worker is considered "busy" and should be excluded from routing decisions.

Variable	Description	Default	Recommended	Notes
`DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD`	Busy threshold based on KV cache block utilization	Not set	`0.85`	Value between 0.0-1.0. Workers exceeding this percentage are excluded. Can update via `/busy_threshold` API
`DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD`	Busy threshold based on prefill token count	Not set	`10000`	Absolute token count. Workers exceeding this are excluded. Uses OR logic with threshold fraction

Infrastructure Configuration

Variable	Description	Default	Notes
`ETCD_ENDPOINTS`	etcd server URL for worker discovery	N/A	Format: `http://localhost:2379`. Multiple endpoints separated by comma
`NATS_SERVER`	NATS server URL for KV event propagation	N/A	Format: `nats://localhost:4222`. Required for KV event streaming
`DYN_EVENT_PLANE`	Event plane backend for KV cache events	N/A	Set to `nats` to enable NATS-based event publishing
`PYTHONHASHSEED`	Deterministic hash seed for KV event IDs	N/A	Set to `0` for reproducible KV event ID generation
`DYN_SYSTEM_PORT`	Port for worker system status server	System-dependent	Used for health checks and metrics. Set per-worker to avoid conflicts (e.g., `8081`, `8082`)

Parameter Tuning Guidelines

For TTFT Optimization (Low Latency):

Increase DYN_KV_OVERLAP_SCORE_WEIGHT to 1.5-2.0 (prioritize cache hits).
Lower busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.70, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=500.
Keep DYN_ROUTER_TEMPERATURE low (0.0-0.1) for deterministic selection.

For Throughput Optimization (High Load):

Lower DYN_KV_OVERLAP_SCORE_WEIGHT to 0.5-1.0 (balance cache hits with load).
Higher busy thresholds: DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.95, DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=2000.
Increase DYN_ROUTER_TEMPERATURE to 0.3-0.5 for better load distribution.

When KV Events Are Unavailable:

Set DYN_KV_EVENTS=false to use prediction-based routing.
Router tracks state via routing decisions with TTL-based expiration (default 120s).
Less accurate than real-time events but works without backend KV event support.

Set Up the Deployment Environment

Both aggregated and disaggregated serving deployments use the same KV routing configuration. This section prepares the container environment and creates a shared configuration file for all deployment scripts.

1. Export your Hugging Face token (replace YOUR_HF_TOKEN with your actual token):

$ export HF_TOKEN=YOUR_HF_TOKEN

2. Run the vLLM container with GPU access and workspace mounting, using the image tag that matches your CUDA version:

$ ./container/run.sh -it --framework VLLM --mount-workspace --image nvcr.io/nvidia/ai-dynamo/vllm-runtime:0.9.0-cuda13 -e HF_TOKEN=$HF_TOKEN

This starts an interactive container session with GPU support and passes the Hugging Face token to the container.

3. Inside the container, create the common environment configuration file:

$ cat << 'EOF' > ~/kv_routing.env
# Infrastructure endpoints
export ETCD_ENDPOINTS=http://localhost:2379
export NATS_SERVER=nats://localhost:4222
export DYN_EVENT_PLANE=nats

# Set deterministic hash for KV event IDs
export PYTHONHASHSEED=0

# KV Router Configuration
export DYN_ROUTER_MODE=kv
export DYN_ROUTER_TEMPERATURE=0.1
export DYN_KV_OVERLAP_SCORE_WEIGHT=1.5
export DYN_KV_CACHE_BLOCK_SIZE=16
export DYN_KV_EVENTS=true

# Optional: Advanced routing configuration
export DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.85
export DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=10000

echo "Environment configuration loaded:"
echo "  etcd: $ETCD_ENDPOINTS"
echo "  NATS: $NATS_SERVER"
echo "  Router Mode: $DYN_ROUTER_MODE"
EOF

This file contains shared configuration variables for all deployment scripts, including etcd endpoints, NATS server address, and KV routing parameters.

Configure KV Routing for Aggregated Serving

Aggregated serving combines prefill and decode phases on a single worker. KV-aware routing optimizes which worker receives each request based on cached KV blocks, improving performance even when workers handle both phases.

Deploy Aggregated Serving with KV Routing

The deployment script launches multiple vLLM workers with KV event publishing enabled and a frontend configured for KV-aware routing, handling process cleanup and providing routing configuration feedback.

1. Inside the container, create the aggregated serving deployment script:

$ cat << 'EOF' > ~/nemotron_agg_kv_routing.sh
#!/bin/bash
source ~/kv_routing.env

MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}"

pkill -f "dynamo.frontend"; pkill -f "dynamo.vllm"; sleep 2
set -e
trap 'kill 0' EXIT

python -m dynamo.frontend --router-reset-states &
sleep 10

# Worker 1 - GPU 0 (ZMQ port 5557)
DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT1:-8081} \
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --trust-remote-code \
    --connector none \
    --enforce-eager \
    --enable-prefix-caching \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}' &

# Worker 2 - GPU 1 (ZMQ port 5558)
DYN_SYSTEM_PORT=${DYN_SYSTEM_PORT2:-8082} \
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --trust-remote-code \
    --connector none \
    --enforce-eager \
    --enable-prefix-caching \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5558"}' &

sleep 30
echo "Aggregated serving ready at http://localhost:8000 (Model: $MODEL)"
wait
EOF

This script configures aggregated serving with KV-aware routing:

Accepts the model name as a command-line argument (defaults to NVIDIA Nemotron Nano 4B).
Sources the environment configuration created earlier (includes DYN_EVENT_PLANE=nats).
Uses --connector none for aggregated serving (prefill+decode on same worker).
Uses --enforce-eager for fast deployment without compilation.
Enables prefix caching with --enable-prefix-caching for KV cache reuse.
Enables KV event publishing with ZMQ endpoints (ports 5557, 5558) on both workers.
Configures system ports with DYN_SYSTEM_PORT to avoid conflicts (8081, 8082).
Uses --trust-remote-code to support NVIDIA Nemotron models.

KV Event Flow Architecture: vLLM natively publishes KV cache events via ZMQ. Dynamo's bridge component automatically translates these events to NATS for router consumption:

┌─────────────────────┐
│ vLLM Worker 1       │
│ (ZMQ: tcp://*:5557) │───┐
└─────────────────────┘   │                                           ┌─────────────────────┐
                          │    ┌──────────────┐    ┌──────────────┐   │ KV Router           │
                          ├───>│ Dynamo Bridge├───>│ NATS Server  ├──>│ (Builds Radix Tree) │
                          │    └──────────────┘    └──────────────┘   │ Global Cache View   │
┌─────────────────────┐   │                                           └─────────────────────┘
│ vLLM Worker 2       │   │
│ (ZMQ: tcp://*:5558) │───┘
└─────────────────────┘

This architecture provides the router with a unified view of cache state across all workers while allowing vLLM to use its native ZMQ event publishing. You'll see ZMQ endpoints in worker logs, but events flow through the Dynamo bridge to NATS for router consumption.

2. Make the script executable:

$ chmod +x ~/nemotron_agg_kv_routing.sh

3. Run the aggregated serving script:

$ ~/nemotron_agg_kv_routing.sh

The script starts the frontend with KV routing enabled and launches two vLLM workers on GPUs 0 and 1. The frontend connects to NATS and begins consuming KV events from all registered workers, building a prefix (radix) tree to track cached blocks across workers.

To deploy a different model, pass the model name as an argument:

$ ~/nemotron_agg_kv_routing.sh "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5"

Verify KV Routing

The router logs provide insights into worker selection and cost calculations.

1. Open a new terminal session and send a test request to the frontend:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Explain the concept of KV cache in LLM inference."}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'

The output displays the model's response in JSON format with streaming tokens.

2. Check the frontend logs for routing decision details, which show cost calculations for each worker:

Formula for worker_id=7587893399646425890 dp_rank=0 with 0 cached blocks: 5.188 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.125 + 2.000
Formula for worker_id=7587893399646425892 dp_rank=0 with 0 cached blocks: 5.188 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.125 + 2.000
Selected worker: worker_id=7587893399646425892 dp_rank=0, logit: 5.188, cached blocks: 0, tree size: 0, total blocks: 75379

The router calculates cost per worker with cost = overlap_weight × prefill_blocks + decode_blocks. In this example, both workers have 0 cached blocks (cold start), equal prefill costs (2.125 blocks weighted by 1.5), and equal decode loads (2.000). With identical costs (5.188), the router selects a worker based on temperature sampling.

3. Send a second request with a similar prompt prefix:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant."},
      {"role": "user", "content": "Explain the concept of KV cache reuse in distributed LLM serving."}
    ],
    "max_tokens": 100,
    "temperature": 0.7
  }'

4. Check the routing decision for the second request in the logs. If the router detects cached blocks from the previous request with a matching prompt prefix, the logs show a reduced prefill cost:

Formula for worker_id=7587893399646425890 dp_rank=0 with 0 cached blocks: 5.375 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.250 + 2.000
Formula for worker_id=7587893399646425892 dp_rank=0 with 1 cached blocks: 3.875 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 1.250 + 2.000
Selected worker: worker_id=7587893399646425892 dp_rank=0, logit: 3.875, cached blocks: 1, tree size: 8, total blocks: 75379

The router prioritizes the worker with 1 cached block, reducing its cost from 5.375 to 3.875 — the cached block reduces prefill computation needed (1.250 blocks vs 2.250), demonstrating KV cache reuse resulting in faster TTFT for requests with overlapping prompt prefixes.

5. Open a new terminal session on the host (outside the container) and monitor the KV event stream to verify event publishing:

$ docker run --rm -it --network container:deploy-nats-server-1 \
  natsio/nats-box nats stream info namespace-dynamo-component-backend-kv-events

The output displays the JetStream statistics including message count and consumer information:

Information for Stream namespace-dynamo-component-backend-kv-events
......
                Subjects: namespace-dynamo-component-backend-kv-events.*
                Replicas: 1
                Storage: File
......
State:
......
                Messages: 8
                Bytes: 2.3 KiB
        Active Consumers: 1
......

The message count increases as workers publish KV cache events — each message represents a KV cache state change, such as block creation or eviction.

Configure KV Routing for Disaggregated Serving

Disaggregated serving separates prefill and decode phases across different worker pools. The KV Router optimizes both prefill worker selection (based on cache hits) and decode worker assignment (based on load distribution), while coordinating KV cache transfer between phases via NIXL.

Deploy Disaggregated Serving with KV Routing

The deployment script configures the frontend with increased overlap weight for prefill optimization, launches prefill workers with low batch sizes for TTFT optimization, and decode workers with high batch sizes for throughput maximization. The NIXL connector enables efficient KV cache transfer between prefill and decode workers.

1. If the aggregated serving script is still running from the previous section, stop it by pressing Ctrl+C in the container terminal.

2. Inside the same container, create the disaggregated serving deployment script:

$ cat << 'EOF' > ~/nemotron_disagg_kv_routing.sh
#!/bin/bash
source ~/kv_routing.env

MODEL="${1:-nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1}"

pkill -f "dynamo.frontend"; pkill -f "dynamo.vllm"; sleep 2
set -e
trap 'kill 0' EXIT

python -m dynamo.frontend --router-reset-states &
sleep 10

# Decode Worker 1 - GPU 0
CUDA_VISIBLE_DEVICES=0 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-decode-worker \
    --max-model-len 2048 \
    --enable-prefix-caching \
    --trust-remote-code &

# Decode Worker 2 - GPU 1
VLLM_NIXL_SIDE_CHANNEL_PORT=20097 \
CUDA_VISIBLE_DEVICES=1 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-decode-worker \
    --max-model-len 2048 \
    --enable-prefix-caching \
    --trust-remote-code &

# Prefill Worker 1 - GPU 2 (ZMQ port 5559)
CUDA_VISIBLE_DEVICES=2 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-prefill-worker \
    --max-model-len 2048 \
    --enable-prefix-caching \
    --trust-remote-code \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5559"}' &

# Prefill Worker 2 - GPU 3 (ZMQ port 5560)
VLLM_NIXL_SIDE_CHANNEL_PORT=20099 \
CUDA_VISIBLE_DEVICES=3 python3 -m dynamo.vllm \
    --model "$MODEL" \
    --connector nixl \
    --is-prefill-worker \
    --max-model-len 2048 \
    --enable-prefix-caching \
    --trust-remote-code \
    --kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5560"}' &

sleep 30
echo "Disaggregated serving ready at http://localhost:8000 (Model: $MODEL)"
echo "Architecture: 2 Decode Workers (GPU 0-1) + 2 Prefill Workers (GPU 2-3)"
wait
EOF

This script configures disaggregated serving with KV-aware routing and NIXL for KV cache transfer:

Accepts the model name as a command-line argument (defaults to NVIDIA Nemotron Nano 4B).
Sources the environment configuration created earlier (includes DYN_EVENT_PLANE=nats).
Uses --connector nixl to enable NIXL for KV cache transfer between prefill and decode workers.
Sets unique NIXL ports via VLLM_NIXL_SIDE_CHANNEL_PORT for workers 2 and 4 (20097, 20099).
Enables prefix caching on all workers with --enable-prefix-caching for KV cache reuse.
Configures decode workers on GPUs 0-1 to receive KV cache from prefill workers via NIXL.
Configures prefill workers on GPUs 2-3 with ZMQ endpoints (ports 5559, 5560) for KV event publishing.
Uses --trust-remote-code to support NVIDIA Nemotron models.
Uses --max-model-len 2048 for efficient memory usage across all workers.
Frontend automatically detects prefill and decode workers to enable disaggregated routing.

3. Make the script executable:

$ chmod +x ~/nemotron_disagg_kv_routing.sh

4. Run the disaggregated serving script:

$ ~/nemotron_disagg_kv_routing.sh

The script starts the frontend, two decode workers on GPUs 0-1, and two prefill workers on GPUs 2-3. The frontend automatically detects prefill workers and creates an internal prefill router for cache-aware prefill worker selection, while the main router handles decode worker assignment based on load distribution.

To deploy a different model, pass the model name as an argument:

$ ~/nemotron_disagg_kv_routing.sh "nvidia/Llama-3_3-Nemotron-Super-49B-v1_5"

Verify Disaggregated KV Routing

Disaggregated routing involves two routing decisions: prefill worker selection based on cache overlap, and decode worker assignment based on load and available capacity.

1. Open a new terminal session and send a test request to verify the disaggregated flow:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant specialized in distributed systems."},
      {"role": "user", "content": "Explain how KV cache transfer works in disaggregated LLM serving using NIXL."}
    ],
    "max_tokens": 150,
    "temperature": 0.7
  }'

2. Check the frontend logs for prefill routing decisions, which show prefill worker cost calculations with the higher overlap weight (1.5):

INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425930 dp_rank=0 with 0 cached blocks: 4.125 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.750 + 0.000
INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425934 dp_rank=0 with 0 cached blocks: 4.125 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.750 + 0.000
INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425930 dp_rank=0, logit: 4.125, cached blocks: 0, tree size: 0, total blocks: 75379

On the first (cold-start) request, both prefill workers have 0 cached blocks, resulting in equal costs (4.125). The decode blocks component is 0.0 because prefill workers only handle the prefill phase.

3. Check the decode routing decisions in the script output. After prefill completion, the decode router assigns the request to a decode worker:

INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425936 dp_rank=0 with 0 cached blocks: 2.000 = 0.0 * prefill_blocks + decode_blocks = 0.0 * 2.750 + 2.000
INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425936 dp_rank=0, logit: 2.000, cached blocks: 0, tree size: 0, total blocks: 75379

The prefill blocks component is 0.0 in this formula because decode workers don't compute prefills — they receive KV cache from prefill workers via NIXL transfer.

4. Monitor NIXL transfer success in the worker logs displayed by the script:

INFO nixl_connector._nixl_handshake: NIXL compatibility check passed (hash: cae818f9461cb277c305be4b8cd4aca3273e21a6ea98f7399d1a339560e1b3ea)

NIXL successfully transfers KV cache blocks from prefill to decode workers using RDMA for low-latency, high-bandwidth data transfer.

5. Send a second request with a similar prompt prefix to verify cache reuse:

$ curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "messages": [
      {"role": "system", "content": "You are a helpful AI assistant specialized in distributed systems."},
      {"role": "user", "content": "Explain how KV cache reuse works in disaggregated LLM serving."}
    ],
    "max_tokens": 150,
    "temperature": 0.7
  }'

6. Check the prefill routing logs for the second request, which show improved cache hits:

INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425930 dp_rank=0 with 1 cached blocks: 2.250 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 1.500 + 0.000
INFO dynamo_llm::kv_router::scheduler: Formula for worker_id=7587893399646425934 dp_rank=0 with 0 cached blocks: 3.750 = 1.5 * prefill_blocks + decode_blocks = 1.5 * 2.500 + 0.000
INFO dynamo_llm::kv_router::scheduler: Selected worker: worker_id=7587893399646425930 dp_rank=0, logit: 2.250, cached blocks: 1, tree size: 2, total blocks: 75379

The prefill router now sees 1 cached block on the selected worker, reducing its cost from 3.750 to 2.250 — demonstrating KV-aware routing's ability to optimize TTFT by leveraging cached prompt prefixes.

Configure KV Routing with SGLang and TensorRT-LLM

The KV Router environment configuration is universal across all inference backends. The same environment variables used for vLLM deployments work identically with SGLang and TensorRT-LLM, requiring only backend-specific adjustments for block size and KV event publishing.

Universal Environment Variables

These environment variables work across all backends (vLLM, SGLang, TensorRT-LLM) without modification:


# Infrastructure endpoints
export ETCD_ENDPOINTS=http://localhost:2379
export NATS_SERVER=nats://localhost:4222
export DYN_EVENT_PLANE=nats
export PYTHONHASHSEED=0

# Router configuration
export DYN_ROUTER_MODE=kv
export DYN_ROUTER_TEMPERATURE=0.1
export DYN_KV_OVERLAP_SCORE_WEIGHT=1.5
export DYN_KV_EVENTS=true

# Load balancing thresholds
export DYN_ACTIVE_DECODE_BLOCKS_THRESHOLD=0.85
export DYN_ACTIVE_PREFILL_TOKENS_THRESHOLD=10000

Backend-Specific Configuration

While the router environment variables are universal, two aspects require backend-specific configuration.

Block Size Configuration — DYN_KV_CACHE_BLOCK_SIZE must match your backend's actual block size:

Backend	Typical Block Size	Configuration
SGLang	16	`export DYN_KV_CACHE_BLOCK_SIZE=16`
TensorRT-LLM	64 or 128 (varies by model)	`export DYN_KV_CACHE_BLOCK_SIZE=64`
vLLM	16	`export DYN_KV_CACHE_BLOCK_SIZE=16`

Mismatched block sizes between the router and workers cause incorrect cost calculations and routing decisions. Always verify your backend's block size configuration.

KV Event Publishing Configuration — each backend uses different command-line parameters to enable KV event publishing to the router:

Backend	Parameter	Example Value
SGLang	`--kv-events-config`	`nats://localhost:4222`
TensorRT-LLM	`--publish-events-and-metrics`	No value (flag only)
vLLM	`--kv-events-config`	`'{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}'`

SGLang publishes events using a NATS URL format:

--kv-events-config nats://localhost:4222

TensorRT-LLM uses a simple flag to enable event publishing:

--publish-events-and-metrics

vLLM uses a JSON configuration format with a ZMQ endpoint:

--kv-events-config '{"enable_kv_cache_events":true,"endpoint":"tcp://*:5557"}'

Tuning KV Router Parameters

The KV Router provides several parameters to optimize routing behavior for different workload characteristics. Proper tuning balances cache reuse (TTFT optimization) with load distribution (ITL optimization).

Overlap Score Weight

The --kv-overlap-score-weight parameter controls the router's preference for cache hits versus load balancing. Default value: 1.0.

Increase (1.5-2.0) for prefill-heavy workloads — long input prompts (>2000 tokens), short output generation (<200 tokens), and where minimizing TTFT is critical:

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --kv-overlap-score-weight 1.5

Decrease (0.5-0.8) for decode-heavy workloads — short input prompts (<500 tokens), long output generation (>1000 tokens), and where maximizing throughput is critical:

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --kv-overlap-score-weight 0.7

Set to 0.0 to disable cache-aware routing and use pure load balancing:

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --kv-overlap-score-weight 0.0

This mode ignores cached blocks and routes based solely on worker load — useful for workloads with no prefix overlap or for baseline performance comparisons.

Router Temperature

The --router-temperature parameter introduces randomness in worker selection to prevent saturation and improve load distribution. Default value: 0.0 (deterministic).

Temperature = 0.0 (default): Always select the lowest-cost worker. Best for maximizing cache reuse in stable workloads, with the risk of oversaturating workers with the best cache state.
Temperature = 0.2-0.5: Add slight randomness while still favoring low-cost workers. Best for production workloads with variable request patterns, giving better load distribution without sacrificing much cache efficiency:

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --router-temperature 0.3

Temperature = 1.0+: More uniform distribution across workers. Best for debugging, testing, or when cache reuse is less important — approaches random selection at very high temperatures.

KV Events Configuration

The --no-kv-events flag disables real-time KV event tracking and switches to prediction-based routing. Default: KV events enabled.

When enabled, the router predicts cache state based on its own routing decisions instead of receiving events from workers:

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --no-kv-events \
  --router-ttl 120 \
  --router-max-tree-size 1048576 \
  --router-prune-target-ratio 0.8

Additional parameters for prediction mode:

--router-ttl 120: Expire cached blocks after 120 seconds (default: 120).
--router-max-tree-size 1048576: Maximum blocks before pruning (default: 2^20).
--router-prune-target-ratio 0.8: Prune to 80% of max size when threshold exceeded.

Use prediction mode when your backend doesn't support KV event publishing, you want lower infrastructure overhead (no NATS dependency), or your deployment is simple enough that prediction accuracy is acceptable.

When using --no-kv-events, configure backends to disable event publishing:

vLLM: Use --kv-events-config '{"enable_kv_cache_events": false}'.
SGLang: Do not use --kv-events-config.
TRT-LLM: Do not use --publish-events-and-metrics.

Active Block Busy Detection

The --active-decode-blocks-threshold parameter marks workers as busy when their KV cache utilization exceeds a threshold, preventing overload. Default: disabled (no threshold).

$ python3 -m dynamo.frontend \
  --router-mode kv \
  --active-decode-blocks-threshold 0.85 \
  --active-prefill-tokens-threshold 10000

--active-decode-blocks-threshold 0.85: Mark worker busy at 85% KV cache utilization.
--active-prefill-tokens-threshold 10000: Mark worker busy when prefill tokens exceed 10,000.

Busy workers are excluded from routing decisions until their load drops below the threshold, preventing request queueing and maintaining consistent latency. Thresholds can be updated at runtime without restarting the frontend:

$ curl -X POST http://localhost:8000/busy_threshold \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
    "active_decode_blocks_threshold": 0.90,
    "active_prefill_tokens_threshold": 12000
  }'

The response confirms the updated thresholds:

{
  "model": "nvidia/Llama-3.1-Nemotron-Nano-4B-v1.1",
  "active_decode_blocks_threshold": 0.90,
  "active_prefill_tokens_threshold": 12000
}

Router Replica Synchronization

The --router-replica-sync flag enables active block state synchronization between multiple router replicas for consistent load balancing. Default: disabled.

Enable it when launching multiple frontend replicas for fault tolerance and horizontal scaling:


# Router replica 1
$ python3 -m dynamo.frontend \
  --router-mode kv \
  --http-port 8000 \
  --router-replica-sync

# Router replica 2
$ python3 -m dynamo.frontend \
  --router-mode kv \
  --http-port 8001 \
  --router-replica-sync

Each replica maintains its own view of active blocks (ephemeral state); replicas exchange routing decisions via NATS core messaging, and cached block state (persistent) syncs automatically via JetStream. New replicas start with zero active blocks but converge quickly through sync, giving consistent routing decisions and better load distribution across replicas.

Monitor KV Router Performance

To monitor KV Router performance metrics including routing decisions, cache efficiency, and worker utilization, deploy an observability stack (Grafana and Prometheus) alongside your Dynamo deployment. Add the KV Router environment variables from this guide to your deployment configuration, then use Grafana dashboards to visualize KV cache routing performance metrics and Prometheus to track TTFT improvements, cache hit rates, and worker selection patterns.

Next Steps

Tune DYN_KV_OVERLAP_SCORE_WEIGHT and DYN_ROUTER_TEMPERATURE against your real traffic patterns rather than the defaults
Stand up Grafana and Prometheus dashboards to track TTFT, cache hit rate, and worker load over time
Test SGLang or TensorRT-LLM backends with the same router configuration to compare cache efficiency
Enable --router-replica-sync before scaling to multiple frontend replicas in production

For the full guide with additional tips, visit the original article on Vultr Docs.

Automating Bash Scripts on Linux Using Cron and Systemd

Sanskriti Harmukh — Wed, 14 Oct 2026 21:06:31 +0000

Automating Bash scripts on Linux is essential for system administration tasks, such as installing applications and monitoring system logs. Cron is a job scheduler that runs tasks at specific intervals, while systemd is a service manager that handles applications through service profiles. This guide walks through writing two Bash scripts — one for log monitoring and one for backups — then automating them with a cron job and running one as a persistent systemd service. By the end, you'll have both a scheduled cron job and a managed systemd service running your own Bash scripts automatically.

Prerequisites

Before you begin:

Have access to a Ubuntu server as your management workstation.
Access the server using SSH as a non-root sudo user.
Install a LAMP stack (Apache, MySQL, PHP).

Create Bash Scripts

Bash scripts support multiple commands and functions that define how tasks run in a system. Follow the steps below to create two Bash scripts: one to back up critical web application files, and one to monitor system logs for errors.

1. Switch to your user's home directory:

$ cd

2. Create a new log-monitor.sh script:

$ nano log-monitor.sh

3. Add the following contents to the log-monitor.sh file:

#!/bin/bash

LOG_FILE="/var/log/syslog" 
ERROR_LOG="/opt/data/error_log.txt"
WARNING_LOG="/opt/data/warning_log.txt"

tail -F "$LOG_FILE" | while read line; do
    if echo "$line" | grep -q "error"; then
        echo "$line" >> "$ERROR_LOG"
    elif echo "$line" | grep -q "warning"; then
        echo "$line" >> "$WARNING_LOG"
    fi
done

Save and close the file. This script creates error_log.txt and warning_log.txt files in the /opt/data directory, filtering results from /var/log/syslog using a while loop. It writes each new line to the respective error or warning file as it appears.

4. Create another script named backup.sh:

$ nano backup.sh

5. Add the following contents to the file:

#!/bin/bash

DB_USER="user"
DB_PASSWORD="new-password"
DB_NAME="mysql"

DB_BACKUP_DIR="mysql_backup"
WEB_FILES_DIR="web_backup"

mkdir -p "$DB_BACKUP_DIR"
mkdir -p "$WEB_FILES_DIR"

DATE=$(date +"%Y%m%d_%H%M%S")

mysqldump -u"$DB_USER" -p"$DB_PASSWORD" "$DB_NAME" > "$DB_BACKUP_DIR/db_backup_$DATE.sql"
echo "MySQL backup completed: $DB_BACKUP_DIR/db_backup_$DATE.sql"

tar -czf "$WEB_FILES_DIR/web_backup_$DATE.tar.gz" /var/www/html
echo "Web application backup completed: $WEB_FILES_DIR/web_backup_$DATE.tar.gz"

Save and close the file. This script connects to the MySQL database server using the DB_... variables, then creates mysql_backup and web_backup directories to store the database and web application backups.

6. Move the Bash scripts to a system-wide directory like /opt:

$ sudo cp backup.sh log-monitor.sh /opt/

7. Create a new data directory under the system-wide directory:

$ sudo mkdir -p /opt/data

8. Change the directory permissions to 777 to allow all users to write files:

$ sudo chmod 777 /opt/data

Automate the Bash Script Using Cron

Cron creates jobs you can use to schedule and automatically run a Bash script at specific intervals. Follow the steps below to automate the backup.sh script to back up the MySQL database and web application files daily.

1. Open the crontab editor:

$ crontab -e

Select your desired text editor when prompted — for example, select 1 to choose the nano text editor.

2. Add the following directive at the end of the file:

0 2 * * * /opt/backup.sh

Save and close the file. This cron job runs backup.sh daily at 2:00 A.M. In the job:

0: Runs the script at the start of a minute. Set a value between 0 and 59.
2: Runs the script at 2:00 A.M. Specify an hour between 0 and 23.
*: Runs the script every day of the month. Specify a day between 1 and 31.
*: Runs the script every month. Specify a month between 1 and 12.
*: Runs the script daily. Specify a day of the week between 0 and 7.

3. List all cron jobs and verify the new task is available:

$ crontab -l

Output:

# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command

0 2 * * * ~/backup.sh

Automate the Bash Script as a System Service Using Systemd

Systemd starts system services automatically at boot and offers options for stopping and restarting them. Follow the steps below to set up the log-monitoring script as a system service.

1. Create a new system service file:

$ sudo nano /etc/systemd/system/logmonitor.service

2. Add the following contents to the logmonitor.service file:

[Unit]
Description= Automatic System Logs Monitoring and Filtering
After=network.target

[Service]
ExecStart=bash /opt/log-monitor.sh
Restart=always

[Install]
WantedBy=multi-user.target

Save and close the file. This service configuration runs the log-monitor.sh script from /opt to monitor and filter log entries in /var/log/syslog.

3. Reload systemd to apply the service file changes:

$ sudo systemctl daemon-reload

4. Enable the new system service to automatically start at boot:

$ sudo systemctl enable logmonitor.service

Output:

Created symlink /etc/systemd/system/multi-user.target.wants/logmonitor.service → /etc/systemd/system/logmonitor.service.

5. Start the system service:

$ sudo systemctl start logmonitor

6. View the system service status and verify it's running:

$ sudo systemctl status logmonitor

Output:

● logmonitor.service - Automatic System Logs Monitoring and Filtering
 Loaded: loaded (/etc/systemd/system/logmonitor.service; enabled; preset: enabled)
 Active: active (running) since Wed 2024-11-06 13:20:07 UTC; 11s ago
Main PID: 14293 (bash)
   Tasks: 3 (limit: 1062)
  Memory: 784.0K (peak: 1.5M)
     CPU: 40ms
  CGroup: /system.slice/logmonitor.service
          ├─14293 bash /opt/log-monitor.sh
          ├─14308 tail -F /var/log/syslog
          └─14309 bash /opt/log-monitor.sh

Nov 06 13:20:07 Linux-server systemd[1]: Started logmonitor.service - Automatic System Logs Monitoring and Filtering.

7. Run the following commands to simulate new error and warning entries in /var/log/syslog:

$ logger -p user.err "This is a simulated error log message from MySQL"
$ logger -p user.err "This is a simulated warning log message from Apache"
$ logger -p user.err "This is a simulated error log message from PHP"
$ logger -p user.err "This is a simulated Warning log message from MySQL"

8. List all files in /opt/data and verify that error_log.txt and warning_log.txt are available:

$ ls /opt/data

Output:

error_log.txt warning_log.txt

9. View the error_log.txt file and verify that only error entries are present:

$ cat error.log

Output:

2024-11-06T13:33:48.010096+00:00 Server linuxuser: This is a simulated error log message from MySQL
2024-11-06T13:33:48.010096+00:00 Server linuxuser: This is a simulated error log message from PHP

Next Steps

Add more granular filters (by service name or severity) to log-monitor.sh
Extend backup.sh with retention logic to prune old backups automatically
Ship backups to remote or off-site storage so they survive a full server loss
Add alerting (email or webhook) when new entries land in error_log.txt

For the full guide with additional tips, visit the original article on Vultr Docs.

DEV Community: Sanskriti Harmukh

Building Serverless APIs with Kotlin Ktor and Coroutines

Prerequisites

Setting Up Your Development Environment

Understanding Ktor and Coroutines for Serverless

Building Your First Serverless API with Ktor

Handling Concurrency and Performance

Integrating External Services

Ktor HTTP Client (timeouts, retries, JSON)

Minimal "circuit breaker" guard (lightweight)

Database integration tips (JDBC/JPA/R2DBC)

Observability (logs, metrics, traces)

Hardening checklist

Deploying and Monitoring Across Cloud Providers

Package a single, portable JAR

Make the app cloud-friendly (port, health, graceful shutdown)

Observability: logs, metrics, traces (portable setup)

Advanced Tips for Optimizing Ktor in Serverless Environments

Next Steps

Using Rsync to Transfer Files in Linux

Install Rsync

Rsync Command Syntax

Rsync Command Options

Transfer Files and Directories Using Rsync

Transfer Local Files

Transfer Local Directories

Exclude Files or Directories Using Rsync

Synchronize Data to a Remote Machine Using Rsync

Warning on Using the --delete Rsync Option

Next Steps

Using Curl

The Short Answer Version

Using Curl

curl Command Options

Detailed Usage

Download a Webpage

Use FTP Protocol

Save Output to a File

Resume an Interrupted Download

Enable Verbose Output

Making HTTP Requests

Get Only Response Headers

Follow Redirects

Change the HTTP Method

Add Custom Headers

Set a Custom User-Agent

Send Form or JSON Data (POST)

Basic Authentication

Advanced Usage and Tips

Exit on HTTP Errors (--fail)

Measure HTTP Request Time (-w)

Upload Files (-F)

Authenticate with Bearer Token

Quiet or Debug Output (-s, -v)

Next Steps

Using the Linux IP Command to Manage Server Network Interfaces

Prerequisites

How to Use the IP Command

View Network Interface Information

Manage Interface IP Addresses Using the IP Command

Configure Static Routing Using the IP Command

Enable or Disable Network Interfaces

Managing the Network Interface MTU (Maximum Transmission Unit)

Next Steps

Scanning for Open Ports and Vulnerabilities with Nmap

Prerequisites

Install Nmap

The nmap Command

Scan Types

Command Options

Perform a Basic Port Scan

Scan for Service Versions and Operating Systems

Scan for Vulnerabilities Using the Nmap Scripting Engine (NSE)

Steps to Mitigate

Next Steps

Setting Up Firewall Policies using Firewalld

Prerequisites

1. Install Firewalld

2. Set Up Firewalld Zones

3. Set Up Firewall Policies using Firewalld

Warning on Using the `--delete` Rsync Option

`curl` Command Options

Exit on HTTP Errors (`--fail`)

Measure HTTP Request Time (`-w`)

Upload Files (`-F`)

Quiet or Debug Output (`-s`, `-v`)

The `nmap` Command

Use a `.dockerignore` File