DEV Community: Sangmin Yoon

Tired of `kubectl port-forward`? Simplify Kubernetes Local Dev with kforward!

Sangmin Yoon — Mon, 14 Apr 2025 15:05:33 +0000

https://github.com/sanspareilsmyn/kforward

Hello community! 👋

If you're developing applications that interact with services running inside a Kubernetes cluster, you've probably felt the pain. Managing countless kubectl port-forward sessions in separate terminals? Trying to remember which local port maps to which service? Hardcoding localhost:PORT in your configs, only to change it back for deployment? It's a frustrating, error-prone dance.

That's why I built kforward!

kforward is a lightweight, single-binary Go CLI tool designed to streamline your local development workflow against Kubernetes. It acts as a local HTTP proxy that automatically manages kubectl port-forward processes for you.

🤔 The Pain Points kforward Solves

Tedious Port-Forwarding: Opening a new terminal for every single service you need access to? It's messy, easy to forget one, and doesn't scale as your app grows.
Configuration Chaos: Littering your local development config files with localhost:3001, localhost:9200, localhost:5433 makes them differ significantly from how your app actually runs in the cluster, leading to potential "works on my machine" issues.
Complexity Overkill: Tools offering deep integration (like network interception or file mirroring) are awesome but can be complex to set up or might be more than you need if you just want to talk to your backend API or database service.

✨ Enter kforward: Simplicity First

kforward takes a simpler approach.

🚀 Simple & Reliable: It provides a standard HTTP/HTTPS proxy on your local machine. Under the hood, it intelligently uses the battle-tested kubectl port-forward for the actual tunneling.
🤖 kubectl Automation: Point kforward at a specific service (namespace/service-name) or an entire namespace, and it automatically finds ready pods and starts/manages the required kubectl port-forward processes in the background. No more manual juggling!
📦 Single Go Binary: Easy to install and run. Just make sure you have kubectl installed and in your PATH (kforward uses it!).
🌐 Use Native Service Names: This is the magic! Configure your local app (or just your terminal session) to use the kforward proxy. Now you can use standard Kubernetes service DNS names like http://my-api.my-namespace.svc.cluster.local:8080 directly in your code or tools like curl, just like you would inside the cluster.

🏗️ How Does It Work? (The Gist)

kforward runs as a local process that you start from your terminal.

It reads your kubeconfig (or uses the context you specify) to talk to your K8s cluster using client-go.
Based on the services or namespace you target, it figures out which ports need forwarding.
It launches kubectl port-forward commands in the background for each required target, mapping them to available local ports (like localhost:10001, localhost:10002, etc.). It keeps track of this mapping.
It starts a local HTTP/HTTPS proxy (defaulting to localhost:1080).
You configure your local application or terminal environment to use this proxy (e.g., export http_proxy=http://localhost:1080).
When your app makes a request to http://my-service.default.svc.cluster.local, the request hits the kforward proxy.
The proxy identifies the target (my-service.default) and looks up the corresponding local port (e.g., 10001) managed by a background kubectl process.
The proxy forwards the request to localhost:10001, which the kubectl process tunnels securely to the actual pod in your cluster.

Here's a simplified view:

🙌 Get Involved!

kforward is open source and built to solve a common developer pain point. Give it a try and see if it simplifies your Kubernetes development workflow!

Check out the code, detailed architecture, and contribute on GitHub: [Link to your GitHub Repo]
Found it useful? Star the repo! ⭐
Feedback & Ideas: Open an issue or join the discussion!

Let me know what you think in the comments! How do you currently handle local development against Kubernetes services?

FeatureLens: Real-time Go monitor for ML feature pipeline quality & drift detection.

Sangmin Yoon — Sun, 13 Apr 2025 15:05:31 +0000

Project Link

FeatureLens

FeatureLens is a lightweight, high-performance monitoring tool designed specifically for real-time Machine Learning feature pipelines. Built with Go, it focuses on detecting data quality issues and drift efficiently, helping you maintain the health and reliability of your MLOps infrastructure. Think of it as a magnifying glass for your live feature data streams.

🤔 Why FeatureLens?

Machine Learning models are fundamentally dependent on the quality and consistency of their input features. In production environments, real-time feature pipelines (e.g., feeding feature stores or directly serving models) are critical but vulnerable to silent failures:

Data Quality Degradation: Unexpected null values, outliers, or data type changes can creep in due to upstream issues.
Data Drift: The statistical distribution of live features can deviate significantly from the distributions seen during training, silently harming model performance (Training-Serving Skew).
Pipeline Latency: Delays in feature calculation or delivery can impact downstream applications.

While various monitoring solutions exist, they can sometimes be:

Resource-Intensive: Requiring significant CPU/memory, especially when monitoring numerous features or high-throughput streams.
Complex to Deploy/Manage: Often involving heavy dependencies or intricate setups.
Batch-Oriented: Not ideally suited for detecting issues in truly real-time streams as they happen.

FeatureLens aims to address these challenges by leveraging Go's strengths:

🚀 High Performance & Concurrency: Go's goroutines allow FeatureLens to efficiently monitor multiple Kafka topics/partitions and calculate statistics concurrently with minimal overhead.
📉 Low Resource Footprint: Consumes significantly fewer resources compared to many Python-based alternatives, making it cost-effective.
📦 Simple Deployment: Distributed as a single static binary, making containerization (Docker) and deployment extremely straightforward, without Python environment hassles.
⏱️ Real-time Focus: Designed from the ground up for low-latency processing of streaming data.

By providing timely insights into feature health, FeatureLens helps MLOps teams detect problems early, reduce debugging time, and increase trust in their production ML systems.

✨ Features (MVP Scope)

The initial Minimum Viable Product (MVP) will focus on delivering the core value proposition with the following features:

Kafka Consumer: Consume feature data messages from specified Apache Kafka topic(s). Assumes messages are in a common format (e.g., JSON).
Real-time Statistics Calculation (Per Feature):
- Process messages within configurable time windows (e.g., 1-minute tumbling windows).
- Calculate basic data quality metrics for specified feature fields:
  - Null / Missing Value Rate: Percentage of null or missing values.
  - Mean (Numerical Features): Average value within the window.
  - Variance / Standard Deviation (Numerical Features): Measure of data dispersion.
  - (Future: Cardinality for categorical, min/max, etc.)
Threshold-Based Alerting:
- Define acceptable thresholds for calculated metrics in a configuration file ( e.g., null_rate < 0.05, mean between 10 and 20).
- Trigger alerts when metrics violate these thresholds.
Simple Logging / Alert Output:
- Log calculated metrics periodically.
- Log alerts to standard output (stdout) when thresholds are breached. (Future: Integrate with Slack, PagerDuty, etc.)
Configuration: Load settings (Kafka brokers, topics, features to monitor, window size, thresholds) from a configuration file (e.g., YAML) or environment variables.

🏗️ Architecture (MVP)

The MVP architecture is designed to be simple and efficient, focusing on Go's strengths in stream processing and concurrency.

+------------------+      Feature Data Stream      +--------------------------------------+      Alerts / Logs
|   Apache Kafka   | ---------------------------> |        FeatureLens (Go Process)      | ---------------------> stdout / Log File
| (Topics/Brokers) |      (e.g., JSON Msgs)       |                                      |
+------------------+                              |  +---------------------------------+ |
                                                  |  | 1. Kafka Consumer               | |
                                                  |  |    (e.g., segmentio/kafka-go)   | |
                                                  |  +---------------------------------+ |
                                                  |                  |                   |
                                                  |  +---------------------------------+ |      +-----------------+
                                                  |  | 2. Message Parser/Validator     |----->|  Configuration  |
                                                  |  +---------------------------------+ |      | (YAML/Env Vars) |
                                                  |                  |                   |      +-----------------+
                                                  |  +---------------------------------+ |
                                                  |  | 3. Stats Calculator (Windowed)  | |
                                                  |  |    - Null Rate                   | |
                                                  |  |    - Mean/Variance               | |
                                                  |  +---------------------------------+ |
                                                  |                  |                   |
                                                  |  +---------------------------------+ |
                                                  |  | 4. Threshold Checker & Alerter  | |
                                                  |  +---------------------------------+ |
                                                  +--------------------------------------+

Key Components:

Kafka Consumer:
- Utilizes a robust Go Kafka client library (e.g., segmentio/kafka-go or confluent-kafka-go).
- Connects to specified Kafka brokers and consumes messages efficiently from target topic(s).
- Handles partitioning and consumer group offset management automatically.
Message Parser/Validator:
- Decodes incoming messages (MVP assumes JSON format) into a predefined Go struct.
- Performs basic validation to ensure required fields for monitoring are present and have expected basic types.
Stats Calculator (Windowed):
- The core engine responsible for real-time aggregation.
- Calculates essential metrics (Null Rate, Mean, Variance for numerical features) over configurable * time windows* (e.g., 1-minute tumbling windows).
- Manages internal state efficiently for each monitored feature across active time windows.
Threshold Checker & Alerter:
- Compares the calculated statistics from each completed window against the user-defined thresholds in the configuration.
- If a threshold is breached, it generates a structured alert message.
- For the MVP, alerts and periodic statistics summaries are logged to standard output (stdout).
Configuration:
- Loads application settings using a flexible library like viper.
- Supports loading from a configuration file (e.g., config.yaml) and/or environment variables.
- Settings include: Kafka connection details, topic names, features to monitor, window parameters (size, type), and alert thresholds for each metric.

🚀 Getting Started

There are two main ways to run FeatureLens: connecting to your existing Kafka cluster or running everything locally using Docker Compose for quick testing.

Option 1: Connecting to Your Existing Kafka

Prerequisites:
- Go (Version 1.19+ recommended)
- Git
- Access to your Apache Kafka cluster.

Clone & Build:

git clone https://github.com/<your-username>/featurelens.git # Replace with actual repo path
cd featurelens
go build ./cmd/featurelens

Configure:
- Copy the example configuration: cp configs/config.example.yaml config.yaml
- Edit config.yaml:
  - Set kafka.brokers to your Kafka broker addresses (e.g., ["kafka1:9092", "kafka2:9092"]).
  - Set kafka.topic to the topic you want to monitor.
  - Configure the features section with the fields to monitor and their thresholds.
Run:
```
./featurelens -config config.yaml
```

Option 2: Running Locally with Docker Compose (Quick Test)

This option uses Docker Compose to quickly set up a local Kafka cluster and a web UI (AKHQ) for easy experimentation.

Prerequisites:
- Go (Version 1.19+ recommended)
- Git
- Docker & Docker Compose

Clone & Set up Environment:

git clone https://github.com/<your-username>/featurelens.git # Replace with actual repo path
cd featurelens
docker-compose up -d # Starts Kafka & AKHQ UI in background

*   Wait a moment for services to start. AKHQ UI will be available at `http://localhost:8080`.

Create Kafka Topic:

FeatureLens needs a topic to monitor. Create it via the AKHQ UI (http://localhost:8080 -> Topics -> Create Topic) or using the command line:

# Use the default topic name 'feature-stream' or change as needed
docker-compose exec kafka kafka-topics --create --topic feature-stream --bootstrap-server kafka:9093 --partitions 1 --replication-factor 1

Build FeatureLens (and Optional Producer):

go build ./cmd/featurelens
go build ./cmd/producer # Builds the sample data producer

Run:
- (Optional) Start Sample Producer: This sends sample data to the feature-stream topic.
```
./producer
```
  (Press Ctrl+C to stop)
- Run FeatureLens: Use the development config which points to local Kafka.
```
./featurelens -config configs/config.dev.yaml # Ensure this config exists and points to localhost:9092
```
Stop Environment:
```
docker-compose down
```

See CONTRIBUTING.md for a more detailed development setup guide, including pre-commit hooks.

🗺️ Roadmap

(Future enhancements and plans beyond the MVP.)

Enhanced Statistics: Support for Min/Max, Median, Quantiles, Cardinality (for categorical features).
Drift Detection Algorithms: Implement statistical tests like Kolmogorov-Smirnov (KS test), Population Stability Index (PSI).
Data Formats: Support for Protobuf, Avro alongside JSON.
Alerting Integrations: Send alerts to Slack, PagerDuty, OpsGenie, Alertmanager.
Metrics Export: Expose internal metrics and calculated statistics via a Prometheus /metrics endpoint for monitoring and visualization (e.g., with Grafana).
Data Source Flexibility: Add support for consuming from other sources like AWS Kinesis, Google Pub/Sub, or Pulsar.
State Management: Implement more robust state management for windowing, potentially using external stores for fault tolerance.
Advanced Windowing: Support for sliding windows, session windows.
(Optional) Web UI: A simple interface for configuration, viewing status, and recent alerts.