DEV Community: Jan

Hybrid Chatbots and REST API: Custom Development vs. Available Solutions

Jan — Thu, 13 Mar 2025 10:07:43 +0000

Introduction
In today's fast-paced technological landscape, hybrid chatbots are becoming increasingly important tools for communication and automation. These systems combine small language models with vector databases, enabling efficient processing and retrieval of information. A crucial element for successful integration of these components is REST API, which acts as an intermediary between:

Language models (e.g., LLaMA 2, Mistral) that process natural language and generate responses.
Vector databases (e.g., Pinecone, Weaviate, Qdrant) that store and retrieve data based on vector representations.
Chatbot frontend (web or mobile applications) that facilitate user interaction.
The REST API receives user queries, routes them to the appropriate components, retrieves relevant information, and returns it to the user. This process ensures smooth and effective communication within the system.

Options: Custom REST API vs. Available Solutions
Custom REST API Development
Developing a custom REST API offers several advantages that can be crucial for complex or specific projects:

Customizability: A custom API can be tailored precisely to the needs of your system, allowing optimization for specific use cases and requirements.
Security: Full control over data and its processing enables the implementation of custom security measures to protect sensitive information.
Flexibility: Easier integration with other services or specific features that may be critical for your application.
Performance Optimization: A custom API can be optimized for maximum performance and efficiency, which is important for demanding applications.
Disadvantages:

Time-Consuming: Developing and testing a custom API can be time-consuming and resource-intensive.
Maintenance Costs: Regular updates and code management can be financially and time-consuming.
Using Available APIs
Using existing APIs, such as OpenAI API, Hugging Face Inference API, or services from Pinecone, can be a suitable solution for projects that require rapid deployment:

Quick Deployment: Existing APIs are ready to use almost immediately, speeding up development and deployment of the application.
Reduced Technical Burden: No need to manage backend infrastructure, reducing technical overhead and allowing focus on the core application.
Support and Documentation: Detailed documentation and support from providers facilitate integration and problem-solving.
Disadvantages:

Dependency on External Providers: Your solution is tied to the availability and terms of the API, which can be a risk in case of changes or service outages.
Cost: Some APIs can be expensive, especially with higher query volumes, increasing operational costs.
Limited Customization: Functionality is determined by the API provider, which may limit customization options.
Decision Factors
The decision between custom development and using existing APIs depends on several key factors:

Project Size: For smaller projects or prototypes, it is more suitable to use existing APIs that allow for quick deployment and reduced initial costs.
Budget and Resources: If financial or technical resources are limited, it may be better to utilize available solutions.
Need for Control and Flexibility: For larger projects that require a high degree of control and flexibility, it is more suitable to invest in a custom REST API.
Long-Term Goals: If the goal is to reduce costs and gain full control over the system in the long term, it is worth considering developing a custom API.
Conclusion: Tailored Hybrid Chatbots
By combining small language models, vector databases, and a well-integrated REST API, a robust and efficient system is created that can handle current communication and data processing demands. The hybrid approach allows achieving an optimal balance between performance, cost, and flexibility, which is crucial for successful chatbot deployment in real-world environments.

If you are planning to implement your own chatbot, consider all aspects and decide based on the specific needs of your project.

How to Try It Out
If you want to experiment with creating a hybrid chatbot, you can follow this basic guide:

Choose a Language Model:

Start with a pre-trained model like LLaMA 2 or Mistral, available through platforms like Hugging Face.
Set Up a Vector Database:

Choose a vector database like Pinecone, Weaviate, or Qdrant and integrate it into your system.
Create a REST API:

Decide whether to use an existing API or develop your own. For a start, you can use OpenAI API or Hugging Face Inference API.
If developing your own API, use frameworks like Flask or FastAPI for quick deployment.
Integrate with the Frontend:

Create a simple frontend for the chatbot using technologies like React or Vue.js.
Connect the frontend with the REST API for communication with the language model and vector database.
Testing and Optimization:

Test your application with various user scenarios and optimize performance and response accuracy.
This guide provides basic directions for creating a hybrid chatbot. If you need more details or assistance, feel free to reach out!

Dynamic throttle

Jan — Fri, 17 Jan 2025 08:39:01 +0000

Imagine this scenario: You have a router, switch, or wireless gateway that goes into a sort of “sleep mode” after a period of inactivity. When you send the first batch of real data, there’s a small delay because the network is “waking up.” Or your VPN tunnel loses its “steam” over time, and the next connection suffers unnecessary lag.

In this article, we’ll explore a punk-style solution called “dynamic throttle.” The idea is simple:

Monitor the network state (latency, and optionally other indicators).
When the network starts slowing down (latency rising, low real traffic), inject a bit of “artificial” traffic: **dummy (empty) packets.
Once the network picks back up with real traffic, we reduce dummy traffic to minimize overhead.

The goal is to keep the network active* and avoid “sleepy” states in routers, NAT tables, VPN tunnels, and other elements that tend to idle when there’s no data flowing.

Where Can This Help?

Modern Wi-Fi* (e.g., Wi-Fi 6, 6E): These devices can cleverly save power, but occasional traffic might cause extra latency when the radio module reactivates from dozing. Sending a few dummy packets could reduce that wake-up delay.

Mobile and 5G Networks: The same phenomenon can happen — a device may transition to a lower power state (RRC_IDLE) and needs reactivation time. A dynamic throttle might help keep the connection in a more active state if latency starts spiking.

VPN Tunnels (OpenVPN, WireGuard, etc.): If you have prolonged inactivity, the router, NAT, or even the VPN daemon may consider the link dormant. Then the first real packet gets delayed while everything re-initializes.

Edge Computing: Many gateways gather sensor data and send it to the cloud in bursts. A dynamic throttle can maintain the upstream link in a ready state, preventing brief dropouts or delays when establishing a connection.

How It Works: A Simple Outline

Measure latency (and optionally throughput, packet loss, etc.)
- For example, run a small ping every few seconds to a target (DNS server, cloud endpoint, VPN gateway).
Decision Logic
- If latency is low, keep dummy traffic to a minimum.
- If latency/jitter increases and there’s little real traffic, crank up dummy packets to “revive” the link.
Send Dummy Packets
- Usually small UDP packets (50–200 bytes). Send them at some rate (e.g., 2–10 per second), dynamically adjusted according to network conditions.
Check and Repeat
- Once real traffic flows again, reduce the dummy packets so you’re not wasting bandwidth or power.
- Periodically re-evaluate.

A Functional Python Example (Less Theoretical)

Below is a working (though not production-ready) example of how you can implement a “dynamic throttle” on Linux (or any OS with ping and socket support). This script:

Periodically pings (PING_TARGET) to measure latency.
Optionally monitors real traffic via tc command (Traffic Control) — this is a simplified demonstration of how you might read outgoing traffic on an interface.
Dynamically adjusts the amount of dummy packets (UDP) sent to a given target.

Note: The tc part is optional and might require elevated privileges (sudo). If you don’t need it, you can remove that logic.

``python
import subprocess
import re
import socket
import time
import math

-----------------------------

CONFIG

-----------------------------

PING_TARGET = "8.8.8.8" # Where we ping to measure latency
DUMMY_TARGET = ("8.8.8.8", 9999) # Where we send dummy packets
NETWORK_INTERFACE = "eth0" # (Optional) Interface to measure real traffic
PACKET_SIZE = 64 # Size of dummy packets in bytes
PING_INTERVAL = 2.0 # Interval between pings (s)
REGULATION_INTERVAL = 2.0 # Interval for adjusting dummy traffic (s)
LOW_LATENCY_THRESHOLD = 50.0 # (ms)
HIGH_LATENCY_THRESHOLD = 120.0 # (ms)
MAX_DUMMY_PACKETS_PER_SEC = 10 # Max number of dummy packets per second

If ping fails, we assume a default (high) latency

DEFAULT_LATENCY = 999.0

Global variable for the number of dummy packets per second

dummy_packets_per_sec = 0

-----------------------------

FUNCTIONS

-----------------------------

def measure_latency(target: str) -> float:
"""
Measure latency to the target using one ping.
Returns the average RTT in milliseconds (float).
If measurement fails, returns DEFAULT_LATENCY.
"""
try:
output = subprocess.check_output(
["ping", "-c", "1", "-W", "1", target],
stderr=subprocess.STDOUT,
text=True
)
match = re.search(r"rtt min/avg/max/mdev = [\d.]+/([\d.]+)/", output)
if match:
return float(match.group(1))
except subprocess.CalledProcessError:
pass
return DEFAULT_LATENCY

def measure_real_traffic(iface: str) -> float:
"""
(Optional) function to measure outgoing real traffic (in Mb/s).
In reality, you might parse /proc/net/dev, bmon, or ifstat, etc.
Here, we do a simplified example with tc -s qdisc show dev ....
NOTE: This requires sudo and is very naive for demonstration.
"""
try:
output = subprocess.check_output(
["tc", "-s", "qdisc", "show", "dev", iface],
stderr=subprocess.STDOUT,
text=True
)
# Super simplified parsing - we look for 'Sent x bytes ...'
match = re.search(r"Sent\s+(\d+)\s+bytes", output)
if match:
sent_bytes = float(match.group(1))
# Real measurement would store the previous value, compare time deltas, etc.
# We'll just return a "fake" value
return sent_bytes / 1e6 # pseudo MB => not exactly Mb/s
except subprocess.CalledProcessError:
pass
return 0.0

def regulation_logic(latency: float, real_traffic_mbps: float) -> int:
"""
Decide how many dummy packets to send:
- If latency < LOW_LATENCY_THRESHOLD, we minimize dummy packets.
If real traffic is flowing, we hardly need any dummy packets.
- If latency > LOW_LATENCY_THRESHOLD and real traffic is low,
we increase dummy packets to "wake the link up."
- Above HIGH_LATENCY_THRESHOLD, we cut dummy packets so we don't make congestion worse.
"""
# Rough heuristic:
# latency < 50 ms => 0 to 2 p/s
# latency 50-120 ms => 2 to 7 p/s, depending on real traffic
# latency > 120 ms => 0 p/s (network is overloaded)
if latency >= HIGH_LATENCY_THRESHOLD:
return 0 # Network is congested, don't add dummy traffic

if latency <= LOW_LATENCY_THRESHOLD:
    # Network is OK
    if real_traffic_mbps > 0.5:
        # Enough real traffic is flowing
        return 0
    else:
        return 2
else:
    # Medium latency
    if real_traffic_mbps > 0.5:
        # Some real load, don't overdo dummy
        return 2
    else:
        # Very little real traffic, but latency is rising => add more dummy
        return 5

def send_dummy_packet(sock: socket.socket, size=64):
"""
Send one dummy UDP packet of given size.
"""
data = b"x" * size
sock.sendto(data, DUMMY_TARGET)

-----------------------------

MAIN LOOP

-----------------------------

def main():
global dummy_packets_per_sec

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setblocking(False)

last_ping_time = time.time()
last_regulation_time = time.time()
last_dummy_send_time = time.time()

interval_between_dummy = 1.0

while True:
    now = time.time()

    # Ping every PING_INTERVAL seconds
    if now - last_ping_time >= PING_INTERVAL:
        last_ping_time = now
        latency = measure_latency(PING_TARGET)
        print(f"[PING] Latency = {latency:.1f} ms")

    # Adjust dummy traffic every REGULATION_INTERVAL seconds
    if now - last_regulation_time >= REGULATION_INTERVAL:
        last_regulation_time = now

        real_traffic_mbps = measure_real_traffic(NETWORK_INTERFACE)
        current_latency = measure_latency(PING_TARGET)

        dummy_packets_per_sec = regulation_logic(current_latency, real_traffic_mbps)
        dummy_packets_per_sec = min(dummy_packets_per_sec, MAX_DUMMY_PACKETS_PER_SEC)

        if dummy_packets_per_sec > 0:
            interval_between_dummy = 1.0 / dummy_packets_per_sec
        else:
            interval_between_dummy = 9999.0  # effectively off

        print(f"[REGULATION] Setting dummy p/s to: {dummy_packets_per_sec}, "
              f"real traffic ~ {real_traffic_mbps:.2f} Mb/s")

    # Send dummy packets spaced out in time
    if dummy_packets_per_sec > 0:
        if now - last_dummy_send_time >= interval_between_dummy:
            last_dummy_send_time = now
            send_dummy_packet(sock, PACKET_SIZE)

    time.sleep(0.01)

if name == "main":
main()
``

Implementation Notes

measure_latency: Classic ping. If it fails, we set the latency to DEFAULT_LATENCY.
measure_real_traffic: A demonstration of how you might detect whether real traffic is flowing. In a real implementation, you’d parse byte counters (e.g., from /proc/net/dev), track deltas over time, and calculate actual Mbps.
regulation_logic: A simple heuristic. If latency is good, we minimize dummy. If latency goes up and real traffic is low, we add more dummy. Above a high latency threshold, we stop sending dummy to avoid making congestion worse.
Main Loop:
- Periodically pings at PING_INTERVAL.
- Periodically recalculates (every REGULATION_INTERVAL) and adjusts dummy packets.
- Sends dummy packets at the computed interval, so we don’t create big bursts.

Note: This is still a proof-of-concept, but it’s already functional. If you run this on Linux (with tc installed), you’ll see it react to latency and (roughly) to real network traffic.

Use cases for modern devices

Edge AI / IoT Gateways
- Running on a Raspberry Pi or NUC, sending data to the cloud sporadically. A dynamic throttle helps keep your ISP router or local switch from dropping into idle.
Mobile Hotspots
- If your device is operating as a hotspot with LTE/5G, a dynamic throttle might reduce response time. But beware of extra battery drain.
VPN (OpenVPN/WireGuard)
- Point DUMMY_TARGET to the other end of the tunnel, generating keep-alive traffic that keeps NAT/firewalls from dropping the VPN session.
Modern Wi-Fi Mesh
- Small dummy keep-alive packets can help ensure mesh nodes don’t discard routing information during low-traffic periods.

Conclusion

Dynamic throttle is an unconventional (some might say “punk”) technique that uses adaptive dummy packets to keep network traffic slightly active. In modern devices (with various power-saving and QoS features), it may help in scenarios where you dislike “startup delays,” and you’re okay with a slight overhead in bandwidth or power.

Key is dynamism: you only increase dummy traffic when latency is rising and real traffic is idle. Otherwise, you scale dummy down to minimize waste.

In practice, always measure carefully — sometimes it helps; sometimes it’s negligible.

The code above is a real example of how you might start on Linux. You could adapt it, add more refined metrics, or integrate a more sophisticated control algorithm (e.g., PID or eBPF-based solutions).

If you try this, have fun experimenting! Let us know if it actually improves anything in your setup — or if you hit constraints that make it purely theoretical. Even negative results can be valuable, as they push our engineering understanding further.

Tip: Want to take it further? Check out PID controllers or eBPF. You can dynamically respond to latency in even more precise ways in rea