I Spent Hours Parsing Logs Until I Tried This AI Approach

Let me set the scene: I had just inherited a massive collection of log files from a dozen microservices. Each team had invented its own logging format—some used JSON, some used plain text with timestamps in weird places, and one service insisted on logging in haiku (okay, not really, but close). My task was to extract all error messages, their timestamps, and the source service, then produce a unified timeline.

I spent three days writing regex patterns, awk one-liners, and custom Python parsers. Every time I thought I was done, a new edge case appeared. A timestamp would be in ISO format instead of Unix epoch. An error message would span multiple lines. The frustration was real.

What I Tried (and Failed)

• Regex everything: I wrote patterns for each known format. Then a new format showed up. Maintenance nightmare.
• Generic parsers: I attempted to write a flexible parser that could infer structure. That quickly turned into a PhD thesis on log parsing.
• Manual review: Not scalable, and my eyes started bleeding.

None of these approaches worked because the variety was infinite. I needed something that understood meaning, not just formatting.

The Approach That Actually Worked: LLM-Based Extraction

I realized I didn't need to parse logs—I needed to interpret them. Large Language Models (LLMs) are great at understanding text and extracting structured information. So I built a small script that sends each log line (or batch of lines) to an AI API and asks it to return a JSON object with the fields I cared about.

Here’s the core technique: define a simple schema, send the raw log text, and let the model do the heavy lifting.

Code Example (Python)

import openai
import json

# Set up your client - replace with your own endpoint
client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://ai.interwestinfo.com"  # example endpoint
)

def extract_structured_info(log_line: str):
    response = client.chat.completions.create(
        model="gpt-4o-mini",  # or any capable model
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a log parser. Extract fields: timestamp (ISO 8601), "
                    "severity (DEBUG, INFO, WARN, ERROR, FATAL), service_name, "
                    "message (string). Return valid JSON only."
                )
            },
            {
                "role": "user",
                "content": f"Parse this log line: {log_line}"
            }
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    return json.loads(response.choices[0].message.content)

# Example usage
log = "[2024-03-15 10:30:45] [ERROR] [payment-service] Failed to charge card: timeout"
parsed = extract_structured_info(log)
print(parsed)
# Output: {'timestamp': '2024-03-15T10:30:45', 'severity': 'ERROR', 'service_name': 'payment-service', 'message': 'Failed to charge card: timeout'}

This works even for the weirdest formats. The model understands context and can extract timestamps regardless of format.

Batching for Performance

To handle thousands of logs, I batch them:

def batch_extract(log_lines: list[str], batch_size=50):
    results = []
    for i in range(0, len(log_lines), batch_size):
        batch = log_lines[i:i+batch_size]
        prompt = "\n---\n".join(f"Line {j}: {line}" for j, line in enumerate(batch, start=i))
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Extract structured data for each line. Return a JSON array of objects."},
                {"role": "user", "content": prompt}
            ],
            response_format={"type": "json_object"},
            temperature=0
        )
        results.extend(json.loads(response.choices[0].message.content))
    return results

Lessons Learned & Trade-offs

This approach is not a silver bullet. Here’s what I discovered:

• Cost: Each call costs money. For high-volume logs (millions of lines/day), this becomes expensive. I used a cheaper model (like gpt-4o-mini) and batching to reduce cost.
• Latency: Synchronous calls are slow. For near-real-time needs, consider async or streaming.
• Hallucination: Sometimes the model invents fields or misinterprets ambiguous text. I added post-processing validation with Pydantic.
• Context window: Very long log lines (e.g., stack traces) can exceed token limits. I truncate or split them.
• Accuracy: It’s not 100%. For critical systems, you might need a hybrid approach: use AI for fuzzy fields and regex for strict ones.

When NOT to use this

• You have highly structured, predictable logs (e.g., always JSON). Regex is faster and cheaper.
• You need real-time processing at high throughput (network overhead kills performance).
• Sensitive data: sending logs to an external API may violate compliance. Use a local model (Llama, Mistral, etc.) to keep data in-house.

What I'd Do Differently Next Time

First, I’d start with a small sample and validate the schema before processing everything. Second, I’d cache parsed results because many logs repeat. Third, I’d try a local model via Ollama to eliminate latency and cost concerns—though accuracy might drop.

Overall, this technique saved me hours of maintenance. Now when a new microservice appears with yet another log format, I don’t panic. The model handles it.

---

What’s your approach for handling messy logs? Have you tried AI or stuck with regex? I’d love to hear what works for you (and what doesn’t).