From Unstructured Logs to Clean JSON: A Free-Model ETL Case Study

Three months ago I inherited a log file from an old Node.js application. It was 500 lines of mixed formats: some lines had bracketed timestamps, some had ISO 8601 dates, some had no timestamps at all. I needed structured JSON for a simple analytics query. This article is a walkthrough of the normalization pipeline I built using MonkeyCode's free model access and a free server option, including the failure rates and the design decisions that mattered.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem: three formats in one file

The log file looked like this:

[2026-05-12 08:14:22] INFO User 42 logged in
2026-05-12T08:15:01Z ERROR user=7 Failed to connect to database
WARN timeout after 3000ms user=13

Three different formats. No consistent delimiter. No structured logging in the original app. A single regex to handle all of them would have been fragile, and maintaining it would have been worse.

The design: regex first, model as fallback

The pipeline had three stages:

1. Regex pass — handle the common formats with patterns.
2. Model pass — send unmatched lines to a free coding model for extraction.
3. Validation pass — verify every parsed record has the required fields.

The key insight: do not call the model for lines a regex can handle. Models are slow and non-deterministic. Regex is fast and predictable. Use the model only for the long tail.

Implementation

The regex stage is straightforward:

import re
import json
import httpx
import os

PATTERNS = [
    re.compile(
        r"^\[(?P<timestamp>.*?)\] (?P<level>INFO|ERROR|WARN) "
        r"(?:User (?P<user_id>\d+)|user=(?P<user_id2>\w+)) (?P<message>.*)$"
    ),
    re.compile(
        r"^(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) "
        r"(?P<level>INFO|ERROR|WARN) user=(?P<user_id>\w+) (?P<message>.*)$"
    ),
]

def parse_with_regex(line):
    for pattern in PATTERNS:
        match = pattern.match(line)
        if not match:
            continue
        groups = match.groupdict()
        return {
            "timestamp": groups["timestamp"],
            "level": groups["level"],
            "user_id": groups.get("user_id") or groups.get("user_id2"),
            "message": groups["message"],
        }
    return None

The model stage sends the raw line and asks for JSON:

def parse_with_model(line):
    # Pseudocode: check the current MonkeyCode docs for the exact endpoint.
    response = httpx.post(
        os.environ["MONKEYCODE_ENDPOINT"],
        headers={"Authorization": f"Bearer {os.environ['MONKEYCODE_TOKEN']}"},
        json={
            "model": "free-model",
            "messages": [{
                "role": "user",
                "content": (
                    "Extract timestamp, level, user_id, and message from this log line. "
                    "Return valid JSON with exactly these four keys. "
                    "If a field is missing, use null.\n\n" + line
                )
            }],
        },
        timeout=30,
    )
    data = response.json()
    content = data["choices"][0]["message"]["content"]
    return json.loads(content)

The validation stage is where most of the real work happens. The model will happily invent fields or return a timestamp format you did not ask for:

from datetime import datetime

def validate(record):
    required = {"timestamp", "level", "user_id", "message"}
    if not required.issubset(record):
        return False, f"missing fields: {required - record.keys()}"
    if record["level"] not in {"INFO", "ERROR", "WARN", "DEBUG"}:
        return False, f"bad level: {record['level']}"
    try:
        datetime.fromisoformat(record["timestamp"].replace("Z", "+00:00"))
    except ValueError:
        return False, f"bad timestamp: {record['timestamp']}"
    return True, "ok"

Results: 500 lines, three buckets

After running the pipeline, every line fell into one of three buckets:

Bucket	Lines	Percentage
Parsed by regex	387	77.4%
Parsed by model	89	17.8%
Failed validation	24	4.8%

The 24 failures were not random. They clustered into two groups: lines with nested JSON in the message field, and lines where the user ID was a UUID instead of a number. The model consistently confused the UUID with the message text.

The fix was not a better prompt. It was a better regex for the UUID case, which moved 19 of the 24 failures into the regex bucket. The remaining 5 lines were genuinely ambiguous, so I labeled them manually.

These numbers are from my specific run. Your log formats and model behavior will differ.

Token usage and cost

The model pass processed 113 lines (89 successes plus 24 failures). Each call used roughly 150 tokens for the prompt and 50 for the response. Total: about 22,600 tokens for the whole job.

That is a small number, but it exposes a scaling problem: if the file had been 50,000 lines instead of 500, the model pass would have consumed over 2 million tokens. The pipeline only works because the regex stage absorbs most of the volume.

What the parsing data taught me

1. Regex is a feature, not a workaround. A good pattern handled 77% of the input at zero token cost. Write the regex first and let the model handle the tail.
2. Validate everything the model returns. The model invented a user_id of "unknown" in 11 records even though I asked for null. Validation caught every case.
3. Keep the raw line in the output. Every record I stored includes the original line. When a downstream query looks wrong, you can trace it back to the source.
4. Failure clusters are informative. The 24 failures were not random noise. They pointed to specific format gaps that a targeted regex could close.

Where this pipeline breaks

This is a pragmatic hack, not a production ETL system. The free model can hallucinate fields, the free server has no uptime guarantee, and the validation layer only catches the failures I thought to check for.

Do not use this approach for:

• Logs containing personal data, credentials, or regulated information.
• High-volume pipelines where even a 5% model failure rate means thousands of bad records.
• Teams that need a formal schema contract with versioned transformations.

For a one-off data cleanup job, the cost is hard to beat: zero dollars, one afternoon, and a JSON file you can query.

If you have a messy data file sitting around, this pattern is worth trying. MonkeyCode's free model access and free server option can handle the long tail that regex cannot. Read the current docs for the exact limits, and keep your validation layer strict.