DEV Community

IntelliTools
IntelliTools

Posted on

Fix Malformed JSON or CSV Files Without APIs or Cloud Services

#python #automation #datavalidation #cli

When working with data, malformed files can bring your workflow to a halt. Whether it's a JSON file with broken syntax or a CSV that's missing headers, these errors can be frustrating to debug. Most developers rely on APIs or cloud services to validate data, but those approaches often require signing up, managing keys, or dealing with rate limits. What if you could validate and clean your data with just a command-line tool and no external dependencies?

Enter TextFileValidator, a self-contained Python CLI that handles these tasks with minimal setup. It’s designed for developers who need to process data files without the overhead of cloud infrastructure or API calls. Let’s walk through how to use it and how it can help you avoid common pitfalls in data validation.

Why You Need a CLI for Data Validation

Data validation is a common pain point in automation pipelines. When you're processing files, you often need to ensure they're well-formed before moving forward. For example, a JSON file that’s missing a closing bracket can cause your script to crash, and a CSV with inconsistent delimiters can bloat your data processing logic.

The key benefit of a CLI like TextFileValidator is that it’s lightweight and doesn’t require any external services. You can run it directly from your terminal, process the file, and get structured output in seconds. This is especially useful in environments where you can’t use cloud services or APIs, or when you want to keep your data processing local.

How to Use TextFileValidator

Let’s look at a simple example. Suppose you have a malformed JSON file named data.json that looks like this:

{
  "name": "Alice",
  "age": 30,
  "hobbies": ["reading", "coding"]
}
Enter fullscreen mode Exit fullscreen mode

Wait, that’s actually valid. Let’s make it malformed by removing the closing brace:

{
  "name": "Alice",
  "age": 30,
  "hobbies": ["reading", "coding"]
Enter fullscreen mode Exit fullscreen mode

Now, run the TextFileValidator tool on this file:

python textfilevalidator.py --input data.json --output results.json --verbose
Enter fullscreen mode Exit fullscreen mode

This command tells the tool to:

  • Read the input file (data.json)
  • Write the output to results.json
  • Print detailed progress to the console

The tool will process the file, detect the malformed JSON, and write a structured output file that you can use in your pipeline.

Real-World Use Case: Cleaning CSV Files

CSV files are notorious for having inconsistent formats. For example, a file might have missing headers or incorrect delimiters. Let’s say you have a CSV file named users.csv that looks like this:

name,age,city
John,30,New York
Jane,25,Los Angeles
Enter fullscreen mode Exit fullscreen mode

This is valid, but what if the file has a missing header row? Let’s simulate that:

John,30,New York
Jane,25,Los Angeles
Enter fullscreen mode Exit fullscreen mode

Now, run the tool with the --csv flag:

python textfilevalidator.py --input users.csv --output results.csv --csv --verbose
Enter fullscreen mode Exit fullscreen mode

The tool will detect the missing header and attempt to infer it from the first row. It will also handle cases where the delimiter is not a comma, like tabs or semicolons.

Teaching a Real Technique: Data Validation in Pipelines

One of the most valuable techniques you can learn is how to integrate data validation into your automation pipelines. Validating data early in the process ensures that downstream tasks don’t fail due to malformed inputs.

Here’s a simple Python script that uses TextFileValidator as part of a data processing pipeline:

import subprocess
import json

def validate_file(input_path, output_path, verbose=False):
    cmd = [
        "python", "textfilevalidator.py",
        "--input", input_path,
        "--output", output_path
    ]
    if verbose:
        cmd.append("--verbose")

    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        print("Validation failed:", result.stderr)
        return False
    print("Validation successful:", result.stdout)
    return True

# Example usage
validate_file("data.json", "results.json", verbose=True)
Enter fullscreen mode Exit fullscreen mode

This script uses the subprocess module to run the CLI tool and capture its output. It’s a great way to integrate data validation into your Python scripts, especially if you're working in an environment where you can’t use external libraries.

Final Thoughts

Data validation is a critical part of any data pipeline, and tools like TextFileValidator make it easier than ever to handle malformed files. With its CLI interface and minimal setup, you can validate and clean your data without relying on cloud services or APIs.

If you're looking for a self-contained, no-frills way to validate JSON or CSV files, you can find the tool here: https://intellitools.gumroad.com/l/textfilevalidator.

Top comments (1)

Collapse
 
topstar_ai profile image
Luis Cruz •

I appreciate how the article highlights the importance of data validation in automation pipelines and introduces TextFileValidator as a lightweight solution. The example use cases, such as validating malformed JSON and CSV files, demonstrate the tool's effectiveness in handling common data format issues. One potential improvement could be to explore integration with popular data processing frameworks, like Pandas or Apache Beam, to further streamline data validation in larger workflows. Have you considered adding support for other file formats, like XML or Avro, to TextFileValidator?