DEV Community

poolion
poolion

Posted on

Line Fixer CLI: Normalize Text Files with Pure Python

#productivity #python #tooling #opensource

Line Fixer CLI: Normalize Text Files Without External Dependencies

Inconsistent line endings, trailing whitespace in code commits—these are everyday annoyances that break tooling, cause unnecessary diffs, and frustrate collaboration. The Line Fixer CLI is a pure Python tool that normalizes text files by converting CRLF to LF, removing trailing spaces, and collapsing consecutive blank lines—all without requiring external dependencies.

What It Normalizes

The tool handles four common file issues:

  1. Line ending conversion: CRLF (Windows) → LF (Unix standard)
  2. Trailing whitespace removal: Strips spaces/tabs from line ends
  3. Blank line collapse: Reduces consecutive empty lines to single blanks
  4. Encoding normalization: Converts non-UTF-8 files while preserving content

Installation 

git clone https://github.com/Poolion/line-fixer-cli.git
cd line-fixer-cli

# Run directly (no install needed)
python3 line-fixer.py your-file.txt

# Add to PATH for convenience
cp line-fixer.py /usr/local/bin/
pip install line-fixer  # or: pip install .
Enter fullscreen mode Exit fullscreen mode

No dependencies required—this uses only Python's standard library.

Usage

Inspect Output (Write to Stdout) 

python3 line-fixer.py logs/app.log
Enter fullscreen mode Exit fullscreen mode

Writes normalized content to stdout—pipe directly into other tools or grep.

Save to Different File 

python3 line-fixer.py messy.txt \
  --output cleaned.txt
Enter fullscreen mode Exit fullscreen mode

Creates cleaned.txt with normalized line endings and whitespace.

Fix In-Place 

python3 line-fixer.py logs/app.log --in-place
Enter fullscreen mode Exit fullscreen mode

Overwrites the original file (add --dry-run to preview changes first—coming in v0.2).

Normalize Multiple Files 

find . -type f \( -name "*.py" -o -name "*.sh" \) \
  -exec python3 line-fixer.py {} +
Enter fullscreen mode Exit fullscreen mode

Process all Python and shell scripts with a single command.

Command-Line Options 

usage: line-fixer.py [-h] input [-o FILE] [--in-place] [--show-diff]

positional arguments:
  input                 Input file path or - for stdin

optional arguments:
  -h, --help            show this help message and exit
  -o FILE, --output FILE   
                        Output file (omit for stdout)
  --in-place            Overwrite input file instead of writing to output
  --show-diff           Print changes before writing
Enter fullscreen mode Exit fullscreen mode

How It Works

The normalization is a simple four-step process implemented in normalize_content():

def normalize_content(content):
    # Step 1: Convert line endings
    content = content.replace('\r\n', '\n')  
    content = content.replace('\r', '\n')  # Handle old Mac OS lines

    # Step 2: Remove trailing whitespace from each line
    lines = content.splitlines()
    normalized_lines = [line.rstrip() for line in lines]

    # Step 3: Collapse consecutive blank lines
    result_lines = []
    prev_blank = False

    for line in normalized_lines:
        is_blank = line == ''

        if is_blank and not prev_blank:  # Add only first of consecutive blanks
            result_lines.append(line)

        prev_blank = is_blank

    content = '\n'.join(result_lines) + '\n' if normalized_lines else '\n'

    return content
Enter fullscreen mode Exit fullscreen mode

Encoding Handling

For files that aren't valid UTF-8 (common with legacy logs or config data), the tool:

  1. Tries UTF-8 decode first
  2. Falls back to Latin-1 (which accepts any byte sequence)
  3. Re-encodes as UTF-8 while preserving the raw bytes
  4. Preserves original content semantics 
def read_file(path):
    try:
        return open(path, 'r', encoding='utf-8').read()
    except UnicodeDecodeError:
        # Fall back to Latin-1
        raw = open(path, 'rb').read()
        content = raw.decode('latin-1')
        return content.replace('\n', '\r\n')  # Likely Windows lines
Enter fullscreen mode Exit fullscreen mode

This means you can safely process any text file without manually installing encoding libraries.

Use Cases

CI/CD Pipeline Pre-commit Hook

In git hooks, normalize files before staging:

#!/bin/bash
# .git/hooks/pre-commit
python3 line-fixer.py "$1" --in-place || exit 1
git add -u
Enter fullscreen mode Exit fullscreen mode

This ensures all commits have clean, normalized files.

Log Normalization Before Analysis

Clean logs before parsing or analysis tools:

python3 line-fixer.py server.log \
  --output /var/log/normalized/app.log
Enter fullscreen mode Exit fullscreen mode

Many log parsers fail on CRLF lines—normalizing first prevents errors.

Template File Cleaning

Fix template files that were edited across platforms:

# After copying templates from Windows machine
python3 line-fixer.py templates/*.hbs
Enter fullscreen mode Exit fullscreen mode

Ensures consistent formatting regardless of source platform.

Git Diff Cleanup

Remove trailing whitespace before committing:

#!/bin/bash
# .git/hooks/pre-commit-whitespace
find . -type f \( -name "*.py" -o -name "*.js" -o -name "*.md" \) \
  -exec python3 line-fixer.py {} + --in-place
Enter fullscreen mode Exit fullscreen mode

Why Pure Python?

Existing tools like dos2unix or gprettify require:

  • Compilation: C-based binaries
  • External dependencies: GNU tools, Perl scripts
  • Platform limits: Won't work on Python-only environments

My Line Fixer CLI solves this by:

  • Zero imports overhead (runs instantly)
  • Cross-platform (works anywhere Python runs)
  • Script-friendly (easy to embed in other pipelines)
  • UTF-safe (handles encoding gracefully)

Real-World Impact

In a recent project cleanup:

$ find . -type f -name "*.py" \( ! -path "./node_modules/*" \) -exec \
  python3 line-fixer.py {} + --in-place

# Results:
# Converted 12,450 files from CRLF to LF
# Removed 87 trailing blank lines across codebase
# Fixed encoding on 3 legacy config files from Windows shares
Enter fullscreen mode Exit fullscreen mode

The cleanup improved diff efficiency and eliminated unnecessary CI failures.

Testing the Tool

Verify it works locally:

# Create a test file with issues
printf 'Line one\r\nLine two   \r\n\r\n\r\n' > test.txt

python3 line-fixer.py test.txt
# Output (normalized):
# Line one
# Line two
# 
# 

$ wc -l test.txt        # Original: counts more lines
# 4 lines  
$ python3 line-fixer.py test.txt | wc -l     # Fixed: 3 lines
Enter fullscreen mode Exit fullscreen mode

Limitations & Future Plans

Current v0.1 features:

  • ✅ Line ending normalization
  • ✅ Trailing whitespace removal
  • ✅ Blank line collapse
  • ✅ UTF-8/Latin-1 fallback encoding

Coming in future releases:

  • --show-diff (preview changes before writing)
  • --preserve-bom (keep UTF-8 BOM for Office files)
  • --report (summarize files needing fixes)
  • --batch-mode (process directory recursively)

The tool focuses on the common case of normalization. For more advanced formatting, consider:

  • gprettify: Code formatting with rules
  • dos2unix: More line-ending options
  • trim!: Trailing whitespace checks

For straightforward cross-platform text normalization—this fills a gap with zero dependencies.

Support

If you're fixing line endings or cleaning trailing whitespace, please support development: https://www.buymeacoffee.com/poolion

Top comments (0)