DEV Community

poolion
poolion

Posted on

Log Grep: Stream Regex Search for Large Logs Without Loading Entire File

#python #opensource #productivity #tool

Log Grep: Stream Regex Search for Large Logs Without Loading Entire File Into Memory

When working with large log files—common in production server monitoring, CI/CD pipelines, or application debugging—the standard grep command can cause memory issues by loading the entire file into RAM.

Log Grep is a lightweight Python CLI tool that uses streaming mode to handle gigabyte-scale log files efficiently, reading line-by-line without loading everything into memory at once.

What Problem Does It Solve?

Large log files (GBs in size) are common in:

  • Cloud application monitoring
  • Web server access logs
  • Container/container orchestration debugging
  • CI/CD pipeline output

Standard tools like grep or Perl regular expressions often read the entire file into memory, causing out-of-memory (OOM) errors on constrained systems. Log Grep uses Python's streaming file iterator pattern to handle arbitrarily large files using constant memory.

Installation 

# Clone and run directly
git clone https://github.com/Poolion/log-grep.git
cd log-grep
python3 log-grep.py "ERROR" bigfile.log
Enter fullscreen mode Exit fullscreen mode

Or add to your PATH:

cp log-grep.py /usr/local/bin/
chmod +x /usr/local/bin/log-grep.py
log-grep.py "pattern" logfile.log
Enter fullscreen mode Exit fullscreen mode

Usage Examples

Basic Pattern Search 

python3 log-grep.py "ERROR" application.log
# Searches for lines containing "ERROR"
# Shows first 10 matches with line numbers (default)
Enter fullscreen mode Exit fullscreen mode

Limit to Specific Matches 

python3 log-grep.py "connection.*timeout" -c 5
# Only show the first 5 matches
Enter fullscreen mode Exit fullscreen mode

Show All Matches 

python3 log-grep.py "^2024-01-" logfile.log --count 0
# Shows all matching lines (no line count limit)
Enter fullscreen mode Exit fullscreen mode

Suppress Line Numbers 

python3 log-grep.py "failed.*login" --no-line-numbers
# Clean output without line numbers
Enter fullscreen mode Exit fullscreen mode

Pipe from Another Command 

ls -la /var/log/*.log | python3 log-grep.py "ERROR"
# Works with stdin via "-f" flag
Enter fullscreen mode Exit fullscreen mode

Technical Details

The tool uses Python's built-in file iteration, which keeps the file descriptor open and reads one line at a time using buffered I/O. This pattern is memory-efficient:

with open(file_path) as f:
    for line_num, line in enumerate(f, 1):
        # Process each line individually
Enter fullscreen mode Exit fullscreen mode

This is different from loading everything at once with file.readlines() or Python's regex module's non-streaming modes.

The -c (count) parameter limits output to prevent excessive console flooding when searching common patterns in large files.

Pattern Support

Log Grep uses re.search() for pattern matching, supporting:

  • Regular expressions ("ERROR.*timeout")
  • Character classes ("[0-9]{3}")
  • Anchors ("^INFO" or "error$")
  • Quantifiers ("fail[ur]" matches 'fail' and 'fair')

When to Use

Use Log Grep when:

  • Files exceed available RAM
  • Working with high-volume logging infrastructure
  • Need controlled output (count limits)
  • Processing logs in constrained containers/VPS

Alternatives

Standard grep works for small files under 1GB. For larger files, consider:

  • less +g pattern file (GNU grep's stream mode)
  • This tool provides better count control and line number options

Conclusion

Log Grep provides a simple, efficient solution for pattern searching large log files without memory concerns. It's perfect for system administrators, DevOps engineers, and developers debugging production systems.

Repo: https://github.com/Poolion/log-grep

If you find this useful, you can support development: https://www.buymeacoffee.com/poolion

Top comments (0)