DEV Community

poolion
poolion

Posted on

Log Pruner CLI: Find and Clean Oversized Log Files in Python

#python #logging #cli #opensource

Log Pruner CLI: A Minimal Tool for Finding Oversized Logs

Oversized log files clog disk space before rotations fire. This CLI scans directories and flags 10MB+ logs for review or deletion, helps identify which entries still matter before cleanup, and counts candidates to feed automation checks. Uses only Python standard library.

What It Does

  • Size-based flagging: Files >10MB get review, >50MB get delete/rotation
  • Age detection: Report how many days since last modification
  • Quick summary mode: Count candidates without listing all 30 files
  • Deletion help: Generate rm commands for top offenders

Perfect for:

  • Post-build cleanup in CI/CD pipelines
  • Cloud container quota compliance checks (VPS limits)
  • Team directories where logs accumulate without rotation
  • Systems with strict quotas that require proactive management

Usage Examples

Detailed Report 

python log-pruner.py -d /var/log -l 2 -r
Enter fullscreen mode Exit fullscreen mode

Flags oversized or aged logs:

daemon.log                    15.2MB (age=14d) -> review
nginx/access.log              72.4MB (age=90d) -> delete
Enter fullscreen mode Exit fullscreen mode

Color-coded recommendations indicate which files exceed thresholds or need rotation planning. Large files from build artifacts accumulate if pipelines don't clear /tmp before shipping containers or images—this catches them quickly.

Quick Count Only 

python log-pruner.py -c -d /home/app/logs
Enter fullscreen mode Exit fullscreen mode

Feeds dashboards or scripts that enforce quotas without manual intervention:

Total log-type files: 842
>10MB logs: 15
Review candidates (5-10MB): 6
Rotation candidates (>50MB): 2
Enter fullscreen mode Exit fullscreen mode

Count output integrates with CI/CD gates—fail if review candidates exceed policy threshold. When storage quotas tighten before quota expires, automated checks using this count decide whether to alert team or proceed with rotation scripts.

Command Reference

Option Description
-d/--dirs Directory(s) to scan (repeat)
-l/--depth Depth per directory (default=1)
-r/--rotate Flag >50MB for rotation
-c/--count Summary counts only

Multi-Directory Scan

Scan multiple locations:

python log-pruner.py -d /var/log -d /tmp/monitoring
Enter fullscreen mode Exit fullscreen mode

Combines findings across system logs and monitoring agent outputs in single report. Useful for comprehensive audits covering all known log storage paths before compliance reviews or scaling events.

CI/CD Integration Example

After builds fail cleanup, catch lingering files:

python log-pruner.py -c -d /tmp/build-artifacts
# Use count output to trigger alerts when review_candidates > MAX_ALLOWED
Enter fullscreen mode Exit fullscreen mode

Feeds into scripts that decide whether to abort builds or alert for manual attention. When artifacts ship to remote storage and disk quota isn't raised before rotation reduces size below threshold, teams face unexpected failures unless proactive checks detect accumulation first.

Code Example

The scanner detects log files by filename patterns and flags oversized ones:

def scan_logs(directory, depth=2):
    logs = []

    def iterate(path, current=0):
        for fname in os.listdir(path):
            fpath = os.path.join(path, fname)

            if not os.path.isfile(fpath):
                continue

            try:
                stat_info = os.stat(fpath)
                size = stat_info.st_size
                mtime = stat_info.st_mtime

                # Age days since last modification
                age_days = int((time.time() - mtime) / 86400)

                # Simple log pattern detection
                name_lower = fname.lower()
                is_log = (name_lower.endswith('.log') or 
                          any(ext in name_lower for ext in ['_log', '.err']))

                if is_log:
                    recommendation = 'review' if size > 10 * 1024 * 1024 else ''
                    elif size > 50 * 1024 * 512 * 1024:
                        recommendation = 'delete'

                    logs.append({'name': fname, 'size': size, 
                                 'age_days': age_days, 'recommendation': recommendation})
            except IOError:
                continue

    iterate(os.path.abspath(directory))
Enter fullscreen mode Exit fullscreen mode

Uses os.stat() to get byte count and modification timestamp. Age calculation divides seconds since last write by 86400 (seconds per day). Pattern matching detects common log extensions without requiring full file content inspection—saves time scanning directories for known patterns only.

Threshold logic flags files larger than 10MB for review, those exceeding 50MB for rotation or deletion. This balances catching problematic storage consumers while allowing legitimate large logs when teams configure rotation after checking retention relevance.

Use Cases

Production Quota Checks

Cloud environments enforce quotas that auto-delete data beyond limits:

python log-pruner.py -c -d /opt/monitoring
# Alert when rotation_candidates > 5, before quota hits critical threshold
Enter fullscreen mode Exit fullscreen mode

Count output feeds alert systems that warn of approaching disk exhaustion. When monitoring agents write to shared directories without central management, this catches accumulation before quota breach causes service restarts or data loss from auto-deletion policies.

Build Pipeline Cleanup

After each run, ensure logs don't inflate artifact size:

python log-pruner.py -d /tmp/build-artifacts -l 1
# Delete flagged files before archiving to remote storage
Enter fullscreen mode Exit fullscreen mode

Deletion candidates get passed to cleanup scripts that remove them safely. When build artifacts ship without prior rotation or truncation, archive downloads become unnecessarily large—flagged files catch size inflation before it impacts delivery costs or client download times.

Legacy System Review

Systems lacking centralized logging accumulate files in custom directories:

python log-pruner.py -d /home/legacy/logs -r -l 2
# Flag files before applying `logrotate` configuration from manual review
Enter fullscreen mode Exit fullscreen mode

Manual inspection of flagged logs determines whether entries still relevant for debugging recent issues. Old entries from production incidents need archival, while debug info from current failures stays readable until resolution completes or next rotation window passes.

Common Patterns Detected

Pattern Example Files Why Oversized?
*.log daemon.log, app.log No rotation config before write
*_log.txt nginx/access_log.txt Custom service capturing to files
.err stderr.err Error redirection to files
_log in name system_auth_log.log Script output without naming norms

Pattern matching catches variations in naming conventions from different services. Without central logging daemon like journald, application developers write logs directly to files—often without rotation scripts that trim entries before reaching size limits.

Alternatives Compared

  • find -size +10M -delete: Removes everything without checking relevance first
  • du -sh *: Shows folder totals—not which specific logs cause disk pressure
  • This tool: Size + age context, recommendations before deletion

Manual deletion risks removing recent debug entries needed for incident response. Automated tools without rotation awareness lose data before teams can determine retention value. This tool balances safety with space management by flagging first, allowing humans or automation scripts to decide whether deletion applies to specific paths.

Source Code

Public repo with examples. Readable, dependency-free implementation suitable as foundation for security automation or policy enforcement scripts.

🔗 Repo: https://github.com/Poolion/log-pruner-cli

If you find this useful, you can support development: https://www.buymeacoffee.com/poolion

Top comments (0)