Malicious LiteLLM v1.107.0 on PyPI/Docker Hub Executes Arbitrary Code on Python Start: Mitigation Steps Available

Introduction: The Silent Threat in LiteLLM 1.107.0

On October 21, a malicious version of the LiteLLM Python package (v1.107.0) was published on PyPI and Docker Hub. This compromised package introduces a critical security threat by executing arbitrary code on every Python interpreter start. The mechanism? A cleverly placed .pth file and a sitecustomize.py script embedded within the package. These files hijack the Python startup process, allowing attackers to run malicious code before any user-written Python script executes.

How the Attack Works

The attack leverages two key components of Python's initialization process:

• .pth Files: These files instruct Python to add additional directories to the module search path. In this case, the malicious cook.pth file points to a directory containing the sitecustomize.py script, ensuring it runs during interpreter startup.
• sitecustomize.py: This script is automatically executed by Python during initialization. The malicious version in LiteLLM 1.107.0 contains code designed to execute arbitrary commands, effectively compromising the system.

The causal chain is straightforward: Installation of LiteLLM 1.107.0 → .pth file modifies sys.path → sitecustomize.py executes on startup → arbitrary code runs → system compromise.

Scope of Impact

Systems are at risk if they:

• Installed LiteLLM without version pinning between 11:39 and 13:23 UTC on October 21.
• Pulled Docker images with tags latest, main-latest, main-stable, or v1.107.0 between 11:46 and 14:20 UTC.

If either condition is met, the system should be treated as compromised. The risk? Unauthorized access, data breaches, and further exploitation of the system or network.

Mitigation Steps: A Decision-Dominant Approach

Addressing this threat requires a systematic approach. Here’s the optimal strategy:

1. Isolate Affected Systems: Immediately disconnect compromised machines from the network to prevent further exploitation.
2. Search for Malicious Files: Before running Python, search the filesystem for cook.pth and sitecustomize.py. Their presence confirms compromise.
3. Reinstall LiteLLM: Rebuild the environment with a safe version, such as litellm==1.106.2. Avoid using pip directly on compromised systems to prevent triggering the malicious code.
4. Rotate Keys: Assume all credentials and keys on the system are compromised. Rotate them immediately to prevent unauthorized access.

This approach is optimal because it:

• Prevents further damage by isolating the system.
• Confirms compromise through direct evidence (malicious files).
• Restores a safe environment without triggering the attack.

Under what conditions does this solution fail? If the attacker has already exfiltrated data or established persistence through other means. In such cases, a full forensic investigation is necessary.

Lessons Learned and Prevention

This incident highlights critical vulnerabilities:

• Lack of Version Pinning: Installing packages without specifying versions increases the risk of pulling compromised releases.
• Inadequate Security Checks: Package repositories like PyPI and Docker Hub lack robust mechanisms to detect malicious submissions.
• Exploitation of Python’s Startup Process: The use of .pth files and sitecustomize.py for code execution is a known attack vector, yet it remains under-mitigated.

To prevent similar incidents:

• Always Pin Versions: Specify exact package versions in requirements files to avoid installing compromised releases.
• Audit Dependencies: Regularly review and audit third-party packages for suspicious files or behavior.
• Implement Repository Scanning: Advocate for and adopt tools that scan package repositories for malicious content.

Rule of thumb: If you’re not pinning package versions, you’re gambling with your system’s security.

Technical Analysis: How the Malicious Code Operates

The malicious LiteLLM v1.107.0 package exploits Python’s initialization process to execute arbitrary code on every interpreter start. This is achieved through a carefully crafted combination of a .pth file and a sitecustomize.py script. Here’s the step-by-step breakdown of the attack mechanism:

1. Vector: Malicious Files Embedded in the Package

The attack hinges on two files included in the compromised package:

• cook.pth: A .pth file designed to modify Python’s sys.path during startup. This file is a standard mechanism in Python for adding directories to the module search path.
• sitecustomize.py: A script that Python automatically executes during interpreter initialization if present in a directory listed in sys.path.

2. Execution Flow: From Installation to Compromise

The causal chain unfolds as follows:

• Installation: When LiteLLM v1.107.0 is installed, the cook.pth file is placed in a directory where Python searches for .pth files (e.g., the site-packages directory).
• Path Modification: On Python startup, the cook.pth file is processed, appending a malicious directory to sys.path. This directory contains the sitecustomize.py script.
• Arbitrary Code Execution: Python’s initialization process detects sitecustomize.py in the modified sys.path and executes it. This script runs arbitrary code, compromising the system.

3. Impact: System Compromise and Observable Effects

The execution of sitecustomize.py allows the attacker to:

• Run arbitrary commands on the host system.
• Exfiltrate sensitive data, such as credentials or source code.
• Establish persistence by modifying system configurations or deploying additional malware.

Observable effects include unexpected network activity, unauthorized file modifications, or anomalous system behavior. However, the attack is stealthy by design, often leaving no immediate traces beyond the presence of the malicious files.

4. Risk Formation Mechanism

The risk arises from the following factors:

• Lack of Version Pinning: Installing LiteLLM without specifying an exact version (e.g., using pip install litellm) pulls the latest release, which in this case was malicious.
• Inadequate Security Checks: PyPI and Docker Hub lack robust mechanisms to detect and block malicious packages before publication.
• Exploitation of Python’s Startup Process: The .pth and sitecustomize.py mechanisms are legitimate Python features but are easily weaponized if misused.

5. Mitigation: Optimal Solution and Edge Cases

The most effective mitigation strategy involves:

• Isolate Affected Systems: Disconnect compromised machines from the network to prevent further exploitation.
• Search for Malicious Files: Locate and delete cook.pth and sitecustomize.py to halt the attack.
• Reinstall LiteLLM: Rebuild the environment with a safe version (e.g., litellm==1.106.2) on a clean system to avoid re-infection.
• Rotate Keys: Assume all credentials are compromised and rotate them immediately.

Optimal Solution Rule: If a system installed LiteLLM v1.107.0 during the specified timeframes, treat it as compromised and follow the above steps. Avoid using pip on affected systems to prevent re-execution of the malicious code.

6. Prevention Measures: Addressing Root Causes

To prevent future incidents:

• Version Pinning: Always specify exact package versions in requirements.txt or equivalent files.
• Audit Dependencies: Regularly inspect third-party packages for suspicious files or behavior.
• Repository Scanning: Advocate for and implement tools to scan PyPI and Docker Hub for malicious content.

This analysis underscores the critical need for vigilance in open-source ecosystems and the importance of understanding the mechanisms behind attacks to effectively mitigate them.

Impact and Mitigation Strategies

The malicious LiteLLM v1.107.0 package exploits Python’s initialization process to execute arbitrary code, creating a critical security threat. Here’s how it works, what’s at stake, and how to respond effectively.

Mechanisms of the Attack

The attack leverages two key files: cook.pth and sitecustomize.py. When LiteLLM v1.107.0 is installed, cook.pth is placed in the site-packages directory. During Python startup, this .pth file modifies the sys.path variable to include a malicious directory. This directory contains sitecustomize.py, which is automatically executed as part of Python’s initialization process. The result? Arbitrary code runs every time the Python interpreter starts, compromising the system.

Scope of Impact

Systems are affected if:

• LiteLLM was installed without version pinning between 11:39–13:23 UTC on October 21.
• Docker images with tags latest, main-latest, main-stable, or v1.107.0 were pulled between 11:46–14:20 UTC.

If either condition is met, treat the system as compromised. The risk? Data exfiltration, unauthorized access, and further exploitation, as the attacker gains arbitrary code execution capabilities.

Mitigation Steps: What Works and Why

Here’s the optimal mitigation strategy, backed by technical analysis:

• Isolate Affected Systems: Disconnect from the network immediately. This prevents further exploitation and data exfiltration.
• Search for Malicious Files: Locate cook.pth and sitecustomize.py on the filesystem. Their presence confirms compromise.
• Reinstall LiteLLM: Rebuild the environment on a clean system using a safe version (e.g., litellm==1.106.2). Avoid using pip on compromised systems, as the malicious files execute on every Python start.
• Rotate Keys: Assume all credentials are compromised. Rotate keys and revoke access immediately.

Why This Solution Dominates

This approach addresses the root cause by removing the malicious files and rebuilding the environment. Alternatives, like manually deleting files without reinstallation, risk leaving residual backdoors. The solution fails only if the attacker established persistence or exfiltrated data before detection, requiring forensic investigation.

Prevention: Closing the Gaps

To prevent future attacks:

• Version Pinning: Always specify exact package versions in requirements.txt. This avoids pulling compromised releases.
• Audit Dependencies: Regularly inspect third-party packages for suspicious files or behavior.
• Repository Scanning: Advocate for tools to scan PyPI and Docker Hub for malicious content. This addresses the lack of robust security checks in these repositories.

Optimal Solution Rule

If LiteLLM v1.107.0 is installed → treat the system as compromised. Isolate, search for malicious files, reinstall on a clean system, and rotate keys.

Avoid using pip on affected systems, as it triggers the malicious code. This rule minimizes risk and ensures thorough mitigation.