Anthony Leignel

Posted on Nov 15 Edited on Nov 15

🔎 Python Project Scanner Architecture: Static Analysis, Multi-Ecosystem Dependencies, and Technical Mapping

#programming #devtools #architecture #python

Taking over an existing software project often begins with the same questions.

Which technologies does it use? What libraries does it depend on? Which external services are integrated? How do its different components relate to one another?

In a recent, well-documented project, the answers may be relatively easy to find. In a legacy project, a monorepo, or an application composed of multiple services, they are often scattered across configuration files, dependency manifests, Dockerfiles, and source code.

Manual analysis becomes a task of its own.

An alternative approach is to build a technical inventory directly from the filesystem without executing the project being analyzed.

This article presents the architecture of Project Inventory, a static analysis engine built with Python 3.11+, capable of inventorying dependencies across ten ecosystems, identifying subprojects, analyzing Docker configurations, and mapping relationships between components.

1. Analyzing a Project Without Executing It

The first architectural principle is to work exclusively with observable elements within the project.

Software project
       │
       ▼
Filesystem exploration
       │
       ▼
Technical file detection
       │
       ▼
Manifest and configuration analysis
       │
       ▼
Source code reference analysis
       │
       ▼
Consolidation and mapping
       │
       ▼
HTML, JSON, and CSV reports

The scanner does not need to start the application, install its dependencies, or launch its Docker containers.

It builds a technical representation from the available files.

This distinction is important: a static inventory describes what can be observed within a project, but does not necessarily establish its actual runtime behavior.

A service may be declared in Docker Compose without running. A library may be installed without being directly imported. An environment variable may exist without being used.

The report must preserve these distinctions.

2. A Modular Architecture

The engine separates responsibilities across several categories of components.

project-inventory/
│
├── scanner/
│   ├── __init__.py
│   ├── filesystem.py
│   └── project_scanner.py
│
├── detectors/
│   ├── __init__.py
│   ├── environment.py
│   ├── infrastructure.py
│   ├── languages.py
│   ├── packages.py
│   ├── projects.py
│   └── services.py
│
├── analyzers/
│   ├── __init__.py
│   ├── dependencies.py
│   └── relationships.py
│
├── exporters/
│   ├── __init__.py
│   ├── csv_export.py
│   ├── html_export.py
│   └── json_export.py
│
├── docs/
├── output/
│   └── .gitkeep
│
├── LICENCE.md
├── LICENSE.md
└── app.py

This organization corresponds to the documented project structure.

Component	Responsibility
`scanner/`	Exploration and orchestration
`detectors/`	Information extraction
`analyzers/`	Consolidation and relationships
`exporters/`	Report generation
`output/`	Result storage
`app.py`	Application entry point

The separation between detection and analysis is a central aspect of this architecture.

A detector can identify a package, an environment file, or a Docker service. An analyzer can then correlate these observations to establish relationships between components.

This prevents data discovery from being mixed with interpretation.

3. Recursive Filesystem Exploration

The first step is to traverse the selected directory and its subdirectories.

However, not every file within a project is relevant to its technical inventory.

Directories containing installed dependencies, generated files, or development tool data can significantly increase the volume of files to analyze.

The scanner excludes directories including:

.git
.idea
.vscode
__pycache__
node_modules
vendor
.venv
venv
dist
build
coverage

An educational example using the Python standard library could look like this:

import os
from pathlib import Path

EXCLUDED_DIRS = {
    ".git",
    ".idea",
    ".vscode",
    "__pycache__",
    "node_modules",
    "vendor",
    ".venv",
    "venv",
    "dist",
    "build",
    "coverage",
}


def walk_project(root: Path):
    for current, directories, files in os.walk(root):
        directories[:] = [
            directory
            for directory in directories
            if directory not in EXCLUDED_DIRS
        ]

        for filename in files:
            yield Path(current) / filename

This example illustrates recursive traversal and directory exclusions. It does not reproduce the production code of Project Inventory.

Modifying the directories list in place prevents os.walk() from descending into excluded directories when using its default top-down traversal.

This is particularly useful when a project contains a large node_modules directory or Python virtual environment.

The objective is not to analyze every installed file, but to identify the elements that describe the project's architecture.

4. Detecting Technical Files by Name and Extension

A multi-ecosystem scanner must recognize several file formats.

Some have conventional names, such as package.json or requirements.txt. Others are identified by their extension, such as .csproj files.

A simplified representation of the detection rules could look like this:

KNOWN_FILES = {
    "requirements.txt": "python",
    "pyproject.toml": "python",
    "package.json": "node",
    "composer.json": "php",
    "go.mod": "go",
    "Cargo.toml": "rust",
    "Gemfile": "ruby",
    "pom.xml": "java",
    "pubspec.yaml": "dart",
    "mix.exs": "elixir",
}


def detect_ecosystem(path):
    if path.name in KNOWN_FILES:
        return KNOWN_FILES[path.name]

    if path.suffix == ".csproj":
        return "dotnet"

    return None

This example is intentionally incomplete. The documented engine also supports lock files and other configuration formats.

Detecting the file type makes it possible to select the appropriate processing logic.

JSON, TOML, XML, and Gradle configuration files cannot necessarily be interpreted using the same parser.

The scanner identifies relevant files before passing their contents to specialized detectors.

5. Inventorying Dependencies Across Ten Ecosystems

Project Inventory currently supports ten ecosystems.

Ecosystem	Supported manifests and files
Python	requirements.txt, pyproject.toml, poetry.lock, Pipfile, Pipfile.lock
Node.js	package.json, package-lock.json, yarn.lock, pnpm-lock.yaml
PHP	composer.json, composer.lock
Go	go.mod, go.sum
Rust	Cargo.toml, Cargo.lock
Ruby	Gemfile, Gemfile.lock
Java / JVM	pom.xml, build.gradle, build.gradle.kts
.NET	*.csproj
Dart / Flutter	pubspec.yaml, pubspec.lock
Elixir	mix.exs, mix.lock

This diversity requires distinguishing the format of a file from the meaning of the information it contains.

A manifest generally describes the dependencies requested by a project.

A lock file can specify resolved versions and include transitive dependencies.

However, not every file serves exactly the same purpose. For example, go.sum contains module checksums and should not be interpreted as a conventional lock file that independently represents the complete set of active dependencies.

The analysis must account for the semantics of each ecosystem.

Example: Node.js Manifest

{
    "dependencies": {
        "express": "^4.18.2",
        "axios": "^1.7.0"
    },
    "devDependencies": {
        "vite": "^5.0.0"
    }
}

This file declares version constraints.

The presence of express in the manifest does not necessarily mean that the installed version is exactly 4.18.2.

A lock file may provide a more precise resolved version.

The inventory should preserve this distinction rather than replacing both observations with a single value.

6. Consolidating Dependencies Without Losing Their Sources

The same dependency can appear in multiple technical files.

In a Dart project, for example, a package may be declared in pubspec.yaml and locked in pubspec.lock.

The engine consolidates information by ecosystem and package.

Example from the documentation:

dio
    pubspec.lock         ==5.7.0 [locked]
    pubspec.yaml         ^5.7.0

The package appears once, while its different sources remain visible.

This approach avoids two problems:

Counting the same dependency multiple times
Losing the distinction between declared and locked versions

A conceptual representation could look like this:

{
    "ecosystem": "dart",
    "package": "dio",
    "sources": [
        {
            "file": "pubspec.yaml",
            "version": "^5.7.0",
            "type": "declared"
        },
        {
            "file": "pubspec.lock",
            "version": "5.7.0",
            "type": "locked"
        }
    ]
}

This JSON illustrates the consolidation principle. It does not necessarily correspond to the exact schema used by the engine's exports.

A dependency discovered only in a lock file also remains in the inventory.

This makes it possible to identify transitive packages that are not directly declared in the main manifest.

Keeping Ecosystems Separate

Two packages with the same name in different ecosystems do not necessarily represent the same library.

A conceptual consolidation key could therefore be:

key = (ecosystem, package_name)

This distinction prevents a Python package and a Node.js package with identical names from being arbitrarily merged.

In projects containing multiple subprojects, source files and their locations also remain important for identifying the context of each dependency.

7. Identifying Subprojects Within a Single Directory Tree

A software repository may contain several independent applications or components.

For example:

application/
│
├── backend/
│   ├── requirements.txt
│   └── app.py
│
├── frontend/
│   ├── package.json
│   └── src/
│
├── worker/
│   ├── pyproject.toml
│   └── worker.py
│
└── docker-compose.yml

A global dependency inventory alone is insufficient to describe this architecture accurately.

The individual subprojects must also be identified.

application
    │
    ├── backend
    │       └── Python
    │
    ├── frontend
    │       └── Node.js
    │
    └── worker
            └── Python

Project Inventory identifies subprojects from detected manifests and configuration files.

Each subproject can be associated with one or more ecosystems and the technical files that led to its identification.

This step provides the context needed to associate dependencies and certain infrastructure relationships with individual components.

A manifest is still an architectural indicator. Its presence alone does not always establish that a directory represents an independently deployable application.

8. Analyzing Docker Compose Infrastructure

Application dependencies represent only part of the architecture.

An application may also depend on a database, a cache, a message broker, or other services.

Docker Compose files provide a particularly useful source of information.

The engine recognizes four conventional filenames:

docker-compose.yml
docker-compose.yaml
compose.yml
compose.yaml

For each detected service, it can extract information including its image, build configuration, ports, volumes, environment files, and dependencies between services.

Consider this educational example:

services:
    backend:
        build: ./backend
        ports:
            - "5000:5000"
        env_file:
            - .env
        depends_on:
            - database

    database:
        image: postgres:16
        volumes:
            - postgres_data:/var/lib/postgresql/data

volumes:
    postgres_data:

Several elements can be observed in this file:

backend
    ├── build: ./backend
    ├── port: 5000:5000
    ├── env_file: .env
    └── depends_on: database

database
    ├── image: postgres:16
    └── volume: postgres_data

This information is obtained without starting the containers.

However, the declared configuration must remain distinct from the actual state of the infrastructure.

The presence of a port in Docker Compose does not demonstrate that it is currently accessible. Similarly, depends_on describes a declared relationship between services, but does not by itself guarantee that a dependency is ready to accept requests.

9. Enriching Services With Infrastructure Signatures

A Docker image can provide additional evidence about the purpose of a service.

For example:

postgres:16
    → PostgreSQL

redis:7
    → Redis

rabbitmq:3-management
    → RabbitMQ

The engine includes signatures for identifying several known infrastructure components:

PostgreSQL
Redis
MySQL
MariaDB
MongoDB
RabbitMQ

These signatures enrich the discovered information.

They do not replace the analysis of Docker Compose files.

A service using an unknown image should remain in the inventory even when it cannot be associated with a recognized infrastructure component.

This distinction prevents discovery from being limited to technologies already present in a predefined catalog.

10. Analyzing Dockerfiles

A Dockerfile provides additional information about how a component is built and started.

The engine extracts instructions including:

Instruction	Extracted information
`FROM`	Base image
`WORKDIR`	Working directory
`EXPOSE`	Declared port
`CMD`	Default command
`ENTRYPOINT`	Entry point

Example:

FROM python:3.12-slim

WORKDIR /app

EXPOSE 5000

CMD ["gunicorn", "-w", "2", "-b", "0.0.0.0:5000", "app:app"]

The analysis can identify Python as the base environment and Gunicorn as the declared startup command.

This information becomes particularly useful when correlated with Python dependencies.

Gunicorn can be used to start an application without appearing in an import gunicorn statement within the source code.

Source code analysis alone could therefore miss a reference that is present in the infrastructure configuration.

11. Extracting Environment Variables Without Exposing Their Values

Environment files often contain important information for understanding a project's integrations.

Project Inventory analyzes files including:

.env
.env.example
.env.sample

However, these files may contain API keys, passwords, or other sensitive credentials.

The engine extracts only the variable names.

Example output:

OPENAI_API_KEY
ENABLE_PERSISTENCE
MEMORY_MAX_TURNS
STRICT_MODE

The values are not displayed.

An educational extraction example could look like this:

import re

ENV_NAME = re.compile(
    r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*="
)


def extract_env_names(content: str):
    names = set()

    for line in content.splitlines():
        match = ENV_NAME.match(line)

        if match:
            names.add(match.group(1))

    return sorted(names)

This function illustrates basic variable name extraction without retaining the values in its output.

It is not a complete parser for every environment file format.

The analysis can therefore identify the existence of a configuration without directly exposing its sensitive contents.

This precaution must also apply to exports, error messages, and technical logs.

12. Identifying External Services From Evidence

Dependencies and environment variables can provide evidence about the external services integrated into a project.

The engine includes signatures for services such as OpenAI, Stripe, AWS, SendGrid, Twilio, and Sentry.

Documented example:

OpenAI
    package    openai
    variable   OPENAI_API_KEY

Two separate observations associate the project with OpenAI.

A conceptual representation could look like this:

{
    "service": "OpenAI",
    "evidence": [
        {
            "type": "package",
            "value": "openai"
        },
        {
            "type": "environment_variable",
            "value": "OPENAI_API_KEY"
        }
    ]
}

The result retains the evidence that led to the identification.

This approach is more informative than a simple list of external services because it explains why a service appears in the report.

An important limitation remains: the presence of a package and an environment variable does not prove that a connection to the service actually occurs.

The service may be optional, disabled, or simply anticipated in the configuration.

13. Analyzing Python Imports With the AST

Analyzing declared dependencies does not always establish which libraries are referenced in the source code.

For Python, the engine uses the AST, or Abstract Syntax Tree.

The AST represents the syntactic structure of a program without requiring its execution.

Consider this file:

import os
import json

from datetime import datetime
from flask import Flask
from openai import OpenAI

from storage import save_record

A text search might identify some imports, but it could also confuse source code with comments or strings.

The AST makes it possible to identify actual import statements within the program's syntax.

Educational example:

import ast


def extract_imports(source: str):
    tree = ast.parse(source)
    imports = set()

    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            for alias in node.names:
                imports.add(alias.name)

        elif isinstance(node, ast.ImportFrom):
            if node.module:
                imports.add(node.module)

    return sorted(imports)

This function analyzes source text without importing or executing the modules involved.

It could produce:

datetime
flask
json
openai
os
storage

A complete implementation must also account for relative imports, submodules, and files containing syntax errors.

Project Inventory classifies Python imports into three categories:

PYTHON STANDARD LIBRARY
    os
    json
    datetime

EXTERNAL
    flask
    openai

INTERNAL
    storage

This classification distinguishes standard library modules, external dependencies, and modules belonging to the project itself.

14. Declared, Locked, and Referenced Dependencies

A dependency may be present in a manifest without appearing directly in the source code.

Conversely, an import may be detected even though no corresponding dependency is explicitly declared in the analyzed manifest.

Project Inventory correlates declared Python dependencies with references observed in source code and, where relevant, Docker configurations.

Example:

flask
    REFERENCED
    import flask

gunicorn
    REFERENCED
    Docker ["gunicorn", "-w", "2", ...]

These two packages are referenced in different ways.

Flask appears in the source code. Gunicorn appears in a Docker command.

Status	Meaning
Declared	Present in a manifest
Locked	Resolved version observed
Referenced	Static reference detected
Reference not found	No static reference detected

The last status is deliberately different from "unused dependency."

The absence of a static reference can have several explanations:

Dynamic imports
Loading through a plugin system
Indirect use by another library
Commands executed outside Python source code
Configuration outside the analyzed scope

There is also a distinction between the name of a distributed package and the name used to import it.

For example:

Package: python-dotenv
Import:  dotenv

Package: Pillow
Import:  PIL

A naive comparison between package names and imports can therefore produce incorrect results.

Dependency correlation must account for this distinction when the relationship can be established.

The inventory provides evidence for further analysis, but should not be used on its own to automatically remove dependencies.

15. Mapping Relationships Between Components

An inventory becomes more useful when it moves beyond a list of independent elements.

A project can be represented as a collection of components and relationships.

Backend subproject
    │
    ├── declares → flask
    ├── declares → openai
    └── uses_external_service → OpenAI

Backend Docker service
    │
    ├── builds → Backend subproject
    ├── depends_on → Database service
    └── uses_env_file → .env

Database Docker service
    │
    └── mounts → postgres_data

Project Inventory represents relationships including:

Relationship	Meaning
`declares`	Dependency declared by a subproject
`depends_on`	Dependency between Docker services
`builds`	Build associated with a subproject
`uses_env_file`	Use of an environment file
`mounts`	Docker volume mount
`uses_external_service`	Detected external service association

Each relationship retains its type, the types of its source and target components, and the evidence that led to its establishment.

A conceptual representation could look like this:

{
    "type": "depends_on",
    "source": {
        "type": "docker_service",
        "id": "backend"
    },
    "target": {
        "type": "docker_service",
        "id": "database"
    },
    "evidence": [
        {
            "file": "docker-compose.yml",
            "declaration": "services.backend.depends_on"
        }
    ]
}

This example illustrates a structured relationship. It does not reproduce the exact internal format of the engine.

The essential principle is that a relationship should only be created when it can be established from an observed element.

A technical map should not invent connections between components simply because they are present in the same project.

16. Separating Discovery From Enrichment

Discovery and enrichment address two different requirements.

Discovery identifies the elements actually present in the files.

Enrichment interprets certain elements using known signatures.

Project files
       │
       ▼
Dynamic discovery
       │
       ├── Packages
       ├── Variables
       ├── Services
       └── Imports
       │
       ▼
Enrichment
       │
       ├── External service signatures
       ├── Infrastructure signatures
       └── Observable relationships

A scanner does not need to know every existing Python, Node.js, or Rust package in advance to inventory them.

It needs to know where to look and how to interpret the files that declare them.

A signature can then provide additional information.

OPENAI_API_KEY
    → OpenAI

STRIPE_SECRET_KEY
    → Stripe

SENTRY_DSN
    → Sentry

An unknown package remains visible in the inventory even when no signature can enrich it.

This separation allows interpretation capabilities to expand progressively without restricting discovery to a predefined catalog.

17. Generating Multiple Representations of the Same Inventory

The engine produces several output formats from the analysis.

Consolidated results
         │
         ├── Terminal
         │
         ├── HTML
         │
         ├── JSON
         │
         └── CSV

Each format serves a different purpose.

Format	Purpose
Terminal	Immediate inspection
HTML	Structured inventory presentation
JSON	Processing by other tools
CSV	Tabular analysis and processing

The HTML report presents the architecture and detected information in a readable format.

JSON retains detailed data in a structured representation.

CSV files provide specialized views of the inventory.

The documented output structure is:

output/
│
├── project-inventory.json
├── project-inventory.html
│
└── csv/
    ├── dependencies.csv
    ├── dependency_usage.csv
    ├── dockerfiles.csv
    ├── environment_variables.csv
    ├── external_services.csv
    ├── imports.csv
    ├── infrastructure_components.csv
    ├── service_relationships.csv
    ├── services.csv
    ├── source_files.csv
    ├── subprojects.csv
    └── technical_files.csv

Separating exporters makes it possible to produce different representations without reimplementing the entire analysis for each format.

When a tool processes data from external projects, report generation must also consider security.

A filename or extracted value must be properly escaped before being inserted into HTML. CSV exports intended for spreadsheet applications may also require protection against cells being interpreted as formulas.

These are general report generation considerations rather than a description of the engine's internal security controls.

18. Running Locally or on a Server

Project Inventory requires Python 3.11 or later.

Its documented entry point is:

python app.py

The application then requests the path of the project to analyze.

Path of the project to analyze:

The results are displayed in the terminal and automatically exported to the output directory.

The engine can run on Windows, Linux, or macOS, as well as on a server with direct access to the project files.

The analysis relies on the filesystem accessible to the application.

No source code or secrets need to be transmitted to an external service to generate the inventory.

The analyzed project is neither executed nor modified.

However, read-only analysis of the target project must remain distinct from report generation: the engine writes its own output files to its dedicated output directory.

19. The Limitations of Static Analysis

A technical inventory is not an exhaustive representation of a software application's actual behavior.

Some information may remain invisible when it depends on runtime execution.

Static observation	What it does not establish
Declared package	Actual usage
Locked version	Version actually installed
Detected import	Execution of the code path
Declared Docker service	Service currently running
Detected variable	Active configuration
Identified external service	Actual network communication

Static analysis does not replace the observation of running processes, network traffic, or deployed environments.

Nor does it constitute, on its own, a security audit or vulnerability assessment.

Its purpose is to provide a structured foundation for understanding a project, identifying its components, and guiding further investigation.

Conclusion

Technical analysis of a software project involves more than listing the files present in a directory.

It requires recognizing formats, interpreting manifests, consolidating dependencies, identifying subprojects, and correlating information from source code and infrastructure configurations.

A modular Python architecture makes it possible to separate these responsibilities across scanners, detectors, analyzers, and exporters.

The distinction between discovery and enrichment allows the engine to dynamically inventory observed elements, then add information when signatures or relationships can be established.

But the most important principle remains the traceability of observations.

A declared dependency is not necessarily used. A detected variable does not prove a connection. A configured service is not necessarily running.

A reliable technical inventory must therefore distinguish what is discovered, declared, locked, referenced, or interpreted without turning a static observation into a claim about the actual runtime behavior of the software.

Technical Documentation

The engine architecture, supported ecosystems, static analysis mechanisms, and export formats are documented in the GitHub repository.

The repository provides technical documentation rather than downloadable source code or production files.

View the technical documentation on GitHub

https://palks-studio.com

Top comments (2)

Alex Shev • Nov 15

The detector/analyzer split is especially useful because it preserves the difference between evidence and inference. I’d carry that through to the exported JSON with a provenance field for every relationship: source file, parser rule, confidence, and whether it came from a manifest, lock file, or code reference. That makes the inventory reviewable when a dependency appears to be present but is not actually exercised at runtime.

Anthony Leignel • Nov 15

Thanks for the thoughtful feedback! This distinction is actually already a core principle of Project Inventory. The current implementation preserves evidence behind detected relationships and distinguishes between declared, locked, and statically referenced dependencies.

The article is a condensed overview of a much larger project, so not every aspect of the implementation is covered.

I particularly like your suggestion of making provenance even more explicit through parser rules and confidence levels. It fits very well with the approach behind the tool.