How I Debugged and Fixed Memory & Goroutine Leaks in ProjectDiscovery Nuclei Engine 🚀

ThryLox — Sun, 28 Jun 2026 01:20:58 +0000

If you work in cloud security or vulnerability scanning, chances are high that you rely on ProjectDiscovery Nuclei—the gold standard open-source vulnerability scanner powered by YAML templates.

While Nuclei performs exceptionally well as a standalone CLI tool, embedding it as an underlying SDK engine inside long-running microservices or continuous scanning workers introduces unique architectural challenges: memory bloat and goroutine leaks over extended execution loops.

Recently, I investigated and resolved these exact engine lifecycle leaks in Nuclei Issue #7503 and submitted Pull Request #7508. Here is a breakdown of what I discovered under the hood and how I fixed it in Go.

🔍 The Problem: Unbounded State & Orphaned Goroutines

When embedding NucleiEngine into a long-running application loop (where engines are instantiated and closed dynamically per scan target), I noticed that memory consumption climbed steadily over time, and orphaned goroutines remained active long after calling engine.Close().

Upon profiling the engine lifecycle in Go, I identified three primary memory leaks:

Unbounded sync.Map in HTTP-to-HTTPS Port Tracker: The HTTPToHTTPSPortTracker stored host port mapping states in an unbounded sync.Map. Over thousands of target scans, this map grew infinitely without eviction.
Orphaned Per-Host Rate Limiter Goroutines: Global protocol state maintained per-execution rate limit pools (PerHostRateLimitPool). When an engine execution finished, worker background routines were not cleanly shut down or purged.
Cached Template Parsers: Compiled template ASTs (parsedTemplatesCache and compiledTemplatesCache) retained parsed representations in memory between engine instances without an explicit cache purging mechanism during engine teardown.

🛠️ The Solution: Architecture & Code Fixes

1. Bounded Expirable LRU Caching

Instead of holding unbounded host entries in a sync.Map, I replaced the storage structure with an expirable LRU (Least Recently Used) cache configured with a strict capacity bound (4,096 entries) and a 24-hour TTL:

// Replacing unbounded sync.Map with bounded expirable LRU cache
type HTTPToHTTPSPortTracker struct {
    cache *expirable.LRU[string, struct{}]
}

func NewHTTPToHTTPSPortTracker() *HTTPToHTTPSPortTracker {
    return &HTTPToHTTPSPortTracker{
        cache: expirable.NewLRU[string, struct{}](4096, nil, 24*time.Hour),
    }
}

This guarantees that host mappings automatically expire and memory remains strictly bounded regardless of how many millions of URLs are scanned.

2. Lifecycle Cleanup in `protocolstate.Close()`

I updated the global protocol state tear-down procedure in pkg/protocols/common/protocolstate/state.go to release rate limiter worker routines and purge trackers upon Close():

func Close(executionID string) {
    stateLock.Lock()
    defer stateLock.Unlock()

    if state, ok := globalStateMap[executionID]; ok {
        // Release per-host rate limiters and background goroutines
        if state.PerHostRateLimitPool != nil {
            state.PerHostRateLimitPool.Close()
        }
        // Purge HTTP to HTTPS tracker entries
        if state.HTTPToHTTPSPortTracker != nil {
            state.HTTPToHTTPSPortTracker.Purge()
        }
        delete(globalStateMap, executionID)
    }
}

3. Engine Cache Purging Interface

Finally, I added a thread-safe Purge() method to the template parser struct and invoked interface type assertions during NucleiEngine.Close():

// Safely purge compiled template caches on engine close
func (e *NucleiEngine) closeInternal() error {
    if e.parser != nil {
        e.parser.Purge()
    }
    if purger, ok := e.executerOpts.Parser.(interface{ Purge() }); ok {
        purger.Purge()
    }
    return nil
}

⚖️ Technical Trade-offs & Potential Criticisms

When designing solutions for large open-source codebases, evaluating architectural trade-offs is essential:

Fixed LRU Capacity vs. Configuration: Setting a hardcoded 4,096 capacity works as a balanced default for standard worker memory limits. However, in enterprise environments scanning millions of domains concurrently, exposing this bound as a configurable parameter (Options.HTTPToHTTPSCacheSize) would be a clean future addition.
Runtime Interface Assertion: Using runtime type assertions (interface{ Purge() }) keeps the codebase decoupled and preserves backward compatibility for third-party SDK consumers using custom parsers without breaking their implementations.
Memory Reclamation vs. Re-parsing Overhead: Purging compiled template caches on engine teardown prioritizes memory stability over template compilation caching across separate engine instances.

🧪 Results & Verification

I validated these fixes across Nuclei unit test packages (httpclientpool, protocolstate, templates, and lib), verifying 100% success with zero memory accumulation between consecutive engine shutdowns.

fix(engine): resolve memory and goroutine leaks in embedded engine usage (#7503) #7508

ThryLox posted on Jun 27, 2026

Summary

Fixes #7503 by implementing the required leak-prevention cleanup mechanisms outlined in #7502 for long-running embedded engines.

Key Changes

Size-Bounded HTTP-to-HTTPS Tracker: Replaced the unbounded sync.Map in HTTPToHTTPSPortTracker (pkg/protocols/http/httpclientpool/http_to_https_tracker.go) with a size-bounded expirable LRU cache (4096 entries max, 24h TTL) and added Purge().
Per-Host Rate Limiter Pool Cleanup: Updated protocolstate.Close() (pkg/protocols/common/protocolstate/state.go) to release per-host rate-limit pool goroutines and purge the HTTP-to-HTTPS tracker on shutdown.
Template Cache Purging: Updated NucleiEngine.Close() / closeInternal() (lib/sdk.go) and Parser (pkg/templates/parser.go) to purge parsed and compiled template caches on engine close.

View on GitHub

Memory and goroutine leaks in long-running embedded engine usage #7503

coderabbitai[bot] posted on Jun 24, 2026

Summary

The embedded engine can leak memory and goroutines over time during long-running usage.

Required changes

Implement the leak-prevention work described in #7502:

bound the HTTPToHTTPS tracker with an LRU
release the per-host rate-limit pool goroutines on close
purge the template caches on engine close

Rationale

Without explicit cleanup and bounded caching, long-running embedders can accumulate memory usage and leave background goroutines running indefinitely.

Affected areas

HTTPToHTTPS tracking / redirect bookkeeping
per-host rate limit pool lifecycle and shutdown
template cache lifecycle during engine close

Acceptance criteria

The HTTPToHTTPS tracker is size-bounded and evicts old entries.
Per-host rate-limit pool goroutines are released when the engine closes.
Template caches are purged on engine close.
Long-running embedded usage no longer shows continued growth from these resources.

Backlinks

Pull request: https://github.com/projectdiscovery/nuclei/pull/7502
Request comment: https://github.com/projectdiscovery/nuclei/pull/7502#issuecomment-4794470056
Requested by: @Mzack9999

Additional context

PR title: fix leaks

View on GitHub

💡 Key Takeaways for Go Developers

Beware of Unbounded sync.Map in Long-Running Apps: While sync.Map is convenient, it lacks eviction policies. Use LRU caches with TTLs for dynamic lookup tables.
Explicit Teardown Interfaces: When building Go SDKs meant to be embedded, always provide clean Close() / Purge() methods to release background channels and goroutines.
Decoupled Lifecycle Hooks: Interface checks like if purger, ok := obj.(interface{ Purge() }); ok enable clean resource cleanup without introducing rigid package dependencies.

Written by @Thrylox. Connect with me on GitHub!

DEV Community: ThryLox

How I Debugged and Fixed Memory & Goroutine Leaks in ProjectDiscovery Nuclei Engine 🚀

🔍 The Problem: Unbounded State & Orphaned Goroutines

🛠️ The Solution: Architecture & Code Fixes

1. Bounded Expirable LRU Caching

2. Lifecycle Cleanup in protocolstate.Close()

3. Engine Cache Purging Interface

⚖️ Technical Trade-offs & Potential Criticisms

🧪 Results & Verification

fix(engine): resolve memory and goroutine leaks in embedded engine usage (#7503) #7508

Summary

Key Changes

Memory and goroutine leaks in long-running embedded engine usage #7503

Summary

Required changes

Rationale

Affected areas

Acceptance criteria

Backlinks

Additional context

💡 Key Takeaways for Go Developers

2. Lifecycle Cleanup in `protocolstate.Close()`