Codebase Knowledge Base Series (03): Code Embedding Strategies — Raw Code Outperforms Comment-Enhanced?

The Assumption and the Data

The common assumption: docstrings look more like natural language, so embedding them should help vector models understand code better. Most engineers start with comment-enhanced indexing.

The data disagrees.

---

Experiment Design

Dataset: 27 Python functions across 5 business modules (auth, database, cache, payment, notification)

Three strategies, same embedding model (BAAI/bge-large-zh-v1.5):

Strategy A: Raw code
  Embed the complete function body — variable names, library calls,
  control flow all preserved

Strategy B: Signature + docstring only (comment-enhanced)
  Embed only the def line + docstring, strip all implementation code
  Example:
  def validate_jwt_token(token: str)
  """Decode and validate a JWT token. Returns payload if valid."""

Strategy C: Hybrid (comment prefix + code)
  Prepend the docstring as a comment, then include the full code body
  Example:
  # Decode and validate a JWT token. Returns payload if valid.
  def validate_jwt_token(token: str):
      payload = jwt.decode(token, SECRET_KEY, ...)
      ...

All three strategies use the same model — differences come only from input format, not model quality.

Metric: Recall@3 and Recall@5 (12 natural language queries × ground-truth relevant functions)

---

Results

Strategy                       Recall@3    Recall@5    vs A
──────────────────────────── ──────────  ──────────  ──────
A_raw_code                        0.889       0.958    base
B_comment_enhanced                0.847       0.917   -0.041
C_hybrid                          0.847       0.917   -0.041

Strategy A wins. B and C tie, both 0.041 below A.

Per-query breakdown (Recall@5)

Query                                                   A       B       C
────────────────────────────────────────────────── ──────  ──────  ──────
verify user identity and check JWT token validity    1.00    0.50    0.50 ←
encrypt and store user password securely             1.00    1.00    1.00
generate JWT access token for authenticated user     1.00    1.00    1.00
check if user has permission to perform an action    1.00    1.00    1.00
store and retrieve data from Redis cache             1.00    1.00    1.00
limit how many times a user can call an API          1.00    1.00    1.00
execute SQL query safely against the database        1.00    1.00    1.00
process payment and create Stripe charge             0.50    0.50    0.50
issue refund to customer                             1.00    1.00    1.00
send email notification to user                      1.00    1.00    1.00
send mobile push notification                        1.00    1.00    1.00
delete a record without removing it from database    1.00    1.00    1.00

Divergence occurs in two queries:

Q1 (JWT validation): A=1.0, B/C=0.5

Strategy B embeds only:

def validate_jwt_token(token: str)
"""Decode and validate a JWT token. Returns payload if valid."""

Strategy A also embeds:

payload = jwt.decode(token, SECRET_KEY, algorithms=["HS256"])
except jwt.ExpiredSignatureError:
except jwt.InvalidTokenError:

jwt.decode, ExpiredSignatureError, InvalidTokenError tell the embedding model precisely where this function lives in the JWT domain. The docstring says "decode and validate" — accurate but imprecise, dropping the specific technical vocabulary that does the discriminative work.

Q8 (Stripe payment): all three score 0.50

All three strategies retrieve create_payment_intent but miss calculate_order_total. The query is "process payment and create Stripe charge," but calculate_order_total's docstring says "Sum item prices, apply discount, compute tax." In vector space, these are semantically distant. No embedding strategy closes that gap.

---

Why Raw Code Performs Better

Code contains rich technical vocabulary that functions as high-quality semantic signal:

Function and variable names:
  validate_jwt_token → "JWT" "token" "validate"
  hash_password → "hash" "password"
  rate_limit_check → "rate" "limit"

Library and package names:
  import bcrypt → password hashing domain
  jwt.decode() → JWT authentication domain
  stripe.PaymentIntent → Stripe payment domain
  redis.Redis → cache/Redis domain

Exception class names:
  jwt.ExpiredSignatureError → JWT expiry scenario
  stripe.error.SignatureVerificationError → payment signature verification

These technical terms have rich context in pre-training data. The embedding model maps them to precise semantic neighborhoods. Docstrings, despite being "more like natural language," tend to use vague verbs ("process," "handle," "manage") that are less discriminative than the technical vocabulary in the code itself.

The analogy: in medical literature, "fever, cough, sore throat" gives a doctor more diagnostic signal than "feeling unwell" — even though the former isn't "more natural language."

---

When Comment-Enhanced Actually Wins

Strategies B and C aren't useless. They outperform A in specific situations:

Situation 1: Code uses opaque naming

def do_proc_v2(x, y, flag=False):
    """Process user authentication with fallback to legacy mode."""
    result = _run_auth_pipeline(x, y)
    if flag:
        return _legacy_compat(result)
    return result

do_proc_v2, x, y carry no semantic information. The docstring is the only useful signal. Strategy B will significantly outperform A here.

Situation 2: Comments contain business context not visible in code

def calculate_fee(amount: float) -> float:
    """
    Platform fee per the 2024 pricing agreement with Partner XYZ.
    Fee = 2.5% for amounts < 1000, 1.8% for amounts >= 1000.
    """
    return amount * (0.025 if amount < 1000 else 0.018)

"Partner XYZ" and "2024 pricing agreement" are business context invisible in the implementation. Only the docstring contains this information.

Situation 3: High-quality docstring coverage

If the codebase enforces documentation standards and most functions have substantive docstrings (> 60% coverage), Strategies B and C become more competitive.

---

Engineering Recommendations

For most codebases: Start with Strategy A (raw code) — simple, effective, no preprocessing needed.

Consider Strategy C when:

• The codebase has many "semantically opaque" functions (poor naming, meaningless parameter names)
• High-quality docstring coverage exists (> 60% of functions have substantive docstrings)
• Query patterns lean toward business language ("user registration flow") rather than technical terms ("JWT token validation")

Always attach structured metadata to function-level chunks:

{
    "content": func_body,               # text to embed
    "metadata": {
        "name": "validate_jwt_token",
        "module": "auth",
        "file": "services/auth.py",
        "line_start": 12,
        "line_end": 20,
        "has_docstring": True,
        "called_by": ["login_handler", "middleware"],
    }
}

Metadata enables post-retrieval filtering ("search only the auth module") and supports navigation ("jump to definition") without affecting embedding quality.

---

Summary

1. Raw code embedding scores highest (Recall@5=0.958): technical vocabulary in code — library names, exception classes, function names — provides precise semantic signals the embedding model correctly uses
2. Q1 pinpoints the root cause: jwt.decode, ExpiredSignatureError are the specific terms that give Strategy A its advantage over B; comment-only strips precise technical signals and replaces them with vague verbs
3. Q8 reveals semantic gaps vector search can't close: the distance between calculate_order_total and "create Stripe charge" exists in the semantic space regardless of embedding strategy — these cases need complementary approaches (keyword search, call graph) to bridge

---

References

• Full demo code: codebase-kb-03-embedding

---

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage