Your language chart changes when bytes get the votes

Your language chart changes its question when you replace repositories with bytes. That's an awkward discovery when you're about to publish it under “language adoption”.

You count language, then aggregate languages, expecting the breakdown to explain the same distribution. But languages holds byte counts, fetched through an optional enrichment specific to repoDetails.

Pooling language bytes weights repositories by their reported code size rather than giving each repository equal influence.

The arithmetic can be right while the chart title is wrong.

The label and the breakdown answer different questions

A repository label gives you a category to count. A language breakdown gives you quantities to sum, and those quantities introduce a weighting decision that a tidy percentage column won't reveal.

Before calculating anything, decide what you mean by “widely used”.

The same records support these separate paths:

repository records
  |
  +-- language label ----------> share of labelled repositories
  |
  +-- languages byte mapping
        +-- positive entries --> presence across measured repositories
        +-- pool bytes --------> share of reported code bytes
        +-- normalise first ---> mean within-repository share

Presence can overlap because a repository can contain several languages. Those percentages needn't sum to an exclusive distribution.

A pie chart would struggle with the arrangement.

One repository needs a collection boundary

Use repository details for an explicit collection of repository paths. This makes the population easier to describe, though selecting paths still doesn't create a representative sample of public repositories.

Example input: this requests language enrichment for the supplied repositories.

import os
import math
from numbers import Integral, Real

import pandas as pd
import duckdb
from apify_client import ApifyClient

run_input = {
    "mode": "repoDetails",
    "repoUrls": ["openai/whisper", "rust-lang/rust", "pallets/flask"],
    "includeLanguages": True,
    "maxItems": 3,
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "dataspryva/github-weekly-trending-repos-and-org-repo-finder"
).call(run_input=run_input)
if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Collection did not succeed: {run['status']}")

items = list(client.dataset(run["defaultDatasetId"]).iterate_items())

The observation here's a repository record. It isn't a developer, a contribution or a language-specific project.

Keep run_input alongside the analysis so you can distinguish the collection you requested from the records you accepted. Completion status describes the run; coverage requires its own accounting.

Also distinguish “repository observed” from “repository identified reliably”. Repeated records can give the same codebase extra influence even when you intended equal repository weights.

Which fields belong in the denominator?

These controls and fields determine the comparison. The output field list gives their meanings without fixing every scalar type or timestamp format, so the loader makes its acceptance rules explicit.

Field or control	Incoming shape and missingness	Treatment and analytical trap
mode	Required input string; select repoDetails here	Preserve as collection metadata. A language breakdown isn't an enrichment promised across every mode.
repoUrls	Input array of repository links or owner/repo paths	Preserve the requested collection. Returned records alone don't establish complete coverage.
includeLanguages	Boolean input, default true; applies to repoDetails	Enables the breakdown and adds a request per repository. It isn't a per-record coverage flag.
repoId	Repository identifier; no scalar wire type specified	Accept text or integer identifiers and normalise to a string key. Don't recover identifiers through floating-point coercion.
fullName	Repository full name; no formal scalar wire type specified	Accept nonblank text as a fallback identity within this collection. It doesn't establish continuity across collections.
language	Repository language label; empty values are omitted	Use nullable pandas string. Label counts describe labelled repositories, unless you explicitly retain an unknown category.
languages	Language-to-byte breakdown; empty values are omitted	Validate the complete mapping. An omitted mapping doesn't establish that a repository contains no code.
languagesCount	Language count; the supplied example uses an integer	Use nullable Int64 after validation. It can check mapping size, but cannot reconstruct missing entries.
scrapedAt	Collection timestamp; no exact format specified	Parse to UTC and count failures. It doesn't date a language-adoption decision.

Empty values and empty lists are omitted throughout repository records. Absence therefore tells you something about coverage, but doesn't explain why a particular value is unavailable.

That uncertainty belongs in the analysis rather than in a guessed replacement value.

Missing mappings change an average before weighting does

A labelled-only chart describes repositories with an available label. A byte chart describes repositories with usable mappings.

Those populations can differ before any shares are calculated.

Replacing an omitted mapping with an empty dictionary would treat an unknown observation as evidence that every language is absent. Keeping that repository in a presence denominator would then push the estimates down.

Partly valid mappings create a different problem. If you discard malformed entries but keep their neighbours, the remaining bytes become the entire repository total, inflating the surviving languages' within-repository shares.

The conservative choice here's to exclude that mapping as a whole and report the exclusion. You lose coverage, but you avoid manufacturing a denominator from the parts that happened to parse.

For comparisons between collections, inspect whether exclusions concentrate in particular repository groups. An equal-weight average can still describe a selective subset; changing the weights doesn't repair missing observations.

A typed loader keeps exclusions visible

This loader preserves raw fields, builds identity from the original records and parses count strings only when they contain plain decimal digits. There's no specified compact-number format to justify interpreting abbreviated counts.

def nonblank(value):
    return value.strip() if isinstance(value, str) and value.strip() else pd.NA

def whole_number(value, *, accept_text=False):
    if accept_text and isinstance(value, str):
        value = value.strip()
        return int(value) if value.isascii() and value.isdecimal() else None
    if isinstance(value, bool):
        return None
    if isinstance(value, Integral):
        return int(value) if value >= 0 else None
    if isinstance(value, Real):
        number = float(value)
        if math.isfinite(number) and number >= 0 and number.is_integer():
            return int(number)
    return None

def repository_key(record):
    value = record.get("repoId")
    if isinstance(value, str) and value.strip():
        return f"id:{value.strip()}"
    if isinstance(value, Integral) and not isinstance(value, bool):
        return f"id:{value}"
    name = nonblank(record.get("fullName"))
    return f"name:{name}" if pd.notna(name) else pd.NA

fields = ["repoId", "fullName", "language", "languages",
          "languagesCount", "scrapedAt"]
repos = pd.DataFrame(items).reindex(columns=fields)
repos["repo_key"] = pd.array(
    [repository_key(record) for record in items], dtype="string"
)
repos["mapping_key_present"] = ["languages" in record for record in items]
for field in ["fullName", "language"]:
    repos[field] = repos[field].map(nonblank).astype("string")

counts = [whole_number(value, accept_text=True)
          for value in repos["languagesCount"]]
repos["languagesCount"] = pd.array(
    [value if value is not None and value <= 2**63 - 1 else pd.NA
     for value in counts], dtype="Int64"
)
repos["scraped_at_utc"] = pd.to_datetime(
    repos["scrapedAt"], format="mixed", errors="coerce", utc=True
)

missing_identity = repos["repo_key"].isna()
repeated_identity = (
    repos["repo_key"].notna() & repos["repo_key"].duplicated(keep=False)
)
audit = {
    "returned_records": len(repos),
    "missing_identity_records": int(missing_identity.sum()),
    "repeated_identity_records": int(repeated_identity.sum()),
    "unparsed_collection_times": int(
        (repos["scrapedAt"].notna() & repos["scraped_at_utc"].isna()).sum()
    ),
}
repos = repos.loc[~missing_identity & ~repeated_identity].copy()
audit["accepted_repositories"] = len(repos)

Repeated resolved identities are excluded together, rather than silently keeping whichever record happened to arrive last. If you're combining collections, choose an observation-selection rule before calculating repository shares.

The fallback identity also has limits: a record identified by name and another identified by ID won't automatically resolve to the same repository. Reconciliation across collections needs more than a convenient string column.

Timestamp parsing is similarly modest. A successfully parsed scrapedAt establishes a usable collection time, not that every field describes an identical source instant.

Next, validate complete mappings and retain repository identity when expanding them into language rows.

def inspect_mapping(value, present):
    if not present:
        return "omitted", {}
    if not isinstance(value, dict):
        return "invalid_mapping", {}
    if not value:
        return "empty_mapping", {}
    if any(not isinstance(name, str) or not name.strip() for name in value):
        return "invalid_language_name", {}
    cleaned = {name: whole_number(count) for name, count in value.items()}
    if any(count is None for count in cleaned.values()):
        return "invalid_bytes", {}
    return ("usable" if sum(cleaned.values()) > 0 else "zero_total"), cleaned

checks = [
    inspect_mapping(value, present)
    for value, present in zip(repos["languages"], repos["mapping_key_present"])
]
repos["mapping_status"] = [status for status, mapping in checks]
repos["validated_languages"] = [mapping for status, mapping in checks]
audit["mapping_status_counts"] = repos["mapping_status"].value_counts().to_dict()

count_check = (
    repos["mapping_status"].isin(["usable", "zero_total"])
    & repos["languagesCount"].notna()
)
audit["language_count_disagreements"] = int(
    repos.loc[count_check, "languagesCount"].ne(
        repos.loc[count_check, "validated_languages"].map(len)
    ).sum()
)

eligible = repos.loc[repos["mapping_status"].eq("usable")]
eligible_n = len(eligible)
long_rows = [
    (key, name, count, count / sum(mapping.values()))
    for key, mapping in zip(eligible["repo_key"], eligible["validated_languages"])
    for name, count in mapping.items() if count > 0
]
language_rows = pd.DataFrame(
    long_rows, columns=["repo_key", "language", "bytes", "within_repo_share"]
).astype({
    "repo_key": "string", "language": "string",
    "bytes": "float64", "within_repo_share": "float64",
})
audit["byte_eligible_repositories"] = eligible_n

Numeric strings aren't accepted as byte values. Boolean values, negative counts and non-finite or fractional numbers also invalidate a mapping.

A zero-total mapping remains distinct from an omitted mapping, but neither enters this analysis's byte denominator. Without a positive total, a within-repository byte proportion is undefined.

The languagesCount comparison is diagnostic rather than corrective. Agreement doesn't certify the byte values; disagreement doesn't identify which field needs repair.

Floating-point aggregates are suitable for chart proportions here. Keep the validated integer mappings if your task requires exact byte accounting.

Give absent languages their place in the average

Within a usable mapping, positive bytes define presence. A language absent from that mapping contributes no share under this analysis convention.

A repository without a usable mapping contributes no observation.

That distinction is why averaging only the exploded rows gives the wrong answer for an equal-repository average. It calculates the mean among repositories containing that language, excluding eligible repositories where the language is absent.

This cell calculates separately named measures and attaches their denominators.

label_counts = repos["language"].dropna().value_counts()
label_n = int(repos["language"].notna().sum())
audit["labelled_repositories"] = label_n

byte_stats = language_rows.groupby("language").agg(
    presence_count=("repo_key", "nunique"),
    pooled_bytes=("bytes", "sum"),
    within_share_sum=("within_repo_share", "sum"),
)
pooled_total = language_rows["bytes"].sum()
comparison = pd.DataFrame({"label_count": label_counts}).join(
    byte_stats, how="outer"
).fillna(0)

definitions = [
    ("label_share", "label_count", label_n,
     "labelled repository", "equal per labelled repository"),
    ("presence_share", "presence_count", eligible_n,
     "repository", "equal per byte-eligible repository; overlap allowed"),
    ("pooled_byte_share", "pooled_bytes", pooled_total,
     "reported code byte", "proportional to repository byte total"),
    ("mean_within_repo_share", "within_share_sum", eligible_n,
     "within-repository proportion", "equal per byte-eligible repository"),
]
for measure, numerator, denominator, unit, weighting in definitions:
    comparison[measure] = (
        comparison[numerator] / denominator if denominator else float("nan")
    )

measure_notes = pd.DataFrame(
    definitions, columns=["measure", "numerator", "denominator", "unit", "weighting"]
)
report = (
    comparison.rename_axis("language").reset_index()
    .melt(id_vars="language", value_vars=measure_notes["measure"].tolist(),
          var_name="measure", value_name="share")
    .merge(measure_notes, on="measure", validate="many_to_one")
)

Dividing the sum of within-repository shares by all eligible repositories supplies the absent-language contributions implicitly. You don't need to materialise every repository-language combination.

The pooled and mean-within-repository measures use the same eligible collection, so their difference exposes weighting. The label measure uses a separate available-label population, which introduces coverage as another possible explanation.

To compare labels against byte measures on a common population, restrict every calculation to repositories with both a usable mapping and an available label. Keep that result separate from the broader measures, because matching the population costs coverage.

Put the denominator where the chart reader can see it

The worked example uses Dataspryva's GitHub Weekly Trending Repos & Org Repo Finder. The weighting decision applies wherever repository records include language byte counts.

For a SQL workflow, retain the same denominator discipline. This DuckDB calculation includes byte-eligible repositories where a particular language is absent.

connection = duckdb.connect()
connection.register("language_rows", language_rows)
sql_comparison = connection.execute("""
    SELECT
        language,
        COUNT(DISTINCT repo_key) AS presence_count,
        SUM(bytes) / NULLIF(
            (SELECT SUM(bytes) FROM language_rows), 0
        ) AS pooled_byte_share,
        SUM(within_repo_share) / NULLIF(
            (SELECT COUNT(DISTINCT repo_key) FROM language_rows), 0
        ) AS mean_within_repo_share
    FROM language_rows
    GROUP BY language
""").df()
connection.close()

Every eligible repository contributes a positive entry to language_rows, so its distinct repository count supplies the intended denominator.

Carry audit alongside report. A chart caption should identify the collection, the eligibility rule and whether the percentage represents labels, presence, pooled bytes or average composition.

Avoid calling the last measure “average language share” without its weighting. That wording leaves your reader to guess whether a large codebase had more influence than a small one, which is precisely the decision the calculation makes.

“Adoption” still needs evidence outside these fields

These fields describe repository labels and reported code composition within your collected population. They don't measure developer headcount, engineering effort or proficiency.

Repeated collections can describe changes in reported composition, but they don't establish when a team adopted a language or why that decision happened. Nor does equal repository weighting make repositories equivalent organisational units: your collection still decides what gets a vote.

A large byte share can coexist with limited repository presence. That's a useful finding once both measures have honest names.

Bytes get votes. Developers haven't been counted.