DEV Community

StagingBench Team
StagingBench Team

Posted on AI-assisted

Handle three-state fields in a JSON benchmark: True, False and missing

A benchmark reader needs to preserve the difference between an observed negative result and an unknown value. That distinction applies both to feature flags and to measurements attached to individual rooms. Combining them can produce a tidy summary with the wrong meaning.

The published StagingBench aspect ratio records contain 93 True values and 26 False values across 119 observations, with 0 unknown values. Its feature records cover 12 tools and 10 features: 64 True values, 2 False values and 54 unknown values across 120 fields.

Those totals describe different kinds of evidence. An aspect ratio flag records an outcome for a room. A feature flag records the status gathered from vendor pages. They can share counting logic while retaining their separate meanings.

Load the values without flattening them

In JSON, the relevant values are true, false and null. After loading them in Python, they become True, False and None. A missing dictionary field can also be handled as unknown by using .get().

Keep those values intact until counting or display. Converting them to a plain yes or no too early discards information that later steps cannot recover.

The following snippet reads a local copy of the published results at data/results.json. It skips entries whose hidden field is true, merges the room01 dictionary with the rooms list by room id (so room 01 is not counted twice), and counts the expected feature fields.

import json
from pathlib import Path

FEATURES = """
declutter empty_room_staging furnished_restyle day_to_dusk
sky_replacement renovation_recolor tour_360_video api
mls_disclaimer commercial_license
""".split()


def counts(values):
    values = list(values)
    return (
        sum(value is True for value in values),
        sum(value is False for value in values),
        sum(value is not True and value is not False
            for value in values),
    )


data = json.loads(Path("data/results.json").read_text(encoding="utf-8"))

for tool in data["tools"]:
    if tool.get("hidden") is True:
        continue

    rooms = {r["room"]: r for r in tool.get("rooms") or []}
    if isinstance(tool.get("room01"), dict):
        rooms["01"] = tool["room01"]

    features = tool.get("features") or {}
    aspect = counts(r.get("aspect_ratio_kept") for r in rooms.values())
    flags = counts(features.get(name) for name in FEATURES)

    print(f"{tool['id']}: aspect kept/not kept/unknown={aspect}; "
          f"features True/False/unknown={flags}")
Enter fullscreen mode Exit fullscreen mode

Read each count in its stated order

The aspect ratio tuple reports kept, not kept and unknown. The feature tuple reports True, False and unknown. Keeping the labels beside the values avoids making the reader remember an unexplained order.

The identity checks are deliberate. value is True counts only the boolean True, while value is False counts only the boolean False. Under the expected field contract, the remaining state is None, including values returned for missing keys.

Unexpected strings or numbers deserve validation before reporting. This short counter assumes boolean or unknown fields; it does not establish that an incoming document follows that contract. In particular, the string "False" should never silently become a recorded negative observation.

Why if not value loses information

Both False and None satisfy if not value. Using that condition to count failures would put an explicit negative result and an unanswered question in the same bucket. Empty strings and numeric zero would also enter that branch.

For aspect ratio measurements, False means the recorded output did not keep the ratio. Unknown means the field supplies no answer. For features, False records absence, while unknown leaves verification unresolved. Those distinctions should survive every transformation.

Count expected fields, including absent ones

The fixed feature list serves a practical purpose. Iterating only over features.values() would count the entries present in the dictionary and overlook omitted feature keys. Looking up every expected feature with .get() gives missing fields an explicit unknown place in the count.

Room observations need a different boundary. A room dictionary with a missing measurement contributes an unknown observation. An absent room dictionary does not automatically become an observation. Creating one would change the denominator without evidence.

Keep that boundary visible when interpreting totals. Unknown fields belong in the accounting, and missing observations need their own explanation. Preserve both distinctions and the summary remains traceable to what was actually recorded.

Top comments (0)