CanopyOS: Open-Weight Vision for Outdoor Urban Greening Observations

Joy Mbiya — Sun, 11 Oct 2026 15:35:44 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass.

Abstract

Urban greening starts with observing real places, yet a photograph can be difficult to discuss systematically. We built CanopyOS, a mobile-first, open-source prototype that turns an outdoor photograph into an inspectable field note. A locally run, open-weight semantic-segmentation model marks vegetation, hard surfaces, buildings, and open ground. Deterministic code then computes image-pixel indicators and an exploratory Green Score, while a bounded decision layer suggests an intervention to investigate on site. Two openly licensed photographs demonstrate the complete workflow. The results establish that the software can produce traceable reports; they do not validate environmental impact, planting feasibility, or model accuracy across real-world sites.

What I Built

CanopyOS is for residents, student groups, and community teams who want a quick starting point for discussing the greenery around them. The intended interaction begins outside: take a photograph of a street, courtyard, or park; inspect the segmentation; and decide what deserves a closer site visit. The report makes the image's visible surfaces easier to discuss without claiming that a photograph replaces field measurements.

The application offers camera capture with gallery upload as a backup, an original-image/segmentation comparison, pixel-based indicators, a cautious recommendation, and a private recent-scan history. Users can delete scans and their stored images. The interface is in English and is designed for both phones and desktops.

Figure 1. The capture flow. This screenshot uses a credited archival photograph of the National Museum of Lubumbashi; it does not document a current field visit.

How I Built It: Research Question and Method

The practical question was: Can an open-weight vision model support a transparent first-pass observation of an outdoor scene while making the limits of that observation visible?

CanopyOS runs the pinned nvidia/segformer-b0-finetuned-ade-512-512 model locally after its weights are downloaded. Its ADE20K labels are mapped into four reporting groups: vegetation (including trees and grass), hard surfaces (roads and sidewalks), buildings, and open ground. Sky, people, vehicles, and other classes are excluded from the denominator. The report separately shows classified coverage, the fraction of the whole image represented by those four groups.

For classified image pixels, the exploratory score is:

Green Score = round(100 × (0.60 × vegetation_fraction
                         + 0.25 × open_ground_fraction
                         + 0.15 × tree_presence))

tree_presence is binary, based on the model's tree or palm pixels; it is not a count of trees or a measure of shade. The formula is a disclosed heuristic, not a calibrated environmental index. A finite decision layer chooses from permitted actions. In the demonstrated default configuration, it uses explicitly labelled deterministic rules. Optional Jev and Gemma integrations exist, but neither was active in the examples below. The application labels its template explanation when Gemma is unavailable; it never presents that text as a live model response.

The implementation uses Python, FastAPI, PyTorch, HTMX, Tailwind CSS, and SQLite. Uploaded images are re-encoded without EXIF metadata, and scan assets require ownership checks. The current design runs one inference job at a time.

Results: Two Reproducible Demonstrations

We ran the application with two openly licensed outdoor photographs. These are functional case studies, not a statistical evaluation or a field survey.

Demonstration image	Classified coverage	Exploratory Green Score	Reported outcome
National Museum of Lubumbashi	44.7%	31/100	Investigate a green wall after a site survey
Cupertino public park	Report available in the gallery	68/100	Preserve existing greenery

For the museum photograph, 26.0% of the classified pixels were vegetation and 72.8% were buildings. These percentages describe the image, not the site's land area or present condition. The photograph was taken in 2007. The park photograph was taken in 2015. Both were processed by the real segmentation pipeline; the decision and explanation fallbacks are visible in the reports.

Figure 2. Actual application report for the museum demonstration image. Image credit: Mike Rosenberg, CC BY 2.0; the photograph was resized and re-encoded for this demonstration.

Figure 3. Actual application report for the park demonstration image. Image credit: Tomwsulcer, CC0; the photograph was resized and re-encoded for this demonstration.

The software validation record reports 30 passing Python unit/API tests and 12 passing Playwright desktop/mobile tests. Those browser tests use a named vision test double; the two case studies above use the real model. This distinction matters: passing software tests does not establish semantic-segmentation accuracy or good urban-planning advice.

Demo

The desktop and mobile product gallery shows capture, both reports, and private history. The repository includes the two licensed demonstration images and local setup instructions, so the workflow can be repeated. The first scan requires downloading model weights; later inference runs locally.

A short video of the working upload-to-report flow or a verified public deployment must be added here before this article is submitted to the challenge. The challenge requires a demo in one of those forms; the screenshot gallery alone is supporting evidence.

Code

Source code, setup instructions, architecture, validation notes, and the MIT license: github.com/joeylukoji/CanopyOS.

Why Open Innovation Matters

The core image analysis runs with inspectable model weights rather than a proprietary vision endpoint. That lets a community group reproduce the same pipeline, examine the category mapping and score formula, and adapt them to local needs. Inference can run on a local computer after the initial weight download; the demonstrated path does not require sending photographs to a hosted AI provider. The optional hosted decision and explanation services are separated from this core and their absence is visible to the user.

This openness is especially useful for a citizen-science tool: the output should invite scrutiny. Someone can challenge a mislabeled sidewalk, inspect the overlay, or change a heuristic rather than accept an unexplained green number.

Limitations and Next Study

A single camera view misses depth, seasonality, ownership, soil conditions, access, and maintenance constraints. Pixel proportions are not measured surface area. Open ground is not automatically plantable, and tree presence is not shade. No temperature, carbon, biodiversity, or energy benefit has been measured. The two archival images do not represent current conditions in Lubumbashi or Cupertino.

The next evaluation is to photograph multiple current outdoor locations with permission, manually review their masks and recommendations, and record failure cases. A physical-phone camera test over HTTPS, live optional-provider tests, and public-host performance checks also remain pending. Until then, CanopyOS should be treated as a reproducible observation prototype and a prompt for real-world investigation, not an environmental assessment instrument.

DEV Community: Joy Mbiya