DEV Community: Johnny Santamaria

MudMap: A Voice-First Trail Logger That Works Without a Signal

Johnny Santamaria — Sun, 11 Oct 2026 19:47:27 +0000

MudMap: A Voice-First Trail Logger That Works Without a Signal

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

MudMap is an offline voice trail logger. You press Record, talk while walking — "Saw a heron, trail washed out past the creek, blackberries ripe" — and the app transcribes it on-device, extracts species and conditions, and drops a pin on a map you own. No cloud. No account. No signal required.

It exists to make the screen the shortest part of a hike. You don't take out your phone to type notes. You don't open an app to browse a feed. You walk, you talk, you put the phone away. Everything else happens after.

The typical user is someone who already walks outdoors and wants a record of what they saw — birders, foragers, trail runners, gardeners — without a subscription app that uploads their favorite fishing hole to a server they don't control.

Demo

The build produces a working debug APK. It builds cleanly in CI, unit tests pass, and the artifact installs on any modern Android phone (arm64-v8a).

Current state, honestly stated: the software pipeline is proven, but hardware verification on real trails is still pending. That's the next step, and it's why I'm looking for Android testers.

Known limit to flag upfront: the bundled basemap is a San Francisco demo extract only (longitude -122.53 to -122.35, latitude 37.70 to 37.83). Outside SF, pins appear on a blank basemap — your entries still save, you just won't see a basemap under them until you prepare a replacement region.

Code

The repository is at https://github.com/johnny603/MudMap.

A build is produced on every push to main via GitHub Actions, and tagged releases produce a permanent download URL.

How I Built It

MudMap is built on four open-source pieces, each load-bearing:

Layer	Open-source component	What it does
Speech-to-text	whisper.cpp v1.9.5	Transcribes the `.m4a` recording on-device
Map rendering	MapLibre Native 12.3.1	Renders the map and pins
Basemap	Protomaps (PMTiles)	Offline vector tiles bundled in the APK
Storage	Room / SQLite	Persists entries locally

The stack is Kotlin + Android Views, minSdk 26, targeting arm64-v8a. The native whisper layer is compiled against the Android NDK 27 and invoked through a small JNI adapter.

The architecture is deliberately small. There's a Transcriber port and a WhisperCppAdapter behind it; an EntityExtractor port with a RuleBasedExtractor behind it. That means the AI pieces are swappable — if a better on-device model drops tomorrow, I change one adapter, not the app.

Walkthrough of one use case:

User presses Record. The app requests one location fix (10-second timeout) and starts capturing .m4a audio to app-private storage.
User presses Stop. The location fix is frozen — if none arrived, the entry saves with null coordinates.
The AAC audio is decoded to 16 kHz mono float PCM — the format whisper.cpp expects.
whisper.cpp transcribes the PCM on-device. No network call.
The RuleBasedExtractor matches the transcript against a bundled vocabulary and produces tags: species, conditions, landmarks.
The six-field entry is saved to Room.
The map reads entries on launch and after saves. A pin opens a transcript-and-tags bottom sheet.

Every step except installing the app happens with the phone in airplane mode.

Challenges

The build itself turned into the hardest part of this project — harder than any of the app logic. Three problems in particular cost real time, and each needed several attempts before it broke.

Challenge 1: The cryptic `25.0.4.1` build failure

The first Gradle run failed with a message that said nothing useful:

FAILURE: Build failed with an exception.

* What went wrong:
25.0.4.1

No file, no line, no explanation. Just a version number.

Attempt 1 — Re-ran the build. Same failure, same 46-second wait.

Attempt 2 — Searched for the error string. 25.0.4.1 returns nothing useful; it's not a documented Gradle or AGP error code.

Attempt 3 — Read the README's pinned versions table carefully. It specified JDK 17. My shell was on a newer JDK, and the "Restricted method" warnings in the build output were the tell. Gradle 8.11.1 and AGP 8.9.2 do not support JDKs past 17 in this configuration — and instead of saying so, they emit a bare version string.

Resolution: forced JAVA_HOME to JDK 17 for the build shell:

export JAVA_HOME=$(/usr/libexec/java_home -v 17)

The build then proceeded to the next failure. I added this diagnosis to the README under Common build failures so the next person doesn't lose the same hour.

Challenge 2: The asset script couldn't find `pmtiles`

bash scripts/prepare-assets.sh failed with:

Install the pmtiles CLI, or set PMTILES to its absolute path.

Attempt 1 — Installed pmtiles via Homebrew. It installed, but the script still failed.

Attempt 2 — Verified the binary. Ran pmtiles --version — which returned unknown flag --version. That's not a bug; the CLI uses the version subcommand. But it made it unclear whether the install had worked at all. which pmtiles and pmtiles --help confirmed it was on PATH.

Attempt 3 — Passed the path explicitly. The README documents an escape hatch, PMTILES=/absolute/path/to/pmtiles, which bypasses the PATH lookup entirely. That worked, and the asset pipeline completed: it downloaded a 4.1 MB extract, verified it in 308µs, and printed Offline assets ready.

The lesson: a tool being "installed" and a script finding it are different problems. The explicit-path override is the reliable fix.

Challenge 3: The disk-space wall

With JDK and pmtiles resolved, the Android SDK install failed:

Warning: An error occurred while preparing SDK package NDK (Side by side)
27.0.12077973: No space left on device.

The NDK is roughly 3–4 GB. The disk had 662 MB free.

Attempt 1 — Homebrew cleanup. brew cleanup --prune=all reclaimed ~1.1 GB. Good, but nowhere near enough — the full Android toolchain plus Gradle plus the whisper.cpp native compile needs roughly 25–30 GB.

Attempt 2 — Cleared Gradle caches and the Homebrew download cache. Another gigabyte or so. Still an order of magnitude short.

Attempt 3 — Moved the build off the machine entirely. Instead of fighting for 30 GB on a MacBook Air that already had 198 GB used, I wrote a GitHub Actions workflow. The runner has its own disk. The build succeeded in 3 minutes 22 seconds, producing a 147 MB debug APK.

Resolution: CI builds the APK; the local machine never needs the Android toolchain. This also became the distribution mechanism — tagged releases now produce a permanent download URL from the same workflow.

What these three have in common

None of them were problems with MudMap's code. All three were environment problems, and all three had error messages that pointed away from the actual cause. That's the ordinary tax of getting an offline native app to build on a machine that wasn't set up for it — and it's why the README now has a Prerequisites section and a Common build failures table.

Why Does Open Innovation Matter?

The open pieces aren't a nice-to-have here — they're the reason the thing works where it's meant to be used.

It works with no signal. Whisper.cpp runs on the phone. There's no server round-trip for transcription. A closed speech API would need a cell tower, and the trail has none.

Your data stays on your phone. Internet permission is removed entirely from the manifest. Cloud backup and device transfer are excluded. Your trail notes are yours. A closed equivalent would store your location history on someone else's disk, where it becomes part of a data set you don't own.

The map is a file, not a subscription. Protomaps ships offline tiles as a .pmtiles archive bundled in the APK. MapLibre reads it locally. No tile-server bill, no API key, no per-request quota. A closed map SDK would meter every pan and zoom.

You can swap the model. The Transcriber and EntityExtractor ports mean the AI backend is replaceable. If a smaller, faster, better model drops — or if you want to fine-tune one on your own observations — you change one adapter. No rewrite.

It costs nothing to run. There is no server component. There is no monthly bill. The APK does everything.

The "open" part is what makes the whole thing possible offline, on a trail, away from infrastructure. That's the point of the project.

Hacktoberfest 2026 is a month-long celebration of open source, and this year it's specifically about building with open-weight models and open-source AI — which is exactly what MudMap is. The event runs all October, online and through 300+ in-person Fests, and anyone can take part regardless of where they are in their open-source journey. For MudMap, my immediate plan is to cut a v1.0.0 release with a permanent download link, then recruit Android testers to verify the software on real trails — microphone capture, GPS acquisition, transcription latency, and battery use. After that, the next slice of work is the deferred list: sync, delete, undo, and LLM-based entity extraction via Ollama. What I've learned building this is that the open pieces aren't decoration — they're the reason the app works where it's meant to be used, and Hacktoberfest is the right place to share that with people who care about the same thing.

CscGrader and generateRubrics: Open-Source Tools for Faster CS Grading

Johnny Santamaria — Sun, 04 Oct 2026 23:26:53 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I built a two-part toolkit to solve the biggest time-sink in teaching large CS courses: grading massive, complex student projects. This isn't about auto-grading, which often fails for creative or multi-part assignments. Instead, it’s about automating the tedious, mechanical work so instructors can focus on providing meaningful feedback.

I built this for my friend, a CS professor who was spending entire weekends manually compiling, running, and debugging hundreds of student submissions for a single assignment.

The problem is simple: when you have 150 students submitting multi-file Java projects, just getting each one to compile and run with the correct test input is a logistical nightmare. It’s slow, error-prone, and mentally draining.

My solution is a two-part open-source toolkit:

CscGrader: A language-agnostic tool that automates the "collection of evidence" phase. It detects student projects, compiles them, runs them with predefined inputs, and captures all the output (stdout, stderr, exit codes, timing).

generateRubrics: A simple utility that takes a single master rubric file (Excel, PDF, etc.) and generates a personalized, ready-to-grade copy for every student on the roster.

Together, these tools transform a weekend of grunt work into a streamlined, semi-automated workflow, freeing up my friend to do what they do best: teaching.

Demo

The tools are designed for a command-line workflow. Here’s how they would be used together for a single assignment:

Step 1: Generate all the grading rubrics for the class.

bash

studentNames.txt contains a list of all students in the class

python3 generateRubrics.py -inputFile studentNames.txt -fileToCopy "Assignment-03-Rubric.xlsx" --assignment 03
This instantly creates a Rubrics/ folder with perfectly named files like JaneDoe-Assignment-03-Rubric.xlsx.

Step 2: Process all student submissions to collect evidence.

bash

The submissions folder contains one subfolder per student

The assignment profile tells CscGrader what to look for and how to run it

cscgrader batch ./submissions/ --assignment ./csc215-assignment-03.json --input test-input.txt
This processes every submission, compiles the Java code, runs it with the same test input, and saves a structured JSON file with all the results for each student.

Step 3: Grade.

My friend now opens the personalized rubric for a student in one window and the corresponding JSON evidence file in another. All the "did it compile? did it run? what was the output?" questions are already answered. They can now focus on code quality, design, and feedback.

Code

Here are the two repositories:

CscGrader: https://github.com/johnny603/CscGrader

generateRubrics: https://github.com/johnny603/generateRubrics

How I Built It

This project is a prime example of using open-source tools for practical, targeted automation.

Core Logic: Both tools are written in Python, relying only on the standard library. This was a deliberate choice to make them as portable and easy to run as possible for instructors who may not have admin rights or want to deal with virtual environments. The generateRubrics tool treats files as binary data, so it can copy .xlsx rubrics without needing libraries like openpyxl.

AI-Assisted Development: I used an open-weight model as a coding partner throughout the process. It was instrumental in designing the modular architecture of CscGrader, specifically the idea of using "assignment profiles" (JSON files) to make the tool reusable across different courses and languages.

Agent Harness: I used an open-source agent harness to automate the testing of CscGrader. The agent would create dummy Java projects with common errors (e.g., syntax errors, missing dependencies, ClassNotFoundException), run CscGrader on them, and verify that the tool correctly identified the failure category. This saved me hours of manual testing and made the tool far more robust.

The architecture of CscGrader is built around a modular pipeline: Detection → Compilation → Execution → Evidence. This is all orchestrated by a central runner, which uses language-specific "adapters" to handle the unique aspects of each programming language. The agent harness was key to building and validating this complex workflow.

Why Does Open Innovation Matter?

My Agent Session

Prize Categories

Using AI as a Learning Tool: Lessons From the Classroom, STEM, and Software Engineering

Johnny Santamaria — Thu, 01 Oct 2026 04:01:57 +0000

"Hello, I just wanted to check in..."

I did not start thinking seriously about AI from one role. I started seeing it differently as I moved between being a computer science student, teaching STEM, working as an LA/DL, doing educational research, and working at a robotics startup. Each experience showed me a different part of the same problem: how can we use technology to make people better at learning and solving problems without taking the problem-solving away from them?

As an Instructional Student Assistant at San Francisco State University, I work with students on programming logic and software engineering while developing Python automation for academic workflows. As an E-GAISE research assistant, I have analyzed submissions from 34 student teams and contributed to research involving AI-integrated grading systems. Before that, I worked as an Engineering Operator at a robotics startup, where I built Python tools for data ingestion, labeling, and validation.

These experiences led me to my main thesis: AI is most valuable in STEM education when it provides context-aware assistance, creates opportunities for practice and feedback, and keeps humans responsible for reasoning and verification. A student can create a diagram before asking AI for design feedback. An instructor can provide course objectives and common mistakes before using AI to create a Kahoot, Blooket, or debugging game. A developer can provide an existing codebase before asking AI to make a change.

AI Needs Context

One of the most useful ways I see AI being used in a classroom is by giving it context rather than starting with an empty prompt.

For example, imagine that a student is working on a software design assignment. Instead of immediately asking an AI, "How should I build this?", the student could first create a class diagram, flowchart, wireframe, pseudocode outline, or system design.

The student could then give that design to the AI and ask:

Does this design satisfy the requirements?
What relationships am I missing?
Is there unnecessary complexity?
What edge cases should I consider?
What could I improve?
What tradeoffs am I making?

This creates a different learning experience.

The AI now has context about what the student is thinking, and the student has already performed part of the intellectual work. The AI becomes something that can critique and question the student's thinking instead of generating the entire solution.

I think this is especially important in computer science because design is part of the learning process. If a student creates a diagram, writes pseudocode, or explains an algorithm before using AI, there is something concrete to evaluate.

The same principle applies to professional software development. A developer can provide an existing codebase, architecture diagram, API specification, test case, or error log before asking AI for help. The more useful question is not simply, "Can AI write this?" but "What context does AI need to understand what I am trying to accomplish?"

AI and Gamified Learning

Another classroom application I have become interested in is using AI to create short, bite-sized learning activities.

During my LA/DL sessions, I use activities similar to Kahoot and Blooket to review programming concepts and assignments. I have found that these activities can make a 50-minute session more interactive because students are not simply listening to a lecture.

AI could help instructors create variations of these activities.

For example, I could provide AI with the assignment objectives, course concepts, common student mistakes, and examples of the code being taught. AI could then help generate questions for a Kahoot, Blooket, or another classroom game.

The instructor would still need to review the material before using it, but AI could make it much faster to create several versions of an activity.

The activity could also go beyond a traditional multiple-choice quiz. A programming class could have a Code Detective game where students receive clues about a programming bug and identify the problematic line. Another activity could ask students to predict what a program will output before running it. Students could work in teams to determine which algorithm would be most appropriate for a particular problem.

The goal is not to use AI simply because it is available. The goal is to use it to create more opportunities for students to practice.

This connects to my previous experience teaching STEM, where I used project-based activities involving coding, game design, and robotics. I learned that students can understand technical concepts differently depending on how the material is presented. A short game, demonstration, or hands-on activity can sometimes create a better entry point than another explanation.

AI gives educators another way to experiment with those activities.

AI as a Learning Assistant

AI can also be useful when I am the student.

When I encounter a difficult programming problem, I do not necessarily want an AI system to give me the finished solution. Doing that can remove one of the most important parts of computer science education: learning how to reason through a problem.

Instead, I can ask questions such as:

What concept am I misunderstanding?
What edge case am I missing?
Can you give me a hint without giving me the solution?
Why does my approach fail?
What should I test next?
Can you explain this error differently?

I can also give AI my own work first.

For example, I might write pseudocode, create a class diagram, implement a first version, or explain my reasoning before asking for feedback. This gives the AI something concrete to evaluate.

It also forces me to confront my own understanding before asking for help.

That is an important distinction between using AI to learn and using AI to avoid learning.

What My LA/DL Experience Taught Me

My experience as an LA/DL has made this distinction even clearer.

A student does not always need more information. Sometimes they need the information presented differently. Sometimes they need someone to ask a question that helps them identify their own mistake.

For example, if a student is confused about tracing code, I might change the activity and have them diagram the execution of the program. If several students make the same mistake, I can turn that mistake into a group debugging activity.

I have also learned that engagement matters. A 50-minute session is relatively short, so I try to structure the time around an icebreaker, a game or review activity, assignment content, demonstrations, and a workshop where students can ask questions and begin working.

AI could help me create more variations of these activities, but it does not know my students in the same way that I do.

I still have to decide:

What should students learn?
What is appropriate for their current level?
What misconceptions are they experiencing?
How much content can realistically fit into 50 minutes?
Which activity will encourage participation?
Is the material appropriate for this particular class?

This is where the human role remains important. AI can assist with preparation, but the instructor still has to understand the students and make the educational decisions.

AI in Educational Research

My research experience has given me another perspective on AI in education.

At San Francisco State University, I analyzed submissions from 34 student teams across two semesters using Python and Pandas. The work involved extracting chat logs and reflective writing and analyzing performance across multiple rubric dimensions. I also contributed to research involving AI-integrated grading systems and educational technology.

I have also worked on an E-GAISE agentic grading pipeline that uses Python, LangGraph, Pandas, and Pydantic to evaluate submissions against a rubric. The system uses evidence and validation mechanisms rather than simply accepting an AI-generated evaluation as correct.

This experience has made me more interested in how AI is integrated into a workflow, rather than simply whether AI is being used.

For example, an AI-assisted grading system needs a clear rubric, evidence from the student's work, and a way to evaluate whether its conclusions are consistent. The important question is not whether AI can produce a grade, but whether the overall process produces useful and defensible feedback.

That perspective connects closely to my experience in software engineering.

Human-in-the-Loop: What I Learned From Robotics

My experience at a robotics startup helped me understand why this distinction matters.

My work involved data ingestion, labeling, validation, and internal tooling. The technical problem was not simply getting software to run. The larger problem was making sure the data moving through the workflow could actually be trusted.

I learned that automation is valuable because it can reduce repetitive work and make systems faster. But automation can also make mistakes faster.

That lesson carries directly into AI.

An AI system can generate ten explanations in seconds, but if those explanations are incorrect, it has simply produced ten incorrect explanations faster. The solution is not necessarily to avoid automation. Instead, the workflow needs enough context, testing, and human judgment to determine whether the result is useful.

This gives me a model for how I want to use AI:

Context → AI assistance → Human review → Feedback → Iteration

The same workflow can apply to a classroom activity, an AI-generated explanation, an automated grading system, or a software development task.

The human does not have to perform every step manually. Instead, AI can handle parts of the process while the human remains responsible for understanding the goal and deciding whether the result is acceptable.

The Risk of Overusing AI

The biggest concern I have with AI in education is that convenience can become a substitute for learning.

If a student asks an AI to solve every programming assignment, they may complete the assignment without developing the ability to solve a similar problem independently.

There is a major difference between:

"Give me the answer."

and:

"Here is my design. Can you help me find weaknesses in it?"

The second approach preserves the student's role in the learning process.

The student has made decisions, created an artifact, and thought about the problem. AI becomes a source of feedback instead of a replacement for reasoning.

This is why I think educators should focus less on simply asking whether students are using AI and more on teaching students how to use it responsibly.

Students should know how to question AI output, test generated code, verify information, recognize incorrect assumptions, and explain their own work.

What AI Has Taught Me About Learning

My experiences have changed my definition of effective AI use.

I used to think about AI primarily as something that could make tasks faster. Now I think about it more as something that can create additional opportunities for experimentation.

As a student, I can use AI to challenge my reasoning.

As an educator, I can use it to brainstorm activities, create practice problems, and develop different explanations.

As a researcher, I can investigate how AI changes educational workflows and how those systems can be evaluated.

As a software developer, I can use AI to investigate problems while still being responsible for the resulting system.

In all of these situations, the most valuable workflow is not:

Human → AI → Answer

It is closer to:

Human → Context/Design → AI → Feedback → Human Decision → Iteration

That distinction is important.

Conclusion

AI will continue to change STEM education and software engineering, but I do not think the future should be about replacing the human parts of learning.

My experiences in education and software engineering have taught me that context, iteration, feedback, and human judgment matter.

In the classroom, that could mean creating a design before asking AI for feedback. It could mean using AI to turn common programming mistakes into a short debugging game. It could mean generating a Kahoot or Blooket from course material and then adapting it to the needs of the class.

For a student, it could mean creating pseudocode before asking AI for implementation feedback. For a developer, it could mean giving AI enough information about an existing system before asking it to make a change. For an educational researcher, it could mean designing an AI-assisted grading workflow around evidence and clear evaluation criteria.

These approaches have something in common: AI is helping with the process, but humans remain responsible for the thinking.

That is the lesson I am carrying forward from my work as a student, educator, researcher, and engineer.

AI should not make us think less.

It should give us better ways to think, practice, test, create, and learn.

Lessons from a Robotics Startup: What I Learned About Data Pipelines

Johnny Santamaria — Sun, 09 Aug 2026 00:01:46 +0000

"Smile because it happened" — Dr. Seuss

The Setup

Earlier this year, I took on a short-term trial role with an early-stage robotics startup. The premise was straightforward: help with data collection, annotation, and evaluation workflows—essentially the backbone of any modern robotics or embodied-AI system.
The trial didn't work out long-term. I was let go after about two months — a decision that, honestly, came down in part to my bandwidth as a student. Balancing a full course load with a startup trial was harder than I anticipated.
But that's not the story I want to tell. What I do want to share are the technical lessons I took away — lessons about building robust data pipelines, about the gap between theory and practice, and about what I'd do differently next time.

These aren't company secrets. They're about the general engineering challenges that anyone working with robotics data pipelines will encounter — challenges I'd read about in papers but hadn't truly internalized until I was standing in front of them.

1. The Data Pipeline Shape Is Universal—But the Details Aren't

If you've spent any time in ML or robotics, you've seen this described:
Data Collection → Annotation → Evaluation
It's a standard three-stage pipeline. Industry vendors describe it explicitly in their robotics content. Academic projects model this structure. It's the field's shared vocabulary. Companies such as Scale AI and Toloka use similar industry workflows involving data collection, annotation, and evaluation.
What isn't shared are the specifics: the sensor setup, the calibration procedures, the annotation rubric, and the evaluation metrics. Those are where a company's IP lives. The pipeline shape? That's just the map. And the map is public.

What I'd do differently: Simulate before you collect. Data collection is expensive — in time, hardware wear, and cognitive load on operators. Before running a full session, run a feasibility study with a small batch. Verify your sync and capture scripts. I assumed this was standard practice. It wasn't. The cost of that assumption showed up as rework.

2. Annotation Granularity: More Isn't Always Better

There's a natural temptation when designing an annotation schema: capture everything. Every possible label, every edge case, every attribute you might want later. This is a trap.

Over-granular annotation creates:

Higher operator cognitive load → slower throughput, more errors
Greater inconsistency → more labels means more disagreement
Fragile evaluation → you're evaluating against things you might not need
The better approach? Start with the minimal viable schema that answers your core research question. Add complexity only when the data tells you it's necessary.

A concrete heuristic: If you're designing labels for a demonstration-collection task, ask yourself: "Will I actually use this attribute to decide if the demonstration is successful?" If the answer isn't an immediate yes, drop it. Retrofitting granularity is easier than cleaning up inconsistent over-labeling.

3. Automation Still Needs Oversight—Especially in Evaluation

Automation is seductive. The idea that you can "just set up an automated evaluation pipeline and let it run" is appealing, especially in a startup where resources are tight.
I learned the hard way that automated evaluation is not a replacement for human judgment. It's a filter.

Automated checks catch obvious failures (missing data, format errors).
They don't catch subtle issues (context-dependent errors, edge cases).
They create a false sense of confidence without regular spot-checks.
The workflow that actually works: Run your automated eval, spot-check a sample of what passes, find patterns in the failures, add those to your automated checks, and repeat. It's the "human-in-the-loop" pattern. Easy to forget when you're moving fast.

What I'd Tell My Past Self

If I could go back to Day 1 of that trial, here is exactly what I'd say:
Ask for explicit success criteria upfront. "Good work" means different things to different people in different contexts. Don't assume alignment. Get it in writing, or at least in a clear verbal agreement.

Simulate before you scale. A small batch of test data can save days of rework. Verify your entire pipeline end-to-end with a single example before you collect a hundred.
Design for iteration, not perfection. Your first annotation schema will be wrong. That's fine. Make it easy to change. Don't fall in love with your first draft.

Trust automation as a filter, not a replacement. It will never catch everything. Spot-check everything that matters, especially early on.
After the fact, write yourself a retrospective — from memory, not from any retained materials. Capture what worked and what didn't while it's fresh. You'll thank yourself later when you're interviewing or planning your next project.

Final Thoughts

Getting let go from a trial role is never fun. It stings, and there's always the temptation to replay the "why."

But the technical lessons? Those are mine. They're transferable. They're not tied to any particular company's proprietary stack.
A startup trial is an accelerated learning environment — whether it works out or not. The key is to separate what you learned about yourself and your craft from what you learned about that specific company's internal operations. The former is valuable, portable, and belongs to you.
The latter is their business.

Now, over to you:
What's one technical or operational lesson from a short-term role that completely changed how you approach your work today? I'd genuinely love to hear your stories in the comments — drop one below. 👇

From Kilowatt-Hours to Carbon Counts: Shaping Accurate Energy Reporting with OED

Johnny Santamaria — Wed, 10 Jun 2026 00:03:13 +0000

"garbage in, garbage out" - Some quote I heard before

Introduction

Every building tells a story through its energy usage.

A classroom left lit overnight, an HVAC system running longer than expected, or a sudden spike in electricity consumption across campus, these patterns often go unnoticed without the right tools to visualize them. That’s where the Open Energy Dashboard (OED) comes in.

OED is an open-source platform that helps organizations transform raw utility data into meaningful insights through charts, analytics, and real-time monitoring.

Schools, universities, sustainability teams, and facility managers use the dashboard to better understand how energy and resources are consumed across their buildings. Instead of sorting through spreadsheets or complicated meter data, users can quickly identify inefficiencies, compare trends, and make informed decisions that reduce both operational costs and environmental impact.

The development build is similar to the production build. In order to run development, you must install Docker which is a container for images, like screenshots of a build you can run on your particular operating system. Here is the app running using ‘docker compose up’, and to quit the browser I used ‘docker compose down’.

The Issue

issue link

In this contribution, I worked on LR18, an integration test focused on validating energy conversion behavior in the readings API. The test ensures that 15-minute interval energy readings can be correctly converted into kg CO₂ values while maintaining proper daily aggregation over large time ranges. While the change may seem small from the outside, it introduced interesting challenges involving fixture setup, unit conversions, expected CSV outputs, and backend testing consistency across the codebase.

The following files and resources were the most important locations in the codebase for understanding and implementing the LR18 integration test:

src/server/test/web/readingsLineMeterRangeQuantity.js This was the primary file where the LR18 test case was implemented. The file contains integration tests for line chart range (min/max) quantity meter readings across different aggregation ranges and unit behaviors. I used the surrounding LR1–LR7 tests to understand the expected structure, helper utilities, API request format, and CSV validation workflow. LR18 was added here because it specifically validates range aggregation behavior combined with unit conversion into kg CO₂.
testing.md Documentation link

This design document was important for understanding the intended testing requirements and naming conventions used throughout the project. The document outlines the required test scenarios for readings APIs, including line range (min/max) tests, aggregation expectations, and expected behavior across different unit conversions and time ranges. It helped confirm that LR18 belonged in the line range quantity test suite and clarified how expected CSV outputs should be structured.

readings_ri_15_days_75.csv This readings dataset file contains the raw 15-minute interval test readings used for validating aggregation and conversion behavior. The LR18 test loads this dataset into the database during setup to simulate realistic meter readings over a long time range. These readings are later aggregated into daily points by the API.
expected_line_range_ri_15_mu_kWh_gu_kgCO2_st_-inf_et_inf.csv This expected CSV file represents the correct API response output for LR18 after daily aggregation and unit conversion into kg CO₂. The test compares the API response against this file using helper assertions to verify timestamps, minimum/maximum values, and aggregation correctness.
src/server/util/readingsUtils.js This utility file contains important helper functions and shared testing utilities used throughout the readings integration tests. Functions such as:

prepareTest()
parseExpectedCsv()
expectRangeToEqualExpected()
getUnitId() were essential for seeding test fixtures, loading expected CSV data, resolving unit IDs dynamically, and validating API responses against expected outputs.

src/server/test/web/readingsLineMeterQuantity.js This file was used as a reference implementation because it already contained an existing L18 conversion-based test. I compared its fixture setup, conversion seeding, and naming conventions against LR18 to ensure consistency between quantity tests and range quantity tests.
package.json

The package.json file was useful for identifying available npm scripts and test commands used to run the OED test suite. This helped with running targeted test cases and validating the LR18 implementation locally during debugging.

npm run testData This command was referenced in the testing documentation and used to load sample data into the system for validating the readings API behavior during local testing and debugging.

The Product/Codebase

Tech Stack

Redux Toolkit (RTK): Manages the application's state, ensuring that data remains consistent and synchronized across all parts of the dashboard. Handles things like user selections, fetched meter data, UI state, etc.

Plotly.js: Generates the interactive graphs and charts that visualize energy and resource consumption.

React: The core frontend library used to build the user interface. It renders components for dashboards, selectors, and controls.

Node.js: Runs the backend server, handling data API requests and managing business logic and authentication.

PostgreSQL: Stores all historical energy/resource data, meter definitions, user configurations, etc. Acts as the main database.

Express.js: Web server running on Node.js, exposes RESTful API endpoints for the frontend to request and update data.

TypeScript: Provides type safety and improved developer experience for both frontend and backend codebases.

Docker: Used to containerize and deploy the application and its dependencies for consistent environments (optional in some deployments).

Syatem diagram

Use case

“A user wants to see the CO₂ emissions for electricity usage over a selected date range.”

Step-by-step walkthrough:

User selects a building and a date range in the dashboard's UI (React) and clicks "View CO₂ Emissions."
The React UI dispatches an action to Redux Toolkit to update the state with the new selection.
Redux triggers an asynchronous operation (a thunk or saga) to fetch the relevant readings and emission data from the backend.
The browser makes a REST API request to the Node.js/Express API, specifying the building and date range.
The backend API receives the request. It queries the PostgreSQL database for all energy readings within the range and uses internal logic (data services) to convert those readings into kg CO₂, applying the project’s conversion formulas (the logic tested by the LR18 integration test).
The backend returns a JSON object with time-stamped CO₂ emissions data to the frontend.
The Redux store receives the data and the UI updates accordingly.
Plotly.js renders an interactive chart with the emissions data, letting the user explore and analyze the results visually.

Challenges

Challenge 1: Inconsistent Unit Conversion Setup in LR18

One of the main technical challenges I faced was diagnosing why the LR18 integration test did not match the expected API behavior for converting readings into kg CO₂. Originally, the test was still requesting the kWh graphic unit, even though project requirements and the expected CSV output specified kg CO₂.

Self-Directed Problem-Solving Attempts

Compared with Existing Tests: I reviewed LR1–LR7 to look for patterns in how units were handled and where API requests might be constructed differently.
Codebase Search: I searched for kgCO2 and related conversion logic in both backend tests and the conversion fixture seeding code to find established best practices.
Implementation Cross-Reference: I closely examined the relevant section in readingsLineMeterQuantity.js, focusing on how conversion fixtures were prepared and used, to spot differences.

Additional Help

I leveraged GitHub Copilot to reason through discrepancies, generating code suggestions and checking my test’s construction against unit-handling best practices.
I carefully read the project’s testing.md guidelines to ensure my approach to data seeding and test assertions matched the intended design.

Status

Resolved. The LR18 test setup was corrected: the test now requests the API with the correct kg CO₂ graphic unit and seeds the conversion fixtures to align with expectations. The test passes and provides accurate verification for future code changes.

Challenge 2: CSV Naming and Expected Output Consistency

I also struggled with inconsistent naming conventions for expected CSV files (kg_of_CO2 vs. kgCO2) when comparing actual output to reference data files. This led to confusion in test assertions and in locating previous examples for regression testing.

Self-Directed Problem-Solving Attempts

Repository Search: I searched the repo for both naming styles to see which was most commonly used.
Pattern Review: I surveyed surrounding tests (both quantity and range-based) to verify which format appeared in assertion logic and file loading calls.
Assertion Validation: I checked the test utilities (such as readingsUtils.js) to see if there were helper functions that auto-resolved or validated expected outputs.

Additional Help

I referenced conversion-based tests in readingsLineMeterQuantity.js to ensure consistency in both naming and output structure.

Status

Resolved. I updated the LR18 test’s CSV file expectations and assertions to use the kgCO2 convention, eliminating confusion across the codebase and achieving a passing assertion.

Challenge 3: Maintaining Consistent Branch Workflow (Using git cherry-pick)

A key workflow challenge our team encountered was maintaining a consistent branch structure during development. There were instances when new commits were accidentally made to the wrong branch, which could cause confusion, code conflicts, or integration issues if left unaddressed.

Self-Directed Problem-Solving Attempts

Manual Merging: Attempted to merge the changes or re-apply the edits manually onto the correct branch, but this was time-consuming and could introduce mistakes.
Branch Reset & Rebase: Explored using git reset and git rebase to move commits, but these approaches risked rewriting history or affecting unrelated changes.
Git Documentation Review: Consulted the official Git docs and several online tutorials to look for a safer, more targeted solution for moving just the desired commit(s).

Additional Help

Once I identified git cherry-pick as a possible approach, I confirmed the workflow with Copilot and checked with project teammates (if applicable) to ensure this was in line with team conventions.

Status

Resolved. I successfully used git cherry-pick to copy the intended commit(s) to the correct branch without affecting unrelated work. This helped maintain branch hygiene and enabled a clean pull request history.

Solution

To address the issue of ensuring accurate conversion of energy usage data to kilograms of CO₂ (LR18), I developed an integration test that simulates a user requesting CO₂ emissions data over a specific date range. I modeled my implementation outline using the structure and methodology established in existing test cases LR1–LR7, which demonstrated good practices for preparing test data, invoking the API endpoint, and checking results.

In my solution, I set up sample meter readings in the database, triggered the backend logic to request the calculated CO₂ values, and verified that the backend (Node.js/Express) returned the correct results according to the expected conversion rules. This test helps ensure the data displayed in the dashboard (via Redux Toolkit and Plotly.js in the frontend) will always be calculated correctly, catching any regressions as the codebase evolves.

I relied heavily on Redux Toolkit to manage the application state in the frontend, this ensures that, once the backend API returns the correct CO₂ data, the entire dashboard updates consistently and all visualizations generated by Plotly.js reflect accurate numbers.

On the backend, I focused on the API logic and data services component (see System Diagram: Node.js/Express and Data Services) that handles the data transformation and communicates directly with the PostgreSQL database. My test verifies that conversions at this backend layer, before the data even reaches Redux or Plotly, are correct.

I verified the solution using automated test runners via Mocha. After implementing the LR18 test, I ran the full test suite and confirmed that all tests, including my new one, passed. This proves that both existing and new requirements for data conversion are met.

See the full solution here

Keeping Your Secrets in Sync: Environment Lifecycle Management That Actually Works

Johnny Santamaria — Sat, 14 Feb 2026 01:25:14 +0000

Introduction

Keyshade is an open-source real-time secret and configuration management platform that helps developers securely store, sync, and rotate sensitive data like API keys, passwords, and environment variables across multiple environments. It uses end-to-end encryption and live updates to eliminate insecure .env file sharing, manual dashboard updates, and secret sprawl. Designed for teams and solo developers alike, Keyshade improves collaboration, auditability, and security by providing version history, access controls, and seamless integrations with modern deployment platforms.

Try it out on https://keyshade.io/

The Problem: Silent Failures in Cloud Integrations (#1254)

The problem was that the integrations with Vercel or AWS Lambda would only respond to updates in secrets and variables. It would completely ignore situations where the user had deleted the environment or renamed the environment. This meant that when a user deleted an environment in Keyshade, the secrets and variables would still be available in Vercel or AWS Lambda but would be orphaned. This presented a problem from a security point of view. Also, when the environment was renamed, the integrations would be out of sync.

Why This Bug Mattered to Production Teams

This problem was significant because of several factors:

Security Risk: When a user deleted an environment using Keyshade (like an environment in staging mode), the secrets became active in Vercel and AWS Lambda. This meant the secrets were exposed on another platform. That is, credentials became potential attack vectors.
Data Integrity: The disconnect between Keyshade’s environment state and the external platforms led to a situation where users couldn’t be assured of data integrity. It defeats the purpose of having a reliable tool like Keyshade.
User Experience: It will be the responsibility of the consumers of Keyshade to manually log in to the Vercel or AWS Lambda interface to clean up after environment changes, thereby defeating the purpose of the integration in the first place. Such an interface might discourage the use of Keyshade.
Production Readiness: The integrations must be prepared to handle the full lifecycle of environments, not only the secrets within static environments, for Keyshade to be considered viable in production, especially by teams with high rates of environment changes (e.g., CI/CD workflows creating ephemeral environments). It was marked as "priority: high" and "difficulty: 4." This was my hardest PR that took around two months.

Under the Hood: Key Code Changes

keyshade/apps/api/src/integrations/plugins/vercel.integration.ts
Purposes: Integrates the Vercel platform - responsible for syncing Keyshade secrets/variables to Vercel projects as environment variables.
Key sections to edit:
getPermittedEvents(). Added ENVIRONMENT_UPDATED and ENVIRONMENT_DELETED to the set of events the integration subscribes to.
added case handlers to switch of emitEvent() for routing environment lifecycle events.
New functions added: handleEnvironmentDeleted(): This cleans up project-level environment variables scoped to the deleted environment and breaks the integration relationship handleEnvironmentUpdated(): Renames custom Vercel environments if available (pro plan users) & update integration metadata

apps/api/src/integration/plugins/aws-lambda.integration.ts
Purpose: Implements the AWS Lambda integration, syncing Keyshade secrets to Lambda function environment variables
Key sections modified:
getPermittedEvents(): Added the two environment events
emitEvent() switch statement: Added routing for environment events
New functions added:
handleEnvironmentDeleted(): Removes environment-scoped keys from Lambda function configuration and disconnects the environment relationship
handleEnvironmentUpdated(): No-op handler (Lambda has no environment rename concept) that logs an audit event

apps/api/src/common/util.ts
Purpose: Added retryWithBackoff() helper function to handle transient failures when calling external APIs (Vercel/AWS)
Why important: Cloud APIs can fail temporarily; this ensures cleanup operations are resilient and retry with exponential backoff

apps/api/src/integration/reconciler.ts
Purpose: Lightweight reconciliation system that processes failed cleanup operations stored in integration.metadata.pendingCleanup
Why important: When retries fail after multiple attempts, the reconciler can later replay these operations, ensuring eventual consistency even if external APIs are temporarily unavailable

apps/api/src/event/event.types.ts
Purpose: Defines TypeScript types for event metadata
Relevant types:
EnvironmentUpdatedEventMetadata: Contains name and description fields for environment updates
EnvironmentDeletedEventMetadata: Contains the environment name for deletion events

Test files:
apps/api/src/integration/plugins/vercel.integration.spec.ts: tests covering environment lifecycle handlers
apps/api/src/integration/plugins/aws-lambda.integration.spec.ts: tests covering Lambda-specific handlers
apps/api/src/integration/reconciler.spec.ts: testing the reconciler's pending cleanup processing

Supporting materials/artifacts
Before the fix:
When a user deleted an environment in Keyshade, the integration would:

Not receive or process the ENVIRONMENT_DELETED event
Leave all environment-scoped secrets active on Vercel/AWS Lambda
Not update the integration metadata to reflect the environment was gone
Create orphaned credentials that users would have to manually clean up

Error behavior:
While there wasn't a specific error message, the silent failure was the problem. Users would delete an environment expecting it to be cleaned up everywhere, but the secrets would persist on external platforms with no warning or indication that cleanup failed.
After the fix:
The integration now:
Subscribes to ENVIRONMENT_UPDATED and ENVIRONMENT_DELETED events
Automatically removes environment variables from Vercel projects when environments are deleted
Removes environment-scoped keys from AWS Lambda function configurations
Updates integration metadata to stay in sync with environment name changes
Uses retry logic with exponential backoff for resilience against transient API failures
Persists failed operations in pendingCleanup for later reconciliation
Logs audit events for compliance and debugging

Key Implementation Detail:
For Vercel's ENVIRONMENT_UPDATED handler, there's a special consideration: Vercel only allows renaming custom environments on Pro plans, not the standard "Preview", "Production", or "Development" environments. The code includes logic to attempt the rename and gracefully handle cases where it's not supported.

Challenges

While preparing my pull request, I ran into several real-world challenges that highlight what contributing to an active open-source project actually looks like behind the scenes. The first issue was merge conflicts between my branch (attempt-1254) and the target develop branch. In a fast-moving repository with many contributors, this is almost inevitable. I resolved the conflicts by pulling the latest upstream changes and manually reviewing each conflicting section to understand both sides. My goal was to preserve other contributors’ work while ensuring my environment lifecycle handler implementation remained intact. After merging, I tested everything locally to confirm nothing regressed, then committed with a clear explanation of the resolution. It was a reminder that merge conflicts aren’t just mechanical fixes — they’re moments where you have to understand the intent of multiple developers and reconcile them thoughtfully.
The next challenge surfaced during code review. A maintainer questioned my use of any type casts in TypeScript, particularly when accessing integration.id and working with the pendingCleanup metadata array. They pointed out that this could introduce long-term technical debt and suggested using @ts-expect-error instead. That feedback was valuable. While the any casts were originally a temporary measure to unblock a critical bug fix, the reviewer highlighted how @ts-expect-error is self-validating: TypeScript warns you if the suppressed error disappears later, which prevents stale workarounds from lingering in the codebase. I acknowledged the concern, added TODO comments recommending a future typed refactor, and committed to either replacing any with @ts-expect-error where appropriate or creating a follow-up issue to properly type the metadata structures. This exchange was a great example of how open-source review culture improves not just correctness, but long-term maintainability.
The biggest blocker came from the CI/CD pipeline. My PR was failing two checks: a Snyk security scan and an API validation build. The frustrating part was that I couldn’t reproduce the failures locally. All my local tests passed — dependencies installed cleanly, the build succeeded, and every unit test ran successfully. But Snyk repeatedly failed in CI with errors related to workspace handling and package manager compatibility. The CLI didn’t fully support the project’s pnpm workspace setup in the same way the CI environment did, and I couldn’t access the GitHub Actions logs directly from my development environment. I tried multiple workarounds, including running Snyk through pnpm and downloading the standalone CLI, but nothing mirrored the CI behavior. Even rerunning the workflows with empty commits didn’t resolve the issue.
At that point, I documented everything I had tried in a detailed PR comment and asked for help. A maintainer with admin access reviewed the CI logs and identified the root cause: a vulnerability in a transitive dependency that required updating the root package.json and regenerating the lockfile. They also explained that the project’s Snyk integration uses a specialized workspace configuration that isn’t trivial to replicate locally. This led to a broader discussion about improving CI transparency for external contributors, including the possibility of automatically surfacing relevant log snippets in PR comments. The experience reinforced an important lesson: as an external contributor, you won’t always have full visibility into infrastructure, and clear documentation of your debugging efforts is essential for maintainers to step in effectively.

Solution (PR #1255)

After working through these issues collaboratively, the pull request was successfully merged. The final solution included environment lifecycle event handlers for both Vercel and AWS Lambda, retry logic with exponential backoff, a reconciliation system for eventual consistency, and comprehensive test coverage. All CI checks passed, and review feedback was fully addressed. More importantly, the contribution closed a high-priority bug that had been blocking production adoption of cloud integrations. It was a strong reminder that persistence, careful communication, and respect for the review process are just as important as writing the code itself. Open-source success isn’t just technical, it’s collaborative.

What Did I Learn?

How to read official Typescript documentation to break down confusing code
How to ask beneficial questions to maintainers
How to navigate CI/CD failures as an external contributor
Understanding Event-Driven API Architecture
Snyk improves security in development

Keyshade Debugging: Mastering Workspace Role Tests and API Repair

Johnny Santamaria — Sat, 08 Nov 2025 01:06:38 +0000

"A good tester prevents problems; a great tester finds them." Keith Klain, Director of Quality Engineering at KPMG UK

Introduction

Four disabled tests. One cryptic comment: 'flaky.' When I dove into Keyshade's workspace role tests, I discovered the failures weren't random at all—they revealed a fundamental misunderstanding of how the API validates permissions.

Keyshade is a modern solution for secure, centralized credential and configuration management, designed for developers, DevOps engineers, and open-source communities who need to manage secrets without the headache.

The Issue (#1122)

I decided to continue my open source journey on Keyshade since my blog post going over how I solved issue 998, which was that the Keyshade command line interface’s “keyshade run” command fails to give a proper error that users can understand. Now I decided to explore Keyshade’s API since I have explored how the CLI functions in its codebase.

I found an issue where included a bug in workspace role tests, which meant the problem stemmed from how workspace roles were being set up and asserted in tests, some test fixtures and assertions depended on implicit ordering and shared state, which made the tests fragile and allowed timing or state-dependent flakes to slip into CI (Continuous Integration). By making the setup deterministic and isolating role state for each test (explicitly creating the roles and their relationships, avoiding shared mutable fixtures, and tightening assertions to check the actual role assignments rather than relying on incidental data), the tests now reliably exercise the API behavior they were intended to cover.

The result is a more robust test suite that accurately guards against regressions in role handling for workspaces, reduces CI noise from flaky failures, and gives us greater confidence in future changes to authorization and workspace logic. Here are some key locations in the codebase we found out to be useful in solving the issue:

The apps/api/src/workspace-role/ folder includes the workspace-role feature module — contains controller, service, DTOs, and possibly unit/integration tests. Focus on business logic, dependency wiring, and any in-module role constants.

Examples:

Inspect the service methods (assignRole, removeRole, listRoles) for edge-case logic (no-op on duplicate role, default role applied on creation).
Check module exports and DI config: ensure the controller gets the expected implementation of the role service (mock vs real) in tests.
Example check: verify if role constants are defined here (e.g., 'admin', 'member', 'viewer') and whether they match values used in DB/entities or frontend.

The apps/api/src/workspace-role/workspace-role-controller.ts file includes the controller / HTTP handlers for workspace-role endpoints — review request DTOs/validation, how the controller calls services, status codes, and permission checks performed at the controller layer.

Examples:

Example request mapping: POST /workspaces/:id/roles -> controller.assignRole(req.params.id, req.body). Check expected body shape (userId, role).
Example validation: if controller uses decorators or a schema, show the required fields (e.g., { userId: string, role: 'admin'|'member' }).
Example permission check: controller may call guard like ensureWorkspaceAdmin(currentUser, workspaceId) — confirm where that check happens and what error it throws (403 vs 401).
The apps/api/src/workspace-role/workspace-role/workspace-role.e2e.spec.ts file has end-to-end tests for workspace-role flows — tests that spin up the app (or a test server), seed DB data or fixtures, run real HTTP requests, and assert full-stack behavior and responses.

Examples:

Test setup/teardown: show DB seeding (create users, create workspace) and cleanup steps to avoid cross-test leakage.
Example test case: an e2e test that POSTs role assignment and then GETs membership list to assert the new role is present; include expected status codes and response shapes.
Flakiness checks: look for timeouts, async waits, or hard-coded delays that may make these tests flaky (and thus failing intermittently).

General Diagram Walkthrough of the Issue

Reproducing the Problem

Here are steps regarding how I reproduced the problem before assessing my plan, implementation, and solution

Scenario 1 — should not be able to update the workspace role with environments that do not belong to the project Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project A in W and capture PROJECT_A_ID.
Create Project B in W and capture PROJECT_B_ID.
Create Environment A in Project A (ENV_A_ID) and Environment B in Project B (ENV_B_ID).
Create a workspace role R for Project A that includes only ENV_A_ID.
Configure your CLI to target workspace W and authenticate with a token that can update roles.
Use the CLI (or send an API request) to update role R to set environmentIds = [ENV_A_ID, ENV_B_ID].
Observe the response/error shown by the CLI.

Expected behaviour

The API should validate that ENV_B_ID does not belong to Project A and return a 4xx validation error (e.g., 400 or 422) with a clear, formatted message such as: "environment ENV_B_ID does not belong to project PROJECT_A_ID"
The CLI should print that formatted message to the user (no opaque stack trace or generic 500).
The update must be rejected and the role R must remain unchanged (still only ENV_A_ID).

Scenario 2 — should be able to add environment access for projects to the role with READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT authorities Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project P in W and ensure it has at least one environment (ENV_P_1, …).
Create a workspace role R (it may initially have no project-level environment access).
Obtain an auth token that has the authorities: READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT.
Configure your CLI to use workspace W and that token.
Use the CLI (or send an API request) to add project-level environment access for Project P to role R (the endpoint that grants project environment access).
Observe the response/error shown by the CLI and then GET the role R to inspect its environment access.

Expected behaviour

The API should accept the request and return 200/201 with a clear success body describing the new project-level environment access (or list of environmentIds that were added).
The CLI should print a readable success message and show the role now has access to Project P environments.
If the token lacks any required authority, the API should return 403 Forbidden with a clear formatted error and the CLI should print that message.

Scenario 3 — should be able to add environment access for projects to the role (permission variation) Steps to reproduce the behaviour:

Create (or use) a workspace W, a Project X with environments, and a role R.
Prepare two auth tokens:

* AUTH\_ADMIN: token with UPDATE\_WORKSPACE\_ROLE (and related read authorities).

* AUTH\_RESTRICTED: token without UPDATE\_WORKSPACE\_ROLE.

Configure your CLI to use workspace W.
With AUTH_ADMIN, run the CLI command (or API request) to add Project X environment access to role R. Verify success.
With AUTH_RESTRICTED, attempt the same CLI command (or API request) to add Project X environment access to role R. Observe the response/error shown by the CLI.

Expected behaviour

With AUTH_ADMIN: API returns 200/201 and the role R gains access to Project X environments; CLI prints a readable success confirmation.
With AUTH_RESTRICTED: API returns 403 Forbidden with a clear formatted message like "missing authority: UPDATE_WORKSPACE_ROLE" and the CLI prints that message; role R must not be modified.

Scenario 4 — should not be able to create workspace role where environments do not belong to the project Steps to reproduce the behaviour:

Create (or use) a workspace W.
Create Project A and Project B in W.
Create Environment B in Project B (ENV_B_ID).
Configure your CLI to target workspace W with a token that can create roles.
Use the CLI (or POST API) to create a workspace role with:

* projectId = PROJECT\_A\_ID

* environmentIds = \[ENV\_B\_ID\]

Observe the response/error shown by the CLI.

Expected behaviour

The API must reject the create request with a 4xx validation error (e.g., 400 or 422) and a clear message such as: "environment ENV_B_ID does not belong to project PROJECT_A_ID"
The CLI should print that formatted error.
No role should be created.

Keyshade’s Codebase

Here is Keyshade's codebase and API related to this issue, and how would I use the code to diagnose and fix the failing workspace-role tests, displaying the importance of code organization.

High-level overview (relevant subsystems)

Schema / client shapes:
- packages/schema/src/workspace-role/index.ts defines the WorkspaceRole payload shape (projects + environments arrays, authorities, etc.). That tells you the expected request/response shape for workspace role create/update operations.
API service layer:
- apps/api/src/workspace-role/workspace-role.service.ts contains the logic that parses project -> environment inputs and applies DB operations. The key function is parseProjectEnvironmentsDBOperation which:
  - verifies that an environment slug exists and belongs to the provided project
  - throws NotFoundException if an environment is not part of the project
  - calls authorizationService.authorizeUserAccessToEnvironment to ensure the acting user has READ_ENVIRONMENT on that environment
Authorization checks:
- apps/api/src/auth/service/authorization.service.ts and apps/api/src/auth/service/authority-checker.service.ts implement the authority checks and higher-level authorization plumbing. authorizationService exposes methods such as authorizeUserAccessToEnvironment and authorizeUserAccessToSecret; these both call into the authority checker and also verify workspace membership.
Tests & client usage:
- packages/api-client/tests/workspace-role.spec.ts contains tests that exercise the workspace-role APIs from a client perspective (create workspace, create project, create environments, then create/update roles).
- apps/platform and controller-instance reference the WorkspaceRoleController from the generated API client; you can use the API client or the CLI to reproduce e2e scenarios.

Why the tests were failing (what the issue description lists)

The failing/disabled tests are focused on cases where environments provided for a workspace role do not belong to the project, and cases that require multiple authorities (READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT).
The service code enforces two separate checks:

1. Ownership check: the environment must belong to the provided project — otherwise parseProjectEnvironmentsDBOperation throws NotFoundException with a message like "Environment X is not part of project Y".

2. Authorization check: after confirming the environment belongs to the project, the service calls authorizationService.authorizeUserAccessToEnvironment which requires the user to have READ\_ENVIRONMENT on that environment (and also checks workspace access).

If tests tried to use an environment that belongs to a different project, the service will intentionally return 404/NotFound before attempting to grant access; tests must assert that behavior.
If tests expected that adding environment access requires specific authorities, the test setup must give the test-user those authorities (or use a workspace-admin) before calling the endpoint.

How to use the codebase to reproduce, diagnose, and fix the tests

Reproduce the failing case manually (fastest way to see precise error):

* Use packages/api-client/tests/workspace-role.spec.ts as a reference for how tests set up the workspace/project/environment/role chain.

* From an integration/e2e test or an API client, create:

    * workspace A

    * project P1 inside workspace A

    * project P2 inside workspace A

    * environment E1 in P1 and environment E2 in P2

* Then attempt to create/update a workspace role that includes project P1 but lists environment E2 (an environment that belongs to P2).

* Observe the response: workspace-role.service.ts is expected to detect the mismatch and return a 404 Not Found with message "Environment ... is not part of project ...".

Check the authority path:

* If you provide a correct project-&gt;environment mapping, but the requestor lacks READ\_ENVIRONMENT (or other expected authorities like UPDATE\_WORKSPACE\_ROLE), authorizationService.authorizeUserAccessToEnvironment will throw Unauthorized/Forbidden. Ensure test user has the required authorities when the test expects success.

Fix the tests (common issues & fixes):

* If a test was disabled because it used an environment from a different project but expected a different error (or success), restore the test and:

    * Either adjust the test to expect NotFound (because parseProjectEnvironmentsDBOperation intentionally rejects cross-project environments).

    * Or adjust test fixtures so the environment is created in the same project as the project entry in the request.

* For authority-related tests:

    * Make sure the test user has the necessary authorities before calling the API. Grant a role to the test user with READ\_WORKSPACE, UPDATE\_WORKSPACE\_ROLE, READ\_PROJECT, READ\_ENVIRONMENT when the test expects role creation/update to succeed.

    * If a test is verifying denial paths (should not be able to create/update), ensure the user has fewer/missing authorities and assert the API returns Unauthorized/Forbidden appropriately.

Where to change code if the behavior is incorrect:

* If you decide the server should validate authority first (instead of ownership first), modify parseProjectEnvironmentsDBOperation ordering; but be careful: the current behavior (check membership/ownership first, then authorization) is consistent with tests that expect NotFound for cross-project envs.

If messages or status codes are inconsistent with tests, adjust the thrown exceptions in parseProjectEnvironmentsDBOperation (NotFoundException) or in authorizationService (Unauthorized/Forbidden) to align with the test expectations — but prefer changing tests to reflect correct domain behavior unless an API bug is proven.

Useful files to inspect or instrument:

* packages/schema/src/workspace-role/index.ts (payload schema)

* apps/api/src/workspace-role/workspace-role.service.ts (parseProjectEnvironmentsDBOperation and create/update logic)

* apps/api/src/auth/service/authorization.service.ts (authorizeUserAccessToEnvironment)

* apps/api/src/auth/service/authority-checker.service.ts (lower-level authority checks)

* packages/api-client/tests/workspace-role.spec.ts (test setup and expected assertions)

Concrete actionable checklist to resolve the issue in the repository

Re-enable the disabled tests.
For each disabled test, verify which of the two server behaviors it is validating:
- Ownership validation (environment belongs to project) → expect 404 Not Found. Ensure test builds a request with a mis-matched environment and asserts a 404 and the message text if desired.
Authorization validation (user needs the listed authorities) → ensure test grants the required authorities to the requester when expecting success; when expecting failure, ensure the user lacks at least one required authority and assert Unauthorized/Forbidden.
Run the tests locally (or in CI):
- The workspace-role e2e test file: packages/api-client/tests/workspace-role.spec.ts is a good starting point.
- Add logging in workspace-role.service.ts parseProjectEnvironmentsDBOperation if you need to trace why an environment was rejected.
If the test failure shows unexpected status codes or messages, adjust the service's thrown exceptions to match the documented API or update the tests to assert the documented behavior.
If you change server code, add unit tests in apps/api for parseProjectEnvironmentsDBOperation to cover:
- environment not in project → NotFound
- environment in project but user lacks READ_ENVIRONMENT → Unauthorized/Forbidden
- environment in project and user has authority → success path produces correct DB ops

Short example of the expected behavior (mapping to the tests listed in the issue)

"should not be able to update the workspace role with environments that do not belong to the project":
- Call update role with project P1 + env from P2 → server should return 404 Not Found (environment not part of project).
"should be able to add environment access for projects to the role with READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT authorities":
- Ensure requestor has those authorities (assign a role in test fixture) → call create/update and assert success and that the role contains the project->environment mapping in response.
"should be able to add environment access for projects to the role":
- Same as above but likely using normal granted authorities; ensure fixtures grant appropriate permissions.
"should not be able to create workspace role where environments do not belong to the project":
- Create role payload with mismatched environment → expect 404 Not Found.

Where to look in code to make changes or confirm behavior

parseProjectEnvironmentsDBOperation in:
- apps/api/src/workspace-role/workspace-role.service.ts
authorization plumbing:
- apps/api/src/auth/service/authorization.service.ts
- apps/api/src/auth/service/authority-checker.service.ts
request/response schema:
- packages/schema/src/workspace-role/index.ts
tests:
- packages/api-client/tests/workspace-role.spec.ts

Use Case

Goal: Create/update a workspace role that grants access to specific projects and, within those projects, specific environments.
Client payload shape (what the API expects)

1. name, description, authorities (array), projectEnvironments: \[{ projectSlug, environmentSlugs: \[slug\] }\]

Server behavior (what the code enforces)

1. For each projectEnvironments entry, resolve projectSlug -&gt; projectId (getProjectSlugToIdMap).

2. For each environmentSlug in environmentSlugs:

    * Verify that environmentSlug exists AND belongs to that project (prisma.environment.findFirst where slug + projectId). If not, throw NotFoundException (404) with a body "Environment not found" / "Environment X is not part of project Y".

Authorize the acting user has READ_ENVIRONMENT for that environment (authorizationService.authorizeUserAccessToEnvironment). This also verifies workspace-level access and permitted authorities (via authorityCheckerService).

Finally, upsert projectWorkspaceRoleAssociation rows connecting roleId<>projectId and connect the environment slugs.

Why the workspace-role tests were failing

The failing/disabled tests covered two distinct types of validation:
- Ownership validation: environment must belong to the project in the request. If not, server should reject with 404 (Not Found).
- Authorization validation: even if environment belongs to project, the acting user must have proper authorities (READ_ENVIRONMENT, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_WORKSPACE_ROLE) to add environment access to a role.
Typical causes for failures:
- Tests that assert 404 were previously commented out or used incorrect fixtures (e.g., environment not created, or created but in same project), so the server behavior didn't match test expectations.
- Tests that assert successful updates weren't ensuring the test actor had the required authorities (so server returned Unauthorized/401/403 instead of 200).
My PR re-enabled those tests and fixed fixtures by:
- Creating environments in the correct projects (dev, stage).
- Explicitly setting the membership role authorities (e.g., update the workspace member role to include READ_ENVIRONMENT and UPDATE_WORKSPACE_ROLE) when the test expected updates to succeed.
- Using consistent test header 'x-e2e-user-email' (alice.email) for requests.

Walkthrough of the two failing test categories and what to change

"Environment doesn't belong to the project" (ownership)

Server side: parseProjectEnvironmentsDBOperation checks project membership:
- It runs prisma.environment.findFirst({ where: { slug: environmentSlug, projectId } })
- If not found, it throws: throw new NotFoundException(constructErrorBody('Environment not found', Environment ${environmentSlug} is not part of project ${pe.projectSlug}))
Tests expecting 404 must:
- Create a workspace and two projects in that workspace.
- Ensure the environment used in the request was created in THE OTHER project, or not created at all (but prefer creating in other project to avoid false positives).
- Make a POST/PUT with projectSlug set to projectA and environmentSlugs containing an environment slug that actually belongs to projectB.
- Assert statusCode === 404.

"Adding environment access requires specific authorities" (authorization)

Server side: after ownership check, parseProjectEnvironmentsDBOperation calls: await this.authorizationService.authorizeUserAccessToEnvironment({ user, slug: environmentSlug, authorities: [Authority.READ_ENVIRONMENT] })
That code path will call authorityCheckerService.checkAuthorityOverEnvironment which checks the acting user's permitted authorities for that environment (it can be from project-level or workspace-level roles).
Tests expecting success must ensure the acting user has:
- READ_WORKSPACE (if required by the flow), UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT — in practice, the tests can mutate an existing workspace role for the test user to include these authorities and also connect appropriate project/environment associations so the user has project-scoped authority.

Exact test changes — enable one disabled test and adjust fixture

Below is a concrete enabled test snippet for packages/api-client/tests/workspace-role.spec.ts that re-enables the “should not be able to create workspace role where environments do not belong to the project” scenario.
This patch enables the test and makes the fixture explicit: create two projects, create an environment in projectB, then attempt to create a workspace role that references projectA + the environment slug from projectB. The test asserts 404.

packages/api-client/tests/workspace-role.spec.ts

// (This is the test snippet to add/enable within the existing test suite)

it('should not be able to create workspace role where environments do not belong to the project', async () => {

// Precondition: two projects exist in the workspace

// projects[0] -> projectA, projects[1] -> projectB (test fixture established earlier in the file)

// Create an environment in projectB that should NOT be valid for projectA

await prisma.environment.create({

Notes about placement:
- Put this enabled test near the other workspace-role create tests after the workspace + project fixtures are created and available.
- Ensure projects variable references are the ones created during beforeAll/beforeEach steps in that test file.

A directly analogous enabled e2e test (what PR 1156 did)

PR 1156 enabled very similar tests in apps/api/src/workspace-role/workspace-role.e2e.spec.ts — it:
- Created dev/stage environments in the proper projects,
- Updated the Member workspace role to add project associations and required authorities for the actor,
- Re-enabled the previously commented tests and asserted status codes 200 or 404 as appropriate.

Small server-side unit test to lock in parseProjectEnvironmentsDBOperation behavior

I recommend adding a unit test that calls parseProjectEnvironmentsDBOperation (or exercises the public create/update API) with a project-environment mismatch and asserts a NotFoundException is thrown.
Example test file (unit) to confirm rejection for cross-project environment:

apps/api/src/workspace-role/workspace-role.service.spec.ts

import { Test, TestingModule } from '@nestjs/testing'

import { WorkspaceRoleService } from './workspace-role.service'

import { AuthorizationService } from '@/auth/service/authorization.service'

import { PrismaService } from '@/prisma/prisma.service'

import { NotFoundException } from '@nestjs/common'

Proposed server code changes — minimal / optional

The current server code in parseProjectEnvironmentsDBOperation is already correct about ownership-first validation (it checks slug + projectId before calling authorizeUserAccessToEnvironment). The PR you provided does not require a behavior change on the server — tests should reflect the server semantics.
If you did want to change server behavior, the main two options are:

1. Keep current semantics (ownership check first, then authorization). Pros: deterministic 404 for cross-project envs; avoids running authorization queries for environment slugs that don't belong to the provided project. Cons: reveals that the environment slug exists in a different project (server replies 404 with message referencing project mismatch).

2. Run authorization first (i.e., check the caller's permissions to read the environment before checking project membership). Pros: could collapse some error channels and avoid disclosing cross-project existence if the caller lacks permission. Cons: often you still need to fetch the environment by slug (so you still learn about its existence) — and the authorization check typically requires fetching the environment anyway to know project/workspace id. It also changes the status codes returned for the cross-project case (may return 401 instead of 404), which is a breaking API semantics change for clients that currently rely on 404 in that case.

Recommendation: do NOT change the ordering of checks unless there's a clear security/privacy reason. The current behavior (404 on ownership mismatch) matches the tests in PR 1156 and is explicit. Prefer to fix test fixtures and grant/revoke authorities in test setup rather than altering server semantics.

Why the PR's approach (what you already did in PR 1156) is correct

You re-enabled tests and fixed fixtures instead of changing server semantics. Specifically:
- Created missing environments in the appropriate projects before calling the API.
- Updated the test user/role authorities where tests expected success.
- Asserted the correct status codes: 404 for ownership mismatch and 200 for successful updates when authorities exist.
This is minimal and non-breaking.

Trade-offs if you change server behavior (ownership vs authorization ordering)

Changing the order to authorize first:
- May return 401/403 instead of 404 for the cross-project environment case — tests and clients must be updated.
- Could obscure whether an environment exists in another project (slight security benefit).
- Could introduce additional database lookups if you need to check membership as well; may not yield significant performance gain.
Keeping ownership-first (current):
- Clear, deterministic 404 response for mismatched environment ownership.
- Tests remain simple: assert 404 for the cross-project case.
- Slight information disclosure (you show that environment exists but in a different project), which may be acceptable in an internal system like Keyshade.

Summary of exact artifacts to change (concrete)

Add/enable the test in packages/api-client/tests/workspace-role.spec.ts as above (or in apps/api/src/workspace-role/workspace-role.e2e.spec.ts — PR 1156 already did the same).
Add a small unit test in apps/api/src/workspace-role/workspace-role.service.spec.ts to lock in behavior of parseProjectEnvironmentsDBOperation for cross-project envs (example above).
Ensure any tests that expect successful updates explicitly:
- create the environments in the projects they want to reference
- grant the test actor role(s) the required authorities (UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT) before calling create/update

Challenges

I reviewed PR #1156 and walked the tests and service code. Below I list concrete challenges we encountered while making this PR and practical mitigations. I narrate what I checked and what I'd recommend next.

What I did (quick recap)

Inspected the workspace-role service logic (parseProjectEnvironmentsDBOperation) and the authorization flow.
Reviewed the test files that were re-enabled/modified in the PR (workspace-role e2e/spec files).
Confirmed the PR fixes were basically test-fixture and authority-setup issues (creating environments in right projects, granting the test user required authorities), not major server-behavior changes.
Verified the PR merged successfully but noted a set of practical challenges that often show up in this work.

Challenges encountered (with examples and mitigations)

Test fixture brittleness and DB state coupling

Problem: E2E tests depend on precise workspace/project/environment setup and roles. Small differences in fixture creation order or leftover DB rows cause flakiness.
Example: A test assumed "production" env exists in projectA but CI used a different test run that already had production in projectB -> unexpected 200 instead of 404.
Mitigation:
- Always create and assert fixtures in test setup (do not assume names).
- Use randomized slugs (e.g., slug-<timestamp>-<random>) to avoid collisions across parallel CI jobs.
- Add teardown/cleanup that removes created rows or run tests inside isolated databases/transactions.

Authority/permission complexity in tests

Problem: Tests that check success paths require several authorities (READ_WORKSPACE, UPDATE_WORKSPACE_ROLE, READ_PROJECT, READ_ENVIRONMENT). If the test doesn't explicitly grant them, the server returns Unauthorized and the test fails.
Example: Test enabling environment access but not updating the Member role to include UPDATE_WORKSPACE_ROLE.
Mitigation:
- Explicitly set up the actor’s roles/authorities per test case.
- Add helper functions in tests for "grantMemberAuthoritiesForEnvironmentAccess" to make fixtures clear.
- Keep negative/positive authority tests separate and deterministic.

Merge conflicts (simple but annoying)

Problem: Enabling tests and modifying the same test files or fixtures on different branches led to conflicts (e.g., two branches each uncommented different blocks of tests).
Example: Two branches changed apps/api/src/workspace-role/workspace-role.e2e.spec.ts: one enabled a test, another refactored a beforeEach. Merge resulted in conflict markers around large commented blocks.
Mitigation:
- Rebase frequently on the develop branch to reduce long-lived divergences.
- Split large test-enabling changes into smaller PRs (enable + fix fixtures + authority wiring).
- When resolving conflicts, run the full test suite locally before pushing the merge commit.

Creating temporary test files or helper tests "to be removed later" causing issues

Problem: Developers often create throwaway tests / temporary files to debug behavior and forget to remove them. These can be picked up by CI and cause nondeterministic runs or duplicate test names.
Example: Added apps/api/src/workspace-role/tmp-debug.spec.ts to iterate quickly, later removed locally but remained in another branch → CI fails or duplicate DB setup occurs.
Mitigation:
- Avoid committing throwaway test files. If necessary, keep them in local-only branches or protect with a naming convention excluded from CI.
- Add CI test discovery rules to ignore test files matching tmp-*.spec.ts, if you must have temporary tests.
- Review diffs carefully before commit and run git status to ensure no debug/test files are included.

Test duplication across packages (api-client vs apps/api)

Problem: Similar workspace-role tests existed in both packages/api-client/tests and apps/api e2e test files; enabling one copy and forgetting to update the other caused inconsistent behavior.
Example: One test in packages/api-client used the API client and different headers than the e2e server-side test; runtime differences led to mismatched assertions.
Mitigation:
- Consolidate canonical e2e tests in one place (prefer apps/api e2e that hits the server directly), keep api-client tests focused on client behavior.
- If duplication is unavoidable, keep test expectations aligned and use shared helpers/mocks.

Race conditions in setup/cleanup (parallel tests)

Problem: Creating projects/environments concurrently across parallelized test jobs (or even within same job) created timing-based failures.
Mitigation:
- Make tests idempotent and isolate data per test.
- Use unique slugs per test run.
- Consider serializing tests that share DB resources or use test DB instances per worker.

Error/status-code semantics and client expectations

Problem: Tests rely on specific status codes (404 vs 401). If server-side code ordering changes (ownership check vs authorization check) the codes change and many tests break.
Example: If you move authorization first, a cross-project environment case might return 401 instead of 404.
Mitigation:
- Keep server semantics stable; if a change is required, update all tests and document the change.
- Add unit tests that assert the contract (e.g., cross-project environment -> NotFound).

CI flakiness from environment variables and headers

Problem: Some tests rely on headers like 'x-e2e-user-email'. If CI environment variable email differs or parallel runs reuse the same email, tests can mutate shared state.
Mitigation:
- Generate a test-specific email per run (e.g., alice+<run-id>@keyshade.xyz).
- Make test harness read and propagate headers from the same fixture source.

Commit noise and history (multiple tiny commits while fixing test)

Problem: WIP commits to enable tests, debug, revert, add files, remove files created a messy branch and bigger diffs, increasing review overhead and merge conflicts.
Mitigation:
- Squash local fixup commits before opening PR or use interactive rebase to tidy commits.
- Keep PRs small and focused: one PR for enabling tests and fixtures, another for refactors.

Accidental API contract changes vs test fixes

Problem: When tests fail, there’s a temptation to change server code to make tests pass instead of fixing tests. That can mask real bugs or break clients.
Mitigation:
- Prefer fixing tests/fixtures if the server behavior matches documented contract.
- If server needs change, write unit tests to cover new contract and get reviewer approval.

Concrete, practical steps I recommend next

Add a small checklist to PR templates for test-enabling PRs:
- Confirm unique slugs in fixtures
- Confirm actor authorities are set explicitly
- Run full test suite locally (or run the specific e2e group) before pushing
- Clean up any temporary test files
Add a helper in tests for deterministic unique slugs and user emails to avoid collisions in CI.
Consider adding a lint/CI guard that fails the build when "tmp-*.spec.ts" or "debug" test files are found in the commit.
If you expect future merges to touch the same big e2e files, coordinate branches or split PRs (e.g., create "enable tests" PR and a follow-up "refactor fixtures" PR).

Solution

Here is a summarized plan of my solution:

Lock the contract with unit tests by adding unit tests that assert parseProjectEnvironmentsDBOperation rejects cross-project environment slugs with NotFound. This prevents accidental server reordering from changing semantics. Make tests deterministic by ensuring every test explicitly creates its fixtures (workspace, project, environment) and set unique slugs per run, which will explicitly grant the required authorities to the test actor in success-path tests.

Prevent accidental temporary test files by adding the pre-commit hook above and a CI check to refuse pushes that include tmp/debug test files. Keep good merge strategies and PR hygiene by keeping PRs small and focused (enable tests separately from refactors). Make sure to rebase frequently and squash debug commits and add a PR/TODO checklist that requires: unique slugs, explicit authority setup, no temporary files, and local e2e run before requesting review. Consolidate test suites by deciding canonical location for workspace-role e2e tests (prefer apps/api e2e) and remove duplicates, or centralize test helpers. Here is my final PR solving the issue over a month:

PR #1156

Conclusion

In conclusion, Keyshade’s clean, well-structured codebase and deterministic tests make development faster, safer, and more collaborative. Clear intent, consistent conventions, and reliable CI build trust between maintainers and contributors—enabling focused changes, quicker reviews, and confident releases.

Fixing CLI Error Handling: A Deep Dive into Keyshade's WebSocket Communication Bug

Johnny Santamaria — Tue, 12 Aug 2025 15:00:00 +0000

“If you build software, every error message is marketing” - Jason Fried, Rework

Introduction

In today’s fast-paced software development landscape, managing credentials and sensitive configurations securely is more critical—and more challenging—than ever. Dispersed secrets, manual processes, and unplanned storage not only slow teams down but also increase the risk of breaches and operational errors. Keyshade addresses these challenges head-on with a centralized, automated approach to credential and configuration governance. Purpose-built for engineering teams, DevOps practitioners, startups, and open-source maintainers. Keyshade streamlines secret management while enhancing security, collaboration, and efficiency.

Keyshade is an advanced platform for centralized credential and configuration governance, designed to safeguard sensitive data, eliminate human error, and streamline operational complexity. By automating secret management, it fortifies security posture, amplifies cross-team collaboration, and liberates developers from the inefficiencies of fragmented secret storage. Its primary constituency—engineering teams, DevOps practitioners, early-stage startups, and open-source maintainers—adopts Keyshade to mitigate secret proliferation, minimize security liabilities, and embed robust secret orchestration seamlessly into their development pipelines. Users gravitate toward Keyshade for its ability to render secret management secure, intuitive, and collaborative, without imposing friction upon established workflows or toolchains. Below is a flow of control diagram discussing how users would typically use the product:

The user can send commands via keyshades terminal or on keyshades web app
On the web, the user can sign up, and initialize projects within workspaces
- Users can do the same in the CLI
Once the a project is made a http request is made to the API websocket service
The websocket service then retrieves that data to put in the Keyshade’s database to save the data for future use

Keyshade is currently in alpha and not live so there really are not any users as of posting this.

The issue

I partnered with a member of the Computing Talent Initiative with the task of solving issue 998 in Keyshade for a Summer Open Source Experience sponsored by . The main problem we noticed was that the Keyshade command line interface’s “keyshade run” command fails to give a proper error that users can understand. We would want to extract an error message and output it to the user. The issue is a bug that seems urgent to solve right now for users to understand errors they might make when working with Keyshade’s CLI, which is the main feature of Keyshade. Here are some key locations in the codebase we found out to be useful in solving the issue:

run.command.ts

Handles CLI logic for running commands, including WebSocket client interactions
Within a CLI folder

change-notifier.socket.ts

Contains the handleRegister method, which processes the register-client-app event
Handles authentication, authorization, and client registration logic
Emits the client-registered response to the client and logs errors
Within an API folder

run.types.d.ts

ClientRegisteredResponse describes the shape of the response sent by the server after registration (success, message)
Ensures type safety for client-side code handling server responses
Within a CLI folder

Here are steps regarding how we reproduced the problem before finding the important locations in the codebase

Steps to reproduce the behaviour:

Create a project on keyshade
Configure your CLI to tap into this project
Delete the project
Use keyshade run <> command
See the error

Expected behaviour

We would want the CLI to display what actually went wrong. The API should send the formatted message to the CLI. The CLI should print the error.

Keyshade’s Codebase

Before delving into the problem itself, and after pinpointing the critical components, we took a step back to develop a comprehensive understanding of the codebase’s architecture. Keyshade’s backend is architected with NestJS (a progressive Node.js framework) and TypeScript, leveraging Prisma as its database ORM, interfacing with a PostgreSQL relational database. It also integrates Redis for high-performance caching and publish/subscribe messaging, as well as Socket.IO for real-time, bidirectional communication between its command line interface and application programming interface.

On the frontend, the stack comprises React, TypeScript, JavaScript, and Next.js, presumably to power a performant and SEO-friendly web interface. Development workflows are streamlined through Nx (for monorepo orchestration), ESLint and Prettier (for code quality and style enforcement), alongside the TypeScript compiler for rigorous static type checking.

For security, Keyshade employs Elliptic Curve Cryptography for robust public-key encryption, JWT for token-based authentication, and API key authentication for service-to-service communication. The infrastructure is containerized via Docker and designed with cloud deployment in mind. A high-level system diagram encapsulates the overall architecture of the codebase as follows:

Here is a use case, using the components in the diagram to give you a better understanding of it.

Use Case: Developer Fetching Live Configuration Updates

Step 1: Developer Initiates CLI Command

keyshade run \--workspace my-company \--project web-app \--environment production

What happens: The developer runs the CLI command to start their application with live configuration updates from Keyshade.

The CLI documentation can be found [here](https://docs.keyshade.xyz/getting-started/installing-the-cli)

Step 2: CLI Processes Command (run.command.ts)

The run.command.ts file:

Parses the command arguments (workspace, project, environment)
Reads the local configuration file to get API credentials
Validates the required parameters using types from run.types.d.ts
Prepares to establish WebSocket connection for live updates

Step 3: WebSocket Connection Establishment

The CLI creates a websocket connection:

const socket = io('ws://api.keyshade.xyz/change-notifier')

What happens: The client connects to the NestJS backend's WebSocket gateway (change-notifier.socket.ts).

Step 4: Client Registration Request

The CLI emits the registration event:

socket.emit('register-client-app', { workspaceSlug: 'my-company', projectSlug: 'web-app', environmentSlug: 'production'});

What happens: This triggers the handleRegister method in change-notifier.socket.ts

Step 5a: Authentication

const userContext \= await this.extractAndValidateUser(client)  
Extracts API key from WebSocket headers (x-keyshade-token)

Queries PostgreSQL database via Prisma to validate the API key
Verifies user account is active

Step 5b: Authorization checks

await this.authorizationService.authorizeUserAccessToWorkspace({...})await this.authorizationService.authorizeUserAccessToProject({...})await this.authorizationService.authorizeUserAccessToEnvironment({...})

Checks if user has READ_WORKSPACE, READ_VARIABLE, READ_SECRET permissions

Validates access to the specific project

Confirms environment access rights

Step 5c: Client Registration

await this.addClientToEnvironment(client, environment.id)  
Creates entry in PostgreSQL mapping socket ID to environment ID

Adds client to Redis set for real-time notifications

Step 6: Success Response

What happens: Server sends confirmation back to CLI using the ClientRegisteredResponse interface from run.types.d.ts.

Step 7: Configuration Fetching

After successful registration:

CLI fetches initial configuration values (secrets, variables)
Application starts with these environment variables
CLI continues listening for real-time updates

Step 8: Real-Time Updates (Ongoing)

Another developer updates a secret value through the web interface.  

await this.redisSubscriber.subscribe(CHANGE\_NOTIFIER\_RSC, this.notifyConfigurationUpdate.bind(this))  
The web interface publishes change to Redis channel

All API instances receive the notification

this.server.to(clientId).emit('configuration-updated', data)

Server identifies all clients registered for that environment

Sends update notification to active CLI instances

CLI receives the configuration-updated event

Updates local environment variables

Application continues running with new values

Challenges

One specific technical challenge I encountered while attempting to solve issue 998 was figuring out why my changes in change-notifier.socket.ts were not being reflected when I tested them through the CLI. Despite updating the logic inside the Socket.IO handlers, the CLI behavior remained unchanged, and it was unclear whether the API changes were even being registered. Here is an attempt to solve this challenge and other challenges I encountered.

So I tried debugging the issue after reproducing the issue and found the debug information for keyshade issue 998

@johnny603 ➜ /workspaces/keyshade (teamAttempt/#998) $ DEBUG=* keyshade run "echo 'test'"INFO: Checking API key validity...INFO: API key is valid!INFO: Connecting to socket...socket.io-client:url parse wss://api.keyshade.xyz/change-notifier +0mssocket.io-client new io instance for wss://api.keyshade.xyz/change-notifier +0mssocket.io-client:manager readyState closed +0mssocket.io-client:manager opening wss://api.keyshade.xyz/change-notifier +0msengine.io-client:socket creating transport "websocket" +0msengine.io-client:socket options: {"path":"/socket.io/","agent":false,"withCredentials":false,"upgrade":true,"timestampParam":"t","rememberUpgrade":false,"addTrailingSlash":true,"rejectUnauthorized":true,"perMessageDeflate":{"threshold":1024},"transportOptions":{},"closeOnBeforeunload":false,"autoConnect":false,"extraHeaders":{"x-keyshade-token":"ks_a3adff2a9db8bf3c42fc817880ac4125983b83c171f0d04d"},"transports":[null],"hostname":"api.keyshade.xyz","secure":true,"port":"443","query":{"EIO":4,"transport":"websocket"},"socket":{"binaryType":"nodebuffer","writeBuffer":[],"_prevBufferLen":0,"_pingInterval":-1,"_pingTimeout":-1,"_maxPayload":-1,"_pingTimeoutTime":null,"secure":true,"hostname":"api.keyshade.xyz","port":"443","transports":["websocket"],"_transportsByName":{},"opts":{"path":"/socket.io/","agent":false,"withCredentials":false,"upgrade":true,"timestampParam":"t","rememberUpgrade":false,"addTrailingSlash":true,"rejectUnauthorized":true,"perMessageDeflate":{"threshold":1024},"transportOptions":{},"closeOnBeforeunload":false,"autoConnect":false,"extraHeaders":{"x-keyshade-token":"ks_a3adff2a9db8bf3c42fc817880ac4125983b83c171f0d04d"},"transports":[null],"hostname":"api.keyshade.xyz","secure":true,"port":"443"},"readyState":"opening"}} +0msengine.io-client:socket setting transport websocket +6mssocket.io-client:manager connect attempt will timeout after 20000 +8msengine.io-client:socket socket receive: type "open", data "{"sid":"qxJTaYpvOE4a6kIaAAB8","upgrades":[],"pingInterval":25000,"pingTimeout":20000,"maxPayload":1000000}" +756msengine.io-client:socket socket open +0mssocket.io-client:manager open +755mssocket.io-client:manager cleanup +0mssocket.io-client:socket transport is open - connecting +0mssocket.io-client:manager writing packet {"type":0,"nsp":"/change-notifier"} +1mssocket.io-parser encoding packet {"type":0,"nsp":"/change-notifier"} +0mssocket.io-parser encoded {"type":0,"nsp":"/change-notifier"} as 0/change-notifier, +0msengine.io-client:socket flushing 1 packets in socket +1msengine.io-client:socket starting upgrade probes +1msengine.io-client:socket socket receive: type "message", data "0/change-notifier,{"sid":"OpbhAQ0UYcfklhBoAAB9"}" +249mssocket.io-parser decoded 0/change-notifier,{"sid":"OpbhAQ0UYcfklhBoAAB9"} as {"type":0,"nsp":"/change-notifier","data":{"sid":"OpbhAQ0UYcfklhBoAAB9"}} +251mssocket.io-client:socket socket connected with id OpbhAQ0UYcfklhBoAAB9 +251mssocket.io-client:manager writing packet {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"}+251mssocket.io-parser encoding packet {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"} +0mssocket.io-parser encoded {"type":2,"data":["register-client-app",{}],"options":{"compress":true},"nsp":"/change-notifier"} as 2/change-notifier,["register-client-app",{}] +0msengine.io-client:socket flushing 1 packets in socket +1mssocket.io-client:socket draining queue +1msengine.io-client:socket socket receive: type "message", data "2/change-notifier,["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}]" +261mssocket.io-parser decoded 2/change-notifier,["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}] as {"type":2,"nsp":"/change-notifier","data":["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}]} +262mssocket.io-client:socket emitting event ["client-registered",{"success":false,"message":{"response":{"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","error":"Not Found","statusCode":404},"status":404,"options":{},"message":"{\"header\":\"Workspace not found\",\"body\":\"Workspace undefined not found\"}","name":"NotFoundException"}}] +261msERROR: Error registering to API: [object Object]

I came to the conclusion that the client is sending an empty object {} in the registration message, but the server expects a ChangeNotifierRegistration object with workspace, project, and environment slugs. This results in undefined values being passed to the authorization check, causing the “Workspace undefined not found” error. The code correctly creates the registrationData object with workspaceSlug, projectSlug, and environmentSlug

But the debug output shows it’s sending an empty object {}

This suggests that the data parameter being passed to connectToSocket has undefined values for workspace, project, and environment. Our team reached out to the issue maintainer radjip-b with a question regarding our process of the issue so far.

This small technical challenge was solved.

Solution

The solution for Keyshade’s issue 998 took five and a half weeks to complete during the eight-week internship. In the remaining time, we addressed additional related issues, including issue 989 and 988, which involved adding Slack and Discord integration documentation, as well as contributing to PR 1093.

Changes Introduced by PR #1093:

Slack Integration Documentation: A brand-new “How It Works” modal was added to the Keyshade platform’s UI, providing a visual, step-by-step guide—including screenshots—on how to set up Slack integration. This includes selecting Slack in the Integrations tab, entering credentials (signing secret, Bot token, channel ID), and testing with sample secrets and variables.
Modal Improvements & Developer Feedback: The modal component underwent adjustments—such as replacing a bare <DialogClose asChild /> (which caused rendering issues) with a proper close button, and revising instructions for better clarity (“Bot Token from Slack”). Additionally, an unused function (_handleClose) was cleaned up and textual instructions were streamlined.
Discord Integration Documentation: The scope of the PR was expanded to include documentation for Discord integration as well—hence the updated title to “feat: added slack and discord integration docs.”
Internal Routing and Formatting Refactor: Responding to reviewer feedback, the team refactored the integration documentation routing logic—removing the DOCS_SLUG_MAP, simplifying routing, standardizing layout for project creation and event triggers, and cleaning up the formatting and positioning of images.
Lint Fixes: Minor linting errors (such as extraneous braces in index.tsx) were corrected to ensure code consistency and avoid build issues.

Summary Table

Area	What Changed
Docs & UI	“How It Works” modal with guide and screenshots
Component Cleanup	Dialog close functionality fixed, redundant code removed
Discord Docs	Documentation for Discord integration added
Slack Docs	Documentation for Slack integration added
Documentation Routing	Routing logic simplified, layout standardized, visuals aligned
Code Quality	Linting issues resolved

Solution Description For Issue #998

Our team resolved a bug where the socket notifier and run command error handling would display [object Object] instead of a meaningful error message. The root cause was improper formatting of the error object before logging it to the user. The fix ensures that error messages are now properly stringified or formatted, providing clearer and more helpful feedback during CLI failures. To verify the fix, you can run the keyshade run command with an invalid configuration or project setup that triggers a connection error. This update significantly improves the developer experience by making error outputs more understandable.

We determined that only three files required direct changes to address the issue:

NestJS WebSocket Gateway – The server-side component uses NestJS’s @WebSocketGateway decorator with Socket.IO to handle real-time communication between the CLI and API server. The handleRegister method in change-notifier.socket.ts processes client registration requests and can return different error message formats depending on the failure type (authentication, authorization, or database errors).
CLI Command Handler – The run.command.ts file contains the client-side logic that connects to the WebSocket server and handles the registration response. This component already included robust error parsing logic capable of handling both string and object formats, but the TypeScript interface was preventing proper type checking.
Type Checker – The run.types.d.ts file was updated to allow both string and object types, since the server can send either format.

After the maintainer approved the fix, we made additional updates to the codebase by modifying two more files: changelog.md to document the changes, and package.json to bump the semantic version to 3.2.3.

What is in the file changes

These were changes found using git’s diff command

What has been changed can also be found here

changelog.md:

🐛 Bug Fixes* **api, cli:** ambiguous error messages on keyshade run failure ([#1087](https://github.com/keyshade-xyz/keyshade/issues/1087)) ([206dda7](https://github.com/keyshade-xyz/keyshade/commit/206dda702b3f3f0cabeb73e4f74dd2ccfc962631))

apps/cli/package.json: "version": "3.2.2-stage.2",
apps/cli/src/types/command/run.types.d.ts:

message: string | object // Allow both string and object types since server can send either

apps/cli/src/commands/run.commands.ts:

Accurate WebSocket URL Parsing:
The CLI’s socket connection logic now uses the full host from the base URL (via the URL class), ensuring reliable and correct WebSocket connections especially for complex URLs.

Registration Timeout Added:A 30-second timeout was introduced for socket registration. If the server does not respond within this window, users receive a clear error message and the process exits gracefully. This prevents hanging connections and improves user feedback.

Enhanced Error Message Extraction:The registration response error handling is now more robust: It intelligently extracts meaningful error messages whether the server responds with a string, an object, or a nested structure. Attempts to parse JSON error details for more informative output. Falls back to stringifying unknown error types, ensuring users always get useful feedback. All error messages are now logged in a consistent, user-friendly format.

Minor Cleanup:Removed a redundant blank line after the socket connection initiation for improved code readability.

apps/api/src/socket/change-notifier.socker.ts:

1. Improved Error Handling & Feedback The registration process now provides much more detailed error messages to clients. Instead of a generic error or raw message, errors are parsed, logged, and emitted in user-friendly formats.

2. Custom Authentication Logic for Socket Connections The previous decorator-based guards (@UseGuards(AuthGuard, ApiKeyGuard) and @RequiredApiKeyAuthorities) have been replaced by custom methods. New methods (extractAndValidateUser and convertToAuthenticatedUser) manually handle authentication and authority checks, mimicking guard behavior but allowing for more granular error handling and feedback.

3. API Key Authority Checks The code now checks if the API key has the required authorities (READ_WORKSPACE, READ_PROJECT, and READ_ENVIRONMENT) directly within the socket handler. If the user has ADMIN authority, individual checks are bypassed.

4. Enhanced Logging Connection logs now include the presence or absence of authentication headers. Errors during registration are logged with more context, including error types and constructors.

5. Dependency Injection Updates The SlugGenerator service is now injected into the class, preparing for potential future use or slug-related operations.

6. Cleaned-Up Imports Removed unused decorators and guards, reflecting the shift to manual authentication and authority verification.

7. User Context Construction The code now constructs a detailed user context object for socket connections, including IP address, workspace, and API key authorities, improving downstream authorization and personalization.

Testing the solution

While testing Keyshade, I encountered an issue that was easy to reproduce: create a project in Keyshade, configure your CLI to connect to it, delete the project, and then run the keyshade run <> command. This sequence would trigger an unclear error message, leaving the cause ambiguous. The expected behavior was for the CLI to clearly display the actual problem, with the API sending a properly formatted error message that the CLI would then print for the user. After applying the solution, the CLI behaved as intended, providing a clear and informative error message, making the debugging process much smoother.

Another testing approach involved building custom WebSocket test clients designed to send invalid authentication data, deliberately triggering error conditions to assess system resilience. We closely monitored server-side logs to ensure errors were correctly detected and handled, while also verifying that the client could parse various error message formats. This end-to-end validation—from CLI input, through WebSocket communication and API processing, to the final CLI response—confirmed the integrity and reliability of the entire flow. This comprehensive process not only confirmed the integrity and reliability of the entire system but also successfully closed issue #998 with our PR #1087.

Conclusion

We resolved a complex WebSocket bug by fixing authentication handling, type safety, and error messaging across the backend and CLI. They used systematic debugging, targeted fixes to three core components, and custom test clients to replace vague errors with clear feedback, improve authentication timeouts, and update integration docs—showcasing technical excellence, thorough testing, and a user-focused, collaborative approach.

Product sense - Intro to Keyshade

Johnny Santamaria — Sat, 12 Jul 2025 02:42:42 +0000

Product sense is the intuitive understanding of what makes a product successful—the ability to identify user needs, spot market gaps, and build solutions that people actually want to use. It's the difference between building features and building the right features.

At its core, product sense combines three critical elements: deep empathy for users, keen awareness of market dynamics, and the judgment to prioritize what matters most. It's what separates products that solve real problems from those that collect digital dust.

The Anatomy of Strong Product Sense

Product sense isn't just about having good ideas—it's about having the right ideas at the right time. It manifests in several ways:
User-Centric Thinking: Understanding not just what users say they want, but what they actually need. This means digging beneath surface-level requests to uncover underlying pain points.

Market Awareness: Recognizing where opportunities exist in the competitive landscape and how user behavior is evolving.
Execution Wisdom: Knowing which features to build first, how to sequence development, and when to pivot based on feedback.

Keyshade: Product Sense in Action

Keyshade exemplifies strong product sense by addressing a fundamental yet often overlooked developer pain point: secure environment variable management. While many developers resort to scattered .env files or basic cloud solutions, Keyshade's creators recognized that teams needed something more robust yet accessible.

The product sense behind Keyshade reveals itself in several key decisions:
Solving a Real Problem: Rather than building another generic developer tool, Keyshade targets the specific friction developers face when managing secrets across environments. This isn't a nice-to-have—it's a daily necessity that most teams handle poorly.
Open Source Strategy: By making Keyshade open source, the team demonstrates understanding of developer culture and trust requirements. When it comes to handling sensitive data, transparency isn't just appreciated—it's essential.
Developer-First Design: The focus on seamless integration and straightforward workflows shows awareness of how developers actually work. Complex security tools often go unused because they create more friction than they solve.

Why This Matters for Product Builders

Keyshade's approach illustrates how product sense translates into tangible decisions. Instead of building a comprehensive DevOps platform (which would be feature-rich but unfocused), they identified one specific workflow that teams consistently struggle with and built a targeted solution.
This focused approach is classic product sense—recognizing that doing one thing exceptionally well often beats doing many things adequately. It's the difference between building what's technically possible and building what's strategically valuable.
The lesson for product builders is clear: strong product sense isn't about having all the answers upfront. It's about asking the right questions, staying close to user problems, and having the discipline to build solutions that matter rather than solutions that impress.

Product sense develops through experience, but it starts with genuine curiosity about user problems. Tools like Keyshade succeed because they're built by people who understand both the technical challenges and the human workflows they're trying to improve.

[Boost]

Johnny Santamaria — Mon, 23 Jun 2025 23:22:28 +0000

Ben Halpern

Jun 23 '25

Meme Monday

#discuss #jokes #watercooler

1 min read

Level Up Your Linux Skills

Johnny Santamaria — Mon, 23 Jun 2025 23:17:19 +0000

Deep Dive into OverTheWire Bandit

“So many books, so little time.” - Frank Zappa

How a gamified approach to cybersecurity education transforms learning from tedious to thrilling

Introduction

In today’s fast-paced digital world, cybersecurity skills are no longer optional — they’re essential. But learning them through static lessons, long-winded lectures, or dry textbooks? That often leads to burnout or boredom.

That’s where OverTheWire Bandit comes in — a brilliant learning platform that transforms traditional Linux security training into a hands-on, puzzle-based game. It’s not just engaging — it’s effective, because it teaches you by doing.

I started playing Bandit right after finishing a web development class, where I had learned how to navigate files and search through directories using Linux. Jumping into this wargame felt like leveling up — taking those foundational skills and applying them in a way that was both fun and deeply practical.

Unlike passive learning platforms, Bandit drops you straight into a live Linux environment. Every command has consequences. Every file might contain a hidden clue. Every level challenges you to solve real problems with real tools — just like in the field.

What Makes Bandit Special?

The Progressive Learning Model

Bandit follows a carefully crafted progression that mirrors real-world security scenarios. Each level builds upon the previous one, introducing new concepts while reinforcing established skills. This isn't just random command execution – it's a structured journey through the fundamentals of:

File system navigation and permissions
Text processing and pattern matching
Network protocols and services
Cryptography and encoding
Process management and privilege escalation
Version control systems

Real Environment, Real Skills

What sets Bandit apart from simulated environments is its authenticity. You're working with actual Linux systems, real SSH connections, and genuine security tools. The skills you develop aren't theoretical – they're immediately applicable to:

System administration tasks
Security auditing and penetration testing
Digital forensics investigations
DevOps and automation workflows

Technical Highlights: Key Learning Concepts

File System Mastery

Early levels focus on fundamental file operations, but with security-minded twists. You'll encounter:

# Working with special characters in filenames
cat ./-

# Handling spaces in filenames
cat "spaces in this filename"

# Finding hidden files and understanding permissions
ls -la
find . -type f -name ".*"

These aren't just command exercises – they simulate real scenarios where attackers hide malicious files or administrators need to locate specific system files.

Advanced Text Processing

As you progress, Bandit introduces sophisticated text manipulation techniques essential for log analysis and data extraction:

# Pattern matching for specific data
grep "pattern" large_file.txt

# Sorting and finding unique entries
sort data.txt | uniq -c

# Working with binary data
strings binary_file | grep "readable_text"

Cryptography in Practice

Rather than abstract mathematical concepts, Bandit presents cryptography through practical applications:

Base64 encoding/decoding for data transmission
ROT13 cipher for basic obfuscation
File compression and decompression chains
SSL/TLS communication with live services

Network Security Fundamentals

Advanced levels introduce network concepts that security professionals use daily:

# Port scanning and service identification
nmap -p port_range target

# Network communication
nc (netcat) for various protocols

# SSL/TLS client connections
openssl s_client -connect host:port

The Learning Psychology Behind Gamification

Immediate Feedback Loop

Each level provides instant validation. Success grants access to the next challenge, while failure requires problem-solving and research. This tight feedback loop maintains engagement and builds confidence progressively.

Problem-Based Learning

Instead of memorizing commands, you're solving authentic problems. Need to find a file with specific properties? You'll learn find command naturally. Need to analyze network traffic? netstat and nmap become tools, not just commands to memorize.

Community and Collaboration

The OverTheWire community fosters collaborative learning. Forums, IRC channels, and documentation encourage discussion without spoiling solutions, creating an environment where learners help each other grow.

Real-World Applications

For Aspiring Security Professionals

Bandit provides foundational skills for:

Penetration testing methodologies
Incident response procedures
Digital forensics investigations
Security operations center (SOC) work

For System Administrators

The challenges develop crucial administrative skills:

Log analysis and troubleshooting
Service configuration and management
Automation scripting capabilities
Security hardening techniques

For Developers

Modern development requires security awareness:

Secure coding practices
DevSecOps integration
Container security fundamentals
Infrastructure as code security

Advanced Concepts and Progression

Privilege Escalation

Later levels introduce sophisticated privilege escalation techniques, teaching:

SUID/SGID binary exploitation
Cron job manipulation
Service account abuse
Environment variable exploitation

Version Control Security

Git-based challenges demonstrate:

Repository forensics for sensitive data
Branch and tag analysis
Commit history investigation
Remote repository interaction

Building Your Learning Path

Getting Started Strategy

Set up your environment with proper SSH clients and tools
Document your progress – maintain notes on techniques learned
Join the community for support and additional resources
Practice consistently – regular engagement builds muscle memory

Advanced Techniques

As you progress, develop:

Automation scripts for repetitive tasks
Custom tools for specific challenges
Deep diving into manual pages and documentation
Cross-referencing with real-world security frameworks

The Technical Foundation for Cybersecurity Careers

Industry Relevance

The skills developed through Bandit directly align with industry certifications:

CompTIA Security+ foundational knowledge
CEH (Certified Ethical Hacker) practical skills
OSCP (Offensive Security Certified Professional) methodology
CISSP security management concepts

Portfolio Development

Completing Bandit demonstrates:

Self-directed learning capability
Problem-solving methodology
Technical documentation skills
Persistence and attention to detail

Conclusion: More Than Just a Game

OverTheWire Bandit represents a paradigm shift in technical education. By combining gamification with authentic security challenges, it creates an engaging learning environment that builds genuine, applicable skills.

The platform's genius lies not just in its technical content, but in its understanding of how people learn best – through doing, failing, researching, and ultimately succeeding. Each level completed isn't just a game achievement; it's a step toward genuine cybersecurity competence.

Whether you're starting your security journey or looking to sharpen existing skills, Bandit offers a structured, challenging, and ultimately rewarding path to technical mastery. The investment in time and effort pays dividends in real-world capability and career advancement.

Ready to begin your journey? The terminal is waiting, and the first challenge is just an SSH connection away.

OverTheWire Bandit

Note: Always approach learning platforms like OverTheWire with respect for their educational mission. Share knowledge and techniques, but avoid sharing specific solutions that might diminish the learning experience for others.

I suggest having a notepad ready! Whether it’s a physical notebook or a digital document, tracking your commands, mistakes, and lessons learned helps reinforce your understanding and gives you something to reference later.

[Boost]

Johnny Santamaria — Sat, 29 Mar 2025 05:28:39 +0000

Jess Lee for The DEV Team

Mar 5 '25

Join Our First-Ever WeCoded Challenge – Celebrating Underrepresented Voices in Tech Through Stories & Code!

#devchallenge #wecoded #dei #career

165

4 min read

DEV Community: Johnny Santamaria

MudMap: A Voice-First Trail Logger That Works Without a Signal

MudMap: A Voice-First Trail Logger That Works Without a Signal

What I Built

Demo

Code

How I Built It

Challenges

Challenge 1: The cryptic 25.0.4.1 build failure

Challenge 2: The asset script couldn't find pmtiles

Challenge 3: The disk-space wall

What these three have in common

Why Does Open Innovation Matter?

CscGrader and generateRubrics: Open-Source Tools for Faster CS Grading

What I Built

Demo

studentNames.txt contains a list of all students in the class

The submissions folder contains one subfolder per student

The assignment profile tells CscGrader what to look for and how to run it

Code

How I Built It

Why Does Open Innovation Matter?

My Agent Session

Prize Categories

Using AI as a Learning Tool: Lessons From the Classroom, STEM, and Software Engineering

AI Needs Context

AI and Gamified Learning

AI as a Learning Assistant

What My LA/DL Experience Taught Me

AI in Educational Research

Human-in-the-Loop: What I Learned From Robotics

The Risk of Overusing AI

What AI Has Taught Me About Learning

Conclusion

Lessons from a Robotics Startup: What I Learned About Data Pipelines

The Setup

1. The Data Pipeline Shape Is Universal—But the Details Aren't

2. Annotation Granularity: More Isn't Always Better

3. Automation Still Needs Oversight—Especially in Evaluation

What I'd Tell My Past Self

Final Thoughts

From Kilowatt-Hours to Carbon Counts: Shaping Accurate Energy Reporting with OED

Introduction

The Issue

The Product/Codebase

Tech Stack

Syatem diagram

Use case

Challenges

Challenge 1: Inconsistent Unit Conversion Setup in LR18

Self-Directed Problem-Solving Attempts

Additional Help

Status

Challenge 2: CSV Naming and Expected Output Consistency

Self-Directed Problem-Solving Attempts

Additional Help

Status

Challenge 3: Maintaining Consistent Branch Workflow (Using git cherry-pick)

Self-Directed Problem-Solving Attempts

Additional Help

Status

Solution

Keeping Your Secrets in Sync: Environment Lifecycle Management That Actually Works

Introduction

The Problem: Silent Failures in Cloud Integrations (#1254)

Why This Bug Mattered to Production Teams

Under the Hood: Key Code Changes

Challenges

Solution (PR #1255)

What Did I Learn?

Keyshade Debugging: Mastering Workspace Role Tests and API Repair

Introduction

The Issue (#1122)

Reproducing the Problem

Use Case

Challenges

Solution

Conclusion

Fixing CLI Error Handling: A Deep Dive into Keyshade's WebSocket Communication Bug

Introduction

Challenge 1: The cryptic `25.0.4.1` build failure

Challenge 2: The asset script couldn't find `pmtiles`