This is a submission for the GitHub Copilot CLI Challenge
Problem I choose to solve
As developers, there are occasions when we’ve spent countless hours staring at API responses in JSON format. Thousands of lines of nested brackets and quotes used to stare back, hiding that one single piece of data we actually needed.
The web has many JSON formatters, viewers, and query tools. They perform well, but often fall short for practical use. Most solutions struggle with large enterprise-scale files. Others don't offer an intuitive, interactive query system. I realized that for most of us, the solution was a frustrating cycle of copying, pasting into a formatter, and manually hunting for keys.
🚀 What I Built as Solution
I wanted a tool that combined fast visualization with a powerful query engine. So, I decided to build JSON Query Pro. It is a comprehensive data exploration environment. It isn't just a viewer; it's a high-performance workbench designed to help developers navigate and reshape massive JSON datasets.
At its core, the application features a virtualized tree explorer that remains fluid even with millions of rows, paired with a dual-tier query engine that supports both standard JavaScript paths and advanced JSONata transformations. It’s a local-first, privacy-conscious tool that brings professional-grade data manipulation directly to the browser.
Demo
PFB a short demo of the app.
PFB relevant links:
Code Repository: https://github.com/priyabratapaul/json-query-pro
Deployed App Link: https://json-query-pro.vercel.app
📋 The Blueprint: What I Set Out to Build
Before writing a single line of code, I mapped out exactly what a "pro" JSON tool should look like. I didn't want a passive viewer; I wanted a dynamic workspace. My primary goals were:
- Virtualized Tree Engine: The UI had to remain fluid (60fps) regardless of whether the file was 1KB or 500MB. This meant only rendering what was visible in the viewport.
- A "Point-and-Click" Selection Mode: I wanted to eliminate the need to manually count array indices or type out long, error-prone property paths. Clicking a node should instantly populate the query editor.
- Dual-Tier Querying: A hybrid approach using native JS pathing for simple lookups and the industry-standard JSONata engine for complex filters, counts, and data reshaping.
- Local-First Persistence: Data should be stored safely in the browser's IndexedDB, ensuring that accidentally closing a tab didn't mean losing hours of investigative work.
🧩 The Wall: Obstacles Faced During Implementation
However, moving from blueprint to browser revealed several harsh engineering realities. As I began implementing these features, I ran into several hurdles. A few of them are mentioned below:
1. The "Where Am I?" Problem
Even with a clean tree view, working with deeply nested objects (like large database exports) made it impossible to keep track of a field's context. A field named "status" might exist in five different parent objects at different levels.
How I solved it: I integrated the selection logic directly into the virtualization engine. Now, every node "knows" its exact absolute path from the root. When you click it, the app doesn't just show you the data; it calculates the precise breadcrumb path and writes the corresponding JSONata code for you.
2. The Performance Wall
Early on, the browser simply died when I loaded a 100MB file. The main thread would freeze for 10 seconds just parsing the JSON, and then another 5 seconds trying to calculate the tree structure.
How I solved it: I offloaded the entire data lifecycle to a Web Worker. The "brain" of the app now lives in a separate thread. While the worker parses, queries, and flattens the JSON, the main UI remains completely interactive. This was the only way to support professional-scale datasets.
3. The 1GB String Ceiling and V8 Buffer Limits
During implementation, I hit the hardest wall of all: the "allocation size overflow." I discovered that Chromium-based browsers (Chrome, Edge) have a hard internal limit on string length—roughly 512MB. Even if a machine has 64GB of RAM, the JavaScript engine cannot create a single string long enough to represent a massive JSON file.
How I solved it: I implemented a defensive loading strategy. Instead of relying on a single large string, the worker uses modern streaming APIs (Response.json()) to parse data in chunks. I also added explicit memory cleanup, nulling out large buffers the moment they are no longer needed to maximize the available heap.
Screenshots
Main page: https://json-query-pro.vercel.app/#/Main
Help page: https://json-query-pro.vercel.app/#/Help
Sample Queries
🚀 Sample Data
All examples below reference the default "Tech Innovations" dataset:
- Company: Tech Innovations (San Francisco)
-
Structure:
company→departments[]→employees[]→projects[]
🔍 Basic Retrieval (Simple Queries)
| Goal | Standard PATH Syntax | JSONata Syntax |
|---|---|---|
| Get Company Name | $.company.name |
company.name |
| First Department | $.company.departments[0] |
company.departments[0] |
| All Dept Names | Not supported (requires index) | company.departments.name |
| First Employee Name | $.company.departments[0].employees[0].name |
company.departments[0].employees[0].name |
⚡ Advanced Selection (Complex Queries)
Filtering Data (JSONata Only)
Find specific items without knowing their index.
-
Engineering Dept:
company.departments[name="Engineering"] -
Software Engineers:
company.departments.employees[position="Software Engineer"] -
High Value Clients:
company.departments.employees.clients[contract_value > 60000]
Deep Search (The Double Star)
Find data anywhere in the file regardless of depth.
-
Find "Alice":
**[name="Alice"] -
Find all Projects:
**.projects
Reshaping Output
Create a custom report from the data.
company.departments.{
"department": name,
"headcount": $count(employees),
"totalValue": $sum(employees.clients.contract_value)
}
💡 Lessons Learned
- Performance IS the experience: Users don't care how many features you have if the app lags. Background processing is non-negotiable for pro tools.
-
The V8 Buffer Limit: I learned that the hard way: if your JSON file is 500MB+, standard
JSON.parseis a gamble in Chromium-based browsers. - Ghost Truncation: I found that some streaming APIs don't always give you a nice error message when they fail under memory pressure; they often just stop reading.
- Dynamic Engine Feedback: We implemented runtime browser detection in our Background Worker to explain memory restrictions specifically when they occur.
🚀 Future Plans
- Breaking the 1GB Barrier: While JSON Query Pro is currently optimized for datasets up to 512MB (Chromium) and 1GB (Firefox), the next frontier is breaking through this browser-imposed ceiling.
- Desktop app: Also I am planning to create desktop version of this app for different OS.
🤖 My Experience with GitHub Copilot CLI
Building a high-performance system like JSON Query Pro required more than just standard frontend knowledge—it required deep dives into browser internals, memory management, and concurrency. Throughout this process, I leveraged GitHub Copilot as a senior partner.
Whether it was brainstorming the communication protocol between the main thread and the Web Worker or debugging the subtle nuances of structured cloning limits, GitHub Copilot provided instant, high-context feedback that significantly accelerated the development cycle.
It acted as a force multiplier, allowing me to iterate on complex features like the virtualized tree rendering and IndexedDB persistence layer in a fraction of the time it would have taken to research these topics in isolation. The impact on my development experience was profound; it turned what could have been a month-long research project into a week-long implementation sprint, all while maintaining a focus on clean, scalable code.
Conclusion
I built JSON Query Pro to turn a large json text into a clear, searchable map. By combining background processing with powerful query languages, I've created a workspace where json data isn't just something you see—it's something you can easily query.
👉Link to app: https://json-query-pro.vercel.app
Happy Querying!
Top comments (2)
I use the utility jq.
Yeah. JQ is a indeed very powerful and very very lightweight command-line tool. I guess I need to rethink on future plan of releasing desktop support of this app. Will it be even relevant. Anyway, will check on it later.
Thanks for the mention though.👍
Always open to learning. 😊