DEV Community: Flude team

System Header Hell: Teaching libclang to Understand GCC and Survive MSVC

Flude team — Fri, 30 Oct 2026 03:48:01 +0000

After migrating the C++ parser to libclang, we expected to see a perfectly accurate Abstract Syntax Tree (AST). But when we unleashed it on our first real project, instead of a beautiful hierarchy of classes, we got a tree consisting entirely of CXCursor_UnexposedDecl and fatal syntax errors.

The parser stumbled on the very first line: #include <vector>.

A Deaf Compiler

It turned out that libclang (being merely a linkable library, not a full CLI compiler) by default has no idea where standard header files are located in the operating system.

If the parser doesn't see system headers, it doesn't understand what std::string or std::vector is. And since everything in modern C++ relies on standard library templates, parsing fails in a cascade.

In the Enterprise world, developers usually pass giant strings of -I/usr/include/... flags to the parsers themselves. But we were building a SaaS and a user-friendly CLI utility. We couldn't force users to manually search for compiler paths on their machines. The parser had to become smart.

The GCC Discovery Quest

On Linux systems, we had to write our own Auto-Discovery mechanism.

We implemented a heuristic that ran the system compiler in the background with clever flags: gcc -print-file-name=include (falling back to cc). This trick forces GCC (or the system Clang) to spit out the path to the internal directory it uses to search for system headers by default into the console.

Our Python code intercepted this output, parsed it, filtered out non-existent directories, and dynamically converted it into an array of -I flags, which were then fed into libclang. The parser gained its "sight"—it began to perfectly understand Linux projects.

The MSVC-STL Nightmare on Windows

But if Linux was solved with an elegant hack, a real hell awaited us on Windows.

Windows developers use the Microsoft Visual C++ (MSVC) compiler and its proprietary standard library implementation—MSVC-STL. Integrating Clang with the intricacies of Windows code (including the monstrous windows.h) is pure pain.

MSVC historically forgives many syntax liberties and uses specific macros, pragmas, and extensions. Strict libclang looked at MSVC-STL code and simply refused to parse it, generating hundreds of errors.

To reconcile them, we had to dig deep into the LLVM documentation. We selected a specific set of compatibility flags (-fms-compatibility, -fms-extensions, and also spoofing macros like -U__clang_major__ -D__clang_major__=19 to bypass version checks in the MSVC STL) that forced libclang to pretend to be a Microsoft compiler. Only after fine-tuning this virtual environment did the parser stop panicking at the sight of Windows source code.

Conclusion

We spent a ton of time not on parsing the client code itself, but on configuring the environment in which that code lives. But the result was worth it: our parser learned to read developers' real code "out of the box" on any operating system, without requiring complex configurations from them.

But "seeing" the code doesn't mean understanding its structure. When the parser finally read all the headers, we discovered that it loses half the methods if a developer dared to define them outside the class (Out-Of-Line). How we manually stitched torn methods back to templates—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/libclang-system-headers-gcc-msvc

Also read us:

Firing Doxygen: Native Parser Architecture and Adopting libclang

Flude team — Tue, 27 Oct 2026 04:10:13 +0000

Doxygen had been with us since day one. At the MVP stage, it was a conscious and absolutely correct choice: it allowed us not to write a C++ parser from scratch, but to immediately focus on business logic and documentation rendering.

But as we transitioned from local scripts to a cloud SaaS, Doxygen turned into a burden. It couldn't cache the AST, its binary was too bulky, and its internal parser broke from C++ templates (remember the SFINAE incident from the previous episode, which forced us to write crutches via INPUT_FILTER).

It was time to do what we planned from the very beginning: fire Doxygen and migrate the system to native parsers.

Divide and Conquer

We never harbored illusions about creating a "universal parser". Just like with Doxygen dialects, we initially designed the system for separate parsers for each language. A tool that equally well understands hellish C++ templates with SFINAE and dynamic Python decorators simply doesn't exist.

Therefore, we conducted research and selected the perfect, native tool for each language:

Python: The built-in ast module. No one will parse Python better than the interpreter itself (planned).
C# and Java: tree-sitter—an incredibly fast and modern incremental parser that has become an industry standard (planned).
C++: Our most complex language and the first candidate for implementation.

Boss Fight: C++ and libclang

We started the implementation of native parsers specifically with C++, as it was the most critical vector. Trying to parse modern C++ with regexes or lightweight parsers is pure suicide. Preprocessors, macros, templates, and operator overloading will break any homegrown tool.

That's why we took libclang. Only a true compiler frontend can output a 100% accurate AST for C++. Yes, integrating with a C++ API from Python is not an easy task, but it's the only guarantee that we will correctly understand all the client's code.

The Triumph of Planned Architecture (Intermediate Representation)

The most beautiful part about moving to libclang was that we didn't have to rewrite the renderers (that same hugo_markdown we talked about in the previous article).

This became possible because our architecture, from day one, was based on an Intermediate Representation (IR). All parsers are required to convert the result of their work into a unified ProjectCatalog structure.

The new parser based on libclang simply started assembling the exact same ProjectCatalog that Doxygen used to output. The rendering layer didn't even know that the engine under the hood had changed. The architecture laid down at the start of the project worked exactly as intended.

Conclusion

The core was freed from a heavy third-party dependency that prevented caching implementation. We closed a huge security hole (by removing INPUT_FILTER) and significantly lightened our cloud workers.

Transitioning to a native parser is the technical step that finally turned an MVP utility into a serious product.

But as soon as we connected libclang to real projects, a surprise awaited us. The native parser turned out to be blind: it flat-out refused to see the C++ standard libraries. How we searched for paths to GCC system headers on Linux and survived with MSVC-STL on Windows—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/firing-doxygen-native-cpp-parser

Also read us:

C++ Crash Test: Hidden SFINAE Traps and Parser Silence

Flude team — Fri, 23 Oct 2026 03:07:26 +0000

We were still trying to squeeze the maximum out of good old Doxygen in our cloud pipeline when we suddenly encountered a classic problem of harsh enterprise C++: the parser started silently "eating" entire chunks of the codebase.

It all began when we noticed an anomaly: important structures (like BITMAPINFO and RGBQUAD) had vanished without a trace in the generated documentation.

The scariest part was that the pipeline passed successfully. Doxygen returned exit code 0, didn't issue a single warning, but the XML output for these files was practically empty. It simply swallowed the code and stayed silent.

Investigation by Binary Search

When a parser is silent, the only way to find the cause is the bisection method. We would take a problematic header file of several thousand lines, delete half of it, run the parser, and see if the necessary structures appeared.

Narrowing the search down to 50 lines, we found the culprit. It turned out to be a classic C++ trick—SFINAE (Substitution Failure Is Not An Error). The codebase extensively used the construct:

typename std::enable_if<CONDITION, bool>::type = VALUE

To a C++ compiler, this is absolutely valid code. But the legacy Doxygen parser went crazy at the sight of such a pile of angle brackets (< >). Its internal state machine broke, and it stopped understanding C++ syntax until the very end of the file. This is exactly why structures declared below the ill-fated SFINAE template simply fell out of reality.

A UNIX-style Solution

We couldn't go to the product developers and say: "Rewrite all your C++ code because our parser doesn't understand it." We needed a systemic solution that didn't touch the source files.

Fortunately, Doxygen has the INPUT_FILTER mechanism. This is a script through which every file is run before the parser sees it.

We wrote a small Python script (sfinae_filter.py) with a clever regular expression. The script caught all std::enable_if constructs on the fly and replaced them with a syntax equivalent in the number of arguments: bool = VALUE.

At the same time, we strictly observed an important rule: the script calculated the number of line breaks in the removed block and added exactly the same number of blank lines to the replacement. Thanks to this, the line numbers in the source code did not shift, and all parser warnings continued to point to the correct lines in the developers' IDEs.

Conclusion

A small Python filter solved the problem in dozens of files, returning the lost classes to the documentation. We won again.

But this incident became the "last straw." Crutches with regular expressions on top of C++ code is a road to nowhere. The language is too complex to parse as text, and the unsafe INPUT_FILTER opened holes in cloud security.

It was time to do what we had avoided for so long: fire Doxygen for good. How we migrated to the native libclang compiler frontend and why we didn't have to rewrite half the project—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/crash-test-sfinae-doxygen

Also read us:

The Butterfly Effect: How Renaming One String Broke Our Distributed System

Flude team — Tue, 20 Oct 2026 04:49:01 +0000

When we designed the SaaS version of Flude, we were very proud of our architecture. We split the logic into independent components: the parser core (engine), the API gateway (control-plane), the cloud workers (job-runner), and the thin clients for CI integration (flude-action and similar).

All of this lived in different repositories and was deployed independently. We thought we had achieved perfect "loose coupling". We were certain that changes in one module would never bring down the entire system.

Until one harmless refactoring proved us wrong.

A Harmless Refactoring (The First Flap of a Wing)

It all started with perfectionism in the core repository. I noticed a technical inaccuracy: our parser generated not just classic Markdown, but a highly specific dialect with metadata (frontmatter) and shortcodes, tailored strictly for the Hugo static site generator.

Leaving the output format under the generic name markdown was incorrect—it would mislead future users if they tried to feed this code into a different generator (like Docusaurus).

I did the most logical thing in the world: I renamed the CLI argument and the internal format identifier from markdown to hugo_markdown. The core tests predictably turned green, and the commit flew into main.

Chain Reaction

In that very minute, our "loosely coupled" microservice architecture began to fold like a house of cards. It turned out that the word "markdown" was not just a local variable, but an implicit hard contract wired into all layers of the system.

Strike #1: Control Plane
The API gateway, which receives jobs from users, fell first. It suddenly started throwing HTTP 422 Unprocessable Entity errors. Why? Because at the entrance stood a strict OpenAPI schema that validated the format field. The schema allowed only two values: ["markdown", "html"]. Receiving the new parameter hugo_markdown from the client, the API simply rejected the request as invalid. We had to urgently patch the validation schemas.

Strike #2: Cloud Workers
As soon as the API was fixed and requests passed through, the workers themselves (Data Plane) crashed. A worker would successfully run the updated parser core, but upon completion, it tried to find the out/markdown directory on the disk to pack it into a ZIP archive. Naturally, the folder wasn't there—the updated engine was now carefully putting the result in out/hugo_markdown. The workers crashed with a FileNotFoundError.

Strike #3: Thin Clients (CI Scripts)
The workers were updated. Archives started forming. But the pipelines on the users' side continued to burn red. Our thin clients (running in GitHub, GitLab, etc.) sent a request to download the artifact, expecting to receive the markdown.zip file. The server, however, was now returning hugo_markdown.zip. The scripts simply couldn't find the needed file and failed the build.

An Expensive Lesson

To roll out the renaming of just one variable, we had to coordinate a synchronous release across four independent repositories.

That day, we learned an important lesson. Microservices do not save you from tight coupling if your contracts are based on implicit Magic Strings smeared across the code. If one service changes a data format and another hardcodes paths to files—that's not a distributed system. It's a distributed monolith, which is much more painful to break than a regular one.

What's Next?

We stabilized the contracts, covered them with end-to-end tests, and finally brought the SaaS infrastructure to a working state. The system was ready for real loads.

But one fundamental architectural problem of the core, which started it all, wouldn't let us go—Doxygen. We tolerated its flaws: the impossibility of proper AST caching, its slowness, and the huge size of its binary in containers. But one fine day, it simply capitulated to complex C++ code, and this became the beginning of the end.

How a harmless SFINAE template trick forced Doxygen to silently delete classes from the project, and why we had to save ourselves with unsafe INPUT_FILTER scripts—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/butterfly-effect-microservices-rename

Also read us:

Surviving the JS Ecosystem: Auth, Netlify, and the New Rules of Next.js 16

Flude team — Fri, 16 Oct 2026 05:56:32 +0000

A Python backend is predictable and stable. Infrastructure on Google Cloud means strict rules and transparent logs. But as soon as we stepped into the territory of modern frontend to build an API key issuance portal for our SaaS, we found ourselves on the rollercoaster of the JavaScript ecosystem.

We just needed a single Next.js page where we could embed the <UserProfile /> component from Clerk. On paper, it looked like a two-hour task. In reality, we spent a day fighting hosting providers, outdated documentation, and implicit conventions.

Choosing a Host: Cloudflare, Vercel, or Netlify?

The first task was to find a home for our frontend (app.flude.guide). We were choosing between three options:

Cloudflare Pages: This was our first candidate, as all our static sites (including the blog) were already running there successfully. But a SaaS app requires Server-Side Rendering (SSR). To run Next.js on Cloudflare, you need Workers and adapters like OpenNext. It turned out that the fresh Next.js 16 and the @clerk/nextjs library are strictly tied to the Node-specific async_hooks module, which simply doesn't exist in the Cloudflare Workers runtime. We couldn't do a static export (output: 'export') because Clerk requires a live server. This option was out.
Vercel: The native harbor of Next.js. We could have launched on the free Hobby tier, but their Fair Use Guidelines are categorical: commercial use is prohibited. Even if the project isn't making money yet, having paid developers on staff already violates the rules. The paid Pro tier for $20/month would be safe, but why pay for infrastructure for a page with near-zero traffic at launch?
Netlify: Unlike Vercel, their free tier officially allows commercial use (confirmed on support forums). Next.js SSR is natively supported, and the Clerk + Netlify combo is battle-tested. We chose them.

The War with Netlify Scanners

Having chosen Netlify, we configured the pipeline and pushed the code to release. The build immediately crashed with a security error.

It turned out that Netlify's built-in secrets scanner saw the string NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY in the configuration and blocked the deploy, deciding that we had committed a private token. We had to spend time digging through the documentation to manually add the SECRETS_SCAN_OMIT_KEYS exception to netlify.toml. We were literally arguing with the host that a public key is, in fact, public.

The Silent Break: Next.js 16 and File Renaming

Having solved the deployment issue, we moved on to authorization. According to the official Clerk documentation, route protection is configured in the middleware.ts file. We set everything up, pushed the code, but authorization simply didn't work—the page let everyone in. No errors in the logs, just silence.

We killed several hours before we realized what was wrong. In Next.js 16, internal conventions had changed: the middleware file is now required to be named proxy.ts. As soon as we renamed the file, authorization worked. Because of one changed word in the framework's convention, the security of the entire application silently disabled itself.

Sudden Deprecation: Chasing Standards

In that very proxy.ts file, we had configured protection for our /account page using a special Clerk function—createRouteMatcher. This was the recommended approach.

But the joy was short-lived. We accidentally discovered that this pattern of route protection at the middleware level was already being marked as deprecated. Clerk was shifting its philosophy towards "Resource-based auth".

We had to urgently rip out all routing logic from proxy.ts, leaving only a "bare" context passthrough (export default clerkMiddleware();), and move the protection inside the server components. Now the /account/[[...rest]]/page.tsx page itself calls await auth.protect() before rendering. Yesterday's "Best Practices" standard turned out to be legacy even before we could release the project.

What's Next?

With much grief, we pushed the frontend through. The SaaS acquired working billing, secure cloud workers, and a portal for getting API keys. The architecture was ready.

But just as we relaxed, we decided to rename the base export format of our core from markdown to hugo_markdown. This harmless decision sent ripples across the entire distributed system, breaking contracts between the engine, workers, and clients. How we eliminated this "butterfly effect"—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/surviving-js-ecosystem-clerk-nextjs

Also read us:

SaaS Economics: Why We Overpaid 10x for Auth and How We Protected the Cloud from Spam

Flude team — Tue, 13 Oct 2026 05:14:37 +0000

Having developed the secure cloud worker architecture described in the previous article, we solved the main engineering task—our parser's core was secure. But now we faced a product problem.

How could we allow developers to test Flude for free without letting them bankrupt us on GCP Cloud Run bills? Every second of parsing costs real money. We needed a Free-Tier, but with ironclad protection against Abuse loads. And the foundation for this protection became the authentication system.

The Foundation: Choosing an Auth Provider

We needed to manage users, quotas, and B2B organizations. We considered three options: Auth0, Supabase Auth, and Clerk.

Auth0 dropped out first: it lacked a native integration with GitLab, and for our CI tool, GitHub, GitLab, and Bitbucket had to be absolutely equal platforms "out of the box."

That left Supabase Auth and Clerk. According to preliminary calculations for a scale of 100,000 monthly active users (MAU), Supabase would have cost us pennies, while the bill from Clerk would be roughly ten times larger.

And yet, we chose the more expensive Clerk. Why?

First, development savings were the deciding factor. Our future Team pricing plan implied multi-tenancy: the entity of an organization, roles (admin and member), and shared billing. Clerk provided the Organizations feature and ready-made UI components "out of the box"—this fit our model perfectly. With Supabase, we would have had to build a B2B role model from scratch.

Second—architectural purity. Supabase is a full-fledged BaaS with its own PostgreSQL database. Our project was already firmly tied to the Google Cloud stack (specifically, Firestore). Dragging a second DBMS into the project just for cheap authentication would have been a mistake.

In the context of projected revenue, the overpayment for Clerk would amount to only a couple of percent. We consciously traded money for months of saved engineering time.

Defense in Depth (Free-Gate)

Equipped with a powerful identification tool, we started setting limits. Our philosophy during the beta phase was simple: we want to support indie developers and their small Open Source projects, while cutting off giant enterprise mono-repos.

To do this, we built a three-level "gate" that cuts off excess load even before the heavy parsing process starts in the cloud:

Entity Limit: The API token is checked by the server. If a request comes from an account linked to an organization—it is rejected. Free access is open only for personal projects, and support for corporate teams is reserved for future paid plans.
Privacy Limit: We make a cheap preliminary request to the public API (e.g., GitHub) before starting a heavy repository clone. If the code is private—rejection (rejected_private).
Greed Metric (5MB Limit): The most important check happens inside the worker after a shallow clone (git clone --depth 1). We don't trust the repository size from the API (it might contain gigabytes of garbage). We weigh only those files that actually match the include_patterns. If the net volume of code exceeds 5 MB, the worker immediately aborts the job.

Dynamic Rate-Limit and the Window of Loyalty

Even small Open Source projects can trigger CI on every tiny commit 100 times a day. To avoid parsing all of this at our expense, we implemented a dynamic rate-limit based on Firestore.

For new accounts, we open a "window of loyalty": for the first 7 days, we allow up to 5 builds per day so the developer can calmly configure CI and play with the Flude configuration.
But a week later, harsh reality sets in—the limit drops strictly to 1 free build per day. If you want to parse documentation on every PR, you'll have to wait for the launch of paid plans, which we are currently only designing.

As a final chord, we embedded a global "kill switch" in Firestore (system_status/global.enabled), which allows us to disable all free builds with a single button in case of any infrastructure attack.

What's Next?

We designed great business logic protecting our servers from spam and integrated a powerful authorization provider. But, as often happens in the JS ecosystem, third-party dependencies have a habit of suddenly breaking.

The integration of Clerk and the transition to the new Next.js 16 standards turned our frontend development into a real quest with deprecated functions and broken routes. About this (and how a single renamed line broke contracts across the entire distributed system)—in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/saas-economics-abuse-protection

Also read us:

The Configuration Trap: How Enterprise Solutions Become SaaS Vulnerabilities

Flude team — Fri, 09 Oct 2026 13:15:44 +0000

After deciding to split the product into Enterprise and SaaS, we started designing the cloud architecture. During the planning phase, the task seemed straightforward: spin up an API, configure container-workers in the cloud, and simply pass the code to them for processing.

However, during a detailed security audit, we discovered that architectural patterns which worked flawlessly in a trusted corporate environment turn a public SaaS into an open hacking sandbox.

Anatomy of a Vulnerability: RCE via Configuration

In the Enterprise version, the configuration logic was maximally flexible: the engine simply read the flude_*_config.json configuration file directly from the client's repository.

The problem lay in exactly how these parameters were translated into parser settings under the hood. Our pipeline relied on Doxygen, which we extended with our own scripts to bypass C++ bugs. To make these filters work, the engine uses a standard Doxygen directive—INPUT_FILTER, which allows executing arbitrary shell commands for processed files.

In an isolated client perimeter, reading settings from the repository was absolutely safe. But when transferring this logic to a public cloud (GCP Cloud Run), a critical vector for Remote Code Execution (RCE) emerged. If a server blindly trusts a flude_*_config.json file from a third-party repository, an attacker can pass through overrides for parser parameters. By injecting a command like rm -rf / or a reverse-shell via INPUT_FILTER substitution, an attacker would force our system to legitimately execute this code on our own servers.

Zero-Trust Architecture and Breaking the Trust

Realizing this risk forced us to completely rethink our approach to configuration. In a public cloud, you cannot trust a single line coming from the outside. We implemented a strict Zero-Trust principle:

Abandoning Client Configurations: The SaaS version will never read flude_*_config.json from a user's repository.
Server Templates: Our cloud worker operates exclusively on hardcoded, versioned server templates for Doxyfiles and configurations.
Secure Parameter Passing: A user can only pass a strictly limited set of safe data-parameters (e.g., directories to include or exclude) via the API, which are rigidly validated on the Control Plane side. Control over which tools and filters run in the container has been permanently handed over to the server.

Fortifying the Perimeter: SSRF and Workload Identity

Having plugged the RCE hole, we turned our attention to network security. Cloning user repositories by a server is a classic Server-Side Request Forgery (SSRF) risk. To prevent attackers from forcing our workers to scan the internal GCP network under the guise of a git repository, we implemented a strict Allowlist: cloning is allowed only via HTTPS and strictly from the domains github.com, gitlab.com, and bitbucket.org.

Additionally, to minimize damage in the hypothetical event of a container breach, we completely abandoned static keys (Service Account Keys), transitioning all workers to dynamic authentication via GCP Workload Identity Federation.

Deferred Threats: Attack via Source Code

Having secured the configuration and network, we asked the next question: what if the attack vector becomes the very source code we are parsing?

We analyzed several highly specific scenarios:

Doxygen Tag Abuse: An attacker could write standard tags like \include, \verbinclude, or \htmlinclude into a code comment with a path like ../../../etc/passwd. These will force the parser to read a file outside the project directory and insert its contents into the final documentation, which is then returned to the client. Our strict directory validation is powerless here, as it only checks the declared input paths, not what Doxygen decides to read during the parsing process.
Malicious Symlinks: If an attacker commits a symlink pointing to system files of the container, and the parser follows it while traversing directories—this is another way to extract data outside the repository sandbox.
Denial of Service (DoS): In our system, the call to the Doxygen binary is limited to 60 seconds, but the parent flude_engine compile process had no timeout. A specially crafted source file (with infinite template recursion or explosive macro expansion) could permanently hang the cloud worker process.

After considering these vectors, we made a conscious decision to defer fixing them.

First, architecturally it was unclear at what level to implement these defenses—especially in light of the upcoming introduction of native parsers. We honestly recorded this debt in the project roadmap under the "Input Validation & Sanitization" section.

Second, the actual risk was significantly reduced thanks to the Data Plane architecture. All workers in Cloud Run are isolated via gVisor, and the containers themselves are absolutely ephemeral. Even in the event of a successful exploit of a symlink or an \include tag, an attacker would read files from a disposable, sterile environment without access to neighboring tasks or the host.

The Accumulating Dissatisfaction with Doxygen

that the incidents with potential RCE via INPUT_FILTER and the vulnerability of standard tags like \include became more than just cloud security issues. They were another wake-up call reminding us of Doxygen's fundamental architectural flaws. Its inflexibility, reliance on shell calls, and poor performance made it a burden for a scalable SaaS solution (and for the architecture as a whole).

Running ahead, we can say: we did not put up with this technical debt. In a couple of weeks, as soon as we finish the series of articles on SaaS infrastructure, we will explain in detail why we finally fired Doxygen.

But before that, we had to solve another pressing cloud problem. We had designed a secure Data Plane, protected from hackers, but now we needed to protect it from bankruptcy. How we built a Free-Tier, protected the API from abuse loads, and designed billing—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/zero-trust-cloud-config-trap

Also read us:

A Tale of Two Fludes: The Architectural Split of Enterprise and SaaS Models

Flude team — Tue, 06 Oct 2026 06:06:10 +0000

Extracting specific code into an independent private submodule, as described in the previous article, allowed us to stabilize the engine's core. The legal boundaries were drawn, and the base architecture—a smart parser and two-level caching—proved its effectiveness.

The next logical step was transitioning from an internal tool to a commercial product. However, at this stage, we faced a fundamental contradiction between the corporate operational model and the demands of the mass market.

The Illusion of Universal Distribution

Initially, Flude's architecture was shaped by the needs of a large corporation. Security requirements in such environments are categorical: the tool must function within a closed perimeter, guaranteeing that the source code never leaves the company's servers.

To solve this, we integrated directly. A package with scripts was handed over to the client and run locally in their CI pipeline, reading configuration files directly from the repository. Within a corporate environment, this delivery model (whether source or bytecode) is ideal: the closed perimeter and strict legal obligations reliably guarantee the safety of intellectual property.

However, when planning to scale to the public market, it became obvious that directly copying the Enterprise distribution approach was technically and conceptually inapplicable.

Losing Control over Distribution

The closed-contract model works perfectly in the Enterprise sector, but to ensure stable product growth, we needed to enter the mass B2B/B2C market: small teams and independent developers.

The obvious problem was that handing over source Python scripts or a ready-made Docker image to users outside of corporate agreements inevitably leads to losing control over product distribution. Without strict legal barriers, open distribution technically allows bypassing licensing mechanisms and uncontrollably replicating the system's core.

Consequently, the approach that is standard for a trusted corporate environment turned out to be structurally incompatible with a public business model.

The Divergence

To protect the product's core, we had to rethink the delivery paradigm itself. Flude's architecture was physically divided into two independent models:

Enterprise Flude: An autonomous, local solution. Delivered exclusively under targeted contracts. Functions offline, processes code locally, and integrates with the configuration directly in the client's repository.
SaaS Flude (Public Market): A cloud-oriented model. The parser's core is never handed over to end users. Instead, a "thin client" (lightweight CI scripts) is provided, whose sole task is packing source files and securely transmitting them via API. All computational load and parsing logic remain encapsulated on our servers.

Moving to the Cloud and the Unexpected Benefit of Binaries

By adopting the SaaS model, we secured our algorithms, but significantly complicated the technical implementation. We had to design a distributed system from scratch capable of receiving and processing third-party code.

Interestingly, it was at this stage that the concept of a "single binary", which we had rejected for public distribution, turned out to be our own salvation. To avoid managing complex Python environments and dependencies on every cloud worker, we compiled our core. A standalone binary became the perfect solution for quick startups inside our own SaaS infrastructure (and later, a convenient delivery format for Enterprise clients).

How we designed the cloud with the Zero-Trust concept in mind, and why configuration files that worked flawlessly locally turned into a critical attack vector (RCE) in SaaS, we'll tell you in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/tale-of-two-fludes

Also read us:

Clean Room: A Refactor That Answered a Legal Question, Not an Architectural One

Flude team — Fri, 02 Oct 2026 04:34:36 +0000

Code born inside the core engine rarely has to stay there forever. This specific move started with a question about legal boundaries. We needed a hard line between our own original logic and knowledge acquired from an external format. One of Flude's renderer families—the one built to precisely mirror an outside documentation tool—sat dangerously close to that line.

Two Risks a Refactor-in-Place Misses

Carving out a module inside the same repository does practically nothing here. A shared commit history leaves the biggest vulnerabilities wide open.

Authorship is the first major headache. Code mirroring an external format gets thoroughly mixed up with the rest of the engine. Proving independent development after the fact becomes incredibly messy. You build a credible evidence trail through total separation—a dedicated repository and a fresh commit history.

The logic itself carries its own risks. The way code bends to match an external format is highly specific knowledge. Tightly weaving it into the engine blurs the line between a general tool and client-specific insights.

A physical split cleanly shuts down both risks. A simple folder rename just papers over the cracks.

How It Played Out in Practice

We pulled the renderer family into a standalone git repository. It plugs into the main project as a private submodule. The engine interacts with it through the standard renderer interface. The class simply lives in a new zip code. The rest of the system barely noticed the change.

Standing up the new repository brought plenty of friction. The test harness relies heavily on the engine's core models. We had to check out the main engine inside the new CI pipeline. Shared relative paths completely broke down. Nailing the dependency paths took a few frustrating attempts. We temporarily dropped the markdownlint gate before restoring it to maintain project standards. Turning the CI pipeline green took some serious trial and error.

The quality checks moved right alongside the code. The entire infrastructure for comparing results against reference documents shifted over. Leaving it behind would have made it impossible to spot rendering drift.

The Result

The main engine feels much leaner now. The core and the remaining renderer family can evolve at their own natural pace. The extracted code operates completely independently. It boasts its own CI and a bulletproof line of provenance. We no longer have to worry about proving its origins retroactively.

Originally published on our blog: https://blog.flude.guide/blog/clean-room-private-plugin

Also read us:

Ghost in the Wire: Our Two-Day Hunt for a Network Bug That Didn't Exist

Flude team — Tue, 29 Sep 2026 03:38:53 +0000

Our self-hosted runner serves CI for five repositories on one physical VM. It started acting completely crazy. The engine job sat queued in GitHub Actions for over a day. It just hung there doing absolutely nothing. The supervisor meant to pick it up was restarting itself every two and a half to three minutes around the clock. We called this endless loop "the carousel" and wasted two days chasing ghosts.

The Symptom, and the First False Lead

We already had a documented network bug in this exact infrastructure. A classic VirtualBox NAT "black hole" used to kill TCP connections inside the guest VM silently. The guest OS would consider the connection alive forever. We tracked down and closed that issue a week and a half earlier.

When the supervisor started hanging on the host side, our reflexes kicked in. We assumed it was the exact same disease in a fresh disguise. That feeling was convincing enough to steer the whole investigation sideways for a full day. The watchdog detected the hang, killed the process, and brought it back up. Every new instance hung reliably after 150-190 seconds. That unnatural precision should have made us suspicious way earlier.

Escalation: Packets, Memory Dumps, Power Settings

A thorough and completely useless sweep through network theories followed. We disabled a third-party VirtualBox NDIS filter attached to the Wi-Fi adapter. We reset the MIMO power-saving settings. Nothing made a dent in the problem.

Running pktmon during an actual hang showed a completely healthy network. New TCP connections to GitHub established in 60 milliseconds. The "hung" process supposedly couldn't send a single byte. We wrongly concluded the hang was happening before the packet got sent.

A memory dump of the process via WinDbg showed twenty-two threads. None of them were sitting inside an HTTP call. We spotted suspicious modules like antivirus drivers and crypt32 certificate validators. Every single one of these plausible candidates turned out to be a dead end.

We did notice one genuinely interesting detail. The host machine dropped into Modern Standby right when the hangs occurred. We disabled sleep and screen timeouts entirely. The restarts kept happening on the exact same schedule. The correlation was totally real—but completely unrelated.

A Second Pair of Eyes Solves Half the Problem

Another session looked at the logs and caught what we missed. The watchdog was treating plain silence as a fatal hang. The GitHub API poll writes nothing when the queue is empty. It simply sleeps for twenty seconds and tries again. A few quiet cycles looked exactly like a dead process to the watchdog.

We verified this by pulling the I/O counters directly. The bytes were genuinely moving. The process was perfectly healthy with nothing to log. We updated the watchdog to check those exact counters before killing anything. The restarts stopped completely. The engine job still refused to dispatch.

The Real Cause

We ran a direct test using the exact token the supervisor uses. Querying runs?status=queued for engine returned zero results. Running the same query with a personal token showed the job instantly. Access rights were perfectly fine.

The answer was buried in the API response itself. The run had a "pending" status. The job inside that run was correctly marked as "queued". Our code filtered runs at the top level. GitHub had split the run's status from the job's status. One bad line of filtering made the code completely blind to a waiting job.

We dropped the top-level filter completely. The nested check handled everything perfectly on its own. The very first poll picked up the stuck jobs and ran the full suite beautifully.

A Small Postscript: Fixing Our Own Fix

Removing the filter spawned a new headache. The code started pulling dozens of recent runs and making separate job requests for each one. The call volume spiked massively. The carousel of restarts came back because I/O genuinely dropped to zero. We added client-side filtering to skip runs marked as completed. Everything stabilized immediately.

What We're Taking Away From This

Solid diagnostic tools will gladly prove the wrong theory. WinDbg and packet sniffers correctly showed a healthy network. We just kept interpreting those results to fit our initial bias. A simple API call with the right token proved infinitely more useful than all the heavy debugging gear.

Next time, we get into legal boundaries and clean room design. We ended up moving an entire renderer family into a private plugin. It was the only way to firmly separate our core engine from logic shaped by an external client format.

Originally published on our blog: https://blog.flude.guide/blog/ghost-in-the-wire

Also read us:

Growing Pains with Five Repositories: Gitlinks and Dual CI

Flude team — Fri, 25 Sep 2026 04:02:27 +0000

Usually, infrastructure stories start with the words: "Our project got too big, and we decided to split it into parts." But with us, everything was different.

From the very first minute of the project, we consciously chose an architecture consisting of a scatter of independent repositories. The Pipeline, engine, design-docs, and user-docs projects were created almost simultaneously, and the .gitmodules configuration file sat in the very first commit of the parent repository.

For the sake of code cleanliness, CI isolation, independent history, and separation of access rights, we built the system on Git Submodules from day one. The idea looked flawless on paper. But here is what we had to pay for it in practice.

The Gitlink Trap: The Ghost of a Local Commit

The first price we paid was the non-obvious mechanics of state synchronization. When you use submodules, the workflow demands strict discipline. If you edit code inside engine/ (while physically located in the folder of the parent repository Pipeline), you need to make two pushes. First, a git push from the folder of the submodule itself, and then, after moving up to the root of Pipeline—you must commit the updated pointer (gitlink) to the new engine commit and push Pipeline itself.

It is incredibly easy to forget the first push. On your local disk, the commit physically exists. Everything builds, the tests are green, and Pipeline happily goes to the cloud. But this only surfaces on a fresh checkout in CI. The git submodule update --init command fails with the text:

fatal: remote error: upload-pack: not our ref <sha>

ext

GitHub honestly replies: "This commit does not exist in the remote repository."

One day, three submodules fell into this trap simultaneously (design-docs, engine, user-docs). The cause was identical—small commits were made right inside the submodule folders, the gitlink in Pipeline was synchronized, but the local commits never left the developer's laptop.

There was also a colorful nuance: the fourth submodule, ude_promotion (which hosts the site), broke so badly that a normal push couldn't fix it—its local branch fell so far behind upstream that an interactive rebase was required before pushing.

Dual CI: One Code in Two Personalities

The second reckoning came from CI. The engine repository (our core) had two personalities. On the one hand, it was a standalone repository with its own build process, which checked out only this repository. On the other—it was a submodule inside Pipeline, which was tested in the context of the entire parent project.

These two personalities inevitably clashed, and this happened in two different stages.

The First Strike: Missing Dependencies
The simplest case. The test_integration_scripts.py test tried to reach the verify_pages/check_links modules, which existed only in the parent Pipeline repository. During a standalone checkout of engine, these files simply weren't on the disk. We fixed it roughly but effectively—we added a single line to the ci.yml configuration, strictly ignoring this test from the standalone run (pytest --ignore=...). But this was a one-off patch, not a systemic solution.

The Fatal Directory Ascent
Soon the bug returned in a new guise. One by one, tests started failing that used a classic trick to find parent artifacts:

Path(__file__).resolve().parents[2]

ext

In a nested run, this code ascended exactly to the root of the parent repository. But in the standalone build of engine, it jumped above the root of the repository, straight into the system filesystem of the CI runner, and crashed with a hard AssertionError.

It was after the second occurrence of this bug that we realized point patches no longer worked. We needed a general convention.

We introduced an elegant marker hack: tests began checking for the presence of our system marker directory (let's call it .workspace_config) at the ascended level. If the directory is there—it means we are in a nested context, and the tests run. If it's not there (standalone)—we politely use pytest.mark.skipif, and the tests are quietly skipped without failing with a red status.

Conclusion

A Multi-repo architecture does not give you magical "loose coupling" for free on day one.

It demands flawless push synchronization discipline and forces you to design code so that it is ready to work in several different execution contexts simultaneously. These lessons cost us broken pipelines, but without them, the system would not have survived further growth.

But the problems with Git and dual CI turned out to be just a prelude. The real mysticism began when the network intervened, and our pipeline started crashing because of a "ghost" in the wires. How we investigated an elusive network bug—read about it in the next episode.

Originally published on our blog: https://blog.flude.guide/blog/multi-repo-gitlinks-dual-ci

Also read us:

Escaping the Cloud: How Five Repositories Stopped Sharing One Minute Limit

Flude team — Tue, 22 Sep 2026 04:13:31 +0000

Splitting the monorepo granted each of our five components an independent continuous integration pipeline. Architecturally, we achieved perfect component isolation. The problem hid in the strict limits of free GitHub Actions minutes. A unified build line used to efficiently compile everything in a single run. Five separate pipelines began burning through our starting two thousand minutes like wildfire by launching independent jobs for every tiny commit.

One Physical Machine for Five Clients

We flatly refused to buy extra cloud minutes and deployed our own self-hosted runner on a standard Windows office laptop. Inside ran an Ubuntu virtual machine powered by VirtualBox. This single virtual machine methodically took turns servicing tasks for all five repositories.

Official GitHub documentation heavily advises against attaching local runners to public projects due to remote code execution risks via malicious pull requests. Our repositories are completely private, keeping us safe from outside attacks. We nevertheless implemented paranoid hygiene to guarantee build reproducibility. A single runner processes code from five different projects, and leftover file debris from a previous build could easily ruin the next one.

We isolated the guest system through classic NAT (nic1=nat), blocking its access to the host's local network. Before every new job, a script ruthlessly rolled the virtual machine back to a clean snapshot. The runner itself launched with the --ephemeral flag for one-time registration, while an additional Unregister-OrphanedRunner.ps1 script scrubbed hanging records via the API.

The Price of Genuine Isolation with a Custom Supervisor

Many wonder why we wrote a custom polling script when GitHub offers built-in Organization Runners. The standard mechanism certainly allows sharing runner pools across multiple repositories. It completely fails at managing the lifecycle of the virtual environments themselves.

The --ephemeral flag successfully removes a runner from the pool after task completion. It does nothing to restore the filesystem to its original pristine state. To wrap every single task in a strict cycle of snapshot rollbacks and VM launches, we still needed an external controller. The custom REST API polling script became a mandatory tax for ensuring a clean environment for every build.

The Watchdog Monitoring Itself

The single server became a critical point of failure for the entire project. Any freeze in the supervisor instantly paralyzed code checks and documentation publishing. For the first few weeks the script ran unsupervised and regularly demanded manual restarts whenever VirtualBox spawned zombie processes. We wrote a separate watchdog script tasked with automatically restarting the frozen controller.

The logic was primitive: if the supervisor.log file stopped updating and the VBoxHeadless process wasn't running, the watchdog pulled the plug. We shot ourselves in the foot on day one. The watchdog diligently wrote its own diagnostic logs into the very same supervisor.log file. It constantly updated the file modification time, reliably blocking its own check condition.

We quickly separated the logs into different files, and the system appeared to run smoothly. The following morning the watchdog lost its mind and began systematically killing a perfectly healthy supervisor for no apparent reason. This sent us down a two-day detective investigation into a network ghost in the machine, which we will cover in the next part.

Originally published on our blog: https://blog.flude.guide/blog/escaping-to-self-hosted-runners

Also read us: