DEV Community

Cover image for From Monolith to Modular Architecture: Refactoring a 2,353-Line PyQt6 Desktop App Without Regressions
freerave
freerave

Posted on

From Monolith to Modular Architecture: Refactoring a 2,353-Line PyQt6 Desktop App Without Regressions

#python #architecture #opensource #softwareengineering

How we transformed DotGhostBoard from a monolithic desktop app into a decoupled, test-driven modular architecture — decomposing monolithic classes, isolating SQLite, and adding 86 tests with zero regressions.

A 2,353-line Dashboard class is rarely just a large file. It is usually a symptom that storage, business logic, security, synchronization, and UI behavior have all collapsed into the same boundary.

That was the state of DotGhostBoard v1.5 — an open-source, privacy-first clipboard manager for Linux built with Python and PyQt6.

What began as a snappy, native desktop tool had accumulated layers of complexity: local AES-GCM encryption, dynamic database queries inside widget click handlers, regex secret detectors on keystrokes, and peer-to-peer LAN sync. Modifying a single button callback risked tripping up the entire Qt event loop.

To prepare the codebase for its upcoming v2.0 milestone (native Wayland protocol support, end-to-end device sync, and extensible plugins), we undertook a comprehensive four-phase refactoring initiative.

The outcome:

  • ui/dashboard.py shrank from 2,353 to 1,535 lines (-35% LOC).
  • The automated test suite expanded from 220 to 306 tests across 25 modules (+39% tests).
  • The entire refactor passed the test suite with zero detected user-facing regressions.

Here is the exact architectural blueprint, design patterns, and lessons learned.

🛑 The "Before" Architecture: The Desktop App Debt Trap

Desktop applications built with GUI frameworks like Qt, Tkinter, or wxWidgets are notoriously susceptible to the monolithic "Blob" antipattern:

[ THE MONOLITHIC ARCHITECTURE ]
┌──────────────────────────────────────────────────────────────────┐
│ Dashboard (ui/dashboard.py) — 2,353 Lines                        │
│  - Raw SQL queries (`cursor.execute(...)`)                       │
│  - AES-GCM crypto encryption/decryption                          │
│  - Card widget lifecycle & pagination                            │
│  - LAN socket broadcasting                                       │
│  - Password prompt modals & session key caching                  │
└──────────────┬──────────────────┬─────────────────┬──────────────┘
               │                  │                 │
      Direct Storage Calls   Qt Event Loop    Tightly Bound Polling
               ▼                  ▼                 ▼
     [ Monolithic SQLite ]   [ UI Signals ]   [ QClipboard Event ]
Enter fullscreen mode Exit fullscreen mode

Three critical pain points made future feature work fragile:

  1. The 2,353-Line Monolithic Dashboard: Button callbacks were directly reading from SQLite, instantiating cipher suites, managing session timers, and firing off network broadcasts.
  2. The 1,100-Line Storage Monolith (core/storage.py): One file handled connection lifecycle, schema migrations, clip CRUD, tag management, collection associations, and peer credentials.
  3. Platform-Tied Clipboard Listener: Capturing clipboard data was tied directly to PyQt's QClipboard.dataChanged event loop. Headless unit testing without an X11 display server was difficult, and supporting Wayland's native wlr-data-control protocol was functionally impossible.

🎯 The Core Architectural Principle

To untangle this, we established a strict rule of separation:

Controllers coordinate UI behavior. Services own business rules. Repositories own persistence. Backends own platform integration.

UI (Widgets / Dialogs)
      │
      ▼
Controllers (ui/controllers/ - QObject Behavioral glue)
      │
      ▼
Services (core/services/ - Headless business logic)
      │
      ▼
Repositories (core/storage/repositories/ - Domain data access)
      │
      ▼
Database (core/storage/database.py - SQLite context & migrations)
Enter fullscreen mode Exit fullscreen mode

Phase 1: Modularizing Storage with the Repository & Facade Patterns

Rather than allowing UI components to issue raw SQL queries, we decomposed core/storage.py into a focused package:

core/storage/
├── __init__.py           # 100% Backward-compatible Facade
├── database.py           # Context-managed connection layer & test paths
├── migrations.py         # Transactional schema migration engine
└── repositories/
    ├── clips.py          # CRUD, AES-256-GCM encryption, image deduplication
    ├── tags.py           # Normalization, tag extraction, rename/delete
    ├── collections.py    # Grouping, filtering, cascade rules
    ├── peers.py          # LAN device trust & credentials
    └── stats.py          # Aggregations & performance metrics
Enter fullscreen mode Exit fullscreen mode

The Backward-Compatible Facade Pattern

A common blunder when refactoring persistence is forcing a sweeping rewrite across every caller at once.

To keep Git diffs manageable and avoid breaking other branches, core/storage/__init__.py serves as a facade, re-exporting original function signatures:

# core/storage/__init__.py (Facade)
from .database import get_db, init_db, close_db
from .repositories.clips import save_clip, get_clips, delete_clip
from .repositories.tags import get_all_tags, normalize_tag
from .repositories.collections import get_collections

# Existing callers continued to function without changing an import!
Enter fullscreen mode Exit fullscreen mode

Transactional Schema Migrations

We implemented an in-house, zero-dependency migration engine (core/storage/migrations.py):

  • Every schema migration executes within an atomic transaction.
  • Legacy schema detection auto-promotes pre-existing v1.5 databases.
  • Includes downgrade guards to prevent older versions from corrupting newer schemas.

Phase 2: Decoupling the Clipboard Engine with Protocols

Previously, the clipboard listener was embedded inside Qt widget callbacks. We extracted this into a headless policy engine:

[ Raw Clipboard Event ]
          │
          ▼
┌─────────────────────────────────────────────────────────────┐
│                 ClipboardPipeline (Pure Logic)              │
│                                                             │
│  1. Validation      ──► Ignore empty or malformed payloads  │
│  2. App Whitelist   ──► Filter out sensitive applications   │
│  3. Paranoia Filter ──► Check private browsing & incognito  │
│  4. Secret Detector ──► Scan for API keys, tokens, SSH keys │
└─────────────────────────────┬───────────────────────────────┘
                              │
                    CaptureDecision (Action)
                              │
          ┌───────────────────┼───────────────────┐
          ▼                   ▼                   ▼
    [ SAVE_NORMAL ]    [ SECRET_CANDIDATE ]    [ IGNORE ]
Enter fullscreen mode Exit fullscreen mode

Protocol-Based Backends

By specifying a clean Python typing.Protocol, the pipeline doesn't care whether events originate from X11, Qt, or Wayland:

# core/clipboard/backend.py
from typing import Protocol, Callable, Optional
from .events import ClipboardEvent

class ClipboardBackend(Protocol):
    def start(self, callback: Callable[[ClipboardEvent], None]) -> None: ...
    def stop(self) -> None: ...
    def get_current_text(self) -> Optional[str]: ...
Enter fullscreen mode Exit fullscreen mode

Benefits:

  • We wrapped Qt's clipboard event handling into QtClipboardBackend.
  • We introduced a lightweight MockClipboardBackend for unit tests, enabling headless test execution in under 4 seconds without Xvfb.
  • Wayland readiness: The architecture is decoupled so a native Wayland backend can be introduced cleanly in v2.0.

Phase 3: Domain Service Layer & Cryptographic Hardening

Desktop apps frequently leak business logic into UI event listeners. For instance, calculating whether a snippet should trigger an "Auto-Pin" recommendation based on copy frequency was embedded inside a widget slot.

We extracted pure Python service classes into core/services/:

  • HistoryService: Manages pagination windows, debounced search, duplicate suppression, and copy thresholds (PIN_SUGGESTION_THRESHOLD = 5).
  • CollectionService: Handles categorization rules and cascade unlinking.
  • SecurityService: Manages master password verification, session lock timeouts, and key rotation.
  • SyncService: Validates LAN peer trust handshakes.

Cryptographic Rigor & Envelope Encryption

DotGhostBoard features an encrypted storage mode (The Vault). We hardened key derivation and isolation:

  1. Two-Tier Key Derivation Chain: $$\text{Master Password} \xrightarrow{\text{PBKDF2-HMAC-SHA256 (600,000 iter + Salt)}} \text{Base Key} \xrightarrow{\text{HKDF-SHA256 (Domain Context)}} \text{Vault KEK}$$
    • Passwords first pass through PBKDF2-HMAC-SHA256 with 600,000 iterations and a persistent 256-bit random salt.
    • The resulting base key is then fed through HKDF-SHA256 with domain context b"dotghostboard:vault:v2". This guarantees that the Vault key is cryptographically independent from other application keys.
  2. Physical Database Isolation & Envelope Encryption:
    • Vault items live in a physically distinct SQLite database (vault.db).
    • Items are encrypted using a fast Data Encryption Key (DEK). The DEK is encrypted with the Key Encryption Key (KEK).
    • Instant Password Rotation: Changing the master password only requires re-wrapping the 32-byte DEK (a constant-time operation taking ~2ms in our local benchmarks), rather than re-encrypting gigabytes of history data.
  3. Defense-in-Depth:
    • Added secure_zero() providing best-effort memory scrubbing for sensitive mutable bytearray buffers after cryptographic operations.

Phase 4: Decomposing the 2,353-Line Monolithic Qt Dashboard

Decomposing the central Dashboard window was the most delicate step.

Rather than letting the main window handle every event, we introduced Behavioral QObject Controllers:

                    ┌────────────────────────┐
                    │       Dashboard        │
                    │  (ui/dashboard.py)     │
                    │   Layouts & Visuals    │
                    └───────────┬────────────┘
         ┌───────────────┬──────┴────────┬───────────────┐
         ▼               ▼               ▼               ▼
┌─────────────────┐ ┌──────────┐ ┌───────────────┐ ┌───────────┐
│HistoryController│ │Collection│ │SecurityContr. │ │SyncContr. │
│• Infinite scroll│ │• Sidebar │ │• Vault unlock │ │• LAN P2P  │
│• Search debounce│ │• Drag/drop│ │• Secret reveal│ │• Pairing  │
│• Card lifecycle │ │• Category│ │• Session timer│ │• Discovery│
└─────────────────┘ └──────────┘ └───────────────┘ └───────────┘
Enter fullscreen mode Exit fullscreen mode

Each controller is a dedicated QObject wired through dependency injection:

  • HistoryController: Manages card pagination, card lifecycle, and 150ms search query debouncing.
  • CollectionController: Manages category navigation, drag-and-drop targeting, and folder CRUD dialogs.
  • SecurityController: Oversees session lock/unlock events, secret masking, and unlock prompts.
  • SyncController: Coordinates peer discovery listings, pairing dialogs, and network status styling.

The Impact:

  • ui/dashboard.py shrank by 818 lines (2,353 → 1,535 lines).
  • Zero raw SQL or crypto primitives remain in the UI layer.
  • Extracted PinSuggestionToast into its own reusable widget module (ui/widgets/pin_toast.py).

🐛 4 Subtle Bugs Discovered (and Fixed) During Refactoring

A comprehensive refactor with high test coverage inevitably unearths long-standing subtle bugs:

1. The Recursive Clipboard Sync Loop

  • Issue: When a user clicked an existing snippet card in the GUI to copy it to their system clipboard, the clipboard watcher detected the change and triggered a LAN broadcast as if it were a new external capture, creating an infinite echo loop between paired devices.
  • Fix: Enforced strict signal boundaries. The LAN sync broadcast is triggered only by physical captures originating from Watcher, never from internal UI card copy actions.

2. The $O(N)$ In-Memory Collection Calculation

  • Issue: Populating collection card counts loaded the entire history table into Python and computed counts using list comprehensions.
  • Fix: Replaced with a single SQL query utilizing LEFT JOIN and COUNT(clips.id) GROUP BY collections.id, slashing dashboard initialization time from ~210ms to under 8ms.

3. N+1 Deletes in Periodic Cleanup

  • Issue: The routine cleaning old clips performed individual DELETE FROM clips WHERE id = ? calls inside a Python iteration.
  • Fix: Refactored to bulk DELETE FROM clips WHERE id IN (...) executed inside a single transaction.

4. Encapsulation Leaks in Session Management

  • Issue: Widgets were directly reading and mutating private session key attributes across controller instances.
  • Fix: Replaced with a validated set_session_key() public method enforcing explicit lifecycle transitions.

📊 Before vs. After: Metrics

Metric Before (v1.5.7) After (v1.6.0 Phantom) Impact
Dashboard File Size 2,353 lines 1,535 lines -35% LOC (-818 lines)
Storage Architecture 1 Monolithic File 7 Decoupled Modules Repository Pattern + Facade
Automated Tests 220 tests 306 tests +39% tests (+86 tests across 25 modules)
Test Execution Time 14.2s (with Qt displays) 3.8s (Mock Backends) 3.7x faster test execution
User-Facing Regressions Baseline 0 detected regressions Verified via test suite & manual pass
Wayland Readiness Qt-coupled Protocol-abstracted Architecture prepared for native backend

📐 How We Measured

To ensure our metrics were empirical and reproducible:

  • Lines of Code: Measured using cloc excluding comments, docstrings, and blank lines.
  • Test Counts & Speed: Measured via pytest run times on an identical Ubuntu 24.04 environment.
  • Database Latencies: Profiled with sqlite3 EXPLAIN QUERY PLAN and Python's time.perf_counter_ns().
  • Regression Verification: Confirmed across 306 automated unit/integration tests and a full manual smoke test on X11 and Wayland sessions.

💡 Key Takeaways for Refactoring Desktop Apps

  1. Do Not Start Refactoring Without a Fast Safety Net: Our initial 220-test suite gave us a fast safety net and exposed breaking changes as the refactor progressed. Build your test harness before you touch architecture.
  2. Use Facades to Keep PRs Manageable: Maintain backwards compatibility at module boundaries. Your refactoring doesn't need to break every import path simultaneously.
  3. Decouple Event Loops from Business Logic: Never let UI event loops govern application state. Pure Python classes with Protocol definitions make your business logic portable, maintainable, and lightning-fast to test.
  4. Isolate Signal Flow Early: GUI architectures crumble when signals cascade unpredictably. Define explicit component boundaries for what generates data versus what consumes it.

🚀 What's Next?

With the architecture of DotGhostBoard v1.6.0 "Phantom" stabilized, we are actively developing the v2.0.0 "Cerberus" roadmap:

  • Native Wayland clipboard engine (wlr-data-control).
  • Authenticated peer-to-peer device sync.
  • Extension and plugin system.

🔗 Links & Resources

Have you ever refactored a monolithic desktop application? What patterns helped you untangle complex legacy UI code? Share your thoughts in the comments below!

Top comments (0)