Autonomous Database Evolution in Flutter/Dart: Dynamic Schemas, Zero-Downtime Migration & Self-Healing for AI Agents

Abstract: As AI agents and automated operations (O&M) systems transition into the era of long-lived continuous operation, the traditional persistence paradigm—freezing data models at compile time and executing scripted migrations inside maintenance windows—is breaking down. Cloud-based autonomous exploration loops synthesize new memory entities at runtime, while edge agents and embodied devices running continuously for months or years require schema changes to take effect immediately, encryption keys to rotate without disrupting live traffic, accumulated transient data to self-metabolize, and machine-readable diagnostics to drive unattended self-healing. Grounded in established systems standards (Google F1's asynchronous schema-change protocol, schema forward/backward compatibility rules), cryptographic compliance (NIST SP 800-57, RFC 8439/5869, OWASP MASVS), and the concurrency constraints of Dart's single-threaded event loop and Flutter's frame budgets, this article presents an end-to-end architecture for online autonomous database evolution and runtime schema crystallization in Flutter/Dart. After an objective cross-section of the Dart persistence ecosystem, we demonstrate concrete production patterns—spanning runtime schema inference, zero-downtime migration, anti-entropy self-tuning, and an OODA self-healing loop backed by the ResultStatus diagnostic contract—using ToStore as our reference runtime engine.

Keywords: Online Schema Evolution; Dynamic Table Creation; Zero-Downtime Migration; Google F1 Protocol; Dart Event Loop; Envelope Key Rotation; Storage Entropy Governance; ResultStatus Automated Diagnostics; Flutter; Dart

---

1. Always-On Services: Three Contracts Rewritten at Runtime

A mobile release can wait for an app update; a live autonomous agent serving external requests cannot.

Autonomous exploration loops in the cloud continuously uncover new entity classes that never existed during initial design reviews: tool-invocation traces, episodic memory snapshots, dynamic skill registries, and pending verification hypotheses. Edge and client-side systems face an even tighter constraint: processes embedded inside domestic robots, in-vehicle terminals, or local desktop copilots routinely endure OS kills and cold restarts, yet physical-world interactions cannot pause for a "scheduled maintenance window." To external callers and end users, what matters is continuous availability—not which integer schema version the binary happens to sit on.

Multi-year continuous operation introduces a second, quieter failure mode: transient exploration traces, intermediate reasoning contexts, and debug telemetry accumulate orders of magnitude faster than durable knowledge. Left unchecked, scratch records eventually bury core memories. Offloading cleanup to ad-hoc cron scripts that contend with foreground write locks does not constitute autonomous tuning; it merely defers storage bloat until disk exhaustion or a P99 tail-latency cliff triggers an outage.

Treating "database upgrades" as release artifacts only works when contracts change strictly between versions. In a long-lived autonomous system, at least three distinct contracts mutate continuously within a single deployment:

1. The Schema Contract: Tables, columns, indexes, primary-key topologies, and integrity constraints define which facts the system recognizes and enforces. Because agent ontologies are open-ended, pre-allocating speculative nullable columns into an initial table merely accumulates sparse bloat and scattered application-layer checks. Schemas must evolve and materialize dynamically at runtime.
2. The Secrecy Contract: Cryptographic keys protecting data at rest carry strict compliance lifetimes and must rotate periodically without pausing concurrent reads or writes. If every scheduled rotation forces a full-database ciphertext rewrite, security hygiene collapses under availability SLAs.
3. The Entropy Contract: Scratch reasoning buffers, telemetry logs, and working caches have bounded lifetimes and footprint ceilings. Retention windows and capacity caps must be encoded directly into the schema contract so the engine can metabolize waste smoothly and tighten thresholds autonomously under pressure.

All three contracts must stay active and closed-loop simultaneously. Evolving columns without retention governance causes unbounded bloat; purging data without schema alignment forces agents to cram rich facts back into opaque JSON blobs—pushing indexing, querying, and validation complexity straight back into application code.

---

2. Why Conventional Persistence Paths Break Down: Internals, Dart Concurrency, and Ecosystem Trade-Offs

Before designing an autonomous evolution layer in Flutter/Dart, we need to examine the underlying storage mechanics and Dart runtime constraints that bottleneck conventional persistence approaches.

2.1 Three Structural Bottlenecks in Traditional Relational Engines

1. Exclusive Locks and Full-Table Copy Rebuilds: Embedded relational engines such as traditional SQLite do not support lock-free in-place alterations for most structural changes (dropping columns, tightening NOT NULL constraints, altering data types, or changing primary keys). Under the hood, they typically run a blocking transaction: create shadow table → INSERT INTO shadow SELECT ... → drop original table → rename shadow table. At millions of rows, this monopolizes write locks, stalls foreground traffic, and doubles peak disk consumption. While server systems often emulate online DDL via gh-ost or pt-osc patterns (shadow tables + dual-write triggers + chunked cursor backfills), maintaining trigger-based write amplification, deadlock risks, and double-storage spikes is brittle on resource-constrained edge nodes.
2. Fragility of Linear Version Chains: Scripted onUpgrade(oldVersion, newVersion) callbacks assume schema evolution follows a single, developer-known monotonic line. When autonomous agents synthesize tables dynamically at runtime across decoupled nodes, schema histories branch like a graph rather than a line—quickly breaking version-number scripts via skipped versions and state drift.
3. Compile-Time Codegen Lock-In: Modern client ORMs commonly rely on build-time code generation (build_runner) to guarantee static type safety. That trades away runtime mutability: once compiled, the domain model is frozen, making it impossible for a running agent to synthesize new tables or adjust constraints dynamically in production.

2.2 Dart's Single-Threaded Event Loop vs. Flutter Frame Budgets

A widespread misconception in cross-platform engineering is that wrapping database migrations or bulk cleanups inside async/await prevents UI jank. In reality, Dart's execution model imposes strict scheduling limits:

• Dart runs on a single-threaded event loop with memory-isolated Isolates. Unless explicitly offloaded to worker Isolates, UI layout, JSON serialization, and synchronous database work all compete for the Root Isolate's event queue. At 60Hz, each frame allows just 16.6 ms; on 120Hz displays, that budget shrinks to 8.3 ms.
• Even inside an async function, any synchronous computation segment between two await suspension points—such as row re-encoding, index reconstruction, or large scan filtering—that exceeds 8.3 ms will cause dropped frames. Conversely, yielding indiscriminately after every handful of rows floods the event queue and crushes batch throughput on headless servers or robotics controllers.

A robust Dart/Flutter non-blocking architecture therefore relies on two complementary mechanisms:

1. Worker Isolate Offloading with Zero-Copy Boundaries: Isolate CPU-intensive encryption, compression, and index rebuilds onto background worker Isolates, passing large byte buffers via TransferableTypedData to avoid redundant memory copies. On Flutter clients, non-urgent housekeeping can further leverage SchedulerBinding.instance.scheduleTask(..., Priority.idle) to execute inside inter-frame idle windows.
2. Cooperative Time-Slicing: Chunk long-running loops by elapsed wall-clock budgets rather than fixed row counts—holding ~8 ms slices on UI-facing clients while relaxing to ~50 ms throughput slices on headless server or agent workers.

import 'dart:async';

/// Engine-agnostic cooperative time-slicer guarding UI frame deadlines (~8ms)
/// or headless server throughput windows (~50ms).
class CooperativeTimeSlicer {
  CooperativeTimeSlicer({required this.budgetMs}) : assert(budgetMs > 0);

  final int budgetMs;
  final Stopwatch _watch = Stopwatch();

  void start() {
    _watch
      ..reset()
      ..start();
  }

  /// Yields to the Dart event loop via a zero-duration timer (unlike microtasks,
  /// Future.delayed(Duration.zero) allows pending UI frames and I/O events to run).
  Future<bool> yieldIfBudgetExceeded() async {
    if (_watch.elapsedMilliseconds < budgetMs) return false;
    await Future<void>.delayed(Duration.zero);
    _watch
      ..reset()
      ..start();
    return true;
  }
}

2.3 Cross-Section of the Dart Persistence Ecosystem

Evaluating Dart/Flutter storage stacks purely by their technical mechanisms clarifies where each fits best:

Persistence Approach	Representative Tools	Core Mechanism	Engineering Boundary for Online Autonomous Evolution
Raw SQLite bindings	sqflite, etc.	Direct SQL strings with manual version callbacks	High flexibility, but complex DDL risks blocking table-copy rebuilds; requires bespoke triggers, WAL management, and error parsing
Compile-time typed ORMs	drift, etc.	Code-generated type-safe tables, DAOs, and reactive queries	Excellent for conventional apps with static schemas; build-time codegen locks models against runtime table synthesis
Local object / KV stores	isar, hive, etc.	Fast object serialization and compact local indexes	Strong local performance, but built around compile-time annotations or schemaless KV without full online relational evolution or machine diagnostic contracts
Runtime-evolving engine	ToStore	Native runtime/declarative evolution, time-sliced async migration, envelope key rotation, and structured ResultStatus specs	Currently one of the few Dart-native engines unifying dynamic tables, zero-downtime migration, hot rotation, self-cleaning retention, and machine diagnostics

For conventional apps with predictable schemas, compile-time ORMs remain an outstanding choice. However, when a unified Dart codebase must power both cloud services and edge AI agents requiring runtime table synthesis, non-blocking refactoring, hot key rotation, autonomous retention, and self-healing diagnostics, ToStore is currently the only Dart-native option covering the full closed loop out of the box. Throughout the remainder of this article, we pair engine-agnostic algorithms with concrete ToStore (package:tostore/tostore.dart v3.6.1 public contract) implementations—strictly bounded to public APIs and observable behavior without exposing internal storage internals.

---

3. Practice I: Dynamic Table Synthesis — From Runtime Schema Crystallization to Materialization

AI agents should not invent arbitrary DDL on every prompt; instead, mature agents follow a Memory Crystallization pipeline:

• L0 Unstructured Exploration Buffer: Fast ingestion of raw tool calls and environmental observations as JSON payloads.
• L1 Runtime Schema Inference: Once sufficient samples accumulate, a background analyzer applies type-lattice widening (integer ⊂ double ⊂ text), ISO-8601 datetime detection, nullability ratios, string length bounds, and cardinality estimation for index selectivity.
• L2 Schema Materialization: Emit an engine-agnostic intermediate representation (IR) blueprint, validate it, and materialize it into a constrained, indexed physical table.

3.1 Engine-Agnostic Runtime Schema Inferencer in Pure Dart

The following RuntimeSchemaInferencer requires zero database dependencies and translates raw agent exploration batches into structured schema blueprints ready for any target persistence backend:

import 'dart:math' as math;

enum InferredFieldType { integer, doubleNum, boolean, datetime, array, json, text }

class InferredFieldSpec {
  const InferredFieldSpec({
    required this.name,
    required this.type,
    required this.nullable,
    this.maxLength,
    this.suggestIndex = false,
  });

  final String name;
  final InferredFieldType type;
  final bool nullable;
  final int? maxLength;
  final bool suggestIndex;
}

class InferredSchemaBlueprint {
  const InferredSchemaBlueprint({
    required this.tableName,
    required this.fields,
  });

  final String tableName;
  final List<InferredFieldSpec> fields;
}

/// Engine-agnostic runtime schema inferencer (Memory Crystallization L0 -> L2).
/// Derives field types via a widening lattice, nullability, bounds, and index selectivity hints.
class RuntimeSchemaInferencer {
  static final RegExp _isoDatePattern =
      RegExp(r'^\d{4}-\d{2}-\d{2}([T ]\d{2}:\d{2}:\d{2})?');

  static InferredSchemaBlueprint inferFromSamples({
    required String tableName,
    required List<Map<String, dynamic>> samples,
    double indexDistinctFloor = 0.05,
    double indexDistinctCeiling = 0.90,
  }) {
    if (samples.isEmpty) {
      return InferredSchemaBlueprint(tableName: tableName, fields: const []);
    }

    final Map<String, List<dynamic>> columnValues = {};
    final Set<String> allKeys = {};
    for (final row in samples) {
      allKeys.addAll(row.keys);
    }

    for (final key in allKeys) {
      if (key == 'id' || key.startsWith('_system_')) continue;
      columnValues[key] = [for (final row in samples) row[key]];
    }

    final specs = <InferredFieldSpec>[];
    for (final entry in columnValues.entries) {
      final values = entry.value;
      final nonNulls = values.where((v) => v != null).toList();
      final bool nullable = nonNulls.length < values.length;

      if (nonNulls.isEmpty) {
        specs.add(InferredFieldSpec(name: entry.key, type: InferredFieldType.text, nullable: true));
        continue;
      }

      InferredFieldType inferred = _classifyValue(nonNulls.first);
      int observedMaxLen = 0;
      final Set<Object?> distinct = {};

      for (final val in nonNulls) {
        inferred = _widenType(inferred, _classifyValue(val));
        if (val is String) {
          observedMaxLen = math.max(observedMaxLen, val.length);
        }
        if (val is num || val is bool || (val is String && val.length <= 128)) {
          distinct.add(val);
        }
      }

      // Recommend B-Tree indexing only for selective scalar fields (skip blobs/JSON/constants).
      final double cardinalityRatio = distinct.length / nonNulls.length;
      final bool isScalarIndexable = inferred == InferredFieldType.integer ||
          inferred == InferredFieldType.datetime ||
          (inferred == InferredFieldType.text && observedMaxLen <= 128);
      final bool suggestIndex = isScalarIndexable &&
          nonNulls.length >= 20 &&
          cardinalityRatio >= indexDistinctFloor &&
          cardinalityRatio <= indexDistinctCeiling;

      final int? paddedMaxLength = inferred == InferredFieldType.text && observedMaxLen > 0
          ? math.max(64, observedMaxLen * 2)
          : null;

      specs.add(
        InferredFieldSpec(
          name: entry.key,
          type: inferred,
          nullable: nullable,
          maxLength: paddedMaxLength,
          suggestIndex: suggestIndex,
        ),
      );
    }

    return InferredSchemaBlueprint(tableName: tableName, fields: specs);
  }

  static InferredFieldType _classifyValue(dynamic v) {
    if (v is int) return InferredFieldType.integer;
    if (v is double) return InferredFieldType.doubleNum;
    if (v is bool) return InferredFieldType.boolean;
    if (v is List) return InferredFieldType.array;
    if (v is Map) return InferredFieldType.json;
    if (v is String && _isoDatePattern.hasMatch(v) && DateTime.tryParse(v) != null) {
      return InferredFieldType.datetime;
    }
    return InferredFieldType.text;
  }

  /// Type widening lattice: int -> double -> text; datetime -> text; heterogeneous -> text.
  static InferredFieldType _widenType(InferredFieldType a, InferredFieldType b) {
    if (a == b) return a;
    if ((a == InferredFieldType.integer && b == InferredFieldType.doubleNum) ||
        (a == InferredFieldType.doubleNum && b == InferredFieldType.integer)) {
      return InferredFieldType.doubleNum;
    }
    if ((a == InferredFieldType.datetime && b == InferredFieldType.text) ||
        (a == InferredFieldType.text && b == InferredFieldType.datetime)) {
      return InferredFieldType.text;
    }
    if ((a == InferredFieldType.json || a == InferredFieldType.array) &&
        (b == InferredFieldType.json || b == InferredFieldType.array)) {
      return InferredFieldType.json;
    }
    return InferredFieldType.text;
  }
}

3.2 Materializing Blueprints into Runtime Tables

Once synthesized, InferredSchemaBlueprint maps cleanly onto live runtime schemas. Two structural invariants apply in multi-tenant or multi-agent environments: separate schema existence probes (tableExists) from row presence, and remember that non-global table schemas (isGlobal: false) are shared across all isolated Spaces while row data stays isolated per Space. Adopting PrimaryKeyType.timestampBased ensures distributed edge and cloud nodes generate collision-resistant keys without centralized ID coordination:

import 'dart:developer' as dev;
import 'package:tostore/tostore.dart';

DataType _mapDataType(InferredFieldType type) {
  switch (type) {
    case InferredFieldType.integer:
      return DataType.integer;
    case InferredFieldType.doubleNum:
      return DataType.double;
    case InferredFieldType.boolean:
      return DataType.boolean;
    case InferredFieldType.datetime:
      return DataType.datetime;
    case InferredFieldType.array:
      return DataType.array;
    case InferredFieldType.json:
      return DataType.json;
    case InferredFieldType.text:
      return DataType.text;
  }
}

/// Materializes an engine-agnostic blueprint into a live runtime table and verifies persistence.
Future<bool> materializeInferredTable(
  ToStore db,
  InferredSchemaBlueprint blueprint,
) async {
  if (await db.tableExists(blueprint.tableName)) return true;

  final fields = [
    for (final f in blueprint.fields)
      FieldSchema(
        name: f.name,
        type: _mapDataType(f.type),
        nullable: f.nullable,
        maxLength: f.maxLength,
        createIndex: f.suggestIndex,
      ),
    // Append monotonic ingest timestamp for downstream retention governance
    if (!blueprint.fields.any((f) => f.name == 'created_at'))
      const FieldSchema(
        name: 'created_at',
        type: DataType.datetime,
        nullable: false,
        defaultValueType: DefaultValueType.currentTimestamp,
        createIndex: true,
      ),
  ];

  final schema = TableSchema(
    name: blueprint.tableName,
    primaryKeyConfig: const PrimaryKeyConfig(
      name: 'id',
      type: PrimaryKeyType.timestampBased,
    ),
    fields: fields,
  );

  final DbResult res = await db.createTable(schema);
  if (res.hasErrors) {
    dev.log('Dynamic table rejected [${res.firstType.codeKey}]: ${res.message}');
    return false;
  }
  return (await db.getTableSchema(blueprint.tableName)) != null;
}

---

4. Practice II: Zero-Downtime Evolution for Schemas and Encryption Keys

4.1 The Google F1 Protocol and Schema Compatibility Standards

Changing a live table's structure without halting reads and writes is fundamentally a concurrency safety problem—specifically, preventing version skew between concurrent code paths from corrupting stored records.

The landmark Google F1 paper (Online, Asynchronous Schema Change in F1, VLDB) proved that concurrent database nodes cannot safely jump directly from schema V1 to V2. When old and new writers coexist during a transition window, an uncoordinated leap allows an outdated path to drop or overlook an index entry written by a newer path, creating orphaned records or corrupt indexes. Safe online evolution must enforce bounded version skew across deterministic intermediate transitions.

Paired with serialization industry standards (such as Avro and Protobuf Backward, Forward, and Full Compatibility rules), every online schema change should follow the Expand → Backfill → Contract lifecycle:

flowchart LR
  Diff["Schema Diff & Compatibility Check"] --> Expand["Phase 1: Expand (Atomic Logical Cutover)"]
  Expand --> DualRead["Read-Time Schema Adapter (Old & New Layouts)"]
  Expand --> NewWrite["New Writes Follow Target Schema Immediately"]
  DualRead --> Backfill["Phase 2: Time-Sliced Watermark Backfill"]
  NewWrite --> Backfill
  Backfill --> Contract["Phase 3: Contract (Prune Obsolete Layout & Activate Index)"]

• Expand: New non-nullable columns must supply a default value; column/table renames preserve entity identity via immutable logical IDs (tableId/fieldId); destructive lossy casts (e.g., text -> integer) are blocked.
• Backfill: Time-sliced background workers bring historical rows and shadow indexes up to date using cost-stratified write modes—metadata-only (none), shadow index builds (indexOnly), row re-encoding (tableOnly), or full table-and-index refactoring (tableAndIndex).
• Contract: Once historical rows converge, new index views activate and obsolete layouts are pruned. Critically, supervisory logic must never treat a newly requested index as active for query routing until its asynchronous migration task reports isCompleted; otherwise, normal scan latency during index construction will be misdiagnosed as a performance regression.

4.2 Idempotent Evolution Coordinator & Edge Storage Compliance

For client-side Flutter deployments, OWASP MASVS and mobile OS guidelines mandate storing persistent databases in an OS-managed application documents directory (never in temporary cache folders that the OS may silently evict), paired with an 8 ms UI slice budget and explicit startup progress reporting. Cloud services operate with 50 ms slices and drive evolution idempotently at runtime:

import 'package:path/path.dart' as p;
import 'package:path_provider/path_provider.dart';
import 'package:tostore/tostore.dart';

/// Edge/Flutter client bootstrap: binds OS-managed persistent storage and 8ms UI slice budget.
Future<ToStore> openEdgeAutonomousStore(
  List<TableSchema> declarativeSchemas, {
  required void Function(double progress, DbStartupStage stage) onProgress,
}) async {
  final root = p.join(
    (await getApplicationDocumentsDirectory()).path,
    'autonomous_runtime',
  );
  return ToStore.open(
    dbPath: root,
    schemas: declarativeSchemas,
    config: DataStoreConfig(
      yieldDurationMs: 8,
      enableJournal: true,
    ),
    onStartupProgress: onProgress,
  );
}

class SchemaEvolutionCoordinator {
  SchemaEvolutionCoordinator(this.db);
  final ToStore db;

  /// Idempotent runtime evolution driven by inferred blueprint additions.
  Future<SchemaUpdateResult?> applyInferredDelta(InferredSchemaBlueprint target) async {
    final TableSchema? live = await db.getTableSchema(target.tableName);
    if (live == null) return null;

    final existingNames = {for (final f in live.fields) f.name};
    final newFields = target.fields.where((f) => !existingNames.contains(f.name)).toList();
    if (newFields.isEmpty) return null;

    var builder = db.updateSchema(target.tableName);
    for (final f in newFields) {
      // Forward/backward compatibility guard: new runtime columns default to nullable
      // unless an explicit defaultValue is supplied.
      builder = builder.addField(
        f.name,
        type: _mapDataType(f.type),
        nullable: true,
        maxLength: f.maxLength,
      );
      if (f.suggestIndex) {
        builder = builder.addIndex(fields: [f.name]);
      }
    }

    return builder.allowAfterDataMigration();
  }

  /// Online topology swap: promote an existing unique non-null key to primary key.
  /// Must NOT combine with setPrimaryKeyConfig in the same chain.
  Future<SchemaUpdateResult> promoteNaturalKeyOnline({
    required String tableName,
    required String uniqueNonNullField,
    required String targetPkName,
  }) {
    return db.updateSchema(tableName).promoteFieldToPrimaryKey(
          sourceFieldName: uniqueNonNullField,
          targetPrimaryKeyName: targetPkName,
        );
  }

  Future<void> observeMigration(
    SchemaUpdateResult cutover, {
    required void Function(List<ResultStatus> errors) onErrors,
    required void Function(String taskId) onCompleted,
  }) async {
    final String? taskId = cutover.taskId;
    if (cutover.hasValidationErrors || taskId == null) return;

    while (true) {
      final MigrationStatus? status = await db.queryMigrationTaskStatus(taskId);
      if (status == null) return;
      if (status.hasErrors) {
        onErrors(status.errors);
        return;
      }
      if (status.isCompleted) {
        onCompleted(taskId);
        return;
      }
      await Future<void>.delayed(const Duration(seconds: 2));
    }
  }
}

Engineering Boundaries: Keep getVersion() / setVersion() strictly for application protocol bookkeeping—never as migration triggers. While promoteFieldToPrimaryKey runs in the background, concurrent allowLargeScaleOperation calls on the same table are blocked by safety guards. When promoting a field declaratively via open(schemas:), configure the target PrimaryKeyConfig with PrimaryKeyType.none and bind fromFieldId to the source column's fieldId.

4.3 Online Key Rotation Aligned with NIST SP 800-57 and RFC 8439

In long-running secure systems, NIST SP 800-57 key-management guidelines call for separating cryptoperiods across a two-tier envelope hierarchy:

• KEK (Key Encryption Key / encryptionKey): Wraps internal key envelopes and rotates frequently on compliance schedules (e.g., every 30–90 days). Root secrets should live in hardware-backed enclaves (iOS Secure Enclave / Keychain, Android StrongBox / Keystore per OWASP MASVS) or cloud KMS via RFC 5869 HKDF, never hardcoded in source or container images.
• DEK (Data Encryption Key / encodingKey): Encrypts table pages, index nodes, and WAL payloads across a longer cryptoperiod. When a DEK must be replaced, storage pages track Key-ID version tags so reads tolerate prior tags while new writes use the active key and background workers lazily re-encrypt historical pages.
• Algorithm Selection on Edge Hardware: Between RFC 8439 (ChaCha20-Poly1305 AEAD) and NIST SP 800-38D (AES-256-GCM), pure-software Dart paths or edge ARM chips lacking ARMv8 Crypto Extensions generally favor ChaCha20-Poly1305 for its constant-time execution (mitigating timing side-channels) and predictable software throughput.

/// Rotates KEK online (O(1) envelope re-wrap, zero table payload rewrite).
/// Invariant: Must NOT execute while a background DEK (encodingKey) migration is active.
Future<bool> rotateMasterKeySafely({
  required ToStore db,
  String? currentKek,
  required String nextKek,
  required Future<void> Function(String newKey) commitToHardwareVault,
}) async {
  final DbResult res = await db.rotateEncryptionKey(
    oldKey: currentKek,
    newKey: nextKek,
  );
  if (res.hasErrors) return false;

  await commitToHardwareVault(nextKek);
  return true;
}

---

5. Practice III: Anti-Entropy Cleanup and Self-Tuning for Long-Term Stability

5.1 Three Quantitative Metrics of Storage Degradation

Multi-year deployments degrade nonlinearly along three measurable storage-engine axes:

1. Space Amplification (SAF) and Tombstone Amplification: Frequent row-level DELETE calls accumulate logical tombstone markers that range scans must repeatedly skip. In traditional SQLite-backed edge apps, two classic pitfalls silently exhaust disk space: omitting PRAGMA auto_vacuum = INCREMENTAL (causing freed pages never to return to the OS) and long-lived concurrent readers starving checkpoints, causing unbounded WAL growth unless periodically truncated via PRAGMA wal_checkpoint(TRUNCATE).
2. Write Amplification Factor (WAF) and Index Bloat: Secondary indexes accumulated over successive evolutions multiply write I/O; stale indexes with low query selectivity (matched_rows / scanned_rows) hurt write throughput for negligible read gain.
3. Google SRE P99 Tail-Latency Spikes: Scheduled mass DELETE jobs saturate disk bandwidth and lock queues, breaching latency SLOs.

A resilient anti-entropy architecture pushes retention boundaries straight into the table schema, throttling evictions per cycle and pruning low-value indexes based on live footprint telemetry:

const toolTraceSchema = TableSchema(
  name: 'tool_trace',
  primaryKeyConfig: PrimaryKeyConfig(type: PrimaryKeyType.timestampBased),
  fields: [
    FieldSchema(
      name: 'created_at',
      type: DataType.datetime,
      nullable: false,
      defaultValueType: DefaultValueType.currentTimestamp,
      createIndex: true,
    ),
    FieldSchema(
      name: 'kind',
      type: DataType.text,
      nullable: false,
      defaultValue: 'trace',
      createIndex: true,
    ),
    FieldSchema(name: 'payload', type: DataType.json),
    FieldSchema(
      name: 'last_active_at',
      type: DataType.datetime,
      nullable: false,
      defaultValueType: DefaultValueType.currentTimestamp,
    ),
  ],
  ttlConfig: TableTtlConfig(
    ttlMs: 14 * 24 * 60 * 60 * 1000,
    sourceField: 'created_at',
  ),
  capacityConfig: CapacityConfig(
    maxEntries: 200000,
    maxBytes: 256 * 1024 * 1024,
    policy: CapacityEvictionPolicy.lru,
    sourceField: 'last_active_at',
    evictRatio: 0.30, // P99 latency smoothing: evict ~30% per cycle instead of full stalls
  ),
);

• TTL vs. FIFO/LRU Constraints: TTL sourceField must be a non-null datetime with DefaultValueType.currentTimestamp (background scan intervals are floored at 60000 ms to avoid idle churn). Under fifo with monotonic primary keys (anything other than PrimaryKeyType.none), eviction follows ascending primary keys and sourceField must be omitted; under lru, applications must update last_active_at on logical hits. Keep memory cache budgets (cacheMemoryBudgetMB) strictly separate from table storage caps (CapacityConfig).
• Namespace-Level Reclamation and Out-of-Tx Purges: Isolate multi-step scratch tasks inside temporary Spaces (switchSpace) and drop the entire namespace via deleteSpace upon completion to eliminate B-Tree fragmentation in near O(1) time. For semantic bulk cleanups that cannot be time- or LRU-inferred, execute .allowLargeScaleOperation() strictly outside transaction blocks (and include .allowUpdateAll() / .allowDeleteAll() when operating unconditionally).

---

6. Practice IV: AI-Native Automated O&M and the ResultStatus Diagnostic Loop

6.1 Three Failure Modes of Autonomous Agents

Delegating evolution and tuning authority to unattended daemons or LLM-driven agents exposes three systemic failure modes:

1. Retry Amplification (Retry Storms): As highlighted in Google SRE practices, treating non-retryable contract/schema defects like transient lock timeouts causes an agent to hammer the engine at millisecond intervals with the exact same invalid payload—turning a localized validation rejection into a CPU and logging meltdown.
2. Schema Hallucination: LLMs drawing on stale training memory frequently synthesize illegal combinations (e.g., setting sourceField on PK-ordered fifo eviction, or embedding large-scale deletes inside transactions).
3. Unbounded Self-Mutation Cascades: Without rate quotas, oscillating feedback loops can trigger back-to-back updateSchema refactors that saturate storage I/O.

6.2 Machine-Readable Diagnostics and ResultStatus as a First-Class O&M Indicator

The decisive boundary between brittle log scraping and truly autonomous, human-free O&M is whether the persistence engine exposes a unified, machine-readable structured status code protocol (conceptually aligned with OpenTelemetry error domains and RFC 9457 / gRPC structured status payloads). In our reference architecture, ResultStatus serves as this foundational diagnostic indicator:

flowchart TD
  Act["Autonomous Evolution or O&M Action"] --> Channel{"Unified Result Channel"}
  Channel -- "Success (hasErrors == false)" --> Observe["OODA Observe: Telemetry & Metrics"]
  Channel -- "Result Errors or DbException" --> Orient["OODA Orient: ResultStatus O(1) Classifier"]

  Orient --> Crit{"isCriticalError (Disk Full / OOM)"}
  Orient --> Dev{"isDeveloperError (Contract / Schema Guard)"}
  Orient --> Biz{"isBusinessError (Constraint Violation)"}
  Orient --> Sys{"isSystemError (Transient Lock / Conflict)"}

  Crit -- "Alert & Graceful Degradation" --> Reclaim["Reclaim Spaces, Tighten Bounds & Page Ops"]
  Dev -- "Extract tableName / field / wrongValue" --> Breaker["Trip Breaker Immediately; Self-Patch via Grounded Spec"]
  Biz -- "Map via batch index & conflictingKeys" --> Sanitize["Quarantine Dirty Row or Upsert Fallback"]
  Sys -- "Tx Conflict / Lock Timeout" --> Backoff["Exponential Backoff with Full Jitter"]
  Observe -- "SAF / WAF / Memory Threshold" --> Act

A structured status system like ResultStatus transforms opaque error strings into deterministic OODA state-machine transitions:

• Unified Dual-Channel Semantics and O(1) Routing: Both everyday result objects (DbResult, SchemaUpdateResult, TransactionResult) and fatal DbException throws carry the exact same ResultStatus contract—combining hierarchical numeric codes (code), stable semantic keys (codeKey), and in-memory O(1) domain predicates (isBusinessError, isDeveloperError, isSystemError, isCriticalError). Supervisors route cleanly without fragile regex parsing.
• 1:1 Batch Attribution and Structured Coordinates: Batch writes align status.index 1:1 with the input list, allowing partial-error pipelines to quarantine single malformed rows instead of discarding entire exploration batches. On schema rejections, SchemaValidationStatus supplies exact tableName, field, and wrongValue coordinates—allowing the controller to trip its circuit breaker immediately, halt blind retries, and feed precise repair coordinates back to the agent.

6.3 Evolution Rate Limiters, Build-Time Contract Grounding (llms-full.txt), and Policy Gates

To keep long-running autonomous loops safe in production, surround the diagnostic core with three operational guardrails:

1. Evolution Rate Limiter & Sandbox Dry-Runs: Cap the maximum number of structural mutations permitted per sliding window, and replay candidate migrations in an isolated temporary instance or sandbox Space before mutating production tables.
2. Build-Time Spec Grounding (llms-full.txt): When agents dynamically propose new TableSchema definitions or tuning configs, ground their prompts against self-contained machine specs (such as llms-full.txt, raw text raw/llms-full.txt, and result_status_specification.md). Production Security Rule: Never fetch external GitHub URLs live at runtime inside 7×24 services (avoiding network stalls, upstream version drift, and supply-chain injection); instead, SHA-256 pin and bundle the version-matched spec into the agent's local read-only context at build time.
3. Human Approval Policy Gates & Checkpoints: Keep irreversible actions (dropTable, deleteDatabase, destructive restore(..., cleanupBeforeRestore: true)), primary-key strategy changes on populated tables, and concurrent DEK/KEK rotations behind elevated human-approval gates. Before complex multi-step migrations, capture a compressed snapshot via db.backup(compress: true, scope: BackupScope.currentSpaceWithGlobal) for deterministic restore rollback.

6.4 Complete End-to-End Autonomous O&M Reference Implementation

The reference coordinator below ties together our pure-Dart EvolutionRateLimiter, RuntimeSchemaInferencer, OODA ResultStatus classifier, LogLevel.critical alerting hook, dynamic index pruning, and online KEK rotation:

import 'dart:async';
import 'dart:math' as math;
import 'package:tostore/tostore.dart';

typedef AlertSink = void Function(String message, String? codeKey);
typedef TelemetrySink = void Function(Map<String, dynamic> mem, Map<String, dynamic> space);

/// Engine-agnostic sliding-window rate limiter preventing runaway AI DDL mutations.
class EvolutionRateLimiter {
  EvolutionRateLimiter({
    required this.maxOpsPerWindow,
    required this.window,
  });

  final int maxOpsPerWindow;
  final Duration window;
  final List<DateTime> _timestamps = [];

  bool tryAcquire() {
    final now = DateTime.now();
    _timestamps.removeWhere((t) => now.difference(t) > window);
    if (_timestamps.length >= maxOpsPerWindow) return false;
    _timestamps.add(now);
    return true;
  }
}

/// Global log probe installed BEFORE ToStore.open; routes critical exhaustion to human ops.
void installOpsLogProbe({
  required AlertSink onCriticalInterventionNeeded,
  required void Function(LogLevel level, String message, String? codeKey) onLog,
}) {
  ToStore.setLogConfig(
    enableLog: true,
    logLevel: LogLevel.warn,
    logLabel: 'autonomous_ops',
    onLog: (LogRecord log) {
      // LogLevel.critical indicates resource exhaustion (disk full, OOM, fatal IO) needing intervention.
      if (log.level == LogLevel.critical || log.status?.isCriticalError == true) {
        onCriticalInterventionNeeded(log.message, log.status?.codeKey);
        return;
      }
      onLog(log.level, log.message, log.status?.codeKey);
    },
  );
}

class AutonomousRuntimeCoordinator {
  AutonomousRuntimeCoordinator({
    required this.db,
    required this.evolutionCoordinator,
    required this.rateLimiter,
    required this.onAlert,
    required this.onTelemetry,
    required this.commitRotatedKek,
  });

  final ToStore db;
  final SchemaEvolutionCoordinator evolutionCoordinator;
  final EvolutionRateLimiter rateLimiter;
  final AlertSink onAlert;
  final TelemetrySink onTelemetry;
  final Future<void> Function(String newKey) commitRotatedKek;

  bool _evolutionCircuitOpen = false;
  final math.Random _rng = math.Random();

  /// Request Loop: writes explored samples with batch 1:1 ResultStatus quarantine.
  Future<void> ingestExploredMemories(
    String tableName,
    List<Map<String, dynamic>> batch,
  ) async {
    if (batch.isEmpty) return;
    if (!await db.tableExists(tableName)) {
      final blueprint = RuntimeSchemaInferencer.inferFromSamples(
        tableName: tableName,
        samples: batch,
      );
      if (!await materializeInferredTable(db, blueprint)) return;
    }

    final DbResult res = await db.batchInsert(tableName, batch, allowPartialErrors: true);
    if (!res.hasErrors) return;

    for (final ResultStatus s in res.statuses) {
      if (s.type == ResultType.success) continue;
      if (s.isDeveloperError) {
        // Trip circuit breaker immediately to prevent SRE retry amplification.
        _evolutionCircuitOpen = true;
        onAlert('Contract mismatch at batch[${s.index}]: ${s.message}', s.codeKey);
        return;
      }
      if (s is ConstraintStatus) {
        // Business constraint conflict: isolate dirty row via s.index rather than halting batch.
        onAlert(
          'Quarantined batch[${s.index}] on ${s.tableName}.${s.fields}: keys=${s.conflictingKeys}',
          s.codeKey,
        );
      }
    }
  }

  /// Supervision Loop: executes one OODA cycle (inferred evolution, tuning, cleanup, KEK rotation).
  Future<void> runSupervisionCycle({
    required String targetTable,
    required List<Map<String, dynamic>> recentSamples,
    required bool kekRotationDue,
    required String? currentKek,
    required String nextKek,
    required int softSpaceBudgetBytes,
    required int maxTraceRowsBudget,
  }) async {
    // 1. Inferred Runtime Evolution (protected by Circuit Breaker & Rate Limiter)
    if (!_evolutionCircuitOpen && recentSamples.isNotEmpty && rateLimiter.tryAcquire()) {
      final blueprint = RuntimeSchemaInferencer.inferFromSamples(
        tableName: targetTable,
        samples: recentSamples,
      );
      final cutover = await evolutionCoordinator.applyInferredDelta(blueprint);
      if (cutover != null) {
        if (cutover.hasValidationErrors) {
          _tripBreakerWithCoordinates(cutover.validationStatuses);
        } else if (cutover.taskId != null) {
          unawaited(
            evolutionCoordinator.observeMigration(
              cutover,
              onErrors: _tripBreakerWithCoordinates,
              onCompleted: (_) {},
            ),
          );
        }
      }
    }

    // 2. OODA Observe & Self-Tuning: monitor SAF / WAF / low-memory markers
    final MemoryInfo mem = await db.status.memory();
    final SpaceInfo space = await db.status.space(useCache: false);
    final TableInfo? traceInfo = await db.status.table('tool_trace');

    if (mem.isLowMemoryMode || space.totalSizeBytes > softSpaceBudgetBytes) {
      await _reclaimEntropyAndTuneStructure(traceInfo, maxTraceRowsBudget);
    }

    // 3. Secrecy Contract: online KEK rotation with full-jitter transient retry
    if (kekRotationDue) {
      await _retryOnTransient(() async {
        final ok = await rotateMasterKeySafely(
          db: db,
          currentKek: currentKek,
          nextKek: nextKek,
          commitToHardwareVault: commitRotatedKek,
        );
        if (!ok) {
          onAlert('KEK rotation deferred (mismatch or active DEK migration)', null);
        }
        return ok;
      });
    }

    onTelemetry(mem.toJson(), space.toJson());
  }

  Future<void> _reclaimEntropyAndTuneStructure(
    TableInfo? traceInfo,
    int maxTraceRowsBudget,
  ) async {
    // Drop expired task sandboxes cleanly as whole namespaces
    for (final String spaceName in await db.listSpaces()) {
      if (spaceName.startsWith('tmp_task_') && spaceName != db.currentSpaceName) {
        await db.deleteSpace(spaceName);
      }
    }

    if (traceInfo == null) return;

    // Dynamically tighten capacity threshold when record footprint grows
    final int? currentCap = traceInfo.schema.capacityConfig?.maxEntries;
    if (traceInfo.totalRecordCount > maxTraceRowsBudget && currentCap != maxTraceRowsBudget) {
      await db.updateSchema('tool_trace').setCapacityConfig(
            CapacityConfig(
              maxEntries: maxTraceRowsBudget,
              policy: CapacityEvictionPolicy.lru,
              sourceField: 'last_active_at',
              evictRatio: 0.35,
            ),
          );
    }

    // Prune bloated non-unique secondary index when index SAF exceeds table data
    if (traceInfo.totalIndexDataSizeBytes > traceInfo.totalTableDataSizeBytes) {
      final bool hasAux = traceInfo.schema.indexes.any(
        (idx) => idx.fields.length == 1 && idx.fields.first == 'aux_tag' && !idx.unique,
      );
      if (hasAux) {
        await db.updateSchema('tool_trace').removeIndex(fields: ['aux_tag']);
      }
    }
  }

  /// Large-scale semantic cleanup strictly outside transactions.
  Future<DbResult> purgeScratchDrafts() {
    return db.delete('tool_trace').where('kind', '=', 'scratch').allowLargeScaleOperation();
  }

  void _tripBreakerWithCoordinates(List<ResultStatus> statuses) {
    for (final ResultStatus s in statuses) {
      if (s.isCriticalError || s.isDeveloperError) {
        _evolutionCircuitOpen = true;
      }
      if (s is SchemaValidationStatus) {
        onAlert(
          'Circuit open -> fix ${s.tableName}.${s.field} (wrongValue=${s.wrongValue}): ${s.message}',
          s.codeKey,
        );
      } else {
        onAlert('Diagnostic -> ${s.message}', s.codeKey);
      }
    }
  }

  /// Google SRE exponential backoff with full jitter for transient lock/system contention.
  Future<void> _retryOnTransient(Future<bool> Function() action, {int maxAttempts = 3}) async {
    for (int attempt = 0; attempt < maxAttempts; attempt++) {
      if (await action()) return;
      final int capMs = 200 * (1 << attempt);
      await Future<void>.delayed(Duration(milliseconds: _rng.nextInt(capMs + 1)));
    }
  }
}

---

7. Conclusion

The core challenge of data evolution in long-lived intelligent systems is not which query syntax looks cleverest, but whether the architecture can weave schema growth, key rotation, data metabolism, and self-healing into normal runtime operation without ever dropping service.

By combining Google F1's asynchronous transition principles, Avro/Protobuf compatibility discipline, Dart event-loop and Flutter frame-budget time-slicing, NIST SP 800-57 dual-layer envelope rotation, quantitative WAF/SAF self-tuning, and an OODA circuit-breaker loop anchored on machine-readable ResultStatus contracts and build-time pinned specifications, Flutter and Dart engineering teams can build data foundations capable of running, evolving, and healing autonomously across cloud and edge environments for years on end.