DEV Community

Cover image for ACAI — Chapter 40: File & Object Storage Architecture
Black Shadow Team ©
Black Shadow Team ©

Posted on

ACAI — Chapter 40: File & Object Storage Architecture

#ai

40.1 Introduction

Chapters 34–39 established the database, authentication, authorization, and API foundations of the ACAI platform.

The next major subsystem is file storage.

An AI platform may process many types of user-owned data:

Images
Videos
Audio
Documents
PDF files
Text files
Generated media
Processed outputs
Temporary processing artifacts
Enter fullscreen mode Exit fullscreen mode

These objects should not normally be stored directly inside PostgreSQL.

Instead, the system should separate:

Metadata
   ↓
PostgreSQL

Large binary objects
   ↓
Object Storage
Enter fullscreen mode Exit fullscreen mode

The resulting architecture is:

                    ACAI
                     │
              ┌──────┴──────┐
              │             │
          PostgreSQL     Object Storage
              │             │
          Metadata       Binary Data
Enter fullscreen mode Exit fullscreen mode

This separation improves scalability, performance, lifecycle management, and security.

40.2 Database vs Object Storage

PostgreSQL is appropriate for structured information such as:

fileId
ownerId
projectId
fileName
mimeType
size
status
storageKey
createdAt
Enter fullscreen mode Exit fullscreen mode

Object storage is appropriate for:

photo.jpg
video.mp4
audio.wav
document.pdf
generated-image.png
Enter fullscreen mode Exit fullscreen mode

The database therefore acts as the authoritative metadata layer.

40.3 File Architecture

The basic flow is:

User
 │
 ▼
Upload Request
 │
 ▼
Authentication
 │
 ▼
Authorization
 │
 ▼
Upload Policy
 │
 ▼
Object Storage
 │
 ▼
Metadata Record
 │
 ▼
Processing Pipeline
Enter fullscreen mode Exit fullscreen mode

For larger files, the architecture can use a direct or multipart upload pattern:

Client
  │
  ▼
API requests upload authorization
  │
  ▼
Server validates request
  │
  ▼
Temporary upload authorization
  │
  ▼
Object Storage
  │
  ▼
Upload completed
  │
  ▼
Server records/updates metadata
Enter fullscreen mode Exit fullscreen mode

The client should not receive unrestricted storage credentials.

40.4 Storage Key Design

A file should have an internal storage key rather than relying only on its original filename.

For example:

users/{userId}/projects/{projectId}/files/{fileId}/original
Enter fullscreen mode Exit fullscreen mode

A generated asset could use:

users/{userId}/projects/{projectId}/generations/{generationId}/output
Enter fullscreen mode Exit fullscreen mode

The exact format can vary, but the key should be:

  • unique;
  • predictable only where necessary;
  • independent from user-provided filenames;
  • free of secrets;
  • compatible with lifecycle policies.

40.5 Original Filename vs Storage Key

Suppose the user uploads:

my vacation photo.jpg
Enter fullscreen mode Exit fullscreen mode

The application can store:

name = "my vacation photo.jpg"
Enter fullscreen mode Exit fullscreen mode

while using an internal key such as:

users/<user>/projects/<project>/files/<file>/original
Enter fullscreen mode Exit fullscreen mode

This separation prevents user-provided filenames from becoming the primary identity of the stored object.

40.6 File Metadata

The File model introduced earlier can be extended.

Example:

model File {
  id          String     @id @default(uuid())

  name        String
  mimeType    String
  sizeBytes   BigInt

  status      FileStatus @default(UPLOADING)

  ownerId     String
  owner       User       @relation(fields: [ownerId], references: [id], onDelete: Cascade)

  projectId   String?
  project     Project?   @relation(fields: [projectId], references: [id], onDelete: SetNull)

  storageKey  String?

  checksum    String?

  metadata    Json?

  createdAt   DateTime   @default(now())
  updatedAt   DateTime   @updatedAt

  @@index([ownerId])
  @@index([projectId])
  @@index([status])
}
Enter fullscreen mode Exit fullscreen mode

Potential metadata includes:

width
height
duration
pageCount
encoding
processing status
Enter fullscreen mode Exit fullscreen mode

Only appropriate metadata should be stored.

40.7 File Lifecycle

A file should have an explicit lifecycle.

Example:

UPLOADING
    ↓
UPLOADED
    ↓
PROCESSING
    ↓
READY
Enter fullscreen mode Exit fullscreen mode

Failure path:

PROCESSING
    ↓
FAILED
Enter fullscreen mode Exit fullscreen mode

Deletion:

READY
    ↓
DELETED
Enter fullscreen mode Exit fullscreen mode

This allows the application to distinguish between:

File exists but is still processing
Enter fullscreen mode Exit fullscreen mode

and:

File processing failed
Enter fullscreen mode Exit fullscreen mode

40.8 Upload Validation

A secure upload system should validate at least:

File size
Declared MIME type
Actual file characteristics
Extension
Application-supported format
Ownership
Destination project
Upload authorization
Enter fullscreen mode Exit fullscreen mode

The client-provided filename or MIME type should not automatically be treated as authoritative.

40.9 File Size Limits

Different resources can have different limits.

For example:

Avatar
    ↓
Small limit

Document
    ↓
Medium limit

Video
    ↓
Large limit
Enter fullscreen mode Exit fullscreen mode

The exact limits should be defined by product requirements, infrastructure capacity, and abuse testing.

The server must enforce the limit rather than relying only on the UI.

40.10 MIME Type Validation

A browser may report:

image/jpeg
Enter fullscreen mode Exit fullscreen mode

but the server should not blindly trust that declaration.

Where appropriate, the processing pipeline should inspect the actual file characteristics.

The principle is:

Declared metadata
       +
Content inspection
       +
Application policy
Enter fullscreen mode Exit fullscreen mode

40.11 Extension Handling

User-controlled filenames can contain unusual or misleading extensions.

The application should not construct sensitive storage behavior directly from an untrusted filename.

Instead:

Original filename
       ↓
Stored as metadata
       ↓
Internal object key
Enter fullscreen mode Exit fullscreen mode

The storage key should be generated by the application.

40.12 File Ownership

Every private file must have an ownership boundary.

Conceptually:

File
 │
 ├── ownerId
 └── projectId
Enter fullscreen mode Exit fullscreen mode

Before access:

Current User
      ↓
Project/File
      ↓
Ownership or Membership
      ↓
Permission
Enter fullscreen mode Exit fullscreen mode

Only after successful authorization should the application provide access.

40.13 Private Storage

Private user files should generally remain inaccessible through anonymous public URLs.

The preferred pattern is:

User
 │
 ▼
Authenticated API
 │
 ▼
Authorization Check
 │
 ▼
Temporary Authorized Access
 │
 ▼
Object Storage
Enter fullscreen mode Exit fullscreen mode

This prevents possession of an arbitrary URL from automatically becoming permanent access.

40.14 Temporary Access

For private objects, the application can issue a short-lived authorized access mechanism when supported by the storage platform.

Conceptually:

Request
   ↓
Authenticate
   ↓
Authorize
   ↓
Generate temporary access
   ↓
Client retrieves object
Enter fullscreen mode Exit fullscreen mode

The lifetime should be limited to the actual use case.

40.15 Upload Workflow

A robust upload sequence is:

1. User selects file
        ↓
2. Client requests upload authorization
        ↓
3. Server authenticates user
        ↓
4. Server validates project ownership
        ↓
5. Server validates requested file policy
        ↓
6. Server creates file metadata
        ↓
7. Upload authorization issued
        ↓
8. Client uploads object
        ↓
9. Upload completion recorded
        ↓
10. Processing job created
Enter fullscreen mode Exit fullscreen mode

This creates a traceable lifecycle.

40.16 Processing Pipeline

After upload:

Object Storage
      ↓
Processing Queue
      ↓
Worker
      ↓
Validation
      ↓
Extraction
      ↓
Transformation
      ↓
Result
      ↓
Database
Enter fullscreen mode Exit fullscreen mode

For a document:

PDF
 ↓
Text Extraction
 ↓
Normalization
 ↓
Chunking
 ↓
Embedding
 ↓
Vector Index
Enter fullscreen mode Exit fullscreen mode

For an image:

Image
 ↓
Metadata Extraction
 ↓
Validation
 ↓
Optional Processing
 ↓
Generated/Processed Asset
Enter fullscreen mode Exit fullscreen mode

40.17 Malware and Unsafe File Handling

File-processing infrastructure should assume that uploaded files are untrusted.

The processing environment should therefore be isolated from privileged infrastructure as much as practical.

Important principles include:

Untrusted file
     ↓
Controlled processing environment
     ↓
Limited permissions
     ↓
No unnecessary network access
     ↓
Validated output
Enter fullscreen mode Exit fullscreen mode

The exact security controls depend on deployment architecture.

40.18 Processing Isolation

A file-processing worker should not automatically have access to:

Production database credentials
Authentication secrets
Payment credentials
Administrative APIs
Unrelated user files
Enter fullscreen mode Exit fullscreen mode

The worker should receive only the permissions required for its task.

This follows least-privilege design.

40.19 Temporary Files

Processing may require temporary local storage.

For example:

Object Storage
      ↓
Worker temporary area
      ↓
Processing
      ↓
Output Storage
      ↓
Temporary data removed
Enter fullscreen mode Exit fullscreen mode

Temporary files should not remain indefinitely.

A lifecycle policy should define:

When created
Where stored
Who can access
When deleted
Enter fullscreen mode Exit fullscreen mode

40.20 Generated Assets

AI-generated outputs should be treated as first-class resources.

For example:

Generation
 │
 ├── request metadata
 ├── model information
 ├── status
 └── output File
Enter fullscreen mode Exit fullscreen mode

A generation may create:

image
video
audio
document
Enter fullscreen mode Exit fullscreen mode

The output should have an identifiable relationship to the generation record.

40.21 File Versioning

Editing systems may need versions.

Example:

Original
   ↓
Edited Version 1
   ↓
Edited Version 2
   ↓
Edited Version 3
Enter fullscreen mode Exit fullscreen mode

A future model can represent:

FileVersion
 ├── id
 ├── fileId
 ├── versionNumber
 ├── storageKey
 ├── createdAt
 └── metadata
Enter fullscreen mode Exit fullscreen mode

This is especially useful for creative applications where users expect undo, revision history, or recovery.

40.22 Derivative Assets

One source file may generate several derivatives.

For example:

Original Image
   │
   ├── Thumbnail
   ├── Preview
   ├── Optimized Version
   └── Edited Version
Enter fullscreen mode Exit fullscreen mode

The database should maintain relationships between the source and derivative assets.

This prevents the application from losing track of how an output was created.

40.23 Content Hashing

A checksum can help detect accidental corruption or identify identical content.

Conceptually:

File
 ↓
Hash Function
 ↓
Checksum
Enter fullscreen mode Exit fullscreen mode

The checksum can be stored in metadata.

However, a checksum should not automatically be treated as a security mechanism for every purpose.

Its exact role should be clearly defined.

40.24 Storage Quotas

Users may receive storage limits.

Conceptually:

User
 │
 ├── Storage Used
 ├── Storage Limit
 └── Available Storage
Enter fullscreen mode Exit fullscreen mode

Before accepting an upload:

Requested Size
       ↓
Current Usage
       ↓
Quota Check
       ↓
Allowed / Rejected
Enter fullscreen mode Exit fullscreen mode

Quota calculations should be performed server-side.

40.25 Project-Level Quotas

The same principle can apply to projects.

Account Quota
      │
      ├── Project A
      ├── Project B
      └── Project C
Enter fullscreen mode Exit fullscreen mode

A project can optionally have its own resource limit.

This is useful for collaborative or organizational environments.

40.26 File Deletion

Deleting a file should be a controlled workflow.

Delete Request
      ↓
Authenticate
      ↓
Authorize
      ↓
Mark Deleted
      ↓
Storage Cleanup
      ↓
Audit Event
Enter fullscreen mode Exit fullscreen mode

For important data, a delayed deletion strategy may be preferable:

ACTIVE
  ↓
MARKED_FOR_DELETION
  ↓
RETENTION PERIOD
  ↓
PERMANENT DELETION
Enter fullscreen mode Exit fullscreen mode

This can provide a recovery window.

40.27 Orphaned Object Detection

Distributed systems can create inconsistencies.

For example:

Database record exists
but object is missing
Enter fullscreen mode Exit fullscreen mode

or:

Object exists
but database record was never completed
Enter fullscreen mode Exit fullscreen mode

A reconciliation process can periodically detect such cases.

Conceptually:

Database
   ↕
Object Storage
   ↕
Reconciliation Worker
Enter fullscreen mode Exit fullscreen mode

Possible states:

CONSISTENT
MISSING_OBJECT
MISSING_METADATA
PROCESSING_STUCK
Enter fullscreen mode Exit fullscreen mode

40.28 Storage Lifecycle Policies

Object storage should use lifecycle rules where appropriate.

Example:

Temporary upload
      ↓
Delete after defined retention period
Enter fullscreen mode Exit fullscreen mode

Processing artifacts:

Temporary artifact
      ↓
Automatic cleanup
Enter fullscreen mode Exit fullscreen mode

Old versions:

Inactive version
      ↓
Long-term retention or deletion
Enter fullscreen mode Exit fullscreen mode

Lifecycle policies reduce unnecessary storage cost and operational clutter.

40.29 Download Authorization

A download endpoint should not simply accept:

/api/files/{id}/download
Enter fullscreen mode Exit fullscreen mode

and return the object.

Instead:

Request
  ↓
Authenticate
  ↓
Find File
  ↓
Check Owner/Member
  ↓
Check Permission
  ↓
Generate Authorized Access
  ↓
Return
Enter fullscreen mode Exit fullscreen mode

This preserves the same authorization principles established in Chapter 38.

40.30 File API

A possible API structure is:

POST   /api/files
GET    /api/files
GET    /api/files/{id}
DELETE /api/files/{id}
POST   /api/files/{id}/complete
POST   /api/files/{id}/process
Enter fullscreen mode Exit fullscreen mode

For larger uploads, the API may separate:

Create upload
Complete upload
Process file
Enter fullscreen mode Exit fullscreen mode

from ordinary file metadata operations.

40.31 File API Security

Every private file endpoint should consider:

Authentication
Authorization
Ownership
Project membership
File status
Quota
Rate limits
Input validation
Audit logging
Enter fullscreen mode Exit fullscreen mode

This is especially important because file identifiers are often easy for clients to obtain.

Knowing an identifier must not automatically grant access.

40.32 AI Document Pipeline

The file architecture becomes especially important when documents are used by AI.

The secure pipeline is:

Upload
  ↓
Validation
  ↓
Storage
  ↓
Text Extraction
  ↓
Chunking
  ↓
Embedding
  ↓
Vector Storage
  ↓
Authorized Retrieval
  ↓
AI Context
Enter fullscreen mode Exit fullscreen mode

The retrieval stage must respect the same ownership and permission boundaries as the original document.

40.33 Document Authorization

Suppose:

Project A
 ├── Document 1
 └── Document 2

Project B
 └── Document 3
Enter fullscreen mode Exit fullscreen mode

If a user has access only to Project A, an AI retrieval query must not accidentally return Document 3.

Therefore:

Semantic relevance
        +
Authorization filter
Enter fullscreen mode Exit fullscreen mode

must both be satisfied.

A relevant document is not automatically an authorized document.

40.34 Retrieval Security Principle

The correct model is:

Candidate Documents
        ↓
Authorization Filter
        ↓
Eligible Documents
        ↓
Relevance Ranking
        ↓
AI Context
Enter fullscreen mode Exit fullscreen mode

An alternative implementation can apply authorization before ranking.

The critical requirement is that unauthorized documents never reach the model context.

40.35 Media Processing

For video and audio, processing can be asynchronous.

Example:

Upload
  ↓
Metadata Extraction
  ↓
Queue
  ↓
Worker
  ↓
Transcoding / Analysis
  ↓
Output
Enter fullscreen mode Exit fullscreen mode

The user interface can display:

Uploading
Processing
Completed
Failed
Enter fullscreen mode Exit fullscreen mode

rather than waiting for a long HTTP request.

40.36 Progress Tracking

A processing job can expose:

0%
25%
50%
75%
100%
Enter fullscreen mode Exit fullscreen mode

However, progress should be reported only when it can be estimated meaningfully.

A system should not create false precision simply to make the interface look active.

40.37 Storage and Privacy

Files can contain highly sensitive information.

Therefore:

Access Control
Encryption
Retention
Deletion
Audit
Enter fullscreen mode Exit fullscreen mode

should be considered together.

The system should collect and retain only what is required for its intended functionality.

40.38 Backup Strategy

Object storage and database backups should be considered separately.

PostgreSQL Backup
        +
Object Storage Protection
Enter fullscreen mode Exit fullscreen mode

A database backup without the corresponding objects may be incomplete.

Likewise, objects without their metadata may be difficult to interpret.

A recovery plan should define how both layers are restored together.

40.39 Disaster Recovery

A recovery workflow may look like:

Incident
   ↓
Infrastructure Recovery
   ↓
Database Restore
   ↓
Object Storage Verification
   ↓
Metadata/Object Reconciliation
   ↓
Application Recovery
   ↓
Integrity Verification
Enter fullscreen mode Exit fullscreen mode

Recovery procedures should be tested rather than merely documented.

40.40 File Security Checklist 

[ ] Files are treated as untrusted input
[ ] File ownership is recorded
[ ] Project access is verified
[ ] Size limits are enforced
[ ] File types are validated
[ ] User filenames are not used as object identity
[ ] Storage keys are generated server-side
[ ] Private objects are not anonymously accessible
[ ] Temporary access is time-limited
[ ] Processing workers use least privilege
[ ] Temporary files are cleaned up
[ ] Deleted objects follow a defined lifecycle
[ ] Orphaned objects can be detected
[ ] Storage quotas are enforced
[ ] File operations are auditable
[ ] Database and storage recovery are coordinated
Enter fullscreen mode Exit fullscreen mode

40.41 Complete Storage Architecture

The complete design is:

                         USER
                           │
                           ▼
                    AUTHENTICATION
                           │
                           ▼
                    AUTHORIZATION
                           │
                           ▼
                       FILE API
                           │
               ┌───────────┴───────────┐
               │                       │
          File Metadata           Upload Policy
               │                       │
               ▼                       ▼
          PostgreSQL             Object Storage
               │                       │
               └───────────┬───────────┘
                           │
                           ▼
                    Processing Queue
                           │
                           ▼
                       Worker
                           │
              ┌────────────┴────────────┐
              │                         │
        Document Pipeline          Media Pipeline
              │                         │
              ▼                         ▼
       Text/Embeddings             Derived Assets
              │                         │
              └────────────┬────────────┘
                           │
                           ▼
                     AI Services
Enter fullscreen mode Exit fullscreen mode

40.42 Architectural Significance

The file system is not merely a place to put uploaded files.

It is a security and data-governance subsystem.

It determines:

Who owns the data?
Who can access it?
How long is it retained?
How is it processed?
Where is it stored?
What happens when it is deleted?
Can it be recovered?
Can AI systems retrieve it?
Can processing workers access it?
Enter fullscreen mode Exit fullscreen mode

These questions must be answered before the platform handles significant user data.

40.43 Conclusion

Chapter 40 established the object-storage architecture.

The system now has a clear separation:

Structured metadata
        ↓
PostgreSQL

Large binary objects
        ↓
Object Storage
Enter fullscreen mode Exit fullscreen mode

and a controlled lifecycle:

Upload
 ↓
Validate
 ↓
Store
 ↓
Process
 ↓
Authorize
 ↓
Retrieve
 ↓
Retain
 ↓
Delete
Enter fullscreen mode Exit fullscreen mode

The most important principle is:

A file being stored does not automatically make it accessible.

Every access must pass through identity, authorization, ownership, and policy boundaries.

This architecture prepares the platform for the next major subsystem: document ingestion, text extraction, chunking, embeddings, vector storage, retrieval, and Retrieval-Augmented Generation (RAG).

END OF CHAPTER 40

Top comments (0)