DEV Community: Ray Forever

Building a TypeScript SDK for Secure Direct Uploads to AWS S3 and ImageKit

Ray Forever — Mon, 03 Aug 2026 08:30:30 +0000

Introduction

Handling file uploads in web applications often involves a trade-off between security, performance, and developer experience. A new open-source TypeScript SDK, Upload SDK, addresses these concerns by enabling secure direct uploads from the browser to cloud storage providers such as AWS S3 and ImageKit.

How It Works

The SDK shifts the responsibility of file transfer away from the application server. The workflow is as follows:

The Node.js server validates the upload request.
The server generates a short-lived, signed upload target.
The browser uploads the file directly to the storage provider using the signed details.

This approach ensures that the application server never needs to stream or proxy the actual file data, reducing bandwidth usage and server load.

Configuration and Usage

Developers define named upload types (e.g., avatar, invoice) to manage different file categories. Each type includes configuration rules:

Allowed MIME types and file extensions
Maximum file size
Expiry time for the upload request
Storage destination and key prefix

To initiate an upload, the browser sends the asset name and file metadata to the server. The server then calls prepareUpload(), which validates the request, generates a collision-resistant storage key, and returns signed multipart POST details to the client.

Multiple Storage Profiles

The SDK supports multiple storage profiles within the same application. For instance, developers can configure:

One S3 bucket for public files
Another S3 bucket for private documents
ImageKit for image-specific storage

This configuration maintains a consistent upload flow across different providers.

Security with Signed Policies

For Amazon S3, the SDK includes security constraints directly in the signed POST policy. This policy can enforce:

File size limits
Content type validation
Specific storage keys
Expiration times

These restrictions are enforced by S3 during the upload process, adding an additional layer of security.

Next Steps

The project is open for feedback on its current API and for suggestions regarding which storage providers to support next.

Xberg v1: A High-Performance Content Intelligence Framework

Ray Forever — Mon, 03 Aug 2026 07:53:35 +0000

Overview

Xberg v1 has been released as the successor to Kreuzberg, representing a significant evolution in content intelligence processing. The framework handles a wide range of input types including documents (101 formats), code and data (367 types), audio/video transcription, and URLs with static or JavaScript-rendered content.

Performance and Architecture

Xberg is engineered as a high-performance engine with substantial improvements over its predecessor. For PDF and image processing, it handles native PDFs with high performance and accuracy, and includes multiple OCR engines that match the quality of leading Python libraries such as docling, PaddleOCR, and RapidOCR, while offering better performance and stability.

Key Technical Changes

The transition from Kreuzberg v4 to Xberg v1 introduces several major architectural changes:

Pure-Rust PDF backend (pdf_oxide) replaces pdfium, eliminating native pdfium dependencies
Layout-aware pipeline with ONNX layout detection using PP-DocLayoutV3/RT-DETR and Docling-style predecessor-graph reordering
Per-page scanned-page detection with selective OCR, plus AcroForm/XFA form field support and outline-based headings
Optimized OCR and PDF extraction with improved memory discipline, pooled model sessions, and streamed conversions
Native PaddleOCR backend supporting PP-OCRv6 with medium/small/tiny tiers alongside Tesseract

Inference and Model Support

The framework includes multiple inference paths:

Pure-Rust Candle OCR/VLM stack supporting TrOCR, GLM-OCR, GOT-OCR, DeepSeek-OCR, and PaddleOCR-VL without ONNX Runtime or native Tesseract dependencies
Secondary inference path via tract enabling ONNX-free execution for in-browser (WASM) and mobile inference
Named-entity recognition natively in Rust (GLiNER2) extensible to all bindings, including in-browser WASM models with no server round-trip

Structured Extraction and Retrieval

Xberg provides comprehensive content processing capabilities:

Structured LLM extraction (extract_structured/split_and_extract) with rasterization, chunking, citations, caching, and configurable policies
Audio and video transcription via Whisper ONNX engine supporting .mp3, .wav, .m4a, .mp4, and .webm formats
Retrieval building blocks including sparse embeddings (SPLADE), ColBERT late-interaction retrieval, cross-encoder reranking, and dense embeddings
Text intelligence features including reversible redaction, summarization, translation, VLM image captioning, QR-code detection, document diffing, and page/chunk classification

Web and Format Support

URL and web ingestion with sitemap discovery (map_url) and batched multi-URL crawling
New document formats including WordPerfect (.wpd/.wp/.wp5), HEIC/HEIF/AVIF, OpenDocument Presentation (.odp), Quarto/R Markdown, and configurable Jupyter cell rendering
Expanded code intelligence with tree-sitter coverage growing from 248 to 367+ languages

Cross-Platform and Bindings

Four new language bindings (Dart/Flutter, Swift, Kotlin/Android, and Zig) bring the total to 15 language bindings over one engine. The framework now includes full mobile support for Flutter, Android, and iOS, with cross-compilation capabilities.

Benchmark Results

Performance benchmarks for PDF and image processing demonstrate Xberg's capabilities:

Composite Quality (Markdown Pipeline, higher is better):

Xberg (layout): 0.958 (Native PDF), 0.836 (Scanned PDF)
docling: 0.779 (Native PDF), 0.762 (Scanned PDF)
mineru: 0.408 (Native PDF), 0.792 (Scanned PDF)

Structure and Layout Fidelity (SF1: tables and reading order, higher is better):

Xberg: 0.949 (Native PDF), 0.531 (Scanned PDF)
docling: 0.612 (Native PDF), 0.366 (Scanned PDF)

For native PDFs, Xberg leads on quality (0.958 vs 0.837 for the next-best framework) and table/reading-order fidelity (SF1 0.949 vs 0.612 for docling). On scanned PDFs, it ranks first on both quality and raw text fidelity.

Migration and Support

A migration guide is available for transitioning from Kreuzberg to Xberg. Kreuzberg remains in LTS mode until the end of the year with continued bug fixes and security updates. The full changelog documents all changes, with over 150 bugs fixed during the 1.0 cycle plus security hardening including bounded RTF/PDF allocations, redaction leak fixes, and Excel DDE warnings.

Testing title

Ray Forever — Fri, 31 Jul 2026 13:40:57 +0000

Testing body