DEV Community: refaat Al Ktifan

Pimcore Upgrade 10 to 12: The Real Migration Path

refaat Al Ktifan — Wed, 08 Jul 2026 11:03:18 +0000

Why You Can't Skip Pimcore 11

The first thing every team asks: can we jump straight from 10 to 12? No. Pimcore 11 is the mandatory stepping stone. The framework changes are too large to absorb in one step, and Pimcore's own migration tooling assumes the intermediate version is in place.

Here's the full scope of what changes across the two major versions:

Component	Pimcore 10.6	Pimcore 11	Pimcore 12
PHP	8.0+	8.1+	8.3+
Symfony	5.4	6.2+	6.4 / 7.x
Doctrine DBAL	2.x / 3.x	3.x	4.x
Admin UI	Built-in (ExtJS 6)	Separate bundle (ExtJS 6/7)	Separate bundle (ExtJS 7) + Studio UI (React)
License	GPLv3	GPLv3	POCL (Pimcore Open Core License)
Core bundles	Monolithic	Extracted	Extracted
Search backend	Built-in	Simple Backend Search Bundle	Generic Data Index + OpenSearch
WYSIWYG	TinyMCE built-in	TinyMCE bundle	TinyMCE or Quill bundle
Product registration	None	None	Required (offline validation)
Event system (JS)	Plugin Broker	Event Listeners	Event Listeners + Studio UI events
Collation	utf8mb4_general_ci	utf8mb4_general_ci	utf8mb4_unicode_520_ci (required)
Cache	Pimcore adapter	Symfony adapter	Symfony adapter (config change)
Full page cache	Built-in	Built-in	Separate bundle option
Asset storage	Local / Flysystem	Local / Flysystem	Local / Flysystem (with bugs on remote)

We've done this upgrade on multiple enterprise Pimcore installations ranging from B2B PIMs to multi-site CMS platforms. The patterns described here are product-generic. For context on how we approach PIM implementations and system architecture, those guides cover our broader methodology. Our methodology page explains how we plan these kinds of high-risk migrations.

┌─────────────────┐     ┌─────────────────┐     ┌─────────────────┐
│  Pimcore 10.6   │ ──▶ │  Pimcore 11.x   │ ──▶ │  Pimcore 12.x   │
│  Symfony 5.4    │     │  Symfony 6.2+   │     │  Symfony 6.4/7  │
│  PHP 8.0+       │     │  PHP 8.1+       │     │  PHP 8.3+       │
│  DBAL 2.x/3.x   │     │  DBAL 3.x       │     │  DBAL 4.x       │
│  ExtJS 6        │     │  ExtJS 6/7      │     │  ExtJS 7 +      │
│                 │     │                 │     │  Studio UI      │
└─────────────────┘     └─────────────────┘     └─────────────────┘

Phase 1: Preparation (Before Touching Composer)

This phase takes the longest and saves the most time. Every hour spent here saves days of debugging later.

1.1 Pre-Upgrade Checklist

# Verify current version
bin/console --version

# Ensure all migrations are current
bin/console doctrine:migrations:up-to-date
bin/console doctrine:migrations:migrate

# Empty all message queues before upgrade
# Workers must finish processing before you change the codebase
bin/console messenger:consume --limit=0

Back up everything: database (full mysqldump), files (assets directory, var directory, config directory), and create a dedicated upgrade branch in version control. Test your restore procedure before starting.

1.2 Deprecated Doctrine Methods (DBAL 2/3 to 4)

This is the most labor-intensive preparation step. Every repository class, every raw SQL query, every custom service that touches the database needs updating. In a typical enterprise installation, expect 50-150 places.

Deprecated (DBAL 2/3)	Replacement (DBAL 4)	Notes
`$db->query($sql)`	`$db->executeQuery($sql)`	Return type changes to `Result`
`$db->executeUpdate($sql)`	`$db->executeStatement($sql)`	Returns affected row count as `int`
`$db->fetchRow()`	`$db->fetchAssociative()`	Returns associative array or `false`
`$db->fetchAll()`	`$db->fetchAllAssociative()`	Returns array of associative arrays
`$db->fetchCol()`	`$db->fetchFirstColumn()`	Returns indexed array
`$db->fetchColumn()`	`$db->fetchOne()`	Single value fetch
`$db->fetchPairs()`	`Pimcore\Db\Helper::fetchPairs()`	Pimcore helper
`$db->insertOrUpdate()`	`Pimcore\Db\Helper::upsert()`	Pimcore helper
`$db->quoteInto()`	`Pimcore\Db\Helper::quoteInto()`	Pimcore helper
`$db->deleteWhere()`	`$db->executeStatement()` with DELETE	Manual SQL
`$db->updateWhere()`	`$db->executeStatement()` with UPDATE	Manual SQL
`$db->quote($val, $type)`	`$db->quote($val)`	Strings only in DBAL 4, no type param
`Pimcore\Db\ConnectionInterface`	`Doctrine\DBAL\Connection`	Type hint change in DI
`Pimcore\Db\Connection`	`Doctrine\DBAL\Connection`	Class replacement

# Find all deprecated method calls in your codebase
grep -rn "->query(" src/ --include="*.php"
grep -rn "->fetchRow(" src/ --include="*.php"
grep -rn "->fetchAll(" src/ --include="*.php" | grep -v "fetchAllAssociative"
grep -rn "->fetchCol(" src/ --include="*.php"
grep -rn "->insertOrUpdate(" src/ --include="*.php"
grep -rn "->quoteInto(" src/ --include="*.php"
grep -rn "->deleteWhere(" src/ --include="*.php"
grep -rn "->executeUpdate(" src/ --include="*.php"
grep -rn "Pimcore\\Db\\Connection" src/ --include="*.php"

Full migration example for a repository class:

// BEFORE (Pimcore 10 / DBAL 2-3)
use Pimcore\Db\ConnectionInterface;

class PriceExporter
{
    public function __construct(
        private readonly ConnectionInterface $connection
    ) {}

    public function export(int $limit, int $offset): array
    {
        $sql = "SELECT p.o_id, p.price FROM object_store_1 p WHERE p.o_published = 1 LIMIT ? OFFSET ?";
        return $this->connection->query($sql)->fetchAll();
    }
}

// AFTER (Pimcore 12 / DBAL 4)
use Doctrine\DBAL\Connection;

class PriceExporter
{
    public function __construct(
        private readonly Connection $connection
    ) {}

    public function export(int $limit, int $offset): array
    {
        $sql = "SELECT p.id, p.price FROM object_store_1 p WHERE p.published = 1 LIMIT ? OFFSET ?";
        return $this->connection->executeQuery($sql, [$limit, $offset])->fetchAllAssociative();
    }
}

Update your services.yaml if using explicit dependency injection:

services:
    App\Export\PriceExporter:
        arguments:
            $connection: '@doctrine.dbal.default_connection'

1.3 Return Type Declarations

Pimcore 11+ enforces strict return types on model classes. Every class extending Pimcore base classes needs updating:

// Before (Pimcore 10)
public function save()
{
    // custom logic
    parent::save();
}

// After (Pimcore 11+)
public function save(array $parameters = []): static
{
    // custom logic
    return parent::save($parameters);
}

This affects every DataObject model class, every custom listing, and every service that overrides Pimcore methods. Check all classes in your src/Model/ directory.

1.4 Remove the o_ Prefix

Pimcore 11 removes the o_ prefix from database column names in the objects table. Every raw SQL query and listing condition that references these columns needs updating:

Old (Pimcore 10)	New (Pimcore 11+)	Context
`o_id`	`id` (or `oo_id` in object store tables)	Primary key
`o_creationDate`	`creationDate`	Timestamp
`o_modificationDate`	`modificationDate`	Timestamp
`o_path`	`path`	Tree path
`o_key`	`key`	Object key/slug
`o_published`	`published`	Publication flag
`o_parentId`	`parentId`	Parent reference
`o_type`	`type`	Object type
`o_className`	`className`	Class name
`o_classId`	`classId`	Class ID
`o_userOwner`	`userOwner`	Creator user ID
`o_userModification`	`userModification`	Last modifier user ID
`o_versionCount`	`versionCount`	Version counter

# Find all o_ prefix usage across PHP and JavaScript
grep -rn "o_id\|o_path\|o_key\|o_published\|o_parentId\|o_type\|o_className\|o_classId" src/ --include="*.php"
grep -rn "o_className\|o_id" public/ --include="*.js"

Example listing condition migration:

// Before
$listing->setCondition('o_path LIKE ? AND o_published = 1', ['/products/%']);

// After
$listing->setCondition('path LIKE ? AND published = 1', ['/products/%']);

1.5 PHP 8.3 / 8.4 Breaking Changes

Beyond what Pimcore requires, PHP 8.3+ introduces its own breaking changes:

Implicitly nullable parameters (deprecated in 8.4):

// DEPRECATED: implicit nullable
public function findByLocale(string $locale = null): array

// FIXED: explicit nullable
public function findByLocale(?string $locale = null): array

This affects every method signature with = null as default. In a typical enterprise codebase, expect 50+ occurrences. Use Rector to automate:

// rector.php
use Rector\Config\RectorConfig;
use Rector\Php84\Rector\Param\ExplicitNullableParamTypeRector;

return static function (RectorConfig $rectorConfig): void {
    $rectorConfig->paths([__DIR__ . '/src']);
    $rectorConfig->rule(ExplicitNullableParamTypeRector::class);
};

# Preview changes
vendor/bin/rector process --dry-run

# Apply changes
vendor/bin/rector process

Other PHP 8.3+ features you can adopt:

// Typed class constants (PHP 8.3)
public const string DEFAULT_LOCALE = 'en';
public const array SUPPORTED_TYPES = ['product', 'category'];

// #[\Override] attribute (PHP 8.3)
#[\Override]
public function normalize(mixed $object, ?string $format = null, array $context = []): array

// json_validate() (PHP 8.3)
if (json_validate($input)) { /* ... */ }

1.6 JavaScript Event System Migration

The old plugin broker system is removed in Pimcore 11. All admin UI JavaScript must migrate:

// BEFORE (Pimcore 10 - Plugin Broker)
pimcore.plugin.datasheet = Class.create(pimcore.plugin.admin, {
    getClassName: function () {
        return "pimcore.plugin.datasheet";
    },
    initialize: function () {
        pimcore.plugin.broker.registerPlugin(this);
    },
    postOpenObject: function (object, type) {
        if (object.data.general.o_className === 'Product') {
            // Add toolbar button
        }
    }
});

// AFTER (Pimcore 11+ - Event Listeners)
(function() {
    'use strict';
    document.addEventListener(pimcore.events.postOpenObject, (e) => {
        let object = e.detail.object;
        if (object.data.general.className === 'Product') {  // o_className -> className
            object.toolbar.add({
                text: t('Generate PDF'),
                iconCls: 'pimcore_icon_pdf',
                handler: function () {
                    Ext.Ajax.request({
                        url: '/api/generate-pdf/' + object.id,
                        success: function (response) {
                            pimcore.helpers.showNotification(
                                t("success"), t("PDF generated"), "success"
                            );
                        }
                    });
                }
            });
            pimcore.layout.refresh();
        }
    });
})();

Additional JavaScript changes:

Old (Pimcore 10)	New (Pimcore 11+)	Notes
`ts()`	`t()`	Translation function
`pimcore.helpers.addCsrfTokenToUrl()`	Remove entirely	CSRF handled differently
`o_className` in JS	`className`	Matches PHP change
`Class.create(...)`	Still supported, but prefer ES6 classes	Future-proofing
`/admin/tags/*`	May change to `/pimcore-admin/tags/*`	Check your admin routes

1.7 ExtJS 7 Changes (Pimcore 12)

Pimcore 12 uses ExtJS 7.x. Key breaking changes for custom admin UI code:

// ExtJS 6 (Pimcore 10/11)
var store = new Ext.data.Store({
    proxy: {
        type: 'ajax',
        reader: {
            type: 'json',
            root: 'data'  // DEPRECATED in ExtJS 7
        }
    }
});

// ExtJS 7 (Pimcore 12)
var store = new Ext.data.Store({
    proxy: {
        type: 'ajax',
        reader: {
            type: 'json',
            rootProperty: 'data'  // Use rootProperty
        }
    }
});

Verify all custom grid operators, admin plugins, and data importer extensions for ExtJS 7 compatibility. Common issues:

root property renamed to rootProperty in store readers
Some grid column configs changed
Prototype patching (Ext.override) may break if internal class structure changed
Check all iconCls references still exist in Pimcore 12's icon set

1.8 Pimcore Studio UI (React)

Pimcore 12 introduces the new Studio UI built with React, running alongside the classic ExtJS admin. This is not a replacement yet but an addition:

Studio UI handles some new features (Generic Data Index UI, some asset management)
Classic Admin UI (ExtJS) remains the primary interface for most operations
Custom ExtJS plugins continue to work in the classic admin
New extension points for Studio UI use React components
Long-term direction is toward React, but full migration is not required for P12

If you have custom admin UI extensions, they will continue to work in the classic admin. No immediate migration to React is needed. But if you're building new admin functionality, consider building it for Studio UI instead. We wrote a detailed walkthrough on building a production-grade Pimcore 12 bundle with Studio UI integration that covers the full process from bundle skeleton to React component registration.

1.9 Remove SensioFrameworkExtraBundle

This bundle is deprecated and removed in Symfony 6. Replace all annotations:

// Before (Sensio)
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;

/** @Template("default/default.html.twig") */
public function defaultAction()
{
    return [];
}

// After (Symfony 6+)
public function defaultAction(): Response
{
    return $this->render('default/default.html.twig', []);
}

1.10 Symfony Route Annotations to Attributes

All 15+ controller files need this change:

// Before (Symfony 5.4 annotations)
use Symfony\Component\Routing\Annotation\Route;

/**
 * @Route("/{_locale}/{slug}_p{id}",
 *     name="product_detail",
 *     methods={"GET"},
 *     requirements={"slug"="[\w\s\-\.]+", "id"="[\w\. ]+"}
 * )
 */
public function __invoke(Request $request, string $id): Response

// After (Symfony 6.4+ attributes)
use Symfony\Component\Routing\Attribute\Route;

#[Route(
    path: '/{_locale}/{slug}_p{id}',
    name: 'product_detail',
    methods: ['GET'],
    requirements: ['slug' => '[\w\s\-\.]+', 'id' => '[\w\. ]+']
)]
public function __invoke(Request $request, string $id): Response

Update config/packages/routing.yaml:

# Change type from 'annotation' to 'attribute'
controllers:
    resource:
        path: ../src/Controller/
        namespace: App\Controller
    type: attribute

1.11 Symfony Serializer Changes (Symfony 7.x)

If you have custom Normalizers (common for search indexing, API serialization), Symfony 7.x requires a new method:

// Before (Symfony 5.4/6.x)
class ProductNormalizer implements NormalizerInterface
{
    public function normalize(mixed $object, string $format = null, array $context = []): array
    {
        // ...
    }

    public function supportsNormalization(mixed $data, string $format = null): bool
    {
        return $data instanceof Product;
    }
}

// After (Symfony 7.x)
class ProductNormalizer implements NormalizerInterface
{
    public function normalize(mixed $object, ?string $format = null, array $context = []): array
    {
        // ...
    }

    public function supportsNormalization(mixed $data, ?string $format = null, array $context = []): bool
    {
        return $data instanceof Product;
    }

    // NEW: Required in Symfony 7.x
    public function getSupportedTypes(?string $format): array
    {
        return [
            Product::class => true,
        ];
    }
}

If you have 10+ normalizers (typical for MeiliSearch or OpenSearch indexing), this is a repetitive but critical change. Missing getSupportedTypes() causes a fatal error on Symfony 7.x.

1.12 Request::get() Deprecation

Request::get() is deprecated in Symfony 6.x. Use specific accessor methods:

// Deprecated
$value = $request->get('param');

// Use specific methods
$value = $request->query->get('param');       // GET parameters
$value = $request->request->get('param');     // POST parameters
$value = $request->attributes->get('param');  // Route parameters
$value = $request->headers->get('param');     // Headers
$value = $request->query->getBoolean('flag'); // Boolean GET parameter

1.13 Install Compatibility Bridge

composer require --no-update pimcore/compatibility-bridge-v10
composer require --no-update symfony/dotenv

Phase 2: Upgrade to Pimcore 11

2.1 Composer Update

{
  "require": {
    "pimcore/pimcore": "^11.0",
    "pimcore/admin-ui-classic-bundle": "^1.0"
  }
}

composer remove --no-update sensio/framework-extra-bundle
COMPOSER_MEMORY_LIMIT=-1 composer update -W

Expect dependency resolution to struggle with third-party bundles. Contact bundle vendors for P11-compatible versions or find alternatives.

2.2 Register the Admin Bundle

// src/Kernel.php
public function registerBundlesToCollection(BundleCollection $collection): void
{
    $collection->addBundle(new \Pimcore\Bundle\AdminBundle\PimcoreAdminBundle(), 60);
}

2.3 Install Extracted Bundles

Many core features are now separate bundles. Install only what you actually use:

composer require \
  pimcore/static-routes-bundle \
  pimcore/custom-reports-bundle \
  pimcore/application-logger-bundle \
  pimcore/seo-bundle \
  pimcore/simple-backend-search-bundle \
  pimcore/glossary-bundle \
  pimcore/tinymce-bundle \
  pimcore/xliff-bundle \
  pimcore/word-export-bundle

2.4 Update Route Configuration

# config/routes.yaml
_pimcore:
    resource: "@PimcoreCoreBundle/config/routing.yaml"  # was: Resources/config/routing.yml

# config/routes/dev/routes.yaml
_pimcore:
    resource: "@PimcoreCoreBundle/config/routing_dev.yaml"  # was: Resources/config/routing_dev.yml

2.5 Messenger Transport Configuration

Pimcore 11 introduces new async message transports:

framework:
    messenger:
        transports:
            # Your existing transports...

            # NEW for Pimcore 11+
            pimcore_scheduled_tasks:
                dsn: "%messenger.dsn%/pimcore_scheduled_tasks"
                retry_strategy:
                    max_retries: 3
                    delay: 1000
                    multiplier: 2
            pimcore_image_optimize:
                dsn: "%messenger.dsn%/pimcore_image_optimize"
            pimcore_asset_update:
                dsn: "%messenger.dsn%/pimcore_asset_update"
            pimcore_search_backend_message:
                dsn: "%messenger.dsn%/pimcore_search_backend_message"

If using RabbitMQ (AMQP), each transport creates a separate queue. If using Doctrine transport, each creates a separate table partition. Plan worker deployment accordingly.

2.6 Run Migrations and Rebuild

bin/console doctrine:migrations:migrate
bin/console pimcore:cache:clear
bin/console cache:clear
rm -rf var/cache/*

Test everything at this checkpoint before proceeding.

If you're also redesigning workflows during this upgrade, read our guide on Pimcore enterprise workflow design for the architectural patterns.

Phase 3: Upgrade to Pimcore 12

3.1 License and Product Registration

Pimcore 12 introduces two mandatory changes:

License: POCL (Pimcore Open Core License) replaces GPLv3. Community Edition with Admin UI Classic Bundle requires an ExtJS license (approximately EUR 1,480).

Edition	Price
Community (POCL)	Free (revenue under EUR 5M)
Professional	EUR 8,400/year
Enterprise Self-Hosted	EUR 25,200/year
Enterprise PaaS	From USD 39,900/year

Product Registration: Mandatory. Three environment variables required:

Variable	Purpose
`PIMCORE_ENCRYPTION_SECRET`	Defuse encryption key, generated once per installation
`PIMCORE_PRODUCT_KEY`	Registration key from license.pimcore.com
`PIMCORE_ENTERPRISE_TOKEN`	Composer auth token for private packages

Registration steps:

# 1. Generate encryption secret (once per installation, NEVER change after)
vendor/bin/generate-defuse-key

# 2. Calculate registration hash
php -r "echo hash_hmac('sha256', 'your-instance-id', 'your-encryption-secret');"

# 3. Register at license.pimcore.com with the hash
# URL: https://license.pimcore.com/register?instance_identifier=ID&instance_hash=HASH
# 4. Receive product key, set as PIMCORE_PRODUCT_KEY env var

# config/config.yaml
pimcore:
    encryption:
        secret: '%env(PIMCORE_ENCRYPTION_SECRET)%'
    product_registration:
        product_key: '%env(PIMCORE_PRODUCT_KEY)%'
        instance_identifier: 'your-instance-id'

The validation is completely offline. It runs at every Symfony container build (cache clear, app start). It verifies the product key's ECDSA signature against a public key shipped with Pimcore. No internet required. If validation fails, InvalidConfigurationException is thrown and the app refuses to start.

Critical: Never change the encryption secret after initial setup. All encrypted data in the database depends on it. Treat it like a database password. Each separate installation (staging, production) can share the same key if they share the same instance identifier.

3.2 Composer Update

composer require -W pimcore/pimcore:^12.0

3.3 Security Configuration

security:
    # REMOVE (default in Symfony 6.4+):
    # enable_authenticator_manager: true

    firewalls:
        pimcore_admin_webdav:
            pattern: ^/asset/webdav  # was: ^/admin/asset/webdav

    access_control:
        - { path: ^/asset/webdav, roles: ROLE_PIMCORE_USER }

3.4 WYSIWYG Editor

# Option A: TinyMCE (Professional/Enterprise)
composer require pimcore/tinymce-bundle

# Option B: Quill (free, Community Edition)
composer require pimcore/quill-bundle

3.5 Doctrine DBAL 4 Final Changes

Beyond method replacements from Phase 1, update the Doctrine config:

# config/packages/doctrine.yaml
doctrine:
    dbal:
        connections:
            default:
                driver: "pdo_mysql"
                server_version: "mariadb-10.11"  # Be specific
                charset: utf8mb4
                default_table_options:
                    charset: utf8mb4
                    collate: utf8mb4_unicode_520_ci  # P12 default

Redis Configuration Deep Dive

Redis serves four critical functions in a production Pimcore 12 deployment. Getting the configuration wrong causes silent failures that are hard to diagnose.

1. Application Cache (Tag-Aware)

# config/packages/cache.yaml
framework:
    cache:
        # Pimcore 10 (OLD)
        # pools:
        #     pimcore.cache.pool:
        #         tags: true                                    # REMOVE
        #         adapter: pimcore.cache.adapter.redis_tag_aware  # OLD adapter name

        # Pimcore 11+/12 (NEW)
        pools:
            pimcore.cache.pool:
                public: true
                default_lifetime: 31536000  # 1 year
                adapter: cache.adapter.redis_tag_aware   # Changed adapter name
                provider: 'redis://%env(REDIS_HOST)%:%env(REDIS_PORT)%'

The adapter name change from pimcore.cache.adapter.redis_tag_aware to cache.adapter.redis_tag_aware is silent. The old name doesn't throw an error. Caching just silently falls back to filesystem, and you wonder why your multi-pod deployment has stale data everywhere.

Without Redis caching, every page render hits the database for navigation trees, site settings, class definitions, translations, and more. On a multi-pod Kubernetes deployment, each pod maintains its own filesystem cache with no invalidation coordination. Redis makes cache invalidation consistent across all pods.

2. Session Storage

# config/packages/framework.yaml
framework:
    session:
        handler_id: '%env(REDIS_SESSION_DSN)%'
        # Example: redis://redis:6379/1

For multi-pod deployments, sessions must be shared. Without Redis sessions, users get logged out when their request hits a different pod (round-robin load balancing). This is the most visible symptom of a missing Redis configuration.

3. Full Page Cache

Pimcore's full page cache stores rendered HTML output for anonymous users. In Pimcore 12, it can use Redis as its backend:

# config/packages/pimcore.yaml
pimcore:
    full_page_cache:
        enabled: true
        lifetime: 7200  # 2 hours
        exclude_patterns:
            - '/admin'
            - '/api'
        exclude_cookie: 'pimcore_admin_sid'

The full page cache dramatically reduces database load for public-facing pages. A page that takes 200ms to render from the database serves in under 5ms from cache. On high-traffic sites, this is the difference between needing 2 web pods and needing 10.

Cache invalidation happens automatically when editors publish content. But be careful with personalized content: any page that varies by user context (login state, locale, A/B test variant) must be excluded from full page caching or use ESI fragments.

4. Messenger Transport (Optional)

framework:
    messenger:
        transports:
            pimcore_core:
                dsn: 'redis://%env(REDIS_HOST)%:%env(REDIS_PORT)%/messages'

Redis Streams can replace RabbitMQ as the message transport. Simpler to operate (one fewer service), but lacks dead letter queues, message priorities, and exchange routing. For installations with complex worker topologies, RabbitMQ remains better. For simpler setups, Redis Streams works fine.

Redis Sizing

Pimcore Scale	Redis Memory	Notes
Small (< 10K objects)	256 MB	Single instance
Medium (10K-100K objects)	1-2 GB	Single instance, persistence enabled
Large (100K+ objects)	4-8 GB	Consider Redis Sentinel or Cluster

Monitor used_memory and evicted_keys. If keys are being evicted, your cache is too small and you're losing performance.

The Collation Migration (The Hardest Part)

This is the section that saves the most time. The collation migration is the most dangerous and least documented part of the entire upgrade.

Why It Breaks

Pimcore 12 migrations create new tables and alter existing columns with utf8mb4_unicode_520_ci collation. Your existing database from Pimcore 10 uses whatever the MySQL server default was. On Azure MySQL 8.0, that's utf8mb4_0900_ai_ci. On older setups, it might be utf8mb4_general_ci.

The result: mixed collations within the same table and across tables. When Pimcore runs UNION queries across objects, assets, and documents tables:

-- This is what Pimcore's getRelationData() runs
SELECT r.dest_id, r.type, o.className as subtype, concat(o.path, o.`key`) as `path`
  FROM objects o, object_relations_X r WHERE ...
UNION
SELECT r.dest_id, r.type, a.type as subtype, concat(a.path, a.filename) as `path`
  FROM assets a, object_relations_X r WHERE ...
UNION
SELECT r.dest_id, r.type, d.type as subtype, concat(d.path, d.`key`) as `path`
  FROM documents d, object_relations_X r WHERE ...

If objects.className is utf8mb4_unicode_520_ci (from P12 migration) but assets.type is utf8mb4_0900_ai_ci (from Azure default), MySQL throws:

SQLSTATE[HY000]: General error: 1267 Illegal mix of collations for operation 'UNION'

This error appears when opening any DataObject in the admin UI. The system is effectively broken.

Actual Distribution You'll Find

On a database migrated from P10 through P11 to P12 on Azure MySQL 8.0:

Collation	Tables	Columns	Source
`utf8mb4_0900_ai_ci`	~580	~5,700	Azure MySQL 8.0 default
`utf8mb4_unicode_ci`	~87	~155	Old bundle tables
`utf8mb4_unicode_520_ci`	~6	~994	P12 migration-created
`utf8mb3_general_ci`	~4	~31	Very old legacy tables
`utf8mb4_general_ci`	~1	~3	Specific bundle tables

# Check your current distribution
bin/console doctrine:query:sql "SELECT TABLE_COLLATION, COUNT(*) AS cnt FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_COLLATION IS NOT NULL GROUP BY TABLE_COLLATION ORDER BY cnt DESC"

# Column-level check (this is where the real problem hides)
bin/console doctrine:query:sql "SELECT COLLATION_NAME, COUNT(*) AS cnt FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = DATABASE() AND COLLATION_NAME IS NOT NULL GROUP BY COLLATION_NAME ORDER BY cnt DESC"

Five Special Cases That Break Simple ALTER TABLE

1. Path-indexed tables (assets, documents, objects, http_error_log):
Composite indexes on path + key/filename exceed the 3072-byte InnoDB index limit after collation conversion (utf8mb4 = 4 bytes/char, 255 chars x 4 = 1020 bytes per column, composite = over limit).

-- Drop indexes, convert, recreate with prefix lengths
DROP INDEX fullpath ON objects;
DROP INDEX type_path_classId ON objects;
ALTER TABLE objects CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;
CREATE INDEX fullpath ON objects (path(255), `key`(255));
CREATE INDEX type_path_classId ON objects (type, path(255), classId);

2. Classification store tables: FK constraint between object_classificationstore_data_* and object_classificationstore_groups_*. The FK must be dropped before conversion and recreated after. The column name may be id (P12) or o_id (P10/11) depending on migration state.

3. Translation tables (translations_admin, translations_messages): Collation change can expose case-insensitive duplicates. If you have both "MyKey" and "mykey" as separate translation keys (valid under case-sensitive collation), converting to case-insensitive collation causes a unique constraint violation. We found ~1,600 duplicates in a single staging database. Duplicates must be removed before conversion.

4. Assets table: Must recreate the fullpath index as non-unique. Case-different filenames can exist (e.g., EEVA.tif and Eeva.tif are legitimate distinct files). A unique index would reject the second one after collation change.

5. Large tables: Tables with millions of rows (localized query tables, for example, can have 30+ tables per class with one per locale) take significant time to ALTER. On a database with 680 tables, the full conversion takes ~18 minutes.

Running the Migration

Run during a maintenance window with web pods scaled down:

# Scale down to avoid lock contention on translation tables
kubectl scale deployment pimcore pimcore-frontend -n <namespace> --replicas=0
kubectl get pods -n <namespace> -l app=pimcore -w  # Wait for termination

After running the automated migration script (which handles all five special cases):

# Rebuild class definitions (restores FK constraints)
PIMCORE_CLASS_DEFINITION_WRITABLE=1 bin/console pimcore:deployment:classes-rebuild -c

# Verify only one collation remains
bin/console doctrine:query:sql "SELECT TABLE_COLLATION, COUNT(*) FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = DATABASE() AND TABLE_COLLATION IS NOT NULL GROUP BY TABLE_COLLATION"

# Scale back up
kubectl scale deployment pimcore -n <namespace> --replicas=1
kubectl scale deployment pimcore-frontend -n <namespace> --replicas=3

Quick Reference: Collation Errors

Error	Cause	Fix
`Illegal mix of collations for operation 'UNION'`	Mixed collations across tables in relation queries	Convert all tables to `utf8mb4_unicode_520_ci`
`Specified key was too long; max key length is 3072 bytes`	Composite path+key index exceeds limit	Drop index, convert, recreate with prefix lengths
`Referencing column and referenced column are incompatible`	FK between tables with different collations	Drop FK, convert both tables, rebuild classes
`Duplicate entry for key 'PRIMARY'`	Collation change makes case-different keys identical	Remove duplicates before converting
`Lock wait timeout exceeded`	App holds locks on translation tables	Scale down pods before ALTER

Database Optimization After Migration

After collation migration, run OPTIMIZE on all tables to defragment storage and rebuild indexes:

# Core tables
bin/console doctrine:query:sql "OPTIMIZE TABLE assets, documents, objects, versions, translations_admin, translations_messages, search_backend_data, properties, dependencies"

# Check for high fragmentation
bin/console doctrine:query:sql "SELECT TABLE_NAME, TABLE_ROWS, ROUND(DATA_LENGTH/1024/1024,2) AS data_mb, ROUND(DATA_FREE/1024/1024,2) AS free_mb, ROUND(DATA_FREE/(DATA_LENGTH+1)*100,1) AS frag_pct FROM information_schema.TABLES WHERE TABLE_SCHEMA = DATABASE() AND DATA_FREE > DATA_LENGTH * 0.1 AND DATA_LENGTH > 1048576 ORDER BY DATA_FREE DESC"

MySQL SSL Configuration

For managed MySQL services (Azure, AWS RDS) with require_secure_transport=ON:

# config/packages/prod/doctrine.yaml
doctrine:
    dbal:
        connections:
            default:
                options:
                    !php/const PDO::MYSQL_ATTR_SSL_CA: true
                    !php/const PDO::MYSQL_ATTR_SSL_VERIFY_SERVER_CERT: false

MYSQL_ATTR_SSL_CA: true enables SSL without specifying a certificate path. The server provides its certificate and the client trusts it. MYSQL_ATTR_SSL_VERIFY_SERVER_CERT: false skips hostname verification (safe for internal networking).

For how we handle database migrations in our broader data engineering practice, that page covers our methodology.

OpenSearch Integration

Pimcore 12 introduces the Generic Data Index bundle, which uses OpenSearch (or Elasticsearch) for backend search and data indexing.

Index Setup (Must Happen Before Migrations)

# Create empty indices manually (required before migrations run)
curl -X PUT "http://opensearch-cluster:9200/pimcore_document" -H 'Content-Type: application/json' -d '{}'
curl -X PUT "http://opensearch-cluster:9200/pimcore_asset" -H 'Content-Type: application/json' -d '{}'

# DO NOT manually create Portal Engine statistics indices
# They auto-create with time-based naming: {name}__{year}_{month}
# Manual creation conflicts with the alias pattern

This is critical because some P12 migrations trigger Document->save(), which fires the DocumentIndexUpdateSubscriber event. If the index doesn't exist: index_not_found_exception followed by cascading SAVEPOINT errors that corrupt the migration state.

After migrations complete:

# Create indices for all DataObject classes
php -d memory_limit=2G bin/console generic-data-index:deployment:reindex --no-interaction

# Available commands
bin/console list generic-data-index
# generic-data-index:deployment:reindex    Creates/updates indices for classDefinitions
# generic-data-index:reindex               Triggers native reindexing
# generic-data-index:update:index          Updates index/mapping

OpenSearch Cluster Sizing

Installation Scale	Nodes	Memory/Node	Storage/Node	Java Heap
Small (< 10K objects)	1	4 GB	50 GB SSD	2 GB
Medium (10K-100K)	2	8 GB	100 GB SSD	4 GB
Large (100K+)	3 (master-eligible)	16 GB	200 GB SSD	8 GB

If replacing Elasticsearch 7.x, OpenSearch is a drop-in replacement. The API is compatible. Only the client library and configuration paths change. For how we design search architecture more broadly, see our ecommerce platforms guide.

OpenSearch Security

For internal cluster communication (pods in the same Kubernetes namespace), you can disable the security plugin:

# opensearch-cluster.yaml
additionalConfig:
    plugins.security.disabled: "true"

For external access or multi-tenant clusters, configure proper authentication.

Flysystem and Remote Storage Bugs

If you use remote asset storage (Azure Blob, AWS S3, Google Cloud Storage), Pimcore 12 has two bugs you need to know about.

Bug 1: getDimensions() Downloads Every Image on Every Page Render

Affects: Pimcore 10.6+, 11.x, 12.x with remote Flysystem storage

Every thumbnail()->html() call triggers getDimensions(), which calls readDimensionsFromFile() before trying getEstimatedDimensions(). On remote storage, each file read costs 50-100ms of network I/O.

Measured impact: A page with ~80 image references took 5,735ms (79 remote storage calls at ~65ms each). After fixing the method order, the same page rendered in 170ms with zero remote calls.

Root cause in ImageThumbnailTrait::getDimensions():

// Current order (WRONG for remote storage)
// Step 1: Check DB thumbnail cache (fast, but often a miss)
// Step 2: readDimensionsFromFile()  → DOWNLOADS FILE FROM CLOUD (~65ms)
// Step 3: getEstimatedDimensions() → NEVER REACHED (step 2 always succeeds)

// Fixed order
if (empty($dimensions) && $config && $asset instanceof Image) {
    $dimensions = $config->getEstimatedDimensions($asset);  // ~0ms, pure math
}
if (empty($dimensions) && $this->exists()) {
    $dimensions = $this->readDimensionsFromFile();  // 50-100ms, last resort only
}

getEstimatedDimensions() calculates dimensions from the original image dimensions (stored in the database) and the thumbnail transformation config. Pure math, zero I/O. In production, 95%+ of images have stored dimensions. This fix eliminates virtually all remote storage I/O during page rendering.

Bug 2: Folder Preview Thumbnails Stuck on Loading Spinner

Affects: admin-ui-classic-bundle 2.2.x, 2.3.x

After uploading images, the Preview tab shows a loading spinner that never resolves. The browser caches the placeholder GIF permanently because the URL uses modificationDate as cache buster (never changes), and no async thumbnail generation is dispatched for folderPreview origin (unlike treeNode which does dispatch it).

Fix: Add Cache-Control: no-store to the placeholder response and dispatch AssetPreviewImageMessage.

Kubernetes Deployment

Pod Architecture

Pod	Purpose	Replicas
`pimcore-web`	Nginx + PHP-FPM, admin + frontend	2-4
`pimcore-worker`	Symfony Messenger consumers	1-3
`pimcore-ops`	Maintenance CLI pod (migrations, rebuilds)	1
`redis`	Cache, sessions, optional transport	1+
`mysql`	Database (or managed service)	1+
`opensearch`	Search indices	2-3
`rabbitmq`	Message broker (if not using Redis)	1-3

Secrets Management

apiVersion: v1
kind: Secret
metadata:
  name: pimcore-secrets
type: Opaque
data:
  PIMCORE_ENCRYPTION_SECRET: <base64>
  PIMCORE_PRODUCT_KEY: <base64>
  PIMCORE_ENTERPRISE_TOKEN: <base64>
  DATABASE_HOST: <base64>
  DATABASE_NAME: <base64>
  DATABASE_USER: <base64>
  DATABASE_PASSWORD: <base64>
  REDIS_HOST: <base64>
  REDIS_PORT: <base64>
  APP_SECRET: <base64>

Use Sealed Secrets (Bitnami) or external secret operators for production. Never commit plain secrets to Git.

Every init container (create-assets, create-classes, db-migrations, clear-cache) must have ALL secretRefs in its envFrom list. Missing secrets cause silent Environment variable not found errors.

Worker Deployment

spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: worker
          command: ["php", "bin/console", "messenger:consume",
            "pimcore_core", "pimcore_maintenance", "pimcore_scheduled_tasks",
            "pimcore_image_optimize", "pimcore_asset_update",
            "pimcore_search_backend_message",
            "--time-limit=3600", "--memory-limit=512M"]

--time-limit and --memory-limit ensure workers restart periodically, preventing PHP memory leaks. Kubernetes restarts the pod when the process exits gracefully.

Deployment Order

1.  Deploy Kubernetes resources
2.  Shell into ops pod
3.  Install missing bundles (PimcoreAdminBundle, GenericExecutionEngine, etc.)
4.  Create OpenSearch indices manually (pimcore_document, pimcore_asset)
5.  Run OpenSearch DataObject reindex
6.  Run database migrations (with PIMCORE_CLASS_DEFINITION_WRITABLE=1)
7.  Run collation migration (maintenance window, web pods scaled down)
8.  Run database optimization (OPTIMIZE TABLE)
9.  Post-deploy commands (classes-rebuild, reindex, assets:install, cache:clear)
10. Verify: open DataObject in admin, check assets, check frontend

Our cloud services cover Kubernetes deployment and operations for Pimcore and other platforms.

Event System Architecture in Pimcore 12

Understanding Pimcore's event system is critical for both the upgrade and for building extensions.

PHP Events (Backend)

Pimcore uses Symfony's EventDispatcher. Key events for DataObjects:

use Pimcore\Event\DataObjectEvents;

// Available events
DataObjectEvents::PRE_ADD      // Before first save
DataObjectEvents::POST_ADD     // After first save
DataObjectEvents::PRE_UPDATE   // Before every save
DataObjectEvents::POST_UPDATE  // After every save
DataObjectEvents::PRE_DELETE   // Before deletion
DataObjectEvents::POST_DELETE  // After deletion

Creating an event subscriber in Pimcore 12:

use Pimcore\Event\Model\DataObjectEvent;
use Symfony\Component\EventDispatcher\EventSubscriberInterface;

class ProductEventSubscriber implements EventSubscriberInterface
{
    public static function getSubscribedEvents(): array
    {
        return [
            DataObjectEvents::POST_UPDATE => 'onPostUpdate',
        ];
    }

    public function onPostUpdate(DataObjectEvent $event): void
    {
        $object = $event->getObject();
        if (!$object instanceof Product) {
            return;
        }
        // Index to search, generate assets, etc.
    }
}

JavaScript Events (Admin UI)

Pimcore 12 uses a different event subscription pattern than P10:

// Pimcore 12 admin events
document.addEventListener(pimcore.events.preOpenObject, (e) => {
    // Before object editor opens
});

document.addEventListener(pimcore.events.postOpenObject, (e) => {
    // After object editor opens, e.detail.object available
});

document.addEventListener(pimcore.events.preSaveObject, (e) => {
    // Before save, can cancel with e.preventDefault()
});

document.addEventListener(pimcore.events.postSaveObject, (e) => {
    // After save completes
});

Extending Pimcore 12

Custom bundles in Pimcore 12 follow Symfony bundle conventions:

// src/CustomBundle/CustomBundle.php
namespace App\CustomBundle;

use Pimcore\Extension\Bundle\AbstractPimcoreBundle;

class CustomBundle extends AbstractPimcoreBundle
{
    public function getJsPaths(): array
    {
        return [
            '/bundles/custom/js/startup.js',
        ];
    }

    public function getCssPaths(): array
    {
        return [
            '/bundles/custom/css/admin.css',
        ];
    }
}

# config/packages/pimcore_admin.yaml
pimcore_admin:
    assets:
        js:
            - '/bundles/custom/js/startup.js'
            - '/bundles/custom/js/grid-operators.js'
        css:
            - '/bundles/custom/css/admin.css'

Post-Upgrade Verification

Command Sequence

PIMCORE_CLASS_DEFINITION_WRITABLE=1 php -d memory_limit=2G bin/console pimcore:deployment:classes-rebuild -c
php -d memory_limit=2G bin/console generic-data-index:deployment:reindex --no-interaction
bin/console assets:install public
php -d memory_limit=2G bin/console pimcore:search-backend-reindex
bin/console messenger:setup-transports
bin/console cache:clear
bin/console cache:warmup

Functional Test Matrix

Area	What to Test	How to Verify
Admin login	Login, 2FA, logout	Manual in browser
DataObjects	Create, edit, save, publish, delete, versions	Open any object, edit a field, save
Assets	Upload, thumbnails, metadata, download	Upload an image, check thumbnail generation
Documents	Edit pages, areabricks, save, publish	Edit a page with areabricks
Relations	Object relations, asset relations	Open object with relations, verify UNION query works
Search	Backend search, OpenSearch indexing	Search for an object in the admin toolbar
Messenger	Queue processing, all transports	Check `bin/console messenger:stats`
Cache	Redis connectivity, invalidation	Publish content, verify change visible on other pod
Full page cache	Anonymous page caching, invalidation	Load page, check response headers for cache hit
ERP integration	Import processing, field mapping	Run a test import
Permissions	RBAC, workflow permissions	Login as restricted user, verify access
Custom JS	Grid operators, admin plugins	Open grid view, check custom columns work

Performance Benchmarks

Metric	Acceptable	Warning	Critical
Admin page load	< 2s	2-5s	> 5s
DataObject open	< 3s	3-8s	> 8s
Asset upload (10MB)	< 5s	5-15s	> 15s
Search response	< 500ms	500ms-2s	> 2s
Frontend page (cached)	< 50ms	50-200ms	> 200ms
Frontend page (uncached)	< 500ms	500ms-2s	> 2s
Thumbnail render (remote storage)	< 200ms	200ms-1s	> 1s (check getDimensions bug)

Common Pitfalls

Skipping Pimcore 11. You cannot jump from 10 to 12. The intermediate version handles critical schema changes, bundle extraction, and prefix removal.
Running migrations without OpenSearch indices. Some migrations trigger event subscribers that query OpenSearch. index_not_found_exception cascades into corrupted SAVEPOINT state. Create indices first.
Not fixing collations. Mixed collations cause UNION query failures when opening any DataObject. This breaks the admin UI silently until an editor opens a product.
Losing the encryption secret. If lost after deployment, all encrypted data in the database becomes unrecoverable. This is not recoverable. Treat it like the master database password.
Silent Redis cache fallback. The adapter name change from pimcore.cache.adapter.redis_tag_aware to cache.adapter.redis_tag_aware doesn't throw errors. Caching silently falls back to filesystem. Multi-pod deployments break.
Not updating Messenger transports. Missing P11 transports cause scheduled tasks, asset processing, and search indexing to silently stop working.
Running collation migration with web pods up. Translation table conversion deadlocks under concurrent access. Scale down first.
Manually creating Portal Engine statistics indices. The auto-creation uses time-based index names with aliases. Manual creation conflicts with the alias pattern.
Testing on empty database. Schema migration on an empty database hides 90% of issues. Use a full production data clone.
Ignoring the getDimensions() bug on cloud storage. Every uncached page render is 3-7 seconds slower than necessary until this is patched. The fix is a two-line reorder.
Missing getSupportedTypes() on normalizers. Symfony 7.x requires this method. Missing it causes a fatal error, not a deprecation warning.
Not updating Request::get() calls. Deprecated but still works in Symfony 6.x. Will break in 7.x.
Forgetting PIMCORE_CLASS_DEFINITION_WRITABLE=1. Migrations and class rebuilds silently fail without this env var. The error message is clear, but it's easy to forget.
ExtJS 7 root vs rootProperty. Custom store readers using root: 'data' silently fail to parse responses in ExtJS 7.

Key Takeaways

Three phases, no shortcuts. Preparation (fix all deprecated code), Pimcore 11 (framework upgrade + bundle extraction), Pimcore 12 (registration, DBAL 4, collation, event system). Each phase is a checkpoint.
Collation migration is the hardest part. Five types of special tables need custom handling. Budget a maintenance window. Use an automated script. Verify at both table and column level.
Product registration is mandatory and offline. No internet needed for validation. But the encryption secret is irrecoverable if lost.
Redis is not optional in multi-pod deployments. Cache consistency, session sharing, full page cache, and optionally message transport all depend on it. The adapter name change is a silent breaking change.
OpenSearch indices must exist before migrations. Create document and asset indices manually, then run data object reindex. The order matters.
The event system migration is mechanical but critical. Plugin Broker to Event Listeners, o_className to className, @Route to #[Route], Request::get() to specific accessors. Rector can automate 80% of it.
Remote storage exposes Pimcore bugs. The getDimensions() ordering bug adds seconds to every page render. The folder preview cache bug makes thumbnails permanently stuck. Both have simple fixes.
Test on a production data clone. Every time. An empty database upgrade takes 20 minutes and succeeds trivially. Real issues appear with 70K assets, 680 tables with mixed collations, and 50K+ data objects.

This is exactly the kind of software engineering work our team handles regularly. Full guide: https://oronts.com/en/guides/pimcore-upgrade-10-to-12

Modern TypeScript Backends: Hono, ElysiaJS, and What Comes After NestJS

refaat Al Ktifan — Fri, 01 May 2026 09:00:10 +0000

Why We Stopped Starting New Projects with NestJS

NestJS is a great framework. We've built enterprise systems on it (Vendure is NestJS under the hood, and we love Vendure). But for new standalone backend services, we've moved to Hono. Not because NestJS is bad. Because Hono is better for the way we work now.

The shift happened gradually. We started a new API service, considered NestJS, and asked ourselves: do we need decorators, modules, providers, and the Angular-inspired dependency injection container for a 15-endpoint API? The answer was no. We needed a fast HTTP framework that runs on any runtime, has good TypeScript support, and doesn't prescribe an architecture.

Hono delivered that. Then we tried ElysiaJS on Bun for a performance-critical service. That delivered something different. This article is an honest comparison from someone who runs all three in production.

For broader context on how we approach software engineering, that guide covers our principles. For how these frameworks fit into commerce architecture, see our Vendure production guide.

The Three Frameworks

Feature	NestJS	Hono	ElysiaJS
Runtime	Node.js	Node, Bun, Deno, Cloudflare Workers, Lambda, Vercel	Bun (primarily)
Architecture	Opinionated (modules, controllers, providers)	Minimal (routes, middleware)	Minimal (routes, plugins)
Type safety	Decorators + runtime validation	Middleware-based, manual	Built-in validation (typebox)
DI container	Built-in (Angular-style)	None (bring your own)	None (plugins)
Bundle size	Large (~50MB node_modules)	Tiny (~14KB core)	Small (~2MB with Bun)
Cold start	Slow (2-5s on Lambda)	Fast (< 100ms on Lambda)	Fast (< 50ms on Bun)
Middleware ecosystem	Large (passport, class-validator, etc.)	Growing (auth, cors, jwt, etc.)	Growing (swagger, jwt, etc.)
Learning curve	Steep (modules, decorators, DI, guards, pipes)	Minimal (express-like)	Minimal (express-like with validation)
Best for	Large enterprise apps, Vendure plugins	Multi-runtime APIs, edge functions, microservices	Bun-native high-performance APIs

Hono: The Framework That Runs Everywhere

Hono is a web framework built for the edge. It runs on Node.js, Bun, Deno, Cloudflare Workers, AWS Lambda, Vercel Edge, and Fastly. The same code deploys to any runtime without changes.

import { Hono } from 'hono';
import { cors } from 'hono/cors';
import { jwt } from 'hono/jwt';
import { logger } from 'hono/logger';

const app = new Hono();

// Middleware
app.use('*', logger());
app.use('*', cors({ origin: ['https://app.example.com'] }));
app.use('/api/*', jwt({ secret: process.env.JWT_SECRET! }));

// Routes
app.get('/api/products', async (c) => {
    const products = await productService.findAll();
    return c.json({ data: products });
});

app.get('/api/products/:id', async (c) => {
    const product = await productService.findById(c.req.param('id'));
    if (!product) return c.json({ error: 'Not found' }, 404);
    return c.json({ data: product });
});

app.post('/api/products', async (c) => {
    const body = await c.req.json();
    const product = await productService.create(body);
    return c.json({ data: product }, 201);
});

export default app;

Why We Like It

Runtime portability. We deploy the same Hono codebase to Lambda (API Gateway), Bun (container), and Cloudflare Workers (edge). No code changes between runtimes. This is Hono's unique strength. No other framework can do this.

14KB core. Cold starts on Lambda are under 100ms. NestJS cold starts are 2-5 seconds. For serverless deployments, this difference is the entire user experience.

No opinions on architecture. Hono gives you routing, middleware, and context. You decide how to structure services, repositories, and business logic. For small to medium APIs (5-50 endpoints), this is exactly right. You don't need a module system for 20 routes.

tRPC integration. We use Hono + tRPC for TypeScript-only stacks. End-to-end type safety from server to client without GraphQL:

import { trpcServer } from '@hono/trpc-server';
import { appRouter } from './trpc/router';

app.use('/trpc/*', trpcServer({ router: appRouter }));

Where It Falls Short

No built-in DI. For large applications with 50+ services and complex dependency graphs, you need to bring your own DI container (tsyringe, inversify) or wire dependencies manually. This is fine for small services. It becomes tedious for large ones.

Smaller middleware ecosystem. NestJS has passport, class-validator, class-transformer, swagger, and hundreds of community packages. Hono's ecosystem is growing but not there yet. You'll write more custom middleware.

No conventions for large apps. NestJS modules give you a structure that scales to 100+ files. Hono doesn't prescribe structure. For teams with junior developers, the lack of conventions can lead to inconsistent codebases.

ElysiaJS: When Bun Performance Matters

ElysiaJS is built specifically for Bun. It takes advantage of Bun's native APIs, FFI bindings, and optimized HTTP server to achieve performance that Node.js frameworks can't match.

import { Elysia, t } from 'elysia';
import { swagger } from '@elysiajs/swagger';
import { jwt } from '@elysiajs/jwt';

const app = new Elysia()
    .use(swagger())
    .use(jwt({ name: 'jwt', secret: process.env.JWT_SECRET! }))
    .get('/products', async () => {
        return productService.findAll();
    })
    .post('/products', async ({ body }) => {
        return productService.create(body);
    }, {
        body: t.Object({
            name: t.String(),
            price: t.Number({ minimum: 0 }),
            description: t.Optional(t.String()),
        }),
    })
    .listen(3000);

Why It's Interesting

Built-in validation with TypeBox. Schemas defined inline, automatically generate OpenAPI docs. No separate validation library needed. The types flow from schema to handler to response.

Bun-native performance. ElysiaJS on Bun handles 2-3x more requests per second than Express on Node.js for simple JSON APIs. For compute-light, I/O-heavy workloads (most APIs), the difference is less dramatic but still measurable.

End-to-end type inference. The request body type is inferred from the TypeBox schema. The response type is inferred from the handler. The client can use Eden (ElysiaJS's tRPC-like client) for type-safe API calls.

Where It Falls Short

Bun-only. If you need to deploy to Node.js, Lambda, or Cloudflare Workers, ElysiaJS is not an option. Bun is production-ready but not universally supported in all hosting environments.

Smaller ecosystem. Bun's package compatibility is good but not 100%. Some Node.js packages use native addons that Bun doesn't support. Database drivers, image processing libraries, and native crypto modules can be problematic.

Less battle-tested. Bun and ElysiaJS are newer. The community is smaller. When you hit an edge case, there are fewer Stack Overflow answers and fewer people who've solved the same problem.

Drizzle vs Prisma vs TypeORM: The ORM Reality

The framework choice is half the decision. The ORM choice is the other half.

Feature	Drizzle	Prisma	TypeORM
Query style	SQL-like (explicit)	Client API (generated)	Active Record or Data Mapper
Type safety	Excellent (inferred from schema)	Excellent (generated client)	Good (decorators)
Migration	SQL-based, manual control	Auto-generated, managed	Auto-generated or manual
Performance	Fast (thin wrapper over SQL)	Good (query engine)	Moderate (heavy abstraction)
Raw SQL	First-class (sql`` template)	Possible but awkward	Possible via query builder
Edge/Serverless	Works (lightweight)	Needs Prisma Accelerate for edge	Too heavy for edge
Learning curve	Low (if you know SQL)	Low (API docs are excellent)	Medium (decorators, relations)
Bundle size	Small (~500KB)	Large (~15MB with engine)	Medium (~5MB)
Best with	Hono, ElysiaJS	Any framework	NestJS, Vendure

Our Stack Choices

Hono + Drizzle + tRPC: For new TypeScript APIs. Lightweight, fast, type-safe end-to-end. Drizzle's SQL-like syntax gives us explicit control over queries without the abstraction overhead of TypeORM or the engine overhead of Prisma.

`typescript // Drizzle: SQL-like, explicit, type-safe const products = await db .select() .from(productsTable) .where(and( eq(productsTable.tenantId, tenantId), eq(productsTable.status, 'active'), gt(productsTable.price, 1000), )) .orderBy(desc(productsTable.createdAt)) .limit(20); `

NestJS + TypeORM: For Vendure plugins and large enterprise applications where the module system and DI container justify their weight. TypeORM integrates deeply with NestJS and Vendure.

Prisma: When we need auto-generated migrations and the team values Prisma's documentation and developer experience over raw query control.

Deployment Flexibility

This is where the framework choice has the biggest practical impact.

Deployment Target	NestJS	Hono	ElysiaJS
Docker/Kubernetes	Works	Works	Works (Bun image)
AWS Lambda	Slow cold starts (2-5s)	Fast (< 100ms)	Not supported
Cloudflare Workers	Not supported	Works natively	Not supported
Vercel Edge	Not supported	Works natively	Not supported
Deno Deploy	Not supported	Works natively	Not supported
Traditional VPS	Works	Works	Works (needs Bun)
Bun native	Partial (most NestJS works on Bun)	Works	Native, optimized

If you deploy to containers only, all three work fine. If you deploy to serverless or edge, Hono is the only framework that gives you every option. ElysiaJS is Bun-only. NestJS is Node-only (practically).

For how we deploy services to cloud infrastructure including Kubernetes, Lambda, and edge functions, that page covers our approach.

When NestJS Is Still the Right Choice

Despite our preference for Hono on new projects, NestJS is the right choice when:

You're building Vendure plugins. Vendure IS NestJS. The plugin system, services, resolvers, and guards all use NestJS patterns. Fighting the framework is worse than using it.
Your team knows Angular. NestJS's module/provider/controller pattern mirrors Angular. If your team comes from Angular, the learning curve is near zero.
You need a large middleware ecosystem. Passport strategies, class-validator, Swagger auto-generation, GraphQL code-first. NestJS has it all built or well-integrated.
The application will grow to 100+ files. NestJS modules provide structure that scales. Without conventions (Hono, ElysiaJS), large teams need discipline to maintain consistency.
You need WebSocket support. NestJS has built-in WebSocket gateways with the same decorator pattern. Hono's WebSocket support is more manual.

Common Pitfalls

Choosing Hono and then rebuilding NestJS. If you end up adding a DI container, decorator-based routing, module system, and guard pattern to Hono, you've just built a worse NestJS. Use NestJS.
Choosing ElysiaJS for production before evaluating Bun compatibility. Check that all your dependencies work on Bun. Native addons, specific database drivers, and some crypto libraries may not.
Ignoring cold start times on serverless. NestJS on Lambda is unusable for user-facing APIs without provisioned concurrency. Hono on Lambda is fast enough without it.
Using TypeORM with Hono. TypeORM is designed for NestJS's DI patterns. Without DI, managing TypeORM connections and repositories is awkward. Use Drizzle or Prisma with Hono.
Premature framework switching. If your NestJS app works and your team is productive, don't switch to Hono for the sake of being modern. Switch when you have a concrete reason (serverless deployment, edge functions, new standalone service).
No validation layer. Hono doesn't include request validation. Use Zod middleware or build your own. Shipping an API without input validation is a security vulnerability.

Key Takeaways

Hono is the most versatile TypeScript backend framework. 14KB core, runs on every runtime, fast cold starts, no opinions on architecture. Our default for new standalone APIs.
ElysiaJS delivers the best performance on Bun. Built-in validation, great DX, type-safe end-to-end. But Bun-only limits where you can deploy it.
NestJS is still the right choice for large enterprise apps and Vendure. Modules, DI, guards, and a massive ecosystem. The learning curve pays off at scale.
Drizzle is our preferred ORM with Hono. SQL-like syntax, explicit queries, lightweight bundle. Prisma for teams that prefer auto-generated clients. TypeORM for NestJS/Vendure.
The deployment target drives the framework choice. Serverless and edge need Hono. Containers work with anything. Bun-native needs ElysiaJS.
Don't switch frameworks without a reason. If NestJS is working, keep using it. Switch when you need serverless, edge, or a new standalone service where the lightweight approach is genuinely better.

OpenTelemetry in Production: Traces, Context, and What Actually Matters

refaat Al Ktifan — Tue, 28 Apr 2026 17:48:50 +0000

Why OpenTelemetry Won

Three years ago, the observability landscape was fragmented. Jaeger for tracing, Prometheus for metrics, Fluentd for logs, each with its own SDK, its own protocol, its own vendor lock-in. OpenTelemetry unified them into a single standard: one SDK, one protocol (OTLP), one collector that routes to any backend.

We adopted OpenTelemetry across our production systems. This article covers the patterns that actually matter in production, not the setup tutorial. For the broader observability strategy (when to alert, what to log, how to structure metrics), see our AI observability guide. This article goes deeper on OpenTelemetry-specific implementation.

Context Propagation: The Hardest Part

A single user request might cross 5 services, 3 message queues, 2 worker processes, and an LLM call. Context propagation ensures that the trace follows the request across all of them.

HTTP Propagation (Easy)

OpenTelemetry auto-instruments HTTP clients and servers. The trace context propagates via traceparent and tracestate headers. This works out of the box.

// Auto-instrumented: no code needed for HTTP propagation
// The SDK adds traceparent header to outgoing requests
// The receiving service extracts it and continues the trace

Queue Propagation (Hard)

Message queues break automatic propagation. When you enqueue a message, the trace context must be serialized into the message headers. When a worker dequeues it, the context must be extracted and the trace continued.

// Producer: inject trace context into message headers
import { context, propagation } from '@opentelemetry/api';

async function enqueueMessage(queue: string, payload: any) {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);

    await messageQueue.send(queue, {
        body: payload,
        headers: carrier, // Contains traceparent, tracestate
    });
}

// Consumer: extract trace context from message headers
async function processMessage(message: QueueMessage) {
    const parentContext = propagation.extract(context.active(), message.headers);

    await context.with(parentContext, async () => {
        const span = tracer.startSpan('process_message', {
            attributes: {
                'messaging.system': 'rabbitmq',
                'messaging.operation': 'process',
                'messaging.destination': message.queue,
            },
        });

        try {
            await handleMessage(message.body);
            span.setStatus({ code: SpanStatusCode.OK });
        } catch (error) {
            span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
            throw error;
        } finally {
            span.end();
        }
    });
}

This pattern works for RabbitMQ, Kafka, BullMQ, SQS, and Symfony Messenger. The message headers carry the trace context. The consumer extracts it and creates child spans under the original trace.

Worker Process Propagation

For Vendure's BullMQ workers, Pimcore's Symfony Messenger workers, and similar background job systems, the pattern is the same: serialize context into the job payload, extract on the worker side.

// BullMQ: add trace context to job data
async function addJob(queue: Queue, data: any) {
    const carrier: Record<string, string> = {};
    propagation.inject(context.active(), carrier);

    await queue.add('process', {
        ...data,
        _traceContext: carrier,
    });
}

// BullMQ: extract trace context in worker
worker.on('process', async (job) => {
    const parentContext = propagation.extract(context.active(), job.data._traceContext || {});

    await context.with(parentContext, async () => {
        const span = tracer.startSpan(`job:${job.name}`);
        try {
            await processJob(job.data);
        } finally {
            span.end();
        }
    });
});

Tracing LLM Calls

LLM calls are the most expensive operations in AI systems. Tracing them with proper attributes enables cost tracking, latency analysis, and quality monitoring.

async function tracedLlmCall(prompt: string, options: LlmOptions): Promise<string> {
    const span = tracer.startSpan('llm.generate', {
        attributes: {
            'llm.provider': options.provider,           // "openai", "anthropic"
            'llm.model': options.model,                 // "gpt-4o", "claude-sonnet-4-20250514"
            'llm.temperature': options.temperature,
            'llm.max_tokens': options.maxTokens,
            'llm.prompt_tokens': estimateTokens(prompt), // estimate before call
        },
    });

    try {
        const response = await llmClient.generate(prompt, options);

        span.setAttributes({
            'llm.response_tokens': response.usage.completionTokens,
            'llm.total_tokens': response.usage.totalTokens,
            'llm.finish_reason': response.finishReason,
            'llm.cost_usd': calculateCost(response.usage, options.model),
        });

        span.setStatus({ code: SpanStatusCode.OK });
        return response.text;
    } catch (error) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
        span.setAttribute('llm.error_type', error.constructor.name);
        throw error;
    } finally {
        span.end();
    }
}

Do NOT put prompt text in span attributes. Prompts contain PII. Span attributes are sent to your observability backend (Jaeger, Grafana Tempo, Datadog). Instead, log the prompt hash or token count. For the full PII-safe logging architecture, see our AI data leakage guide.

LLM Span Attributes Convention

Attribute	Type	Example
`llm.provider`	string	"openai"
`llm.model`	string	"gpt-4o"
`llm.temperature`	float	0.7
`llm.prompt_tokens`	int	1250
`llm.response_tokens`	int	340
`llm.total_tokens`	int	1590
`llm.finish_reason`	string	"stop"
`llm.cost_usd`	float	0.023
`llm.error_type`	string	"RateLimitError"
`llm.cache_hit`	boolean	false

Sampling Strategies

In high-volume AI systems, tracing every request is too expensive. Sampling reduces volume while preserving visibility into important traces.

Head-Based Sampling

Decide at the start of the trace whether to sample it. Simple but lossy.

// Sample 10% of all traces
const sampler = new TraceIdRatioBased(0.1);

// Always sample errors (override ratio for error traces)
const compositeSampler = new ParentBasedSampler({
    root: new TraceIdRatioBased(0.1),
    // Errors always sampled via span processor
});

Tail-Based Sampling (Recommended for AI)

Decide after the trace completes whether to keep it. Keeps all interesting traces (errors, slow responses, high cost) and drops routine ones.

// OpenTelemetry Collector: tail-based sampling config
processors:
    tail_sampling:
        decision_wait: 10s
        policies:
            # Keep all errors
            - name: errors
              type: status_code
              status_code: { status_codes: [ERROR] }

            # Keep slow traces (> 5 seconds)
            - name: slow
              type: latency
              latency: { threshold_ms: 5000 }

            # Keep expensive LLM calls (> $0.10)
            - name: expensive_llm
              type: string_attribute
              string_attribute:
                  key: llm.cost_usd
                  values: []  # Custom: filter in pipeline
                  enabled_regex_matching: true

            # Sample 5% of everything else
            - name: baseline
              type: probabilistic
              probabilistic: { sampling_percentage: 5 }

Tail-based sampling requires the OpenTelemetry Collector. The collector buffers complete traces, evaluates policies, and forwards only sampled traces to the backend. This adds latency (the decision_wait period) but dramatically reduces storage costs while keeping all interesting data.

Privacy-Safe Spans

Span attributes, span names, and span events are all sent to your observability backend. If any of these contain PII, your tracing infrastructure becomes a data protection liability.

// BAD: PII in span attributes
span.setAttribute('user.email', 'sara.mustermann@beispiel.de');
span.setAttribute('user.name', 'Sara Mustermann');
span.setAttribute('request.body', JSON.stringify(requestBody)); // Contains PII

// GOOD: token IDs and aggregate data only
span.setAttribute('user.id', 'usr_abc123');  // Opaque ID, not PII
span.setAttribute('entities.detected', 3);
span.setAttribute('entities.types', ['person', 'email', 'phone']);
span.setAttribute('policy.applied', 'german-support');

Rules for privacy-safe tracing:

User IDs: opaque identifiers only (not emails, not names)
Request bodies: never include raw content. Log entity counts and types.
LLM prompts: never include. Log token counts and prompt hash.
Error messages: sanitize before attaching to spans. Strip any user data.

The Baggage API

OpenTelemetry Baggage carries key-value pairs across service boundaries. Unlike span attributes (which stay on the span), baggage propagates to all downstream services automatically.

import { propagation, context, baggage } from '@opentelemetry/api';

// Set baggage at the API gateway
const bag = propagation.createBaggage({
    'tenant.id': { value: 'tenant_acme' },
    'request.priority': { value: 'high' },
    'feature.flags': { value: 'new-checkout,beta-search' },
});
const ctx = propagation.setBaggage(context.active(), bag);

// Downstream services can read baggage
const tenantId = propagation.getBaggage(context.active())?.getEntry('tenant.id')?.value;

Useful for:

Tenant ID propagation (every downstream service knows which tenant)
Feature flags (propagate experiment assignments across services)
Priority routing (high-priority requests get different queue treatment)
Debug markers (mark specific requests for verbose logging)

Baggage travels with the trace context in HTTP headers and message metadata. Every service that extracts the trace context also gets the baggage.

Collector Architecture

The OpenTelemetry Collector is the central routing layer between your applications and your observability backends.

┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│  Service A   │  │  Service B   │  │  Worker C    │
│  (OTLP gRPC) │  │  (OTLP HTTP) │  │  (OTLP gRPC) │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       ▼                 ▼                 ▼
┌─────────────────────────────────────────────────┐
│              OTel Collector                      │
│                                                  │
│  Receivers: OTLP (gRPC + HTTP)                  │
│  Processors: batch, tail_sampling, attributes    │
│  Exporters: Tempo, Prometheus, Loki             │
└─────────────────────────────────────────────────┘
       │                 │                 │
       ▼                 ▼                 ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│  Grafana     │ │  Prometheus  │ │  Grafana     │
│  Tempo       │ │              │ │  Loki        │
│  (traces)    │ │  (metrics)   │ │  (logs)      │
└──────────────┘ └──────────────┘ └──────────────┘

The collector handles batching (reduces network calls), sampling (reduces storage), attribute processing (adds/removes attributes), and routing (different signals to different backends). Deploy it as a sidecar or as a central service depending on your infrastructure.

For cloud deployment patterns including observability infrastructure, that page covers our approach.

Common Pitfalls

No context propagation across queues. HTTP propagation is automatic. Queue propagation is not. If you don't inject/extract context in message headers, traces break at every queue boundary.
PII in span attributes. Your tracing backend indexes everything. If spans contain emails, names, or request bodies, your Grafana Tempo cluster is a PII store.
Tracing every request in production. At 1000 RPS, full tracing generates terabytes of data. Use tail-based sampling to keep errors, slow traces, and expensive operations.
No LLM-specific attributes. Without token counts, cost, model ID, and finish reason on LLM spans, you can't track AI costs or diagnose quality issues.
Head-based sampling dropping errors. If you sample 10% of traces and an error happens in the 90% you drop, you never see it. Use tail-based sampling or always-sample-errors policies.
Baggage for large payloads. Baggage travels with every request. Large values increase header size on every HTTP call. Keep baggage values small (IDs, flags, priorities).

Key Takeaways

Context propagation across queues is the hardest part. HTTP is automatic. Queues require manual inject/extract of trace context in message headers. This is where most distributed tracing implementations break.
Trace LLM calls with cost and token attributes. Model, provider, token counts, cost, finish reason. These attributes enable AI cost dashboards and quality monitoring.
Tail-based sampling for AI workloads. Keep all errors, slow traces, and expensive operations. Drop routine traces. Reduces storage by 90%+ while keeping all interesting data.
No PII in spans. Opaque user IDs, entity counts, token types. Never raw content, emails, names, or request bodies.
Baggage propagates tenant context. Set tenant ID, feature flags, and priority at the edge. Every downstream service reads it from baggage without explicit parameter passing.

Event-Driven Architecture in Practice: What Actually Goes Wrong

refaat Al Ktifan — Sat, 25 Apr 2026 15:26:51 +0000

The Event Storm Nobody Expects

Event-driven architecture looks clean on a whiteboard. Service A emits an event. Service B consumes it. Loose coupling. Independent scaling. Beautiful diagrams.

Then you deploy to production. One product save triggers 8 event subscribers. Each subscriber dispatches 3 async messages. Each message handler loads the product, modifies a field, and saves. Each save triggers 8 subscribers again. Within seconds, your message queue has thousands of messages for a single product update, your workers are saturated, and your database is under lock contention.

We've operated four different event-driven systems in production. Each used a different message broker. Each had different failure patterns. This article covers what actually goes wrong and how to fix it. For broader context on how we approach system architecture, that guide covers our methodology.

The Four Failure Patterns

Every event-driven system eventually encounters these:

Pattern	What Happens	Severity
Event storm	One action cascades into thousands of messages	Critical (can saturate infrastructure)
Bidirectional sync loop	System A syncs to B, B syncs back to A, infinite loop	Critical (infinite message growth)
Duplicate processing	Same message processed twice, creates duplicate data	High (data integrity)
Dead letter accumulation	Failed messages pile up with no resolution strategy	Medium (operational debt)

1. Event Storms

The most common and most dangerous pattern. It happens when event handlers trigger new events, and those events trigger more handlers.

Product save
  -> EventSubscriber A dispatches MessageA
  -> EventSubscriber B dispatches MessageB
  -> EventSubscriber C dispatches MessageC
  -> ...8 subscribers total

WorkerA processes MessageA
  -> Modifies product, calls save()
  -> Product save triggers 8 subscribers again
  -> Each dispatches more messages

WorkerB processes MessageB (same pattern)
  -> 8 more messages

Result: 1 save -> 8 messages -> 8 saves -> 64 messages -> ...

The solution: EventSubscriberSupervisor. A process-scoped control that disables event subscribers during worker saves.

// Worker handler: disable subscribers before saving
async function handleGenerateThumbnail(message: ThumbnailMessage) {
    const product = await productService.findById(message.productId);

    subscriberSupervisor.disableAll();
    try {
        product.thumbnail = await generateThumbnail(product);
        await product.save(); // No subscribers fire, no new messages dispatched
    } finally {
        subscriberSupervisor.enableAll();
    }
}

The order matters. The save must happen inside the disabled scope. If you re-enable subscribers before saving, the save triggers the cascade.

For a deeper look at how we apply this pattern specifically in Pimcore, see our Pimcore workflow design guide.

2. Bidirectional Sync Loops

When two systems sync data bidirectionally, each update from A triggers a sync to B, which triggers a sync back to A. Without loop prevention, this runs forever.

System A (Commerce)         System B (Operations)
     │                           │
     │  Customer updated         │
     │ ────────────────────────▶ │
     │                           │  Sync received, save customer
     │                           │  Customer updated event fires
     │  Sync received            │
     │ ◀──────────────────────── │
     │  Save customer            │
     │  Customer updated event   │
     │ ────────────────────────▶ │
     │           ...forever...   │

The solution: source tracking. Every message carries an x-source header that identifies the originating system. When a system receives a message from itself (via the other system), it ignores it.

// When publishing a sync message
async function publishCustomerSync(customer: Customer, source: string) {
    await messageQueue.publish('customer.updated', {
        customerId: customer.id,
        data: customer,
        source: source,  // "commerce" or "operations"
    });
}

// When consuming a sync message
async function handleCustomerSync(message: CustomerSyncMessage) {
    // Ignore messages that originated from this system
    if (message.source === THIS_SYSTEM_ID) {
        logger.debug('Ignoring self-originated sync', { customerId: message.customerId });
        return; // ACK the message, don't process
    }

    const customer = await customerService.update(message.customerId, message.data);

    // When saving, mark the source so downstream events carry it
    await publishCustomerSync(customer, THIS_SYSTEM_ID);
}

An alternative approach: message deduplication by content hash. Hash the message payload and check if the same hash was processed recently. If the data hasn't changed, the sync is a no-op.

3. Duplicate Processing

Network failures, worker restarts, and at-least-once delivery guarantees mean the same message can be delivered more than once. Without idempotency, you get duplicate orders, duplicate emails, duplicate database records.

The solution: idempotency stores with business-key deduplication.

// Idempotency store with business key
class IdempotencyStore {
    async checkAndAcquire(key: string, scope: string): Promise<boolean> {
        try {
            await this.db.insert('idempotency_keys', {
                key,
                scope,
                status: 'PROCESSING',
                created_at: new Date(),
                expires_at: new Date(Date.now() + 24 * 60 * 60 * 1000), // 24h TTL
            });
            return true; // Acquired, proceed with processing
        } catch (error) {
            if (isDuplicateKeyError(error)) {
                return false; // Already processed, skip
            }
            throw error;
        }
    }

    async complete(key: string, scope: string): Promise<void> {
        await this.db.update('idempotency_keys',
            { status: 'COMPLETED', completed_at: new Date() },
            { key, scope }
        );
    }
}

// Usage in a worker
async function handleNotification(message: NotificationMessage) {
    const dedupeKey = `${message.recipientEmail}:${message.category}:${message.entityRef}:${todayBucket()}`;

    const acquired = await idempotencyStore.checkAndAcquire(dedupeKey, 'notification');
    if (!acquired) {
        logger.info('Duplicate notification skipped', { dedupeKey });
        return; // ACK the message
    }

    try {
        await emailService.send(message);
        await idempotencyStore.complete(dedupeKey, 'notification');
    } catch (error) {
        await idempotencyStore.fail(dedupeKey, 'notification');
        throw error; // NACK, let the queue retry
    }
}

The dedupe key must be business-meaningful. For notifications: recipient + category + entity + day bucket (prevents sending the same notification twice in one day but allows it the next day). For orders: tenant + product + date. For imports: source record ID + import batch ID.

4. Dead Letter Handling

Messages that fail repeatedly end up in a dead letter queue. Most teams set up the DLQ and then ignore it. Dead letters accumulate. Months later, someone discovers thousands of unprocessed messages.

// Dead letter handler with classification
async function processDeadLetter(message: DeadLetterMessage) {
    const failureType = classifyFailure(message.error);

    switch (failureType) {
        case 'TRANSIENT':
            // Network timeout, service temporarily down
            // Requeue with exponential backoff
            await requeue(message, { delay: calculateBackoff(message.retryCount) });
            break;

        case 'PERMANENT':
            // Invalid data, schema mismatch, business rule violation
            // Log, alert, archive. Do not retry.
            await archiveDeadLetter(message);
            await alertOps('Permanent failure in dead letter', message);
            break;

        case 'POISON':
            // Message itself causes crashes (malformed payload)
            // Archive immediately, never retry
            await archiveDeadLetter(message);
            await alertOps('Poison message detected', message);
            break;
    }
}

function classifyFailure(error: string): 'TRANSIENT' | 'PERMANENT' | 'POISON' {
    if (error.includes('ECONNREFUSED') || error.includes('timeout')) return 'TRANSIENT';
    if (error.includes('SyntaxError') || error.includes('Cannot parse')) return 'POISON';
    return 'PERMANENT';
}

The key insight: not all dead letters are the same. Transient failures should be retried. Permanent failures should be archived and investigated. Poison messages should be quarantined immediately.

For how we handle failure patterns in AI systems specifically, see our AI failure modes guide.

Choosing a Message Broker

We've used four different brokers in production. Each has a distinct sweet spot.

Broker	Best For	Not Great For	Deployment Complexity
Kafka	High-throughput event streaming, event sourcing, replay	Simple job queues, low-volume systems	High (ZooKeeper/KRaft, partitions, consumer groups)
RabbitMQ	Routing, dead letters, priority queues, complex topologies	Very high throughput (millions/sec), event replay	Medium (clustering, HA policies)
BullMQ (Redis)	Job queues, delayed jobs, rate limiting, simple setups	Complex routing, message replay, multi-consumer patterns	Low (just Redis)
Symfony Messenger	PHP/Symfony apps, Pimcore, simple async dispatch	Non-PHP ecosystems, complex broker features	Low (uses Doctrine, AMQP, or Redis as transport)

When to Use Kafka

You need event replay (consumers can re-read from any offset)
You have multiple consumer groups processing the same events differently
Throughput exceeds 100K messages per second
You're doing event sourcing or building a change data capture pipeline
You need guaranteed ordering within a partition

When to Use RabbitMQ

You need complex routing (topic exchanges, headers-based routing)
Dead letter queues with automatic retry policies are important
Message priority matters (urgent messages processed first)
You need request-reply patterns (RPC over messages)
Your system has multiple services in different languages

When to Use BullMQ

You're in a Node.js/TypeScript ecosystem (Vendure, NestJS)
Job scheduling with delays and cron patterns
Rate limiting per queue or per job type
You already have Redis for caching and sessions
Your system is small to medium scale (under 10K messages/sec)

When to Use Symfony Messenger

You're in a PHP/Symfony/Pimcore ecosystem
You need simple async dispatch for background tasks
Worker infrastructure follows Symfony conventions (supervisord)
Transport flexibility (switch between Doctrine, AMQP, Redis without code changes)

For our Pimcore projects, we use Symfony Messenger with RabbitMQ transport. For Vendure projects, we use BullMQ. For high-throughput data ingestion platforms, we use Kafka. The broker choice follows the ecosystem, not the other way around.

Message Ordering

Most teams assume message ordering matters. Usually it doesn't.

Scenario	Ordering Needed?	Why
Send notification email	No	Emails are inherently unordered
Generate thumbnail	No	Latest version wins
Update search index	No	Latest state wins (eventual consistency)
Process payment steps	Yes	Charge must happen before capture
Order state transitions	Yes	Can't ship before payment confirmed
Event sourcing replay	Yes	Events must replay in causal order

When ordering matters, use FIFO queues (SQS FIFO, Kafka partitions keyed by entity ID, RabbitMQ with single consumer). When it doesn't, parallel processing is faster and simpler.

// Kafka: ordering within a partition (keyed by product ID)
await producer.send({
    topic: 'product-updates',
    messages: [{
        key: productId,  // All updates for same product go to same partition
        value: JSON.stringify(event),
    }],
});

// BullMQ: no ordering needed for thumbnails
await thumbnailQueue.add('generate', { productId }, {
    removeOnComplete: true,
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
});

Retry Strategies

Not all retries are equal. The strategy depends on the failure type.

Strategy	When to Use	Example
Immediate retry	Transient glitch (race condition)	Optimistic lock failure
Exponential backoff	Service temporarily down	External API timeout
Fixed delay	Rate limiting	API returns 429
No retry	Permanent failure	Invalid payload, business rule violation

// BullMQ retry configuration
const queue = new Queue('notifications', {
    defaultJobOptions: {
        attempts: 5,
        backoff: {
            type: 'exponential',
            delay: 60000, // 1min, 2min, 4min, 8min, 16min
        },
        removeOnComplete: { count: 1000 },
        removeOnFail: false, // Keep for dead letter analysis
    },
});

After all retries are exhausted, the message moves to a dead letter queue. Never retry infinitely. Set a maximum attempt count and handle the failure explicitly.

Monitoring Event-Driven Systems

The hardest part of event-driven systems is debugging. When something fails, the error might be 5 services and 12 messages away from the original cause.

Correlation IDs

Every message carries a correlation ID that links it to the original action:

// Original API request generates correlation ID
const correlationId = generateUUID();

// Every message in the chain carries it
await queue.add('process-order', {
    orderId: order.id,
    correlationId,
});

// Workers pass it to downstream messages
async function handleProcessOrder(job) {
    const { orderId, correlationId } = job.data;

    // ... process order ...

    // Downstream messages carry the same correlation ID
    await notificationQueue.add('send-confirmation', {
        orderId,
        correlationId, // Same ID, traceable back to the original request
    });
}

When debugging, query all logs and events by correlation ID to see the full chain. For more on distributed tracing patterns, see our AI observability guide.

Queue Health Metrics

Metric	Healthy	Warning	Critical
Queue depth	< 100	100-1000	> 1000
Processing time (p95)	< 5s	5-30s	> 30s
Failure rate	< 1%	1-5%	> 5%
Dead letter count	0	1-10	> 10
Consumer lag (Kafka)	< 1000	1K-10K	> 10K

Alert on queue depth trends, not absolute values. A queue depth of 500 that's been stable for hours is fine. A queue depth of 50 that was 0 an hour ago and is growing is a problem.

Common Pitfalls

No subscriber control during worker saves. The single biggest source of event storms. Worker saves must not trigger the same event subscribers that dispatch work to workers.
No source tracking on bidirectional sync. Without x-source headers, bidirectional sync between two systems loops forever.
Using message ID for idempotency. Message IDs change on redelivery in some brokers. Use business keys (entity ID + action + time bucket) instead.
Infinite retries. Set a maximum retry count. After exhaustion, move to dead letter queue. Never retry forever.
Ignoring dead letter queues. Dead letters are operational debt. Classify them (transient, permanent, poison) and handle each type differently.
Assuming message ordering. Most workloads don't need ordering. Parallel processing is faster. Only use FIFO when state transitions or causal ordering require it.
Same retry strategy for all failures. A timeout needs exponential backoff. An invalid payload needs zero retries. A rate limit needs fixed delay.
No correlation IDs. Without them, debugging a failure chain across 5 services is impossible.

Key Takeaways

Event storms are the most dangerous failure mode. One uncontrolled save cascade can saturate your entire infrastructure. The EventSubscriberSupervisor is not optional.
Bidirectional sync needs source tracking. Every message carries the originating system ID. When you receive a message from yourself (via the other system), ignore it.
Idempotency uses business keys, not message IDs. Recipient + category + entity + day bucket for notifications. Tenant + product + date for orders. The key must be business-meaningful.
Dead letters need classification. Transient failures get retried. Permanent failures get archived. Poison messages get quarantined. Never ignore the dead letter queue.
The broker follows the ecosystem. Kafka for high-throughput streaming, RabbitMQ for complex routing, BullMQ for Node.js job queues, Symfony Messenger for PHP. Don't pick a broker and then build around it.
Most workloads don't need ordering. Parallel processing is simpler and faster. Use FIFO only when state transitions demand it.

AI Browser Automation Without BrowserBase: What We Built Instead

refaat Al Ktifan — Sat, 25 Apr 2026 15:17:00 +0000

The Paid Browser Automation Market

BrowserBase, Browserless, and similar services charge per-minute or per-session for managed headless browsers. For AI workflows that need to interact with web pages (filling forms, extracting structured data, navigating multi-step processes), these services handle the infrastructure: browser instances, anti-detection, proxies, and session management.

The pricing adds up fast. At $0.10-0.50 per session-minute, a workflow that processes 1,000 pages per day at 2 minutes each costs $200-1,000 per day. For an AI system that runs continuously, that's $6,000-30,000 per month just for browser infrastructure.

We built a self-hosted alternative using Playwright + LLM for page understanding. It handles 90% of the use cases at a fraction of the cost. This article covers the architecture. For how we build AI workflow systems and agentic AI more broadly, those guides cover the higher-level patterns.

The Architecture

┌─────────────────────────────────────────────────────────┐
│                  AI Browser Engine                       │
│                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  │
│  │  Task Queue   │  │  Instance    │  │  Session      │  │
│  │  (BullMQ)     │  │  Pool        │  │  Manager      │  │
│  │               │  │  (Playwright │  │  (cookies,    │  │
│  │  Prioritized  │  │   browsers)  │  │   localStorage│  │
│  │  Retry logic  │  │              │  │   auth state) │  │
│  └──────┬───────┘  └──────┬───────┘  └──────┬───────┘  │
│         │                 │                  │           │
│         ▼                 ▼                  ▼           │
│  ┌──────────────────────────────────────────────────┐   │
│  │              Page Interaction Layer                │   │
│  │                                                    │   │
│  │  1. Navigate to URL                               │   │
│  │  2. Wait for page load                            │   │
│  │  3. Extract page structure (accessibility tree)   │   │
│  │  4. Send structure to LLM for understanding       │   │
│  │  5. LLM returns action plan (click, type, select) │   │
│  │  6. Execute actions via Playwright                │   │
│  │  7. Extract structured data from result           │   │
│  └──────────────────────────────────────────────────┘   │
│                                                          │
└─────────────────────────────────────────────────────────┘

Instance Pooling

Running a new browser for every task is expensive (cold start: 1-3 seconds, memory: 200-400MB per instance). A pool reuses browser instances across tasks.

class BrowserPool {
    private available: Browser[] = [];
    private inUse = new Map<string, Browser>();
    private maxInstances: number;

    constructor(options: { maxInstances: number }) {
        this.maxInstances = options.maxInstances;
    }

    async acquire(): Promise<{ browser: Browser; id: string }> {
        // Reuse an available instance
        if (this.available.length > 0) {
            const browser = this.available.pop()!;
            const id = crypto.randomUUID();
            this.inUse.set(id, browser);
            return { browser, id };
        }

        // Create new if under limit
        if (this.inUse.size < this.maxInstances) {
            const browser = await chromium.launch({
                headless: true,
                args: [
                    '--no-sandbox',
                    '--disable-setuid-sandbox',
                    '--disable-dev-shm-usage',
                    '--disable-gpu',
                    '--single-process',
                ],
            });
            const id = crypto.randomUUID();
            this.inUse.set(id, browser);
            return { browser, id };
        }

        // Pool exhausted: wait for one to be released
        return new Promise((resolve) => {
            this.waitQueue.push(resolve);
        });
    }

    async release(id: string): Promise<void> {
        const browser = this.inUse.get(id);
        if (!browser) return;

        this.inUse.delete(id);

        // Clear state between tasks
        const pages = browser.contexts();
        for (const context of pages) {
            await context.close();
        }

        // If someone is waiting, give them this instance
        if (this.waitQueue.length > 0) {
            const resolve = this.waitQueue.shift()!;
            const newId = crypto.randomUUID();
            this.inUse.set(newId, browser);
            resolve({ browser, id: newId });
        } else {
            this.available.push(browser);
        }
    }
}

Pool Sizing

Workload	Pool Size	Memory Required
Light (< 100 pages/hour)	2-3 instances	1-2 GB
Medium (100-500 pages/hour)	5-10 instances	3-5 GB
Heavy (500+ pages/hour)	10-20 instances	5-10 GB

Each Chromium instance uses 200-400MB of RAM. The pool size determines your throughput ceiling and memory requirements. Start small and scale based on actual load.

Session Management

Many workflows require maintaining login state across multiple page interactions. The session manager persists cookies, localStorage, and authentication tokens between tasks.

class SessionManager {
    private sessions = new Map<string, SessionState>();

    async createSession(id: string, options: SessionOptions): Promise<BrowserContext> {
        const context = await browser.newContext({
            viewport: { width: 1280, height: 720 },
            userAgent: options.userAgent || this.getRandomUserAgent(),
            locale: options.locale || 'en-US',
            timezoneId: options.timezone || 'Europe/Berlin',
        });

        // Restore previous session state if exists
        const existing = this.sessions.get(id);
        if (existing) {
            await context.addCookies(existing.cookies);
            // localStorage restored via page.evaluate after navigation
        }

        return context;
    }

    async saveSession(id: string, context: BrowserContext): Promise<void> {
        const cookies = await context.cookies();
        const pages = context.pages();
        let localStorage = {};

        if (pages.length > 0) {
            localStorage = await pages[0].evaluate(() => {
                const data: Record<string, string> = {};
                for (let i = 0; i < window.localStorage.length; i++) {
                    const key = window.localStorage.key(i);
                    if (key) data[key] = window.localStorage.getItem(key) || '';
                }
                return data;
            });
        }

        this.sessions.set(id, {
            cookies,
            localStorage,
            lastUsed: Date.now(),
        });
    }
}

LLM-Driven Page Understanding

The core innovation: instead of writing CSS selectors or XPath queries for every page, send the page's accessibility tree to an LLM and let it decide which elements to interact with.

async function extractPageStructure(page: Page): Promise<string> {
    // Get the accessibility tree (structured, compact representation)
    const tree = await page.accessibility.snapshot();

    // Convert to a text format the LLM can understand
    return formatAccessibilityTree(tree, {
        maxDepth: 5,
        includeRoles: ['button', 'link', 'textbox', 'combobox', 'checkbox', 'heading'],
        includeText: true,
        includeLabels: true,
    });
}

function formatAccessibilityTree(node: any, options: any, depth = 0): string {
    if (depth > options.maxDepth) return '';
    if (!options.includeRoles.includes(node.role) && depth > 1) {
        // Skip non-interactive elements, but recurse into children
        return (node.children || []).map(c => formatAccessibilityTree(c, options, depth + 1)).join('');
    }

    const indent = '  '.repeat(depth);
    let result = `${indent}[${node.role}] ${node.name || ''}`;
    if (node.value) result += ` value="${node.value}"`;
    result += '\n';

    for (const child of node.children || []) {
        result += formatAccessibilityTree(child, options, depth + 1);
    }
    return result;
}

LLM Action Planning

Send the page structure to the LLM with the task description. The LLM returns a sequence of actions:

async function planActions(pageStructure: string, task: string): Promise<Action[]> {
    const response = await llm.generate({
        model: 'gpt-4o-mini', // Fast model for action planning
        messages: [
            {
                role: 'system',
                content: `You are a browser automation assistant. Given a page structure and a task,
                return a JSON array of actions to accomplish the task.
                Available actions: click(selector), type(selector, text), select(selector, value),
                wait(ms), extract(selector).
                Use the element text/labels to identify targets, not CSS selectors.`,
            },
            {
                role: 'user',
                content: `Page structure:\n${pageStructure}\n\nTask: ${task}`,
            },
        ],
        responseFormat: 'json',
    });

    return JSON.parse(response.text);
}

// Example task: "Fill in the contact form with name Sara Mustermann and email sara.mustermann@beispiel.de"
// LLM returns:
// [
//   { "action": "type", "target": "Name input field", "value": "Sara Mustermann" },
//   { "action": "type", "target": "Email input field", "value": "sara.mustermann@beispiel.de" },
//   { "action": "click", "target": "Submit button" }
// ]

Resolving LLM Actions to Playwright Commands

The LLM returns human-readable targets ("Name input field"). A resolver maps them to Playwright selectors:

async function resolveAndExecute(page: Page, actions: Action[]): Promise<void> {
    for (const action of actions) {
        // Find the element matching the LLM's description
        const element = await findElementByDescription(page, action.target);

        if (!element) {
            throw new ActionError(`Could not find element: ${action.target}`);
        }

        switch (action.action) {
            case 'click':
                await element.click();
                await page.waitForLoadState('networkidle');
                break;
            case 'type':
                await element.fill(action.value);
                break;
            case 'select':
                await element.selectOption(action.value);
                break;
            case 'wait':
                await page.waitForTimeout(action.value);
                break;
            case 'extract':
                const text = await element.textContent();
                results.push({ field: action.target, value: text });
                break;
        }
    }
}

async function findElementByDescription(page: Page, description: string): Promise<ElementHandle | null> {
    // Try multiple strategies to find the element
    const strategies = [
        // By aria-label
        () => page.$(`[aria-label*="${description}" i]`),
        // By placeholder
        () => page.$(`[placeholder*="${description}" i]`),
        // By visible text
        () => page.$(`text=${description}`),
        // By label association
        () => page.$(`label:has-text("${description}") + input, label:has-text("${description}") input`),
        // By role and name
        () => page.getByRole('textbox', { name: new RegExp(description, 'i') }).first().elementHandle(),
        () => page.getByRole('button', { name: new RegExp(description, 'i') }).first().elementHandle(),
    ];

    for (const strategy of strategies) {
        try {
            const element = await strategy();
            if (element) return element;
        } catch {
            continue;
        }
    }

    return null;
}

Anti-Detection Basics

Some websites detect and block headless browsers. Basic countermeasures:

const context = await browser.newContext({
    // Randomize viewport
    viewport: {
        width: 1280 + Math.floor(Math.random() * 200),
        height: 720 + Math.floor(Math.random() * 100),
    },

    // Rotate user agents
    userAgent: getRandomUserAgent(),

    // Set realistic locale and timezone
    locale: 'de-DE',
    timezoneId: 'Europe/Berlin',

    // Realistic geolocation
    geolocation: { latitude: 48.1351, longitude: 11.5820 },
    permissions: ['geolocation'],
});

// Override navigator.webdriver (headless detection)
await page.addInitScript(() => {
    Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

Note: anti-detection is an arms race. For sites with sophisticated bot detection (Cloudflare, Akamai), self-hosted Playwright will eventually be detected. This is where paid services like BrowserBase add value: they invest continuously in anti-detection. For most business automation tasks (internal tools, partner portals, public data), basic anti-detection is sufficient.

When Paid Tools ARE Worth It

Scenario	Self-Hosted	Paid Service
Internal tool automation	Best choice (no anti-detection needed)	Overkill
Public data extraction (simple)	Good (basic anti-detection works)	Unnecessary
Sites with bot detection	Possible but constant maintenance	Worth it (they handle anti-detection)
High-volume scraping (10K+ pages/day)	Complex (proxy rotation, IP management)	Worth it (managed infrastructure)
Regulated data (GDPR, compliance)	Better (data stays on your infrastructure)	Risk (data goes through third party)
One-time migration	Good (temporary workload)	Unnecessary cost

The decision framework: if you're automating internal workflows or processing public data from sites without aggressive bot detection, self-host. If you're doing high-volume extraction from sites with Cloudflare-level protection, pay for a service that handles anti-detection as their core business.

Cost Comparison

Component	Self-Hosted (monthly)	BrowserBase (monthly)
Compute (5 instances)	$50-100 (container/VPS)	N/A
LLM calls (action planning)	$20-50 (GPT-4o-mini)	N/A
BrowserBase sessions	N/A	$500-2,000
Proxy service (if needed)	$50-200	Included
Maintenance	2-4 hours/month	None
Total (1,000 pages/day)	$120-350/month	$500-2,000/month
Total (10,000 pages/day)	$300-800/month	$3,000-10,000/month

Self-hosting is 3-10x cheaper at scale. The trade-off is maintenance time and anti-detection capability.

Common Pitfalls

No instance pooling. Launching a new browser per task wastes 1-3 seconds on cold start and 200-400MB of RAM. Pool and reuse instances.
Hardcoded CSS selectors. Pages change their DOM structure regularly. LLM-based element identification is more resilient than hardcoded selectors.
No session persistence. Multi-step workflows that require login fail when the session state is lost between steps.
Ignoring anti-detection entirely. Even basic measures (random viewport, user agent rotation, webdriver override) prevent detection on most sites.
Using a large model for action planning. GPT-4o-mini or Claude Haiku are fast enough for page understanding. A large model adds latency without better accuracy for this task.
No timeout on page loads. Some pages load indefinitely (infinite scrolling, slow third-party scripts). Set a navigation timeout and handle it.
Running in production without monitoring. Track success rate, average execution time, and error types per workflow. Alert when success rate drops.

Key Takeaways

Self-hosted Playwright + LLM handles 90% of browser automation use cases. For internal tools, partner portals, and public data without aggressive bot detection, this is the right approach.
Instance pooling is essential. Reuse browser instances across tasks. Cold starts and memory allocation are the biggest performance bottleneck.
LLM page understanding replaces brittle selectors. Send the accessibility tree to a fast model. Let it decide which elements to interact with. More resilient to page changes than hardcoded CSS selectors.
Paid services earn their cost on anti-detection. If your target sites have Cloudflare or similar protection, BrowserBase invests continuously in bypassing it. That's their core business. Don't try to compete.
Self-hosting is 3-10x cheaper at scale. But you pay in maintenance time and anti-detection limitations. Make the trade-off consciously.

FIND MORE: https://oronts.com/en/guides/browser-automation-ai-without-paid-tools

CodeRanedeer: An AI-Powered Programming Assistant

refaat Al Ktifan — Mon, 08 May 2023 19:14:30 +0000

Enhance Your Coding Experience with CodeRanedeer: An AI-Powered Programming Assistant (GPT 4 prompt)

AI Code Buddy - CodeRanedeer

Discover how CodeRanedeer, an AI-powered language model (GPT 4 prompt), can help you write, fix, review, and explain code with personalized learning experiences and interaction styles.

Introduction:

Whether you’re a beginner or an experienced programmer, having a helping hand while writing, fixing, or reviewing code can make a significant difference in your productivity. CodeRanedeer, an AI-powered programming assistant developed by Refaat Al Ktifan, is designed to cater to your coding needs with personalized learning experiences and interaction styles. This blog post will introduce CodeRanedeer, walk you through its features, and provide examples to showcase its capabilities. Check out the GitHub repository here: https://github.com/Refaat-alktifan/CodeRanedeer

Section 1:

Overview of CodeRanedeer CodeRanedeer is an AI language model that assists with various code-oriented tasks such as writing, fixing, reviewing, and explaining programming code. It can also serve as a pair programming buddy, enhancing your coding experience through collaboration. CodeRanedeer offers personalization options to cater to individual learning preferences and interaction styles.

Section 2:

Personalization Features CodeRanedeer provides personalized learning experiences and interaction styles based on user preferences. It offers the following personalization options:

Depth: Determines the level of detail in explanations and assistance, ranging from basic concepts to advanced techniques.
Interaction Styles: Specifies the AI’s role in the interaction, such as providing guidance or collaborating as a pair programming buddy.
Explanation Styles: Defines how the AI presents information, catering to different levels of programming expertise and learning preferences.

Section 3:

Commands and Configuration CodeRanedeer features a range of commands to help you perform code-oriented tasks. These include:

/write_code: Write code based on user’s specifications.
/fix_code: Identify and fix issues in the provided code.
/review_code: Review the provided code and offer suggestions for improvement.
/explain_code: Explain the provided code or concepts to the user.
/pair_program: Engage in pair programming with the user.

You can also configure CodeRanedeer according to your preferences, including depth, interaction style, explanation style, and language.

Section 4:

Example Use Cases Here are some examples showcasing CodeRanedeer’s capabilities:

Writing code: CodeRanedeer can write a Python function to calculate the factorial of a given number.
Reviewing code: CodeRanedeer can review your Python code, offering suggestions for improvement or optimization.
Fixing code: CodeRanedeer can identify issues in your JavaScript code and provide fixes, ensuring your code is working correctly.
Explaining code: CodeRanedeer can break down and explain Java code, making it easier to understand the logic and flow of the program.
Pair programming: CodeRanedeer can help you create a Python script to find the greatest common divisor (GCD) of two numbers, engaging in pair programming to enhance your coding experience.

Conclusion:

CodeRanedeer is an invaluable AI-powered programming assistant (GPT 4 prompt) that can help you with various code-oriented tasks while catering to your learning preferences and interaction styles. With CodeRanedeer, you can improve your coding skills, enhance your understanding of programming concepts, and boost your productivity. Don’t forget to visit the GitHub repository to explore CodeRanedeer further: https://github.com/Refaat-alktifan/CodeRanedeer
AI Code Buddy - CodeRanedeer

Keywords: CodeRanedeer, AI-powered, programming assistant, personalized learning, chatgpt, gpt4, ai, free ai

0–7 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 16:05:04 +0000

Pipes are an essential feature of NestJS, used for handling data transformation and validation in a clean and maintainable way. Pipes are classes that implement the PipeTransform interface, which requires a single method: transform(). This method takes an input value and returns a transformed value or throws an exception if the value is invalid.

There are two primary use cases for pipes in NestJS:

Validation: Pipes can validate incoming data, ensuring it meets specific requirements before it’s passed to a controller method. If the data is invalid, the pipe can throw an exception, which NestJS will convert into an appropriate HTTP error response.

For example, you can use the built-in ValidationPipe to automatically validate incoming data against a class-validator schema:

    import { Controller, Post, Body, UsePipes } from '@nestjs/common';
    import { ValidationPipe } from '@nestjs/common';
    import { CreateCatDto } from './dto/create-cat.dto';

    @Controller('cats')
    export class CatsController {
      @Post()
      @UsePipes(ValidationPipe)
      create(@Body() createCatDto: CreateCatDto) {
        // Implementation of the create method
      }
    }

Data Transformation: Pipes can transform incoming data, allowing you to modify or normalize it before it reaches a controller method. This is useful for cases like converting strings to numbers or formatting date strings.

For instance, you can create a custom pipe to parse a string into a number:

    import { PipeTransform, Injectable, BadRequestException } from '@nestjs/common';

    @Injectable()
    export class ParseIntPipe implements PipeTransform {
      transform(value: string) {
        const val = parseInt(value, 10);
        if (isNaN(val)) {
          throw new BadRequestException('Validation failed');
        }
        return val;
      }
    }

To use this custom pipe, apply it to a controller method with the @UsePipes() decorator:

    import { Controller, Get, Param, UsePipes } from '@nestjs/common';
    import { ParseIntPipe } from './parse-int.pipe';

    @Controller('cats')
    export class CatsController {
      @Get(':id')
      findOne(@Param('id', ParseIntPipe) id: number) {
        // Implementation of the findOne method
      }
    }

Pipes help maintain a clean and organized codebase by handling validation and transformation concerns separately from the core business logic.

Next, let’s explore the concept of interceptors in NestJS. How do interceptors work, and what are some common use cases for them in a NestJS application?

to be continued

0–6 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 16:02:35 +0000

Dependency Injection (DI) is a design pattern used in NestJS to manage dependencies between classes, promoting decoupling and making it easier to test and maintain the code. In NestJS, the DI system is built around the concept of providers and tokens.

Providers: A provider is a class, value, or factory function responsible for creating an object that can be injected into other classes. Providers are typically registered in the providers array of a module. Here's an example of a provider class:

    import { Injectable } from '@nestjs/common';

    @Injectable()
    export class CatsService {
      // Implementation of the service
    }

Tokens: Tokens are unique identifiers used by NestJS to locate providers. By default, a class’s token is the class itself. When a class needs a dependency, it requests it from the DI system using a token.
Injecting Dependencies: To inject a dependency, you can use the @Inject() decorator in the constructor of a class. The DI system will then resolve the dependency and provide an instance of the requested class. Here's an example:

    import { Injectable, Inject } from '@nestjs/common';
    import { CatsService } from './cats.service';

    @Injectable()
    export class AppService {
      constructor(private readonly catsService: CatsService) {}
    }

In this example, the AppService class requests an instance of CatsService in its constructor. The DI system locates the CatsService provider, creates an instance, and injects it into the AppService class.

Dependency Injection in NestJS provides the following benefits:

Decoupling: DI promotes loose coupling between classes by allowing them to depend on abstractions rather than concrete implementations. This makes it easier to modify or replace parts of the application without affecting other components.
Testing: DI makes it simpler to test classes by allowing you to replace dependencies with mock implementations during testing.
Reusability: By managing dependencies centrally, DI encourages the reuse of providers across multiple parts of the application.

Now that you understand Dependency Injection in NestJS, let’s discuss the role of pipes and their use cases. How can pipes help with validation and data transformation in a NestJS application?

to be continued

0–5 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 15:56:12 +0000

Modules and decorators are fundamental concepts in NestJS that contribute to its modular architecture, helping you create well-structured and maintainable applications.

Modules: In NestJS, a module is a class annotated with the @Module() decorator, responsible for organizing application components such as controllers, providers, and other modules. By grouping related components together, modules make it easier to manage large applications and promote code reusability.

A typical NestJS module looks like this:

    import { Module } from '@nestjs/common';
    import { CatsController } from './cats.controller';
    import { CatsService } from './cats.service';

    @Module({
      controllers: [CatsController],
      providers: [CatsService],
    })
    export class CatsModule {}

Decorators: Decorators are a TypeScript feature that allows you to add metadata to classes, properties, and methods. In NestJS, decorators are used extensively to enhance classes and provide additional information to the framework. Some common decorators in NestJS include @Controller(), @get(), @post(), and @Inject().

For instance, a basic NestJS controller using decorators could look like this:

    import { Controller, Get } from '@nestjs/common';

    @Controller('cats')
    export class CatsController {
      @Get()
      findAll() {
        return 'This action returns all cats.';
      }
    }

The @Controller('cats') decorator marks the CatsController class as a controller and associates it with the 'cats' route. The @get() decorator is applied to the findAll method, indicating that it should handle HTTP GET requests.

By leveraging modules and decorators, NestJS enforces a clean, modular structure for your application, making it easier to manage, extend, and maintain.

Now, let’s explore a crucial aspect of NestJS: dependency injection. Can you explain how dependency injection works in NestJS and how it helps with decoupling and testing?

to be continued

0–4 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sun, 07 May 2023 15:54:46 +0000

Interfaces in TypeScript are a powerful way to define contracts for the shape of objects, allowing you to enforce specific structures for objects and create reusable type definitions. They can also be used for type checking and autocompletion support in code editors.

Here’s an example of an interface in TypeScript:

    interface Person {
      firstName: string;
      lastName: string;
      age: number;
      greet(): string;
    }

This interface defines a Person object with firstName, lastName, and age properties, as well as a greet method. You can use the interface as a type for an object like this:

    const john: Person = {
      firstName: "John",
      lastName: "Doe",
      age: 30,
      greet() {
        return `Hello, my name is ${this.firstName} ${this.lastName}.`;
      },
    };

If you try to create an object that doesn’t match the Person interface, TypeScript will throw a type error during compilation.

Now let’s shift our focus to NestJS. Can you explain the importance of modules and decorators in NestJS and how they contribute to the framework’s modular architecture?

to be continued

0–3 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sat, 06 May 2023 15:56:26 +0000

Static typing and dynamic typing are two approaches to handling data types in programming languages.

In static typing, the data type of a variable is known at compile time, meaning that the type is explicitly specified or inferred when the code is written. This allows for type checking before the code is executed, which can help catch potential errors early and improve code reliability. Languages with static typing include TypeScript, Java, and C++.

In dynamic typing, the data type of a variable is determined at runtime, meaning that it’s determined when the code is executed. Variables can change types during the execution of a program, which can make the code more flexible but also more prone to errors. Languages with dynamic typing include JavaScript, Python, and Ruby.

Now that we’ve briefly discussed static typing, let’s delve into TypeScript basics. TypeScript introduces static typing to JavaScript, providing several data types such as number, string, boolean, null, and undefined. In addition, TypeScript has more advanced types like any, unknown, never, tuple, and enum.

For instance, declaring a variable with a specific type in TypeScript looks like this:

let age: number = 25;

If you try to assign a non-number value to the age variable, TypeScript will throw a type error during compilation.

Are you familiar with interfaces in TypeScript? If so, can you provide an example?

to be continued

0–2 NestJS Hunter: Slaying Complexity in Node.js with TypeScript.

refaat Al Ktifan — Sat, 06 May 2023 15:55:24 +0000

Using TypeScript, NestJS, and Node.js together is beneficial for modern web development for several reasons:

Type Safety: TypeScript’s static typing helps catch potential errors during development, leading to more reliable and maintainable code. It can also enhance the development experience with better autocompletion and documentation support in code editors.
Consistent Language: Since Node.js allows you to write server-side code in JavaScript, using TypeScript across both the front-end and back-end of an application enables you to maintain a consistent language throughout the entire development stack.
Modular Architecture: NestJS’s modular approach, inspired by Angular, encourages a well-structured and organized codebase. This makes it easier to manage large applications and promotes code reusability.
Scalability and Performance: Node.js’s event-driven, non-blocking I/O model allows for efficient resource usage and high performance when handling multiple concurrent connections. NestJS enhances this by providing an abstraction layer that simplifies building scalable applications.
Growing Ecosystem: The combination of TypeScript, NestJS, and Node.js has a vibrant and growing ecosystem with a vast number of libraries, tools, and resources available to help you build powerful applications more quickly and efficiently.

Now that we’ve discussed the benefits of using TypeScript, NestJS, and Node.js together, let’s move on to exploring TypeScript basics. Are you familiar with static typing, and can you explain how it differs from dynamic typing?

to be continued