DEV Community: Rizwan Saleem

Changesets for monorepo versioning: managing changelogs and publishing across packages

Rizwan Saleem — Tue, 23 Jun 2026 08:01:36 +0000

Changesets for monorepo versioning: managing changelogs and publishing across packages

Understanding the Core Concepts

Changesets provide a pull-request-friendly approach to versioning packages in monorepos. Developers add changeset files describing their changes, and automated release processes consume them to generate changelogs and publish updated packages.

Developer productivity is determined more by environment and tools than by individual effort. Investing in your tooling, workflow, and development environment compounds over time. The best productivity investments pay back every day.

Practical Implementation Strategies

When implementing solutions in this area, start with a clear understanding of the requirements and constraints. What problem are you solving, and what does success look like? Define measurable outcomes before choosing your approach. This clarity prevents over-engineering and ensures you are solving the right problem.

Begin with a simple implementation that addresses the core requirements. The simplest working solution teaches you more than a complex partially-built one. You can iterate and improve once you have a foundation that works end-to-end.

Test your implementation thoroughly before deploying to production. Write tests that cover normal operation, edge cases, and failure scenarios. Automated testing gives you confidence that your system behaves correctly and catches regressions when you make changes.

Monitor your implementation in production. Collect metrics on performance, error rates, and resource utilization. Set up alerting for conditions that require human intervention. Observability data tells you whether your system is behaving as expected and helps you diagnose issues when they arise.

Common Challenges and Solutions

One of the most common challenges in this area is underestimating complexity. Systems that seem simple at first often reveal hidden complexity when you start implementing them. Break down complex problems into smaller, manageable pieces. Each piece should be independently testable and deployable.

Another frequent challenge is over-engineering the solution. It is tempting to build for scale you do not need yet or adopt patterns that add complexity without immediate benefit. Build for what you know you need and refactor when you learn more.

Technical debt accumulates when shortcuts are taken without a plan to address them. The key is to track technical debt consciously and allocate time to address it before it slows down development.

Real-World Applications

The patterns and practices discussed here have been proven in production systems at companies of all sizes. Startups use them to move fast without creating disasters. Large enterprises use them to maintain reliability at massive scale. The principles are universal, though the implementation details vary by context.

When applying these concepts to your own work, consider your specific context. A five-person startup has different constraints than a five-hundred-person enterprise. The right solution depends on your team size, risk tolerance, and growth trajectory. Adapt patterns to your situation rather than adopting them blindly.

Learn from the experiences of others. Case studies, conference talks, and engineering blogs share hard-won lessons from real implementations. Studying what went wrong is often more valuable than studying what went right.

Key Takeaways

The most important principle: keep it simple. Complexity is the enemy of reliability, maintainability, and velocity. Simple systems are easier to understand, debug, and change. Every abstraction, pattern, and tool you add should earn its place by solving a concrete problem that you actually have.

Second principle: measure before you optimize. Without data, you are guessing about what matters. With data, you can identify the actual bottlenecks and focus your energy where it has the most impact.

Third principle: invest in your team capabilities. The best architecture is worthless if your team cannot operate it effectively. Choose technologies and patterns that your team understands and can maintain.

Conclusion

Mastering this area of software engineering takes time and practice. The concepts build on each other, and understanding deepens with experience. Start with the fundamentals, practice consistently, and learn from both successes and failures. The journey of continuous improvement is what defines great engineers.

Share what you learn with your team and the broader community. Writing about your experiences, both successes and failures, helps others avoid your mistakes and builds your reputation as a thoughtful engineer.

Getting Started

If you are new to this topic, start with the fundamentals. Understand the core concepts before diving into advanced patterns. Build a simple implementation that works end-to-end. Then gradually add sophistication as you understand the tradeoffs involved.

The best way to learn is by doing. Pick a small project that exercises the concepts discussed here. Implement it, deploy it, and operate it. The lessons you learn from a real implementation will be deeper than anything you can learn from reading alone.

Pro Tips

Document your decisions and the reasoning behind them. Architecture Decision Records capture the context, options, and rationale for significant technical choices. This documentation helps future team members understand why things are the way they are and avoids repeating past mistakes.

Automate everything that can be automated. Manual processes are error-prone and do not scale. Every manual step in your workflow is an opportunity for automation. Invest in automation early, and it will pay dividends throughout the life of your system.

Action Plan

This week: audit your current systems and practices in this area. Identify the biggest gap between where you are and where you want to be. Pick one improvement that you can make this week.

This month: implement the improvement you identified. Measure the impact. Share what you learned with your team. Document the changes and the reasoning behind them.

This quarter: review and refine your approach. What worked well? What could be improved? Update your practices based on what you have learned. Continuous improvement is the key to mastery.

Rizwan Saleem | https://rizwansaleem.co

WebAssembly in production: running high-performance code in the browser and beyond

Rizwan Saleem — Mon, 08 Jun 2026 04:20:19 +0000

WebAssembly in production: running high-performance code in the browser and beyond

Understanding the Core Concepts

Frontend engineering has evolved from simple page rendering to complex application development. Modern frontend applications manage state, handle real-time updates, and deliver rich user experiences. The complexity has shifted from the server to the client.

Practical Implementation Strategies

Begin with a simple implementation that addresses the core requirements. Resist the temptation to add features or optimizations before you have a working system. The simplest working solution teaches you more than a complex partially-built one. You can iterate and improve once you have a foundation that works end-to-end.

Common Challenges and Solutions

Another frequent challenge is over-engineering the solution. It is tempting to build for scale you do not need yet, add abstractions that obscure the simple path, or adopt patterns that add complexity without immediate benefit. Build for what you know you need and refactor when you learn more.

Technical debt accumulates when shortcuts are taken without a plan to address them. Not all shortcuts are bad: sometimes shipping quickly is the right business decision. The key is to track technical debt consciously and allocate time to address it before it slows down development.

Real-World Applications

Key Takeaways

Third principle: invest in your team capabilities. The best architecture in the world is worthless if your team cannot operate it effectively. Choose technologies and patterns that your team understands and can maintain. Train your team continuously. The capability of your team is the most important factor in your system success.

Conclusion

Getting Started

Pro Tips

Action Plan

This week: audit your current systems and practices in this area. Identify the biggest gap between where you are and where you want to be. Pick one improvement that you can make this week.

This month: implement the improvement you identified. Measure the impact. Share what you learned with your team. Document the changes and the reasoning behind them.

This quarter: review and refine your approach. What worked well? What could be improved? Update your practices based on what you have learned. Continuous improvement is the key to mastery.

Rizwan Saleem | https://rizwansaleem.co

Building technical authority: speaking, writing, and leading in your domain

Rizwan Saleem — Sun, 07 Jun 2026 13:30:17 +0000

Building technical authority: speaking, writing, and leading in your domain

Understanding the Core Concepts

Career growth in software engineering is not linear. The skills that make you effective as a junior engineer differ from those needed at senior and staff levels. Understanding the inflection points in your career helps you invest in the right skills at the right time.

Technical leadership is about enabling others to do their best work. It requires balancing technical excellence with people development. The best leaders create conditions for their teams to succeed rather than trying to solve every problem themselves.

Practical Implementation Strategies

Common Challenges and Solutions

Real-World Applications

Key Takeaways

Conclusion

Getting Started

Pro Tips

Action Plan

This week: audit your current systems and practices in this area. Identify the biggest gap between where you are and where you want to be. Pick one improvement that you can make this week.

This month: implement the improvement you identified. Measure the impact. Share what you learned with your team. Document the changes and the reasoning behind them.

This quarter: review and refine your approach. What worked well? What could be improved? Update your practices based on what you have learned. Continuous improvement is the key to mastery.

Rizwan Saleem | https://rizwansaleem.co

Architecture Decision Records: documenting technical decisions that stick

Rizwan Saleem — Sun, 07 Jun 2026 11:50:17 +0000

Architecture Decision Records: documenting technical decisions that stick

Understanding the Core Concepts

Documentation is how knowledge outlives the people who created it. Good documentation accelerates onboarding, reduces support burden, and preserves institutional knowledge. Writing documentation is an investment that pays compounding returns.

Software engineering is as much about process and people as it is about code. The best technical solutions fail if they do not consider the human factors: team dynamics, organizational constraints, and user needs. Engineering is ultimately about delivering value.

Practical Implementation Strategies

Common Challenges and Solutions

Real-World Applications

Key Takeaways

Conclusion

Getting Started

Pro Tips

Action Plan

This week: audit your current systems and practices in this area. Identify the biggest gap between where you are and where you want to be. Pick one improvement that you can make this week.

This month: implement the improvement you identified. Measure the impact. Share what you learned with your team. Document the changes and the reasoning behind them.

This quarter: review and refine your approach. What worked well? What could be improved? Update your practices based on what you have learned. Continuous improvement is the key to mastery.

Rizwan Saleem | https://rizwansaleem.co

Web components in production: a practical guide for modern frontends

Rizwan Saleem — Sun, 07 Jun 2026 09:50:20 +0000

Web components in production: a practical guide for modern frontends

Web components are native browser APIs for creating reusable UI components. They work across all frameworks React, Vue, Angular, or no framework at all. After years of mixed browser support, web components are now production-ready and supported in all modern browsers.

A web component is a custom HTML element defined using the Custom Elements API. You create a class that extends HTMLElement, register it with customElements.define, and the element works anywhere you use HTML. The Shadow DOM provides style encapsulation that prevents CSS conflicts. This native approach eliminates framework lock-in for UI components.

Web components are ideal for shared design systems. Build your buttons, form inputs, modals, and navigation as web components. They work in any framework and any project. A team using React and a team using Vue can share the same component library. This is the primary use case for web components in production.

Performance considerations matter. Each web component creates a Shadow DOM tree, and creating many components can be slow. Use lightweight components for frequently created elements. Consider lit-html or LitElement for efficient rendering. Performance testing should be part of your web component development process.

Interoperability with frameworks requires some adaptation. React has issues with custom events and form elements. Vue and Angular handle web components better. Most frameworks provide guidance on integrating web components. Test your web components with the frameworks your teams actually use.

Web components are not a replacement for your framework. They're a complement for specific use cases: shared design systems, embedding widgets in third-party sites, and long-lived applications that outlive framework trends. Use web components strategically where they provide the most value.

Testing web components is straightforward. They're regular DOM elements, so you can test them with standard testing tools. Set attributes, dispatch events, and assert on the rendered output. Playwright and Cypress handle web components natively, making testing consistent with the rest of your application.

Practical Implementation

Start with a solid component architecture. Break your UI into small, focused components that each do one thing well. Use a component library as a foundation to avoid rebuilding basic elements. Invest in a good design system that enforces consistency across your application.

Optimize for the user experience before developer convenience. A fast, accessible, responsive application beats a clever architecture every time. Measure Core Web Vitals in CI and set performance budgets that fail the build if exceeded.

Common Challenges

State management is the most over-engineered aspect of frontend development. Most state should be local. Server state belongs in a dedicated data-fetching library like TanStack Query or SWR. Only use global state management for truly cross-cutting concerns like authentication and theming.

Bundle size grows insidiously. Every import adds bytes. Use bundle analysis tools to track what you ship. Code-split at the route level. Lazy-load heavy components. Remove unused dependencies. A modern frontend application should ship under 200KB of JavaScript for the initial load.

Real-World Application

A typical modern stack: React or Vue with TypeScript for the framework, TanStack Query for server state, Zustand for global state, Tailwind CSS for styling, Vite for building, Playwright for E2E tests, and Vitest for unit tests. This stack provides a productive developer experience with good performance characteristics.

Key Takeaways

Simple state management is best. Optimize for users first. Measure bundle size in CI. Test with real browsers. The best frontend architecture is invisible to the user.

Advanced Implementation

Implement client-side error tracking to catch issues that users experience but do not report. Tools like Sentry capture JavaScript errors with full context browser version, operating system, user actions leading up to the error, and the stack trace. Error tracking turns silent failures into actionable bug reports.

Optimize the loading sequence for your application. Identify the resources needed for the initial render and load them first. Defer everything else. Use resource hints like preload, prefetch, and preconnect to tell the browser about important resources early. A good loading sequence can cut perceived load time in half.

Accessibility and Internationalization

Build accessibility into your component architecture from the start. Use semantic HTML, manage focus correctly, and test with screen readers. Accessibility is not a feature it is a fundamental quality of a good frontend.

Internationalize early, even if you do not have immediate plans for multiple languages. String formatting, date/time handling, and text direction are much harder to retrofit than they are to build in from the start. Use a mature i18n library and externalize all user-facing strings from day one.

Common Mistakes and How to Avoid Them

The most common frontend mistake is optimizing for developer experience at the expense of user experience. Heavy frameworks, complex state management, and excessive dependencies make the developer happier but the user slower. Always optimize for the user first measure Core Web Vitals in CI and set performance budgets.

Another frequent error is ignoring the non-JavaScript experience. Your application should work without JavaScript, at least for core content. This improves SEO, accessibility, and resilience. Progressive enhancement starting with HTML, then CSS, then JavaScript builds more robust applications.

Conclusion

Frontend development is evolving rapidly, but the fundamentals remain the same: build fast, accessible, and reliable interfaces for users. Choose tools that help you achieve these goals rather than tools that are popular or trendy. The best frontend is invisible it just works, quickly and reliably.

Getting Started

If you are new to frontend development, start with the fundamentals: HTML, CSS, and JavaScript. Understand the DOM, the event loop, and the rendering pipeline. Build a simple page without frameworks. Adding React or Vue is easier when you understand what they are abstracting.

Learn responsive design principles. Use CSS Grid and Flexbox for layouts. Use relative units (rem, em, %) instead of absolute units (px) for flexibility. Use media queries for breakpoints. A responsive design that works on mobile, tablet, and desktop is a fundamental requirement for modern web applications.

Pro Tips

Use TypeScript for all frontend projects. TypeScript catches entire categories of bugs at compile time: null reference errors, incorrect function arguments, and missing properties. The initial investment in typing pays for itself many times over through fewer runtime errors and better developer experience.

Optimize the critical rendering path. Identify the minimum resources needed to display the first meaningful paint. Inline critical CSS, defer non-critical JavaScript, and use preload hints for important resources. A fast initial render is the most important performance optimization for user experience.

Related Concepts

Understanding browser rendering helps you build faster applications. Learn about the critical rendering path, layout thrashing, and compositing. Understanding how the browser processes HTML, CSS, and JavaScript helps you write code that renders efficiently.

Web accessibility is not optional it is a fundamental quality of a web application. Learn WCAG guidelines, ARIA attributes, and screen reader testing. An accessible application is usable by everyone, including people with disabilities. Accessibility also improves SEO and overall user experience.

Action Plan

This week: run a performance audit on your application. Use Lighthouse to measure Core Web Vitals. Identify the top three performance issues and fix them.

This month: audit your application for accessibility issues. Use axe-core or Lighthouse to find violations. Fix the critical issues. Test with a screen reader.

This quarter: implement a component library or design system. Consistent components improve development speed, visual consistency, and maintainability. Document component usage and build a shared library that all teams can use.

Rizwan Saleem | https://rizwansaleem.co

How to explain technical concepts to anyone: a step by step guide

Rizwan Saleem — Sat, 06 Jun 2026 23:40:20 +0000

How to explain technical concepts to anyone: a step by step guide

The ability to explain technical concepts to non-technical stakeholders is one of the most undervalued skills in engineering. Good explanations build trust, align teams, and help you sell your ideas. Engineers who communicate well are more influential, get promoted faster, and have more impact.

Start with the "why" why does this matter to your audience? A product manager cares about delivery timelines and business impact, not about the elegance of your architecture. Frame every explanation in terms of outcomes they care about. "We need to refactor this module" becomes "We need to spend two days improving this code so we can add new features three times faster."

Use analogies grounded in everyday experience. A caching layer is like a pantry you keep frequently used items close so you don't have to go to the supermarket every time. A load balancer is like a restaurant host who seats guests at the next available table. A database index is like a book's index it helps you find information without reading every page. Good analogies create instant understanding.

Layer complexity deliberately. Start with the simplest possible explanation that is still accurate. Then add detail layer by layer, checking for understanding at each level. The listener should be able to repeat back your explanation in their own words before you add the next layer. This technique, sometimes called "progressive disclosure," prevents cognitive overload.

Visuals beat text for communicating structure. Draw architecture diagrams, data flow charts, and before/after comparisons. A well-drawn diagram communicates in seconds what paragraphs of text cannot. Tools like Excalidraw, Miro, or even a whiteboard are more effective than slides full of bullet points.

Check for understanding continuously. Ask "Does that make sense?" and wait for a real answer. Watch for glazed eyes, which signal you've lost them. If someone asks a question that shows they're thinking at a deeper level, you've succeeded. The goal is not to deliver information but to create understanding.

End with a clear ask or decision. Technical explanations should drive action. What do you need from your audience? Approval? Resources? Simply awareness? Make it explicit and give them a way to say yes or no. A conversation without a clear outcome is a missed opportunity.

Practical Implementation

Invest in your career systematically, not reactively. Set quarterly goals for skill development, network building, and personal projects. Review progress monthly and adjust your strategy based on what is working. Treat your career like a product you are building.

Build a portfolio of work that demonstrates your skills. Write blog posts, contribute to open source, speak at meetups, and share what you learn on social media. The most successful engineers are known for what they create and share, not just for their job titles.

Common Challenges

The biggest career mistake is optimizing for salary at the expense of growth. Early in your career, prioritize teams and projects where you will learn the most. The salary growth will follow the skill growth. Later, optimize for autonomy, impact, and working conditions.

Another common mistake is staying too long in a comfortable role. If you are not learning anymore, it is time to move. The market rewards engineers who continuously grow their skills and take on new challenges.

Real-World Application

A 5-year career plan: year 1-2, join a fast-growing company and learn from strong seniors. Year 3, take on technical leadership for a medium-sized project. Year 4, specialize in an area with high demand (AI/ML, security, or distributed systems). Year 5, consider staff engineer track or transitioning to management.

Key Takeaways

Invest in learning early. Build in public. Network authentically. Move when you stop growing. The best career investment is becoming someone others want to work with.

Advanced Implementation

Build your personal brand through consistent, valuable contributions to your professional community. Write one blog post per month, share insights on social media weekly, and speak at one conference or meetup per quarter. Consistency matters more than volume a steady stream of quality contributions builds trust and recognition over time.

Develop a mentorship network both as mentor and mentee. Teaching others deepens your own understanding and builds leadership skills. Having mentors provides guidance, perspective, and opportunities. The best engineers are lifelong learners who actively seek out both roles.

Strategic Career Planning

Track your career metrics like a product: skills acquired, network size and quality, speaking engagements, published writing, and compensation growth. Review these metrics quarterly and adjust your strategy based on what is working. A career that is deliberately managed grows faster than one left to chance.

Understand the difference between your role and your function. Your role is your job title; your function is the value you create. The most successful engineers focus on maximizing their function, not optimizing for a specific title. When your function grows, the appropriate role follows.

Common Mistakes and How to Avoid Them

The most common career mistake is optimizing for short-term gains at the expense of long-term growth. A high salary at a company where you learn nothing is a bad trade. A title without real responsibility is empty. Prioritize learning, impact, and growth opportunities, especially early in your career.

Another frequent error is not negotiating. Many engineers accept the first offer or never ask for a raise. Negotiation is a skill that compounds over your career a 10 percent difference on your first salary grows to hundreds of thousands of dollars over a career.

Conclusion

Your career is one of your most important investments. Manage it deliberately, invest in learning continuously, build authentic relationships, and do not be afraid to make changes when you stop growing. The best careers are not planned in detail they are built through a series of good decisions over time.

Getting Started

If you are early in your career, focus on building a strong technical foundation. Master one programming language deeply before learning others. Understand data structures, algorithms, and system design. Build projects that demonstrate your skills. The first few years are an investment period prioritize learning over compensation.

Develop communication skills early. Write documentation, give presentations, and participate in code review discussions. Technical skills get you in the door; communication skills determine how far you go. The most senior engineers are often the best communicators, not just the best coders.

Pro Tips

Keep a brag document a running list of your accomplishments, impact metrics, and positive feedback. Update it monthly. Use it when writing performance reviews, updating your resume, or preparing for interviews. Your memory of what you accomplished six months ago is less reliable than your brag document.

Build your network before you need it. Connect with people at conferences, meetups, and online communities. Share what you learn through blogging, social media, or talks. A strong network provides job opportunities, mentorship, and support throughout your career.

Related Concepts

Understanding the business side of engineering helps you make better career decisions. Learn how product management works, how engineering impacts business metrics, and how to communicate with non-technical stakeholders. Engineers who understand the business context are more valuable and more influential.

Financial literacy is important for maximizing your career earnings. Understand equity compensation, tax implications of stock options, and long-term investment strategies. Many engineers leave significant money on the table because they do not understand their compensation package.

Action Plan

This week: create or update your brag document. List your accomplishments from the last three months. Identify gaps in your skills and create a plan to address them.

This month: publish one piece of content a blog post, a social media thread, or a talk recording. Share something you have learned. The act of creating content clarifies your thinking and builds your reputation.

This quarter: review your career trajectory. Are you learning? Are you growing? Are you on track for your long-term goals? If not, identify what needs to change and make a plan. Your career is your most important investment manage it deliberately.

Rizwan Saleem | https://rizwansaleem.co

Building a Self-Cheering Microservice: Observability-Driven Resilience in a Real-Time Analytics Pipe

Rizwan Saleem — Thu, 04 Jun 2026 16:30:20 +0000

Building a Self-Cheering Microservice: Observability-Driven Resilience in a Real-Time Analytics Pipe

Building a Self-Cheering Microservice: Observability-Driven Resilience in a Real-Time Analytics Pipeline

In this thought-leadership piece, I’ll share a senior engineer’s perspective on a concrete project I built: a real-time analytics microservice designed to celebrate its own health and performance as it runs. The core idea is to embed observability into the architecture so the system “self-cheers” when things go well and gracefully flags, quarantines, and recovers from trouble. This approach reduces MTTR, improves operator confidence, and provides a blueprint for other teams aiming to ship resilient, observable services without sacrificing velocity.

Overview of the project

Domain: Real-time event analytics for a streaming platform
Service: Lightweight, polyglot microservice written in Go (core ingest and enrichment) with a Node.js sidecar for optional user-facing dashboards
Communication: gRPC for internal calls, Apache Kafka for event streaming
Observability stack: OpenTelemetry + Prometheus + Grafana, with a custom “health cheer” signal pipeline
Deployment: Kubernetes with autoscaling, canary releases, and circuit breakers

Why observability-driven resilience matters

Traditional health checks tell you if a process is alive, not if it’s delivering value. Observability-focused design surfaces the exact corners where value degrades (throughput, latency, error budgets).
A system that can measure itself and announce its status reduces ambiguity during incidents and speeds remediation.
Proactive detection (slo-based alerts, self-healing retries, and adaptive load shedding) keeps service-level objectives within reach even under load spikes.

Design goals

End-to-end traceability: correlate ingestion, enrichment, and downstream processing across services
Low-latency path: sub-100ms latency for typical events, with predictable tail latency handling
Resilience by design: automatic retry, backoff, circuit-breaking, and graceful degradation
Self-cheering health signals: clear, visible indicators of healthy operating conditions
Maintained velocity: minimal operational overhead; easy to onboard new teams

Project architecture

Ingest tier (Go microservice)
- gRPC API for internal ingestion, with protobuf definitions including a health Cheer field
- Kafka producer for event persistence and downstream consumption
- Enrichment pipeline: lightweight transformations, deterministic for idempotency
Sidecar (Node.js)
- Web UI to visualize health signals, metrics, and recent event stats
- Optional live dashboard exposing a simple API for external dashboards
Observability layer
- OpenTelemetry instrumentation in both services
- Metrics exposed to Prometheus; dashboards in Grafana
- Distributed tracing across components via OTLP export
Resilience primitives
- Retries with exponential backoff and jitter
- Circuit breakers for downstream dependencies
- Rate limiting and backpressure signals to Kafka producer

Step 1: Instrumentation plan

Define a minimal, consistent trace across components:
- Trace ID, span IDs propagated through gRPC calls and Kafka produce/consume paths
- Key spans: ingest_request, enrichment_step, kafka_publish, downstream_call
Collect metrics at three levels:
- Service-level: request_rate, success_rate, error_rate, p95/p99 latency
- Pipeline-level: events_in_flushed, events_enriched, events_published, downstream_success
- Resource-level: cpu_usage, memory_usage, gc_pause_seconds
Implement health cheer signals:
- Health metrics: healthy = true when error_budget_remaining > threshold, latency p90 within target, queue depths below saturation
- Cheer events: counter increments with every successful batch, latency within target, and all downstream calls healthy
Code example (Go) for tracing and metrics (simplified):
- Initialize OTLP exporter and tracer
- Wrap handlers with tracing:
- func handleIngest(ctx context.Context, req *IngestRequest) (*IngestResponse, error) { ctx, span := tracer.Start(ctx, "ingest_request"); defer span.End(); ... }
- Prometheus metrics:
- var ( ingestsTotal = prometheus.NewCounter(prometheus.CounterOpts{Name: "ingests_total", Help: "Total ingest requests"}) ingestLatency = prometheus.NewHistogram(prometheus.HistogramOpts{Name: "ingest_latency_ms", Help: "Ingest latency in ms", Buckets: prometheus.LinearBuckets(1, 10, 20)}) )
- prometheus.MustRegister(ingestsTotal, ingestLatency)

Step 2: End-to-end tracing and propagation

Propagate trace context through gRPC metadata and Kafka message headers
Use OpenTelemetry semantic conventions for spans
Example snippet (Go gRPC interceptor):
- func unaryServerInterceptor(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp interface{}, err error) { ctx, span := otel.Tracer("ingest-service").Start(ctx, info.FullMethod) defer span.End() // inject trace context into outbound gRPC or Kafka messages return handler(ctx, req) }
Kafka integration:
- Before producing a message, inject trace context into the message headers
- On the consumer side, extract the trace context to continue the trace

Step 3: Resilience patterns in code

Retries with backoff:
- Use a retry policy with max attempts, exponential backoff, and jitter
- Apply retries to transient errors (network hiccups, 429 responses)
Circuit breaker:
- Wrap downstream calls in a breaker (e.g., with goresilience or Sony’s/Go’s circuit-breaker implementations)
Backpressure and graceful degradation:
- If the Kafka producer channel is saturated, temporarily bound the rate and signal degraded mode in sidecar UI
Idempotency:
- Upsert on a unique event key to handle duplicates
Example (pseudo-Go) for a retry loop:
- for attempt := 1; attempt <= maxRetries; attempt++ { err := publishToKafka(event) if err == nil { break } backoff := minDelay * 2^(attempt-1) + jitter() time.Sleep(backoff) }

Step 4: Metrics-driven alerts and dashboards

Alerts:
- Error rate > 1% for 5 minutes
- P95 latency above target for 10 minutes
- Downstream circuit breaker open for longer than threshold
Dashboards:
- Global health ticker: a “cheer-meter” displaying green/yellow/red with recent cheer events
- Pipeline flow: ingest → enrich → publish with throughput and latency bars
- Resource usage: CPU, memory, GC pauses
Example PromQLs:
- rate(ingests_total[5m])
- histogram_quantile(0.95, le_ingest_latency_ms_seconds_bucket)

Step 5: Observability-driven release process

Canary-based rollouts with feature flags for health cheer
On each deploy, automatically run a synthetic data path to validate health cheer signals
If cheer count drops or health metrics degrade, fail fast and roll back
Document-run: keep a changelog of observed health signals tied to releases

Step 6: Git structure and code organization

Monorepo approach (for small teams) or separate repos with clear API contracts
Directory layout (Go example):
- /cmd/ingest-service
- /internal/enrichment
- /pkg/observability
- /third_party/proto
- /deploy/k8s
Instrumentation package
- /pkg/observability/tracing.go
- /pkg/observability/metrics.go
Lightweight validation tests
- Unit tests for enrichment logic
- Integration tests for trace propagation and end-to-end flow (can run in a test cluster)

Step 7: Measurable impact and how to quantify success

Latency and throughput:
- target: 99th percentile ingestion latency under 120 ms; throughput at least 1000 events/s per pod
Reliability metrics:
- Error rate under 0.5% for all critical paths
- SLOs: 99.9% or better for end-to-end latency
Operational efficiency:
- MTTR: reduced from hours to minutes thanks to unified health signals
- Time to onboard: new teams can instrument new services in a day or less using the same patterns
Business impact:
- Real-time analytics enable decisions with millisecond latency, improving user experience during peak usage

Code snippets: end-to-end trace and metrics glue (synthetic, compact)

Protobuf example (simplified):
- message IngestRequest { string event_id = 1; string payload = 2; int64 timestamp = 3; }
Go gRPC server interceptor (simplified):
- func unaryServerInterceptor(...) { ctx, span := otel.Tracer("ingest-service").Start(ctx, info.FullMethod); defer span.End(); // handler(ctx, req) }
OpenTelemetry initialization (simplified):
- func initTracer() { exporter, _ := otelx.NewOTLPExporter(ctx, otlpURL); tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exporter)); otel.SetTracerProvider(tp) }

Lessons learned

Start with a small, measurable health signal: you don’t need a perfect system to begin; track a few core signals and expand
Tie business outcomes to observability: demonstrate how health cheer translates to lower MTTR and more reliable user experience
Keep instrumentation lightweight yet consistent: use shared libraries for tracing and metrics to reduce duplication across services
Automate validation: synthetic tests that exercise health signals ensure you don’t drift during rapid releases
Foster a culture of blameless incident reviews: focus on improving the system, not assigning fault

Concrete next steps for teams

Pick a small, mission-critical microservice and implement a health cheer loop around one upstream and one downstream dependency
Establish a shared observability library for tracing, metrics, and health signals across services
Create a lightweight Node.js sidecar dashboard to visualize health cheer in real time
Deploy with canary releases and fatigue tests to validate resilience under simulated failures

Call to action
If you’re an engineer who cares about reliable, observable systems, let’s connect. I’d love to discuss concrete patterns for embedding self-cheering signals in your microservices, review your observability stacks, and share templates for training your teams to ship with resilience in mind. Reach out with a brief note about your current challenges and a link to your project or repo, and we’ll set up a time to dive in.

Rizwan Saleem | https://rizwansaleem.co

Building resilient edge data pipelines with probabilistic routing and verifiable compensation

Rizwan Saleem — Thu, 04 Jun 2026 15:50:20 +0000

Building resilient edge data pipelines with probabilistic routing and verifiable compensation

Edge computing often means moving computation closer to the data source, but it also introduces new challenges: intermittent connectivity, heterogeneous devices, and evolving data contracts. In this thought-leadership piece, I’ll walk through a concrete project I led to design and implement a resilient edge data pipeline that leverages probabilistic routing, local decision-making, and verifiable compensation. The goal is to share actionable techniques you can reuse in real-world edge scenarios, with metrics to guide adoption and clear lessons learned for the engineering community.

The project at a glance

Objective: Enable real-time anomaly detection at the edge, with eventual consistency guarantees when connectivity to the cloud is unreliable.
Scope: A fleet of 2000+ distributed IoT gateways collecting sensor data, with periodic bursts of high-frequency telemetry during events.
Core innovations:
- Probabilistic routing to local edge clusters to reduce latency and preserve bandwidth during congestion.
- Local decision logic for immediate responses with pluggable policy modules.
- Verifiable compensation and reconciliation when gateways reconnect to central services.
Measurable impact:
- 48-72% reduction in uplink data during stable periods.
- 25-40 ms latency for edge-triggered alerts in normal operation, vs 150-300 ms if routed through centralized services.
- 99.98% data integrity after reconciliation cycles, across simulated outage scenarios. ### Architectural overview

The system is organized into three planes:

Edge plane: lightweight runtime on gateways (Rust/wasm optional, but picks for safety and performance). Responsibilities include:
- Local feature extraction and anomaly scoring.
- Probabilistic routing to nearby edge clusters or cloud, based on network conditions and policies.
- Local queues with bounded memory to absorb bursts.
- Compensation log for reconciliation after connectivity is restored.
Control plane: a central orchestration service that:
- Publishes routing policies, routing tables, and anomaly definitions.
- Monitors edge health, network quality, and policy drift.
- Safely reconciles divergent states during re-attachment.
Data plane: the storage and processing backbone (real-time streaming at cloud-facing endpoints and optional local stores for offline mode).

Key design ideas:

Probabilistic routing: instead of deterministically sending all data to one destination, route a fraction of traffic to alternative paths during congestion or faults. This smooths load and preserves bandwidth for critical events.
Local decision policies: allow edge devices to make immediate, bounded decisions without waiting for cloud feedback. Policies can be swapped at deployment time for different environments.
Verifiable compensation: maintain append-only logs on the edge and a compact delta state in the cloud so that, upon reconnection, the system can replay or compensate missing records with a clear audit trail.

Core components and data flows
Data model (simplified):
- Telemetry { id, device_id, timestamp, metrics..., metadata }
- Snapshot { id, device_id, timestamp, store_version, state_hash }
- CompensationEntry { local_seq, remote_seq, action, status }
Edge data pipeline:
1. Ingest telemetry from sensors.
2. Compute lightweight features (e.g., moving average, rate-of-change).
3. Run anomaly score and decide local action.
4. Enqueue to an adjustable routing bucket and a local durable queue.
5. Route some data probabilistically to nearby edge hub and some to cloud, depending on policy and network metrics.
6. Persist a compensation log locally.
Reconciliation workflow:
- When connectivity is restored, the edge sends a reconciliation bundle containing:
- Local compensation entries not yet acknowledged by the cloud.
- A state hash and version markers.
- Cloud side applies compensations idempotently, updates central state, and confirms.
- If conflicts are detected, the system escalates to a deterministic reconciliation protocol with human-in-the-loop if needed.
Observability:
- Metrics: uplink bandwidth used, edge latency, local processing time, reconciliation success rate, data integrity rate.
- Traces: end-to-end latency from sensor to cloud acknowledgment, including routing path. ### Step-by-step implementation guide

1) Define policies and routing strategy

Decide target latency and preferred routing paths based on network topology.
Implement a probabilistic router:
- Given a data item, compute a routing score for each destination (edge hub, cloud).
- Use a Bernoulli trial with probability p to route to the primary destination; route to secondary with probability (1-p) or per-burst logic.
Example: during normal operation, p_primary = 0.8, p_secondary = 0.2. During congestion, p_primary reduces to 0.5.

2) Build the edge runtime

Language: choose a safe, low-footprint language (Rust with wasm for plug-ins is common; you can also use Go or C++ depending on device constraints).
Modules:
- Ingress: sensor adapters
- Feature extractor: simple stats and windowed calculations
- Decision engine: policy evaluation
- Router: probabilistic routing logic
- Queue and storage: bounded ring buffers, local on-device store with write-ahead log
- Compensation log: append-only, tamper-evident store
Data serialization: use compact formats (e.g., MessagePack or Protobuf) with versioned schemas.

3) Implement local decision policies

Start with a few policy templates:
- Anomaly-first policy: raise alert locally if score exceeds threshold; only forward summarized data to cloud.
- Bandwidth-preserving policy: prioritize essential telemetry and alerts; down-sample normal telemetry when bandwidth is constrained.
- Time-bounded policy: maintain a shortest-path path for a window; switch to cloud fallback after timeout.
Provide a policy runtime that can hot-swap without redeploying firmware.

4) Establish verifiable compensation mechanics

Edge logs:
- Use a monotonically increasing local sequence number per device.
- Log entries include a state_hash capturing the device state at commit time.
Cloud acknowledgments:
- Cloud maintains a minimal state map of acknowledged local_seq ranges per device.
- Reconciliation protocol uses sequence numbers to replay or confirm missing entries.
Security:
- Sign logs locally; verify signatures on the cloud side.
- Encrypt data in transit with mutual TLS; at rest, use device-specific keys.

5) Data routing and throttling

Implement rate-limiting at the edge:
- Token bucket per destination with configurable burst allowance.
Probing mechanism:
- Periodically measure round-trip time to edge hub and cloud; adapt routing probabilities accordingly.

6) Reconciliation choreography

Outage handling:
- During outages, queue items locally with bounded size; drop oldest if needed, but preserve critical events (configurable).
Rejoin flow:
- Exchange a compact reconciliation payload: local_seq_min, local_seq_max, state_hash, list of compensation entries.
- Cloud applies compensations in order and confirmsled state back to edge.

7) Observability and testing

Unit tests:
- Verify routing decisions under different simulated network conditions.
- Validate compensation application idempotence.
Integration tests:
- Simulate outages with a mock cloud service to verify reconciliation and state convergence.
Performance tests:
- Stress edge with burst data to measure latency and queue depths.
- Evaluate end-to-end latency with varying routing probabilities.

8) Deployment and rollout

Canary rollout:
- Start with a subset of gateways.
- Monitor key metrics for signs of instability.
Policy tuning:
- Use A/B testing to refine routing probabilities and policy thresholds.
Rollback plan:
- Maintain capability to revert to previous policy versions if anomalies occur.

Code example: probabilistic router (pseudo-Rust)

This illustrates the core idea of probabilistic routing and bounded queues.

use std::random::{ThreadRng, thread_rng};
use std::time::{Duration,Instant};

struct RoutePolicy {
primary_dest: Destination,
secondary_dest: Destination,
primary_prob: f64, // 0.0 - 1.0
rng: ThreadRng,
}

enum Destination {
EdgeHub(String), // hub identifier
Cloud(String),
}

impl RoutePolicy {
fn new(primary: Destination, secondary: Destination, primary_prob: f64) -> Self {
Self { primary_dest: primary, secondary_dest: secondary, primary_prob, rng: thread_rng() }
}

fn choose_dest(&mut self) -> Destination {
let r: f64 = self.rng.gen();
if r < self.primary_prob {
self.primary_dest.clone()
} else {
self.secondary_dest.clone()
}
}
}

// Usage in edge pipeline
fn route_telemetry(policy: &mut RoutePolicy, item: Telemetry) {
let dest = policy.choose_dest();
send_to(dest, item);
}

Note: In production, avoid cloning destinations; use refs and avoid allocations in hot path.

Resource-aware tuning tips:

Start with conservative defaults for primary_prob and burst capacity.
Expose metrics to adjust in real time without redeploying.
Consider device heterogeneity: lighter devices may have higher default routing to cloud to ensure reliability.

Measurable outcomes you can aim for
Latency:
- Edge-triggered alerts: aim for sub-50 ms processing, <100 ms end-to-end in good networks.
- Cloud routing only when necessary to reduce round-trips.
Bandwidth:
- Target 40-70% uplink reduction during normal operation by prioritizing essential data and local processing.
Data integrity:
- Reconciliation reliability > 99.99% across outage/reconnect cycles in tests.
Reliability:
- System remains within bounded memory usage under burst scenarios; watchdogs trigger safe shutdowns otherwise. ### Lessons learned (community-friendly takeaways)
Start simple, then layer probabilistic routing on top: a small, well-audited router module is easier to maintain than a complex, fully dynamic routing stack.
Local decision autonomy pays off: enabling edge devices to act on local data dramatically reduces latency and network load, but requires careful policy governance to avoid drift.
Verifiable reconciliation is non-negotiable for trust: without a clear audit trail, reconciliation under failures becomes fragile and error-prone.
Observability is a first-class feature: you can’t optimize what you can’t measure; invest in end-to-end tracing and per-item routing visibility.
Test with realistic outage scenarios: simulate long outages, intermittent connectivity, and “split-brain” conditions to understand edge-cloud interactions.

Practical example: a small rollout plan
Month 1: implement edge runtime and a single policy (bandwidth-preserving).
- Metrics to monitor: uplink usage, edge latency, queue depth.
Month 2: add probabilistic routing with cloud fallback and basic reconciliation.
- Metrics: reconciliation success rate, duplicate detections, state convergence time.
Month 3: broaden policy library, introduce canaries for new routes, improve observability dashboards.
Month 4+: scale to full fleet, refine thresholds based on telemetry and incident postmortems.

How to get started in your team
Define success metrics early: latency targets, bandwidth budgets, reconciliation SLAs.
Build a minimal viable edge runtime first, focusing on robust queues and a simple router.
Design for upgradeability: make policies hot-swappable and tuneable without firmware updates.
Invest in a clear reconciliation protocol from day one; treat edge logs as verifiable, append-only records.
Create playgrounds that simulate outages and measure real resilience, not just theoretical specs.
If you’d like, I can tailor this blueprint to your environment (device specs, network topology, preferred tech stack) and draft a concrete project plan with milestones, risk register, and a starter code repository scaffold.

Would you be open to a quick call or an async write-up to align on your gateway capabilities and the exact data contracts you’re targeting? I’m happy to adapt the design to your constraints and share more concrete code samples for your stack.

Rizwan Saleem | https://rizwansaleem.co

Designing a Resilient Event-Driven Data Pipeline with Change Data Capture

Rizwan Saleem — Thu, 04 Jun 2026 14:30:19 +0000

Designing a Resilient Event-Driven Data Pipeline with Change Data Capture

In modern data platforms, near-real-time insights often hinge on how quickly and reliably changes in source systems propagate downstream. A well-architected event-driven data pipeline that uses Change Data Capture (CDC) can provide low-latency data updates, robust exactly-once semantics, and simple audit trails. This tutorial walks through a practical, maintainable design for an event-driven data pipeline built around CDC, streaming, and modular data processing services. It includes an actionable blueprint, recommended components, and example code to help you implement in production.

Overview of the problem and goals

Goal: Build a scalable data pipeline that captures changes from a relational source (or a set of sources), streams those changes to downstream consumers, and provides reliable processing guarantees, observability, and fault tolerance.
Constraints:
- Low latency: minutes or seconds from source change to downstream availability.
- Reliability: fault-tolerant with at-least-once or exactly-once semantics where feasible.
- Observability: end-to-end visibility with tracing, metrics, and replay capability.
- Modularity: clean separation between CDC capture, streaming backbone, and processing jobs.

High-level architecture

CDC capture layer: Detects changes in source databases and emits change records.
Streaming backbone: Publishes change events to a scalable log-based system.
Processing/services layer: Downstream services consume events, perform enrichment, materialization, and analytics.
Storage and sinks: Maintain durable stores for history, snapshots, and queryable views.
Observability and operations: Centralized logging, metrics, traces, and a workflow for schema evolution and replay.

Key components and rationale

Change Data Capture (CDC) source
- Polling-based CDC or log-based CDC depending on database support.
- Keeps track of a replication slot/offset or binary log position to ensure incremental changes.
Streaming platform
- A durable, scalable log such as Apache Kafka, Kinesis, or Pulsar.
- Provides partitioning, offset management, and replay capabilities.
Processing layer
- Stream processors or microservices that subscribe to topics, apply business logic, and emit results to sinks.
- Choose between stateful stream processing (for aggregation, windowing) or stateless microservices with idempotent behavior.
Sinks and materialized views
- Data warehouses, data lakes, or operational databases to serve dashboards and BI.
- Consider maintaining a changelog or upsert-capable sink to support replays and corrections.
Governance and schema evolution
- Central schema registry and versioned schemas to handle evolving data models.
- Backward/forward compatibility strategies and migration plans.
Operational primitives
- Exactly-once or de-duplication mechanisms.
- Retries, backpressure handling, and dead-letter queues for unprocessable events.
- Monitoring, tracing, alerting, and runbook automation.

Detailed design and steps

1) Define the data domain and CDC strategy

Identify source systems: relational databases (PostgreSQL, MySQL), SaaS apps, or other data stores.
Decide on CDC approach:
- Database log-based CDC (preferred for latency and fidelity) using tools like Debezium, Maxwell, or native CDC connectors.
- Trigger-based polling if log access is unavailable, with a careful handling of, e.g., tombstones and deletes.
Model change events
- Each event should include: operation type (insert/update/delete), primary key, before/after images (as applicable), timestamp, and a transaction/offset identifier for ordering.
- Use a consistent envelope format, e.g.,:
- type: "insert"/"update"/"delete"
- database, table
- primary_key: { ... }
- before: { ... } or null
- after: { ... } or null
- ts: source timestamp
- _offset: CDC offset or transaction id

2) Establish the streaming backbone

Choose a durable log: Kafka is a common default due to ecosystem, but Kinesis or Pulsar are valid alternatives depending on team constraints.
Topic layout
- Per-table topics or a single topic with a table field, depending on cardinality and security requirements.
- Use compacted topics for changelog-like views where appropriate.
Partitioning strategy
- Partition by primary key hash to parallelize consumption while preserving in-order guarantees per key.
Exactly-once considerations
- Use idempotent producers and transactional writes if your platform supports it (e.g., Kafka transactions).
- Produce changelog events in a single, atomic batch where possible to reduce duplicates.

3) Processing layer design

Stateless microservices vs. stateful stream processors
- Stateless services: simple enrichment, routing, and materialization to sinks.
- Stateful processors: windowed aggregations, aggregations over keys, or joins with reference data.
Processing guarantees
- Exactly-once: use transactional sinks and idempotent operations in the processing layer.
- At-least-once with de-duplication: implement idempotency keys in downstream sinks and a dedupe window in the processor.
Processing patterns
- Enrichment: join events with reference data (e.g., dimension tables) loaded into a fast store (Redis, in-memory databases) or materialized in a stream-table abstraction.
- Materialization: write computed views to a sink (e.g., a data warehouse, a NoSQL store) for dashboards.
- Error handling: route unprocessable events to a dead-letter topic for manual inspection.

4) Sinks and materialized views

Operational databases or search indexes
- Used for serving queries with low latency; ensure you can handle upserts and deletes.
Data warehouse or lakehouse
- For analytics and long-term retention; support incremental loads and schema evolution.
Data quality checks
- Implement post-load validation with counters, row counts, and sampling to detect drift between source and sink.

5) Schema management and evolution

Use a centralized schema registry (e.g., Confluent Schema Registry) to manage Avro/JSON schemas.
Versioned schemas
- Backward-compatible changes (adding fields with defaults) allow rolling updates without breaking producers/consumers.
- Forward-compatible changes (removing fields) require careful handling and deprecation windows.
Migration plan
- Deploy schema updates in a controlled sequence: producers first, then consumers, with feature flags to switch parsing logic.

6) Observability and operations

End-to-end tracing
- Use distributed tracing (OpenTelemetry) to track events from source to sink.
Metrics
- Track lag between CDC and processing, event throughput, error rates, and processing latency.
Logging and alerting
- Centralize logs, set thresholds for lag, and alert on spikes or failures.
Replay and rollback
- Be able to replay a range of events from a given offset if a bug is discovered or a data correction is needed.

7) Security, compliance, and data governance

Access control
- Least privilege for producers and consumers; encrypt data in transit and at rest.
Data redaction and PII handling
- Mask or tokenize sensitive fields where appropriate; apply field-level security in the processing layer.
Compliance
- Maintain audit trails for changes, provide data lineage, and enforce data retention policies.

8) Operational workflow and recovery planning

Deployment
- Use blue/green or canary deployments for critical components to minimize risk.
Failure modes
- CDC source failure: implement retry/backoff with circuit breakers; fall back to a safe paused state.
- Streaming failure: monitor consumer lag and partition health; automatically reassign partitions if needed.
- Sink outages: implement buffering with backpressure and dead-letter queues.
Disaster recovery
- Regularly back up important state stores; keep an immutable log of events for rehydration.

Practical implementation example

Scenario: A PostgreSQL database is the source, Kafka is the streaming backbone, and a downstream analytics service materializes a customer activity view into a data lake (S3) and a fast, queryable cache (Redis). We’ll outline a minimal, working example using popular open-source tools.

Stack

Source: PostgreSQL with Debezium for CDC
Stream: Apache Kafka
Processor: Kafka Streams (or Apache Flink) for enrichment and materialization
Sinks: Amazon S3 (lakehouse-like data lake) and Redis (fast lookup)
Schema: Avro with Confluent Schema Registry
Observability: OpenTelemetry, Prometheus, Grafana

Step-by-step guide

1) Set up CDC on PostgreSQL with Debezium

Enable logical replication on PostgreSQL and create a replication user.
Run Debezium CDC connector pointing to PostgreSQL, configured to produce change events to a Kafka topic, e.g., dbserver1.public.customer_changes.
Ensure the Debezium connector outputs a consistent envelope with fields like before and after, op, and ts.

2) Create a Kafka topic strategy

Topic: customer_changes
Partitions: based on customer_id modulo partitions
Retention: long enough to cover expected replay windows (e.g., 7 days) plus some headroom

3) Implement a Kafka Streams processing job

Read from customer_changes
Apply business logic: for example, compute a derived field like last_seen, and join with a static reference dataset (customer_segments) loaded from a small in-memory store or a compacted topic
Materialize a target topic: customer_view
Persist to sink
- Write a parquet/gzipped files to S3 for analytics
- Update Redis with a real-time view for fast lookups

Java/Scala snippet (simplified)

Kafka Streams topology outline (pseudo):

KStream source = builder.stream("customer_changes", Consumed.with(Serdes.String(), jsonSerde));

KTable segments = builder.table("customer_segments", Consumed.with(Serdes.String(), jsonSerde));

KStream enriched = source
.filter((k,v) -> v.get("op").asText() != "d") // filter deletes if desired
.join(segments, (change, seg) -> {
ObjectNode out = JsonNodeFactory.instance.objectNode();
out.set("after", change.get("after"));
out.set("segment", seg);
return out;
}, Joined.with(Serdes.String(), jsonSerde, jsonSerde))
.mapValues(v -> {
// compute derived fields
((ObjectNode) v).put("last_seen", System.currentTimeMillis());
return v;
});

enriched.to("customer_view", Produced.with(Serdes.String(), jsonSerde));

4) Materialize to S3

Use a sink connector or a small Spark job that reads customer_view from Kafka (or a parquet writer streaming) and writes daily/hourly partitions to S3 as parquet.
Ensure schema compatibility across writes; use a stable Avro schema for the parquet files.

5) Populate Redis for fast reads

A separate consumer subscribes to customer_view and writes the latest state for key customer_id into Redis:
- Key: customer:
- Value: JSON or a compact struct with essential fields
Implement a TTL if appropriate to avoid stale data; ensure eventual consistency with the lakehouse.

6) Observability

Instrument producers and consumers with OpenTelemetry traces that propagate through Kafka and the processing job.
Expose metrics like:
- Ingested events per second
- Processing latency
- Lag between CDC and processing per partition
- S3 write throughput and failure rate
Set up dashboards in Grafana for end-to-end visibility.

7) Handling schema evolution

Register both source and destination schemas in the registry.
Add fields with defaults for non-breaking changes.
When removing fields, phase them out with a deprecation period and update all producers/consumers in a controlled rollout.

8) Testing and validation

Unit tests for individual processing steps with mock input events.
End-to-end integration tests using a local CDC source and a test Kafka cluster.
Fuzz tests for schema changes and message drift.
Replay tests: store a known offset range and verify that reprocessing yields identical final state.

9) Security and governance

Use TLS for all connections; enforce authentication between components.
Limit access to Kafka topics with ACLs and to Redis/AWS resources with IAM roles.
Keep an audit log of CDC changes and data edits for compliance.

When to use this architecture

Real-time dashboards that need fresh data with low latency reports.
Scenarios requiring a single source of truth with a robust audit trail.
Environments where data quality and schema evolution must be managed rigorously.

Common pitfalls and how to avoid them

Pitfall: At-least-once processing causing duplicates.
- Solution: Use idempotent sinks, deduplication windows, and transactional producers where supported.
Pitfall: Schema drift breaking downstream consumers.
- Solution: Enforce backward-compatible schema changes and a strict release plan for schema evolution.
Pitfall: Lag growing unbounded under backpressure.
- Solution: Implement backpressure-aware processing, auto-scaling, and robust DLQ handling.

Illustrative analogy

Think of CDC as a meticulous librarian watching a shelf for every book moved, added, or removed. The streaming backbone is the courier network delivering change cards in real time. The processing layer is a team of librarians who read each card, enrich it with context, and place it into a new, well-organized cabinet (the data lake or a cache) where analysts can retrieve up-to-date information quickly. Observability instruments act like a security camera and inventory system, ensuring we can trace every change from origin to its final resting place.

If you’d like, I can tailor this blueprint to your tech stack (e.g., AWS native services, Google Cloud, Azure, or a Kubernetes-based setup) and provide a more concrete code sample using your preferred language and tooling. Would you like a version aligned to a particular cloud provider or a pure open-source stack?

Rizwan Saleem | https://rizwansaleem.co

Building a Personal Knowledge Graph for Software Engineers

Rizwan Saleem — Thu, 04 Jun 2026 13:50:20 +0000

Building a Personal Knowledge Graph for Software Engineers

A personal knowledge graph (PKG) is a lightweight, structured map of your technical knowledge, projects, and learning intent. It helps you organize concepts, track skills, and connect ideas across domains-accelerating learning, career planning, and collaboration. This tutorial walks you through designing, implementing, and using a PKG tailored for software engineers.

Why a PKG matters for engineers

Smoother onboarding: quickly connect new tools, stacks, and domain concepts to existing knowledge.
Clear learning path: see gaps, set learning goals, and measure progress.
Better decision making: evaluate tech choices by mapping tradeoffs and dependencies.
Collaboration power: share a succinct map with teammates, mentors, or recruiters.

A PKG isn’t a static repo of bookmarks; it’s a living graph that evolves with your career.

Core concepts

Nodes: The building blocks of your PKG. Types include: concepts, skills, projects, articles, tools, patterns, and goals.
Edges: Relationships between nodes. Examples: “learns,” “implements,” “depends-on,” “used-in,” “influences,” “replaces.”
Attributes: Properties on nodes (level, proficiency, date learned, difficulty, confidence, notes).
Projections: Views or slices of the graph focused on a domain (e.g., “Frontend Performance,” “Distributed Systems,” “Machine Learning for Engineers”).
Provenance: Track sources and reasoning: links to articles, snippets, or code.

Think of it as a directed, labeled multigraph with metadata attached to nodes and edges.

Step 1: Choose your data model and storage

Option A: Lightweight local graph

Data format: JSON or YAML
Storage: plain files in a git repo for version control
Pros: simple, portable, offline
Cons: limited querying, manual wiring needed for complex relationships

Option B: Embedded graph database

Data format: Graph database (e.g., Neo4j, Dgraph, or an embedded option like ArangoDB) or a lightweight graph store
Storage: local or remote server
Pros: fast traversals, complex queries, scalable
Cons: setup/maintenance overhead

Option C: Hybrid

Use JSON/YAML for author’s primary knowledge and a graph tool for advanced queries (e.g., a local Neo4j instance with a syncing mechanism)

For many engineers starting out, a hybrid approach works well: store nodes and edges as structured YAML in a git repo, and optionally mirror to a small graph database for advanced querying later.

Suggested starter schema (conceptual):

Node types: Concept, Skill, Tool, Project, Article, Pattern, Goal
Edges: learns, uses, teaches, related-to, depends-on, implemented-in, example-of, planned-for

Example snippet (YAML):
concepts:

id: functional-programming name: Functional Programming description: "A paradigm that treats computation as the evaluation of mathematical functions and avoids changing state." level: 2 skills:
id: polyglot-programming name: Polyglot Programming description: "Ability to work across multiple languages and paradigms." level: 3 projects:
id: reactive-dashboard name: Reactive Dashboard description: "A dashboard built with reactive streams and WebSockets." technologies: [react, rx, websockets]

edges:

from: functional-programming to: polyglot-programming relationship: related-to
from: polyglot-programming to: reactive-dashboard relationship: implemented-in
from: reactive-dashboard to: frontend-performance relationship: depends-on ### Step 2: Define a minimal, extensible schema for starting

Start with a small set of node types and a couple of relationships. You can always grow the graph.

Node types (essential)
- Concept: abstract ideas or domains
- Skill: concrete abilities (e.g., "Rust memory safety," "CI/CD pipelines")
- Tool: software or platforms (e.g., "Docker," "Kubernetes")
- Project: actual work items or experiments
- Article/Note: reading notes or references
- Goal: short- and mid-term career objectives
Relationship types (essential)
- learns: Skill/Concept learned
- uses: Tool used in a Skill/Project
- implemented-in: Project implements a Skill
- depends-on: Skill/Tool requirement
- related-to: Concept-to-Concept connections
- references: Article/Note references

Example starter data (conceptual):

Node: Concept - Microservices
Node: Skill - API design
Node: Tool - Docker
Node: Project - Shopping-cart microservice
Node: Article - "Domain-Driven Design in Practice"

Edges:

Microservices related-to API design
API design depends-on Docker
Shopping-cart microservice implements API design
Shopping-cart references Domain-Driven Design in Practice ### Step 3: Set up a minimal repo structure

If you go with a local YAML/JSON approach, a compact repo helps you version, review, and share.

.pkg/
- concepts/
- functional-programming.yaml
- skills/
- polyglot-programming.yaml
- tools/
- docker.yaml
- projects/
- reactive-dashboard.yaml
- articles/
- fp-notes.md
- edges.yaml
- goals/
- learn-rust-in-2026.yaml

Example edge entry (edges.yaml):
from: functional-programming
to: polyglot-programming
relationship: related-to

Keep data human-friendly: use IDs, names, and short descriptions. Use dates for learning milestones.

Step 4: Create practical workflows

Daily micro-updates: add one small node or update an edge after a learning session (e.g., “read article X,” “implemented Y in Z project”).
Weekly review: prune stale edges, reassess goals, and highlight new domains to explore.
Milestone mapping: align PKG with career goals (e.g., “Be proficient in distributed systems within 12 months”). Attach a plan edge like goal-achieved with date.

Sample weekly ritual:

Pick one new concept or tool to add as a node.
Create at least two edges that connect it to existing nodes.
Update a goal with progress metrics. ### Step 5: Practical code examples

This example shows a small Python utility to add and query a YAML-based PKG.

File: pkg_cli.py
Dependencies: PyYAML

Code (conceptual, Python 3.9+):

import sys
import yaml
from pathlib import Path
from datetime import date

BASE = Path(".pkg")

def load(pkg_path):
path = BASE / pkg_path
if not path.exists():
return {}
with open(path, "r", encoding="utf-8") as f:
return yaml.safe_load(f) or {}

def save(pkg_path, data):
path = BASE / pkg_path
path.parent.mkdir(parents=True, exist_ok=True)
with open(path, "w", encoding="utf-8") as f:
yaml.safe_dump(data, f, sort_keys=False)

def add_concept(concept_id, name, description):
data = load("concepts/{}.yaml".format(concept_id))
data.update({"id": concept_id, "name": name, "description": description})
save("concepts/{}.yaml".format(concept_id), data)

def add_edge(from_id, to_id, relation):
edges = load("edges.yaml") or []
edges.append({"from": from_id, "to": to_id, "relationship": relation})
save("edges.yaml", edges)

def main():
# simple CLI demo: python pkg_cli.py add-concept fp "Functional Programming" "A paradigm..."
if len(sys.argv) < 2:
print("Usage: python pkg_cli.py [args]")
return
cmd = sys.argv
if cmd == "add-concept":
add_concept(sys.argv, sys.argv, sys.argv)
elif cmd == "add-edge":
add_edge(sys.argv, sys.argv, sys.argv)
else:
print("Unknown command")

if name == "main":
main()

Usage examples:

python pkg_cli.py add-concept functional-programming "Functional Programming" "A paradigm..."
python pkg_cli.py add-edge functional-programming polyglot-programming related-to

Note: This is intentionally simple. As your PKG grows, you can switch to a graph database or add richer metadata and validation.

Step 6: Build useful views or reports

Even a small PKG benefits from focused views. Here are a few practical ones you can implement with minimal tooling:

Proficiency snapshot: list of skills with Level and last-learned date
- Output: a table or markdown list
Learning plan by quarter: identify concepts/tools to learn in the next 12 weeks
- Output: a plan table with goals, prerequisites, and success criteria
Dependency map: show which skills depend on others
- Output: a simple graph visualization (you can export to Graphviz DOT or use a JS library for a web view)
Project map: connect projects to the skills they exercise
- Output: a dashboard of projects vs. skills

Example DOT graph snippet (for Graphviz):
digraph PKG {
"Functional Programming" -> "Polyglot Programming" [label="related-to"];
"API Design" -> "Shopping-cart microservice" [label="implemented-in"];
"Docker" -> "Shopping-cart microservice" [label="used-in"];
}

This can render a visual map to inspire learning paths.

Step 7: Integrate with your daily tools

Notes apps: Link notes to corresponding PKG nodes for quick cross-referencing.
IDEs: Some editors can open a PKG as a lightweight knowledge sidebar (e.g., a Markdown/JSON view in VS Code).
Version control: Store the PKG in a dedicated branch or repo; use PRs to review changes, reinforcing deliberate learning.

If you prefer a lightweight browser-based view, you can build a small static site that reads the YAML data and renders interactive graphs with a library like D3.js or Vis.js. For a quick start, export edges to a Graphviz DOT file and view it with any Graphviz tool.

Illustration: Simple PKG view concept

Node: Microservices
- Skills: API design, distributed tracing
- Tools: Docker, Kubernetes
- Projects: Shopping-cart service
- Related Articles: Domain-Driven Design in Practice
Edges connect as: Microservices -depends-on→ API design; Shopping-cart service -uses→ Docker; API design -related-to→ Distributed tracing

This mental map helps you see how improvements in API design ripple through projects and tooling.

Step 8: Maintain quality and guardrails

Keep IDs stable: once you create a node ID, avoid renaming it; update descriptions or add new edges instead.
Be explicit about provenance: store a citation or reference for each concept or claim you add (e.g., article URL, date learned).
Prioritize readability: write clear descriptions; a PKG should be understandable to future you or a colleague.
Regular cleanup: quarterly prune of obsolete edges and consolidation of duplicates. ### Step-by-step plan to start today

1) Pick storage: YAML-based PKG in a Git repo (local-first, easy to iterate).
2) Define the first 6 node types and 6 relationship types.
3) Create a starter dataset: 5 concepts, 5 skills, 3 tools, 2 projects, 2 articles.
4) Implement a tiny CLI (or use the sample Python script) to add nodes and edges.
5) Build a simple view for yourself: export a Markdown summary of your current PKG.
6) Schedule a 30-minute weekly PKG session to add 1 concept, 1 edge, and update a goal.

Example starter dataset (illustrative)

Concepts
- id: functional-programming name: Functional Programming description: A paradigm focused on composing pure functions and avoiding shared state. level: 2
- id: distributed-systems name: Distributed Systems description: Systems designed to run on multiple machines with reliability and consistency. level: 2
Skills
- id: polyglot-programming name: Polyglot Programming description: Write code across multiple languages without paralysis. level: 1
- id: api-design name: API Design description: Designing robust, usable, and scalable interfaces. level: 2
Tools
- id: docker name: Docker description: Containerization for reproducible environments.
- id: kubernetes name: Kubernetes description: Orchestrates containers at scale.
Projects
- id: shopping-cart-ms name: Shopping Cart Microservice description: A small microservice illustrating clean API boundaries.
Articles
- id: dd-in-practice name: Domain-Driven Design in Practice url: https://example.org/dd-in-practice
Edges
- from: functional-programming to: polyglot-programming relationship: related-to
- from: polyglot-programming to: shopping-cart-ms relationship: implemented-in
- from: docker to: shopping-cart-ms relationship: used-in
- from: api-design to: shopping-cart-ms relationship: depends-on
- from: dd-in-practice to: api-design relationship: references If you’d like, I can tailor a PKG template to your current tech stack and goals (e.g., frontend-heavy, backend, or data engineering focus) and generate a ready-to-run YAML/JSON scaffold plus a tiny CLI script. Tell me your preferred focus area and whether you want a local-first setup or one that scales with a lightweight graph database.

Would you like me to draft a starter PKG tailored to your Carlisle, England context and your current tech stack, with an initial 5 concepts, 5 skills, and 3 projects?

Rizwan Saleem | https://rizwansaleem.co

Sources

https://example.org/dd-in-practice

Building a Developer-Friendly Automation Studio: End-to-End Local Pipelines for Modern CI/CD

Rizwan Saleem — Thu, 04 Jun 2026 13:30:20 +0000

Building a Developer-Friendly Automation Studio: End-to-End Local Pipelines for Modern CI/CD

Automated workflows aren’t just about pushing code; they’re about turning repetitive toil into reliable, visible processes that you can reason about, extend, and share. This guide gives you a practical blueprint to design, implement, and maintain a developer-friendly automation studio: a local-first, extensible pipeline system that you can grow with your team’s needs, from quick checks to full-blown release pipelines.

Illustrative goal

You want a reproducible, fast feedback loop from code changes to verifiable results.
You want pipelines that are easy to understand for developers across disciplines.
You want to minimize bake-time for common tasks and maximize trust in automation.

What you’ll build

A lightweight local automation studio powered by a small, self-contained orchestrator.
A set of recipe modules for common tasks: linting, type checks, unit tests, integration tests, build artifacts, and deployment previews.
A simple UI (CLI-first with optional web dashboard) to view, run, and monitor pipelines.
A “local-first” philosophy: run pipelines locally, with knobs to mirror CI behavior.

Overview of the architecture

Orchestrator: The brain. It discovers, schedules, and tracks jobs. Keeps a deterministic pipeline graph.
Tasks: Small, composable units that do one thing well (lint, test, build, publish, etc.).
Runners: Engines that execute tasks in isolation (local subprocesses, containers, or VM-like sandboxes).
Configuration: Declares pipelines as recipes in YAML or TOML, with reusable templates.
State store: Keeps logs, artifacts, and metadata locally; optionally syncs to a remote store for sharing.
UI layer: CLI commands and an optional minimal web UI to observe pipeline runs.

Step 1: Define the requirements and success criteria

Local-first operation: can run without network access; caches results to speed up re-runs.
Observability: each task emits structured logs, exit codes, and artifacts.
Modularity: add or remove tasks without rewriting pipelines.
Reproducibility: deterministic environments per task (version pins, lockfiles, container images).
Extensibility: easy to add new task types or integrate with existing tooling.

Step 2: Choose the core primitives

Task: a function-like unit with inputs, outputs, and idempotent behavior.
Pipeline: an ordered graph of tasks with dependencies.
Runner: a isolated process runner that executes a task and reports status.
Context: a workspace context containing repository paths, env vars, and caches.
Config: human-readable YAML with reusable templates.

Pseudo-code sketch of a task interface

Task(name, run_fn, inputs, outputs, env)
run_fn(context, inputs) -> (success, outputs, logs)

Example tasks you’ll implement

lint: runs eslint/flake8/ruff, collects style diagnostics
type-check: runs TypeScript or Python type checkers
unit-test: executes unit tests with coverage
integration-test: spins up lightweight services and verifies APis
build-artifact: compiles/bundles artifacts (e.g., a Docker image or a static bundle)
publish-preview: publishes a temporary preview URL or artifact to a staging area
notify: sends a summary to your chat or email

Step 3: Create a minimal local orchestration engine

Language choice: Python is beginner-friendly and has strong tooling; Node.js works well if you’re JS-heavy. For speed/portability, a small Rust or Go runner can be employed later.
Core components:
- Graph builder: parses pipeline YAML into a DAG and resolves dependencies.
- Ticketing: each task run is a “ticket” with status, start/end times, and logs.
- Executor: runs tasks in sequence respecting dependencies; supports parallelism where possible.
- Caching: memoizes task outputs keyed by inputs + environment to avoid rework.

A compact Python example (high level)

You would typically store pipelines in pipelines.yaml:
- name: ci tasks:
  - lint
  - type_check
  - unit_test
  - build_artifact
- name: deploy_preview depends_on: [ci] tasks:
  - publish_preview

Minimal runner sketch (conceptual)

def run_pipeline(pipeline, context):
- for task_name in pipeline.sequence respecting dependencies:
- if task_already_ran_and_cached(task): continue
- status = run_task(task, context)
- if not status.success: halt and report

Step 4: Build reusable task templates

Template approach: declare common constraints per language/runtime; then fill in specifics.
Example templates:
- lint_template(language): runs language-appropriate linters with strict rules
- test_template(framework, command): runs tests with coverage, failing on warnings if configured
- build_template(target): creates artifacts, with deterministic build steps and versioning

Concrete examples

Python lint task (ruff + black check)
- Command: ruff check exit-zero; black check .
- Outputs: lint-report.json
Type-check task (mypy)
- Command: mypy src explicit-package-bases
- Outputs: mypy-report.json
Unit tests (pytest with coverage)
- Command: pytest maxfail=1 disable-warnings cov=src
- Outputs: coverage.xml, test-results.json

Step 5: Configure environments and reproducibility

Pin tooling versions explicitly:
- Use poetry/pyproject.toml or pip-tools for Python; package.json/volta/nvm for Node. Use containerized runners for consistent environments when needed.
Lockfiles:
- Commit lockfiles to repo to guarantee identical dependencies.
Environment isolation:
- Each task runs in its own virtual environment or isolated container.
Artifact naming:
- Tag artifacts with pipeline run-id and version to avoid confusion.

Step 6: Implement a minimal UI and UX

CLI first approach
- Commands:
- studio run ci
- studio list
- studio logs
- studio artifacts
- Features:
- Dry-run mode to validate pipeline syntax without executing
- Parallel execution where tasks have no dependencies
Optional lightweight web dashboard
- Shows pipeline graphs, recent runs, status, and links to logs
- Can be a small static site and a REST API

Step 7: Add observability and failure handling

Structured logs:
- Each task outputs JSON lines with timestamp, level, message, and fields (task, run-id, artifacts).
Retriability:
- Support limited retries with exponential backoff for flaky steps (e.g., network calls).
Notifications:
- Slack/Teams/Email when a pipeline fails, with a link to logs.
Artifacts and traces:
- Store artifacts in a local artifacts/ directory; optionally push to a remote store for sharing.

Step 8: Design for scalability and maintenance

Module registry:
- Keep a directory like tasks/ with well-documented interfaces.
- Allow third-party contributors to publish new tasks as plugins.
Versioned pipelines:
- Pipelines can declare a version; pipelines evolve without breaking older runs.
Test harness:
- Include a small suite of integration tests for the orchestration engine itself.

Step 9: Real-world example: a sample ci pipeline

Pipelines.yaml (simplified)
- pipelines:
- name: local-ci description: Local quick feedback cycle stages:
  - lint
  - type_check
  - unit_test
  - build_artifact
- name: preview depends_on: local-ci stages:
  - publish_preview
Tasks definitions (pseudo):
- lint:
- run: ["ruff", "check", "."]
- on_error: halt
- type_check:
- run: ["mypy", "src"]
- unit_test:
- run: ["pytest", "maxfail=1", "disable-warnings", "cov=src"]
- build_artifact:
- run: ["npm", "run", "build"] or ["python", "setup.py", "bdist_wheel"]
- publish_preview:
- run: ["deploy-preview", "artifact", "build/artifact.zip"]

Step 10: Step-by-step plan to implement in your stack
Phase 1: MVP

Build a small Python-based runner with the ability to execute simple shell commands per task.
Create a YAML config for a single pipeline with three tasks: lint, unit_test, and build_artifact.
Implement basic logging, run tracking, and a CLI to trigger runs and view logs.
Ensure re-runs reuse cached results when inputs haven’t changed.

Phase 2: Improve reliability

Add environment isolation via venvs or subprocess namespaces.
Implement a simple dependency graph resolver to ensure tasks run in the correct order, with parallelism where possible.
Introduce artifacts directory and a basic artifact naming convention.

Phase 3: Observability and UX

Add structured JSON logs, a log viewer in CLI, and a minimal web dashboard.
Implement basic notifications on failure or success.
Create a small plugin system to add new tasks without editing core engine.

Phase 4: Extensibility

Build a task registry and plugin API.
Add templates for common languages (JS/TS, Python, Go) to accelerate setup.
Prepare a sample project demonstrating cross-language pipelines.

Tips and best practices

Start small: first automate a single CI-like pipeline for your repo, then expand.
Favor idempotent tasks: ensure repeated runs don’t cause unintended side effects.
Store secrets securely: use your environment’s secret manager or a local vault; avoid hard-coding credentials.
Document pipelines: keep pipeline definitions in version control with clear comments.
Keep artifacts small and meaningful: only store what you’ll need for debugging or review.

Common pitfalls and how to avoid them

Overcomplicating the first version: a simple, reliable MVP beats a perfect but heavyweight system.
Non-deterministic tasks: avoid tasks that depend on wall-clock time or random network variability unless you can parametrize and stabilize them.
Hidden dependencies: ensure the DAG clearly specifies all inputs/outputs so re-runs are predictable.

What an example run looks like

You run: studio run local-ci
You’ll see a sequence: lint → type_check → unit_test → build_artifact
Each step prints structured logs; on success you see an artifact path; on failure you see a failure reason and a link to logs.

A quick starter repository layout

studio/
- pipelines.yaml
- tasks/
- lint.py
- type_check.py
- unit_test.py
- build_artifact.py
- runners/
- executor.py
- ui/
- cli.py
- web_dashboard/
- config/
- pyproject.toml (or package.json/yarn.lock, depending on stack)
- logs/
- artifacts/

Optional: integrating with existing ecosystems

If you already use GitHub Actions, you can mirror or supplement workflows with your local studio to speed up development and local testing.
Use the same coding standards and linting rules across tasks to reduce cognitive load.

Conclusion

A developer-friendly automation studio empowers you to bring consistency, speed, and clarity to your workflow. By focusing on a local-first, modular architecture with clear pipelines and observability, you enable faster feedback loops, easier onboarding, and better collaboration across teams. Start with a minimal MVP, iterate on reliability and UX, and grow with a plugin-friendly design that scales as your project and team do.

Would you like a concrete starter repository template in Python (with a minimal CLI and YAML pipeline) that you can clone and adapt to your project stack? If so, tell me your preferred language runtime (Python, Node.js, or a language you prefer), and I’ll tailor a ready-to-run starter complete with example tasks and a simple web dashboard.

Rizwan Saleem | https://rizwansaleem.co

Building Accessible Frontend Components: A Practical Guide to Inclusive UI Patterns

Rizwan Saleem — Thu, 04 Jun 2026 09:50:20 +0000

Liquid syntax error: Variable '{{% raw %}' was not properly terminated with regexp: /\}\}/