DEV Community: Xusheng Cao

Building a Real-Time Customer Support System in .NET - Prologue: Architecture, Constraints, and Engineering Decisions

Xusheng Cao — Thu, 26 Feb 2026 15:20:55 +0000

Most people think a customer support system is just a chat box.

It isn’t.

Behind that small floating widget lies a real-time messaging core that must deliver messages with low latency under unpredictable traffic. There is concurrency control to manage thousands — sometimes hundreds of thousands — of simultaneous connections. There is memory management under pressure, where inefficient allocation patterns can quietly introduce latency spikes. There is multi-tenant isolation to ensure that one customer’s workload never impacts another’s stability. There are deployment strategies that differ fundamentally between SaaS and on-premises environments. There are reliability guarantees that determine whether a message is delivered once, at least once, or lost under edge conditions. And beneath all of that, there are long-term architectural trade-offs that shape how the system evolves over years, not weeks.

What appears simple on the surface is, in reality, a layered system balancing performance, isolation, reliability, and maintainability. The interface may look minimal, but the engineering beneath it is anything but.

Over the past few years, I built a real-time customer support system called ShenDesk. It supports both SaaS and on-premises deployment, and it is designed around high concurrency, maintainability, and architectural clarity. What started as a minimal prototype gradually evolved into a production-grade system with a real messaging pipeline, tenant isolation, visitor tracking, extensible APIs, and a structure designed to adapt rather than accumulate accidental complexity.

This series is a technical record of that evolution — the decisions, the constraints, the revisions, and the lessons learned while building and refining a real-time infrastructure system from the ground up.

Why Write This Series?

Engineering transparency builds trust — especially in infrastructure-level software.

There are many products in the customer support space, including established platforms such as Zendesk and Intercom. They are mature systems, backed by significant resources, and refined through years of iteration. From the outside, they present clean interfaces and well-designed user experiences.

What often remains invisible, however, is the engineering reasoning that makes such systems viable at scale. The UI may appear straightforward, but the infrastructure beneath it must operate continuously under load, tolerate bursts of traffic, isolate tenants reliably, and evolve without collapsing under its own complexity. Those design choices — the trade-offs, constraints, and revisions — rarely surface in public discussions.

How do you design a real-time messaging core in .NET that remains stable under sustained concurrency? How do you manage six-figure WebSocket connections without introducing latency spikes? How do you reduce GC pressure in a system where allocation patterns directly influence responsiveness? What does multi-tenant isolation look like beyond theoretical diagrams? And what architectural compromises emerge when enterprise clients require on-premises deployment instead of pure SaaS delivery?

These questions are not answered by feature lists. They are answered through architectural thinking shaped by constraints.

This series will explore those questions from an engineering perspective. It will examine decisions that seemed reasonable at the time but later required revision. It will discuss trade-offs between simplicity and scalability, between abstraction and performance, and between immediate delivery and long-term maintainability.

It is not a marketing campaign, and it is not a step-by-step tutorial that hands out production-ready code. Instead, it is a structured exploration of architectural decisions, constraints encountered in practice, and lessons learned while building and evolving a real-time system over time.

What This Series Will Cover

This series will follow the system’s evolution from its earliest assumptions to its more mature architectural form. It will begin with the fundamental question of why building a customer support system from scratch made sense at all — and what constraints justified that decision. From there, it will move through the transition from a minimal prototype to a production-grade architecture, examining how early simplicity gradually gave way to structural rigor.

The discussions will cover the design of a real-time messaging core in .NET and the practical realities of handling high-concurrency WebSocket connections under sustained load. They will explore memory allocation patterns and GC behavior in latency-sensitive environments, not as theoretical topics but as forces that directly influence user experience. Stability mechanisms such as backpressure, queue management, and failure isolation will be examined in the context of preventing cascading breakdowns.

As the system expanded, architectural concerns shifted toward multi-tenant isolation, deployment flexibility, and reliability guarantees. Supporting both SaaS and on-premises environments introduced different operational constraints and forced explicit decisions about configurability, automation, and environmental assumptions. Messaging reliability models — and the compromises between theoretical guarantees and operational practicality — became central considerations.

Later articles will reflect on rewriting core components when early design decisions proved insufficient, designing extensible APIs without over-engineering abstractions, and integrating AI capabilities into a latency-sensitive, real-time pipeline without destabilizing the underlying system.

Each article will focus on engineering reasoning rather than surface-level features. Where appropriate, I will discuss trade-offs, rejected alternatives, and architectural compromises — because real systems are shaped more by constraints than by ideals. Architectural diagrams may suggest clarity, but clarity often emerges only after revision.

This series is not organized around what a product can do. It is organized around how a system survives growth, load, and time.

The planned topics include:

Why Build a Customer Support System from Scratch?
From MVP to Production Architecture
Designing a Real-Time Messaging Core in .NET
Handling High-Concurrency WebSocket Connections
Memory Optimization and GC Pressure in Real-Time Systems
Backpressure and Stability in Messaging Pipelines
Multi-Tenant Architecture: Logical vs Physical Isolation
Designing for On-Premises Deployment
Message Reliability and Delivery Guarantees
Lessons from Rewriting Core Components
Open API Design for Extensibility
Integrating AI into a Real-Time Support System
Each article will focus on engineering reasoning rather than surface-level features.

Where appropriate, I will discuss trade-offs, rejected alternatives, and architectural compromises — because real systems are shaped more by constraints than by ideals.

Scope and Boundaries

This series will focus on architecture, design thinking, and engineering lessons drawn from building and evolving a real-time system in production. The emphasis will be on reasoning: why certain decisions were made, what constraints shaped them, and how those decisions held up over time.

It will not disclose sensitive security details, customer data, deployment-specific configurations, or internal proprietary implementation mechanisms. Real systems operate within operational contexts that cannot — and should not — be exposed publicly. Stability and security are not theoretical concerns; they are practical responsibilities.

Where examples are provided, they will be abstracted to illustrate patterns rather than reveal internal structure. Benchmarks may be discussed in directional terms rather than exact production numbers. Architectural diagrams, if included, will represent conceptual models rather than literal infrastructure layouts.

The intention is to share thinking, not to expose operational risk. Engineering transparency does not require publishing every line of code or every deployment detail. It requires clarity about the decisions that shape a system and honesty about the trade-offs that accompany them.

Boundaries are not limitations of the discussion; they are part of responsible engineering practice.

Who This Series Is For

This series may be useful if you are building real-time systems where latency, concurrency, and reliability are not abstract concerns but daily engineering constraints. It may resonate with you if you are designing SaaS infrastructure and finding that architectural decisions rarely remain isolated — they ripple outward into deployment models, tenant isolation, operational complexity, and long-term maintainability.

If you are exploring multi-tenant architecture beyond theoretical diagrams, or working with WebSocket-heavy workloads where connection lifecycle management and memory behavior directly affect system stability, the topics discussed here may align with the problems you are facing. Likewise, if you care about designing systems that remain understandable and adaptable after years of iteration — not just weeks of development — this series is written with that perspective in mind.

This is not content optimized for speed of consumption. It assumes a willingness to think in terms of trade-offs rather than absolutes, and in terms of evolution rather than quick solutions. The focus is not on feature comparison, but on structural reasoning. It is not about building something that works once, but about building something that continues to work as complexity grows.

If you are simply looking for a quick “how to build a chat app in 10 minutes” guide, this will not be that. There are many excellent resources for rapid prototyping. This series is concerned with what happens after the prototype begins to encounter real traffic, real constraints, and real operational responsibility.

A Long-Term Engineering Record

Software systems evolve.

They accumulate complexity as features are added and edge cases surface. They encounter unexpected bottlenecks that were invisible in early designs. They force you to revisit assumptions that once seemed entirely reasonable. What appears clean in an architectural diagram often becomes layered and negotiated in production reality.

No system remains identical to its first version. Concurrency models are adjusted. Data structures are replaced. Message pipelines are rewritten. Deployment workflows are simplified — and sometimes complicated again — as new requirements emerge. Over time, the challenge is no longer just making the system work, but ensuring that it continues to work without collapsing under accumulated decisions.

This series is an attempt to document that journey — not as a polished success story, but as an ongoing engineering process. It will include revisions, structural corrections, and moments where initial confidence gave way to deeper understanding. Stability is rarely achieved through a single correct decision; it is usually the result of iteration, constraint, and refinement.

In infrastructure-level SaaS systems, evolution is not optional. Real traffic exposes hidden weaknesses. Real customers introduce requirements that reshape architecture. Real operational responsibility changes how trade-offs are evaluated. Over time, engineering becomes less about adding capabilities and more about preserving clarity while adapting to change.

If you are interested in real-time architecture, distributed design trade-offs, or the realities of building and maintaining systems that must operate continuously, you may find value in following this record.

Closing Notes

If you find these discussions useful, I welcome your thoughts, corrections, and alternative perspectives. Engineering benefits from dialogue, and thoughtful disagreement often sharpens architectural clarity. Thank you in advance to those who choose to follow this series and engage with it seriously.

ShenDesk is not a conceptual project. It is a real product, used in production environments by real users. The ideas discussed here are grounded in practical implementation rather than theoretical modeling. If you are curious about the product itself, you can learn more at:

https://www.shendesk.com

Conversations around system design are rarely complete within a single article. If you are building similar infrastructure, facing comparable trade-offs, or simply interested in discussing real-time architecture in depth, I would be glad to connect and exchange ideas.

Engineering is rarely a solitary endeavor — even when the system is built by a single developer.

From GC Jitter to Stable Low-Latency: High-Performance Optimization with Span and Memory in ShenDesk

Xusheng Cao — Tue, 24 Feb 2026 07:27:27 +0000

Background

ShenDesk is an enterprise-grade, real-time customer service and visitor behavior tracking system, supporting both SaaS and On-Premises deployments. Built on the .NET ecosystem, its core architecture features high-concurrency WebSocket channels, real-time message processing, visitor tracking data stream analysis, and extensible Open APIs.

As the creator of ShenDesk, I have overseen the system’s evolution from its initial "proof-of-concept" phase to its current state: a robust solution capable of supporting demanding commercial environments. Along this journey, the platform has undergone multiple performance bottlenecks and architectural refactorings.

Driven by these real-world engineering challenges, I began a deep dive into the .NET memory model and high-performance programming techniques.

Introduction

In an era where high concurrency and low latency have become the default expectations, modern .NET applications can no longer afford "unconscious memory allocations." Especially in big data processing, file parsing, network communication, and real-time systems, frequent array copying and string allocations are often the true culprits behind performance bottlenecks.

While developing the ShenDesk server application, we encountered a textbook challenge:
High-frequency WebSocket message processing and visitor tracking data stream parsing generated a massive volume of short-lived objects during peak hours. This led to spiked GC pressure, noticeable latency jitter, and capped throughput.

If we had stuck to traditional array manipulations and string concatenation patterns, our window for optimization would have been extremely narrow.

This is precisely why Span<T> and Memory<T> were introduced to C#.

These types allow us to slice and manipulate memory without triggering additional heap allocations. By leveraging stack allocation and controlled memory reference mechanisms, we can:

Eliminate unnecessary array copying.
Reduce GC invocation frequency.
Mitigate latency jitter.
Enhance throughput stability.

By integrating Span<T> into ShenDesk’s real-time messaging channels and protocol parsing layers, we significantly reduced transient allocations, allowing the system to remain steady under heavy concurrent loads.

Drawing from real-world engineering scenarios, this article will provide a deep dive into:

The underlying mechanics of Span<T> and Memory<T>.
How they facilitate allocation-free memory management.
Best practices for Web API, WebSocket, and I/O processing.
Tangible performance gains observed in production systems.

If you are building high-performance .NET systems—or if you are currently battling GC stutters and latency spikes—understanding these two types is no longer an "advanced trick"; it is an engineering essential.

What is a Span?

A Span is a lightweight value type that represents a contiguous region of memory. Unlike traditional array operations, Span allows developers to directly access and manipulate memory data without the need for data copying.

A Span can point to various memory sources, including:

Arrays
Stack memory
Native memory (unmanaged memory)
Strings

The key advantage of Span is its ability to avoid unnecessary memory allocations. While traditional array or string operations often create new objects and copy data, Span operates directly on existing memory. This architectural shift significantly improves application performance and efficiency.

// Example: Operating on an array using Span
int[] array = new int[100];
Span<int> span = array.AsSpan();

// Directly modify the original array
span[0] = 10;
span.Slice(10, 20).Fill(1); // Fill a specific segment of data

The Importance of Span

Traditional operations, such as extracting substrings or creating array slices, typically result in the creation of new objects in memory. These additional memory allocations not only increase the workload of the Garbage Collector (GC) but also degrade overall application performance.

Span addresses this issue by creating a "view" of existing memory rather than copying the data. This approach offers several key advantages:

Faster execution speeds
Reduced memory footprint
Enhanced overall system throughput
Lower Garbage Collection frequency

Span is particularly indispensable in high-performance scenarios, such as:

Web Servers (e.g., handling high-concurrency requests)
Data Parsers (e.g., JSON or binary protocol parsing)
Real-time Systems (where latency is critical)
Large-scale Data Processing

Core Features of Span

Span provides several critical features that make it the ideal choice for high-performance memory management:

Stack-only Allocation: As a ref struct, Span is not allocated on the managed heap, which significantly boosts execution efficiency.
Slicing Support: It allows for direct manipulation of specific memory segments without the overhead of data duplication.
Type and Memory Safety: Span provides indexed access with bounds checking, preventing common pitfalls like buffer overflows while maintaining performance.
Reduced GC Pressure: By minimizing transient object creation, it drastically reduces the frequency and duration of Garbage Collection cycles.
Optimized for Big Data: It excels in scenarios involving massive datasets where traditional object-oriented overhead becomes a bottleneck.

// Example: Slicing operations with Span
byte[] buffer = new byte[1024];
Span<byte> bufferSpan = buffer.AsSpan();

// Process the first 512 bytes
ProcessData(bufferSpan.Slice(0, 512));

// Process the remaining 512 bytes
ProcessData(bufferSpan.Slice(512));

What is Memory?

While Memory<T> shares many similarities with Span<T>, it is specifically designed for a broader range of scenarios—most notably asynchronous programming. The key distinctions are:

Stack vs. Heap: While Span<T> is a "stack-only" type restricted to synchronous methods, Memory<T> can be stored on the heap.
Asynchronous Compatibility: Memory<T> can be used within async methods and stored as a class field, allowing it to persist across await boundaries.
Lifecycle Management: It represents a memory region that can be safely passed between asynchronous operations.
Performance Trade-off: Although Memory<T> is slightly slower than Span<T> due to its extra layer of abstraction, it offers significantly more flexibility while still maintaining excellent performance compared to traditional arrays.

// Example: Using Memory in an asynchronous method
async Task ProcessDataAsync(Memory<byte> dataMemory)
{
    // Processing data asynchronously
    // Notice we convert to .Span at the point of synchronous execution
    await Task.Run(() => ProcessData(dataMemory.Span));

    // Memory can be stored for later use, unlike Span
    _storedMemory = dataMemory;
}

When to Use Span

Span<T> is the optimal choice in the following scenarios:

Synchronous Workflows: Ideal for processing arrays, buffers, or strings within a single execution thread.
Data Parsing: High-performance parsing of network protocols, custom file formats, or structured logs.
File I/O Operations: Efficiently reading from or writing to large files without intermediate allocations.
Massive Datasets: When your application requires low-latency processing of large-scale data.
Performance-Critical Paths: Any "hot path" in your code where memory optimization and GC pressure are significant concerns.

// Example: Parsing a string using Span
string s = "127.0.0.1:8080";
ReadOnlySpan<char> span = s.AsSpan();

int colonPos = span.IndexOf(':');
if (colonPos > 0)
{
    // These slices do not create new string objects
    var ipSpan = span.Slice(0, colonPos);
    var portSpan = span.Slice(colonPos + 1);

    // Process the IP and Port as views of the original string
}

When to Use Span

Span<T> is the optimal choice in the following scenarios:

Synchronous Workflows: Ideal for processing arrays, buffers, or strings within a single execution thread.
Data Parsing: High-performance parsing of network protocols, custom file formats, or structured logs.
File I/O Operations: Efficiently reading from or writing to large files without intermediate allocations.
Massive Datasets: When your application requires low-latency processing of large-scale data.
Performance-Critical Paths: Any "hot path" in your code where memory optimization and GC pressure are significant concerns.

// Example: Parsing a string using Span
string s = "127.0.0.1:8080";
ReadOnlySpan<char> span = s.AsSpan();

int colonPos = span.IndexOf(':');
if (colonPos > 0)
{
    // These slices do not create new string objects
    var ipSpan = span.Slice(0, colonPos);
    var portSpan = span.Slice(colonPos + 1);

    // Process the IP and Port as views of the original string
}

Real-World Application Scenarios

Span<T> and Memory<T> have been widely adopted across various high-performance domains, including:

High-Performance Web APIs: ASP.NET Core leverages Span internally to maximize throughput.
File Processing Systems: Efficiently handling large-scale file I/O with minimal overhead.
Networking Applications: Protocol parsing and low-level packet processing.
Real-Time Systems: Low-latency data manipulation where every millisecond counts.
Game Development: High-efficiency memory management for rendering and physics engines.
Parsers and Serializers: Rapid data transformation for formats like JSON, XML, or Protobuf.

For instance, ASP.NET Core utilizes Span extensively within its internal pipeline to optimize request handling, particularly in the following areas:

Request Header Parsing: Splitting and validating headers without string allocations.
URL Decoding: In-place transformation of encoded characters.
JSON Serialization/Deserialization: Using System.Text.Json for high-speed data binding.
Response Buffering: Directly writing to the output stream buffer.

Performance Benefits

Implementing Span<T> and Memory<T> yields significant performance gains across several key metrics:

Minimal Memory Allocation: Eliminates redundant allocations and data duplication.
Accelerated Execution: Direct memory access removes intermediate processing steps, shortening the execution path.
Reduced GC Pressure: Drastically lowers both the frequency of garbage collection and the duration of "stop-the-world" pauses.
Optimized Resource Utilization: Ideally suited for high-throughput, high-concurrency applications.

Benchmark data indicates that transitioning to Span can achieve:

Over 90% reduction in memory allocations.
2x to 5x faster execution speeds in intensive processing tasks.
Significant reduction in Garbage Collection (GC) pause times.

// Performance Comparison: String Processing
// Traditional Approach - Allocates a new string object
string substring = bigString.Substring(start, length);

// Using Span - Zero-allocation "view" of the original string
ReadOnlySpan<char> span = bigString.AsSpan().Slice(start, length);

Conclusion

High performance is not about "showing off" technical tricks; it is a fundamental engineering mindset.

At their core, Span<T> and Memory<T> solve more than just memory allocation issues. They challenge us to rethink how data flows through our systems: Does this data really need to be copied? Is a heap allocation truly necessary? Can we eliminate Garbage Collection (GC) pressure at the source?

When you begin to scrutinize your code through the lenses of "memory lifecycle" and "allocation cost," you quickly realize that many so-called performance bottlenecks are simply the result of unconscious programming habits.

Throughout the evolution of ShenDesk, the benefits of this structural shift in thinking have far outweighed any single micro-optimization. Our systems have become more resilient under high concurrency, our latency curves have smoothed out, and our resource utilization has become far more predictable.

If you are building .NET systems for real-world users, my advice is straightforward:

Don't wait for a performance crisis to start firefighting. Understand these low-level capabilities early on, and make "avoiding unnecessary allocations" your default principle.

Engineering quality is never decided at the moment of deployment; it is forged by the choices we make every time we write a line of code.

Wrapping up

ShenDesk is still evolving.

If you’ve ever built or deployed a real-time chat system, I’d genuinely love to hear your experience—
how you handled live updates, load balancing, or flexible deployment models in production.

Let’s compare notes.

If you’re curious

I’ve been building ShenDesk, a customer support chat system designed to run reliably
both online and on your own infrastructure.

🌐 Website: https://shendesk.com
📘 Documentation: https://docs.shendesk.com

You can try it for free, whether you prefer a hosted setup or self-hosting.

Feedback from developers interested in self-hosted systems, real-time communication,
and customer experience engineering is always welcome.

UI snapshots

Visitor side
Fast loading, no message loss

Agent side
Reliable, feature-rich, built for real-world support work

Web admin panel

2025 in Review: Running a Real-World Customer Support System

Xusheng Cao — Fri, 02 Jan 2026 09:53:41 +0000

2025: A Quiet Year, A Brutal One

Over the past few years, the same topics keep circulating in tech communities: AI, global expansion, SaaS, indie developers.
The buzzwords change, the narratives evolve, but for those actually running systems in production, the experience feels much more concrete—and much more unforgiving.

For me, 2025 was not a year of dramatic turning points.
There was no explosive growth, no sudden pivot.
It was simply a year of building, encountering real problems, and steadily correcting my assumptions.

This post is a modest retrospective.
It records the choices I made, the mistakes I ran into, and a few lessons that only became clear after spending a full year working on a customer support system.

1. 2025: A Year That Looked Ordinary, But Wasn’t

From the outside, 2025 didn’t feel especially turbulent.
It wasn’t like 2020, nor did it carry the uncertainty of 2022.

But underneath the surface, something had clearly changed:

Opportunities still existed, but tolerance for mistakes dropped sharply.

Organic traffic became harder to come by
Users were less willing to “grow with your product”
Technical advantages turned into endurance tests

You could still ship features and release versions—but every decision carried more weight.
Reversibility became expensive.

Calm on the surface, acceleration underneath

2025 felt like a dividing line.

Large companies narrowed their focus to highly predictable bets
Small teams realized storytelling and funding couldn’t sustain them forever
Indie developers either became more professional—or quietly disappeared

“It runs” was no longer impressive.
“Can it survive?” was the real question.

Technology still mattered—but it stopped being the moat

One realization stood out:

Technology didn’t lose value.
But relying on technology alone became a much narrower path.

Frameworks improved. Tooling matured. Shipping became faster than ever.
But so did imitation.

What separated products wasn’t innovation—it was:

Stability
Maintenance
Boring, invisible decisions made over time

Most projects didn’t fail because they were impossible to build.
They failed because the long road after launch was underestimated.

For indie builders, 2025 was a year of clarity

The romantic phase ended.

You start calculating:

Infrastructure costs
Support overhead
The real price of “free users”

And eventually you face a hard question:

Will anyone use this long-term—and pay for it?

Nothing dramatic happened for me that year.
I didn’t quit. I didn’t break out.

I just realized that if I kept going,
I had to treat this as long-term, sometimes boring work.

That’s the context in which I kept building ShenDesk

In 2025, I continued working on something decidedly unglamorous:
a customer support system.

Not because it was trendy—but because it was honest.

It demands engineering discipline.
It exposes trade-offs.
And it reflects reality very quickly.

Everything that follows comes from this premise:

2025 was no longer a “let’s try and see” year.

2. Why Build a Customer Support System in 2025?

From a market perspective, it’s a strange choice.
Customer support software is not new, not exciting, and definitely not a blue ocean.

But precisely because of that, it became a very honest problem to work on.

A customer support system is a truth machine

Support systems don’t generate growth.
They absorb problems.

They don’t shine when everything works.
They are used when things break.

That makes two things unavoidable:

Stability and real-time behavior matter enormously
You can’t hide behind marketing

You know whether it works within days.

“Red ocean” doesn’t mean “no real problems”

What I kept seeing:

Feature-heavy products with fragmented long-term experience
Great demos that struggled under real usage
Tools optimized for sales—not for engineers

For small and mid-sized teams, the pain was consistent:

SaaS versions felt restrictive
Self-hosted versions were painful to operate
Diagnosing issues was unnecessarily hard

It wasn’t about lack of options.
It was about lack of trust.

I wanted to test one thing

The question wasn’t “can this be a big product?”

It was simpler—and harsher:

If you design everything around controllability and maintainability,
can you still build something users genuinely like?

That choice forces you to prioritize:

Observability
Clear system boundaries
Predictable behavior

These things don’t show well in demos.
But they show up after the 500th use.

ShenDesk is just the container for that experiment

ShenDesk isn’t a concept-first product.
It’s a long-running testbed for real-world decisions.

What to automate.
What to leave manual.
Where SaaS ends and self-hosting begins.

Most decisions weren’t best practices.
They were consequences.

Why 2025?

Earlier, I would have rushed.
Later, I might have played too safe.

In 2025:

The tech was mature enough
Users were more rational
I cared less about impressing people

And more about this question:

Can this system earn long-term trust?

3. Five Technical Mistakes I Actually Made in 2025

These weren’t theoretical mistakes.
They surfaced under real usage.

Mistake 1: Confusing “feature complete” with “usable”

A support system is not a feature product.
It’s a stability product.

It’s judged by:

Peak-time behavior
Network edge cases
Recovery after crashes

Checklists don’t reveal that.

Mistake 2: Underestimating real-time complexity

On paper: WebSockets + messages.
In reality: state convergence under chaos.

Ghost sessions, partial disconnects, race conditions—
hard to reproduce, impossible to ignore.

Mistake 3: Treating logs as a post-mortem tool

If you need to reproduce a support issue, you’re already late.

Without structured observability,
you’re guessing.

Mistake 4: Thinking SaaS and self-hosted were just deployment choices

They aren’t.

They are different products with different assumptions.

Mistake 5: Ignoring non-functional requirements

Performance, reliability, diagnosability—
you don’t “add them later.”

They decide whether later exists.

4. Three Product Lessons That Felt Backwards

Lesson 1: Users want fewer surprises—not more power

Predictability beats capability.

Lesson 2: The best systems disappear

Presence is friction.

Lesson 3: For small teams, control beats automation

Black boxes erode trust.

5. The Key Trade-offs I Made in ShenDesk

SaaS and self-hosted—not one or the other
Fewer features, clearer boundaries
Diagnosability over clever automation
Internationalization as a constraint, not a patch
Building a long-term system, not a feature bundle

ShenDesk isn’t meant to prove intelligence.
It’s meant to survive reality.

6. Three Small, Certain Things I’ll Focus on in 2026

Turning stability into a system capability
Running real international usage—not just translations
Defining who ShenDesk is not for

No big promises.
Only things that can be verified.

7. For Those Still Building Slowly

This isn’t a success story.
It’s a record of choosing to continue—clear-eyed.

If you’re building something quiet, slow, and hard to explain,
you’re not alone.

ShenDesk is built that way on purpose.

If it fits you, you’ll understand why.
If not, that’s fine too.

Continuing to build carefully, in this phase,
is already rare enough.

Wrapping up

ShenDesk is still evolving.

Let’s compare notes.

If you’re curious

I’ve been building ShenDesk, a customer support chat system designed to run reliably
both online and on your own infrastructure.

🌐 Website: https://shendesk.com
📘 Documentation: https://docs.shendesk.com

You can try it for free, whether you prefer a hosted setup or self-hosting.

Feedback from developers interested in self-hosted systems, real-time communication,
and customer experience engineering is always welcome.

UI snapshots

Visitor side
Fast loading, no message loss

Agent side
Reliable, feature-rich, built for real-world support work

Web admin panel

Why We Chose WebView Over SDK for In-App Customer Support

Xusheng Cao — Thu, 25 Dec 2025 15:34:30 +0000

A Practical Engineering Perspective from Building ShenDesk

When integrating an online customer support system into a mobile app, I went through a long and sometimes frustrating process of technical exploration and trade-off analysis.

At the beginning, I was strongly attracted to several “technically ideal” options.
For example:

Letting app teams fully build their own UI while I only provide APIs, maximizing flexibility
Or integrating a native SDK to pursue the most “pure” native experience

On paper, these options look elegant.
In practice, as the project evolved, I gradually realized that solutions driven purely by technical ideals often fail to meet the complexity of real-world business scenarios.

This realization didn’t come from theory, but from feedback.

I had in-depth conversations with customers across more than a dozen industries, including e-commerce, education, SaaS, finance, and government-related organizations. Through real production deployments and continuous feedback loops, several critical decision factors became very clear.

Key Constraints That Actually Matter in Production

1. Time-to-launch is the primary bottleneck

Most teams want to launch customer support within one to two weeks, without disrupting their existing app flows.

“Can this be integrated quickly and painlessly?”
This question outweighed almost every other consideration.

2. UI consistency directly impacts user trust

Users expect the support interface to feel like a natural part of the app:

No jumping outside the app
No visually fragmented third-party windows

Even subtle inconsistencies in style or interaction can significantly degrade perceived quality and trust.

3. Functional completeness is non-negotiable

Modern customer support is not just about sending text messages.

Users expect:

Image and file uploads
AI bot integration
Queueing and agent transfer
Satisfaction ratings and feedback

These are no longer “advanced features”; they are baseline expectations.

4. Operational capacity is limited

Most engineering teams are small.

Maintaining a fully custom-built support module—especially real-time communication, state synchronization, and reconnection logic—quickly becomes a long-term operational burden rather than a one-time engineering effort.

5. Multi-platform UI becomes a hidden cost multiplier

If teams are asked to build and maintain their own chat UI across Android, iOS, and H5:

Development timelines effectively double
Maintenance complexity compounds over time

What looks like “maximum flexibility” at the beginning often turns into long-term technical debt.

The Strategy That Emerged

Based on these constraints, the technical direction became clear:

For most companies, the optimal path is a solution that is highly controllable, fast to integrate, functionally complete, and reusable across platforms.

Such a solution must provide enough flexibility—style customization, theme configuration, contextual parameters—while also relying on a mature and stable server-side foundation: queueing logic, agent assignment, chat history, notifications, and satisfaction surveys.

This philosophy is what guided the design of ShenDesk.

In the following sections, I’ll compare several common approaches for integrating customer support into mobile apps, and explain why we ultimately committed to WebView-based embedding, including real pitfalls we encountered and concrete implementation details.

SDK Integration or API + Custom UI?

Both SDK integration and API-driven custom UI can make sense in very specific, highly customized scenarios.

However, in real production environments, they often introduce significant hidden costs and risks.

Common Pitfalls of SDK-Based Integration

“Native SDK integration” is frequently marketed as the best possible user experience.
In actual engineering work, however, it often comes with underestimated complexity and long-term maintenance costs.

Below, I’ll break this down from three perspectives: development complexity, performance impact, and maintainability.

1. High integration complexity and strong platform coupling

SDKs typically require separate integration for Android and iOS, along with a non-trivial set of dependencies, permissions, and lifecycle hooks.

A small misconfiguration can easily result in runtime errors or crashes.

A common Android integration example:

<!-- AndroidManifest.xml -->
<uses-permission android:name="android.permission.INTERNET"/>
<application>
    ...
    <activity android:name="com.sdk.chat.ChatActivity"
              android:theme="@style/SDKTheme"
              android:exported="true"/>
</application>

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)

    ChatSDK.initialize(
        apiKey = "your_key",
        userId = user.id,
        context = this
    )
}

Typical problems include:

SDK upgrades changing initialization parameters or permission requirements
Internal services or broadcasts interfering with host app behavior
Incomplete ProGuard/R8 configuration causing crashes in release builds

2. Noticeable binary size and performance overhead

Many third-party support SDKs bundle:

WebSocket clients
Image caching libraries
Local databases
Built-in UI templates, fonts, and animations

This often results in:

App size increases of 3–5 MB
Slower startup time, especially on low-end devices
Dependency conflicts and method count issues

3. Limited UI customization and fragmented UX

Most SDKs expose a closed, self-contained “support UI module.”

val intent = Intent(this, ChatSDK.getChatActivityClass())
startActivity(intent)

You cannot embed it like a Fragment, nor can you control UI details at the DOM or layout level.

This leads to:

Visual inconsistency with the rest of the app
Restricted navigation and interaction patterns
Poor control over localization and theming

4. Opaque updates and difficult debugging

Because SDKs are maintained by third parties, their internal logic is largely invisible.

E/ChatSDK: WebSocket failed to connect.
E/ChatSDK: Internal message parser error.

You cannot step into the source code, inspect state machines, or analyze message queues and retry logic.

In some cases, heavy obfuscation turns the SDK into a complete black box.

5. Weak integration with business context

Suppose you want your support system to understand what product or order the user is currently viewing and route the chat to the appropriate agent group.

With SDK-based integration, this is often extremely difficult:

Contextual data passing is limited or unsupported
Real-time business events cannot be injected
Server-side routing logic is not exposed

ChatSDK.sendMessage("{ \"type\": \"product\", \"id\": \"123456\" }")

Structured messages like this are often rejected or silently reformatted, making advanced product-level features impossible.

The Trap of “API + Custom UI”

At first glance, building your own chat UI using APIs sounds flexible and empowering.
In reality, you quickly discover that a customer support system is not just a chat box, but a high-complexity asynchronous real-time system.

It involves message states, queueing, reconnection, file uploads, agent transfers, and satisfaction feedback—each with its own edge cases.

1. Message synchronization is deceptively complex

A real chat system must handle:

Sending / sent / failed states
Read receipts
Unread counters
Out-of-order messages
Deduplication

function sendMessage(content: string) {
    const msgId = generateClientMsgId();

    renderMessageLocally({ id: msgId, status: 'sending', content });

    sendToServer(content, msgId)
        .then(() => updateStatus(msgId, 'sent'))
        .catch(() => updateStatus(msgId, 'failed'));
}

This logic breaks easily under network interruptions, retries, or mismatched acknowledgements.

2. Reconnection and message recovery are full of edge cases

Ensuring “no message loss” requires far more than reconnecting a WebSocket.

You need:

Last-message tracking
Incremental sync APIs
Deduplication and ordering guarantees

socket.onopen = () => {
    const lastTimestamp = getLastMessageTimestamp();
    fetch(`/api/messages/since?ts=${lastTimestamp}`).then(syncMessages);
}

Clock drift, latency, and concurrent sends can quickly create inconsistent states.

3. UI state management easily spirals out of control

A chat UI is effectively a finite state machine:

Before session
Queueing
Active conversation
Finished

Network failures, agent transfers, or server restarts can trigger unexpected state transitions, and every edge case must be handled manually.

4. Critical logic lives on the server, not in the API docs

Agent routing, business hours, evaluation eligibility, rate limits—many behaviors are governed by server-side rules that APIs alone do not fully describe.

Triggering UI actions prematurely often leads to failed or confusing user experiences.

5. File uploads are surprisingly painful

Uploading images or files requires:

Storage signing (S3, OSS, etc.)
Validation, progress tracking, retries
Message linking and expiration handling

What looks simple quickly becomes one of the most error-prone parts of the system.

WebView Embedding: A Lightweight but Powerful Approach

After extensive evaluation, we chose WebView-based embedding as the default integration model for ShenDesk.

In this approach, the app embeds a hosted support page inside a WebView, while preserving UX continuity.

This allows us to achieve:

Faster integration
Lower client-side complexity
Greater customization flexibility
Stronger system-level control

Key Advantages

Low integration cost
Apps only need to open a WebView and pass basic parameters.

Easy updates
UI changes do not require app updates or store reviews.

Cross-platform consistency
One implementation works across iOS, Android, web, and other channels.

Context-aware integration
User identity and business context can be injected via URL parameters or cookies.

Final Thoughts

In theory, SDKs and fully custom UIs promise control and performance.
In practice, WebView offers a more balanced solution for most teams operating under real-world constraints.

This is not a shortcut—it is a deliberate engineering trade-off.

Wrapping up

ShenDesk is still evolving.

Let’s compare notes.

If you’re curious

I’ve been building ShenDesk, a customer support chat system designed to run reliably
both online and on your own infrastructure.

🌐 Website: https://shendesk.com
📘 Documentation: https://docs.shendesk.com

You can try it for free, whether you prefer a hosted setup or self-hosting.

Feedback from developers interested in self-hosted systems, real-time communication,
and customer experience engineering is always welcome.

UI snapshots

Visitor side
Fast loading, no message loss

Agent side
Reliable, feature-rich, built for real-world support work

Web admin panel

A UTF-8 Trap I Hit While Building a Customer Support Chat System

Xusheng Cao — Mon, 15 Dec 2025 14:24:53 +0000

For years, I’ve been building an independent customer support chat system in my spare time.

It didn’t start big. In the early days, only a handful of users tried it out of curiosity. Over time, it slowly grew into something real: a system running in production, used daily, deployed both as a hosted service and in on-premises environments. Along the way, I learned a lot—not just about writing code, but about operating and maintaining a long-lived product.

More than once, after releasing a new version, I caught myself thinking:

“This one is rock solid. Nothing can break it.”

And almost every time, reality responded a few weeks later with:

“Wait… how is this even possible?”

A few years ago, those moments happened all the time.
This year, only occasionally.
And recently, after upgrading several customer environments to the latest version without a single incident, I genuinely thought:

“Alright. This might finally be stable.”

Even my own production environment had changed.
User numbers were growing. Concurrent visitors were increasing.
Yet bug reports had almost disappeared.

That’s usually when you get careless.

“Why Is Yesterday’s Chat History Completely Gone?”

The report came in quietly:

“Why are yesterday’s conversations missing?”

No crash.
No alerts.
No errors reported by the application.

I started digging through logs late at night and found something unsettling:
the database was returning zero records—cleanly, normally. No timeout. No permission error.

It was as if the conversation had never existed.

Scrolling further back, I noticed something interesting.
The last message sent by the agent was:

“Alright, please wait a moment 🥲”

That single emoji turned out to be the culprit.

One small character had silently destroyed the entire chat record.

The Symptom: INSERT Looked Successful, but Data Was Gone

At first glance, everything looked fine.
The application logged “message saved successfully”.
No exception bubbled up.

But under the hood, the SQL statement had failed.

Here’s how the table was originally created:

CREATE TABLE chat_message (
    id BIGINT AUTO_INCREMENT PRIMARY KEY,
    visitor_id VARCHAR(50),
    content TEXT CHARACTER SET utf8 COLLATE utf8_general_ci,
    created_at DATETIME
);

Looks reasonable, right?

Until a user sends a message containing a 4-byte character—like an emoji.

INSERT INTO chat_message (visitor_id, content, created_at)
VALUES ('A123', 'Alright, please wait 🥲', NOW());
-- Error: Incorrect string value: '\xF0\x9F\xA5\xB2' for column 'content'

The insert failed.
But the application didn’t notice.

The Hidden Problem: “utf8” in MySQL Is Not UTF-8

This is the trap.

In MySQL, utf8 is not full UTF-8.
It only supports 1–3 byte characters (the BMP).

Emoji live outside that range.

🥲 = U+1F972
UTF-8 bytes = F0 9F A5 B2

Result: the database rejects the value.

And here’s where it got worse.

The “Fake Success” at the Application Layer

I was using a .NET MySQL connector with default settings.
Combined with incomplete exception handling, the driver swallowed the failure.

try
{
    await db.ExecuteAsync(
        "INSERT INTO chat_message (visitor_id, content, created_at) VALUES (@v, @c, @t)",
        new { v = visitorId, c = content, t = DateTime.UtcNow }
    );
    logger.Info("Message stored: " + content);
}
catch (Exception ex)
{
    // Only logging ex.Message, no SQL error code
    logger.Warn("Message error: " + ex.Message);
}

From the application’s point of view: success.
From the database’s point of view: nothing was saved.

So the agent thought the message was stored.
The next day, they opened the chat history—and found nothing.

The Fix: Switching Everything to utf8mb4

The solution itself was straightforward, once the root cause was clear.

Step 1: Update database and tables

ALTER DATABASE mychat
  CHARACTER SET = utf8mb4
  COLLATE = utf8mb4_unicode_ci;

ALTER TABLE chat_message
  CONVERT TO CHARACTER SET utf8mb4
  COLLATE utf8mb4_unicode_ci;

Step 2: Explicitly set charset in the connection string

var connStr =
  "Server=localhost;Database=mychat;Uid=root;Pwd=xxx;CharSet=utf8mb4;";

Step 3: Verify with real emoji data

var testMessage = "Welcome to our support system 🥳🔥";

await db.ExecuteAsync(
    "INSERT INTO chat_message (visitor_id, content, created_at) VALUES (@v, @c, @t)",
    new { v = "T001", c = testMessage, t = DateTime.UtcNow }
);

var result = await db.QuerySingleAsync<string>(
    "SELECT content FROM chat_message ORDER BY id DESC LIMIT 1"
);

Console.WriteLine(result);
// Output: Welcome to our support system 🥳🔥

This time, it worked.

One More Trap: Index Length and utf8mb4

Switching to utf8mb4 introduced another surprise.

ALTER TABLE chat_message
ADD INDEX idx_v_c(visitor_id, content);
-- Error: Specified key was too long; max key length is 767 bytes

Each utf8mb4 character can take up to 4 bytes.
Indexing a long VARCHAR or TEXT column can easily exceed InnoDB limits.

The fix was to use a prefix index:

ALTER TABLE chat_message
ADD INDEX idx_v_c(visitor_id, content(100));

Or, depending on the use case, a full-text index.

Lessons Learned (the Hard Way)

Never assume utf8 means UTF-8 in MySQL. It doesn’t.
Always log real SQL error codes. Silent failures are worse than crashes.
Test with emoji. Real users will use them—constantly.

I ended up adding this test to our deployment pipeline:

it("should store emoji without errors", async () => {
    const message = "Emoji test 🐱🐶🔥";
    await api.sendMessage({ visitorId: "U999", content: message });
    const saved = await api.getLastMessage("U999");
    expect(saved.content).toBe(message);
});

It runs every time we deploy.

Because no system—no matter how many years it’s been running—
should ever lose history to a single 🥲 again.

Wrapping up

ShenDesk is still evolving.

Let’s compare notes.

If you’re curious

I’ve been building ShenDesk, a customer support chat system designed to run reliably
both online and on your own infrastructure.

🌐 Website: https://shendesk.com
📘 Documentation: https://docs.shendesk.com

You can try it for free, whether you prefer a hosted setup or self-hosting.

Feedback from developers interested in self-hosted systems, real-time communication,
and customer experience engineering is always welcome.

UI snapshots

Visitor side
Fast loading, no message loss

Agent side
Reliable, feature-rich, built for real-world support work

Web admin panel

Building ShenDesk: a self-hostable real-time chat system for web & mobile apps

Xusheng Cao — Sat, 25 Oct 2025 11:36:17 +0000

Hi everyone,
I’m the developer behind ShenDesk, a real-time customer support and marketing system designed for modern teams who want full control and flexibility.

I started building ShenDesk because most live chat tools are either too expensive, too closed, or simply not designed for developers who want to self-host. Many small teams told me they wanted to own their data, integrate support directly into their websites and mobile apps, and still have the option to use SaaS if needed.

So I built ShenDesk to make that possible.

💡 What makes ShenDesk different

Works on both web and mobile apps
The visitor interface is a pure HTML page, so it can be embedded directly into mobile apps (via WebView) or any website — making it a single, unified support system across platforms.
Free SaaS and free self-hosted deployment
You can try ShenDesk online for free, or deploy it privately at no cost.
It supports Windows, Linux, and Docker deployments out of the box.
Full feature set
Real-time visitor tracking, multi-agent live chat, message handling, open APIs, and integrations.
Agents can monitor online visitors, initiate chats, or respond instantly from a web-based console.
Built for developers
Server Application: .NET 8
Agent Console: WPF
Web Admin Panel: Vue 3
Everything runs in containers, with open APIs for further customization.

🔧 Why I built it

I believe small teams should have the same quality of real-time engagement tools as large enterprises — but with freedom and ownership.
With ShenDesk, you can choose to host everything yourself, integrate it seamlessly into your app or site, and extend it however you want.

🚀 Try it out

🌐 Official Website: https://shendesk.com
📘 Documentation: https://docs.shendesk.com

It’s completely free to try — both online and on your own server.
I’d love to get feedback from developers who care about self-hosted tools, real-time communication, or building better customer experiences.

👋 Closing

I’m still improving ShenDesk and would love to hear your ideas.
If you’ve built or integrated live chat systems before, how did you approach the challenges of real-time updates, load handling, and deployment flexibility?

Let’s talk.

UI Screenshots

Visitor Interface: instant loading, no message loss.

Agent Console: reliable and fully featured.

Web Admin Panel

Why I insist on creating "products" rather than just focusing on "technology"

Xusheng Cao — Fri, 21 Mar 2025 16:22:56 +0000

Because the value of a product can be passed down over time.

When I worked at Fujitsu, I participated in some Japanese software projects that are still using technology from over a decade ago. When reviewing and modifying code, I often saw comments written by my predecessors more than ten years ago. These software systems are still delivering their value, and their economic and social benefits have not been affected by the "outdated" technology. To put it bluntly, the benefits they generate are unrelated to the technical issues.

Software products are a summary, extraction, and synthesis of knowledge and processes within an industry. The knowledge in these fields often has a very long lifespan. Over time, as industry knowledge and processes accumulate, the competitiveness of the product gradually increases. Technology is just a method, not the end goal. Even if there is a need to update the "method," it can be iterated on the already constructed product. It is the product that gives value to the "method."

In the context of rapid technological updates, focusing solely on technology itself and trying to always chase the latest or more "advanced" technologies is like chasing the flowers in the mirror or the moon in the water. From the trends of the past few years, technology is no longer updated within a fixed system, but rather, it is constantly overthrown by entirely new ideas and methods. What we can do is truly make technology work for us, transforming technological methods into product value that can exist in a specific field for a long time.

My Free .NET Resource File Management Tool: Shen Resources Tool

Xusheng Cao — Tue, 18 Mar 2025 15:35:55 +0000

This is a tool I developed to assist in the development of multilingual user interfaces for .NET applications. Although many years have passed, I still use it, and I hope it will be useful to you as well. 😀

You can download and use it for free from my website:
https://sr.shendesk.com.

To enable multi-language support in software, .NET provides resource files to store different language resources, which are then loaded and used in the program. However, for large projects or team development—especially with continuously evolving versions—multi-language development can become extremely cumbersome, leading to many uncontrollable issues, such as:

The traditional way of retrieving resources relies on string-based keys, which can result in spelling errors when developers access resources.
When modifying resource files, existing resource identifiers may be changed without being properly synchronized across the program.
During resource file modifications, existing entries may be mistakenly deleted, or entries assumed to be unused may still be needed elsewhere.
It is difficult to determine the usage status of a resource: whether it is still in use and whether it is used in multiple places.
When adding new language resources, ensuring full synchronization across all languages becomes challenging, especially when there are a large number of resources with frequent updates.

If resource keys are referenced as strings, none of these issues can be detected at compile time, making errors difficult to identify.

Additionally, two more challenges make resource file management particularly troublesome during development:

In Visual Studio, resource files can only be edited one by one in separate windows. For instance, if you have both English and Japanese resource files, you must open two independent windows for editing. More importantly, Visual Studio does not check whether the entries in both resource files match. There is no linkage or mapping between them, meaning discrepancies such as missing or extra entries, or case-sensitive key mismatches, go unnoticed.
There is no efficient way to coordinate with translators for tasks such as resource import, export, and automatic validation. There is no reliable way to hand off language resources to translators and then re-import them while ensuring that thousands of resource entries have no omissions or inconsistencies.

Here is an introduction to my tool:

Supports Google automatic translation.

Generate .NET code and resource files, and use a single interface to constrain and invoke them.

Generate JSON files for referencing them in a Web project.

Export to or import from an Excel file to facilitate the work of translators.

You can download and use it for free from my website:

https://sr.shendesk.com.

My Journey in Software Development

Xusheng Cao — Mon, 17 Mar 2025 16:54:11 +0000

I am a seasoned programmer from China with over 10 years of experience in software development. I have a strong ability to develop and implement products from 0 to 1, transforming ideas into fully functional and scalable solutions. My expertise lies in .Net/C#, where I specialize in WPF, WinForms, and ASP.NET, enabling me to build high-performance desktop and web applications.

In addition to my .Net expertise, I am highly proficient in web front-end technologies, including native JavaScript and Vue.js, as well as back-end service development. I am also familiar with Java and Python, having worked on projects involving these technologies, though they are not my primary focus.

With in-depth research and hands-on experience in complex technical architectures, I am adept at designing and optimizing systems for scalability, maintainability, and efficiency. I am passionate about solving challenging technical problems, improving system performance, and leveraging technology to drive innovation.

I am delighted to meet you all and look forward to engaging in meaningful discussions and exchanging ideas with you.