DEV Community: Joung Park

Migrate AMQ to Kafka

Joung Park — Tue, 27 Oct 2026 02:41:16 +0000

As our system moved from a monolith towards microservices, services needed a way to communicate without depending on one another directly. We used ActiveMQ, managed through Amazon MQ, for that messaging. Later, we moved to Kafka on Amazon MSK.

This was not simply a change of hosting provider. ActiveMQ and Kafka have different messaging models, so the migration made us revisit how we publish, consume, and operate events. Here is why we made the move, what changed, and what I would keep in mind when planning a similar migration.

Why move from ActiveMQ to Kafka?

Amazon MQ gave us a managed ActiveMQ broker and a familiar message-broker model. That was a reasonable starting point: services could send messages through a broker instead of calling each other directly.

As we moved further towards event-driven microservices, Kafka became a better fit for the kind of communication we wanted to build around. Kafka stores records in durable, partitioned logs. Consumers read those records at their own pace, and separate consumer groups can each process the same stream independently.

That model is useful when an event should be available to more than one downstream service, or when we want consumers to be able to catch up by reading retained events. It also gives us a common platform for event-driven workflows and stream processing.

The key point is that Kafka was not automatically better for every message. We chose it because its log-and-consumer model fit the direction of the architecture. A traditional broker can still be the right choice for workloads that need its queueing and delivery semantics.

The important difference: queue versus log

The biggest conceptual change was moving from thinking of messages as work handed to a broker queue to thinking of events as records in a log.

With a queue, a message is typically delivered to a consumer, acknowledged, and removed from the work waiting to be processed. With Kafka, a record remains in a topic according to its retention policy. Each consumer group tracks its own position, so multiple groups can independently process the same records.

ActiveMQ queue
Producer ──► Queue ──► Consumer

Kafka topic
Producer ──► Topic (retained log) ──► Consumer group A
                              └─────► Consumer group B

This changes what consumers are responsible for. A Kafka consumer tracks offsets and must handle the possibility of seeing a record again, for example after a restart or a failure before its progress is committed. In practice, consumers need to be designed for safe retries and duplicate delivery; moving to Kafka does not make processing exactly-once by default.

Kafka also orders records within a partition, not across an entire topic. If related events need to be processed in order, the producer needs to use a consistent key so those events are routed to the same partition. Ordering is a design decision, not something to assume globally.

Considerations when moving from ActiveMQ to Kafka

Two requirements shaped our migration: keep the live system running without downtime and preserve the ordering our scoring logic depends on. Changing the broker was only useful if we could maintain both.

Migrating a live system without downtime

Our system was already live, so we needed to move traffic without interrupting message processing. We brought Kafka online, switched the producer over, and kept the ActiveMQ listener available during the monitoring and rollback period:

Create the Kafka topic with the required configuration.
Start the Kafka listener and confirm it is healthy before sending it production messages.
Stop publishing to ActiveMQ and start publishing to Kafka. This switches new messages to the new path without intentionally sending each message to both brokers.
Monitor the ActiveMQ queue to confirm no new messages are arriving and that any messages already queued are draining.
Keep the ActiveMQ listener running while monitoring Kafka. If an issue appears during the cutover, switch publishing back to ActiveMQ while investigating and debugging the Kafka path.
Stop the ActiveMQ listener only after the Kafka path has been verified and the rollback window has passed.
Continue monitoring Kafka. Once the new path is stable, remove the old ActiveMQ queue and publishing/listener logic.

Keeping the ActiveMQ listener available gives us a way to back out while the old path is still available. Before stopping it, confirm that the old queue has drained and no messages remain in flight. Monitor Kafka consumer lag, processing errors, and successful downstream score updates—not just whether the listener is running. Growing lag can mean scores are delayed even when the brokers appear healthy.

Tip: keep message handling separate from business logic

One thing that made this migration easier was separating the service layer from the controller/presentation layer. Our scoring logic lived in the service layer, while the message listener acted as an entry point that translated an incoming message into a service call.

Instead of changing the scoring service, we added a Kafka listener alongside the existing ActiveMQ listener. Both called the same service:

ActiveMQ listener ──┐
                    ├──► Scoring service
Kafka listener ─────┘

This kept the migration focused on the messaging boundary:

Controller/presentation layer: receive and translate messages, and handle broker-specific acknowledgements or offset commits.
Service layer: apply the scoring rules without depending on whether the request came from ActiveMQ or Kafka.

We could deploy the Kafka entry point before switching publishers, keep the ActiveMQ entry point available for rollback, and remove it after the cutover without rewriting the scoring logic. The practical tip is to add a new entry point, not a second copy of the business logic.

Ordering and grouping messages

Our scoring flow needs related messages to be processed in strict order. A delta between snapshots needs the previous state, and a win streak depends on the order of outcomes. Processing those messages concurrently or out of order could cause race conditions and inconsistent score calculations.

The grouping key must identify the scoring subject—for example, a player ID, match ID, or another stable identifier for the state being calculated. We used:

ActiveMQ: JMSXGroupID
Kafka: the message key

The technology changed, but the reason for grouping did not: all events contributing to the same score must be processed sequentially.

How it works under the hood

ActiveMQ (`JMSXGroupID`)

JMSXGroupID is a message property that enables JMS message grouping. The broker keeps messages with the same group ID associated with one consumer at a time and delivers them in order within that group.

The broker maintains the group-to-consumer association. If a consumer disconnects, the group can be reassigned. This provides consumer affinity for a group, while allowing different groups to be processed by different consumers.

Kafka (message key)

The message key determines which partition receives a record. With the same partitioning strategy and partition count, records with the same key are routed to the same partition, where Kafka preserves their order:

AMQ:    JMSXGroupID = player-123  ──► same consumer at a time, in order
Kafka:  key = player-123          ──► same partition, in order

Within a Kafka consumer group, each partition is assigned to only one consumer at a time. A rebalance can assign that partition to a different consumer, so Kafka provides partition-level ordering rather than permanent key-to-consumer affinity.

Implications

Ordering is scoped differently: AMQ groups messages by consumer affinity; Kafka orders records within a partition.
Parallelism is across groups or partitions: one group is processed sequentially, while separate groups or partitions can be processed concurrently.
Hot keys can limit throughput: a busy scoring subject can become a hotspot. In Kafka, adding consumers does not increase parallelism beyond the number of partitions, and uneven key distribution can create hot partitions.
Choose the key from the state being updated: use the same stable scoring-subject identifier for all events that contribute to that calculation.
Preserve order in the handler too: do not process events for the same scoring subject concurrently. Partition ordering does not prevent application code from completing work out of order.
Design retries for stateful processing: duplicate delivery must not apply a score change twice. A failed event also needs an explicit recovery path; skipping it and processing the next snapshot or outcome could produce the wrong score.

The cutover needs the same care. An older event still being processed from ActiveMQ could finish after a newer event from Kafka. Keeping the same scoring identifier in both systems does not create ordering across the two brokers. Before allowing newer events to update a subject's score, ensure its earlier ActiveMQ work has completed. The same concern applies when switching publishing back during rollback.

If the Kafka message key is not the right fit, alternatives depend on the required guarantee:

Use a separate topic when the message flow needs independent scaling, retention, or failure handling—not just because it has a different message type.
Choose an explicit partition only when fixed placement is required and the coupling to the topic's partition layout is acceptable. A stable key is generally easier to maintain as the topic scales.
Handle ordering in the application with sequence numbers or version checks when strict broker-level ordering is not required. Consumers must then detect and deal with out-of-order or duplicate events.

Key takeaway

In both systems, correctness depends on choosing a stable grouping identifier that matches the state being calculated. In our scoring flow, we used JMSXGroupID with ActiveMQ and the same scoring-subject value as the Kafka message key, so events for that subject stay ordered.

The outcome

Moving from ActiveMQ on Amazon MQ to Kafka on Amazon MSK was part of a broader shift in how our services communicate. Kafka gave us a durable event log that multiple consumer groups can read independently, which fits the event-driven architecture we were moving towards.

The two requirements stayed the same throughout: keep processing live traffic and keep score calculations correct. Separating the listeners from the service layer let us change the messaging entry point without rewriting the scoring logic. Keeping the old path available gave us a rollback option, while choosing the scoring subject as the Kafka key carried our grouping requirement into the new system.

The main lesson for me is that a messaging migration is not just a broker change. It is about preserving the behavior the application depends on while changing how messages reach it.

Step-Up Verification: OTP, Magic Link

Joung Park — Mon, 26 Oct 2026 04:50:35 +0000

We already use Firebase Authentication for primary login—ID tokens on the client, verified on the backend with the Admin SDK. That answers who the user is. It does not always prove they still control the account at the moment they confirm a payment, change payout details, or export sensitive data. Step-up verification—email one-time passwords (OTP) or magic links—closes that gap.

Step-up does not replace Firebase. Users keep sending the same ID token; sensitive routes additionally require recent proof via email. Because Firebase is already in the stack, Firebase email-link sign-in was the obvious first idea. This post compares that shortcut with a custom OTP service layered on the same auth.

Primary login vs step-up

Layer	Responsibility
Firebase (existing)	Sign-in, session, ID token, `firebaseUid` → internal user
Step-up (new)	Short-lived proof before payments, billing changes, exports, etc.

The backend pattern stays familiar: verify the Firebase token first, resolve the user, then—for protected actions—check a step-up credential or reject with 403 and prompt for verification.

When step-up matters

Typical triggers:

Confirming or changing payment methods and billing settings
Approving a transfer or purchase above a threshold
Exporting or deleting account-held data
Any action where a stolen session should not be enough

For those flows the API usually needs two answers:

Is the caller authenticated (session or ID token)?
Have they recently verified via a channel you control (email OTP or equivalent)?

The shortcut we tried first: trigger Firebase email-link sign-in, then read last sign-in time and sign_in_provider from the ID token or Admin SDK user record. If the provider indicated email link and sign-in fell within a window (for example 15 minutes), allow the action—no new OTP tables or send endpoints.

That only works—if the metadata matches what the user actually did.

Options considered

	Firebase email-link step-up	Custom OTP
Implementation	Reuse Firebase send/link APIs + session metadata	`POST /otp/send`, `POST /otp/verify`, DB, email
User steps	Open email, tap link (device matters)	Open email, type or paste code (any device)
Proof on backend	Last login time + `sign_in_provider`	OTP record + short-lived step-up token
Ops burden	Low if metadata is trustworthy	You own security, delivery, and abuse controls

Security concerns

Firebase email-link step-up

Weak backend proof — If you gate on sign_in_provider and last login, a normal password sign-in can look like step-up; policy mistakes can skip re-verification entirely.
Link as bearer secret — Anyone with the magic link can consume it; forwarded email or a shared inbox completes the flow.
Full sign-in side effects — Email link is real authentication, not a scoped step-up token—sessions refresh, audit trails mix login with “confirm payment,” and revocation semantics are coarser.
Device binding — Links opened on the wrong device can strand or confuse users; support pressure rises before security does.

Custom OTP

Your threat model — Brute force, credential stuffing on /otp/verify, and OTP spraying via /otp/send need rate limits, lockouts, and monitoring you implement and operate.
Storage and logs — Plaintext OTPs in the database or logs are a common failure mode; hash at rest and never log codes.
Email channel — Same mailbox risks as a magic link; a short numeric code can be easier to phish or shoulder-surf if UX is careless.
Step-up token lifecycle — Issue tight TTL, bind to user (and optionally device/session), and reject replay; otherwise a verified OTP window becomes a long-lived bypass.

Option 1: Firebase sign-in via email link

Firebase documents this flow as passwordless email-link authentication on the web—see Authenticate with Firebase Using Email Link.

Why it is attractive

Reuse existing Firebase auth — same project, Admin SDK, and email patterns as primary login.
No OTP product to build — Firebase generates and validates the link; you focus on when to require step-up.
Deep links — mobile clients can resume in-app after the user taps the link.
Continue URLs / query params — return the user to the payment or settings screen that triggered verification.

Where it breaks down

Cross-device UX — magic links often complete on the device that opened the email. Starting checkout on phone and opening mail on a laptop breaks the flow unless you invest in handoff.
Session edge cases — expired links, stale tabs, and multiple active sessions need explicit testing.
POC friction — a quick spike often surfaces integration details that are not obvious from the docs; Firebase handles sign-in, not necessarily your step-up policy.
The blocker: sign_in_provider — after email-link sign-in, Firebase reports sign_in_provider as password, not a distinct email-link provider. You cannot tell “just completed email link step-up” from “signed in with email/password” using provider alone. Last login time + provider is an unreliable gate before a payment or similar action.

Unless you add another durable signal (custom claims from a Cloud Function on link completion, a dedicated step-up flag, etc.), Option 1 does not give the backend clean, auditable step-up proof.

Option 2: Custom OTP

Firebase stays primary auth. Step-up is a narrow API beside it: send a code to the email on the Firebase user (or your internal profile), verify it, issue a short-lived step-up token (or a server-side “verified until” timestamp keyed by user id).

Pros

Explicit proof — the backend knows an OTP was sent, verified, consumed, and when.
UX control — codes work when email is read on another device; resend, cooldown, and validation UI are yours.
One pattern, many gates — the same flow can guard payments, billing changes, and exports.
No Firebase metadata quirks — step-up does not depend on sign_in_provider semantics.

Cons

Build and maintain — generation, hashing, expiry, single-use, templates, monitoring (see security notes above).

Flow

sequenceDiagram
    participant U as User
    participant C as Web / Mobile Client
    participant A as Backend API
    participant D as Database
    participant E as Email Service

    Note over U,E: Send OTP
    U->>C: Start protected action (e.g. confirm payment)
    C->>A: POST /otp/send (Firebase ID token)
    A->>A: Verify token, resolve user
    A->>A: Generate OTP, store hash + expiry
    A->>E: Send email with code
    E-->>U: Email delivered
    A-->>C: 200 OK

    Note over U,E: Verify OTP
    U->>C: Enter code
    C->>A: POST /otp/verify (code)
    A->>D: Load pending OTP for user
    alt Valid and not expired
        A->>D: Mark OTP used
        A->>A: Issue step-up token or session flag
        A-->>C: 200 OK
        C-->>U: Continue protected action
    else Invalid or expired
        A-->>C: 401 / 400
        C-->>U: Retry or resend
    end

In words:

Send — Client calls /otp/send with the usual Firebase ID token. Backend verifies it, resolves the user, stores a hashed OTP with expiry, sends email. Never store plaintext codes.
Verify — Client submits the code. Backend checks hash, expiry, and single-use, marks the OTP consumed, returns a step-up credential (JWT claim, cookie, or server-side “verified until”).
Protected APIs — Require a fresh step-up credential within the policy window in addition to normal auth—for example before submitting a charge or updating a payout account.

Implementation notes (custom OTP)

Hash OTPs at rest; short TTL (for example 10–15 minutes); one active send per user with resend cooldown.
Rate limit send and verify; cap attempts per OTP before invalidation.
Email — state the action (“Confirm payment”, “Update billing”) so users know why they received a code.
Audit — log send/verify outcomes without logging the code.
Clients — one backend contract; web and mobile differ only in how they collect the code.

Takeaway

We wanted step-up without ripping out Firebase. Email-link step-up is the natural fit, but sign_in_provider reporting password breaks the simple “provider + last login” rule. Patching that with custom claims or Cloud Functions adds moving parts—and still blurs sign-in with step-up.

Custom OTP is more code, but it sits cleanly on top of existing Firebase auth: same ID token at the edge, plus an explicit step-up proof the API can enforce before payments and other sensitive work.

A practical rollout: ship a thin vertical slice (send + verify + one protected endpoint, such as payment confirmation) behind a feature flag, and load-test rate limits before widening coverage.

Share a Kafka Topic Only When Order Matters

Joung Park — Mon, 19 Oct 2026 05:16:34 +0000

We merged Kafka topics because the same services consumed the messages together. It felt simpler. Two incidents showed what we had really done: one shared topic means one shared queue, one failure domain, and one on-call problem—not a tidy folder in a repo.

My default rule: share a topic only when order matters for a clear stream (same partition key, same lifecycle). If urgency, retries, batch vs live traffic, or ownership differ, split the topics. Below I explain why we merged, what broke in production, when one topic is still right, and defaults that help teams avoid the same mistakes.

Why we merged topics

We had several topics. The same services read the events. Dashboards grouped them. We used the same domain words (“orders,” “billing,” “notifications”). Merging felt reasonable:

Fewer topic names and fewer ACL/schema subjects to manage
One obvious place per business area
One mental pipeline for producers and consumers

We also followed a familiar rule of thumb: combine topics that are mostly consumed together. If the same consumers normally subscribe to all of them, one topic can be simpler. Consumers can skip message types they do not handle—that is acceptable. Fetching from Kafka is cheap; ignoring even a large share of messages on a topic is often not a big cost. You only need a separate topic when consumers would ignore the vast majority of messages.

We put message type in the header so each record could be parsed and routed to the right handler. That made “one topic, many processes” feel safe: separate code paths without separate topic names.

That made merging feel low-risk. The incidents below showed we had still missed something.

What that advice does not cover

Co-consumption answers who reads the stream. It does not spell out what guarantees the log must provide—ordering, retention, retries, and what happens when one message type misbehaves.

You can join streams in application code, materialized views, or analytics. A Kafka topic is not a join key. It is an append-only log with contracts for ordering and retention. Headers that distinguish message type help you dispatch inside the consumer; they do not give you separate partitions, consumer lag, or retry boundaries. When message types need different contracts, one topic still ties them together. You often only notice that under load or during failure—not when consumers happily skip or route messages in code.

Incident 1: Bulk traffic blocked important work

Setup. Topic billing.events, partition key accountId, already handled live payment-captured and invoice-issued. Later we added ledger-export-row to the same topic so the export job could reuse the same pipeline and accountId keying. One consumer group still processed every message type in partition order.

What happened. After ledger-export-row joined billing.events, a heavy export run—or backlog after an outage—filled partitions for busy accountId values. payment-captured and invoice-issued kept flowing as before, but new records queued behind export rows on the same partitions. The consumer could not prefer the live types that had been fine on their own; offset order now applied to ledger-export-row too. Lag on payment-captured and invoice-issued climbed while the group chewed through export rows at a fixed rate.

Symptoms.

End-to-end latency alerts on payment-captured / invoice-issued flows while ledger-export-row lag was high on billing.events
Support reports: payments or invoices “stuck for hours” with no export job visibly “broken”
Healthy brokers, growing consumer lag on the shared group consuming billing.events

Mitigation. We stopped the ledger-export-row producer, watched lag on billing.events, and waited until the consumer drained enough backlog that payment-captured and invoice-issued caught up. It worked, but only as a manual lever—identify the bulk source, turn it off, hope you chose the right one.

Lesson. Live types on a topic do not stay safe when you add bulk later. Do not park ledger-export-row-class export on the same partition-ordered topic as payment-captured and invoice-issued unless you accept that slow export blocks fast billing for that key. Move export back to its own topic (for example billing.export-replay) or isolate it with a separate consumer group.

Incident 2: One broken process blocked everything on the topic

Setup. Topic order.events, partition key orderId. A message-type header routed each record—for example order-placed, payment-authorized, and return-initiated—to its own handler with different validation and idempotency rules. Separate code paths, but still one topic and one consumer group per partition, processed in offset order.

What happened. A return-initiated message was built from bad data in the database (invalid refund amount on the return record). The returns handler rejected it, the message retried, and the consumer could not commit past that offset on the partition. Later records on the same partition—including order-placed and payment-authorized for other orders that hashed to that partition—waited behind the failing return-initiated. Headers picked the right handler; they did not unblock the shared queue.

Symptoms.

Log spikes for return-initiated validation failures on order.events
Stalled processing for order-placed / payment-authorized on the same partition despite healthy payloads
Retries on the bad return-initiated record amplifying load; on-call debating skip, seek, or redeploy on order.events

Mitigation. We fixed the invalid data in the database—the source of the bad refund amount—then replayed or retried until the return-initiated handler could succeed and the partition offset could move. order-placed and payment-authorized traffic resumed on that partition. Patching DB rows to unblock a Kafka partition was slow, risky, and coordination-heavy; it should not be the routine way to recover order.events.

Lesson. Routing return-initiated by header is not the same as isolating it. One topic still shares retry policy and poison-message risk across types—and a bad row in the database can stall unrelated message types on the same partition. A dedicated topic (for example order.returns) or an isolated consumer + DLQ for return-initiated shrinks blast radius so returns work does not hold up fulfillment or payment events.

The rule: share a topic only when order matters

Share a topic when all of this is true:

Events form one logical sequence for one id (aggregate, entity, or workflow instance).
Consumers must process that sequence in order to stay correct—not just for easier debugging.
You can name the partition key for that sequence (orderId, accountId, shardId, …).

Kafka orders messages within a partition. It does not order across keys. It does not let you prioritize message type A over type B on the same partition.

When one topic is the right model

Example: account lifecycle for one account.

Topic: account.events (or your environment prefix)
Key: accountId
Events: account-opened → limit-set → debit-applied → account-closed

If you rebuild account state from the log, you need order per account. One topic, stable keys, and documented schema changes fit well.

When the same domain still needs multiple topics

You may still say “account” or “order” in conversation—but need different guarantees:

Concern	Split because
Live lifecycle vs bulk backfill	Different urgency and lag tolerance
Commands vs analytics fan-out	Strict order for state; best-effort for metrics/search
Process A vs process B handlers	Different failures, retries, and owners
Notifications vs ledger	Different SLOs and idempotency rules

Example split:

order.lifecycle — key = orderId, state machine, strict order
order.backfill — bulk replay, isolated consumers, not on the live path
order.notifications — key = orderId or userId, own retry/DLQ

For experienced teams: Draw topic boundaries around SLOs and failure domains. DDD boxes can guide naming; the log contract is what you must get right.

Recommendations

Do not share a topic when…

Interactive and batch/export/replay traffic need different latency.
One message type can poison the queue or retry forever while others are fine.
Different teams ship on different cycles—you do not want one bad deploy to block another team’s events.
One event type can spike 10× and define lag for everything else.
“We read them together” is only true in a UI or report—join in the app, not in the broker.
A message-type header (or similar) routes work in code—it does not replace separate topics when lag, retries, or failure domains must differ.

Creating a new topic per feature is usually correct

Teams ask: “Will we drown in topics?” Creating topics is cheap. Coupled outages are expensive. A feature with its own producers, consumers, retention, and on-call owner deserves its own topic name and a clear ordering story—not a slot on a crowded legacy topic because the noun matches.

What grows with topic count is governance: ACLs, Schema Registry subjects, dashboards, naming rules—not a cliff where the cluster stops accepting topics. Invest there instead of minimizing topic count for its own sake.

Defaults worth putting in team standards

Document what the partition key means for “order.”
One consumer group per service that owns processing; do not share groups to save machines.
DLQ and retry policy per pipeline—not one policy for unrelated handlers on a mega-topic.
Backfill on dedicated topics or controlled replay; do not mix with live traffic without a design review.
Runbooks: owner, retention, key, ordering guarantee, and what to do when lag spikes—without “patch the database to unblock the partition” as step one.

Request/reply and channel separation

In many microservice setups (including NestJS/Kafka-style patterns), request and reply use two topics. That is on purpose: different consumers, correlation rules, and scaling. Same idea—order may matter per correlation id on each channel, not “one topic for all RPC-shaped traffic.”

Closing

We merged topics because consumption looked shared. Incident 1: bulk and urgent traffic must not share an ordered queue if recovery means stopping producers and waiting. Incident 2: unrelated processes on one topic share retries and poison messages—and fixing bad rows in the database to unblock the log is a bad habit, not a strategy.

Default: Share a topic only when order matters for a defined key and lifecycle. Split on urgency, failure isolation, and ownership before the next 2 a.m. drain.

This week, pick one busy shared topic and ask: mixed urgency? mixed handlers? one retry policy for everything? If yes, plan a split—the incidents above are what that review prevents.

Find Similar Sentences Using Embeddings and a Vector Database

Joung Park — Thu, 15 Oct 2026 12:05:56 +0000

Why I Started Exploring Semantic Search

I'm building Second Memory, a personal memory application that allows users to save their memories and retrieve them later through natural-language questions.

One challenge is that users don't necessarily remember the exact words they used when saving a memory. They might remember its context or meaning instead.

For example, a user might have saved these memories:

"I forgot my umbrella and got soaked in the rain."
"I lifted weights at the gym after work."
"I caught up with John over coffee yesterday."
"I stayed up late debugging my application."

Later, the user asks:

"Who did I meet recently?"

The most relevant memory would be:

"I caught up with John over coffee yesterday."

A traditional keyword search might miss this because "meet" and "caught up" are different expressions.

This is where semantic search comes in. Instead of relying on exact keyword matches, we can search for text with similar meanings using embeddings.

In this post, I'll explore how to generate embeddings, calculate cosine distance, find relevant sentences, and perform vector searches using PostgreSQL with pgvector.

The examples use a locally runnable embedding model, so no API key or paid embedding service is required.

Generate Embeddings

An embedding is a numerical representation of text. An embedding model converts text into a vector, allowing us to compare sentences mathematically.

For this demonstration, I'll use sentence-transformers/all-MiniLM-L6-v2, a small, open-source embedding model that runs locally and produces 384-dimensional vectors.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

sentences = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

embeddings = model.encode(sentences)

print(embeddings.shape)
# (4, 384)

print(embeddings)
# [[ 0.04035296  0.10288464  0.09306896 ... -0.00502765 -0.13130508  0.05253862]
#  [-0.07009698  0.04319851  0.00160278 ... -0.06155823 -0.09415536 -0.01941661]
#  [-0.04802262 -0.01395144  0.12706237 ... -0.0639021  -0.03755342 -0.06052431]
#  [ 0.00379441  0.01192298 -0.02876158 ... -0.02021407 -0.0892337   0.05982817]]

The model converts each sentence into a vector containing 384 floating-point values.

The shape (4, 384) represents four sentences, each with a 384-dimensional embedding.

Each number is a coordinate in a high-dimensional vector space. Individual dimensions don't necessarily correspond to specific concepts such as people, places, or activities. Instead, semantic information is distributed across the vector.

The output is abbreviated with ..., but each embedding contains all 384 values.

Looking at these numbers alone doesn't tell us which sentences are semantically related. To find that out, we need to compare their vectors.

Using an Embedding API

Embeddings can also be generated through an API rather than running a model locally.

For example, OpenAI provides an embeddings endpoint:

curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-small",
    "input": "I caught up with John over coffee yesterday."
  }'

The response contains an embedding vector, just like the local model.

The concept is the same, although different models produce different vector dimensions and values. Embeddings generated by different models generally shouldn't be mixed in the same similarity search.

For the rest of this post, I'll continue using the local model so the examples remain reproducible without an API key.

Calculate Cosine Distance

Now that we have embeddings, we need a way to measure how closely two sentences are related.

One commonly used metric is cosine distance, which compares the directions of two vectors rather than their magnitudes.

Let's calculate the distance between two sentences.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

embeddings = model.encode([
    "I forgot my umbrella and got soaked in the rain.",
    "Who did I meet recently?"
], normalize_embeddings=True)

# Cosine distance using normalized vectors
distance = 1 - np.dot(embeddings[0], embeddings[1])

print(f"Cosine distance: {distance:.4f}")
# Cosine distance: 0.7981

The normalize_embeddings=True option scales each vector to unit length.

For normalized vectors, the dot product (np.dot()) is equivalent to cosine similarity. Subtracting it from 1 gives us cosine distance:

cosine distance = 1 - cosine similarity

Sentence Transformers also provides model.similarity(), which calculates cosine similarity by default. The equivalent calculation would be:

distance = 1 - model.similarity(
    embeddings[0:1], embeddings[1:2]
).item()

However, using NumPy keeps the distance calculation independent of Sentence Transformers. The same approach works with embeddings returned by OpenAI or other embedding APIs.

A smaller cosine distance indicates greater similarity. A distance of 0 means the vectors point in the same direction.

Why Cosine Distance?

There are several ways to compare vectors:

Metric	Description
Cosine distance	Compares vector directions, ignoring magnitude
Euclidean distance	Measures straight-line distance between vectors
Inner product	Measures vector alignment and is affected by magnitude

For text embeddings, we're generally interested in semantic relationships rather than differences in vector magnitude, making cosine distance a practical default.

When embeddings are L2-normalized, cosine distance, Euclidean distance, and inner product produce equivalent rankings. The choice of metric matters more when embeddings aren't normalized or when the model recommends a particular metric.

We'll use cosine distance throughout this post because pgvector supports it directly through the <=> operator.

Find the Most Relevant Memory

Now let's apply cosine distance to our original example and find the memory most relevant to the question:

"Who did I meet recently?"

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

query = "Who did I meet recently?"

memory_embeddings = model.encode(texts, normalize_embeddings=True)
query_embedding = model.encode(query, normalize_embeddings=True)

# Cosine distance using normalized vectors
distances = 1 - np.dot(memory_embeddings, query_embedding)

for i in np.argsort(distances):
    print(f"{distances[i]:.4f} - {texts[i]}")
# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.

This time, we're comparing a single query against multiple candidate memories.

The results are sorted in ascending order because smaller cosine distance indicates greater relevance.

As expected, the memory about meeting John ranks first, even though the question uses "meet" while the memory uses "caught up".

This demonstrates semantic retrieval: finding related meanings rather than matching exact words.

However, the closest result isn't guaranteed to be relevant in every case. The ranking depends on the embedding model and the available memories.

In a production system, we'd need to evaluate retrieval quality using representative queries and data.

Introducing pgvector

So far, we've generated embeddings and calculated distances in Python. But generating embeddings for every stored memory on each query would be inefficient.

A better approach is to generate embeddings when memories are created or updated, persist them, and search against the stored vectors.

For this, I'll use pgvector, a PostgreSQL extension that adds vector storage and similarity search capabilities.

Before using it with text embeddings, let's look at a simple example with two-dimensional vectors.

Store Vectors

First, enable the extension and create a table:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE examples (
    id SERIAL PRIMARY KEY,
    embedding VECTOR(2)
);

Insert three vectors:

INSERT INTO examples (embedding)
VALUES
    ('[1,0]'),
    ('[0.8,0.6]'),
    ('[0,1]');

Using two-dimensional vectors makes it easier to see how pgvector calculates distances.

Search by Cosine Distance

Let's find the vectors closest to [1,0].

SELECT
    embedding,
    embedding <=> '[1,0]' AS distance
FROM examples
ORDER BY distance;

The <=> operator calculates cosine distance.

The results are:

 embedding | distance
-----------+----------
 [1,0]     | 0
 [0.8,0.6] | 0.2
 [0,1]     | 1

The first vector has a distance of 0 because it points in the same direction as the query.

The second vector has a distance of 0.2, while the third is orthogonal to the query and has a distance of 1.

The important point is that pgvector doesn't understand the meaning of text. It stores vectors and performs mathematical operations on them.

The embedding model is responsible for producing vectors that represent semantic relationships.

Vector Indexes

Without an index, PostgreSQL performs an exact nearest-neighbor search.

For larger datasets, pgvector supports approximate nearest-neighbor indexes such as HNSW and IVFFlat.

For example, we can create an HNSW index:

CREATE INDEX examples_embedding_hnsw_idx
ON examples
USING hnsw (embedding vector_cosine_ops);

The search query itself doesn't need to change.

PostgreSQL can use the index to accelerate searches, although the query planner may choose a sequential scan for small datasets.

Approximate indexes can improve search performance by trading off some retrieval accuracy.

Embedding Dimensions and Index Limitations

One practical consideration when using pgvector is the number of dimensions in an embedding.

pgvector supports storing vectors with up to 16,000 dimensions, but standard HNSW and IVFFlat indexes using the vector type support up to 2,000 dimensions.

This becomes relevant when using models such as OpenAI's text-embedding-3-large, which produces 3,072-dimensional embeddings by default.

One option is to use a shorter embedding supported by the model.

OpenAI's text-embedding-3 models use Matryoshka Representation Learning (MRL), which allows shorter embeddings to retain useful semantic information.

For example, we can request a 1,536-dimensional embedding from text-embedding-3-large:

curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "text-embedding-3-large",
    "input": "I caught up with John over coffee yesterday.",
    "dimensions": 1536
  }'

This reduces storage requirements and allows the embedding to fit within pgvector's standard vector index dimension limit.

Another option is pgvector's halfvec type, which supports indexing up to 4,000 dimensions using half-precision values.

The right approach depends on retrieval quality, storage, precision, and performance requirements.

For our example using 384-dimensional embeddings, the standard vector type is sufficient.

Store and Search Embeddings with pgvector

Now that we've covered embedding generation, cosine distance, and pgvector, let's bring everything together.

We'll store the four memories with their embeddings in PostgreSQL and retrieve them using cosine distance.

import psycopg
from pgvector.psycopg import register_vector
from sentence_transformers import SentenceTransformer

DB_URL = "postgresql://user:password@localhost:5432/mydatabase"

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

texts = [
    "I forgot my umbrella and got soaked in the rain.",
    "I lifted weights at the gym after work.",
    "I caught up with John over coffee yesterday.",
    "I stayed up late debugging my application.",
]

query = "Who did I meet recently?"

DIMENSIONS = 384

with psycopg.connect(DB_URL) as conn:
    conn.execute("CREATE EXTENSION IF NOT EXISTS vector")
    register_vector(conn)

    conn.execute(f"""
        CREATE TABLE IF NOT EXISTS journal (
            id BIGSERIAL PRIMARY KEY,
            content TEXT NOT NULL,
            embedding VECTOR({DIMENSIONS}) NOT NULL
        )
    """)

    embeddings = model.encode(texts)

    # Reset the demo data
    conn.execute("TRUNCATE TABLE journal RESTART IDENTITY")

    for content, embedding in zip(texts, embeddings):
        conn.execute(
            """
            INSERT INTO journal (content, embedding)
            VALUES (%s, %s)
            """,
            (content, embedding)
        )

    conn.commit()

    query_embedding = model.encode(query)

    results = conn.execute(
        """
        SELECT
            content,
            embedding <=> %s AS distance
        FROM journal
        ORDER BY embedding <=> %s ASC
        LIMIT 4
        """,
        (query_embedding, query_embedding)
    ).fetchall()

    print(f"Query: {query}")

    for content, distance in results:
        print(f"{distance:.4f} - {content}")
# Query: Who did I meet recently?
# 0.6097 - I caught up with John over coffee yesterday.
# 0.7981 - I forgot my umbrella and got soaked in the rain.
# 0.8151 - I lifted weights at the gym after work.
# 0.8454 - I stayed up late debugging my application.

There are two key differences from the in-memory implementation.

1. Embeddings are persisted

pgvector provides the VECTOR(384) data type, allowing embeddings to be stored alongside their original text.

register_vector(conn) integrates pgvector with psycopg so NumPy arrays can be passed directly as parameters.

In a real application, embeddings would be generated when memories are created or updated, rather than on every query.

2. Similarity search moves into PostgreSQL

Instead of loading every embedding into Python to calculate distances, we let PostgreSQL perform the search.

The <=> operator calculates cosine distance, and the query returns memories ordered by their distance from the query embedding.

As shown in the output, the ranking matches our earlier Python example, with the memory about meeting John appearing first.

For larger datasets, vector indexes can improve search performance without changing the core retrieval logic.

Conclusion

We've explored how embeddings represent sentence meanings numerically, how cosine distance helps identify related text, and how to retrieve relevant sentences even when their wording differs.

We started with a locally runnable embedding model, calculated distances in Python, and then introduced pgvector using simple two-dimensional vectors.

Finally, we combined everything into a PostgreSQL-backed semantic search implementation.

The same retrieval approach can be used in a Retrieval-Augmented Generation (RAG) system, where retrieved text is provided to an LLM as context for generating an answer.

There are further considerations for production systems, including vector indexing, hybrid search, relevance thresholds, and retrieval evaluation.

But the core mechanism is straightforward: generate embeddings, calculate distances, and retrieve the closest matches.

Why I Chose Third-Party (Firebase) Authentication

Joung Park — Fri, 09 Oct 2026 06:12:00 +0000

Authentication is essential to Second-Memory, but it isn't its core business.

The goal of Second-Memory is to help users capture their thoughts and retrieve meaningful information from them. Building an authentication system doesn't directly advance that goal.

There's also a user-experience consideration. People already manage many online accounts. Asking them to create and remember another password adds friction, while social login lets them sign in with an account they already have.

Build vs. Buy

I considered two approaches: building authentication in-house or using an existing service.

Consideration	In-house authentication	Firebase Authentication
Initial development	Build authentication flows and supporting infrastructure	Integrate an existing service
Security	Implement and maintain security controls	Firebase manages core authentication infrastructure; the application still has security responsibilities
Maintenance	Handle security updates and operational issues	Google maintains the service
User experience	Users may need another account and password	Users can sign in with a supported existing identity
Engineering cost	More development and ongoing maintenance	Less implementation and maintenance work

For a project like Second-Memory, building authentication from scratch would mean taking on additional complexity without providing much differentiation.

What Does In-House Authentication Involve?

Building authentication is more than verifying a username and password. It means taking responsibility for several areas.

1. Security

Store passwords using an appropriate password-hashing algorithm.
Protect against brute-force attacks and credential stuffing.
Implement secure session management, including expiration and revocation.
Protect account recovery and verification flows against account takeover.
Manage secrets, security logging, monitoring, and incident response.

2. Account lifecycle

Registration and email verification
Login and logout
Account recovery and password resets
Account deletion and related data cleanup
Session revocation and suspicious-login handling

3. Social login integration

Supporting social login involves integrating with external identity providers and handling their authentication flows. Standards such as OpenID Connect (OIDC) and OAuth 2.0 are commonly involved, depending on the provider and flow.

The application must also handle identity mapping, account linking where appropriate, and the relationship between an external identity and its own user records.

4. Ongoing operations

Authentication requires continued attention after launch. Security vulnerabilities emerge, dependencies need updates, and authentication flows must be tested as the application evolves.

The effort doesn't end when login starts working.

The Decision

I don't have specific authentication requirements that justify building and maintaining my own system.

I'll use Firebase Authentication and make social login the preferred way to sign in. This avoids asking users to create another password and lets me rely on an established authentication service.

There may be usage costs and some provider dependency, but building authentication in-house also has costs in development time, maintenance, and security responsibilities.

For Second-Memory, Firebase Authentication is a practical choice because it meets my needs without requiring me to build and operate an authentication system myself.

However, using a third-party authentication service doesn't eliminate application security responsibilities. Second-Memory still needs to validate identities correctly, enforce authorization in the backend, and ensure users can access only their own data.

How Authentication Works in Second-Memory

Second-Memory has web and mobile clients, plus backend services. Both clients use Firebase Authentication, while the backend verifies each request and maps the Firebase identity to an internal user record.

The authentication flow

sequenceDiagram
    participant C as Web / Mobile Client
    participant F as Firebase Authentication
    participant A as Backend API
    participant D as Second-Memory Database

    C->>F: Sign in
    F-->>C: Firebase ID token
    C->>A: API request + Bearer token
    A->>A: Verify token and extract uid
    A->>D: Find user by firebaseUid
    D-->>A: Return internal user record
    A->>A: Authorize request
    A-->>C: API response

The flow has six steps:

Sign in: The client signs in with Firebase Authentication and obtains a Firebase ID token.
Call the API: The client sends the token in the Authorization: Bearer <token> header.
Verify the token: The backend verifies the token using the Firebase Admin SDK. It must not trust a token simply because it can decode it.
Extract the Firebase UID: After verification, the backend reads the user's Firebase UID from the token.
Resolve the internal user: The backend looks up the user record using firebaseUid.
Authorize the request: The backend uses the internal user ID to enforce access controls and scope data operations to that user.

Why maintain our own user table?

Firebase provides authentication, but Second-Memory still needs its own application-level user record.

For example:

id	firebaseUid
`usr_123`	`firebase-uid-abc`
`usr_456`	`firebase-uid-xyz`

The id is Second-Memory's internal user ID, while firebaseUid links the record to Firebase Authentication. The firebaseUid column should have a unique constraint.

This separation has a few advantages:

Domain independence: Memories and other application entities can reference Second-Memory's internal user ID.
Consistent authorization: Backend services can use the same internal identity when checking data access.
Flexibility: If the authentication provider changes in the future, the application's domain model doesn't need to use provider-specific identifiers everywhere.

The backend must still enforce authorization on every relevant request. A valid Firebase token proves the caller's authenticated identity; it does not automatically grant access to a particular memory or resource. The backend should derive identity from the verified token, not trust a user ID supplied by the client.

How this fits the client and backend stack

The same authentication principle applies across the different parts of Second-Memory, although their implementations differ.

Next.js web app: Configure Firebase for the web client, implement sign-in, and attach the ID token to API requests.
Expo mobile app: Configure Firebase Authentication for React Native, handle session persistence, and attach the ID token to API requests.
Backend services: Use the Firebase Admin SDK to verify ID tokens, resolve the internal user, and enforce authorization.

The important distinction is that the clients obtain tokens, but the backend is responsible for verifying them and deciding what the authenticated user is allowed to access.

Conclusion

For Second-Memory, authentication is necessary infrastructure, not a feature I need to reinvent.

Firebase Authentication reduces the amount of authentication code I need to build and maintain. Keeping an internal user record also lets Second-Memory separate external identity verification from application-level identity and authorization.

The goal isn't to eliminate security responsibilities. It's to use an established service for authentication while keeping control of the application's own data and access rules.

From Monorepo to Turborepo: Managing Multiple Projects

Joung Park — Fri, 09 Oct 2026 04:42:43 +0000

Why a Monorepo?

A monorepo is a single repository containing multiple distinct projects, with well-defined relationships. — Monorepo Explained

A monorepo makes sense when multiple projects have meaningful relationships:

Shared libraries and utilities — reuse common code across projects.
Shared contracts — keep API types consistent between producers and consumers.
Cross-project changes — update related projects in a single change.
Shared tooling and rules — maintain consistent development practices.

If projects have no meaningful relationships, separate repositories may be simpler.

For Second-Memory, Web, Mobile, and Backend need to agree on API contracts. Sharing TypeScript API types helps keep those contracts consistent.

Web uses React/Next.js, while Mobile uses React Native/Expo, so their UI implementations remain separate. Code is shared only where it genuinely works across both platforms.

Keeping related projects together makes coordinating changes and managing shared contracts easier.

How do we manage their packages and dependencies?

Why Workspaces?

Workspaces are a set of features in the npm CLI that provide support for managing multiple packages from a local file system within a single top-level root package. — npm Workspaces

A monorepo can work without workspaces. Each project can have its own package.json and manage dependencies independently.

For example:

second-memory/
├── apps/
│   ├── web/
│   └── mobile/
├── services/
│   ├── memory-service/
│   ├── ask-service/
│   └── embedding-worker/
└── packages/
    ├── api-types/
    └── ui-component/

Suppose both Web and Memory Service depend on api-types.

Without workspaces, each project needs to manage its dependencies and local references separately. A project might reference the shared package using a local file dependency:

{
  "dependencies": {
    "@second-memory/api-types": "file:../../packages/api-types"
  }
}

The relative path depends on the consuming project's location. Dependencies must be installed and managed in each project, and keeping local references and updates consistent requires additional care.

With workspaces, the root configuration defines which directories contain packages.

For example, with pnpm:

pnpm-workspace.yaml

packages:
  - "apps/*"
  - "services/*"
  - "packages/*"

Each package declares its own dependencies in its package.json. For example, Web can depend on the shared API types package:

apps/web/package.json

{
  "dependencies": {
    "@second-memory/api-types": "workspace:*"
  }
}

The shared package identifies itself in its own package.json:

packages/api-types/package.json

{
  "name": "@second-memory/api-types",
  "version": "1.0.0"
}

The workspace:* protocol explicitly tells pnpm to use the local workspace package.

Dependencies can then be installed from the repository root:

pnpm install

pnpm recognises the workspace packages and links local dependencies, making it easier to manage relationships between projects.

Without Workspaces vs With Workspaces

Area	Without workspaces	With workspaces
Install dependencies	Install separately in each project	Install from the root
Reference shared packages	Configure local file dependencies or links	Declare dependencies between workspace packages
Manage package relationships	Maintain references across projects	Define participating packages in the root configuration
Develop shared packages	Manage local references and updates carefully	Work with linked local packages
Run builds and tests	Run scripts manually or maintain root scripts	Still requires scripts or task orchestration

A workspace reduces manual package and dependency management. It provides a way to define packages and their local dependencies within the monorepo.

However, managing packages is only part of the problem. As the number of projects and dependencies grows, coordinating builds, tests, and other tasks becomes another challenge.

Why Task Orchestration?

With only a few projects, running build and test commands manually may be enough. As the repository grows, two problems become more noticeable.

Problem 1: Running Only the Necessary Tasks

The dependencies in Second-Memory look like this:

flowchart TD
    AT[api-types] --> W[apps/web]
    AT --> M[apps/mobile]
    AT --> MS[memory-service]
    AT --> AS[ask-service]

    UI[ui-component] --> W
    UI --> M

    EW[embedding-worker]

Suppose I change packages/ui-component.

Web and Mobile depend on it, but Memory Service, Ask Service, and Embedding Worker do not.

Ideally, I want to run checks for the affected projects without wasting time checking unrelated services.

With workspaces alone, I can run individual commands using package filters:

pnpm --filter web test
pnpm --filter mobile test

But I still need to know which projects depend on the changed package and decide which commands to run.

As the dependency graph grows, maintaining this selection manually becomes more difficult.

Task orchestration can automate the selection of relevant tasks based on package relationships, configured task dependencies, and changes.

Problem 2: Managing Dependencies Between Tasks

Some tasks need to run before others.

For example, if a service consumes a shared package's compiled output, that package needs to be built before the service.

I could maintain a script to enforce the order:

{
  "scripts": {
    "build:memory": "pnpm --filter api-types build && pnpm --filter memory-service build"
  }
}

This works for a simple case, but the scripts become harder to maintain as more packages and task dependencies are added.

Ideally, I want to define task relationships once and let a tool determine the correct execution order.

How Task Orchestration Helps

Task orchestration adds a layer above workspace package management. It coordinates tasks such as builds, tests, and linting across packages.

Task Graph

A task graph describes dependencies between tasks and helps determine their execution order.

For example, if api-types must be built before its consumers:

flowchart TD
    A[Build api-types] --> B[Build memory-service]
    A --> C[Build ask-service]
    A --> D[Build web]
    A --> E[Build mobile]

The graph should reflect the actual build requirements. If consumers compile the shared package's source directly, a separate build step may not be necessary.

Turborepo uses the workspace package dependency graph together with configured task dependencies to determine execution order. The package relationships provide the foundation; task configuration defines how tasks depend on one another.

Affected Tasks

When a package changes, Turborepo can use Git change information and the dependency graph to select affected packages and their relevant tasks.

Change api-types: Web, Mobile, Memory Service, and Ask Service may be affected.
Change ui-component: Web and Mobile may be affected.
Change embedding-worker: Its checks can run independently of the other projects.

This reduces the need to maintain manual build and test sequences as the repository grows. The result depends on the configured task graph, available Git change information, and the tasks defined by each package.

Additional Benefits

Technique	Benefit
Caching	Reuse valid task results instead of repeating unchanged work
Parallel execution	Run independent tasks concurrently while respecting dependencies
Remote caching	Share cached results across developers and CI when configured

These capabilities can improve development and CI performance, although the primary goal is to manage task relationships and execution more reliably.

Comparing the Approaches

Capability	Workspaces alone	Turborepo	Nx
Manage packages and local dependencies	Yes	Uses a workspace package manager	Yes
Run scripts across packages	Yes, using scripts and filters	Yes	Yes
Coordinate task dependencies	Manual scripts or additional tooling	Built-in task orchestration	Built-in task orchestration
Identify affected tasks	Manual selection or custom tooling	Supported	Supported
Cache task results	Requires additional tooling	Built-in	Built-in
Run independent tasks in parallel	Manual or script-based	Built-in	Built-in
Broader project tooling and plugins	Limited to package management	Task-focused	More extensive options

Workspaces alone can be sufficient for a small repository with simple build and test requirements. Turborepo and Nx become useful when coordinating tasks across project dependencies requires more automation.

Both provide task orchestration, affected-task capabilities, caching, and parallel execution. Nx offers a broader set of project tooling and plugins, while Turborepo focuses more directly on task orchestration.

For Second-Memory, I wanted the task orchestration capabilities without needing a broader project-management system.

Monorepo or Polyrepo?

Joung Park — Wed, 07 Oct 2026 05:52:16 +0000

After deciding to use microservices, the next question was:

How should I organise the codebase?

I chose a monorepo.

Second-Memory has several applications and services:

apps/
  web/
  mobile/

services/
  memory-service/
  ask-service/
  embedding-worker/

packages/
  api-types/
  ui-components/

They are independently deployable, but they are all part of the same product.

Monorepo vs Polyrepo

There is no universally better choice.

The trade-offs become clearer when looking at common development tasks:

Area	Monorepo	Polyrepo
Share code	Easy — reusable code can live in shared packages	Harder — publish packages or duplicate code
Share types	Easy — TypeScript types can be shared directly	Harder — types need separate versioning
Cross-project changes	Easy — related changes can be made in one PR	Harder — changes may require multiple PRs
Dependency management	Easy — dependencies can be managed consistently	More complex — each repo manages its own dependencies
Local development	Easy — one repository contains the whole system	More setup — multiple repositories need to be configured
CI/CD	More complex — need to detect affected projects	Easy — each repository naturally has its own pipeline
Service isolation	Harder — boundaries need to be enforced within the repository	Easy — repositories provide natural isolation
Access control	Harder — repository access generally covers the whole codebase	Easy — permissions can be managed per repository
Independent releases	Possible — CI/CD can deploy projects independently	Easy — each repository has its own release lifecycle

Neither approach is inherently better.

For Second-Memory, I valued easy development across the whole system more than repository-level isolation.

When to Choose Which

A monorepo can be a good fit when:

Small team or solo developer — one repository is easier to manage than several.
Projects share contracts or code — changes can be made and tested together.
Cross-project changes are common — one PR can update multiple applications or services.
The whole system is closely related — one repository makes it easier to understand and work across the product.

Polyrepo can be a better fit when:

Large or independent teams need clear ownership boundaries.
Strong repository isolation is important.
Projects have different release cycles.
Access control needs to differ between projects.

For Second-Memory, being a sole developer was one of the main reasons I chose a monorepo.

Managing several repositories would add overhead without giving me much benefit. I didn't need repository-level ownership boundaries between teams.

Instead, I wanted to make it easy to work across the web app, mobile app, backend services, and shared contracts.

What Should Be Shared?

A monorepo makes sharing easy.

But easy to share doesn't mean everything should be shared.

I wanted to keep the shared surface area small.

Shared Part	Used By	Why Share It?
API types	Web, Mobile, Backend	Keep API contracts consistent
Web-specific code	Web	Keep it inside `apps/web`
Mobile-specific code	Mobile	Keep it inside `apps/mobile`
Service domain logic	Individual service	Keep ownership within the service
Service implementation	Individual service	Preserve service boundaries

So instead of creating shared packages for everything, I only create a package when there is a genuine reason to share something.

For example:

apps/
  web/
  mobile/
services/
  memory-service/
  ask-service/
  embedding-worker/
packages/
  api-types/
  ui-components/

The API types are shared because they represent a contract between different parts of the system.

But a Web-only component stays in apps/web.

A Mobile-only utility stays in apps/mobile.

And Memory Service domain logic stays inside memory-service.

Share the contract, not the ownership

The fact that services live in the same repository doesn't mean they should share their internal implementation.

The API contract can be shared:

flowchart TB
    T[packages/api-types]

    W[Web] --> T
    M[Mobile] --> T
    B[Backend Services] --> T

while the services remain independently owned:

flowchart TB
    MS[Memory Service]
    AS[Ask Service]

    MS --> MD[Memory Domain]
    AS --> AD[AI Interaction]

    MD --> DB1[(Memory DB)]
    AD --> DB2[(Ask Data)]

Share contracts where it makes sense, but don't share service ownership.

Keeping the Shared Surface Small

There is another practical reason for limiting shared code.

The more projects depend on a shared package, the larger the potential impact of a change.

For example:

flowchart TB
    S[packages/shared]
    S --> W[Web]
    S --> M[Mobile]
    S --> ME[Memory]
    S --> A[Ask]

A change to shared may require all projects to be tested.

But a change here:

flowchart LR
    C[Change in apps/web] --> W[Web]
    W --> CI[Web CI/CD]

only affects Web.

This has implications for both CI/CD and application dependencies.

I don't want a large shared package that contains unrelated functionality used by almost every project.

Instead, I prefer small, focused shared packages with clear reasons for existing.

The CI/CD Trade-off

CI/CD is one area where polyrepo can be simpler.

With polyrepo, a change to memory-service naturally belongs to the memory-service repository.

flowchart TB
    R[memory-service repo]
    R --> CI[CI/CD]
    CI --> T[Test]
    CI --> B[Build]
    CI --> D[Deploy]

In a monorepo, multiple deployable units live in the same repository.

A change to:

services/memory-service/

shouldn't necessarily rebuild and deploy everything.

The pipeline needs to understand:

what changed
what depends on it
what needs to be tested
what needs to be built
what needs to be deployed

This becomes particularly important when shared packages are involved.

A change to ui-components may affect only web and mobile projects, while a change isolated to services/embedding-worker/ should ideally affect only the relevant pipeline.

The repository structure itself is simple, but the build and deployment system needs to become more intelligent as the project grows.

Why Monorepo for Second-Memory?

The main benefits for me were:

Easy sharing of API types
Easy cross-project changes
Easy local development
Easy dependency management
Easy visibility of the whole product
Less overhead for a solo developer

The trade-offs were:

More complex CI/CD
More tooling needed as the repository grows
Less repository-level isolation
The need to carefully control shared dependencies

For Second-Memory, I was willing to accept these trade-offs.

The goal wasn't to share as much code as possible.

The goal was to get the convenience of a monorepo while keeping ownership and dependencies clear.

That led to the next question:

How do I manage builds, dependencies, task execution, and caching across all these projects?

Microservices or Monolith?

Joung Park — Wed, 07 Oct 2026 04:18:28 +0000

The initial architecture had three main backend responsibilities:

Memory Service
Ask Service
Embedding Worker

But there was another architectural decision behind this:

Why make them separate services instead of building one backend?

A monolith would have been simpler.

For a small application, it would probably have been enough.

I still chose a microservice architecture for Second-Memory for three main reasons.

1. Clear boundaries

Memory and Ask have different responsibilities.

The Memory Service owns the memory domain:

create and manage memories
persist memory data
generate/retrieve relevant memories
manage embeddings

The Ask Service owns the AI interaction:

receive questions
retrieve relevant context
construct prompts
call the LLM
return the answer

They work together, but they represent different responsibilities.

I wanted those boundaries to exist at the service level rather than only as modules inside one application.

flowchart LR
    A[API Gateway] --> B[Memory Service]
    A --> C[Ask Service]
    B -.-> D[Memory + Vector]
    C -.-> E[LLM]

The Embedding Worker is separated for a different reason.

Embedding generation is asynchronous work and doesn't need to be part of the request that creates a memory.

flowchart LR
    A[Memory Service] --> B[Outbox]
    B --> C[Queue]
    C --> D[Embedding Worker]

The boundaries therefore follow the responsibilities and execution models of the system.

2. A system for learning AI engineering

Second-Memory is also a project for exploring how AI systems are built.

I didn't want the architecture to stop at:

flowchart LR
    A[Application] --> B[LLM API]

The system gave me an opportunity to work with several patterns that are common in production AI systems:

asynchronous processing
queues
workers
embeddings
vector search
retrieval
service-to-service communication
LLM integration

Using separate services makes these boundaries explicit and gives me a way to experiment with each part independently.

The goal isn't to use microservices simply because they are popular.

The architecture should help me learn and validate how these pieces work together as a real system.

3. Production readiness and scaling

I also wanted to design Second-Memory as something that could eventually grow into a production system.

The different components have different scaling characteristics.

If memory creation increases, I may need more Memory Service instances.

If embedding jobs increase, I can scale the workers independently.

If AI requests become the bottleneck, the Ask Service can scale independently.

This doesn't mean I need to scale everything independently today.

It means the architecture doesn't force unrelated workloads into the same deployment unit.

What about a monolith?

A monolith would still be a reasonable choice for Second-Memory.

It would have:

less infrastructure
simpler deployment
simpler local development
fewer service-to-service calls
less operational overhead

So why accept the additional complexity?

Because my goal wasn't only to build the simplest V1.

I wanted to build a system with clear boundaries that could evolve toward production scale, while giving me a practical environment to explore AI system architecture.

The trade-off was intentional.

I chose microservices not because a monolith couldn't work, but because the boundaries, asynchronous workloads, learning goals, and potential scaling characteristics made the additional complexity worthwhile.

Why the Outbox Pattern, Queue, and Embedding Worker?

Joung Park — Wed, 07 Oct 2026 03:47:01 +0000

The previous post covered why I chose pgvector for semantic search.

But there was another question:

When should a memory be embedded?

At first, it might seem simple:

Create memory
     ▼
Generate embedding
     ▼
Save embedding

But this makes memory creation dependent on the embedding process.

If the embedding provider is slow or unavailable, creating a memory could also fail or become slow.

I wanted to separate these two operations.

The architecture became:

Memory Service
      ▼
 PostgreSQL
      ▼
 Outbox Table
      ▼
 Queue / Job Runner
      ▼
 Embedding Worker
      ▼
 pgvector

1. The Outbox Pattern

When a user creates a memory, the Memory Service writes the memory and an outbox event in the same database transaction.

For example:

Transaction
┌─────────────────────────────┐
│ INSERT memory               │
│ INSERT outbox event         │
└─────────────────────────────┘

The important part is that they succeed or fail together.

This avoids a common problem with distributed systems:

Memory saved
     ▼
Embedding event lost

The outbox gives me a durable record of the work that needs to happen.

The outbox is not the queue itself.

It is a reliable bridge between the database transaction and asynchronous processing.

2. The Queue / Job Runner

A separate process reads pending outbox events and puts jobs onto a queue.

Outbox
   ▼
Queue
   ▼
Embedding Worker

This gives the embedding process some useful properties:

asynchronous processing
retries
independent scaling
failure isolation
no need to block the memory API

The user can save a memory without waiting for the embedding provider.

3. The Embedding Worker

The embedding worker has one main responsibility:

turn memory text into an embedding and store it in pgvector.

flowchart TD
    A[Get Memory] --> B[Generate embedding]
    B --> C[Store Vector]

This keeps embedding-specific logic out of the Memory Service's synchronous request path.

It also gives me a place to evolve the embedding pipeline later.

For example, I could change:

embedding models
chunking strategy
retry behaviour
batch processing
embedding dimensions

without changing the API used to create a memory.

4. Why asynchronous?

There is an important consequence of this design:

A newly created memory may not be immediately searchable.

There can be a small delay between:

Memory created
     │ asynchronous processing
     ▼
Embedding created
     ▼
Memory available for semantic search

I accepted this trade-off.

For Second-Memory, immediate persistence is more important than making the embedding operation part of the user's request.

This is essentially eventual consistency between the memory record and its vector representation.

5. Failure becomes easier to handle

The asynchronous design also changes how failures work.

If the embedding provider temporarily fails:

Memory
  ▼
Outbox
  ▼
Queue
  ▼
Embedding Worker - X -> Retry

The memory itself has already been safely stored.

The embedding job can be retried without asking the user to submit the memory again.

That separation was important to me.

The decision

The final flow became:

graph TD
    CM[Create Memory] --> PG

    subgraph PG [PostgreSQL]
            M[Memory]
            OE[Outbox Event]
    end

    OE --> Q[Queue]
    Q --> EW[Embedding Worker]
    EW --> PV[(pgvector in PostgreSQL)]

Each part has a different responsibility:

Component	Responsibility
Memory Service	Store the memory
Outbox	Reliably record the event
Queue	Deliver asynchronous work
Embedding Worker	Generate embeddings
pgvector	Store and search vectors

This added some complexity compared with simply generating the embedding inside the API request.

But it gave me something more important:

memory creation is no longer tightly coupled to the availability of the embedding pipeline.

That was the trade-off I wanted.

Vector database

Joung Park — Wed, 07 Oct 2026 03:34:17 +0000

Second-Memory stores things a user writes over time.

If a user later asks:

“What did I write about performance problems?”

A normal keyword search isn't always enough.

The memory might say:

“The database becomes slow when thousands of users query it at the same time.”

There may be no exact keyword match between the question and the memory.

This is where semantic search becomes useful.

1. From text to vectors

I can convert a piece of text into an embedding — a vector representation of its meaning.

For example:

"The database becomes slow when thousands of users query it at the same time."
         ↓
[Embedding model]
         ↓
[0.021, -0.183, ...]

The actual vector contains many dimensions, so it isn't meaningful to look at the individual numbers.

What matters is the relationship between vectors.

Texts with similar meanings tend to have vectors that are closer together.

So when a user asks a question, I can:

Question
Embedding
Vector similarity search
Relevant memories

This gives Second-Memory a way to retrieve memories based on meaning, rather than just matching words.

2. Why pgvector?

Once I decided to use embeddings, I needed somewhere to store and search them.

I considered:

	pgvector	Pinecone
Vector search	Yes	Yes
Relational data	Yes	No
Existing PostgreSQL	Yes	No
Separate infrastructure	No	Yes
Operational complexity	Lower	Higher
Good fit for V1	Yes	Yes

I chose pgvector.

The main reason wasn't that pgvector is necessarily better than Pinecone.

It was that Second-Memory already had a natural place for the vectors:

the Memory Service's database.

With pgvector, I could keep the memory and its embedding together.

erDiagram
    "Memory Service" ||--|| "PostgreSQL + pgvector" : utilizes
    "PostgreSQL + pgvector" ||--|{ Memory : contains

    Memory {
        uuid user_id
        text content
        vector embedding
    }

That kept the architecture simple.

3. Why not introduce a vector database?

A dedicated vector database could make sense at larger scale.

But introducing one also creates another system to operate and another boundary to manage.

For V1, I didn't see enough benefit to justify that complexity.

The Memory Service could own:

memory data
embeddings
vector search

all within the same data boundary.

That also reinforced one of the architectural principles from the previous post:

The service that owns the data should own access to it.

The Ask Service doesn't need to know whether semantic search is implemented with pgvector, Pinecone, or something else.

It simply asks the Memory Service for relevant memories.

4. What this gives me

With pgvector, the basic retrieval flow becomes:

User question
      ↓
Generate query embedding
      ↓
Memory Service
      ↓
pgvector similarity search
      ↓
Relevant memories
      ↓
Ask Service
      ↓
     LLM

This is the first important piece of the AI architecture.

The LLM doesn't need to know everything the user has ever written.

Instead, the system retrieves the memories that are most relevant to the current question and uses those as context.

The decision

For Second-Memory V1, I chose:

PostgreSQL + pgvector

because it gave me semantic search without introducing another database.

It was a pragmatic choice:

vector search
relational data
one data boundary
less infrastructure
simpler development

The First Architecture Draft

Joung Park — Wed, 07 Oct 2026 03:24:36 +0000

I had the product definition, user stories, and backlog.

Now it was time to design the system.

I started with the main responsibilities of Second-Memory and came up with my first architecture draft:

flowchart 
    W[Web Client] --> G[API Gateway / BFF]
    M[Mobile Client] --> G

    G --> A[Auth]
    G --> MS[Memory Service]
    G --> AS[Ask Service]

    MS --> D[(Database)]
    MS --> V[(Vector DB)]

    AS --> L[LLM Provider]

    AS --> MS

The diagram itself is fairly simple, but there were several design principles behind it.

1. BFF for the clients

Both the web and mobile clients communicate through the API Gateway / BFF.

I chose the Backend for Frontend (BFF) pattern so that the clients don't need to communicate directly with the internal services.

The BFF provides a client-facing API while hiding the internal architecture.

It also gives me a place to handle things such as authentication, routing, and client-specific requirements.

This means the clients don't need to know whether a request is handled by the Memory Service, Ask Service, or something else.

The internal architecture can evolve without making the clients aware of every change.

2. Each service owns its data

One principle I wanted to follow was:

Don't share databases between services.

A service owns the data that belongs to its responsibility.

In this architecture, the Memory Service owns both the database and vector database.

If another service needs memory data, it doesn't connect directly to either database.

Instead, it calls the Memory Service through an internal API.

For example:

flowchart 
    AS[Ask Service] -->|Internal API| MS
    MS[Memory Service] --> D[(Database)]
    MS --> V[(Vector DB)]

This keeps the data ownership clear and prevents other services from becoming coupled to the Memory Service's storage implementation.

If the storage implementation changes later, the other service shouldn't need to know.

3. Separation of concerns

Each component has a specific responsibility.

The BFF deals with client-facing communication.

The Memory Service deals with memory.

The Ask Service deals with the AI interaction.

The LLM generates the response.

The databases provide persistence.

This separation makes the architecture easier to reason about.

It also means that AI-specific logic doesn't have to spread throughout the rest of the application.

4. DDD and bounded contexts

The separation between Memory Service and Ask Service wasn't just about splitting the code into smaller services.

I was thinking about the different domains and responsibilities involved.

The Memory Service represents the memory domain:

creating memories
retrieving memories
searching memories
managing memory data

The Ask Service represents the AI interaction:

receiving a question
deciding what information is needed
requesting relevant memories
constructing context
calling the LLM
generating the answer

They are closely related, but they are not the same responsibility.

This is where the idea of bounded contexts from Domain-Driven Design helped me think about the boundaries.

The Ask Service uses the Memory Service, but it doesn't become responsible for managing memory itself.

5. Encapsulation and API contracts

The service boundary also creates an important form of encapsulation.

The Ask Service doesn't need to know how the Memory Service stores or searches memories.

It only needs to know what the Memory Service provides through its API.

For example, the Ask Service might ask for relevant memories, without knowing whether the Memory Service uses PostgreSQL, pgvector, a different vector store, or some other implementation in the future.

The API becomes the contract between the services.

This keeps implementation details inside the service that owns them.

These principles gave me the initial structure of Second-Memory.

It wasn't a complicated architecture, and that was intentional.

I wanted clear boundaries without creating unnecessary complexity.

With the boundaries in place, I could start looking at the individual technical decisions.

Putting on the Product Owner Hat

Joung Park — Wed, 07 Oct 2026 02:59:25 +0000

I had the idea for Second-Memory.

But at that point, it was still just an idea. It was too vague to start thinking about architecture or implementation.

I wanted to develop the idea further before I started building it.

So I put on the Product Owner hat.

I followed the kind of process I had experienced in my professional work.

I started by defining the initiative. I wrote down what I was trying to achieve and what I wanted the product to be.

Then I wrote a PRD.

Writing it made me think through the product in more detail: the problem, the user, the main use cases, the scope, and what I wanted to achieve with the first version.

From there, I created user stories.

This helped me look at the product from the user's perspective rather than thinking about how I was going to implement it.

I then organised the stories into an initial backlog and broke the work down into tasks that I could eventually implement.

There was another reason I wanted to document all of this.

I knew I would be using AI tools to help me build the project.

Instead of explaining the project from scratch every time I asked an AI coding tool for help, I could give it the product documents as context.

The PRD and user stories became a shared reference point — for me as the developer, and for the AI tools I was working with.

The idea started in my head as something fairly vague.

By the end of this process, I had something much more concrete:

an idea → a product definition → user stories → work I could build.

DEV Community: Joung Park

Migrate AMQ to Kafka

Why move from ActiveMQ to Kafka?

The important difference: queue versus log

Considerations when moving from ActiveMQ to Kafka

Migrating a live system without downtime

Tip: keep message handling separate from business logic

Ordering and grouping messages

How it works under the hood

ActiveMQ (JMSXGroupID)

Kafka (message key)

Implications

Key takeaway

The outcome

Step-Up Verification: OTP, Magic Link

Primary login vs step-up

When step-up matters

Options considered

Security concerns

Firebase email-link step-up

Custom OTP

Option 1: Firebase sign-in via email link

Why it is attractive

Where it breaks down

Option 2: Custom OTP

Pros

Cons

Flow

Implementation notes (custom OTP)

Takeaway

Share a Kafka Topic Only When Order Matters

Why we merged topics

What that advice does not cover

Incident 1: Bulk traffic blocked important work

Incident 2: One broken process blocked everything on the topic

The rule: share a topic only when order matters

When one topic is the right model

When the same domain still needs multiple topics

Recommendations

Do not share a topic when…

Creating a new topic per feature is usually correct

Defaults worth putting in team standards

Request/reply and channel separation

Closing

Find Similar Sentences Using Embeddings and a Vector Database

Why I Started Exploring Semantic Search

Generate Embeddings

Using an Embedding API

Calculate Cosine Distance

Why Cosine Distance?

Find the Most Relevant Memory

Introducing pgvector

Store Vectors

Search by Cosine Distance

Vector Indexes

Embedding Dimensions and Index Limitations

Store and Search Embeddings with pgvector

Conclusion

Why I Chose Third-Party (Firebase) Authentication

Build vs. Buy

What Does In-House Authentication Involve?

The Decision

How Authentication Works in Second-Memory

The authentication flow

Why maintain our own user table?

How this fits the client and backend stack

Conclusion

From Monorepo to Turborepo: Managing Multiple Projects

Why a Monorepo?

Why Workspaces?

Without Workspaces vs With Workspaces

Why Task Orchestration?

Problem 1: Running Only the Necessary Tasks

Problem 2: Managing Dependencies Between Tasks

How Task Orchestration Helps

Task Graph

Affected Tasks

Additional Benefits

Comparing the Approaches

Monorepo or Polyrepo?

ActiveMQ (`JMSXGroupID`)