DEV Community: Jason Shen

When Your Coding Agent Needs a Scribe, Not a Memory Engine

Jason Shen — Fri, 26 Jun 2026 04:58:25 +0000

Over the past few weeks, I have had several conversations about the right way to give AI coding agents persistent memory.

Some developers ask about AgentMemory. Others ask about Qiju, which I built and maintain.

My usual response is: they are not solving the same problem. But I have not written that distinction down clearly.

This post is my attempt to do that.

The same surface problem

Both tools address the same frustration.

Every new coding session starts without context. The agent does not know which file is authoritative. It does not know what was decided in the previous session. It does not know which approaches have already failed or which assumptions have been verified.

The developer spends the opening minutes of every session reconstructing what the previous session established.

Both tools try to reduce that cost. But they approach it differently, and the difference matters in practice.

What AgentMemory does

AgentMemory runs as a local server and hooks into the lifecycle events of your coding session automatically.

With Claude Code, Codex, or another supported agent, it captures:

every prompt submitted;
every tool call the agent makes;
every result the agent receives;
session start and session end.
Those observations are compressed using a language model, embedded using a local or cloud embedding model, and indexed with BM25 and vector search. At the start of the next session, the most relevant memories are retrieved automatically and injected into the conversation. You do not need to do anything.

AgentMemory has published reproducible benchmark results: 95.2% recall accuracy on LongMemEval-S. Semantic retrieval can find "database performance" when you actually wrote "N+1 query fix." Keyword search cannot do that.

What Qiju does

Qiju does not capture anything automatically.

When I want a record to persist, I or my agent intentionally invokes /qiju-log inside Claude Code, or $qiju-log inside Codex.

That produces a structured handoff record:

title:         Selected token refresh strategy
tags:          auth, architecture
search_terms:  refresh token, 401 retry
next_steps:    implement bounded retry, add expiry regression test
body:          Decision, evidence, rejected alternatives, and handoff notes.

The record is stored in plain JSONL files on the developer's machine. Nothing is embedded. Nothing leaves the machine. Retrieval is deterministic keyword and tag search.

In the next session, the agent searches for what it needs. The developer sees exactly what was recorded and why.

The real distinction

It is tempting to describe AgentMemory as "automatic Qiju," or Qiju as "manual AgentMemory." That is not quite accurate.

The two tools are answering different questions.

AgentMemory asks: what did the agent observe during previous sessions?

Qiju asks: what did the developer judge worth preserving about this project?

An internal log of every tool call is not the same as a structured record of why a decision was made, what evidence supported it, and which approaches were already rejected.

A memory engine accumulates what happened.
A record layer captures what matters.

Why the distinction is practical

Imagine a session in which I investigated an authentication bug, tried three approaches, found that the second one caused a regression, and decided to roll back the change and record a specific boundary condition in the architecture notes.

An automatic memory capture would produce:

observations of every file read;
observations of every command run;
observations of every test failure;
a compressed session summary.
That is useful background.

A deliberate Qiju record would produce:

the specific file that is the current ground truth;
the decision and the reason for it;
the two rejected approaches and why they failed;
the next step for the following agent.
The next agent does not need the full observational trace. It needs to know what was decided, where to find the evidence, and what not to repeat.

Those are different artefacts.

A concrete example

In the post I wrote about Codex's SQLite logging problem, I described this kind of record:

Ground truth:  design/pypi-packaging-execution-plan.md
Decision:      Use one canonical src/qiju package for source and PyPI installs.
Verified:      Wheel and sdist passed artifact inspection and installed-wheel smoke tests.
Rejected:      Renaming scripts to qiju only during the staging build.
Next:          Publish to TestPyPI and repeat the clean-install lifecycle test.

A complete session transcript might contain tens of thousands of words around those conclusions. The durable project record is much smaller.

The next agent does not need the whole session. It needs the right record.

The operational differences

Beyond philosophy, the two tools make different operational assumptions.

AgentMemory requires:

a Rust binary (iii-engine) running as a background daemon;
Node.js installed;
an embedding model, either local or from a configured API provider.
If an external embedding provider is configured, content leaves the machine. The local all-MiniLM-L6-v2 model avoids this but adds setup steps. AgentMemory supports over twenty agents including Claude Code, Codex, Cursor, Cline, Gemini CLI, and others. It runs on macOS, Linux, and Windows.

Qiju requires:

Python 3.11 or later;
nothing else.
Records stay in plain JSONL files on the developer's machine. They can be committed to git alongside the code. Qiju supports Claude Code, Codex, Kiro, and Cursor. It runs on macOS and Linux. Windows is not yet supported.

Where they compose

I do not think of these tools as competitors.

A team could run AgentMemory for automatic background capture — what the agent did, when, and in which files — and run Qiju for the intentional layer — what was decided and why.

AgentMemory recalls what the agent did.

Qiju records what the developer decided.

That is a natural division of responsibility.

Current limits of both

AgentMemory's automation is valuable precisely because most developers will not log anything if the effort is nonzero. But automatic capture at PostToolUse granularity produces high volume. The compression step reduces it, but the signal-to-noise ratio depends on how well the LLM identifies what was important. The daemon and embedding pipeline add operational complexity.

Qiju's intentional capture is a feature to some developers and a friction point to others. If you forget to log something, it is not recorded — though a Stop hook in your agent configuration can instruct the agent to run qiju-log before the session closes, which reduces the chance of an unrecorded session. There is no semantic search. Retrieval depends on using good keywords and tags at log time. The record is only as good as the effort put into writing it.

Neither tool solves all the problems. Both have been honest about their current limits.

My conclusion

The question of agent memory is not settled. The tooling is early and changing quickly.

If zero-friction automatic capture matters most, and you are comfortable with the operational requirements, AgentMemory is worth exploring.

If deliberate, auditable, file-based records matter most, and you want something with no background services and no external dependencies, Qiju fits that model better.

If both matter — the tools compose.

If You Disable Codex’s SQLite Logs, What Keeps Your Project Memory Alive?

Jason Shen — Tue, 23 Jun 2026 05:45:05 +0000

Developers have recently been discussing Codex continuously writing large amounts of diagnostic data into:

~/.codex/logs_2.sqlite

The visible database may remain relatively small while its SQLite WAL file keeps receiving frequent writes in the background.

Some users have reported:

excessive SSD writes;
rapidly growing WAL files;
slower responses;
lock contention;
startup or session reliability problems.

One unofficial workaround is to add a SQLite trigger that ignores new inserts into the logging table.

That may reduce the disk-writing problem.

But it also exposes a deeper architectural question:

If you disable an agent’s internal logs, what preserves the memory required to continue the project?

Diagnostic logs and project memory are not the same thing

An agent’s internal logs are primarily designed for the agent application itself.

They may contain:

network events;
trace output;
runtime state transitions;
low-level debugging information;
internal operational details.

Those logs can be useful for diagnosing Codex.

But they are not necessarily what the next coding agent needs when it resumes your project tomorrow.

The next agent usually needs something much smaller and more meaningful:

Which file is the current ground truth?
What decision was made?
Why was that decision made?
Which approaches have already failed?
What evidence has been verified?
What remains unfinished?
What should happen next?

That is project continuity.

It is not diagnostic telemetry.

What happens when you block the logs?

The trigger workaround attempts to stop new rows from being inserted into Codex’s local logging database.

The intended operational effect is straightforward:

reduce repeated SQLite writes;
slow or stop WAL growth;
reduce SSD pressure;
potentially improve responsiveness.

But it is still an unofficial workaround that modifies Codex’s internal database behavior.

It may reduce diagnostic visibility.

It may behave differently after a future Codex schema migration.

It may interfere with debugging.

It should not be presented as a guaranteed or permanent fix.

More importantly, it should not be treated as a project-memory strategy.

Even if Codex logging worked perfectly, an internal diagnostic database would still be the wrong place to depend on for durable project continuity.

Its schema belongs to Codex.

Its retention policy belongs to Codex.

Its format can change with Codex.

Its purpose is to help Codex operate and diagnose itself—not to preserve the smallest verified context required by the next agent.

The real performance gap is continuity performance

When developers hear “agent performance,” they usually think about:

model quality;
response speed;
token usage;
tool execution;
latency.

There is another kind of performance that matters just as much:

How quickly can a new session recover the correct context and continue working?

I call this continuity performance.

A fast model that spends twenty minutes rediscovering decisions, reopening the wrong files, repeating failed experiments, or misunderstanding the project’s current state is not performing well at the project level.

This becomes especially visible when:

the original session is gone;
context has been compacted;
the transcript is too large;
the agent changes;
local logs are cleared;
an internal database is disabled;
a tool changes its storage format.

The problem is not that every trace event disappeared.

The problem is that the next agent no longer knows what matters.

Where QiJu fits

QiJu is now publicly available on PyPI:

uv tool install qiju

It is a local-first, lossless record layer for AI coding agents.

It works with:

Claude Code;
Codex;
Kiro;
Cursor.

QiJu does not try to preserve every conversation line or every tool event.

It intentionally records the smaller set of facts that allow work to continue:

the ground-truth file or artifact;
the decision that was made;
the evidence supporting it;
the result that was verified;
the unresolved risk;
the next action.

A later Codex session—or Claude Code, Kiro, or Cursor—can search those records and continue from the correct state.

The goal is not to reconstruct the entire old session.

The goal is to help the next agent continue correctly.

QiJu is not “memory”

This distinction is important.

QiJu does not make a model remember more.

It creates an explicit record layer.

Ordinary agent memory tries to answer:

What does the model remember?

QiJu tries to answer:

Why do we know this?
Where is the proof?
Who produced the result?
Was it verified?
What should happen next?
Which agent should continue?

That makes the work auditable and handoffable instead of trapped inside one tool’s internal recall.

A useful way to think about it is:

Codex logs tell Codex what happened internally.
QiJu tells the next agent what matters for continuing the work.

A practical example

Imagine a Codex session has just completed a packaging migration.

During the work, the agent discovered that:

the public package must move to src/qiju/;
pyproject.toml must remain the version authority;
runtime JSON must be loaded through package resources;
uninstalling the CLI must preserve user records;
all supported coding-agent hosts must pass installed-wheel testing.

A complete transcript may contain tens of thousands of words around those conclusions.

The durable project record can be much smaller:

Ground truth:
design/pypi-packaging-execution-plan.md
Decision:
Use one canonical src/qiju package for source and PyPI installs.
Verified:
Wheel and sdist passed artifact inspection and installed-wheel smoke tests.
Rejected:
Renaming scripts to qiju only during the staging build.
Next:
Publish to TestPyPI and repeat the clean-install lifecycle test.

That is enough for the next agent to find the right file, understand the decision, and continue from the correct point.

It does not need every trace message.

It needs the right record.

The new update problem

Once QiJu is installed through PyPI, users can upgrade the CLI with:

uv tool upgrade qiju

But that creates another subtle problem.

The CLI may be updated while the copied SKILL.md files inside existing projects remain stale.

Those skills tell Claude Code, Codex, Kiro, and Cursor how to call QiJu.

So upgrading only the executable is not enough.

QiJu 0.5.3 adds:

qiju update

This refreshes the installed QiJu skills across registered projects.

Typical usage:

uv tool upgrade qiju
qiju update

You can also preview changes:

qiju update --dry-run

Refresh only one host:

qiju update --host codex

Or scan and backfill older projects that were created before the registry existed:

qiju update --scan-projects

This matters because continuity depends not only on preserving records, but also on keeping the agent integration itself current.

How project registration works

When you initialize QiJu in a project:

cd /path/to/project
qiju init --host codex

QiJu registers the project location.

Each project’s own:

.qiju/config.json

remains the source of truth for which hosts are enabled.

The registry tells QiJu where projects are.

The project config tells QiJu how each project is wired.

That separation allows qiju update to refresh projects without relying only on heuristic filesystem scans.

It also means projects outside standard search roots can still be updated, as long as they were registered during initialization.

For older installs, one migration command is enough:

qiju update --scan-projects

After that, ordinary updates can remain fast and deterministic.

What QiJu does not do

QiJu does not:

fix Codex’s SQLite logging behavior;
stop SSD writes by itself;
replace Codex diagnostics;
guarantee that the unofficial database trigger is safe;
preserve the Codex conversation interface;
reproduce every internal event from the previous session;
silently ingest all transcripts or prompts;
use embeddings or vector memory.

Capture is intentional.

You—or the agent under your instruction—decide what is worth recording.

That boundary is important.

QiJu is not another background logger.

It is the opposite: a deliberate, inspectable record of what matters.

A real workflow

Install QiJu:

uv tool install qiju

Connect it to a project:

cd /path/to/project
qiju init --host codex

During the session, ask the agent to record what matters:

$qiju-log

Later, in another session or another agent:

$qiju-search the last packaging decision

After upgrading QiJu:

uv tool upgrade qiju
qiju update

That keeps both the CLI and the installed agent skills current.

Why this matters after disabling logs

Suppose you apply the SQLite trigger workaround and Codex stops writing most of its local diagnostic events.

Operationally, that may reduce disk pressure.

But now the separation becomes clear:

Codex diagnostics
→ reduced or unavailable
QiJu project records
→ still available
Ground-truth files
→ still referenced
Decisions
→ still searchable
Next steps
→ still recoverable

The next agent may not know every internal event from the previous Codex run.

It does not need to.

It needs to know:

what was decided;
why;
where the evidence is;
what was verified;
what to do next.

That is how QiJu helps maintain continuity performance even when Codex’s internal logging is limited or cleared.

The broader design lesson

The Codex logging issue is a reminder that developers often mix several different kinds of memory:

Diagnostics — what happened inside the application.
Conversation history — what the user and agent said.
Repository state — what currently exists in the codebase.
Decision history — why the project reached its current state.
Continuation context — what the next agent needs to do next.

No single SQLite log, transcript, or Git repository fully represents all five.

Git can show what changed, but not always why.

A transcript may show why, but it can be too large and tied to one tool.

Diagnostic logs show runtime behavior, but mostly contain noise from the project’s perspective.

A durable agent workflow needs an explicit project record layer.

Current limits

QiJu is still in developer preview.

At the time of writing:

it supports macOS and Linux;
Windows is not yet supported or tested;
capture is intentional rather than automatic;
retrieval is deterministic keyword, tag, and regex search;
there are no embeddings or semantic ranking;
the CLI and host wiring may still evolve.

Those limits are deliberate.

The current priority is reliability, inspectability, and ownership.

My conclusion

Blocking noisy Codex logs may be a reasonable temporary response to excessive disk writes, provided users understand that it is unofficial and may affect diagnostics.

But the deeper conclusion is more important:

Your project memory should not disappear when an agent’s internal logs are disabled, cleared, corrupted, or redesigned.

Codex logs tell Codex what happened internally.

QiJu records the ground truth, decisions, evidence, and next steps that another agent needs to continue.

That separation lets you reduce dependence on internal diagnostic logs without sacrificing project continuity.

QiJu is available now:

uv tool install qiju

Project page:

https://pypi.org/project/qiju/

Source:

https://github.com/jasonshrepo/qiju

Star if you like QiJu(起居).

The question I am most interested in is:

If you stopped trusting your coding agent’s internal history tomorrow, what would you preserve so another agent could continue correctly?

From Prompting ChatGPT to Orchestrating AI Agents: Two Years as an Ordinary Engineer

Jason Shen — Sun, 21 Jun 2026 05:41:51 +0000

Two years ago, my interaction with AI was mostly limited to asking ChatGPT questions and checking its knowledge cutoff.

Today, I use multiple coding agents, connect company knowledge systems through MCP, run local models inside iOS applications, and maintain a project memory layer so different agents can continue each other’s work.

I am not an AI researcher.

I am an ordinary engineer who kept experimenting until AI gradually became part of both my development workflow and my daily life.

This is not a history of the AI industry. It is a personal engineering retrospective: what changed, what did not, and what I learned while moving from chat-based assistance to agent-driven development.

Stage 1: AI Was Impressive, but Unreliable

My first experience with ChatGPT was similar to that of many people.

The initial response was amazement.

The second response was scepticism.

It did not take long to discover that a language model could present incorrect information with extraordinary confidence. I learned the term “hallucination,” and asking about the model’s knowledge cutoff became part of my normal usage.

I later completed introductory AI courses from Andrew Ng and Microsoft.

I did not understand every technical detail, but those courses gave me enough context to recognise an important limitation:

A model producing a plausible answer is not the same as a system producing a reliable result.

That distinction became more important as I moved from asking questions to using AI to modify software.

Stage 2: Cursor Turned AI Into a Development Partner

For me, the real shift began with Cursor.

Before Cursor, AI was primarily a place I went to ask for help.

With Cursor, AI entered the development environment itself.

One Christmas, I spent three days almost entirely at home, building with Cursor. At the end of those three days, I had created my first iOS application.

Then I built another one.

And another.

Cursor made the feedback loop unusually short:

Describe an idea.
Generate an implementation.
Run it.
Observe the failure.
Ask for a correction.
Repeat.

This was my introduction to what later became widely described as Vibe Coding.

The most important change was not that the model wrote code faster than I did.

It was that ideas remained alive long enough to become prototypes.

Before AI-assisted development, a small idea often died during setup, documentation searches, dependency configuration, or framework research.

With AI, the cost of reaching the first working version dropped dramatically.

Rules Were an Early Form of Context Engineering

The early experience was also frustrating.

The agent forgot previous decisions.

It changed files outside the intended scope.

It solved local problems by creating architectural ones.

It occasionally behaved as if it had never seen the project before.

The community’s response was to rediscover software engineering discipline.

We began writing project rules:

Read the instructions before changing code.
Explain the implementation plan first.
Do not delete files without confirmation.
Follow the existing architecture.
Verify the result after making changes.
Ask questions when requirements are ambiguous.

At the time, this was often described as prompt engineering.

Looking back, it was an early form of context engineering.

We were not merely trying to write better sentences. We were trying to create a stable operating environment for a probabilistic system.

Limited context made this harder. Long conversations regularly lost important information, and opening a new chat was part of the workflow.

I even required the agent to refer to itself using a fixed name in every response. It was an imperfect but useful signal that the original instructions were still present.

The implementation details have changed, but the underlying problem remains:

An agent cannot reliably continue work unless the important context is explicit, accessible, and structured.

AI Exposed My Missing Domain Knowledge

While building my first applications, I noticed that my interfaces were consistently unattractive.

Initially, I treated this as a model limitation.

Later, I realised that the missing component was my own domain knowledge.

AI could generate SwiftUI code, but it could not give me design judgement that I did not possess.

It could implement a button, but I still needed to understand:

Why the button should be placed in a particular location.
How much spacing creates visual hierarchy.
What information a user needs before taking an action.
Why an interface feels confusing even when every feature works.

AI reduced implementation friction, but it did not remove the need for expertise.

In some ways, it made missing expertise more visible.

When implementation is slow, it is easy to blame the tools.

When implementation becomes fast, weak product decisions, unclear requirements, and poor design judgement become much harder to hide.

Stage 3: Moving Beyond the Chat Interface

As models and tools multiplied, I started using APIs and local inference rather than relying entirely on hosted chat products.

I bought a reasonably powerful computer and began downloading models from Hugging Face.

One model that surprised me was Kokoro TTS. Its relatively small footprint showed me how much useful capability could exist outside the largest hosted models.

I later integrated text-to-speech into an iOS application using the Sherpa ecosystem.

This introduced a new category of engineering problems.

Running a model in a desktop experiment is very different from shipping it inside a mobile application.

Suddenly, I had to care about:

Model size.
Memory usage.
Startup latency.
Packaging.
Device compatibility.
Offline execution.
Failure handling.

This experience reinforced another lesson:

A model demo is not a product.

The engineering work begins when the model has to operate inside real constraints.

Stage 4: Spec-Driven and Agent-Driven Development

After spending a long time in free-form Vibe Coding loops, Kiro’s Spec mode stood out to me.

It approached AI development through explicit requirements, design, tasks, and implementation steps.

This felt important because code generation was no longer the main bottleneck.

The real challenges had become:

Maintaining the goal across a long-running task.
Preserving constraints.
Producing work that could be reviewed.
Recording decisions.
Enabling another developer or agent to continue.
Recovering when the agent moved in the wrong direction.

Spec-driven development felt less magical than free-form prompting, but it felt more transferable to professional engineering.

The same transition occurred with project instruction files, skills, goal modes, and agent loops.

The industry moved from:

“Generate this function.”

toward:

“Understand this repository, follow these constraints, complete this goal, test the result, and report what changed.”

That is a fundamentally different type of interaction.

MCP Connected AI to Existing Systems

The Model Context Protocol generated plenty of debate when it appeared.

Some considered it a major architectural shift. Others saw it as an additional abstraction that might eventually be replaced.

In my own work, MCP became useful for a practical reason:

It offered a relatively lightweight way to connect agents to existing tools and data.

I built an MCP integration for my company’s knowledge platform.

The organisation already had nearly ten years of accumulated knowledge. The problem was not the absence of information. The problem was making that information available within an agent workflow.

Once the knowledge platform was connected, the agent could retrieve information in the context of an actual task.

This represented a major change in how I thought about AI.

Two years earlier, I had asked a chatbot when its training knowledge ended.

Now I was designing a system that gave an agent access to the organisation’s current knowledge.

The focus had shifted from model intelligence to system design.

Coding Is Easier, but Engineering Is Not

Modern coding agents can produce remarkable amounts of working code.

It is increasingly possible to define a goal, give an agent access to a repository, and let it iterate through implementation and testing.

That can make programming appear almost trivial.

My experience has been more complicated.

The amount of manual typing has decreased.

The need for engineering judgement has not.

The work has moved toward:

Defining goals precisely.
Breaking large tasks into safe boundaries.
Supplying relevant context.
Preventing unnecessary changes.
Reviewing architecture.
Validating assumptions.
Testing behaviour rather than trusting output.
Diagnosing failures that cross multiple generated changes.

A successful build is not necessarily a correct product.

A passing test suite is not proof that the requirement was understood.

Generated code still becomes code that someone must own.

AI has made implementation faster, but faster implementation increases the importance of deciding what should be implemented.

Multi-Agent Development Created a Memory Problem

I now regularly use more than one agent on a project.

Claude may begin a task. Kiro or Codex may continue it. Another tool may take over when the first one reaches a limit or struggles with a particular type of work.

This sounds like parallel engineering, but the handoff is often the weakest point.

The next agent needs to know:

What the original goal was.
What has already been implemented.
Which decisions were intentional.
Which approaches failed.
Which files are sensitive.
What remains incomplete.
What should not be changed.

Without that information, every handoff becomes a partial restart.

To solve this problem for my own workflow, I built QiJu, written 起居 in Chinese.

QiJu records project changes and provides a reusable memory layer for subsequent agent sessions.

Without it, I spend significant time reconstructing context.

With it, agent handoffs become closer to continuing a shared workflow rather than starting another isolated chat.

I no longer think of multi-agent development as “using several AI tools.”

The engineering challenge is creating continuity between them.

The Most Important Change: Ideas Survive Longer

A few days ago, I spent around thirty minutes building a floating stopwatch for macOS.

It was a small tool I had wanted for a long time.

The result was not technically significant, but the process represented something larger.

Previously, that idea would probably have remained on a list.

Building it would have required enough setup and research that the expected benefit did not justify the activation cost.

With AI assistance, the distance between the idea and a usable first version was short enough that the idea survived.

That is probably the biggest personal impact AI has had on me.

It has not made every idea valuable.

It has made experimentation cheap enough that more ideas get the opportunity to prove whether they are valuable.

AI Expanded My Capabilities and Narrowed My Attention

There has also been a personal cost.

Because AI and software occupied so much of my attention, I realised that I was entering an information bubble.

My daily reading became dominated by models, agents, context windows, IDEs, MCP servers, and product announcements.

To counter that, I began using AI to generate a daily story about something outside my normal interests.

History, art, society, individuals, and unfamiliar parts of the world became part of a collection I call “Stories Outside the Bubble.”

The irony is obvious.

AI expanded what I could build while narrowing what I noticed.

Then I used AI to widen my attention again.

What I Believe After Two Years

Over these two years, my relationship with AI moved through several stages:

AI as a chatbot.
AI as a coding assistant.
AI as an implementation agent.
AI as a component connected to tools and knowledge.
AI as part of a multi-agent engineering workflow.

The most valuable lesson is not that prompts are powerful or that coding agents are fast.

It is that useful AI systems depend on everything around the model:

Context.
Domain knowledge.
Memory.
Tools.
Constraints.
Evaluation.
Human judgement.
Continuity between tasks.

I remain an ordinary engineer.

But AI is now embedded in how I build applications, use company knowledge, record project decisions, and explore new subjects.

I do not know which models or development tools will still be dominant 24 months from now.

I do expect the distance between an idea and an implementation to keep shrinking.

The difficult part will remain deciding which ideas deserve to be implemented, designing systems that can be trusted, and taking responsibility for what the agents produce.

After writing this retrospective by the ocean, I should probably continue taking the morning off.

But I am already thinking about opening Claude Code again.

Monitoring SES Email Events though Configuration Set with SNS, SQS, Lambda and CloudWatch Log Groups

Jason Shen — Fri, 01 Sep 2023 11:28:03 +0000

When you are sending emails through Amazon SES service, you need to track sending events like Bounce and Complaint. You can use those events to adjust your mail list and eventually maintain reputation of your SES account, so you don't have to face unwanted status of your SES account like sending pause and review situation.

Amazon SES provides sending events through email feedback and event notification. When Configuration Set feature is available, now you can publish more events to multiple destinations. For example, you can publish Subscription event to CloudWatch metric.

In this article, I will show you how to set up a serverless workflow to monitor sending events of your SES account by using the following AWS services.

Amazon SES
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
Amazon CloudWatch log groups

1#. Follow SES document "Set up email sending with Amazon SES " and setup your SES account. If your SES account is in sandbox, you will need to verify both sender and recipient email addresses.

2#. Follow SES document "Creating configuration sets in SES" and create a Configuration Set.

3#. Follow SQS document "Create a queue (console)" and create SQS queue with standard queue type.

4#. Follow SNS document "Getting started with Amazon SNS" and create SNS topic.

5#. Create a SNS subscription with Protocol type "SQS", put SQS ARN of SQS queue created in step 3.

6#. Update Access Policy of SQS queue to allow SNS topic to publish events.

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<AWS_Account_ID>:root"
      },
      "Action": "SQS:*",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    },
    {
      "Sid": "__sender_statement",
      "Effect": "Allow",
      "Principal": {
        "Service": "sns.amazonaws.com"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:us-west-2:<AWS_Account_ID>:<SQS_Queue_Name>"
    }
  ]
}

7#. Create a Lambda function using Python version > 3.9

The function will be triggered by SQS queue and store events into CloudWatch log groups based on SES message ID. The function will create CloudWatch log group if not existing.

import json
import boto3
from datetime import datetime
from time import time
import random

def lambda_handler(event, context):
    #print(event)
    for record in event['Records']:
        print("record")
        payload = record["body"]
        print(payload)
        # get lambda running region
        region = context.invoked_function_arn.split(":")[3]
        # get lambda running version
        version = context.function_version
        # get lambda running name
        name = "ses-notification-" + region
        mail = json.loads(payload)['mail']
        #print("mail")
        #print(mail)
        messageID = mail['messageId']
        print(messageID)
        logGroupNamePrefix="/aws/lambda/" + name
        # define CloudWatch log group client in the region
        client = boto3.client('logs', region_name=region)
        # check cloudwatch log group with name "ses-notification-cw-logs" existence
        response = client.describe_log_groups(logGroupNamePrefix=logGroupNamePrefix)
        # if log group not found, create it
        if len(response["logGroups"]) == 0:
            client.create_log_group(logGroupName=logGroupNamePrefix)
            print("Log group created")
        # get current date in form as year/month/day
        currentDay = str(datetime.now().day)
        currentMonth = str(datetime.now().month)
        currentYear = str(datetime.now().year)
        date = currentYear + "/" + currentMonth + "/" + currentDay
        logStreamsName = date + "/[" + version + "]" + messageID
        # create CloudWatch log stream
        responseLogStream=client.describe_log_streams(logGroupName=logGroupNamePrefix,logStreamNamePrefix=logStreamsName)
        if len(responseLogStream["logStreams"]) == 0:
            client.create_log_stream(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName)
            print("Log stream created")
        # put payload into CloudWatch log stream with current timestamp
        #payload = "this is test payload"
        client.put_log_events(logGroupName=logGroupNamePrefix, logStreamName=logStreamsName, logEvents=[{"timestamp": int(time() * 1000), "message": payload}])
        print("Payload put into log stream")

8#. The IAM role of the Lambda function must have the following permission in IAM managed policy.

CloudWatchLogsFullAccess
AWSLambdaSQSQueueExecutionRole

9#. In the SQS queue, you need to follow "Configuring a queue to trigger an AWS Lambda function (console)" and add the Lambda function as trigger.

10#. Create Event Destination in Configuration Set created in step 2 by following "Set up an Amazon SNS event destination for event publishing.

11#. Now you can send an email by using SMTP, AWS CLI or API through your SES account. Remember that you need to specify the Configuration Set in your sending action.

Amazon S3 - Web Based Upload Object with POST request and Presigned URL in Python Boto3

Jason Shen — Tue, 22 Aug 2023 13:21:41 +0000

In this article, I will show you how to generate S3 Presigned URL for HTTP POST request with AWS SDK for Boto3(Python). The unique part of this article is that I will show you how to apply Server Side Encryption with KMS key, Tagging objects, Updating Object Metadata and more with S3 Presigned URL for HTTP POST.

When using S3, there is a scenario about "Broswer-Based Uploads Using HTTP POST". However, it is required to calculate AWS SigV4 Signature to follow the section.

Instead of calculating the signature by you own codes, you can actually use AWS Boto3 SDK with method "generate_presigned_post" to generate S3 PreSigned URL. This is not only saving your time to debug "Signature Mismatch" error with your own codes, you don't have to figure out requirements of crypto modules used by your codes to generate right signature. It will all be handled by AWS SDK.

For example, you owns an S3 bucket in your account. One customer of yours is running a business to allow users of the customer to upload images. The images will be directly uploaded from the customer's website into your S3 bucket. The customer is not familiar with Amazon S3 service and does not own an AWS account, so you need to provide your customer an easy method uploading objects from the customer website directly into your S3 bucket. At the meantime, you don't need to make your bucket public for uploading objects.

This is where S3 Presigned URL is needed. You can generate the S3 Presigned URL for HTTP POST from AWS Lambda function by having these benefits. Then you can provide the S3 Presigned URL with your customer to integrate into the customer's website.

But you might ask this question:

Why are you not using S3 Presigned URL for PutObject API call?

S3 Presigned URL for HTTP POST from broswer-based uploads provides a unique feature. You can define "starts-with" condition in the policy. You and your customers can both have some controls on requirements of the uploaded objects.

For example, you only want your customer to upload text files, so you can use the following "start-with" condition to restrict value of "Content-Type" starting with "plain" in uploading request. The uploading request is created from your customer's website. The value of "Content-Type" request header is set when a file is being uploaded from your customer's website by using your S3 Presigned URL.

["starts-with", "$Content-Type", "plain"],

In document of AWS SDK for Boto3, it did not share much information regarding how to use "Fields" and "Conditions" parameters mentioned at "generate_presigned_post". It took me some time to figure it out, so I added my understanding in the code example.

I hope they will save your time in your code development.

Here is the Python Code Example. Before you test it, you will need to update the constants to match your resources.

import boto3
import requests
from botocore.config import Config

ACCESS_KEY="AKIAIOSFODNN7EXAMPLE"
SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

BUCKET_NAME="example-bucket-name"
OBJECT_NAME="example-key-name"
REGION_LOCATION="ap-southeast-2"

KMS_KEY_ARN="arn:aws:kms:<region>:<account-id>:key/<key-id>"

EXPIRATION_TIME = 60*60 * 12 # 12 hours

TEST_FILE_NAME="/Absolute/Path/To/Local/FileName"

my_config = Config(
    region_name = REGION_LOCATION,
    signature_version = 'v4',
    retries = {
        'max_attempts': 10,
        'mode': 'standard'
    }
)

# define S3 client in ap-southeast-2 region
s3=boto3.client('s3',
                 aws_access_key_id=ACCESS_KEY,
                 aws_secret_access_key=SECRET_ACCESS_KEY,
                 config=my_config)


fields={
    "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>",
    "x-amz-storage-class": "STANDARD_IA",
    "Cache-Control": "max-age=86400",
    "success_action_status": "200",
    "x-amz-server-side-encryption": "aws:kms",
    "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN,
    "x-amz-server-side-encryption-bucket-key-enabled": "True"
    # "acl": "public-read"
    }


conditions=[
    {
        "x-amz-storage-class": "STANDARD_IA"
    },
    ["starts-with", "$Content-Type", "plain"],
    {
        "tagging": "<Tagging><TagSet><Tag><Key>type</Key><Value>test</Value></Tag></TagSet></Tagging>"
    },
    {
        "Cache-Control": "max-age=86400"
    },
    {
        "success_action_status": "200"
    },
    {
        "x-amz-server-side-encryption": "aws:kms"
    },
    {
        "x-amz-server-side-encryption-aws-kms-key-id": KMS_KEY_ARN
    },
    # {
    #     "acl": "public-read"
    # },
    {
        "x-amz-server-side-encryption-bucket-key-enabled": "True"
    }
]

# generate S3 Presigned URL for HTTP POST Request
response_presigned_url_post=s3.generate_presigned_post(
    BUCKET_NAME,
    OBJECT_NAME,
    Fields=fields,
    Conditions=conditions,
    ExpiresIn=EXPIRATION_TIME
)
print(response_presigned_url_post)

# User requests.post to test the URL
post_fields=response_presigned_url_post['fields']

# files={'file': open(TEST_FILE_NAME, 'rb')}
# you will see 403 error
#comment the following line and uncomment the second following line, you will see 200 successful

post_fields["Content-Type"]="application/octet-stream"
#post_fields["Content-Type"]="plain/text"

# file key must be the last key in the "files" parameter(form)
# it is defined at https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html#RESTObjectPOST-requests-form-fields

post_fields["file"]=open(TEST_FILE_NAME, 'rb')
print(post_fields)

# making POST Request
response_post_request=requests.post(response_presigned_url_post['url'], files=post_fields)

# print response, by default status code is 204,
# "success_action_status": "200" change it to 200

print(f'Response Status of POST request with S3 Presigned URL: {response_post_request.status_code}')
print(f'Response Headers of POST request with S3 Presigned URL: {response_post_request.headers}')
print(f'Response Body of POST request with S3 Presigned URL: {response_post_request.text}')

CloudFront Edge Computing - Dynamic At Edge with CloudFront Functions

Jason Shen — Mon, 14 Aug 2023 06:53:53 +0000

Edge Computing feature is definitely one of the things that I like the most with CloudFront service. When CloudFront Functions is launched, it makes the feature even better.

Recently I saw someone asked a question about the following scenario in re:Post. I think it is a perfect scenario for CloudFront Functions.

Files in the S3 origin:

JSON files are under content folder(prefix), e.g. s3://example_bucket/content/a.json
Image files are under a sub-folder of content folder (prefix) e.g. s3://example_bucket/content/image/a.png

The files are requested by viewer request like:

JSON file: https://text.example.com/abc/a.json
PNG file: https://image.example.com/a.png

What is going to happen when there was no edge computing in CloudFront service?

For the JSON file, there might be no other options but move the objects under 'content/abc/' folder. Then origin path will be 'content/' in origin setting of the distribution.

For the PNG file, it will require a separate origin setting in the distribution configuration other than the JSON file because they are not going to share the same origin path. The PNG file will need origin path as 'content/image'.

The JSON file and PNG files will also need different behaviors because they need to call different origin settings.

It looks like lots of work.

With CloudFront Functions, a few lines of codes will resolve the problem.

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var file_name = uri.split('/').pop();

    // Check whether the URI is missing a file name.
    if ( file_name.endsWith('.json')) {
        request.uri = "/content/" + file_name;
        return request;
    } else if (file_name.endsWith('.png')) {
        request.uri = '/content/image/' + file_name;
        return request;
    }

    return request;
}

In the behavior, the function will be added as Viewer Request trigger in Default behavior with an S3 origin pointing to "s3://example_bucket".

That is it!

And if any further requirement is required, e.g. changing path based on the requesting domain name, only a few modifications in codes will meet the requirement.

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var file_name = uri.split('/').pop();
    var host = request.headers.host.value;

    // Check whether the URI is missing a file name.
    if ( host === "text.example.com" ) {
        request.uri = "/content/" + file_name;
        return request;
    } else if ( host === "image.example.com" )) {
        request.uri = '/content/image/' + file_name;
        return request;
    }

    return request;
}

CloudFront Functions are perfect to process those 'small' modifications in viewer requests and responses.

The code I am using is modified from this code example in CloudFront public document.

I am seeing a trend that generative AI tools like ChatGPT or Amazon CodeWhisperer will help to writ those scripts much easier than ever!

Like Lambda@Edge, there are some restrictions on runtime of CloudFront Functions that you should have a read before wring your codes.

The Differences In Sending Email Actions Between SES Version 1 and Version 2 APIs

Jason Shen — Mon, 07 Aug 2023 03:05:28 +0000

If you have been using Amazon SES service for a while, it might not be new to you that Amazon SES is having two versions of API actions available at the moment, API Reference (version 1) and API v2 Reference. I tried to find the announcement of the API v2 in What's New with AWS? page but I did not have the luck.

Amazon SES service API calls can be separated into two types:

Management API actions, which can be recorded in CloudTrail Event History
Sending API actions, which are not recorded in CloudTrail Event History

In this article, you will read about how I took a peek at the sending API actions in Amazon SES API v1 and v2.

What are these API actions?

I can find the API action names in SES API documents. But I find that I can also get the list of API calls by creating an example IAM policy from IAM console.

Here is the list of actions I got when I was creating an IAM policy with service name 'SES' and 'SES v2' from IAM console.

v1 IAM actions

"Action": [
"ses:SendBounce",
"ses:SendBulkTemplatedEmail",
"ses:SendCustomVerificationEmail",
"ses:SendEmail",
"ses:SendRawEmail",
"ses:SendTemplatedEmail"
]
v2 IAM actions

"Action": [
"ses:SendBulkEmail",
"ses:SendCustomVerificationEmail",
"ses:SendEmail"
]

What are the differences?

I used SES SDK Boto3 Document in the following comparison.

To use Amazon SES v2 APIs, the service client must be boto3.client('sesv2')

SES v2 API "SendEmail" provides the same functions in v1 APIs "SendEmail", "SendRawEmail" and "SendTemplatedEmail". But only the API "SendEmail" in SES v2 supports the feature "list management". List Management can help to reduce the chance of get complaints or even hard bounces by sending emails to recipients who have opted out of their previous subscriptions to some email services, e.g. newsletter email.
SES v2 API "SendBulkEmail" has different names of parameters in sending requests with SES v1 "SendBulkTemplatedEmail". But values of those parameters should be no difference.
SES v2 API action "SendCustomVerificationEmail" has no difference with the action in SES v1 API.
SES v2 API actions do not have a definition of action "SendBounce". The action "SendBounce" is used to generate and send a bounce message to the sender of an email which I received through Amazon SES. There is a restriction that we can only use this API action on an email up to 24 hours after receiving it.

How do I restrict usage in SES API v1 and v2?

SES public document has given an example about "Allowing Access to only SES API version 2". Relatively, I could modify the condition forcing to use SES API v1.

Why would I want to do that? One thing in the same document could provide an answer:

The SES SMTP interface uses SES API version 2 of ses:SendRawEmail.

As an IAM user can convert AWS credentials into SMTP username and password, I can use the trick to restrict the IAM user to use or not use SMTP client to send emails through Amazon SES.

One more difference is the message size between using SES API v1 and v2. While size of message (after base64 encoding and including attachments) is 10 MB using SES API v1, the size is 40MB in SES API v2.

SES BYODKIM - Streamline Private and Public Key by Python Script

Jason Shen — Tue, 01 Aug 2023 05:26:28 +0000

In Amazon SES document, it states the following:

You have to delete the first and last lines (-----BEGIN PRIVATE KEY----- and -----END PRIVATE KEY-----, respectively) of the generated private(public) key. Additionally, you have to remove the line breaks in the generated private key. The resulting value is a string of characters with no spaces or line breaks.

source

If the private or public is not streamlined, you won't be able to use them with SES BYODKIM.

At the meantime, you also need to generate a random string as selector in TXT DNS record publishing the public key

Name	Type	Value
selector._domainkey.example.com	TXT	p=yourPublicKey

Replace selector with a unique name that identifies the key.
Here is the Python Script to complete

source

Here is python script can do the both.

import sys, string, random

def streamline_key(keyLocation):
   keyLines=open(keyLocation).readlines()
   keyStream=[]
   for line in keyLines[1:-1]:
      keyStream.append(line.replace('\n', ''))
   key=''.join(keyStream)
   return key

print(sys.argv)

key1Location=sys.argv[-3]
print(key1Location)

key2Location=sys.argv[-2]
print(key2Location)

domain=sys.argv[-1]
print(domain)

key1Streamline=streamline_key(key1Location)
print(key1Location+" streamline:\n" + key1Streamline )
print("\n")
key2Streamline=streamline_key(key2Location)
print(key2Location+" streamline:\n" + key2Streamline)
print("\n")
selector = ''.join(random.choice(string.ascii_lowercase + string.digits) for i in range(32))
print("public key TXT record name:\n" + selector+'._domainkey.'+domain)

Save the code with name Run the code 'remove-newline-in-key.py' in the same folder storing private key and public key. Then run the script as following format in command line. My public key name is public.key.

% python remove-newline-in-key.py private.key public.key example.com

You will get the following result

['remove-newline-in-key.py', 'private.key', 'public.key', 'example.com']
private.key
public.key
example.com
private.key streamline:
MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBAMdPWfzVMYahtkvTvBfXsVr52O2CHUok8JzeoCt1Ou3t1SmDmV3755+ztxGj7nFwUCVFmrT5ZmaaDZ5u7Jd856KmejtlIeuPHBt9wuoaiwI1IohXWZAMGLi+qo+FX1kHk+nKj5nMLNq9dSOE8xXXfmtPcz+B4LACpuQRXNGhqCLlAgMBAAECgYEAkXTq8qdQtrXMSfij3C6xI/kVhPihkZv18jZTZIPw1vXszJhbVIjkWNwarggam7Vg+GKc7pjZT+X8LHU9u60Pio22vi6ZNBQwqe0DlpMx1MtJIht4EwH63CZDSU6jijZUjvdTyKqtoqMHiqUaLz2Iom8LYikmrKImMr6S9PqgBsECQQDsJ+8N4asakc0uUKZkxgQNpoM7fykuFmF9TJcq3K9JHfx8HpvMN9UWNyGDfQqIo/4oFD3LxeheeyltETCNqE91AkEA2A7OE2r+D9uMpAnNyt3SmIRQZzVn+ZHB+0fICFB8L17rt6TnuH2AU6ceuoVzr8vtSWGZ+/sotUGvaIbZvwkHsQJAfoTafv5i8+YfHewZaS3pKAMIlcyHnGhjLITnDBCVXD/TcA/Z+iwDXlaE/vPzu8bYOFK31L8fwdaMGCG4eHwurQJAO2CeO/Hsjrkcxrw3BWi/BtFeM28W+xyWvhM1IyvTZUVl7JtyX16GVPcZ19LzPz4BIWikY/7baiz6IvTkhL7bkQJAZhwo33EVKRcDoavSOWshcWEsp6SNychsdT9R17uEzsZq1RgrB9XNqskTveJvzLeT/aRtuoqB+mpJ1R3ux3VLYw==

public.key streamline:
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHT1n81TGGobZL07wX17Fa+djtgh1KJPCc3qArdTrt7dUpg5ld++efs7cRo+5xcFAlRZq0+WZmmg2ebuyXfOeipno7ZSHrjxwbfcLqGosCNSKIV1mQDBi4vqqPhV9ZB5Ppyo+ZzCzavXUjhPMV135rT3M/geCwAqbkEVzRoagi5QIDAQAB

public key TXT record name:
w2hajm6q1zoe0gw1q993shhmqopj5auy._domainkey.example.com