DEV Community: Kausar Raza

How We Fixed Unused JavaScript and LCP Render Delay in Next.js

Kausar Raza — Mon, 02 Nov 2026 18:25:35 +0000

We build fast sites for clients, so a mediocre PageSpeed score on our own home page is not a good look. This is the report we started from on 19 October 2026, each issue in it, and exactly what we changed - including two fixes that sounded right and made things worse. The site is Next.js 16 behind Cloudflare; most of this applies to any React site, and the last sections cover what changes for an online shop and where security headers fit in.

The starting point

Lighthouse 12.8, mobile profile, against the live home page:

Metric	Result
Performance score	72
First Contentful Paint	1.9 s
Largest Contentful Paint	4.5 s
Total Blocking Time	370 ms
LCP element	the header logo
Unused JavaScript	an 89 KiB chunk, gtag.js 65 KiB, AdSense 35 KiB
Render-blocking CSS	452 ms
DOM size	2,239 elements

The LCP breakdown was the giveaway: TTFB 653 ms, resource load delay 822 ms, load time 66 ms, and render delay 2,918 ms. The logo downloaded in a fraction of a second and then waited almost three seconds to be painted. That is not a network problem, and no CDN setting fixes it.

Unused JavaScript: an 89 KiB chunk nobody asked for

The biggest unused file had a hashed name, so the first job was finding out what was in it. Next.js 16 has a bundle analyser built in:

npx next experimental-analyze -o

It writes per-route data to .next/diagnostics/analyze. The chunk turned out to contain the docx library and PDF helpers - code for our PDF-to-Word and Word-to-PDF tools - and the home page was not even loading it directly. Next.js prefetches the code for routes linked from the page, and the header links to several tools. The tool page imported a registry of all 54 tools, so prefetching any one tool pulled in the code for all of them. Every tool page was shipping about 1 MB of JavaScript for the same reason.

The fix was to load each tool's component on its own with next/dynamic, from a small client module:

"use client";
import dynamic from "next/dynamic";

const tools = {
  "word-counter": dynamic(() => import("./impl/text-tools").then((m) => m.WordCounter)),
  "pdf-to-word":  dynamic(() => import("./impl/pdf-tools").then((m) => m.PdfToWord)),
  // ...one line per tool
};

export function ToolComponent({ slug }) {
  const Tool = tools[slug];
  return Tool ? <Tool /> : null;
}

dynamic() still renders the component on the server, so the HTML and its SEO value are unchanged; the browser just stops downloading the other 53. One trap: the server page cannot call a function exported from a "use client" file - the build fails with "Attempted to call hasTool() from the server but hasTool is on the client" - so the list of tool slugs lives in a plain shared module. Result: a tool page went from 1,059 KiB to 767 KiB of JavaScript (321 to 237 KiB gzipped), and the unused chunk disappeared from the home page report.

If you see "reduce unused JavaScript" on a Next.js site, look for exactly this pattern: one file that imports everything, used by a route that only needs one piece of it. Icon libraries are the other usual suspect; experimental.optimizePackageImports: ["lucide-react"] makes sure only the icons you use are bundled.

LCP render delay: the fade-in that hid the headline

Why was the logo the LCP element, not the big headline? Because the hero animated in with a fade starting from opacity: 0. Chrome does not count an invisible element as a Largest Contentful Paint candidate, so while the headline was fading in, the largest visible thing was the logo - and the measurement only settled once the animation had finished. A 620 ms fade with staggered delays was adding seconds of "render delay" on a throttled phone.

We kept the entrance and dropped the fade: the hero now slides up 10 px without changing opacity.

@keyframes rise-up {
  from { transform: translateY(10px); }
  to   { transform: none; }
}
.rise { animation: rise-up 620ms cubic-bezier(0.22, 1, 0.36, 1) both; }

The headline is now a candidate from its first paint. If your LCP element is something small like a logo while a large headline sits right next to it, check for an entrance animation first.

We also sized the logo to what is actually drawn. It was declared as 260×64 but displayed about 44 px tall, so the image optimiser was choosing a 640 px file for phones. Declared at 180×44 with fetchPriority="high", it downloads the right file first. (On this Next.js version priority alone was not adding the fetchpriority attribute - check the rendered HTML rather than trusting the prop.)

The render delay that remains

Not all of it went away, and it is worth explaining why. In a trace of the page loading in headless Chrome, the main thread was idle for about a second before the first paint. That second was spent on GPU rasterisation - drawing the hero's illustration, gradients and shadows. Removing the illustration as a test brought first paint from about 1.65 s to 1.15-1.45 s. PageSpeed Insights also runs in a headless browser with software rendering, so it sees this cost; a real phone with a GPU sees much less of it. Rebuilding the hero to be cheaper to draw is a design change, not a configuration change, and we have not done it yet.

Third-party scripts: wait for the visitor

gtag.js and AdSense were already loading with strategy="lazyOnload", which runs them after the page's load event. That sounds late, but it is still inside the window Lighthouse uses to measure Total Blocking Time, and Google Analytics' start-up ran as 300+ ms of long tasks there.

Both now load on the visitor's first interaction - a scroll, tap, key press or mouse movement. Analytics also has a fallback: four seconds after the page has loaded, once the browser is idle, so a visitor who reads without touching anything is still counted.

const EVENTS = ["pointerdown", "keydown", "touchstart", "scroll", "mousemove"];

function afterInteraction(start, idleAfterLoadMs) {
  let done = false;
  const run = () => {
    if (done) return;
    done = true;
    EVENTS.forEach((e) => removeEventListener(e, run));
    start();
  };
  EVENTS.forEach((e) => addEventListener(e, run, { once: true, passive: true }));
  if (idleAfterLoadMs !== undefined)
    addEventListener("load", () =>
      setTimeout(() => requestIdleCallback(run, { timeout: 2000 }), idleAfterLoadMs));
}

afterInteraction(() => {
  window.dataLayer = window.dataLayer || [];
  window.gtag = function () { dataLayer.push(arguments); };
  gtag("js", new Date());
  gtag("config", "G-XXXXXXXXXX");
  const s = document.createElement("script");
  s.async = true;
  s.src = "https://www.googletagmanager.com/gtag/js?id=G-XXXXXXXXXX";
  document.head.appendChild(s);
}, 4000);

Be honest about the trade-offs. A visitor who leaves within about four seconds without interacting is not counted. Ads only load once someone interacts, which costs nothing in practice because our ad slots are below the fold. And we send Core Web Vitals to GA, which report TTFB and FCP before GA has loaded - those are now queued and sent once GA is configured, instead of being silently dropped.

Two fixes that made things worse

Inlining the CSS. Lighthouse flagged the stylesheet as render-blocking (452 ms), and Next.js has experimental.inlineCss to put it in the HTML. On this site the stylesheet is 277 KB uncompressed, because one Tailwind file covers the blog, 54 tools and the admin area - and inlining it put the CSS into the page twice, once as a `` tag and again in the React Server Components payload. The home page HTML went from 520 KB to 1.38 MB and First Contentful Paint got worse. Reverted. Inlining only makes sense when the CSS is small.

A modern browserslist. Lighthouse reported 13 KiB of legacy JavaScript (polyfills for features every current browser has). Setting browserslist to Chrome 111, Edge 111, Firefox 111 and Safari 16.4 is the textbook fix. The estimate did not move: those polyfills ship inside the framework's own pre-built code, which your config does not recompile. It does no harm, but it did not fix the warning either.

Smaller fixes

Image sizes. Portfolio cards are about 400 px wide in a three-column grid but declared 33vw, so they were served from a 640 px file. They now declare (max-width: 768px) 100vw, (max-width: 1024px) 50vw, 420px, and we added 480 to the optimiser's imageSizes so there is a file between 384 and 640.
DOM size. 2,239 elements is mostly the home page being long. We wrapped every below-the-fold section in content-visibility: auto with a contain-intrinsic-size, so the browser skips laying them out until they approach the viewport. That does not change the element count Lighthouse reports - it changes how much of it the browser has to process up front.
Cloudflare. Email Address Obfuscation injects its own script; we are switching it off. The full list of Cloudflare settings we use is in Cloudflare speed settings and our Brotli vs zstd vs gzip benchmark.

The result

To compare like with like, both versions were built and tested on the same machine with the same Lighthouse mobile settings:

Metric	Before (2 runs)	After (3 runs)
Performance score	60-62	66-73
Total Blocking Time	350-400 ms	210-340 ms
Largest Contentful Paint	7.4-7.9 s	5.6-5.7 s
JavaScript on a tool page	1,059 KiB	767 KiB

Local numbers are worse than the live site's in absolute terms because the local server fetches its data from a remote API; it is the difference between the columns that counts. With so few runs, treat the ranges as ranges. Lab scores also move from run to run - the field data in Search Console's Core Web Vitals report, from real visitors over 28 days, is what Google ranks on. Our Core Web Vitals guide explains how to read the two side by side.

What changes for an e-commerce site

Everything above applies to a shop, with higher stakes: a slow product page loses the sale, not just the reader. The places shops usually lose time are predictable:

The main product image is the LCP element. Never lazy-load it, give it fetchpriority="high", and serve it in AVIF or WebP at the size it is displayed. Lazy-load the gallery thumbnails below it instead.
Third-party widgets pile up. Review widgets, live chat, marketing pixels, heatmaps and recommendation engines each bring their own JavaScript. Load anything not needed for the first view on interaction, as above - a chat widget nobody has opened does not need to block the page.
Theme and plugin JavaScript. On Shopify and WooCommerce, "reduce unused JavaScript" usually points at apps and plugins loading their scripts on every page, including ones that do not use them. Remove what you no longer use; it is the most effective fix.
TTFB on category and search pages. Faceted filters and sorting often make these the slowest pages on the site and the hardest to cache. Test them, not just the home page - our website speed test can run across every page in your sitemap and list the slowest.
Caching. Cache product and category HTML at the edge, but never cart, checkout or account pages. The Cloudflare rules for that are in the post linked above.
Do not defer payment scripts. Whatever you delay, leave the checkout's payment provider alone.

Security headers: free, and no speed cost

While auditing performance it is worth checking security headers, because they cost nothing in speed: they are a few hundred bytes on each response. Our free security headers checker fetches a page and grades each header with what to add. This site sends:

`text Strict-Transport-Security: max-age=31536000 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN Referrer-Policy: strict-origin-when-cross-origin Permissions-Policy: camera=(), microphone=(), geolocation=(), usb=(), payment=(), browsing-topics=() Content-Security-Policy: frame-ancestors 'self'; base-uri 'self'; object-src 'none'; form-action 'self'; upgrade-insecure-requests `

Two deliberate gaps. HSTS has no includeSubDomains until every subdomain is confirmed to work over HTTPS, because a browser that has seen it will refuse plain HTTP on all of them for a year. And the Content-Security-Policy has no script-src: a strict script policy in Next.js needs a fresh nonce on every request, which makes every page dynamic and throws away the edge caching our TTFB depends on. That is a real trade-off between security and speed, and for a site with no logins or payments we chose speed. A shop taking payments should weigh it the other way, and a payment page in particular deserves a strict policy.

FAQ

How do I reduce unused JavaScript in Next.js?

Find what is in the unused chunk first - npx next experimental-analyze writes a per-route breakdown. The usual cause is one module importing many components that a route only uses one of; load those with next/dynamic. Also use optimizePackageImports for icon libraries, and load analytics and ads after the first interaction.

What causes LCP render delay?

The LCP resource has arrived but is not painted yet. Common causes are an entrance animation starting at opacity 0 (Chrome does not count invisible elements), render-blocking CSS or JavaScript, long main-thread tasks, and an expensive first frame to rasterise.

Why is my logo the LCP element?

Often because the larger element - usually the headline - is invisible when the page first paints, for example fading in from opacity 0. Make the entrance animation move without fading and the headline becomes the LCP candidate from its first paint.

Does loading Google Analytics on interaction lose data?

A little. Visitors who leave within a few seconds without scrolling, tapping or moving the mouse are not counted unless you add an idle-time fallback. We load GA on first interaction or four seconds after load, whichever comes first, and queue Web Vitals so they are not dropped.

Should I use experimental.inlineCss in Next.js?

Only if your CSS is small. With a 277 KB stylesheet it put the CSS into the HTML twice (a style tag and the RSC payload), the home page HTML grew from 520 KB to 1.38 MB, and First Contentful Paint got worse.

How do I speed up an e-commerce website?

Load the main product image first in AVIF or WebP at its displayed size, remove unused apps and plugins, delay chat, review and marketing widgets until interaction, test category and search pages for slow TTFB, and cache product pages at the edge while never caching cart, checkout or account pages.

Do security headers slow down a website?

No - they are a few hundred bytes per response. The only speed trade-off is a strict Content-Security-Policy with per-request nonces, which stops pages being statically cached. The other headers are free.

Originally published on knowledgemarkg.com.

Crawled vs Discovered - Currently Not Indexed: What 63 Unindexed Pages Taught Me

Kausar Raza — Mon, 02 Nov 2026 18:07:36 +0000

Search Console's Pages report puts "Crawled - currently not indexed" and "Discovered - currently not indexed" next to each other, and they sound almost the same. They aren't. One means Google has never fetched the page. The other means it fetched the page and decided not to keep it. The fix for one does little for the other.

On 1 November I checked every URL in this site's sitemap through the URL Inspection API, then asked Google to index 12 of the pages that were missing. Here is what the numbers looked like, what each status meant for my pages, and what changed one day later.

The short version: "Discovered" means Google knows the URL but hasn't crawled it yet. Fix it with internal links from pages that are already indexed, and give it time. "Crawled" means Google read the page and passed on it. Fix it by making the page worth indexing: improve it, merge it into a stronger page, or remove it. Request indexing helped 6 of my 12 pages within a day, but none of the 5 service pages I sent.

What the two statuses mean

Discovered - currently not indexed. Google found the URL in a sitemap or a link, but hasn't fetched it. Google's help page says it usually happens because crawling the site would have overloaded it, so the crawl was postponed. On a small site, it more often means Google doesn't think the URL is important enough to crawl yet. There's no last crawl date, because there was no crawl.

Crawled - currently not indexed. Google fetched the page, read it, and left it out of the index. There's a last crawl date. Google says the page may be indexed later and that you don't need to resubmit it. In practice it's a quality judgement. The page didn't add enough that Google's index doesn't already have.

URL is unknown to Google. This one appears in URL Inspection, not the Pages report. Google has no record of the URL at all, not even from the sitemap.

My numbers: 63 of 219 URLs not indexed

I ran the URL Inspection API over all 219 URLs in the sitemap on 1 November 2026. 156 were indexed. The other 63 broke down like this:

Section	URLs	Indexed	Discovered	Crawled	Unknown
Blog posts	53	41	2	9	1
Free tools	83	66	13	0	4
Tag pages	30	8	14	1	6
Service pages	12	6	4	0	2
Everything else	41	35	3	1	2

One tag page was excluded by its own noindex tag, which is deliberate. The two problems look very different once you split them by section.

What "Crawled - currently not indexed" looked like

Nine blog posts were crawled and rejected. Seven of them have something in common: they're broad, generic guides from 2025. A Git tutorial, a Python tutorial for beginners, a React hooks guide, a full-stack roadmap, a list of backend project ideas, an AI tools roundup, and a piece on AI and junior developers. Six of those seven were crawled on the same day, 19 August. Thousands of sites cover each of those topics already.

Meanwhile the posts that are indexed are mostly narrow and first-hand: a specific error message, the exact cause, what I tested. My guess, and it's only a guess from this pattern, is that Google read the generic guides and found nothing it didn't already have.

That's why resubmitting a crawled page rarely helps on its own. Google already has the page and already decided. What changes the outcome is changing the page. The day before this check I had unpublished 38 off-topic posts from the site's earlier life (computer tips, YouTube guides), because none had a single click in three months.

What "Discovered - currently not indexed" looked like

The Discovered pages were a different crowd: tag pages, the image converter pages (/tools/image-converter/jpg-to-png and so on), tool category pages and service pages. Many had few or no links pointing at them.

The converter pages are the clearest case. An Ahrefs crawl the same day listed them as orphan pages, meaning nothing on the site linked to them. They were in the sitemap and nowhere else. I added a block of links to them from the main image converter page and deployed it that evening.

I requested indexing for 12 pages. Here is what happened

On 1 November I used Request indexing in Search Console for 12 of the missing URLs, picking the pages that matter most: 5 service pages, 4 blog posts, the portfolio and two tool category pages. I sent about 15 requests in total (a few were repeats) and never saw a "Quota exceeded" message. One day later I inspected the same 12 through the API:

Page	Before	One day later
Blog: PHP2Shell explained	Discovered	Indexed
Blog: ChunkLoadError after deploy	Discovered	Indexed
Blog: AI reverse engineering	Crawled	Indexed
Blog: Core Web Vitals guide	Crawled	Indexed
/portfolio	Crawled	Indexed
/tools/image-tools	Discovered	Indexed
/tools/pdf-tools	Discovered	Discovered
2 service pages	Discovered	Discovered
1 service page	Discovered	Unknown
2 service pages	Unknown	Unknown

Six of twelve indexed in a day. Two of those had been "Crawled - currently not indexed", so a request can move a crawled page too, at least when the page is decent. But none of the five service pages was indexed. The two URLs Google had never heard of were still unknown a day after a direct request, and one page went from Discovered to unknown, which I can't explain.

Twelve pages over one day is a small sample, so read this as one site's result, not a rule. What it does show is that Request indexing gets a URL into a queue. It doesn't make Google want the page.

How to check all your URLs at once

The Pages report shows examples, not every URL, and it lags. The URL Inspection API gives the same answer as the URL Inspection tool for one URL per call, and you can script it. The quota is 2,000 inspections a day per property, which covers a small site in one run. It can't request indexing. There is no API for that outside job postings and livestreams.

Create a service account in Google Cloud, enable the Search Console API, and add the service account's email as a user on the property in Search Console. Then:

from google.oauth2 import service_account
from googleapiclient.discovery import build

creds = service_account.Credentials.from_service_account_file(
    "service-account.json",
    scopes=["https://www.googleapis.com/auth/webmasters.readonly"],
)
gsc = build("searchconsole", "v1", credentials=creds)

def coverage(url):
    result = gsc.urlInspection().index().inspect(body={
        "inspectionUrl": url,
        "siteUrl": "https://example.com/",   # or "sc-domain:example.com"
    }).execute()
    status = result["inspectionResult"]["indexStatusResult"]
    return status.get("coverageState"), status.get("lastCrawlTime")

print(coverage("https://example.com/blog/some-post"))

Loop that over the URLs in your sitemap and count the coverageState values. I called the same endpoint with plain HTTP rather than this library, but the request and response are the same. Keep the service account key out of your repository.

What to do about each status

Discovered - currently not indexed

Link to the page from pages that are already indexed. A URL that only exists in a sitemap is a weak signal. Check for orphan pages with a crawler first.
Keep the sitemap clean. Every URL in it should return 200, be canonical and be indexable. My sitemap still listed a noindex tag page, which is a mixed signal.
Request indexing for the few pages that matter. Not for hundreds of URLs.
Check the server. If Google says it postponed crawling because of load, slow responses are part of the problem.

Crawled - currently not indexed

Ask whether the page deserves to be indexed. If it repeats what a hundred other pages say, improve it with something only you can add: your data, your test, your error message.
Merge thin pages into one stronger page and redirect the rest.
Unpublish what you wouldn't miss. Fewer weak pages also means less of your site Google has to judge.
Then request indexing once, after the page has actually changed.

One thing that doesn't help Google here is IndexNow. Bing, Yandex and a few other engines use it, but Google doesn't. I wrote about that in the IndexNow status codes post.

I'll re-inspect the service pages and the rest of the 63 in a week and update this post with what moved.

FAQ

What is the difference between Crawled and Discovered - currently not indexed?

Discovered means Google knows the URL but has not fetched it yet, so there is no crawl date. Crawled means Google fetched and read the page, then decided not to index it. Discovered is usually about priority and links; Crawled is usually about the page itself.

How do I fix Crawled - currently not indexed?

Change the page before asking again. Add something other pages do not have, such as your own data or a tested fix, merge thin pages into a stronger one, or unpublish pages you would not miss. Then request indexing once.

How do I fix Discovered - currently not indexed?

Link to the page from pages that are already indexed, keep only indexable 200 URLs in your sitemap, and request indexing for the few pages that matter most. Orphan pages that exist only in the sitemap are the most common case.

Does Request indexing in Search Console actually work?

Sometimes. On this site, 6 of 12 requested pages were indexed one day later, including two that had been Crawled - currently not indexed. All five service pages stayed unindexed. It puts the URL in a queue but does not guarantee indexing.

Can I request indexing through an API?

Not for normal pages. Google's Indexing API only accepts job posting and livestream pages. The URL Inspection API can check a URL's index status, up to 2,000 calls a day per property, but it cannot request indexing.

How long does Discovered - currently not indexed last?

Google gives no fixed time. Some of this site's pages moved within a day of a request; others stayed Discovered. Internal links from indexed pages are the most reliable way to raise a URL's priority.

Should tag pages be indexed?

Usually only when they hold several posts and add something, like a short introduction. Most of this site's tag pages were Discovered or unknown. A tag with a single post is noindexed, and it should not be in the sitemap either.

Originally published on knowledgemarkg.com.

ASP.NET Core 'Error Unprotecting the Session Cookie': Why Your Keys Disappear on Deploy

Kausar Raza — Sun, 01 Nov 2026 14:45:25 +0000

You deploy an ASP.NET Core app, or IIS recycles its app pool overnight, and the log fills up with this:

warn: Microsoft.AspNetCore.Session.SessionMiddleware[7]
      Error unprotecting the session cookie.
      System.Security.Cryptography.CryptographicException: The key {83f2e736-3392-4263-9f0f-ad943c51fe3e} was not found in the key ring.

At the same time, users report being logged out, and forms fail with "The antiforgery token could not be decrypted". The session warning is just the visible part. The cause is that the app has lost the keys it used to encrypt its cookies. Every cookie issued before that moment is now unreadable.

This happened on the API behind this site. It runs on IIS under Plesk, where the app pool has no loaded user profile, so ASP.NET Core kept its keys in memory and threw them away on every restart. Below is what I reproduced, the exact messages each cause produces, and the fix. The deploy side of IIS and Plesk 500 errors is in the Plesk 500 errors post.

The short fix: store the Data Protection keys in a folder that survives restarts and deploys, and give the app a fixed name: services.AddDataProtection().PersistKeysToFileSystem(new DirectoryInfo(@"App_Data\keys")).SetApplicationName("MyApp"). Then make sure your deploy never deletes that folder. Users who already have old cookies will be logged out once more, and after that it stops.

What the error actually means

ASP.NET Core doesn't store session data, login tickets or antiforgery tokens in plain text. It encrypts them with the Data Protection system, which keeps a set of keys called the key ring. Each encrypted payload carries the ID of the key that made it. When a request comes in, the app looks up that ID in its ring. If it isn't there, you get "The key {...} was not found in the key ring".

Three components report it, in three different ways:

Session middleware logs "Error unprotecting the session cookie" as a warning, then quietly starts a new, empty session.
Cookie authentication logs "Cookies was not authenticated. Failure message: Unprotect ticket failed", but only at Information level. With the default "Microsoft.AspNetCore": "Warning" setting in appsettings.json, you never see it. The user is just signed out.
Antiforgery throws AntiforgeryValidationException: "The antiforgery token could not be decrypted", which usually surfaces as a 400 on form posts.

That's why the session warning is the one people search for. It's the only one that shows up in a default log, even though the logouts hurt more.

What I tested

I built a minimal ASP.NET Core 9 app (runtime 9.0.8, Windows 11, Kestrel) with cookie sign-in, session and antiforgery. I signed in with curl, kept the cookie jar, restarted the app with a different key setup each time, and sent the old cookies back. These are the results.

1. Keys persisted, plain restart: nothing breaks

With keys stored in a folder and the app restarted on the same folder, the user was still signed in and the old antiforgery token still validated. This is the control. One thing did reset: the session value was empty after the restart. That isn't a key problem. AddDistributedMemoryCache() keeps session data in process memory, so a restart loses it even when the cookie decrypts fine. If you need sessions to survive restarts, use SQL Server or Redis for the session store as well.

2. Key folder wiped: "was not found in the key ring"

Pointing the restarted app at an empty folder is exactly what a deploy does when it deletes the old site folder, including App_Data. The user came back anonymous, the antiforgery post failed with "The antiforgery token could not be decrypted", and the log showed the error from the top of this article, naming the old key's ID. The app also silently created a new key, logging:

No XML encryptor configured. Key {53f0f160-...} may be persisted to storage in unencrypted form.

3. Same keys, different application name: "The payload was invalid"

Data Protection isolates apps from one another using an application name. If you don't set one, it is derived from the app's content root path. So moving the site to a different folder, or swapping deployment slots, can change it. I kept the same key folder and changed only SetApplicationName from SiteA to SiteB. The keys were found, but nothing decrypted:

warn: Microsoft.AspNetCore.Session.SessionMiddleware[7]
      Error unprotecting the session cookie.
      System.Security.Cryptography.CryptographicException: The payload was invalid.
info: Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[7]
      Cookies was not authenticated. Failure message: Unprotect ticket failed

The antiforgery error changed too, to "The required antiforgery cookie ".AspNetCore.Antiforgery.6asUPwDiMTw" is not present". The suffix of that cookie name is derived from the application name, so the app was looking for a cookie with a different name. If you see "payload was invalid" rather than "key not found", look at the application name before the keys.

4. Where the keys go by default

With no configuration, a normal Windows user account gets this at startup:

User profile is available. Using 'C:\Users\<you>\AppData\Local\ASP.NET\DataProtection-Keys' as key repository and Windows DPAPI to encrypt keys at rest.

That's why it works on your machine. An IIS app pool doesn't have that folder unless "Load User Profile" is on, and on shared hosting such as Plesk you often can't control that setting. When there is no profile and no registry hive set up for the pool, the framework uses an in-memory ring. I couldn't reproduce that fallback outside IIS, but its warning is in the framework itself (Microsoft.AspNetCore.DataProtection.dll, 9.0.8):

Neither user profile nor HKLM registry available. Using an ephemeral key repository. Protected data will be unavailable when application exits.

If that line is in your startup log, every app pool recycle, idle timeout and deploy logs your users out.

The fix

Three settings, all in Program.cs before builder.Build(). This is the configuration this site's API uses now:

var keysDir = new DirectoryInfo(Path.GetFullPath("App_Data/keys", builder.Environment.ContentRootPath));
keysDir.Create();

var dataProtection = builder.Services
    .AddDataProtection()
    .PersistKeysToFileSystem(keysDir)          // keys survive restarts and recycles
    .SetApplicationName("MyApp");              // fixed, so moving the folder or slot doesn't change it

if (OperatingSystem.IsWindows())
{
    dataProtection.ProtectKeysWithDpapi(protectToLocalMachine: true);  // encrypt key files at rest
}

Notes on each line:

The folder must be outside the release files. If your deploy copies the site over App_Data, or deletes and re-extracts it, the keys go with it. Keep App_Data out of the delete step, the same way you keep uploads.
Use ProtectKeysWithDpapi(protectToLocalMachine: true), not the per-user default. An app pool has no user profile to hold a per-user DPAPI key. Machine scope works without one. The trade-off is that the key files can only be decrypted on the same machine.
Set the application name once and never change it. Changing it logs every user out, the same as losing the keys.

After the change, the first deploy logs everyone out once. That's expected: the old cookies were encrypted with keys that no longer exist, and there's nothing to recover them with. From then on, restarts and recycles keep the same keys.

How to check it worked

Sign in, then recycle the app pool or restart the app.
Check that you are still signed in. If you are, the keys survived.
Check the log. The "User profile is available" line should be gone, and no "Neither user profile nor HKLM registry available" line should appear. Either one would mean the keys went to memory.
Check that App_Data\keys contains key-*.xml files, and that the timestamps don't change on every restart.

FAQ

What does 'The key was not found in the key ring' mean in ASP.NET Core?

ASP.NET Core encrypted a cookie or token with a key that the app no longer has. The key ring is empty or missing that key, so the payload can't be decrypted. The usual cause is keys stored in memory, or a key folder that was wiped during a deploy.

Why am I logged out every time I deploy or recycle the app pool?

The Data Protection keys are kept in memory, or in a folder that the deploy deletes. The new process has different keys, so cookies from before can't be decrypted. Persist the keys to a folder outside the release files, with PersistKeysToFileSystem.

What does 'The payload was invalid' mean in ASP.NET Core Data Protection?

The app found the key but the cookie still couldn't be decrypted. A common cause is a different application name. Check SetApplicationName, and check that the app isn't running from a different folder or deployment slot with a different default name.

What is the difference between 'key not found' and 'payload was invalid'?

'Key not found' means the key ring doesn't contain the key that encrypted the cookie, so the keys were lost or moved. 'Payload was invalid' means the key was found, but the app name or key data doesn't match. Keys first, then application name.

Where should ASP.NET Core store Data Protection keys on IIS?

In a folder that the app pool identity can write to, outside the release folder, for example App_Data/keys. Protect the key files with ProtectKeysWithDpapi(protectToLocalMachine: true), since an app pool usually has no user profile for per-user DPAPI.

What does 'No XML encryptor configured' mean?

The Data Protection keys are being saved to disk without encryption. It's a warning, not an error. Fix it with ProtectKeysWithDpapi on Windows, or with a certificate or Key Vault on other hosts.

Does the session warning 'Error unprotecting the session cookie' mean my users are logged out?

Not directly. It's the session middleware, which starts an empty session. Login is handled by cookie authentication, which logs a separate message at Information level. Check that too, because the warning alone doesn't show it.

Originally published on knowledgemarkg.com.

LinkedIn 'Content Credentials Label Added' (CR Icon): What It Means and How to Remove It

Kausar Raza — Tue, 27 Oct 2026 15:48:11 +0000

I uploaded a banner to a LinkedIn post this week and a small notice appeared in the corner of the editor:

Content credentials label added.

A round cr badge sat on the top-left corner of the image, and it stayed there once the post was published. The same image went to my Facebook Page the same day, and Facebook put "AI content" under the Page name - even though the composer's AI label switch was showing off when I posted.

The image came out of ChatGPT. Neither platform looked at the picture and decided it looked AI-generated. Both read something hidden inside the file. Below is what that something is, the test that proves it, and what you can and can't do about the label.

The short answer: "Content credentials label added" means the image carries a signed C2PA manifest - a small block of provenance data that tools like ChatGPT, Adobe Firefly and some cameras embed in the file. LinkedIn shows it as the CR icon; Facebook and Instagram turn it into an "AI content" / "AI info" label. You can't switch the label off on the platform. It goes away only if the file you upload no longer contains the manifest - and for AI-generated images there are good reasons to leave it on.

What is inside a ChatGPT image

A PNG file is a list of named chunks: IHDR for the dimensions, IDAT for the pixels, IEND to close it. Listing the chunks of my ChatGPT banner (1672 x 941, 1,704,000 bytes) gave one I didn't expect:

IHDR      13 bytes
caBX  23,654 bytes   <- the Content Credentials
IDAT    (pixels)
IEND       0 bytes

caBX is where C2PA stores its manifest in a PNG. 23,654 bytes is about 1.4% of this file. Reading it with the C2PA project's own Python library (c2pa-python 0.38) gave this, trimmed to the parts that matter:

claim generator : OpenAI Media Service API
signed by       : OpenAI OpCo, LLC  (OpenAI Media Service), alg Ps256
actions         :
  c2pa.created              softwareAgent: ChatGPT / gpt-image
                            digitalSourceType: trainedAlgorithmicMedia
  c2pa.converted
  c2pa.watermarked.unbound

Three lines in there do the work:

trainedAlgorithmicMedia is the IPTC term for "created by a generative AI model". This is the value platforms look for.
The signature is OpenAI's. The manifest is cryptographically signed, so anyone can check that it wasn't edited afterwards. (My local check reported the signature as structurally valid but the certificate as "untrusted" - only because I hadn't loaded a trust list, not because anything was wrong.)
c2pa.watermarked.unbound is the one most articles miss. In the C2PA specification (2.3 and later) it means an invisible watermark was inserted into the image itself, without creating a soft binding. That watermark is in the pixels, not the metadata.

The test: same image, with and without the manifest

To prove the label comes from the manifest and not from what the image looks like, I made a copy with identical pixels and no manifest. Re-saving through Pillow does exactly that, because Pillow doesn't carry unknown chunks over:

from PIL import Image

Image.open("banner.png").save("banner-resaved.png")

The re-saved file had no caBX chunk, and c2pa-python reported ManifestNotFound. Then I uploaded both to the LinkedIn post editor, one after the other in the same session, and discarded the draft afterwards:

File	C2PA manifest	LinkedIn editor
Original ChatGPT PNG	Yes, signed by OpenAI	cr badge + "Content credentials label added."
Same pixels, re-saved with Pillow	None	No badge, no notice

Same picture, different result. LinkedIn is reading the file's metadata, and it is not (in this test, at least) detecting the invisible watermark.

Facebook I could only observe, not A/B test, because its label only appears on a published post. The original PNG got "AI content" with the composer's AI switch set to off. Clicking the label explains that it is based on a signal added by the tool that created the content - that is the manifest. The post's "..." menu has an AI content item, which opens the same explanation. There is no option there to remove it.

What LinkedIn shows people who click the CR icon

LinkedIn's help centre calls it the C2PA icon. According to that page, clicking it can show:

whether AI was used to generate or edit the content, as stated by whoever signed it
the app or device used - a camera model or an AI tool
who created the image, and which organisation issued and signed the credential
when it was signed

LinkedIn also says the feature is still rolling out, so not every account sees it, and that not every AI-generated image can be identified. On the published post, the badge is a button labelled "This image has content credentials."

How to check whether an image has Content Credentials

Before you post, you can see what a platform will see:

Online: the Content Authenticity Initiative's Verify tool (contentcredentials.org/verify) reads the manifest in your browser and shows the same fields.
From code: the C2PA project's own library.

# pip install c2pa-python
import json
import c2pa

with open("banner.png", "rb") as f, c2pa.Reader("image/png", f) as reader:
    store = json.loads(reader.json())

manifest = store["manifests"][store["active_manifest"]]
print(manifest["claim_generator_info"][0]["name"])
for assertion in manifest["assertions"]:
    if assertion["label"].startswith("c2pa.actions"):
        for action in assertion["data"]["actions"]:
            print(action["action"], action.get("digitalSourceType", ""))

On a file without a manifest, c2pa.Reader raises ManifestNotFound - which is also the quickest way to confirm a removal worked.

How to remove the CR label - and whether you should

There is no setting on LinkedIn or Facebook to switch a detected label off. The label follows the file, so the only way to avoid it is to upload a file without the manifest. These do that:

Re-saving or converting the image in a tool that doesn't keep C2PA data. Pillow does this by default (tested above, for both PNG and WebP). Many image editors and online converters behave the same way; check the result with the reader rather than assuming.
Taking a screenshot. OpenAI's own documentation says screenshots remove the metadata, and that most social platforms strip metadata from uploads anyway - which is why an image without Content Credentials proves nothing either way.

Two things to weigh before you do it:

The watermark stays. Removing the manifest doesn't remove the invisible watermark the manifest says was added. LinkedIn didn't react to it in my test; that tells you about LinkedIn today, not about every platform or detector tomorrow. Meta said when it launched these labels that it was working on detecting invisible watermarks as well as metadata.
The label is accurate. If ChatGPT made the image, "AI content" is true. For a banner, diagram or illustration, nobody is misled and the label costs you very little. The other case is different: a realistic AI image of a person or an event is exactly what these labels exist for, platforms expect it to be disclosed, and stripping the manifest to pass it off as a real photo is the kind of thing their AI-content rules are written against.

The legitimate reason to strip it is the opposite situation: an image you made yourself, where one small AI-assisted step (an upscale, a background fill) produced a manifest that makes the whole thing read as AI-generated. Facebook's own wording separates "entirely AI-generated" from "only modified with AI" for this reason.

A checklist before you post an AI image

Know where the image came from. If a generator made it, assume it carries a manifest and that LinkedIn, Facebook and Instagram will label it.
Check, don't guess. Run it through the Verify tool or the reader above. Some editing tools keep the manifest, some drop it, and some add their own - the only reliable answer is the file.
Decide whether the label is fair. Fully AI-generated: leave it. Your own photo or design with a small AI-assisted step: removing the manifest is reasonable, and Facebook's own wording draws the same line.
Disclose in words when it matters. For anything realistic - a person, a place, an event - say it is AI-generated in the post itself. Labels can be lost along the way; your caption isn't.
Re-check after converting. If you resize or compress the image for a website, confirm what survived. Conversion can drop the manifest without you asking it to.

What I do now

For this site I keep two versions of every AI-generated banner. The original PNG, with its manifest, goes to LinkedIn and Facebook, where the label is honest and harmless. The site itself serves a WebP version made with Pillow - mainly because WebP at quality 88 is about 150 KB against 1.7 MB for the PNG, which is what decides how fast the page loads. The manifest disappears in that conversion as a side effect, and I've left it that way because a page's featured image isn't where people look for provenance. If you care about page weight, the same reasoning applies - I wrote up how much image size matters for Largest Contentful Paint.

The banner that started all this was for my article on the sqlcmd "scripting variable not defined" error - a reminder that the content of a file and what tools read from it are not always the same thing.

FAQ

What does 'Content credentials label added' mean on LinkedIn?

It means the image or video you uploaded carries a signed C2PA manifest - provenance data embedded by the tool that made it, such as ChatGPT, Adobe Firefly or some cameras. LinkedIn reads it and shows a CR (C2PA) icon on the post that viewers can click for details.

What is the CR icon on LinkedIn posts?

It is LinkedIn's Content Credentials icon, which LinkedIn's help centre calls the C2PA icon. Clicking it can show whether AI was used, the app or device used, who created and signed the content, and when it was signed.

How do I remove the CR label from a LinkedIn post?

There is no setting to switch it off. The label comes from metadata in the file, so it only disappears if you upload a version without the C2PA manifest - for example after re-saving or converting the image in a tool that doesn't keep it, or by using a screenshot. In a test, the same pixels without the manifest got no label.

Why does Facebook say 'AI content' or 'AI info' on my post when I turned the AI label off?

The toggle is for labelling content yourself. Facebook also adds the label automatically when the file contains a signal from the tool that created it, such as C2PA metadata saying it is AI-generated. In a test, a ChatGPT image was labelled 'AI content' with the composer's AI label set to off.

Do ChatGPT images contain C2PA metadata?

Yes. A ChatGPT image tested in October 2026 had a 23,654-byte caBX chunk holding a C2PA manifest signed by OpenAI, recording that it was created by ChatGPT (gpt-image) as trainedAlgorithmicMedia, plus a c2pa.watermarked.unbound action indicating an invisible watermark.

Does removing C2PA metadata remove the AI watermark?

No. The metadata and the watermark are separate. The C2PA manifest is in the file's metadata; the invisible watermark is in the pixels. Re-saving the image removes the first but not the second.

How can I check if an image has Content Credentials?

Upload it to the Content Authenticity Initiative's Verify tool at contentcredentials.org/verify, or read it in code with the c2pa-python library. A file without a manifest returns ManifestNotFound.

Originally published on knowledgemarkg.com.

sqlcmd 'Scripting Variable Not Defined': Why $( Breaks Your Script and How to Fix It

Kausar Raza — Tue, 27 Oct 2026 03:12:08 +0000

If you run SQL scripts through sqlcmd - a deploy step, a data fix, a seed script, a pipeline that loads text into a table - sooner or later one of them prints this:

'Amount' scripting variable not defined.

The name in quotes changes; the cause doesn't. Somewhere in your script there is a $( followed by a name and a ), and sqlcmd has read it as one of its own variables. Usually it's inside a string you meant literally: a price template, a shell command stored in a table, a blog post with a code sample in it. I hit it on this site, whose posts are inserted by a script that pipes generated SQL through sqlcmd - any post with a shell example like $(cygpath -w file) in it tripped over this.

The short fix: if the script doesn't use sqlcmd variables at all, run it with sqlcmd -x (or Invoke-Sqlcmd -DisableVariables in PowerShell). If it does use them, break the pattern inside the SQL text instead: write '$' + '(Amount)' or CHAR(36) + '(Amount)'. And if you build the command in Bash or PowerShell, put the query in single quotes - in double quotes the shell eats $(...) before sqlcmd ever sees it.

That's the fix. What makes this worth an article is that the error message is the least dangerous thing sqlcmd does with $(...) - the dangerous cases print nothing at all.

What sqlcmd is doing

sqlcmd is not just a pipe to SQL Server. Before it sends each batch, it runs its own pre-processor over the text, and one of that pre-processor's jobs is replacing $(Name) with the value of a scripting variable called Name. This is a feature: it lets one script run against different databases or servers.

:setvar TargetDb "Shop_Staging"
USE $(TargetDb);
SELECT COUNT(*) FROM dbo.Orders;

The catch is that the pre-processor doesn't care about T-SQL quoting. It replaces $(Name) inside string literals - '...' and N'...' - exactly as it does in code. To sqlcmd, 'Total: $(Amount)' is a variable reference wrapped in some characters that happen to be quotes.

A value for a variable can come from five places. From lowest to highest precedence, according to Microsoft's documentation:

system-level environment variables
user-level environment variables
variables set in the shell before starting sqlcmd (SET X=Y, export X=Y)
sqlcmd -v X=Y
:setvar X Y inside the script

Only if none of the five has a value do you get "scripting variable not defined". Keep the first three in mind - they are where the silent failures come from.

What I tested

Everything below was run on 26 October 2026 with the classic ODBC-based sqlcmd (version 15.0.1300.359, Windows) against SQL Server 2022, using only SELECT and PRINT. The test script is one line:

SELECT 'Total: $(Amount)' AS r;

1. Without -b, the error doesn't stop anything

Run it plainly and you get the message on stderr - and the query still runs, with the literal text left in:

r
-
Total: $(Amount)

(1 rows affected)
'Amount' scripting variable not defined.

The exit code is 0. If this script were an INSERT, the row would be saved exactly as written, and any CI job checking the exit code would report success. Microsoft's own page contradicts itself here: one sentence says "the execution of the script stops", while a worked example further down shows the row being returned. The example - and my test - are right. It only stops when you pass -b.

2. With -b, it stops - halfway through

With -b, the batch containing the reference is never sent and sqlcmd exits with code 1. But batches before it have already run. A three-batch script printed batch 1 ran, then the error, then nothing. If the script isn't wrapped in a transaction, you are left with the first half applied. That's still the behaviour you want in automation - just know that "failed" doesn't mean "nothing happened".

3. Environment variables fill it in - silently

This is the one that actually corrupts data. I changed the test string to:

SELECT 'path=$(USERNAME)' AS envleak, '$(SQLCMDSERVER)' AS srv;

No message, exit code 0, and the result came back with my Windows user name in place of $(USERNAME) and the server's address in place of $(SQLCMDSERVER). sqlcmd treats every environment variable as a scripting variable, and it defines a set of its own (SQLCMDSERVER, SQLCMDUSER, SQLCMDDBNAME and more). So a stored string that happens to contain $(PATH), $(TEMP), $(COMPUTERNAME) or $(HOME) is quietly rewritten with values from whatever machine ran the script - including, on a build agent, paths and account names you didn't mean to put in your database. Names are case-insensitive too: -v AMOUNT=5 filled in $(amount).

The visible error only appears when the name matches nothing. You only get a warning when you're lucky.

4. A broken pattern is a syntax error

An unclosed $( or a name with a space in it gives a different message:

Sqlcmd: Error: Syntax error at line 1 in file 't7.sql'.

Here the batch is not sent - but without -b the exit code is still 0, so the script "succeeds" having skipped that statement.

5. Things that turned out to be fine

Some things I expected to break and didn't, in this version:

$(Anything) inside a -- or /* */ comment produced no message.
A multi-line string literal containing a line that starts with GO, :setvar or !!echo was kept as text. sqlcmd did not split the batch, set a variable or run a shell command.

So it's specifically $(...) in code and string literals you need to watch.

Fix 1: turn variables off with -x

If the script never uses sqlcmd variables, this is the whole fix:

sqlcmd -S myserver -d Shop -x -b -i load_content.sql

-x makes sqlcmd ignore scripting variables completely - no substitution, no message, the text goes to SQL Server as written. It's an all-or-nothing switch: in my test, -x together with -v Amount=19.99 still produced the literal $(Amount), and :setvar inside the file was ignored as well. If the script relies on $(TargetDb) somewhere, -x breaks that.

Keep -b alongside it, so real SQL errors still fail the run.

PowerShell: Invoke-Sqlcmd -DisableVariables

The SqlServer module's Invoke-Sqlcmd uses the same variable syntax, and its equivalent of -x is -DisableVariables. Microsoft's documentation describes it in nearly the same words as -x - "useful when a script contains many INSERT statements that may contain strings that have the same format as variables". (I didn't have the module installed for this test, so this one is from the docs, not from a run.)

Invoke-Sqlcmd -ServerInstance myserver -Database Shop -InputFile .\load_content.sql -DisableVariables

Fix 2: break the pattern inside the SQL

When the script needs variables and contains literal $(...) text, change how the text is written so the pre-processor never sees $(. Both of these came back as the literal Total: $(Amount) with no message:

SELECT 'Total: $' + '(Amount)'          AS concat_fix,
       'Total: ' + CHAR(36) + '(Amount)' AS char_fix;

CHAR(36) is the dollar sign. Use NCHAR(36) and N'...' for nvarchar data. There is no escape sequence for $( in sqlcmd - splitting it is the workaround. If you generate SQL from code, do the split in the generator so nobody has to remember it by hand:

def sql_literal(text):
    # N'...' with quotes doubled, and "$(" split so sqlcmd can't see a variable
    body = text.replace("'", "''").replace("$(", "$' + N'(")
    return "N'" + body + "'"

The same goes for SQL database projects and SSMS in SQLCMD mode, which use the same $(Name) syntax for their variables. I haven't tested those here, but the split string has no $( in it for any of them to find.

Fix 3: check what your shell did first

A related trap sits one layer earlier, in the shell. Both Bash and PowerShell use $(...) for their own command substitution, and both expand it inside double quotes before sqlcmd starts:

$ q="SELECT 'Total: $(Amount)' AS r"; echo "$q"
bash: Amount: command not found
SELECT 'Total: ' AS r

PowerShell does the same and says The term 'Amount' is not recognized as the name of a cmdlet, function, script file, or operable program, then hands over SELECT 'Total: ' AS r. The query runs, the value is just gone - and sqlcmd never complains because it never saw a $(. If what comes out has an empty gap where your text was, this is why. Use single quotes in the shell (doubling the inner SQL quotes in PowerShell), escape it as `$( in a PowerShell double-quoted string, or - better - put the SQL in a file and pass it with -i.

Which fix to use

Loading content (articles, templates, HTML, anything users wrote): -x or -DisableVariables. You can't predict what text contains, and environment variable substitution makes it unsafe to rely on the error appearing.
Deployment scripts that use variables: keep variables on, split any literal $( with CHAR(36), and always pass -b.
Large or repeated data loads: consider not generating SQL text at all. A parameterised command from your application, bcp or BULK INSERT doesn't put the data through sqlcmd's pre-processor, so there's nothing to escape.

The other sqlcmd defaults that cause trouble

This is the second article I've written about sqlcmd quietly behaving differently from the application that normally talks to the same database. The first was Msg 1934, "SET options have incorrect settings: QUOTED_IDENTIFIER" - the classic sqlcmd starts sessions with QUOTED_IDENTIFIER OFF, which breaks writes to any table with a filtered index. The script that inserts posts on this site now passes both -x and a SET QUOTED_IDENTIFIER ON where it needs one. If you run SQL by hand against a database an ORM normally manages, it's also worth reading about EF Core's "There is already an object named X in the database", which is what happens when hand-run SQL and migration history drift apart.

FAQ

What does 'scripting variable not defined' mean in sqlcmd?

Your script contains $(Name) and sqlcmd treats that as a reference to a scripting variable called Name. It found no value for it in :setvar, -v or the environment, so it prints the message. The text is usually a literal string you didn't mean as a variable.

Does the script stop when sqlcmd says a scripting variable is not defined?

Not by default. In a test with sqlcmd 15, the statement still ran with the literal $(Name) text and the exit code was 0. Only with -b does sqlcmd skip that batch and exit with code 1, and batches before it have already run.

How do I stop sqlcmd from replacing $( in my data?

Run sqlcmd with -x, which turns off scripting variables entirely, or use Invoke-Sqlcmd -DisableVariables in PowerShell. If the script also needs variables, write the dollar sign as CHAR(36) or split the string as '$' + '(Name)' so sqlcmd never sees the $( pattern.

Is there an escape character for $( in sqlcmd?

No. sqlcmd has no escape sequence for the $( pattern. The workarounds are -x to disable variables, or building the text so the two characters are never next to each other in the script, for example CHAR(36) + '(Name)'.

Can environment variables change my data when I run sqlcmd?

Yes. sqlcmd treats environment variables as scripting variables, so a string containing $(USERNAME), $(PATH) or $(TEMP) is silently replaced with that machine's value, with no warning and exit code 0. That's why -x is the safer choice for loading user-written content.

What is the order of precedence for sqlcmd variables?

From lowest to highest: system environment variables, user environment variables, variables set in the shell before starting sqlcmd, the -v option, and :setvar inside the script. Variable names are case-insensitive.

Why is my query empty where $(...) was, with no sqlcmd error?

Your shell probably expanded it first. Bash and PowerShell both treat $(...) inside double quotes as command substitution, so the text is removed before sqlcmd starts. Use single quotes, escape it as `$( in PowerShell, or put the SQL in a file and pass it with -i.

Originally published on knowledgemarkg.com.

SQL Server Msg 1934: 'SET Options Have Incorrect Settings: QUOTED_IDENTIFIER' - Why It Happens and How to Fix It

Kausar Raza — Fri, 23 Oct 2026 17:58:22 +0000

This is the error, exactly as SQL Server printed it to me while I was fixing this site:

Msg 1934, Level 16, State 1, Line 1
UPDATE failed because the following SET options have incorrect settings: 'QUOTED_IDENTIFIER'.
Verify that SET options are correct for use with indexed views and/or indexes on computed
columns and/or filtered indexes and/or query notifications and/or XML data type methods
and/or spatial index operations.

I was fixing a broken profile photo on this site by updating one row in the AspNetUsers table from the command line. The same table is updated every day by the ASP.NET Core application that runs this site, and that application has never once seen this error. Same database, same table, same kind of statement - one client fails, the other doesn't. That difference is the whole explanation, and once you see it the fix takes one line.

The short fix: put SET QUOTED_IDENTIFIER ON; at the top of your script, before the statement that fails. If you run scripts with the classic sqlcmd, add the -I switch instead. If the failing statement is inside a stored procedure, re-create the procedure from a session that has the option on - setting it inside the procedure does nothing. The rest of this article explains why, and how to find every place it can bite.

What the error actually means

Read the long sentence at the end of the message as a checklist. SQL Server is saying: this statement touches one of six features that only work with a particular set of session options - indexed views, indexes on computed columns, filtered indexes, query notifications, XML data type methods, or spatial indexes - and your session has at least one of those options set the wrong way. The option it names in quotes ('QUOTED_IDENTIFIER' here) is the one that is wrong.

The first word changes with the statement. You will see INSERT failed, UPDATE failed, DELETE failed, SELECT failed, CREATE INDEX failed or ALTER INDEX failed - it is the same error, number 1934, every time. And the option in quotes is not always QUOTED_IDENTIFIER; it can also be ANSI_WARNINGS, ANSI_NULLS, ARITHABORT and a few others. The cause is always the same shape: a client connected with a non-default setting.

In my case the feature involved was a filtered index. ASP.NET Core Identity creates one on AspNetUsers:

SELECT OBJECT_NAME(object_id) AS [table], name, filter_definition
FROM sys.indexes
WHERE has_filter = 1;

-- table         name            filter_definition
-- AspNetRoles   RoleNameIndex   ([NormalizedName] IS NOT NULL)
-- AspNetUsers   UserNameIndex   ([NormalizedUserName] IS NOT NULL)

Those two rows are the only filtered indexes in this site's database, and both come from Identity: a unique index on a column that allows NULLs gets an IS NOT NULL filter, so that more than one row can be NULL. So if your project uses ASP.NET Core Identity on SQL Server, you have filtered indexes whether you ever wrote one or not - and any script that updates users or roles from the wrong client will hit this error.

Why one client fails and another doesn't

QUOTED_IDENTIFIER is a session setting, and each client library picks its own default when it connects. That is the entire mystery:

ADO.NET / SqlClient, Entity Framework Core, the ODBC driver, OLE DB, and SSMS connect with it ON. Microsoft's documentation says the ODBC driver and OLE DB provider set it on automatically when connecting. This is why the application never sees the error.
The classic sqlcmd (the ODBC-based one that ships with the SQL Server client tools) starts with it OFF. I checked rather than trusting memory: a plain session from sqlcmd 15.0 reported SESSIONPROPERTY('QUOTED_IDENTIFIER') = 0. Its -I switch turns it on.
The newer sqlcmd (Go) always has it on. Microsoft's docs say its -I switch is ignored because quoted identifiers are always enabled. So whether this error happens can depend on which sqlcmd is first on your PATH.
DB-Library applications default to OFF - old, but still found in legacy tools.

Check what your own session has before guessing:

SELECT SESSIONPROPERTY('QUOTED_IDENTIFIER')       AS quoted_identifier,
       SESSIONPROPERTY('ANSI_NULLS')              AS ansi_nulls,
       SESSIONPROPERTY('ANSI_WARNINGS')           AS ansi_warnings,
       SESSIONPROPERTY('ANSI_PADDING')            AS ansi_padding,
       SESSIONPROPERTY('ARITHABORT')              AS arithabort,
       SESSIONPROPERTY('CONCAT_NULL_YIELDS_NULL') AS concat_null_yields_null;

From sqlcmd on this machine that returned 0 for QUOTED_IDENTIFIER and for ARITHABORT, and 1 for the rest.

What I tested

To be sure which statements fail and which don't, I reproduced it on SQL Server 2022 (compatibility level 150) with a temporary copy of the Identity table - same filtered unique index, nothing touching real data:

SET QUOTED_IDENTIFIER ON;
CREATE TABLE #Users (Id int PRIMARY KEY, NormalizedUserName nvarchar(256) NULL);
CREATE UNIQUE INDEX UserNameIndex ON #Users (NormalizedUserName)
    WHERE NormalizedUserName IS NOT NULL;
GO
SET QUOTED_IDENTIFIER OFF;   -- now behave like a default sqlcmd session
GO
UPDATE #Users SET NormalizedUserName = N'B' WHERE Id = 2;   -- Msg 1934

Statement	Session settings	Result
INSERT, UPDATE, DELETE on the table	QUOTED_IDENTIFIER OFF	Msg 1934 (each names its statement)
Plain SELECT on the table	QUOTED_IDENTIFIER OFF	Works
SELECT using an xml `.value()` method	QUOTED_IDENTIFIER OFF	Msg 1934, "SELECT failed"
CREATE INDEX with a WHERE filter	QUOTED_IDENTIFIER OFF	Msg 1934, "CREATE INDEX failed"
ALTER INDEX ... REBUILD on the filtered index	QUOTED_IDENTIFIER OFF	Msg 1934, "ALTER INDEX failed"
UPDATE	QUOTED_IDENTIFIER ON, ARITHABORT OFF	Works
UPDATE	QUOTED_IDENTIFIER ON, ANSI_WARNINGS OFF	Msg 1934 naming 'ANSI_WARNINGS'
EXEC of a procedure created with QUOTED_IDENTIFIER OFF	caller has it ON	Msg 1934 inside the procedure
UPDATE from `sqlcmd -I`	QUOTED_IDENTIFIER ON	Works

Three of those results are worth pulling out, because they are where people lose time:

Reads usually still work. A plain SELECT on a table with a filtered index did not fail; the index simply can't be used. That is why a script can read everything fine and then fall over at the first write - it looks like a permissions or locking problem when it isn't.
ARITHABORT OFF was not a problem on this server, even though the session had it off. With ANSI_WARNINGS on and the database at compatibility level 90 or higher, SQL Server treats ARITHABORT as on. If you do see 'ARITHABORT' in the message, look at ANSI_WARNINGS and the compatibility level first.
Stored procedures remember their creation setting. The caller's setting is irrelevant - more on that below, because it is the case that survives every other fix.

How to fix it

1. In a script: set the option before the statement

SET QUOTED_IDENTIFIER ON;
SET ANSI_NULLS ON;
GO

UPDATE AspNetUsers
SET    ProfileImageUrl = '/uploads/profiles/new-photo.webp'
WHERE  Id = '...';

Microsoft's documentation is precise about when this takes effect: QUOTED_IDENTIFIER applies at parse time, and a SET inside a batch changes parsing from that point on. So it works on the same line as long as it comes first. Putting it in its own batch with GO, as above, is the version that never surprises anyone.

2. From sqlcmd: use -I, or the Go version

sqlcmd -S myserver -d mydb -I -i fix.sql

-I applies only to the classic ODBC sqlcmd. The newer Go-based sqlcmd has quoted identifiers on permanently, so the switch does nothing there - and the error doesn't happen there either. If you have a deploy script that runs SQL files through sqlcmd, add -I to it now; it costs nothing on the Go version and fixes the ODBC one.

3. In a stored procedure: re-create it, don't SET inside it

When a procedure is created, SQL Server stores the QUOTED_IDENTIFIER and ANSI_NULLS settings that were in force at that moment, and uses them for every later call. A SET QUOTED_IDENTIFIER ON written inside the procedure body has no effect. In my test, a procedure created from an OFF session failed with Msg 1934 even when called from a session that had the option on.

Find every module saved with the wrong setting:

SELECT OBJECT_SCHEMA_NAME(m.object_id) AS [schema],
       OBJECT_NAME(m.object_id)        AS [name],
       o.type_desc,
       m.uses_quoted_identifier,
       m.uses_ansi_nulls
FROM sys.sql_modules AS m
JOIN sys.objects     AS o ON o.object_id = m.object_id
WHERE m.uses_quoted_identifier = 0
   OR m.uses_ansi_nulls = 0
ORDER BY [schema], [name];

For each row, script the module out and run its ALTER (or CREATE OR ALTER) from a session with both options on - SSMS, or sqlcmd -I. The same applies to triggers and functions; they are in sys.sql_modules too. A deployment tool that runs your migration scripts through the classic sqlcmd is the usual way these get saved wrong in the first place, so fix the tool as well as the procedures.

4. If the message names a different option

The features in the error message need this full set, not only QUOTED_IDENTIFIER. Setting them together at the top of a maintenance script means you won't meet the next one:

SET ANSI_NULLS ON;
SET ANSI_PADDING ON;
SET ANSI_WARNINGS ON;
SET ARITHABORT ON;
SET CONCAT_NULL_YIELDS_NULL ON;
SET QUOTED_IDENTIFIER ON;
SET NUMERIC_ROUNDABORT OFF;

What not to do

Don't drop the filtered index to make the error go away. On an Identity table it is the unique constraint that stops two accounts from sharing a username; removing it trades a clear error for silent duplicate users. The same reasoning as any unique index - it is a rule the database enforces for you, which I wrote about in database indexes and idempotency.

Where this came from

The reason I was updating Identity rows by hand at all is worth a sentence, because it is the common route into this error: a deploy had deleted the uploads folder on the server, and the fastest repair for one broken image was a direct UPDATE rather than going through the admin screens. Direct SQL against tables an ORM normally manages is exactly when you leave the client library's defaults behind - the same way EF Core's migration history can surprise you, as in "There is already an object named X in the database". The application had been setting QUOTED_IDENTIFIER ON for me all along; the command line wasn't.

FAQ

What does 'SET options have incorrect settings: QUOTED_IDENTIFIER' mean?

It is SQL Server error 1934. The statement touches a feature that requires QUOTED_IDENTIFIER ON - a filtered index, an indexed view, an index on a computed column, an XML data type method, query notifications or a spatial index - and the current session has it OFF. The word before 'failed' (INSERT, UPDATE, DELETE, SELECT, CREATE INDEX, ALTER INDEX) is just the statement that hit it.

How do I fix Msg 1934 in sqlcmd?

Add the -I switch, which turns QUOTED_IDENTIFIER on for the classic ODBC sqlcmd, or put SET QUOTED_IDENTIFIER ON at the top of the script before the failing statement. The newer Go-based sqlcmd always has quoted identifiers enabled and ignores -I.

Why does my application work but the same SQL fails in sqlcmd?

Each client chooses its own session defaults. ADO.NET, Entity Framework Core, the ODBC driver, OLE DB and SSMS connect with QUOTED_IDENTIFIER ON, while the classic ODBC-based sqlcmd starts with it OFF unless you pass -I. Same database, different session settings.

Why does a stored procedure still fail after I SET QUOTED_IDENTIFIER ON?

Stored procedures, triggers and functions keep the QUOTED_IDENTIFIER and ANSI_NULLS settings that were in force when they were created, and a SET inside the body has no effect. Find them with sys.sql_modules WHERE uses_quoted_identifier = 0 and re-create each one from a session that has the option on.

What is the difference between SET QUOTED_IDENTIFIER ON and OFF?

With ON (the default for most clients) double quotes delimit identifiers such as "Order" and strings must use single quotes. With OFF, double quotes delimit string literals instead. Features like filtered indexes and indexed views require ON, which is why OFF causes error 1934 on those tables.

Does ASP.NET Core Identity use filtered indexes?

Yes. On SQL Server, Identity's unique indexes on nullable columns are filtered with IS NOT NULL - UserNameIndex on AspNetUsers and RoleNameIndex on AspNetRoles. Any INSERT, UPDATE or DELETE on those tables from a session with QUOTED_IDENTIFIER OFF fails with Msg 1934.

Should I drop the filtered index to get rid of the error?

No. The filtered unique index is what stops two users sharing a username. Fix the session setting instead - SET QUOTED_IDENTIFIER ON, sqlcmd -I, or re-create the procedure - and the index keeps doing its job.

Originally published on knowledgemarkg.com.

PHP2Shell Explained: How AI Found a PHP Bug Hiding Since 2012, and How It Was Fixed

Kausar Raza — Thu, 22 Oct 2026 16:41:56 +0000

On 21 October 2026 PHP shipped an emergency release for two bugs in the language itself: CVE-2026-109618 and CVE-2026-109619, together nicknamed PHP2Shell. Both are rated 9.2 (Critical) and both can give an attacker remote code execution with no login - on a server whose application code has no bug at all. Wiz, one of the teams that reported them, says every PHP version released in the last decade is affected by at least one of the two, going back to PHP 5.4 from 2012.

The other reason this one made the evening news is how it was found. Three separate reports reached the PHP security team in August and September, and frontier AI models are credited with a large share of the finding, the exploiting and the checking. This article covers the bug itself, how it was found, how it was fixed, what you need to do, and whether AI is the real danger here.

If you only read one line: update to PHP 8.5.12, 8.4.27, 8.3.36 or 8.2.35 (or your distribution's patched package), then restart every PHP process - Apache, PHP-FPM, queue workers. PHP's own advisory says there is no complete workaround: "updating is the only safe choice."

What PHP2Shell is, in plain English

A PHP server does not start a fresh process for every visitor. It keeps a pool of worker processes alive and hands each new request to one of them. Every request gets its own memory, and PHP is supposed to clean that memory up completely before the worker takes the next request.

Both bugs are failures of that clean-up, triggered at moments PHP did not expect to be interrupted:

CVE-2026-109618 - "RCE via OOM corruption". If a request runs out of memory (hits memory_limit) at the wrong moment, PHP abandons it while some internal data is only half set up. The clean-up code then reads that half-built data as if it were valid.
CVE-2026-109619 - "RCE via stale global state". If a fatal error or an out-of-memory error interrupts a request at the wrong point, some of PHP's global state keeps pointing at memory that belonged to that request. When the same worker picks up the next request, it can use that freed memory before it is reset - a classic use-after-free, across two requests.

Corrupted or reused memory is the raw material for remote code execution: an attacker who can shape what ends up in that memory can steer the process. That is what "2Shell" in the name means - from an HTTP request to a shell on your server.

Why your application does not need a bug

This is the part that makes PHP2Shell unusual. Most PHP security news is about a plugin, a framework or somebody's code. These two bugs are in the interpreter, and the dangerous paths run before your code is reached:

Multipart parsing happens during request startup. When a request arrives with a file-upload style body, PHP parses it to fill $_POST and $_FILES before your script starts. The advisory for CVE-2026-109618 describes three ways an attacker can push that parsing into an out-of-memory error while staying inside limits that look safe: a body with no usable Content-Length that keeps streaming, a part with an unlimited number of headers, and empty file fields that do not count towards max_file_uploads but still allocate memory.
Session upload progress runs without session_start(). The session.upload_progress feature is enabled by default and does its work during upload parsing. The CVE-2026-109619 advisory names it as one of the confirmed paths.
Settings applied per request can trip it. On Apache with mod_php, .htaccess and php_value directives are applied before the request is fully initialised. A warning raised there can read the stale state from the previous request - and the advisory notes that hiding warnings with error_reporting, @ or display_errors does not help.

Wiz says it got remote code execution against the stock WordPress, Drupal and Joomla container images and against a plain "Hello World" page. It also showed a second outcome that needs no code execution at all: leaking a worker's memory back to the attacker inside an oversized PHPSESSID cookie. Worker memory can hold database passwords, API keys and other visitors' data.

Which setups are most exposed

Apache with mod_php is the worst case. It runs request shutdown even when startup failed, so a single unauthenticated request can trigger CVE-2026-109618 before any application code runs. PHP says full RCE was demonstrated here with default settings.
PHP-FPM is less exposed to the startup path, because a child that fails to start exits. The CVE-2026-109618 advisory says no complete FPM exploit was demonstrated; but the PHP Foundation confirms RCE on FPM through application-level code paths, and the CVE-2026-109619 advisory says the session-upload-progress RCE was validated against both mod_php and FPM.
FrankenPHP and LiteSpeed (LSAPI) are judged likely susceptible from source review, not validated.

How it was found

The timeline below comes from the PHP Foundation's post and Wiz's write-up, which agree on the dates they share.

2024: during PHP's first external security audit (by Quarkslab, organised by OSTIF and funded by Germany's Sovereign Tech Agency), a PHP core developer spotted the first signs of the out-of-memory problem while fixing a different report. It was not fixed at the time; reporting since disclosure says the team could not agree on a fix and nobody had shown it was exploitable.
30 July 2026: security firm Calif starts an audit of PHP under OpenAI's "Patch the Planet" programme.
11 August: Calif's Ryan Austin reports an unconditional remote code execution with the out-of-memory root cause. Coverage citing the Wall Street Journal and New York Times says he used OpenAI's GPT-5.6 Sol.
19-24 August: Wiz Research, working independently, finds its first PHP bug on 19 August, confirms unauthenticated RCE against Drupal the next day and WordPress the day after, and reports to PHP on 24 August. Wiz says it used its own Wiz Atlas tool together with Google's Gemini 4 Argon and GPT-5.6 Sol, and that the work was done by a single researcher in about a week.
18 September: OpenAI reports further attack vectors.
21 October: patched releases and public disclosure.

What the AI actually did matters more than the headline. In Wiz's account the models contributed to finding the bugs, writing exploits, adapting those exploits to different PHP versions, configurations and memory layouts, detecting attacks and validating the patch - analysing their own failures and trying again. Humans chained the bugs together and did the per-target debugging. On the receiving end, the PHP security team also uses AI now: to validate incoming reports and group the ones with the same root cause. Their post is explicit that "a person makes every call and works on every patch."

If you want to see what this kind of AI-driven bug hunting looks like in practice - a model driving a disassembler or a debugger on its own - I wrote up the tooling in AI reverse engineering: Ghidra MCP, IDA MCP and REA.

How the disclosure went wrong

Coordinated disclosure means the fix and the announcement land at the same moment, so defenders and attackers start level. PHP2Shell did not get that. In UTC, per the PHP Foundation, Patchstack and the reporting that followed:

17 October: Amazon's MadPot honeypots recorded an exploitation attempt from an anonymising network - four days before disclosure. Amazon says it is unclear how the attacker knew, and advises investigating from at least 17 October, not from the release date.
20 October, ~15:00: the CVE IDs leaked on forums and social media, with a wrong release time and a wrong CVSS score of 10.0. No technical details leaked.
20 October, 19:30: PHP published its pre-announcement early and brought the release forward a day, to 21 October at 14:00.
20 October, ~20:00: Calif mistakenly published a video of a working proof of concept, which the PHP Foundation says accelerated attackers' own proof-of-concept work.
21 October, 07:41: container vendor Chainguard published patched images more than six hours before the coordinated release - letting anyone compare old and new code.
21 October, 07:43: Cloudflare's CTO said it was seeing early attacks and had deployed WAF rules.
21 October, ~13:34-14:00: advisories and releases went out.

Some operators chose to go dark rather than wait. Reported examples include the Dutch government portal Overheid.nl, Finnish public-facing government services, French Interior Ministry websites, parts of Wikimedia's supporting infrastructure, and the Internet Archive briefly.

Is it being exploited? The honest answer on 22 October is "reports conflict". CISA said on 21 October it had not observed exploitation. watchTowr said its honeypots saw exploitation by the same day. Patchstack reported that hosting partners had reported active incidents that morning, while its own telemetry after release showed probing rather than confirmed breaches. Plan as if exploitation is happening.

How it was fixed

The fixes go after the root cause rather than the individual attack paths:

Out-of-memory no longer jumps out of the middle of anything. PHP used a longjmp() to abandon a request the instant an allocation failed - wherever it happened to be. The patch removes that. Hitting the limit now raises a delayed "soft" out-of-memory error at the next safe point in the virtual machine, where no half-built state is in flight, plus an immediate hard exit of the process (_exit()) at a higher threshold. Either way, nothing reads half-initialised data afterwards.
Request state is reset before request memory is freed. The CVE-2026-109619 fix makes sure error, session, object and extension state that points into a request's memory is cleared before that memory is released, so the next request on the worker has nothing stale to find.

Fixed versions are 8.5.12, 8.4.27, 8.3.36 and 8.2.35. PHP 8.1 and older are end of life and get no binaries, but because of the severity the PHP team published minimal source backports for 7.4, 8.0 and 8.1 for people who build from source. Linux distributions and long-term-support vendors received the patches in advance, so your distribution's package may already be fixed under its own version number.

What to do now

1. Find out which PHP actually serves your site

php -v                 # the command-line PHP - NOT necessarily what your web server runs
php-fpm8.3 -v          # each FPM version installed (Debian/Ubuntu naming)

That first command catches people out. The php on your shell path and the PHP behind your website are often different builds, and many servers have several versions installed side by side. The reliable check is from the web side: a phpinfo() page you delete straight afterwards, or the version shown in WordPress → Tools → Site Health → Info → Server. On cPanel, MultiPHP Manager lists the version per domain; Plesk shows it in each domain's PHP settings. Update every installed version, not just the one your site uses.

2. Update, then restart everything

A new package on disk does nothing while old workers are still running the old code in memory. Restart Apache, every PHP-FPM pool and any long-running PHP process (queue workers, Laravel Octane, RoadRunner, Swoole). Rebuild and redeploy container images - a running container does not patch itself.

3. If you genuinely cannot update today

These come from PHP's own announcement. They block the known paths and nothing more; PHP is clear that they do not make a server safe:

; php.ini, FPM pool or server config - not ini_set(), not .user.ini
memory_limit = -1                    ; removes the controllable OOM trigger
post_max_size = 8M                   ; 32M only if you need uploads
upload_max_filesize = 6M             ; example - keep it below post_max_size
session.upload_progress.enabled = 0  ; blocks the confirmed session paths
fatal_error_backtraces = 0           ; PHP 8.5 only

Apply them where PHP reads them before the request starts, and restart every worker. Put request-size limits in front of PHP too, at the web server or proxy, because the limits have to cover bodies with no usable Content-Length. And note the warning in the advisory: do not use RLIMIT_AS or RLIMIT_DATA as a substitute memory cap - they can cause exactly the allocation failure you are trying to avoid. If the server is high-value and none of this is possible, CISA's advice is to take it offline until it is patched.

4. Look back, starting from 17 October

The signs Wiz, Patchstack and CISA list: unusual multipart requests followed by memory-limit errors, bursts of HTTP 500s, oversized or malformed PHPSESSID cookies, unexplained PHP worker restarts, and - the one that matters most - a PHP process starting a shell or downloading tools. None of these proves anything on its own. If you see them, patch anyway, treat the server as possibly compromised, and rotate the secrets it holds, because a memory leak does not leave a broken page behind.

5. "My site isn't PHP" - check the server anyway

This site runs on ASP.NET Core and Next.js, so PHP2Shell does not touch its code. That is exactly the trap for anyone on shared or panel-managed hosting. A Plesk or cPanel server usually has PHP installed whether or not your site uses it - for webmail such as Roundcube, for phpMyAdmin, and for the other customers on the same machine. Ask your host which PHP builds are on your server and when they were patched. If you are deciding between a managed CMS and something custom partly on maintenance grounds, this is the kind of event that belongs in that decision - see WordPress vs a custom website.

Is AI the danger here?

Every headline asked it, so here is my reading of the evidence - and it is a reading, not a fact.

AI did not create these bugs. They were written by people, some of them over a decade ago, and a person had already seen the first sign of one in 2024. What stopped anyone fixing it was not a lack of intelligence; it was disagreement, missing proof of exploitability and too few people. The PHP Foundation says so itself: the delay "was a structural failure, caused by both the security team's decision process and its lack of resources."

What AI changed is the cost. Wiz's account is one researcher, about a week, three of the biggest PHP applications exploited, with the models porting the exploit across versions. Turning "this looks like it might be exploitable" into a working attack used to be the expensive part. Here it was the fast part. The same economics apply to attackers - and they apply to the gap between a patch and an exploit. Security firm Sansec predicted "mass attacks to start within 15 minutes of the release"; Chainguard's early images handed out the patch diff hours ahead. When a model can read a diff and work out the bug, publishing a fix is publishing a map.

The same tools are the defence. All three reports came from defenders using AI, and PHP's own team uses AI to cope with the volume - it says it has handled more than 100 invalid, duplicate or unreproducible reports since July alongside 15 real CVEs. The capability is symmetrical. What is not symmetrical is funding: The Guardian pointed out that the foundation maintaining a language behind a huge share of the web had 13 full- and part-time staff, while the AI work credited with the discovery sits on billions in investment.

So no, AI is not the threat on its own. The threat is the time between "a fix exists" and "your server runs it", and AI is shrinking every other clock in this story. Updates that used to be fine within a week now need to happen within hours, which mostly means automatic security updates, an inventory of everything that runs PHP, and a host that tells you when it has patched. If your process today is "someone remembers to check", PHP2Shell is the reason to change it.

The short version

Two memory-safety bugs in PHP's request clean-up, present since 2012, let an unauthenticated request corrupt or reuse worker memory and take over the process - before your code runs. AI-assisted research found and weaponised them in days; a leaky disclosure gave attackers a head start; PHP fixed the root cause by no longer abandoning requests mid-operation and by resetting state properly. Update to 8.5.12, 8.4.27, 8.3.36 or 8.2.35, restart every worker, and check for signs of trouble back to 17 October.

FAQ

What is PHP2Shell?

PHP2Shell is the nickname for two critical bugs in the PHP interpreter, CVE-2026-109618 and CVE-2026-109619, disclosed and patched on 21 October 2026. Both are rated CVSS 9.2 and can allow unauthenticated remote code execution on a PHP server, even when the application itself has no bug.

Which PHP versions are affected by PHP2Shell?

All versions from PHP 5.4 through 8.5.11, according to Wiz and Google. The fixed releases are 8.5.12, 8.4.27, 8.3.36 and 8.2.35. PHP 8.1 and older are end of life; the PHP team published source backports for 7.4, 8.0 and 8.1, and some Linux distributions and LTS vendors ship their own patched packages.

Is my WordPress site affected by PHP2Shell?

If it runs on an unpatched PHP version, yes - the bug is in PHP, not WordPress, and Wiz demonstrated remote code execution against the stock WordPress image. Check the PHP version under Tools > Site Health > Info > Server, and ask your host to confirm it is 8.2.35, 8.3.36, 8.4.27, 8.5.12 or a patched distribution package.

How was the PHP2Shell vulnerability found?

Three teams reported it independently: Calif (11 August 2026, during an audit under OpenAI's Patch the Planet programme), Wiz Research (24 August) and OpenAI (18 September). Wiz says it used Gemini 4 Argon, GPT-5.6 Sol and its own Wiz Atlas tool, and that AI helped with discovery, exploit development and patch validation. A PHP developer had seen the first sign of the out-of-memory problem in a 2024 audit.

How do I mitigate PHP2Shell if I cannot update?

PHP lists partial measures: memory_limit=-1 set where applications cannot override it, a low post_max_size, session.upload_progress.enabled=0, and fatal_error_backtraces=0 on PHP 8.5, all set in php.ini or server config and followed by a restart of every worker. PHP says these block known paths only and that updating is the only safe choice.

Did AI create the PHP2Shell vulnerability?

No. The bugs were in human-written code dating back to PHP 5.4 in 2012. AI models were used by the researchers who found them, to locate the bugs, build and adapt exploits and validate the fix. The concern is that the same capability lowers the cost of exploitation for attackers and shortens the time between a patch and an attack.

Is PHP2Shell being exploited in the wild?

Reports conflict. Amazon recorded an exploitation attempt on its honeypots on 17 October, four days before disclosure; Cloudflare and watchTowr reported early attacks on 21 October; CISA said it had not observed exploitation as of 21 October. Treat it as actively exploited and check logs back to 17 October.

Originally published on knowledgemarkg.com.

DeepSeek Harness (dsh): What It Is, How to Install It, and What Is Actually Inside

Kausar Raza — Tue, 20 Oct 2026 16:39:56 +0000

DeepSeek Harness, or dsh, is DeepSeek's open-source agent harness: the software that wraps a language model and turns it into an agent that can read files, run commands, search the web and work on a task over many steps. Its design idea is unusual - everything is a plugin, including the agent loop itself.

I installed it on Windows, ran its command-line tool, and dumped the configuration of its web profile to see what it is really made of. This guide covers what an agent harness is, how to install dsh, what is inside it, which models it can use, how its permission modes work, and the safety warnings DeepSeek itself attaches to it.

What is an agent harness?

A language model on its own only produces text. An agent harness is everything around the model that lets it act:

The agent loop - send the conversation to the model, run the tools it asks for, feed the results back, repeat until the task is done.
Tools - read and write files, run shell commands, search, fetch web pages, ask the user a question.
Context management - deciding what goes into each request, and compacting old history when the conversation gets too long.
Safety - sandboxing and approval prompts before risky actions.
A user interface - a terminal, a web app, a desktop app or an API.

Claude Code, OpenAI's Codex CLI, OpenCode and Hermes are all harnesses. The model is the engine; the harness is the rest of the car. Two harnesses running the same model can behave very differently, which is why "harness engineering" has become a topic of its own.

What makes DeepSeek Harness different

Most harnesses have a fixed core and an extension system bolted on. dsh is built on Cordis, a plugin framework in which plugins contribute services, typed events and reversible effects to a shared context. DeepSeek's architecture document says there is "no privileged core to patch": the model adapter, the tool registry, the session log and the agent loop are all plugins, so each one can be swapped through configuration.

In practice that means:

A profile is a named stack of plugin bundles. The shipped profiles are web, headless, sdk, sdk-minimal and acp, and the CLI help also shows a tui example.
A patch file (cordis.patch.yml) overrides or adds plugin rows without forking anything.
The session log is the source of truth: every model-visible input is logged as JSONL, so resume, fork and transcripts are all derived from one record.
Capabilities sit behind seams. Point the filesystem and subprocess providers at a remote sandbox, and Bash, the terminal and language-server tools all move with them.

How to install and run it

You need Node.js. The quickest start is one command:

npx @deepseek-ai/dsh web

That boots the web profile and serves the Web UI at http://127.0.0.1:3080 (add --no-open to stop it opening a browser). Then go to Settings → Models, add an API key, choose a workspace folder, and start a session. The agent cannot do anything until a workspace is selected.

Other ways to run it, from the CLI's own help:

dsh headless "run the tests"              # answer one task, print the result, exit
dsh --profile sdk                         # JSON-RPC server for the Python SDK
dsh --profile acp                         # automation clients over ACP
dsh --profile web --dump-config           # print the composed plugin tree and exit
dsh plugin --profile tui add 
# install a plugin into a profile ```

 On my machine, `npx -y @deepseek-ai/dsh --version` reported **0.2.1-rc.2** and ran fine on Node.js 22.18.0. There is also an official Electron desktop app in the repository, and a crop of unofficial "DeepSeek Harness desktop" wrappers on GitHub - check that you are installing from `deepseek-ai/deepseek-harness` or the `@deepseek-ai/dsh` npm package, because several projects reuse the name.

## What is inside: 193 plugins

Running `dsh --profile web --dump-default-config` prints the full plugin tree without booting anything. On version 0.2.1-rc.2 the web profile is composed of **193 separate `@deepseek-ai/dsh-*` plugin packages**. Grouped by job:

- **Agent core:** `dsh-agent-loop`, `dsh-system-prompt`, `dsh-agent-instructions`, `dsh-agent-preset`, `dsh-plan-mode`, `dsh-persona`, `dsh-goal`.
- **Tools:** filesystem and file search, Bash and PowerShell (each with a persistent variant), web fetch and search, to-do lists, background jobs, scheduling, ask-the-user, skills and a workflow tool.
- **Subagents:** spawn and fork in-process subagents, and control them.
- **Context management:** basic compaction, a tool-result pruner, image offload and a token meter - the parts that keep long sessions from overflowing.
- **Safety:** a local sandbox, a sandbox policy, separate Bash and PowerShell sandboxes, a filesystem sandbox, user approval and permission presets.
- **Models:** a DeepSeek API-key adapter, a DeepSeek account adapter, a retry layer and an adapter called `dsh-llm-pi-ai` for other providers.
- **Sessions and storage:** JSONL persistence, SQLite queries, checkpoints, export, titles and telemetry (OpenTelemetry).
- **MCP:** a `dsh-mcp-resources` plugin is part of the profile.
- **The Web UI itself:** chat, file browser, terminal, document preview, plan view, settings pages - each one a plugin too.

That granularity is the point. If you want a different compaction strategy or a remote sandbox, you replace one row in the patch file rather than forking the project.

## Which models can it use?

DeepSeek's models are built in: **Settings → Models** has a DeepSeek card with a single API-key field, and DeepSeek's models offer "off", "low", "high" and "max" reasoning levels. It is not DeepSeek-only, though. According to the provider guide you can add:

- **Third-party providers** from a catalogue, including ids such as `anthropic`, `openai`, `moonshotai` (Kimi) and `zai` (GLM).
- **Custom endpoints** - a company gateway or a self-hosted server - speaking one of three protocols: OpenAI Chat Completions, OpenAI Responses or Anthropic Messages.

Keys are stored in `$DSH_HOME/.credentials.yaml`, and provider settings live in the profile's `cordis.patch.yml`.

## Permission modes

The dumped configuration shows three permission presets:

| Preset | Sandbox | Approval |
|---|---|---|
| `read-only` | read-only | ask |
| `workspace-write` (default) | workspace-write | ask |
| `danger-full-access` | full access | never |

The default comes from the `DSH_PERMISSION_MODE` environment variable and falls back to `workspace-write`, which means the agent can change files in your chosen workspace but asks before operations that need approval. Only `danger-full-access` turns approvals off - and the name is an honest warning.

## The safety notice: read it before you run it

DeepSeek describes dsh as experimental developer-preview software that has not had a security audit, and warns that there will be compatibility-breaking changes. Its safety notice is blunt: the harness can run model-generated commands, modify or delete files it can reach, and access the network and any credentials made available to it, and third-party plugins may be untrusted. Sandboxing and approvals reduce the risk but "cannot protect resources that the project is allowed to access".

The recommendations are the same ones I would give for any agent harness:

1. Run it in a disposable virtual machine or container for anything important.
2. Give it the narrowest workspace and the fewest credentials possible - never your production keys.
3. Keep backups of anything it can touch.
4. Review plugins and proposed commands before approving them.

## DeepSeek Harness vs Claude Code and OpenCode

People search for these comparisons, so here is what can be said honestly without running a benchmark:

- **Claude Code** is Anthropic's harness, tuned for Claude models, mature and terminal-first, with hooks, skills and MCP support. dsh is younger, open source under MIT, web-first, and designed around replacing any part of itself.
- **OpenCode** is an open-source, provider-agnostic terminal harness. dsh is also provider-agnostic, but its plugin model goes further: even the loop and the session log are swappable.
- **Maturity:** dsh is a developer preview (0.2.x release candidates at the time of writing). For day-to-day client work I would still reach for a mature harness; dsh is the one to study if you want to understand, or build, an agent harness.

I have not measured task success between them, so treat anyone's "X beats Y" chart with care unless they publish the tasks, the model and the settings.

## Who should try it?

- **Developers building their own agents** - the plugin tree is a working reference architecture.
- **Teams that need a self-hosted agent** pointed at their own model gateway.
- **DeepSeek API users** who want an official interface beyond the chat app.

If you only want to run DeepSeek's models locally without an agent, my guide to [running DeepSeek R1 locally on Mac, Windows and Linux](https://knowledgemarkg.com/blog/how-to-run-deepseek-r1-locally-for-free-on-macwindowslinux) is the simpler place to start, and [DeepSeek vs ChatGPT](https://knowledgemarkg.com/blog/deepseek-ai-vs-chatgpt-which-ai-best-fits-your-needs-2025) covers the models themselves. For the other big agent story of 2026 - agents driving disassemblers - see [AI reverse engineering with Ghidra and IDA](https://knowledgemarkg.com/blog/ai-reverse-engineering-ghidra-ida-mcp).

## FAQ

**What is DeepSeek Harness?**

DeepSeek Harness (dsh) is DeepSeek's open-source agent harness: the software around a language model that lets it read and edit files, run commands, search and work on multi-step tasks. It is built on the Cordis plugin framework, so every part - including the agent loop - is a plugin.

**How do I install DeepSeek Harness?**

With Node.js installed, run npx @deepseek-ai/dsh web. It serves the Web UI at http://127.0.0.1:3080. Add an API key in Settings, choose a workspace folder, and start a session.

**What is an agent harness?**

An agent harness is everything around a model that turns it into an agent: the loop that runs tools and feeds results back, the tools themselves, context management, sandboxing and approvals, and the user interface. Claude Code, Codex CLI, OpenCode and DeepSeek Harness are all harnesses.

**Does DeepSeek Harness only work with DeepSeek models?**

No. DeepSeek models are built in, but you can add third-party providers from its catalogue, including Anthropic, OpenAI, Kimi and GLM, or any custom endpoint that speaks the OpenAI Chat Completions, OpenAI Responses or Anthropic Messages protocol.

**Is DeepSeek Harness safe to use?**

Its own safety notice calls it experimental, unaudited developer-preview software that can run commands and modify files it can reach. The default mode asks before risky operations, but you should run it in a VM or container, give it minimal credentials, and keep backups.

**Is DeepSeek Harness free and open source?**

Yes, the harness is open source under the MIT licence. You pay only for the model API you connect, such as DeepSeek's API, unless you point it at a self-hosted model.

**What are the DeepSeek Harness permission modes?**

There are three presets: read-only and workspace-write, which both ask for approval before sensitive operations, and danger-full-access, which turns approvals off. The default is workspace-write, set by the DSH_PERMISSION_MODE environment variable.

---

_Originally published on [knowledgemarkg.com](https://knowledgemarkg.com/blog/deepseek-harness-dsh-guide)._

AI Reverse Engineering in 2026: How LLM Agents Drive Ghidra and IDA (and Where They Fail)

Kausar Raza — Tue, 20 Oct 2026 16:39:17 +0000

A year ago, "AI reverse engineering" meant pasting a decompiled function into a chat window and asking what it did. In 2026 it means something different: an agent such as Claude, Codex or a local model drives the disassembler itself - it lists functions, follows cross-references, renames variables and reads pseudocode, one tool call at a time, through the Model Context Protocol (MCP).

This guide explains how that works, which tools exist for Ghidra, IDA Pro, Binary Ninja and Android, what a real session looks like, what the first serious benchmark found, and the limits you will hit. It also covers the part most tutorials skip: what you are allowed to reverse engineer in the UK, the EU and the US.

What "AI reverse engineering" actually means now

Reverse engineering is recovering how a program works without its source code. The slow part has always been the reading: thousands of unnamed functions like FUN_00401a30, registers instead of variables, and no comments. Language models are good at exactly that kind of pattern reading - they recognise a CRC loop, a string decoder or a licence check far faster than a person scrolling through assembly.

What changed is tool access. An MCP server exposes the disassembler's functions - "decompile this address", "list cross-references", "rename this symbol", "search strings" - as tools the model can call. The model is no longer guessing from a snippet you chose; it navigates the binary itself and comes back with findings and the evidence behind them.

The usual stack has four parts:

The agent - Claude Code, Claude Desktop, Codex, Cursor or another MCP client.
An MCP server - the bridge that turns tool calls into disassembler actions.
The analysis engine - Ghidra, IDA Pro, Binary Ninja, radare2, jadx or Frida.
The target - a native binary, an APK, an Electron app, a firmware image or a network capture.

The main tools, by engine

A community index of AI reverse engineering projects, awesome-ai-reverse-engineering, lists 122 repositories; 74 of them are MCP servers or plugins for disassemblers. The ones you will meet first:

Tool	Engine	What it gives the agent
GhidraMCP (LaurieWired)	Ghidra (free)	Decompile, list functions and strings, rename symbols in a running Ghidra
IDA Pro MCP (mrexodia)	IDA Pro 8.3+	A broad tool set over IDA's analysis; IDA Free is not supported
Hex-Rays IDA MCP (official)	Recent IDA versions	Vendor-maintained server; newer and smaller community
Binary Ninja MCP	Binary Ninja	Plugin with an embedded server; a headless variant exists
jadx MCP	jadx (Android)	Decompiled Java/Kotlin classes, manifest, references
Frida MCP	Frida	Runtime hooks on a live process - observe what the code really does
radare2 MCP	radare2	Command-line engine, scriptable and free
REA (morluto/rea)	Hopper, Ghidra or IDA, plus built-in parsers	One MCP for native binaries, JavaScript/Electron, .NET, APKs, websites and captures

If you only want to try the idea, Ghidra plus GhidraMCP is the zero-cost route. If you already pay for IDA, the IDA servers are more capable. If your targets vary - an Electron app this week, a .NET DLL next week - a multi-engine MCP like REA saves you wiring up five separate servers.

A real session: patching lives in River Raid

The clearest public write-up so far is Quesma's River Raid case study. The author loaded the 8 KB Atari 8-bit ROM into Ghidra as raw 6502 code, connected Claude through GhidraMCP, and asked for infinite lives. What happened is a good summary of the whole field:

It found the right code. Claude identified the Atari 8-bit platform from hardware register addresses and located the routine that loads, decrements and stores the lives counter.
It knew the fix but could not apply it. The patch was replacing DEY ($88) with NOP ($EA) at offset 0x355. The MCP server had no write access, so the human applied it with dd. In the emulator, the lives stayed at 3.
It needed help with setup. Ghidra loaded the ROM at $0000 instead of $A000, so every cross-reference pointed nowhere. Claude diagnosed the problem correctly but could not rebase the program itself.
It was confidently wrong once. It guessed the game was Centipede. The author's conclusion is worth remembering: confidence and accuracy are unrelated.
It was slow. Some tool calls took around 30 seconds - fine for a batch task, painful interactively.

That pattern - excellent at reading and locating, weak at broad context and anything that needs write access - matches what most people report.

What the first benchmark says

CrackMeBench (2026) tests agents on symbol-poor CrackMe binaries built from C, Rust and Go templates. Each agent gets a shell in a network-less Docker sandbox with standard reverse engineering tools, a five-minute budget per task and three scored submissions. The program itself is the judge: the agent must produce a serial or key generator the binary accepts.

Model	Generated tasks (12)	Harder half (6)	Public CrackMes (8)
GPT-5.5	11/12	5/6	3/8
Claude Opus 4.7	7/12	2/6	2/8
Kimi K2	5/12	1/6	1/8

Two things stand out. Results drop sharply on the harder and the real-world public tasks, and the gap between models is large. Treat any single leaderboard as a snapshot, though - the tasks are small, the budget is five minutes, and models change every few months.

Setting it up: the part that takes longest

Every tool follows the same shape: install the engine, install its plugin or extension, start the MCP server, then register that server with your agent. In Claude Code that last step is claude mcp add; in Claude Desktop it is the MCP section of the config file. Follow each project's README for the exact commands - they change between releases, and the Quesma author counted four separate points of failure in the chain (client, MCP server, Ghidra extension, Ghidra).

REA tries to collapse that into one command:

npx rea-agents setup --dry-run     # print the plan, change nothing
npx rea-agents setup               # register the MCP server with your agents

I tried exactly that on Windows with Node.js 22.18.0 to inspect my own website's build output. The dry run planned nothing and returned status: needs_human with the remediation "Install Node.js 22.x (>=22.19), 24.x (>=24.11), or 26+". Two lessons: check the runtime requirement before anything else, and always run the dry run first - a setup tool that edits your agent's configuration should show you its plan before it writes a byte.

Once a server is registered, start small. Good first prompts are "list the exported functions and summarise what each module does" and "find where this string is used and explain the check around it". Ask for addresses and evidence with every claim, so you can verify instead of trusting.

Where AI reverse engineering is genuinely useful

Triage. Getting a first map of an unfamiliar binary - which functions matter, where the network code is, what the strings suggest - in minutes rather than an afternoon.
Naming and annotation. Proposing names for hundreds of functions and variables, which a human then reviews.
Explaining known patterns. Crypto routines, compression loops, custom string encoding, licence checks.
Your own software. Understanding a legacy binary whose source is lost, checking what a build really contains, or finding why a minified JavaScript bundle is so large. As a web developer, that last one is where I would use it most: the same tools that read malware can read your own production bundles.

The limits you will hit

Confident mistakes. Always ask for the address and the instructions behind a claim.
Write access. Many servers are deliberately read-only. Patching, rebasing and retyping often stay manual.
Speed and cost. Large binaries mean many tool calls; each one costs time and tokens.
Obfuscation and packing. Packed or virtualised code defeats the model just as it defeats static analysis.
Prompt injection. A binary can contain strings written to manipulate the agent reading it. Researchers have published attacks on exactly these setups. Never give a reverse engineering agent your real credentials or an unrestricted shell, and analyse untrusted samples in a disposable virtual machine.

Is it legal? UK, EU and US basics

This is general information, not legal advice - get proper advice before reverse engineering anything commercial.

UK: the Copyright, Designs and Patents Act 1988 (section 50B) allows decompiling a program to achieve interoperability with another program, under conditions, and section 50BA allows observing and testing a program's behaviour if you have the right to use it.
EU: the Software Directive (2009/24/EC, Article 6) gives a similar, narrow interoperability exception.
US: reverse engineering for interoperability has been accepted as fair use in several cases, but the DMCA's anti-circumvention rules (section 1201) apply when protection measures are involved, with specific exemptions such as good-faith security research.

Licence terms matter too: many end-user licences forbid reverse engineering, and the AI provider's own usage policy applies to what you ask the model to do. Analysing your own code, open-source software, CTF challenges or malware samples for defence is the safe ground.

Where to start

Install Ghidra and pick a target you are allowed to analyse - a CrackMe or a CTF binary is ideal.
Connect GhidraMCP to Claude Code or Claude Desktop and ask for a function map with addresses.
Verify three of the agent's claims by hand before trusting the fourth.
When your targets diversify, look at a multi-engine MCP such as REA - after checking its Node.js requirement and running the dry run.

The agent-harness side of this story is moving just as fast. If you want to see how an open agent framework is put together from the inside, read my teardown of DeepSeek Harness. And if you are curious what this kind of bundle inspection finds on a real website, here is how I fixed unused JavaScript and LCP render delay in Next.js.

FAQ

What is AI reverse engineering?

It is using a language model to analyse compiled software. In 2026 that usually means an agent such as Claude driving a disassembler like Ghidra or IDA through an MCP server - listing functions, following cross-references and reading decompiled code - and reporting findings with the addresses behind them.

What is Ghidra MCP?

GhidraMCP is an open-source Model Context Protocol server that connects an AI agent to a running Ghidra instance, so the agent can decompile functions, search strings and rename symbols itself instead of working from snippets you paste in.

Can Claude reverse engineer software?

Yes, when it is connected to a disassembler through MCP. It is strong at recognising patterns and locating the code that matters, weaker at broad context, and many MCP servers are read-only, so patching often stays manual. Always ask for evidence and verify its claims.

Is Ghidra MCP or IDA Pro MCP better?

Ghidra with GhidraMCP is free and simple, which makes it the best way to start. IDA Pro MCP offers a broader tool set but needs a paid IDA Pro licence (IDA Free is not supported). There is no published head-to-head benchmark of the two.

What is REA (Reverse Engineer Anything)?

REA is an open-source MCP and CLI that gives an agent one set of reverse engineering tools across native binaries, JavaScript and Electron apps, .NET assemblies, Android APKs, websites and network captures. It uses Hopper, Ghidra or IDA for deep native analysis. Setup is npx rea-agents setup, which needs Node.js 22.19 or newer.

How accurate are AI agents at reverse engineering?

In the CrackMeBench benchmark, GPT-5.5 solved 11 of 12 generated tasks, Claude Opus 4.7 solved 7 and Kimi K2 solved 5, but every model scored far lower on harder and real-world tasks. Treat results as a snapshot and verify the agent's work.

Is it legal to reverse engineer software with AI?

It depends on what and why. The UK, EU and US all allow some reverse engineering for interoperability and research, but licence terms and anti-circumvention laws can restrict it. Analysing your own code, open-source software, CTF challenges or malware for defence is safe ground; get legal advice for anything commercial.

Originally published on knowledgemarkg.com.

There Is Already an Object Named 'X' in the Database: What EF Core Is Really Telling You

Kausar Raza — Wed, 14 Oct 2026 16:10:55 +0000

You run dotnet ef database update against a database that already has a schema,
and it stops with this:

Msg 2714, Level 16, State 1, Line 5
There is already an object named 'BlogPosts' in the database.

The instinct is to go and look at the migration. That is the wrong place. The migration is
almost certainly fine. What has gone wrong is that EF Core's record of what it has already
applied no longer matches the migration files sitting in your project, so it is trying to
run something for the second time.

This happened on this site's own database, and the evidence is still in the table as I write
this. I will show you the real rows further down.

The same bug wears three different faces

Which error message you get depends entirely on what the migration being re-run happens to do.
That is why this problem is so hard to search for - three people hit the identical bug and go
looking for three unrelated strings.

I captured all three from a live SQL Server 2022 rather than quoting them from memory, using a
temporary table so nothing real was touched:

Migration operation	Error	Message
CreateTable	2714	There is already an object named 'X' in the database.
AddColumn	2705	Column names in each table must be unique. Column name 'ResumeUrl' in table 'SiteSettings' is specified more than once.
CreateIndex	1913	The operation failed because an index or statistics with name 'IX_BlogPosts_Slug' already exists on table 'BlogPosts'.

One quirk worth knowing if you ever try to reproduce this: 2714 is raised when the batch is
compiled, not when it runs, so a TRY...CATCH in the same batch will not catch it. The
other two will. It is a small detail, but it is the kind that makes you think your error handling
is broken when it is working exactly as designed.

What __EFMigrationsHistory actually is

EF Core has no magic here. Its entire memory of your database is one small table that it
creates itself:

SELECT MigrationId, ProductVersion FROM __EFMigrationsHistory ORDER BY MigrationId;

Two columns. MigrationId is the filename of the migration without the extension -
a 14-digit timestamp, an underscore, and the class name. ProductVersion is the EF
version of the tooling that applied it.

When you run database update, EF reads every MigrationId out of that
table, compares the list against the migration classes compiled into your assembly, and runs
whatever is in the assembly but not in the table. That is the whole algorithm.

Which means the error is not really "this object exists". It is "I have no row saying I created
this object, so I am going to create it".

The real cause: your file and your history row drifted apart

Here is the actual history table behind this site, trimmed to the interesting part:

MigrationId                                    ProductVersion
---------------------------------------------  --------------
20260719093058_AddLogoDarkUrlToSiteSettings     9.0.0
20260920154152_AddResumeToSiteSettings          9.0.9
20260920154332_AddResumeToSiteSettings          9.0.9
20260922181332_AddLeadFieldsToContactMessages   9.0.9

Look at the middle two. The same migration name, twice, with timestamps one minute and forty
seconds apart. There is only one file on disk -
20260920154332_AddResumeToSiteSettings.cs. The 154152 row is a ghost.

What happened is mundane and extremely common: the migration was generated, applied, then
something about it was wrong, so it was deleted and generated again. The regenerated file got a
fresh timestamp, because the timestamp is simply the moment migrations add ran. As
far as EF is concerned that is a completely different migration which has never been applied - and
its Up method said AddColumn("ResumeUrl") against a table that already
had the column.

That is error 2705, and this is the version of the story where it was caught. Run the same
mistake on a migration that creates tables instead and you get 2714.

Other routes to the same state, all of which I have run into:

The database was built by hand or restored from a backup, and the first migration tries to create what is already there.
Two branches each added a migration, both were applied in different environments, and the merge left one environment's history out of step.
A migration was applied, then reverted in source control but never reverted in the database.
Someone ran database update against the wrong connection string. On shared hosting, with staging and production on one instance, this is easier than it sounds.

Why dotnet ef migrations list will not show you this

This is the part that cost the most time, so I will be blunt about it. The obvious diagnostic
does not work.

dotnet ef migrations list --project MyApp.Core --startup-project MyApp.Api

On the database above, that command printed eighteen migrations. The ghost row was not one of
them. It cannot be: the command builds its list from the migration classes in your assembly and
then annotates which of them are pending. A history row with no matching class is not in the
assembly, so there is nothing to annotate and it never appears.

So migrations list answers "which of my files have been applied". It does not
answer "what does the database think it has applied", and the second question is the one you need.
You have to go and ask the table yourself.

The diagnosis, in one query

Read the history table and compare it with your files. Nothing here writes anything, so it is
safe to run against production:

SELECT MigrationId, ProductVersion
FROM   __EFMigrationsHistory
ORDER  BY MigrationId;

Then list the files:

ls Migrations/*.cs | grep -v Designer | grep -v Snapshot

Three outcomes, and each one has a different fix:

A file with no row. EF will try to apply it on the next update. If the schema it creates is already there, that is your error.
A row with no file. Harmless on its own - EF ignores rows it does not recognise - but it is a loud signal that a migration was regenerated, and the replacement is probably case 1.
Duplicate names with different timestamps. The two above, together. This is the regenerated-migration case and it is the most common one by a distance.

Check ProductVersion while you are in there, too. Mixed values across rows mean
different machines with different EF tooling versions have been applying migrations to the same
database, which is its own source of surprises.

Fixing it without dropping the database

The fix depends on which of two things is true, and you have to decide which before you type
anything.

If the schema is already correct and only the history row is missing - the
usual case - you do not want to run the migration at all. You want to tell EF it has already been
run. Insert the row yourself, idempotently, so re-running the script is harmless:

IF NOT EXISTS (SELECT 1 FROM __EFMigrationsHistory
               WHERE MigrationId = N'20260920154332_AddResumeToSiteSettings')
BEGIN
    INSERT INTO __EFMigrationsHistory (MigrationId, ProductVersion)
    VALUES (N'20260920154332_AddResumeToSiteSettings', N'9.0.9');
END

The MigrationId must match the filename exactly, timestamp included.
ProductVersion should match the EF version you are running; it is informational, but
getting it wrong makes the table lie to the next person who reads it.

If the history row exists but the schema does not - someone restored an older
backup over a newer history, say - then deleting the row is what lets EF apply the migration
properly. Take a backup first. This is the direction in which guessing loses data.

If you have an orphan row and no corresponding file, leaving it alone is a
legitimate choice. EF will not trip over it. The 154152 row is still in this site's
database for exactly that reason: it is harmless, and it documents what happened. If it bothers
you, delete it - but delete it because you have confirmed no file matches it, not because it looks
untidy.

To generate the SQL without applying it, and read what EF intends to do first:

dotnet ef migrations script 20260719093058_AddLogoDarkUrlToSiteSettings 20260920154332_AddResumeToSiteSettings --idempotent --output check.sql

--idempotent wraps every operation in a check against the history table, which
makes the script safe to run more than once. On a production database it is the only form worth
running.

Three fixes to not reach for

dotnet ef database drop ends the error by destroying the data. It is the right
answer on a local development database you can reseed in a minute, and the wrong answer everywhere
else. The two commands sit next to each other in the documentation, which is how accidents
happen.

Deleting the Migrations folder and starting again feels like a reset. It is not -
the history table survives, so a fresh InitialCreate immediately collides with the
existing schema and you are back at 2714 with less information than before.

Adding --ignore-changes, or hand-emptying the Up method to make the
error go away, is the one that causes real damage later. You end up with a migration that claims
to do something and does nothing, and the next environment you deploy to is missing the schema
with no error to tell you.

Keeping it from happening again

One rule covers most of it: never regenerate a migration that has already been applied
anywhere. Once a migration has run against a database you cannot reach and fix by hand,
its timestamp is a fact about the world. If the migration is wrong, add a new one that corrects
it. migrations remove is only safe while the migration is still unapplied
everywhere.

Three more habits that have earned their place:

Script the deployment with migrations script --idempotent rather than running database update against production. You get to read it first, and it is safe to run twice if a deploy is interrupted.
Check __EFMigrationsHistory as part of a release, not as part of an incident. The query above takes two seconds.
Be deliberate about which connection string is active. On shared hosting, where staging and production often sit on the same instance, the only thing between you and the wrong database is which appsettings file happened to load.

If you are working on the same stack, the post
on indexes and idempotency covers why idempotent scripts matter well beyond migrations, and
the ASP.NET Core deployment errors post
covers the failures that show up immediately after this one, when the application starts but the
database will not cooperate.

The short version: when EF says an object already exists, it is not describing your schema. It
is telling you it has lost track of its own history. Read the table, find the disagreement, and
fix the record rather than the migration.

FAQ

Can I just delete the migration file and create a new one?

Only if the migration has never been applied to any database. Once it has run somewhere you cannot reach and correct by hand, regenerating it creates a second migration with a new timestamp that EF treats as unapplied - which is the exact cause of this error. Add a corrective migration instead.

Is it safe to insert rows into __EFMigrationsHistory manually?

Yes, when you have confirmed the schema that migration creates is already present. You are recording a fact that is already true. Use an IF NOT EXISTS guard so the script can be run twice, and make MigrationId match the filename character for character, including the timestamp.

Why does dotnet ef migrations list not show the extra row?

Because it builds its list from the migration classes compiled into your assembly and then marks which ones are pending. A history row with no matching class is not in the assembly, so it never appears in the output. You have to query __EFMigrationsHistory directly to see it.

What is the difference between error 2714, 2705 and 1913?

Nothing, as far as the cause goes - they are the same re-applied migration hitting different operations. 2714 comes from CreateTable, 2705 from AddColumn and 1913 from CreateIndex. Searching the message text sends three people with one problem to three unrelated places.

Does --idempotent fix this error?

It prevents it rather than fixing it. An idempotent script wraps every operation in a check against the history table, so operations EF has already recorded are skipped. It will not repair history that has already drifted - do that first, then use idempotent scripts so it does not drift again.

My database was created by hand and now the first migration fails. What now?

That is case 1: the schema exists, the history row does not. Generate the script for InitialCreate and read it against your actual schema. If they match, insert the history row and skip the migration. If they differ, reconcile the differences in a new migration before recording anything as applied.

Should I run dotnet ef database update against production?

Prefer migrations script --idempotent and apply the SQL deliberately. You get to read exactly what will run, the script is safe to re-run if a deploy is interrupted, and you are not relying on a build machine holding a production connection string.

Originally published on knowledgemarkg.com.

Database Indexes and Idempotency: What They Fix, and How to Get Them Wrong

Kausar Raza — Wed, 14 Oct 2026 15:54:19 +0000

Two things quietly decide whether a small application survives being used: whether its reads are indexed, and whether its writes can be repeated safely. They are usually taught as separate subjects - one belongs to databases, the other to API design. They are the same subject seen twice. An index is how reads survive scale. Idempotency is how writes survive retries.

Every example below is from the application that serves this page, including a failure that happened while it was being written.

Updated 22 October 2026: the idempotency sections now cover concurrent duplicates - two identical requests arriving at the same moment - and why a UNIQUE constraint, not a check in your code, is what actually stops them. They also cover what no database transaction can protect: a card charge or an email sent from inside it. Thanks to Kashif Manzer for pointing out the race in the comments below.

Part one: what an index actually is

The mental model that gets people into trouble is "an index makes the table faster". It does not. An index is a second, ordered copy of some of your columns, with a pointer back to the row. Adding one gives the query planner a new route to the data. It does not change the table.

Two consequences follow immediately, and both matter more than the speedup:

Every index is paid for on write. An insert into a table with four nonclustered indexes writes to five structures, not one. An index nobody reads is a pure tax.
An index only helps if the query can use it. Wrapping the indexed column in a function - WHERE YEAR(PublishedAt) = 2026 - throws the ordering away and the planner scans anyway. So does a leading wildcard in a LIKE.

The problem an index solves, concretely

This site lists blog posts. The query behind that list is, in effect:

SELECT ... FROM BlogPosts
WHERE IsPublished = 1 AND PublishedAt IS NOT NULL
ORDER BY PublishedAt DESC;

Without a useful index the database reads every row to find the published ones, then sorts the survivors. At a hundred rows nobody notices. At fifty thousand you are reading fifty thousand rows and sorting most of them to return ten.

With the right index the database walks a structure that is already filtered and already in the right order, takes the first ten entries, and stops.

Composite indexes, and why column order is the whole game

These three indexes shipped with this application:

IX_BlogPosts_IsPublished_PublishedAt             (IsPublished, PublishedAt)
IX_BlogPosts_IsPublished_IsFeatured_PublishedAt  (IsPublished, IsFeatured, PublishedAt)
IX_BlogPosts_AuthorId_IsPublished                (AuthorId, IsPublished)

Look at the first one. The order is not arbitrary and it is not alphabetical. A composite index is sorted by its first column, then within that by its second - like a phone book sorted by surname, then first name.

(IsPublished, PublishedAt) puts all the published rows together, and inside that group they are already in date order. The query above filters on the first column and sorts on the second, so the database seeks to the published section and reads straight down it. No sort step at all.

Reverse it to (PublishedAt, IsPublished) and the index loses the property that mattered. The rows are now ordered by date first, with published and unpublished rows interleaved, so there is no contiguous block of published rows to seek to. The database can still read the index newest-first and throw drafts away as it goes - cheap while almost every row is published, expensive once most of them are drafts or scheduled posts, because it has to wade through all of them to collect ten survivors. The first arrangement costs the same however many drafts exist.

Same two columns. Same table. One arrangement seeks straight to the answer; the other has to read past everything that is not the answer. The rule worth memorising: columns you filter by with equality come first, the column you sort or range-scan on comes last.

Why the third index exists separately

(AuthorId, IsPublished) could look redundant next to the others. It is not, because of the leftmost-prefix rule: an index on (A, B) can serve queries filtering on A, or on A and B - but it cannot seek on B alone. The author page filters by author first, so it needs an index that leads with AuthorId. No amount of indexes leading with IsPublished will help it.

When not to add one

Indexes are not free and the instinct to add one per column is how tables get slow at writing without getting faster at reading. Skip them on small tables, on a column with very few distinct values on its own (a bit column is a fine leading column of a composite and a poor index by itself), and on anything you have not actually seen in a slow query. Add them from evidence - an execution plan, a slow query log - rather than from intuition.

This table is its own example. Alongside the composites it still carries the single-column IX_BlogPosts_IsPublished from the first migration. By the leftmost-prefix rule, (IsPublished, PublishedAt) can already answer anything that index can, so it is now a write cost with no read benefit - a candidate to drop once the query plans confirm nothing still chooses it.

The other job an index does: a unique index is a rule

Everything above is about indexes as a route to data. A UNIQUE index is also a rule the database enforces on every write: no two rows may share that value, however and whenever they arrive. That second job is the one Part two depends on.

Part two: idempotency, and the problem it actually solves

An operation is idempotent when doing it twice leaves the same result as doing it once. Reading is naturally idempotent. Setting a value is idempotent. Appending is not. Charging a card is emphatically not.

The problem it solves is narrower and more common than "duplicate clicks". It is this: a network gives you no way to tell a request that failed from a response that was lost. Your client sends a request, waits, and gets a timeout. Did the server do the work? You cannot know from the client. And the moment you retry - and HTTP clients, load balancers and message queues all retry - you are betting the outcome on that unanswerable question.

Idempotency removes the question. If the operation is safe to repeat, the retry is safe. You stop needing to know whether the first attempt landed.

The pattern, in SQL

The simplest form is to make the write describe the destination rather than the change. This is not idempotent:

UPDATE Posts SET ViewCount = ViewCount + 1 WHERE Id = 42;   -- runs twice, counts twice

This is:

UPDATE Posts SET ViewCount = 981 WHERE Id = 42;             -- runs twice, same result

When you genuinely need an increment, the idempotency has to come from somewhere else - usually a record of which events you have already counted.

For inserts, the equivalent is to check before writing. The script that published this post runs as an upsert, so running it a second time edits the post rather than creating a duplicate:

IF EXISTS (SELECT 1 FROM BlogPosts WHERE Slug = @slug)
    UPDATE BlogPosts SET Title = @title, Content = @content WHERE Slug = @slug;
ELSE
    INSERT INTO BlogPosts (Title, Slug, Content) VALUES (@title, @slug, @content);

That is a small habit with a large payoff: any script written this way can be re-run after a failure without anyone having to work out how far it got.

But it is only safe when it runs once at a time. Run it twice concurrently and both copies can execute the IF EXISTS, both see nothing, and both insert. Under SQL Server's default READ COMMITTED isolation the check and the insert are two separate steps, and nothing stops another session slipping in between them. For a script you run by hand that is rarely a problem. For the same pattern behind an API endpoint, it is exactly what a double submit does.

Two fixes, and the first one is not optional:

-- 1. Make the database refuse the duplicate. This is the guarantee.
CREATE UNIQUE INDEX UX_BlogPosts_Slug ON BlogPosts (Slug);

-- 2. Optionally, serialise the check so the second caller waits instead of failing.
BEGIN TRAN;
IF EXISTS (SELECT 1 FROM BlogPosts WITH (UPDLOCK, HOLDLOCK) WHERE Slug = @slug)
    UPDATE BlogPosts SET Title = @title, Content = @content WHERE Slug = @slug;
ELSE
    INSERT INTO BlogPosts (Title, Slug, Content) VALUES (@title, @slug, @content);
COMMIT;

UPDLOCK, HOLDLOCK makes the second session wait at the IF EXISTS until the first commits, and then see the row. It is a good habit, but it is a habit - forget the hint once and nothing warns you. The unique index cannot be forgotten: the worst case becomes a clean error instead of a silent duplicate.

On this site the guarantee was there before the script was written. BlogPosts.Slug has carried a unique index, IX_BlogPosts_Slug, since the first migration, so two overlapping runs of the publishing script cannot produce two posts - the second stops with error 2601. The script itself has no lock hint, and has never needed one.

Idempotency keys, and when you need one

Checking before writing works when the row has something naturally unique - a slug, an email, an order reference. When it does not, the client supplies the uniqueness instead: a key generated per intended operation, sent with the request, and stored by the server alongside the result. Stripe popularised the pattern with its Idempotency-Key header, and it is the right answer for anything that moves money or sends a message.

The obvious implementation is: look the key up; if it has been seen, return the stored response; if not, do the work and record the key. That has the same race as the upsert. Two requests with the same key - a double click, or a client retrying while the first attempt is still running - can both look the key up, both find nothing, and both do the work. Every request carried the right key, and the work still happened twice.

The fix is to stop asking the question in code and let the database answer it. Give the key a uniqueness constraint, and make claiming the key the first write:

CREATE TABLE IdempotencyKeys (
    ClientId        int            NOT NULL,   -- whose key it is
    IdempotencyKey  nvarchar(100)  NOT NULL,   -- as sent by the client
    RequestHash     binary(32)     NOT NULL,   -- SHA-256 of the request body
    StatusCode      int            NULL,       -- NULL while the work is still running
    ResponseBody    nvarchar(max)  NULL,
    CreatedAt       datetime2      NOT NULL,
    CompletedAt     datetime2      NULL,
    CONSTRAINT PK_IdempotencyKeys PRIMARY KEY (ClientId, IdempotencyKey)
);

BEGIN TRAN;

-- Claim the key first. A second request with the same key cannot get past this line.
INSERT INTO IdempotencyKeys (ClientId, IdempotencyKey, RequestHash, CreatedAt)
VALUES (@clientId, @key, @requestHash, SYSUTCDATETIME());

-- ... do the database work here: insert the order, reserve the stock ...

UPDATE IdempotencyKeys
SET StatusCode = @status, ResponseBody = @body, CompletedAt = SYSUTCDATETIME()
WHERE ClientId = @clientId AND IdempotencyKey = @key;

COMMIT;

Now the database arbitrates. When two requests race, the second INSERT waits on the first one's uncommitted row and then either fails with a duplicate-key error (2627 for a primary key or unique constraint, 2601 for a unique index) or, if the first rolled back, succeeds and does the work itself. There is no window in which both can believe the key is new.

Two columns in that table are there for reasons that are easy to miss:

The key is unique per client, not globally. Keys come from clients, and two clients can send the same one. With a global key, the second client gets the first client's stored response - someone else's order, returned as theirs.
The request hash catches a reused key. The same key with a different body is a client bug, not a retry. Replaying the old response would hide it; reject it with a 4xx instead.

Two ways to hold the claim

One transaction, for quick database-only work. This is the code above: claim, work and response commit together. A crash anywhere rolls all of it back, claim included, so a retry starts clean. A concurrent duplicate simply waits on the insert and gets the duplicate-key error once the first commits. The cost is that the transaction stays open for the whole operation, so keep it to work that takes milliseconds.

Claim, then work, then complete, for long-running work. Commit the claim in a short transaction of its own, do the work, then record the response in a second one. Locks stay short, but two new situations appear. A repeat can find a claimed row with CompletedAt still NULL: answer it with a 409 and a Retry-After, never by doing the work again. And a crash between claim and completion leaves a row that says "in progress" forever, so give the claim an expiry time and let a later request take over an expired claim - after checking whether the work actually happened.

What no database transaction protects

Charging a card is an HTTP call to a payment provider, and rolling back your transaction does not refund it. If the process dies after the charge succeeds and before your commit, the claim is rolled back with everything else, and the retry charges again. The database did its job; the duplicate happened outside it.

The fix is to carry the idempotency across that boundary. Payment providers take an idempotency key of their own - Stripe accepts one on every POST - so pass one derived from yours, and the provider refuses the second charge the same way your table refuses the second claim. For emails and webhooks, write the message to an outbox table inside the transaction and send it from there afterwards, so "this message should be sent" commits atomically with the work.

The remaining details are smaller but each one breaks the pattern if skipped:

The duplicate-key error is not an error. Catch 2627 / 2601, read the stored row, and return its response. Surfacing it as a 500 turns a retry that already succeeded into an apparent failure.
Return the stored response, not a bare 200. Otherwise the caller cannot tell what happened.
Keys expire. Keep them long enough to cover your clients' retry window - Stripe keeps them for at least 24 hours - then delete them in batches.

The check in code is still useful - it answers a repeat quickly without opening a transaction. But it is an optimisation. The constraint is the guarantee.

Where the two ideas meet: EF Core's migration history

Entity Framework's __EFMigrationsHistory table is an idempotency ledger, though nobody calls it that. It exists so that running dotnet ef database update twice does not apply the same migration twice. Each applied migration's ID is the key, and it is the table's primary key, so the ledger can never hold the same migration twice.

That protects the ledger, not the schema. Two deploys racing could both run the same ALTER TABLE before either writes its history row. EF Core 9 closes that gap separately: Migrate() now takes a database-wide lock before it starts, so only one process migrates at a time. The key stops the bookkeeping duplicating; the lock stops the work duplicating. Same two halves as the idempotency table above.

Which makes its failure mode instructive - and it happened on this project while this article was being drafted. A migration had been applied under one generated ID. Later the migration file was regenerated and picked up a new timestamp in its name. The change was already in the database; the ledger held the old key. EF compared files to history, saw an ID it had no record of, and tried to apply it again:

Column names in each table must be unique.
Column name 'ResumeUpdatedAt' in table 'SiteSettings' is specified more than once.

The schema was correct. Only the bookkeeping was wrong. The fix is to record the key that should have been there - itself written idempotently, so re-running it is harmless:

IF NOT EXISTS (SELECT 1 FROM __EFMigrationsHistory
               WHERE MigrationId = '20260920154332_AddResumeToSiteSettings')
    INSERT INTO __EFMigrationsHistory (MigrationId, ProductVersion)
    VALUES ('20260920154332_AddResumeToSiteSettings', '9.0.0');

Here the check-then-insert race is harmless: MigrationId is the primary key, so a concurrent second run would fail with a duplicate-key error rather than write a second row.

The lesson generalises past EF. An idempotency mechanism is only as good as the stability of its key. Change how the key is derived and the ledger stops protecting you - silently, until something tries to run twice. The mirror-image failure, where EF tries to create something the database already has, is covered in "There is already an object named X in the database": what EF Core is really telling you.

What to take from this

Indexes answer a question about reading: can the database find these rows without looking at all of them? Get the column order right and most of the work disappears; get it backwards and the database reads past rows it was supposed to skip, while you still pay for the index on every insert.

Idempotency answers a question about writing: if this runs twice, is the result the same? Make it so - including for the side effects that leave your database - and the entire category of bug that begins "the request timed out and then everything was duplicated" stops existing.

And the two meet in one place: a unique index. It is how the database finds a key quickly, and it is the only thing that keeps "have I seen this key?" true when two requests ask at the same moment. A check in your code is a hope; a constraint is a guarantee.

Neither is advanced. Both are the difference between an application that works in testing and one that works in production.

FAQ

What is a database index, in simple terms?

A second, ordered copy of some of your columns with pointers back to the rows. It gives the query planner a faster route to the data without changing the table itself. The cost is that every insert and update has to maintain it as well.

Why does column order matter in a composite index?

A composite index is sorted by its first column, then by its second within that - like a phone book by surname then first name. An index on (IsPublished, PublishedAt) lets a query seek straight to the published rows and read them already in date order. Reverse the columns and the database can no longer seek; it has to read past every unpublished row to find the published ones.

When should I not add an index?

On small tables, on a column with very few distinct values on its own, on an index whose columns are the leading columns of another index, and on anything you have not seen in a slow query. Every index is paid for on every write, so one nobody reads is a pure tax. Add them from an execution plan, not from intuition.

What does idempotent mean?

Doing the operation twice leaves the same result as doing it once. Setting a value is idempotent; incrementing one is not. It matters because a network cannot tell you whether a request failed or the response was simply lost.

What problem do idempotency keys actually solve?

They remove the need to know whether a timed-out request landed. The client sends a key with the operation; the server stores it with the result. A repeat of the same key returns the stored response instead of doing the work again. For work outside your database, such as a card charge, pass a key on to the provider as well, because rolling back your transaction does not undo it.

Is checking for the key before doing the work enough?

No. Two requests with the same key that arrive together can both see 'not seen' before either writes, and both do the work. Put a UNIQUE constraint on the key (scoped per client) and insert the key as the first step of the transaction, so the second request fails on that insert. Treat that duplicate-key error (2627 or 2601 in SQL Server) as 'already done' and return the stored response.

How do I make an SQL script safe to re-run?

Write the destination rather than the change, and check before inserting. An upsert - IF EXISTS then UPDATE, ELSE INSERT - can be run any number of times without creating duplicates, as long as the runs do not overlap. If they can, back it with a unique index on the natural key (and optionally WITH (UPDLOCK, HOLDLOCK) on the check), so concurrent runs cannot both insert.

Originally published on knowledgemarkg.com.

ASP.NET Core Web API Performance: From 900ms to 3ms on Cheap Hosting

Kausar Raza — Wed, 14 Oct 2026 15:50:45 +0000

The ASP.NET Core Web API behind this site runs on a budget setup: an ASP.NET Core 9 app on one box, and SQL Server on shared hosting somewhere else entirely. Every query crosses a network I do not control. A cold read of the homepage payload took somewhere between 200 and 1000 milliseconds, and it was not the query plan that was slow. It was the distance.

Warm reads now come back in 2 to 3 milliseconds. Nothing was rewritten in C++ and nothing moved to Redis. Seven changes did it, and every one of them is available to you on the cheapest hosting you can find.

The constraint worth naming first

Most ASP.NET Core Web API performance advice quietly assumes your database is in the same data centre as your app. On managed cloud that is true. On the shared hosting most small projects actually run on, it is not.

When the database is 200 milliseconds away, the winning move stops being "make the query faster" and becomes "make fewer round trips, and stop repeating the ones you have already made". Almost everything below follows from that one sentence.

1. Project into DTOs. Never load entities

The single biggest change in the whole exercise. Every read path in the Web API goes straight from IQueryable into a DTO, so EF Core emits one flat SELECT with only the columns that will actually be serialised:

public static readonly Expression<Func<BlogPost, PostSummaryDto>> ToSummary =
    p => new PostSummaryDto(
        p.Id,
        p.Title,
        p.Slug,
        p.Excerpt,
        p.PublishedAt,
        p.Categories.Select(c => new TermDto(c.Id, c.Name, c.Slug)).ToList());

What this avoids matters more than what it does:

No Include fan-out. Loading a post with its categories and tags as entities pulls every column of every row, including the full article body you were not going to display in a list.
No change tracking. Projections into a non-entity type skip the tracker entirely - there is nothing to snapshot.
No lazy-load surprises. A missing Include cannot turn into an N+1 later, because there is no navigation to walk.

Storing these as Expression fields rather than methods keeps them composable: they sit in one file, and every endpoint reuses the same shape. When the DTO gains a field, one edit covers the whole API.

2. Pool the context, and default to no tracking

Two lines, applied once, at startup:

builder.Services.AddDbContextPool<ApplicationDbContext>(options =>
{
    options.UseSqlServer(connectionString, sql =>
    {
        sql.EnableRetryOnFailure(maxRetryCount: 3, maxRetryDelay: TimeSpan.FromSeconds(5));
        sql.CommandTimeout(30);
    });
    options.UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking);
}, poolSize: 128);

AddDbContextPool reuses context instances instead of building a new one per request, so the model and DI wiring are not re-resolved thousands of times a minute. NoTracking as the default is safe precisely because of the previous section - nothing here needs a change tracker, and the handful of write paths ask for tracking explicitly with AsTracking().

EnableRetryOnFailure is not optional on shared hosting. Remote database connections drop. Without retries that is a 500 for the visitor; with them it is a hiccup nobody sees.

3. Output caching, with tags

This is where the 3 milliseconds come from. Response caching is usually skipped because of the obvious objection: if I cache for five minutes, my edits take five minutes to appear. Tag-based eviction removes that objection completely.

options.AddPolicy("posts-policy", p => p
    .Expire(TimeSpan.FromMinutes(5))
    .Tag("posts")
    .SetVaryByQuery("page", "pageSize", "category", "tag", "search", "sort"));

Then have every write throw away exactly the tags it touched:

await cacheStore.EvictByTagAsync("posts", ct);
await cacheStore.EvictByTagAsync("taxonomy", ct);

The result is a cache with no staleness cost. Publishing a post evicts the post tags and the change is live immediately, while a tag nobody edited keeps serving from memory. A warm read never touches the database at all, which is why the network distance stops mattering.

Get SetVaryByQuery right or the cache becomes a bug. Every query parameter that changes the response has to be listed. Miss one - an author filter, say - and two different requests share one cached body, which is a data leak dressed up as a performance win.

4. One aggregated endpoint instead of five round trips

The homepage needs featured posts, latest posts, popular posts, categories and tags. Five endpoints is the tidy design, and on a remote database it is also five times the latency.

GET /api/v1/site/home returns all five in one response. The client makes one request, the server makes one trip. This is not clever, it is just counting - and on a slow link, counting round trips beats micro-optimising any one of them.

5. Never make a reader wait on a write

Every article view should increment a counter. Doing that inline means the reader waits for a write to a database on the other side of the country, in order to read something that was already rendered.

Instead the view goes onto a bounded channel and a background worker batches them:

private readonly Channel<PostViewEvent> _channel =
    Channel.CreateBounded<PostViewEvent>(new BoundedChannelOptions(10_000)
    {
        FullMode = BoundedChannelFullMode.DropWrite,
        SingleReader = true,
    });

A hosted service drains it every five seconds, up to 200 events at a time, and writes one batch. Two details are deliberate:

Bounded, not unbounded. An unbounded channel under a traffic spike is a memory leak with a queue in front of it.
DropWrite when full. If the choice is between losing a few analytics rows and making real visitors wait, the analytics lose. Say that out loud in a comment so the next person does not "fix" it.

6. The split-query decision, and why the default was wrong here

Post projections pull two collections - categories and tags. EF Core warns about this and suggests AsSplitQuery(), which avoids the cartesian join by issuing separate queries.

On this setup that advice is backwards. A split query trades duplicated rows for extra round trips, and round trips are the expensive thing here. With a handful of terms per post, the duplicated rows cost less than another 200 millisecond hop. So the behaviour is set explicitly:

sql.UseQuerySplittingBehavior(QuerySplittingBehavior.SingleQuery);

Stating it explicitly also silences the warning honestly - the setting is a decision, not an oversight. If the database were local, or if a post had hundreds of tags, the right answer would flip. Measure your own link before copying either choice.

7. Compression, and a cache in front of the cache

Brotli and Gzip on JSON responses, at the fastest compression level rather than the smallest:

builder.Services.Configure<BrotliCompressionProviderOptions>(o =>
    o.Level = CompressionLevel.Fastest);

Maximum compression spends CPU to save bytes that were not the bottleneck. Fastest gets most of the size reduction for a fraction of the cost.

In front of all of it, the Next.js frontend keeps its own data cache, so a page that has not changed does not even ask the API. Two cache layers sound excessive until you notice they fail independently - if one is cold, the other still absorbs the hit.

What I did not bother with

Honesty is more useful than a longer list:

Redis. A second service to run, pay for and monitor, to solve a problem in-memory output caching already solved on one box. Worth it when you scale past one instance. Not before.
Compiled queries. Real, measurable, and completely lost in the noise next to a 200 millisecond network hop.
Hand-written SQL. The projections already generate the query I would have written. Dapper would win microseconds and cost me the migrations.
More indexes, at first. Indexes fix slow queries. My queries were not slow - my network was. Adding indexes before measuring is guessing.

How to find out where your own time goes

Before changing anything, get the number. Turn on EF Core command logging in development and read what it actually emits:

options.LogTo(Console.WriteLine, LogLevel.Information)
       .EnableSensitiveDataLogging();

Then ask three questions in order:

How many queries per request? If a list endpoint issues one query per row, stop here and fix the N+1. Nothing else matters yet.
How wide are they? A SELECT naming forty columns to render a card is a missing projection.
How far away is the database? Time a trivial query - SELECT 1. That figure is the floor under every request you will ever serve, and it decides whether you should be optimising queries or eliminating them.

That last number is the one that reframed this whole exercise. Once I knew the floor was 200 milliseconds, caching stopped being an optimisation and became the architecture.

Worth knowing before you copy any of it

These choices suit a read-heavy Web API on a slow link. A write-heavy internal tool on a local database would want almost the opposite: tracking on, caching off, split queries, real indexes.

The transferable part is not the settings. It is measuring the floor first, then deciding whether your problem is slow work or repeated work. Mine was repeated work, and repeated work is the cheap kind to fix.

If you are deciding where to run any of this, ASP.NET Core does not need Windows hosting - and running SQL Server in a Docker container keeps the whole stack on one cheap Linux box. If you are still assembling the fundamentals, the full stack developer roadmap covers the ground around this.

And if you would rather someone else did it, this is what I do - send me the slow endpoint and I will tell you which of these seven it needs.

FAQ

How do I make an ASP.NET Core Web API faster?

Start by measuring where the time goes. In most content APIs the wins in order are: project queries into DTOs instead of loading entities, enable output caching with tags so repeat reads never reach the database, pool the DbContext with no-tracking as the default, and move writes such as view counters off the request thread.

What is an EF Core projection and why is it faster?

A projection selects straight from IQueryable into a DTO, so EF generates SQL that returns only the columns you use. It avoids Include fan-out, skips change tracking entirely, and removes the chance of an N+1 later because there are no navigations to lazy load.

Does output caching make my API serve stale data?

Not if you use tags. Give each policy a tag, then have every write evict the tags it affects. A published post clears the post tags immediately, so edits appear at once while untouched data keeps serving from memory.

Should I use AsSplitQuery in EF Core?

It depends on where your database is. Split queries avoid a cartesian join but cost extra round trips. If the database is remote and each hop costs 200ms, a single query with some duplicated rows is faster. If the database is local and the collections are large, split queries win. Measure the round trip before choosing.

What does AddDbContextPool actually do?

It reuses DbContext instances between requests instead of constructing a new one each time, so EF does not re-resolve the model and dependency graph on every call. It is a one-line change and is safe as long as you do not keep per-request state on the context itself.

Do I need Redis to cache an ASP.NET Core API?

Not on a single instance. The built-in output cache lives in memory and needs no extra service to run, pay for or monitor. Redis becomes worth it once you scale to more than one instance and need the cache shared between them.

Why is my API slow even though my queries are fast?

Usually network distance. Time a trivial query such as SELECT 1 against your production database - that figure is the floor under every request. If it is 200ms, no amount of query tuning gets you below it, and the answer is to make fewer round trips and cache the ones you have made.

How do I stop page view counting from slowing down requests?

Write it asynchronously. Push the event onto a bounded channel and let a background hosted service flush batches every few seconds. Use a bounded channel that drops writes when full, so a traffic spike costs you analytics rows rather than memory or response time.

Originally published on knowledgemarkg.com.