DEV Community: Umer Amin Durrani

Your PostgreSQL Backup Finished. Can You Actually Restore It?

Umer Amin Durrani — Sat, 07 Nov 2026 08:57:00 +0000

The backup job is green. The file exists. Someone says, “Sorted, bro.”

Lovely. One tiny question before we all go for chai:

Can the application actually use what comes back out of that file?

A completed backup is useful evidence that a backup operation finished. Recovery needs more evidence: that you can access the backup, restore it into a suitable environment, validate the data and bring the service back within acceptable limits.

A charger in your bag is reassuring. Finding out it fits a different phone is a less enjoyable experience at 2% battery.

Here is a small PostgreSQL restore exercise you can adapt. It is a lab walkthrough, not a report of a production incident or a measured recovery benchmark.

What this exercise proves—and what it doesn't

We will create synthetic data, make a logical backup, restore it into a new database, and compare a few expected results.

It does not establish disaster recovery for a whole platform. It does not restore your operating system, object storage, application secrets or external services. A logical dump is one component of a recovery strategy; your needs may call for physical backups, WAL archiving or managed-service recovery features instead.

PostgreSQL documents these distinctions in its backup overview.

1. Start with a disposable environment

Use a non-production PostgreSQL instance and a database role allowed to create databases. The example assumes compatible PostgreSQL client tools are installed and that PGHOST, PGPORT and PGUSER point to that lab instance. Use your normal secure authentication method; keep passwords out of shell history and shared screenshots.

The commands below do not delete an existing database. Keep them in the same Bash session so their variables remain available.

set -euo pipefail
umask 077

: "${PGHOST:?Set PGHOST to your disposable lab server}"
: "${PGUSER:?Set PGUSER to your lab role}"

LAB_DIR=$(mktemp -d)
LAB_SUFFIX="$(date +%s)_$$"
SOURCE_DB="restore_source_${LAB_SUFFIX}"
RESTORE_DB="restore_target_${LAB_SUFFIX}"

createdb --template=template0 "$SOURCE_DB"

Check the host before running this. “I thought that terminal was staging” is a sentence with surprisingly expensive punctuation.

See the createdb documentation for connection and permission details.

2. Create a small dataset with an actual relationship

psql --dbname="$SOURCE_DB" --set=ON_ERROR_STOP=1 <<'SQL'
CREATE TABLE customers (
  id integer PRIMARY KEY,
  name text NOT NULL
);

CREATE TABLE orders (
  id integer PRIMARY KEY,
  customer_id integer NOT NULL REFERENCES customers(id),
  amount_paisa integer NOT NULL CHECK (amount_paisa >= 0)
);

INSERT INTO customers VALUES
  (1, 'Lab customer A'),
  (2, 'Lab customer B');

INSERT INTO orders VALUES
  (101, 1, 1000),
  (102, 1, 2200),
  (103, 2, 1000);
SQL

This gives us something more useful than checking that a database name exists: rows, a foreign-key relationship and a numeric total.

3. Create and inspect the archive

pg_dump \
  --format=custom \
  --file="$LAB_DIR/application.dump" \
  "$SOURCE_DB"

pg_restore --list "$LAB_DIR/application.dump" \
  > "$LAB_DIR/archive-contents.txt"

The first command produces an archive. The second shows its table of contents. Listing the archive is not a successful restore test. It is an intermediate inspection step.

pg_dump handles one database; global objects such as roles require separate planning. Keep client/server version compatibility in mind. PostgreSQL pg_dump reference

4. Restore into a fresh target

createdb --template=template0 "$RESTORE_DB"

pg_restore \
  --dbname="$RESTORE_DB" \
  --single-transaction \
  --exit-on-error \
  --no-owner \
  --no-privileges \
  "$LAB_DIR/application.dump"

For this small lab, one transaction helps avoid leaving a partially restored result if a restore command fails. Larger restores may need a different strategy; transaction scope, resource use and parallelism involve trade-offs.

The owner and privilege options deliberately simplify this lab. They also mean we have not tested restoration of the production permission model. Do not silently carry that assumption into a real recovery plan.

Only restore trusted archives. A database dump can contain executable database definitions. PostgreSQL pg_restore reference

5. Verify useful properties

psql --dbname="$RESTORE_DB" --set=ON_ERROR_STOP=1 <<'SQL'
SELECT count(*) AS customers FROM customers;
SELECT count(*) AS orders FROM orders;
SELECT sum(amount_paisa) AS total_paisa FROM orders;

SELECT count(*) AS orphan_orders
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id
WHERE c.id IS NULL;
SQL

For this synthetic dataset, the expected results are 2 customers, 3 orders, 4,200 paisa and 0 orphan orders. These values were reproduced during draft verification in an isolated PostgreSQL 18 cluster using synthetic data. They are not results from a production recovery test.

Counts and totals are useful checks, but they do not prove every value is correct. For a real system, add domain-specific checks: representative records, critical relationships, known transactions and application workflows.

Then test through the application using its normal restricted role. A restore that works only for an administrator has not yet demonstrated that your service can operate.

The Technical PM part: define “recovered” before the incident

Ask the team to agree on:

Which customer journeys must work before recovery is declared complete?
How much data loss is acceptable? This informs the recovery point objective, or RPO.
How long can the service remain unavailable? This informs the recovery time objective, or RTO.
Who can access backups and keys if the usual operator is unavailable?
Who owns the decision to resume writes?

Measure retrieval, provisioning, restore, validation and application readiness separately. A stopwatch around pg_restore alone does not measure the full recovery experience.

Keep a dated record with the backup identifier, environment, tool versions, checks, timings, failures and follow-up owner. Redact sensitive information before sharing it.

The target isn't “a backup exists.” The target is a recovery process your team has evidence for.

What would you check first after a database restore: a row count, an application login, or the customer journey that makes the business money—and why?

Draft prepared with AI support. The synthetic-data walkthrough was checked in an isolated PostgreSQL 18 cluster during draft preparation. Author review and environment-specific testing are still required before publication.

#PostgreSQL #Linux #DevOps #Beginners

What Evidence Makes You Trust a Backup?

Umer Amin Durrani — Sat, 07 Nov 2026 08:56:18 +0000

You know the friend who says, “I'm five minutes away,” while still looking for their keys?

Sometimes a backup dashboard has that same energy.

“Completed successfully” sounds like the whole journey. Depending on the job, it may only describe one stage.

That doesn't make the dashboard useless. It means we should ask what its green tick actually measures.

Four different kinds of confidence

Here is a discussion framework, not an industry certification scheme:

Evidence	What it helps establish	What remains open
A recent file exists	An artefact is present	Is it complete, trusted and accessible when needed?
The backup operation succeeded	The operation met its configured success criteria	Can we restore it into the required environment?
A restore exercise passed	That tested restoration worked under those conditions	Can the application and its dependencies operate?
A recovery drill met its targets	The tested service recovery met defined criteria	Will changed infrastructure or a different incident break the assumptions?

The distinction matters because a backup tool, a database administrator and a customer may all mean different things by “working.”

The customer probably isn't interested in a screenshot of the cron job. Fair enough, honestly.

My favourite question for a recovery meeting

Not “Do we have backups?”

Try:

“What is the most recent recovery evidence, and what did that exercise leave untested?”

That question makes room for an honest answer. Maybe the data restored correctly, but nobody tested the application role. Maybe the files are solid, but retrieval depends on one person's account.

Those are useful findings. They are not reasons to roast whoever ran the drill.

A practice session that reveals a missing key is doing its job. Finding it during the outage is the inconvenient sequel.

A small action worth trying

At the next review, choose one important user journey. Write down how you would prove it works after restoration, which dependencies it requires and who can run that check.

No dramatic transformation programme needed. Just make one assumption visible.

For PostgreSQL specifically, the official backup and restore chapter is a useful starting point for understanding the different approaches.

Which recovery assumption has surprised you most: permissions, missing files, unavailable credentials, external integrations—or something completely different? Share the lesson without exposing customer data or internal credentials.

Draft prepared with AI support. The examples are discussion scenarios, not claimed personal incidents.

#Discuss #DevOps #Database #Productivity

Before You Run That AI-Generated Bash Script, Read This

Umer Amin Durrani — Sat, 07 Nov 2026 08:50:46 +0000

The assistant writes a Bash script in twelve seconds.

The script has comments, a progress message and a final line saying “Done!”

Looks professional, mate. Unfortunately, confidence is not a permission boundary.

Before executing it, ask one question:

What can this script change, and how do I know those are the things I intended?

You don't need to distrust every AI-generated line. You need a review process that doesn't depend on the assistant sounding sensible.

1. Translate the request into a bounded task

“Clean up my server” is not a specification.

A more reviewable request would be:

List regular .log files older than 14 days inside one specified lab directory. Do not delete, upload, change ownership or follow symbolic-link directories. Print the candidates for human review.

Now you have observable boundaries. Directory, age, file type and permitted action are explicit.

This is the software equivalent of telling someone which cupboard to tidy instead of saying, “Sort the house out.” Your passport should not become collateral damage.

2. Review four things before the syntax

Target: Does the path resolve to the intended directory? Are remote host and account correct?

Authority: Which user executes the script? Can it write outside the target, access secrets or invoke privileged commands?

Inputs: Can filenames, environment variables or untrusted text alter the command's meaning?

Effects: Does it delete, install, change permissions, send data, stop services or alter databases?

A syntactically valid command can still act on the wrong environment. Shell linting cannot decide your business intent for you.

3. Begin with inspection, not modification

This Bash example only lists candidates. It assumes GNU find on a Linux lab machine:

#!/usr/bin/env bash
set -euo pipefail

TARGET=${1:?Usage: inspect-logs.sh /path/to/lab-logs}

if [[ ! -d "$TARGET" || -L "$TARGET" ]]; then
  printf '%s\n' 'Choose an existing, non-symlink lab directory.' >&2
  exit 1
fi

TARGET=$(realpath -- "$TARGET")

if [[ "$TARGET" == / ]]; then
  printf '%s\n' 'The filesystem root is not an acceptable target.' >&2
  exit 1
fi

printf 'Inspection only. Target: %q\n' "$TARGET"
find -P "$TARGET" -xdev -type f -name '*.log' -mtime +14 -print0 |
  while IFS= read -r -d '' candidate; do
    printf 'Candidate: %q\n' "$candidate"
  done

This is a review exercise, not a containment mechanism. A filesystem can change during inspection; path validation alone does not prevent races. -xdev also intentionally avoids descending into other mounted filesystems. GNU find age tests use their documented rounding rules—“older than 14 days” needs a precise definition if a retention policy depends on it.

There is no deletion step hidden at the end. Inspect first; design the mutation separately if it is actually required.

4. Check syntax and common shell mistakes

bash -n inspect-logs.sh
shellcheck inspect-logs.sh

The first checks Bash syntax without executing the script. ShellCheck can identify many common shell issues. Neither proves the script is safe or appropriate for your environment. ShellCheck project

Use a disposable directory with ordinary files, spaces in filenames, unexpected names, a symbolic link and a file that should not match. Decide the expected candidate list before running the script.

That makes the result something you can compare, rather than something you can merely admire.

5. A prompt is not a security control

An AI assistant may read logs, pages or documents containing instructions you did not author. That makes the distinction between data and instructions important. Prompt wording alone should not be your enforcement layer. OWASP prompt-injection guidance

Put limits in the execution environment: restricted credentials, bounded tools and explicit approval for consequential actions. The model proposes; the surrounding system decides what is permitted.

Also: adding sudo because a command failed is not troubleshooting. It's changing the size of the mistake you can make.

The habit worth keeping

Before execution, explain the script in your own words, identify every write or network operation, and describe how you'll verify the result.

If you cannot do that yet, ask for a smaller script or a read-only version. There is no shame in needing an explanation. The server does not award bravery points.

What is the first thing you inspect in an unfamiliar shell script: paths, privilege, network calls, quoting—or something else?

Draft prepared with AI support. The illustrative script passed Bash syntax checking and a disposable-file fixture during draft preparation. ShellCheck and author environment-specific review remain outstanding.

#Linux #Bash #AI #Security

Which Commands Should an AI Agent Never Run Without Your Approval?

Umer Amin Durrani — Sat, 07 Nov 2026 08:48:26 +0000

“Are you sure?”

Yes, bro. About what, exactly?

A confirmation button isn't much help if the user cannot see the action, destination or likely effect.

As assistants gain tools, the interesting question isn't whether they can run commands. It's which decisions we are comfortable delegating, under which conditions.

Don't classify risk by how short the command looks

A long query might only read a local test table. A tiny command might change a production permission.

Likewise, “read-only” does not always mean harmless: reading a credential file and sending its contents somewhere can expose a secret without changing the original file.

I would start an approval design with these questions:

Which environment is affected?
Is the action reversible through an ordinary, tested process?
Could it expose confidential data?
Does it alter access, billing, availability or persisted records?
Is the target bounded, or can it expand unexpectedly?

A proposed decision table

This is a starting policy to adapt, not a universal permission standard:

Example action	Possible treatment
Read synthetic lab logs	Allow within a scoped tool
Inspect production service health	Allow only through approved, limited access
Prepare a deployment diff	Allow preparation; review before applying
Change production access rules	Require specific human approval
Delete retained customer records	Require the authorised workflow and explicit approval
Send logs to an external service	Review the contents, destination and data-sharing permission

OWASP's AI Agent Security guidance discusses scoped tools and human oversight. The practical challenge is turning those principles into permissions your system actually enforces.

Make approval a decision, not a reflex

A useful request would say:

“Apply this reviewed configuration change to the staging service. It changes these two values. The rollback is this previous configuration.”

That gives someone a decision they can inspect.

Compare it with “Proceed?” after a wall of terminal output. That's less informed approval and more an endurance test.

I would also bind approval to the reviewed action. If the target, payload or permissions change, the old approval should not become a blank cheque.

Where would you draw the boundary for your own team: what can an agent execute independently, and what must always return to a human? A concrete example is more useful than “never trust AI” or “automate everything.”

Draft prepared with AI support. The table is an illustrative policy discussion, not a record of a deployed access-control system.

#AI #Security #DevOps #Discuss

Deployment Rollback Gets Complicated When the Database Has Moved On

Umer Amin Durrani — Sat, 07 Nov 2026 08:47:04 +0000

“Just roll it back.”

A beautiful sentence. Four words. Very relaxing.

Then someone asks whether the previous application still understands the new database schema, and suddenly everyone's tea is going cold.

Rolling back an application release is not necessarily the same as restoring the whole system to an earlier state. Writes, migrations, jobs and external effects may have continued after deployment.

Let's work through an illustrative Django application. No customer incident or measured outcome is being claimed here.

The innocent-looking change

Suppose a customer table has a full_name column. A new release introduces separate given_name and family_name fields.

A direct replacement sounds neat:

Remove full_name.
Add the new fields.
Deploy the new application.

But an older application instance may still read full_name. A worker might also be processing jobs created by the previous version.

Now your deployment is a compatibility problem, not just a file replacement.

And real names do not always split cleanly into two fields. The product assumption needs review before the migration gets anywhere near a database.

Define a compatibility window

One approach is to separate expansion from removal:

Stage	Database shape	Application behaviour
Expand	Keep the old field; add new fields	Existing release still operates
Transition	Both representations exist	New code uses a defined synchronisation and fallback strategy
Verify	Reconciliation and behaviour checks pass	All active services and jobs support the new representation
Contract	Remove the old field when safe	Returning to the old release may no longer be supported

This is a proposed design pattern, not a promise that every field migration can be handled this way.

The transition stage deserves most of the thinking. What happens if one write succeeds and another fails? Which field is authoritative? How will old and new application instances coexist? Can a backfill be resumed without corrupting later updates?

“Write both” is a starting thought, not a complete consistency design.

A Django migration isn't automatically reversible

Framework support is useful, but it does not recover information that was discarded.

Django supports migration dependencies and reverse operations, while some operations or custom data migrations can be irreversible. Review what your specific migration does rather than assuming a reverse command makes the business change safe. Django migration documentation

Before release, I would want the team to answer:

Can the previous application version run against the expanded schema?
Does the backfill preserve the original values needed for recovery?
Can old workers consume new jobs, or do queues need version-aware handling?
Which step ends the compatibility window?

That last question is the one that tends to hide behind “we'll tidy it later.” Later has an impressive backlog.

Separate the rollback decisions

There are several different actions people may mean:

Application rollback: route traffic to a previous application artefact.

Configuration rollback: restore previously reviewed settings.

Migration reversal: execute a specific reverse schema or data operation.

Database recovery: restore data from a backup or recovery point.

These actions have different consequences. Database recovery can lose legitimate writes made after the recovery point and may disagree with external systems that already acted on those writes.

A payment provider is unlikely to say, “Ah, you rolled back your database? No worries, we'll forget that transaction too.”

Rehearse a journey, not just a deployment command

For a controlled staging exercise:

Start with the previous release and synthetic data.
Apply the expansion step.
Exercise old and new application behaviour under the planned coexistence model.
Create representative new records and queued jobs.
Return application traffic to the earlier version.
Check reads, writes, authorisation and job processing.
Record exactly which actions are safe and which require a different recovery procedure.

Use acceptance criteria defined before the exercise. “No obvious errors” is a mood, not a test case.

Google's discussion of reliable product launches is useful background on planning and managing launch risk. The specific compatibility matrix above is an illustrative design to adapt to your own system.

The useful release question

Instead of asking only, “How do we deploy?” ask:

“After each step, which earlier state can we safely return to—and what evidence supports that?”

That gives the Technical PM a real coordination task: align application engineers, database owners, operations and product decision-makers around the same recovery options.

Which dependency makes rollback hardest in your environment: schema, background jobs, configuration, third-party APIs or data written during the release?

Draft prepared with AI support. The migration scenario and release plan are illustrative, not a claimed production implementation.

#DevOps #Architecture #Database #Django

When Is Rolling Forward Safer Than Rolling Back?

Umer Amin Durrani — Sat, 07 Nov 2026 08:46:09 +0000

You take a wrong turn. Your passenger says, “Just reverse.”

Solid advice in an empty car park. Slightly different situation on a busy flyover.

A software rollback also depends on what happened after the turn.

The previous release isn't automatically the safer release

Imagine that a new application version has written records the earlier version cannot interpret.

Returning to the previous binary might be quick. Returning to safe behaviour might not be.

On the other hand, a rushed forward fix can introduce another unknown. “I can patch it in five minutes” belongs in the same category as “I'm leaving the house now”: useful information, but worth checking.

A decision framework I would bring to the incident

Question	Why it matters
Does the previous version work with current persisted state?	Compatibility can rule out an ordinary rollback
Can we contain the effect by disabling one feature?	A smaller action may reduce impact without a full release
Is the proposed forward fix narrow and verifiable?	Another speculative deployment can increase uncertainty
Are external systems already acting on the new data?	Reverting local state may not undo those effects
Who can authorise the recovery choice?	Parallel uncoordinated changes make diagnosis harder

There is no clever slogan that replaces these answers.

The Technical PM contribution

During an incident, the useful job is not to shout “rollback” louder than everyone else.

Make the alternatives visible. Capture what is known, who is investigating each unknown and when the team will reassess. Keep customer communication aligned with the actual state of recovery.

Afterwards, turn the uncertainty into a release improvement. If nobody knew whether the old version could read the current schema, add that check to the next release plan.

GitHub's deployment environment controls can help enforce certain deployment gates where available. They do not decide whether an old application is compatible with your current data; that remains a system-specific question.

What fact would make you reject a rollback and choose a forward fix instead? I'd love a concrete scenario, with the sensitive details removed.

Draft prepared with AI support. The scenario is hypothetical and the decision table is a discussion aid.

Discuss #DevOps #Architecture #Productivity

How to Write Acceptance Criteria for AI Features That Actually Mean Something

Umer Amin Durrani — Sat, 07 Nov 2026 08:44:22 +0000

“The AI should answer accurately.”

Great. The food should also taste nice. We still need to know what we're ordering.

A polished demo can make an AI feature feel finished before the team has agreed on what success means.

Useful acceptance criteria describe the behaviour, context, evidence and failure boundaries—not just how convincing the answer sounds.

Let's use an illustrative support assistant for a small online shop. This is a proposed evaluation design, not a claim about a deployed product.

Start with the customer's task

The customer wants to understand a return policy or find out where an order is.

Those tasks require different capabilities:

A policy answer may be grounded in approved documents.
An order-status answer needs current order data and an authorised identity.
A refund changes business state and may require a separate approval process.

Putting all three behind one chat box does not make them the same requirement.

“It's just a chatbot” has done some heavy lifting in software meetings.

Replace adjectives with observable checks

Vague requirement	More useful proposed criterion
Give accurate policy answers	Use the current approved policy and identify the relevant source; do not invent an exception
Handle missing information	Ask for the missing detail or explain the limitation rather than guessing
Respect privacy	Do not reveal an order's information when the requesting user is not authorised to access it
Escalate when needed	Offer the defined human handoff when the task is outside the assistant's permitted scope
Process refunds	Prepare the request but do not execute a refund without the required approval

These are still a starting point. “Current approved policy” needs a source owner, version and update process. “Human handoff” needs a destination that someone actually monitors.

Otherwise, escalation is just the bot politely placing your problem in a decorative cupboard.

Build a small evaluation set before arguing about the model

Start with a deliberately limited set of synthetic examples that cover different failure modes:

A straightforward question with an answer in the policy.
A question whose answer is absent.
An ambiguous request that needs clarification.
A request involving another customer's information.
A stale document conflicting with an approved update.
A request to ignore the policy or execute an unauthorised action.

For each case, write down the acceptable outcome and unacceptable behaviour before generating the response. Keep some representative cases separate from prompt development so you are not only checking examples the team has tuned against.

Anthropic's agent-evaluation guide is useful reading on evaluation structure and different kinds of graders. Our shop-assistant cases are an original illustrative test design, not results from that guide.

Use several checks, not one impressive percentage

A response can be fluent and still expose private information. It can quote a policy correctly while acting on the wrong order.

Track different dimensions separately:

Answer quality: Is the response supported by the approved source?

Task behaviour: Did the assistant ask for clarification or offer a handoff when required?

Action safety: Did it stay within its tool permissions and approval boundaries?

Operational behaviour: What happened to latency, timeouts and provider failures under the tested conditions?

A small test set cannot establish a precise population-wide failure rate. Report counts, cases and limitations rather than dressing a handful of examples up as scientific certainty.

Make action boundaries enforceable

If the assistant is not permitted to issue a refund, don't rely only on “never issue refunds” in its prompt.

The surrounding tools and authorisation layer should enforce what it can do. OWASP's AI Agent Security guidance discusses this separation between agent behaviour and system controls.

For the shop example, the assistant could prepare a structured refund request. An authorised workflow validates the order, amount, eligibility and approval before executing anything.

Same customer conversation. Very different risk boundary.

Define the release decision

The team should decide in advance which failures block a release, which require restricted availability and which can be monitored with a named follow-up owner.

Document the model identifier, prompt version, source snapshot, tool configuration and evaluation conditions. Reassess when those inputs materially change.

The Technical PM doesn't need to pretend to be the model researcher. The job is to make sure product intent, engineering behaviour, security boundaries and release evidence describe the same feature.

If you could add one acceptance criterion to an AI feature tomorrow, would it concern answer quality, privacy, escalation or tool permissions? Why that one?

Draft prepared with AI support. All scenarios and evaluation criteria are illustrative; no benchmark result is claimed.

AI #Testing #Architecture #Productivity

What Does 'The AI Works' Actually Mean?

Umer Amin Durrani — Sat, 07 Nov 2026 08:42:57 +0000

The demo answers one question beautifully.

Someone says, “It works.”

The room nods. A slide gets a green tick. Somewhere, a requirement quietly escapes through the window.

I'm interested in the next question: works for which task, for whom, and under what conditions?

A chat box can hide several different products

An assistant that summarises a public document is different from one that reads customer records. Both are different from one that can change those records.

They might share the same interface. Their acceptance criteria should not be interchangeable.

Think of a kitchen knife and a plastic spoon. Both live in the cutlery drawer. We do not run identical safety tests because they have similar storage arrangements.

Four claims to separate

“It produced a plausible answer.”
That describes the appearance of one output.

“It answered these representative cases acceptably.”
Now we have a test set, conditions and criteria to inspect.

“It stayed inside its permissions when the request was hostile or ambiguous.”
Now we are checking behaviour that a friendly demo may never exercise.

“It continues to meet our service expectations after changes.”
Now we need monitoring and repeatable evaluation, not only a memorable launch-day conversation.

None of these statements automatically proves the next one.

An exercise for the next product meeting

Ask everyone to finish this sentence independently:

“We should not release this assistant if it ______.”

A developer might say “times out on common queries.” Product might say “cannot complete the customer's task.” Security might say “reveals another user's information.” Operations might say “fails without a usable fallback.”

That disagreement is useful. You have discovered different expectations while everyone still has time to do something about them.

It is much cheaper than discovering the same disagreement through an angry support ticket.

For evaluation background, see Anthropic's discussion of agent evals. An evaluation needs to match your task; copying somebody else's benchmark score is not a substitute for defining your own release question.

What is one AI failure your team would treat as a release blocker, even if the assistant gives excellent answers most of the time? Specific examples welcome; confidential details can stay off the internet.

Draft prepared with AI support. The meeting and assistant examples are hypothetical, not claimed firsthand events.

#AI #Discuss #Testing #Productivity