A content automation pipeline can become wasteful surprisingly early.
Imagine pulling stories from several external sources.
Every new item arrives.
Every item gets summarized.
Every item gets sent through an LLM.
Every item gets a social draft.
Maybe an image prompt is generated too.
Then somebody looks at the output and realizes half of it was never useful enough to publish.
The problem started much earlier than generation.
The system never decided which stories deserved generation in the first place.
A cleaner pipeline adds a relevance gate before expensive AI work begins.
Do not make generation the first decision
A simple content pipeline often starts like this:
Sources
↓
Collect items
↓
Generate article
↓
Generate social copy
↓
Review
It works.
It is also doing expensive work before asking whether that work is needed.
A better sequence is:
Sources
↓
Collect items
↓
Normalize
↓
Relevance check
↓
Duplicate check
↓
Candidate queue
↓
Generate
↓
Review
Now generation is reserved for candidates that have already passed cheaper checks.
Normalize before you score
Different publishers rarely describe content the same way.
One source may call the main field title.
Another returns headline.
Another has rich category metadata.
Another provides almost none.
Before relevance logic runs, create a consistent internal object.
A simplified model could look like:
type SourceItem = {
id: string;
sourceId: string;
headline: string;
summary?: string;
url: string;
canonicalUrl?: string;
publishedAt?: string;
categories: string[];
body?: string;
};
The exact fields will vary.
The goal is more important than the schema:
relevance logic should work against one predictable representation instead of learning every source format.
Separate cheap checks from expensive checks
Not every relevance decision needs an LLM.
Some candidates can be eliminated with inexpensive rules.
For example:
function passesBasicRules(item: SourceItem) {
if (!item.headline) return false;
if (isTooOld(item.publishedAt)) return false;
if (blockedSources.has(item.sourceId)) return false;
if (containsExcludedCategory(item.categories)) return false;
return true;
}
This is not sophisticated.
It does not need to be.
Every obviously irrelevant item removed here is an item that never needs a model call later.
Then score the candidates that remain
After basic filtering, assign relevance deliberately.
A content operation may care about things such as:
- geography
- topic
- company or entity
- business category
- recency
- strategic priority
- audience fit
A simple scoring interface could look like:
type RelevanceScore = {
score: number;
reasons: string[];
};
async function scoreRelevance(
item: SourceItem
): Promise<RelevanceScore> {
// Rule engine, classifier, model call,
// or a combination of them.
return {
score: 82,
reasons: [
"target geography",
"relevant property topic",
"recent publication"
]
};
}
The number itself is not magic.
What matters is making the decision explicit.
Now the workflow can say:
if (relevance.score < MIN_GENERATION_SCORE) {
await markSkipped(item.id, relevance);
return;
}
The system knows why generation did not happen.
That is much easier to operate than silently dropping items somewhere downstream.
Deduplication should happen before generation too
External content feeds repeat stories constantly.
The same development may appear:
- on the original publisher
- in a syndicated copy
- in another outlet's rewrite
- in an updated version
- under a slightly different headline
If all of those reach the generation layer, the model produces several variations of essentially the same content.
That wastes model calls and fills the review queue with duplicates.
A duplicate gate can use several signals:
Canonical URL
Headline similarity
Entity overlap
Publication time
Source relationship
Body similarity
A simplified decision could be:
const duplicate = await duplicateDetector.findMatch(item);
if (duplicate) {
await markDuplicate(item.id, duplicate.id);
return;
}
Again, the benefit is not just cost.
The editorial team receives a cleaner candidate queue.
Build a candidate queue, not a generation queue
Once an item has passed:
- normalization
- basic rules
- relevance scoring
- duplicate detection
it becomes a candidate.
That distinction is useful.
A candidate does not automatically mean:
Generate everything now.
It means:
This item is worth considering for content creation.
You can model that explicitly:
type Candidate = {
itemId: string;
relevanceScore: number;
status:
| "candidate"
| "approved_for_generation"
| "skipped"
| "generated";
};
Now editorial policy has somewhere to live.
Some candidates can still wait
Suppose six useful stories arrive within ten minutes.
Generating all six immediately may not make sense.
The operation might only need:
- two website stories today
- three social posts
- one weekly summary
So rank the candidate queue.
Candidate A 94
Candidate B 88
Candidate C 84
Candidate D 79
Candidate E 74
Candidate F 71
Generation capacity can then follow editorial priority.
This creates a much healthier relationship between automation and volume.
The system is no longer asking:
What can we generate?
It is asking:
What is worth generating now?
Make rejection observable
A skipped story should not disappear into nowhere.
Store the decision.
For example:
{
"itemId": "story_482",
"decision": "skipped",
"reason": "low_relevance",
"score": 41,
"evaluatedAt": "2026-08-15T10:30:00Z"
}
Or:
{
"itemId": "story_517",
"decision": "duplicate",
"matchedItemId": "story_503"
}
This gives operators useful answers later.
Why did this story not get generated?
Why did the queue suddenly become small?
Why are two publishers producing similar candidates?
Without decision records, automation becomes difficult to explain.
Generation should receive context, not raw source data
When a candidate finally reaches generation, the model should not be forced to rediscover everything the pipeline already knows.
Pass structured context.
For example:
type GenerationContext = {
sourceItem: SourceItem;
relevance: RelevanceScore;
targetAudience: string;
outputType: "article" | "social" | "summary";
editorialNotes?: string[];
};
Now generation starts from a much cleaner state.
The workflow already knows:
- what the item is
- why it matters
- what output is needed
- who the output is for
That usually improves consistency as much as it reduces wasted processing.
Heavy work belongs behind the gate
Generation is rarely the last expensive operation.
A content platform may also run:
- translation
- asset creation
- rendering
- enrichment
- image processing
- scheduled publishing
Those jobs can live behind worker queues.
The pipeline then becomes:
Source intake
↓
Normalize
↓
Filter
↓
Score
↓
Deduplicate
↓
Candidate
↓
Generate
↓
Human review
↓
Approved?
↓
Background workers
↓
Schedule / publish
The expensive path starts only after the content has earned its way there.
Measure the funnel
Once the gates exist, measure them.
Useful numbers might include:
Items collected
↓
Items passing basic rules
↓
Relevant candidates
↓
Unique candidates
↓
Generated drafts
↓
Approved drafts
↓
Published items
This tells you much more than:
We generated 5,000 pieces this month.
A large generation count can hide a very inefficient pipeline.
A smaller generation count with a strong approval rate may be healthier.
A recent system where this mattered
We recently worked on an AI Real Estate Content Operations Platform that brought source intake, relevance scoring, deduplication, AI content generation, review, scheduling, and background processing into one operational workflow.
The goal was not to push every incoming source item directly into generation.
The system needed a controlled path from external information to content that was actually useful enough to review and publish.
The public project breakdown is here:
Related work:
https://ascentinnovate.com/work/ai-real-estate-content-operations-platform
Generate later than you think
When teams add AI to content operations, generation is naturally the feature that gets attention.
But a good pipeline makes several decisions before the model starts writing.
- Is the source usable?
- Is the story relevant?
- Have we already covered it?
- Is it worth producing now?
- Which output does it actually need?
Those gates protect model spend, editorial attention, and publishing quality at the same time.
Do not generate everything you can collect.
Generate the things that have already earned a place in the queue.
Top comments (0)