Sentiment analysis is the wrong shape for market events

#ai #python #api #machinelearning

Ask a sentiment API about "Government raises import duty on edible oils" and you get something like negative, 0.82.

Negative for whom?

That headline is bad for FMCG margins and good for domestic oilseed processors. Both of those are true at once, and the single number cannot express it. The problem is not that the score is imprecise. It is the wrong shape of answer.

Sentiment collapses a redistribution into a mood

Most market-moving events do not push the whole market one way. They move value from one place to another.

A fuel duty change compresses margins for airlines and lifts them for oil marketing companies. A sugar export quota helps mills and hurts confectionery input costs. A rate hike is bad for rate-sensitive borrowers and fine for banks with floating books.

Every one of those is a redistribution. Impact has to land somewhere. A tool that returns a scalar has already thrown away the only thing you needed: direction, per sector.

What we return instead

For one event — a headline, a circular, a notification — the output is structured:

the event, resolved to what actually happened rather than how it was phrased
sector impact across NSE sectors as a signed vector: winners positive, losers negative, in the same response
stakeholder views, because a producer, a consumer and a trader experience the same event from different sides
a confidence band and supporting evidence, so you can weigh a well-grounded answer differently from a weak one

The signed vector is the part that matters. Both signs appear in one response, which is the only honest representation of an event that helps some sectors and hurts others.

Why a general model struggles here

Two reasons, both about context rather than language.

Indian market structure is specific. Monsoon cycles, MSP announcements, duty and subsidy regimes, the difference between a state-level action and a central one. These are not vocabulary problems. A model trained mostly on US markets will parse the sentence correctly and still map it to the wrong sectors, because the causal structure it learned does not apply.

Headlines are frequently ambiguous. "India needs to import sugar" could be a shortage, a policy shift, or a trade story. The body usually resolves it. Any system reading headlines alone is guessing on a meaningful fraction of inputs, and confident guessing is worse than declining.

Declining is a feature

When an event falls outside what the system can resolve, it declines rather than inventing an answer.

That sounds like a limitation and is closer to the opposite. A pipeline consuming structured output needs to distinguish low confidence from no answer from confident answer. Collapsing all three into a number with a plausible-looking score is how downstream systems end up acting on noise.

We publish a confidence band rather than a bare number for the same reason: it is part of the answer, not decoration.

What it will not do

It interprets events. It does not tell you what to trade, and it returns no execution signals of any kind — that separation is deliberate and is enforced at the API boundary, not left to convention.

DEV Community