A supermarket’s internal sales look healthy, yet its customers are increasingly unhappy and competitors are quietly changing prices. How can an organization recognize a problem that its own databases cannot fully explain?

A business cannot understand its environment by examining its internal history alone. Effective BI must bring external signals into the decision-making process without confusing unreliable information with evidence.
A supermarket can tell you how many vegetables it sold yesterday, what remains in inventory, and how much each transaction earned. If management wants to reconsider prices or plan a promotion, the natural instinct is to query those records.
It is a good instinct. Internal data is familiar, structured, and already available. Building another report from it seems cheaper than looking elsewhere.
But suppose the decision also depends on competitors’ prices, an upcoming regulation, or what customers are saying about product quality. Those signals may exist long before they appear as changes in the supermarket’s own sales.
The database has not ✨failed✨
It was never responsible for recording the whole environment.
The Database Is Accurate About the World It Can See
The previous step in Business Intelligence was organizational memory: retaining transactions, documents, and knowledge so they can be used later. Now comes a different problem. Memory can preserve internal history without explaining what is changing outside the organization.
This matters whenever a decision depends on information the business does not generate itself. A price change may be relevant because competitors have adjusted theirs. A product decision may depend on customer opinions expressed on public websites. A planning decision may depend on market, technological, economic, regulatory, social, or political developments.
The tempting fix is to add a few more feeds and declare the problem solved. Yet a feed is only a route by which information arrives. It does not decide whether the information is relevant, reliable, or even comparable to what the company already knows.
That is the actual responsibility of information integration : making useful evidence from different places and formats work together for a business decision.
“Collect Everything” Is the Next Attractive Mistake
Once external information looks valuable, web scraping, text mining, and market-monitoring tools become difficult to resist. Why not collect every relevant-looking article, customer review, and competitor update?
Because the collection process already contains decisions that software cannot avoid: What question are we trying to answer? Which sources can answer it? Can we access them lawfully? Are they trustworthy? And can we afford to process them?
A sensible integration effort therefore starts with the business need, followed by data availability, organizational expertise, the capabilities of existing BI applications, and budget. Only then does the organization decide how much external information to bring in.
Even after choosing the sources, the data still needs to be understood. Who owns it? What does each field mean? How frequently is it updated? How complete is it? What are its security, privacy, and permitted-use constraints?
That last question is not a bureaucratic afterthought. Combining two datasets can make information more useful, but it can also make its errors more persuasive. An incorrect price, an outdated notice, or a duplicated customer record can travel through the entire analytical process looking remarkably official.
Garbage in, garbage out is still true when the garbage comes from five different websites.
Looking At the Environment Is Not the Same as Looking For Something
The way an organization searches its environment depends on two conditions.
Environmental analyzability describes how predictable or interpretable the organization believes the environment to be. Organizational intrusiveness describes how actively it commits resources or intervenes to obtain information.
Together, they give four modes of environmental scanning:
Passive approach
- Undirected viewing (un-analyzable environment): Notice informal signals without a tightly defined search.
- Conditioned viewing (analyzable environment): Routinely monitor selected issues through established sources.
Active approach
- Enacting (un-analyzable environment): Try a change and observe the response.
- Searching (analyzable environment): Systematically collect and analyze information for a defined question.
Consider four possible approaches to vegetable pricing. Casual customer conversations resemble undirected viewing. Regularly reading an official price publication resembles conditioned viewing. Testing a promotion and observing customer response resembles enacting. Systematically assembling verified price data to forecast a future trend resembles searching.
The difference is not simply which website was opened. It is whether the organization is casually observing, routinely monitoring, experimenting, or conducting a deliberate search.
Enacting is especially interesting. When the environment is difficult to analyze from observation alone, the organization may learn by changing something and watching what happens. That is a different kind of information gathering from waiting for another report.
This four-mode distinction is developed in Chun Wei Choo’s work on environmental scanning. It also exposes a limit: greater scanning effort may reduce uncertainty, but it does not make an uncertain environment perfectly predictable.
The Outside World Rarely Arrives as a Clean Table
Internal sales data might offer rows with product identifiers, dates, quantities, and prices. External evidence often arrives as prose: news articles, reviews, reports, emails, posts, or regulatory announcements.
You cannot reliably answer every question about those documents with a familiar SELECT ... WHERE ... query. First, the useful information must be located and interpreted.
This is where text mining becomes valuable. Three related tasks solve different parts of the problem:
- Information retrieval finds and ranks documents or passages relevant to a query. Unlike an exact database lookup, results may have different degrees of relevance.
- Information extraction identifies useful entities, facts, and relationships within the text, often using Natural Language Processing (NLP).
- Information summarization compresses larger texts into their important ideas, so the decision-maker does not have to read every document in full.
There is a small but important detail in retrieval: a word occurring frequently is not necessarily informative. A term appearing in almost every document tells us less about what makes one document distinctive.
Term Frequency–Inverse Document Frequency (TF-IDF) addresses that by combining the frequency of a term within a document with how uncommon it is across the collection:
tf-idf(term, document) = tf(term, document) × idf(term)
Common implementations use a logarithmic, sometimes smoothed, IDF. The objective is to give more weight to terms that are prominent in a particular document without being ubiquitous everywhere. Scikit-learn’s TF-IDF explanation shows the exact variants.
That score is useful for indexing, retrieval, and classification. It is not a truth score. A distinctive phrase can make a document easy to retrieve without making its claims accurate. NLP, categorization, clustering, and human review still have work to do. IBM’s overview of text mining describes how these techniques turn unstructured language into analyzable information.
The Web Has Content, Connections, and Behaviour
Text mining helps us interpret documents. But the web offers more than documents, and obtaining those documents introduces a separate step.
A web crawler discovers and retrieves pages, often by following hyperlinks. Crawling collects material; web mining attempts to discover useful patterns from web-based data. The distinction matters because downloading an entire website is not the same as understanding it.
Three forms of web mining ask three different questions:
Web content mining examines text, images, audio, video, and metadata to discover what a page contains. Web structure mining investigates links and relationships between pages. Web usage mining examines interactions such as visits and navigation patterns, generally requiring appropriate access to logs or other usage data. These are established categories in Kosala and Blockeel’s survey of web-mining research.
Usage mining itself needs preprocessing , pattern discovery through data-mining methods, and pattern analysis using techniques such as visualization and OLAP. Otherwise, raw interaction events are just another collection of records.
The business connection is Customer Relationship Management (CRM). Operational CRM helps handle sales and service interactions. Analytical CRM uses customer information to discover segments, patterns, and opportunities for more relevant service or campaigns. Web behaviour and feedback can support that analysis, provided the organization has appropriate access and handles customer information responsibly. IBM distinguishes these CRM responsibilities.
A customer score may help prioritize a campaign. It does not establish that the campaign will work, and browsing behaviour alone does not explain every customer motivation. The analytical result still needs an appropriate decision.
Some Signals Come From Objects. Others Are Hidden in Documents.
Not every useful signal is a review or a website visit. Radio Frequency Identification (RFID) can provide information about the movement and presence of tagged goods, supporting inventory tracking and supply-chain visibility. A passive RFID tag typically draws energy from a reader; an active tag has its own power source. Battery-assisted passive tags occupy a middle ground. GS1 explains these differences.
RFID is not web mining, nor does it automatically reveal a competitor’s intentions. It illustrates a broader integration problem: relevant evidence may be generated by physical processes, not just databases or public text.
The same lesson appears in NASA’s Expert Seeker. The challenge was to locate people with particular skills when relevant evidence was scattered across employee information and documents describing their work. Web content mining helped associate names with expertise keywords, including through TF-IDF-based indexing, rather than relying entirely on a manually maintained skills list. NASA’s Expert Seeker report documents that approach.
There is a catch here too. Ranking possible experts does not guarantee that every returned person is relevant, or that every relevant person was found.
That is why retrieval quality is assessed with precision (relevant results divided by all returned results) and recall (relevant results found divided by all relevant results). Better retrieval is not just about collecting more pages. It is about improving what the system can reliably recover from them.
Integration Is Where the Evidence Becomes Usable
At this point the supermarket could have transactions, official price information, customer comments, web interactions, and RFID readings. That sounds like an impressive collection.
It is not yet an integrated BI system.
Data migration moves information between systems. Consolidation brings information together. Master Data Management (MDM) helps keep important shared entities consistent. Synchronization maintains corresponding records across locations. Data warehousing organizes information for analysis. These activities overlap, but none is interchangeable with the entire job of integration.
The practical work starts with requirements: input and output formats, transformation rules, update frequency, permitted access, latency, and what to do when records are wrong. Then come the details of Extract, Transform, Load (ETL): how a source is accessed, what a field really means, how mismatches are corrected, and how the result is loaded without destroying useful history.
A field called price may mean different things in different sources. A review may refer to a product under a name the inventory system does not recognize. A current record may overwrite a value whose historical meaning still matters. These are illustrative examples of the semantic problems integration must address, not failures a particular supermarket is known to have experienced.
The work needs documentation, testing, error handling, monitoring, and data lineage so analysts can tell where a figure came from. Microsoft’s ETL guidance describes why cleaning, validating, and reshaping data are part of the integration process rather than separate cosmetic improvements.
Now the design begins to make sense to The Origami Software Engineer. We did not add external information simply to make the dataset larger. We added selected information because the decision required evidence outside the operational system, then integrated it because evidence with incompatible meanings cannot safely support analysis.
The supermarket’s database was useful all along. It could accurately describe transactions, stock, and margins. It just could not be expected to describe every event, opinion, relationship, or change beyond its boundaries.
A good BI system does not know everything. It knows which questions require looking beyond what it already remembers, and how to bring the answers back without losing their meaning.
That’s not failure.
That’s evolution.
The “I liked this” Starter Pack:
Don’t let your fingers get lazy now.
- Like : It tells me this was worth writing.
- A Comment: Tell me your thoughts, your favorite snack, or a better title for this blog.
- Boost it: Especially with that one developer who definitely needs this.
Thanks for being here. It genuinely helps more than you know!
— Aaroophan Varatharajan
Find me elsewhere:
- Professional stuff: linkedin.com/in/Aaroophan
- Code stuff: github.com/Aaroophan
- UI stuff: aaroophan.dev/Aaroophan
- Life stuff: instagram.com/Aaroophan



Top comments (0)