DEV Community: Amodh

Understanding Shannon entropy: A brief into quantifying Information

Amodh — Mon, 06 Jul 2026 18:01:43 +0000

How randomness became one of the simplest signals in modern data security.

When we hear the word entropy, most of us think of thermodynamics. Surprisingly, the same mathematical concept quietly powers many security systems used by modern software companies.

It isn't used to encrypt data and neither identify secrets with absolute certainty.

Instead, it answers a much simpler and computationally inexpensive question: Does this string look unusually random?

That single question turns out to be incredibly useful when scanning millions of log entries, chat messages, API payloads, and source code files.

What exactly is Shannon entropy?

One of the easiest ways to think about Shannon entropy is as a score assigned to a sequence of characters.

If the characters follow a predictable pattern, the score is low. If they appear highly random, the score is high.

For example, strings like Rahul, Hyderabad, or India have relatively low entropy because their character patterns are easy to anticipate. On the other hand, strings like AD#33@, Sk.Live.9981, or TNF33421 have higher entropy because their characters appear much less predictable.

The more mixed up and less predictable the characters become, the higher the entropy.

That doesn't automatically mean the string is important. It simply means the string looks statistically unusual and may deserve a closer look.

Why would companies care about random-looking strings?

Imagine a system scanning application logs and coming across the following values.

order_number=12345
token=Qx7P1mZa92LkEf8r
ref=7F29ACD83B11E2

Even without knowing what these values represent, most of us instinctively recognize that they don't all look the same.

An entropy calculation attempts to quantify that intuition.

order_number=12345
→ Low entropy
→ Probably harmless

token=Qx7P1mZa92LkEf8r
→ High entropy
→ Suspicious

ref=7F29ACD83B11E2
→ Medium/High entropy
→ Could be a generated reference ID

The entropy score doesn't make the final decision. Instead, it acts as a lightweight suspicion score that helps determine whether a value deserves further inspection.

This is why Shannon entropy is typically used as one signal inside a much larger detection pipeline rather than as a standalone system.

Does high entropy always mean a secret?

This is probably the biggest misconception.

High entropy does not mean a string is confidential.

It simply means the string is statistically less predictable.

Many perfectly harmless values have high entropy, including UUIDs, random database identifiers, cache keys, session IDs, generated filenames, and hashes of public data.

Likewise, many confidential values can have surprisingly low entropy.

For example:

password=Summer2024

That password is clearly sensitive.

Yet it isn't particularly random.

Entropy measures randomness, not sensitivity.

This distinction is what makes entropy useful, but also why it should never be used on its own.

So where does entropy actually fit in?

In real production systems, entropy is almost never used by itself.

Instead, it becomes one component inside a larger sensitive-data detection pipeline.

A typical pipeline might combine regular expressions, dictionary matching, checksum validation, known token prefixes, contextual analysis, and Shannon entropy.

Conceptually, the workflow looks something like this:

Incoming Log -> Regex Detection -> Dictionary Matching -> Checksum Validation -> Entropy Score ->Final Confidence Score

Each stage contributes evidence before the system decides whether something is likely to contain sensitive information.

Entropy simply strengthens that decision.

How do large companies use this in practice?

Organizations processing millions—or even billions—of log entries cannot manually inspect every value flowing through their systems.

Instead, they build automated pipelines that attempt to identify sensitive information before it gets stored, indexed, shared, sent to monitoring platforms, or displayed on dashboards.

Within these pipelines, Shannon entropy helps surface values that look like API keys, access tokens, JWTs, OAuth tokens, encryption keys, password hashes, payment-related identifiers, or random secrets accidentally pasted into logs.

Notice the wording here.

The system isn't concluding that these values are secrets.

It's simply increasing the confidence that they deserve closer inspection.

What might this look like in a codebase?

Many companies implement a small utility that computes a normalized Shannon entropy score for a given string.

That score is then passed to a larger security rules engine instead of making decisions by itself.

The architecture often looks something like this:

Application -> Log Scanner -> Entropy Calculator -> Security Rules Engine -> Alert / Redaction / Block

The entropy calculator is simply another reusable component inside the overall detection pipeline.

Some questions I had while learning about this

Do all companies use Shannon entropy?

Not necessarily, many security products and internal security platforms use entropy-based heuristics, especially for secret detection, data loss prevention, and log redaction.

However, there isn't a universal standard.

Some organizations rely heavily on entropy, while others place more emphasis on contextual analysis, machine learning, proprietary heuristics, or a combination of several techniques.

Does a high entropy score always indicate a valuable or confidential string?

Not at all. A randomly generated UUID may have high entropy while being completely harmless.

On the other hand, a password like Summer2024 has relatively low entropy but is still highly confidential.

Entropy is a heuristic, not proof.

It provides another signal to help a detection system make better decisions, but it should never be treated as definitive evidence.

Bonus point- Was this names after Claude Shannon? The same Clause that Anthropic's AI is named after?

Yes! Anthropic's AI assistant, Claude, is named after Claude Shannon, the mathematician and electrical engineer widely regarded as the father of information theory.

Another day, another concept simplified!

Shannon introduced the concept of information entropy in his landmark 1948 paper, A Mathematical Theory of Communication. His work laid the mathematical foundation for modern data compression, cryptography, digital communications, and many other areas of computer science.

While today's AI models don't generate responses by directly computing Shannon entropy, the name "Claude" serves as a tribute to one of the most influential figures in information theory.

Final thoughts

Before reading about Shannon entropy, I assumed it was some sophisticated technique for identifying secrets.

It turns out the idea is much simpler.

Shannon entropy isn't a secret detector. It's a randomness detector.

On its own, it cannot determine whether a string is confidential. But when combined with regular expressions, contextual analysis, dictionaries, and checksum validation, it becomes an inexpensive yet surprisingly effective signal inside modern security systems.

Sometimes, asking "Does this string look unusually random?" is far more practical than trying to answer "Is this definitely a secret?"—especially when you're scanning billions of strings every day.

Understanding the Luhn Algorithm: Why your browser knows a card number is invalid before the bank does

Amodh — Sat, 04 Jul 2026 19:07:08 +0000

While making an online payment the other day, I accidentally mistyped my card number. The website rejected it immediately, even before there was any indication that it had contacted the issuing bank.

Which lead to the question

How can a software or a portal determine that a number is invalid without querying a database?

The answer lies in one of the simplest and most elegant checksum algorithms ever designed and that I recently got to know about- the Luhn algorithm.

What is the Luhn Algorithm?

The purpose of the Luhn algorithm is not to verify that a number (note: here I am considering any reference number/credit card number, etc when I say number) exists.

Instead, its purpose is to detect whether a number has been typed or transmitted correctly by verifying its checksum (Assuming that the human inputting the number might mistype a digit or two)

It acts as a fast, inexpensive first line of validation before any network request or database lookup is performed.

Assuming that it is primarily used for Credit card number validations, does this validation only work for 16-Digit Numbers?

No.

Although it is commonly associated with payment card numbers, the Luhn algorithm works for identifiers of any length, provided they contain a check digit (the final digit of any number).

Its design is completely independent of the number of digits.

When we day "number is valid", what Does "Valid" actually mean?

This is perhaps the most common misconception.

Passing the Luhn algorithm does not mean that the identifier exists in a databse and holds some value against it.

It simply means the identifier is structurally valid according to the checksum.

In other words, the number is plausibly valid and has been inputted correctly by the user.

A number can satisfy the Luhn algorithm even if it has never been issued by any institution.

To determine whether an identifier actually exists, the issuing organization must still perform its own database lookup and authorization checks (which is honestly a story for another time).

The Luhn algorithm is therefore best thought of as the first and cheapest filter.

Do institutions intentionally generate numbers that satisfy Luhn?

Yes.

When an organization generates an identifier that uses the Luhn algorithm, it deliberately computes the final digit—known as the check digit—so that the complete number satisfies the checksum.

The check digit is not random; it is calculated.

The internal working

Starting from one end of the number (the direction depends on whether the identifier has an odd or even number of digits), every alternate digit is multiplied by 2, producing a repeating multiplication pattern: Assume the below pattern

2 1 2 1 2 1 ...

If doubling a digit produces a two-digit result (for example, 8 × 2 = 16), the digits of the product are added together (1 + 6 = 7). This is equivalent to subtracting 9 from any doubled value greater than 9.

Finally, all transformed digits are summed.

If the total is divisible by 10, the identifier passes the checksum.

Why the 2-1-2-1... Pattern?

The pattern is not arbitrary.

Doubling every alternate digit creates a transformation in which every decimal digit maps to a unique output after digit reduction (Every single-digit number maps to a unique single-digit value after the Luhn transformation: 0→0, 1→2, 2→4, 3→6, 4→8, 5→1, 6→3, 7→5, 8→7, and 9→9, ensuring that no two different digits produce the same output.). Which might not have been possible if any other number other than 2 would have been used.

This uniqueness allows the algorithm to detect all single-digit errors and most adjacent transposition errors.

Alternative multiplication patterns such as 3-1-3-1 or 4-1-4-1 do not preserve this property, making them less effective at error detection.

Why is the checksum based on divisibility by 10?

The designers could have chosen any modulus. Then why 10?

They chose 10 because decimal numbers are what humans naturally work with, and checking whether a sum is divisible by 10 is computationally trivial:

sum % 10 == 0

Using modulo 10 makes both the implementation and the explanation remarkably simple. This is why the Luhn's algorithm is also called the modulo 10

Why not simply store every issued identifier in a database?

Because the goals are different.

A database answers the question:

"Does this identifier exist?"

The Luhn algorithm answers a cheaper question:

"Could this identifier even be valid?"

By rejecting obvious typing mistakes locally, applications avoid unnecessary database queries and network requests.

This improves responsiveness and reduces load on backend systems.

When should an organization use the Luhn Algorithm?

The Luhn algorithm is a design choice, not a requirement.

It is most useful when identifiers are frequently entered manually and there is value in detecting accidental typing errors before performing an expensive lookup.

Whenever humans are expected to repeatedly type long numeric identifiers, adding a single calculated check digit can dramatically improve the user experience while reducing unnecessary backend work.

So this is how a simple typo while making an online payment led me down a rabbit hole into one of the most elegant algorithms I've come across.