Everything runs in this tab and nothing you open is uploaded. The eight specimens are synthetic and seeded, so every visitor sees the same bytes. No malware is involved: the packed files are headers and random bytes.
Entropy lab
Packed and encrypted code looks like noise, and noise is measurable. Entropy is the number scanners use to spot it: bits per byte, from 0 (every byte the same) to 8 (no pattern at all). Five exhibits take it apart, from the sum by hand to a detector you tune yourself, including where it falls over.
Lab exploredThat's entropy, start to finish. The YARA lab picks up where it runs out: rules that key on what entropy can't see.
Count the surprise
Shannon entropy measures how surprised you should be by the next byte. A file of zeros never surprises you: 0 bits. If all 256 byte values are equally likely, you get the maximum: log2(256) = 8 bits. The formula adds up −p·log2(p) for each byte value, where p is that value's share of the data.
Type below. The table is the working: each row is one byte value, its share, and what it adds to the total. Shannon also showed that this number is the size limit for squeezing the data with any code that only looks at how often each byte occurs.
- Bytes
- 0
- Distinct values
- 0
- Entropy
- 0.000
- Ceiling
- Smallest size
| Byte | Count | Share p | −p·log₂p | Running sum |
|---|
The maths
A byte value with share p carries s = −log2 p bits of surprise: a share of ½ costs 1 bit, 1/256 costs 8, and 1 costs nothing. Entropy is the average surprise, each value weighted by how often it turns up.
H = −∑i pi log2 pi = log2 n − (1/n) ∑i ci log2 ci
The second form is the same sum written with counts (pi = ci/n). It's the one the code runs, because counts are whole numbers that are cheap to keep up to date.
Why the ends are 0 and 8. With one value, p = 1 and log2 1 = 0, so H = 0. With k equally common values every p is 1/k and the sum collapses to log2 k, which is 8 when k = 256. Nothing beats equal shares: H is the average of log2(1/p), the logarithm is concave, and Jensen’s inequality caps that average at log2 of the average of 1/p, which is log2 k.
Why it's a size limit. Shannon’s source-coding theorem: if bytes arrive independently with these frequencies, no lossless code can average fewer than H bits per byte, and Huffman or arithmetic coding gets close. So H·n/8 bytes is the floor for any scheme that only looks at how often each byte occurs. A compressor that also uses order, such as zlib on text, can go below it.
In practiceEntropy depends only on the byte counts. Shuffle a file into any other order and it doesn't move, which is why some patterns hide from it later.
DefenceNothing to defend yet, but two facts carry through the lab. Entropy can never exceed log2 of the number of distinct byte values in the data, and it ignores order.
Read a whole file at once
One number for a whole file hides where the interesting part is, so scanners slide a window along the file and measure each stretch. The ring and the line below draw the same measurement: position runs round the ring and along the line, and height is bits per byte. Pick a specimen, point at the plot, and read the bytes there.
Real files aren't uniform. Padding reads near 0, strings and tables sit in the middle, code reads about 6, and compressed or encrypted data climbs past 7.5. Switch on the layout to see what each part holds, or open a file of your own, which never leaves the tab.
Or drop a file anywhere on this card. It is read here, up to 24 MB, and never sent.
Each byte is a point: across is its value, up is the value of the byte after it. Text huddles in one bright block, zeros are a single dot, and random bytes fill the square.
The maths
The scope measures a window of w bytes, slides it on by half a window and measures again. Instead of recounting each window, the code keeps the counts. Sliding one byte out and one in changes two counts by one each, and the sum S = ∑ c log2 c changes by f(c ± 1) − f(c), where f(x) = x log2 x comes from a table built once.
Hw = log2 w − S/w
So the whole profile costs one table lookup per byte of file: O(n) instead of O(n·w). The engine recounts from scratch every 2,048 slides so rounding can't build up.
Why the line sits below 8 even for random data. A window is a small sample, and counts from a small sample are lumpy: some values appear twice, some not at all. Lumpiness always lowers the measured entropy (the same concavity as before), so on average, for k equally likely values and a window of n bytes:
E[Ĥ] ≈ log2 k − (k − 1) / (2n ln 2)
Each of the k − 1 free counts adds a little noise, worth 1/(2n ln 2) bits of shortfall. The formula is good once n is well above k. The ceiling on this page is the exact expectation instead, summed from the binomial counts, which stays right for small windows where the formula doesn't.
In practiceDetect It Easy and PE-bear draw this same profile and report entropy per section. As rules of thumb, English text runs about 4 to 4.5, x86 code 5.5 to 6.5, and past 7.2 in a 1 KB window the data looks random.
DefenceHigh entropy is a reason to look closer. It isn't a verdict, because installers, archives, images and encrypted documents all read high. The next exhibits show how to get more out of the signal.
Window size changes the reading
A window of n bytes can only show so much randomness. Perfectly random data reads about 5.8 in 64-byte windows and 7.2 in 256-byte windows, and needs several kilobytes to get near 8. So a detector looking for “above 7.5” in 256-byte windows can never fire. The opposite mistake is a window so large that a small block is diluted to nothing.
Below, needle.bin is 240 KB of ordinary text with one 3 KB block of random bytes hidden inside. Find it.
The maths
Entropy is one summary of a window’s byte counts. Chi-square is another: it measures whether the counts are flatter or lumpier than chance would make them. With n bytes, chance expects E = n/256 of each value.
X = ∑v=0255 (Ov − E)2 / E, df = 255
For random bytes X averages 255 with a spread of √510, about 22.6. To turn it into a probability the page uses the Wilson-Hilferty approximation: the cube root of X/df is close to normal, so
z = [ (X/df)1/3 − (1 − 2/(9df)) ] / √(2/(9df)), p = 1 − Φ(z)
and p is the chance that noise would look this lumpy or lumpier. The unit tests check it against the exact tail of the chi-square distribution.
What a window can resolve. The two statistics are linked: near a flat histogram, Ĥ ≈ 8 − X/(2n ln 2). So the spread of X becomes a spread in entropy of about √510 / (2n ln 2) ≈ 16.3/n bits: 0.016 at 1,024 bytes, 0.064 at 256. Chi-square itself needs E of about 5 or more per cell, so n ≥ 1,280. Below that, many cells hold 0 or 1 and X is only a rough guide. A window can't see structure smaller than itself: a 3 KB block in 240 KB barely moves a whole-file figure, but a window that fits inside the block sees it at full strength.
In practiceThis is why entropy tools show a profile rather than one figure, and why the window and the step are settings. Half a window is a common step: every stretch of the file is then measured at least twice.
DefenceMeasure in windows no bigger than the smallest block you care about, and set the threshold from what random data reads at that size. Use the whole-file average as a summary and nothing more.
Several different things read high
Compressed data, encrypted data and random data all read close to 8, and so does plenty of ordinary data. Entropy also has blind spots of its own. Text written as base64 or hex reads much lower, capped at 6 and 4 bits. XOR a file with a one-byte key and every byte changes, yet entropy doesn't move at all.
Four unlabelled blobs are below. Use the numbers, the pair plot and the first bytes to decide what each one is, then check your labels. The bench under them shows what XOR does to entropy as the key gets longer.
- Size
- Entropy
- Distinct bytes
- Printable text
- Chi-square
- Serial correlation
The maths
Compressed, encrypted and random data can all read close to 8, because their byte counts are flat, and entropy looks only at counts. What can separate them is order.
Serial correlation asks whether a byte predicts the next one. With m the mean byte value, and each byte paired with its successor (the last with the first):
r = ∑ (xi − m)(xi+1 − m) / ∑ (xi − m)2
It's 0 for independent bytes, give or take 1/√n, and near ±1 for a ramp or an alternation. The digraph plot counts every ordered pair (bi, bi+1) in 256 × 256 cells. Its number is the pair entropy H2, and H2 − H1 is what is still unknown about a byte once you have seen the one before it: 8 bits for independent uniform bytes, far less for anything with structure.
Why high doesn't say what it is. A good cipher is built so that no statistic tells its output from random, so only the file type, a signature or a decoder can. A compressor also reads flat, but leaves a header, block markers and a slightly lumpy histogram (exhibit III’s chi-square). Hex or base64 text uses only 16 or 64 values, so its ceiling is log2 16 = 4 or log2 64 = 6 whatever the bytes underneath.
Pairs need far more data than counts: n − 1 pairs cannot fill 65,536 cells, so H2 can never exceed log2(n − 1). That's why the example sets the blob beside random bytes of the same length.
In practiceChi-square helps: good ciphertext is about as flat as noise, while a compressed stream is slightly lumpy. Compressed formats also begin with a signature, and the decisive test is the decoder. Entropy narrows the search but doesn't finish it.
DefenceDon't rely on entropy alone. Add the file type, the signature, whether a decoder accepts the data and where in the file it sits. A one-byte XOR hides from entropy, and a long key raises it, but the byte-pair plot and repeated-key statistics still show the structure underneath.
Turn the number into a rule
“Flag anything above N” is only the start of a detector. Below are eight files: four carry packed or encrypted payloads, and four are innocent, including an installer and a zip that read as high as any. Set the window, threshold, minimum run and where to look, and the table scores you live. Find the settings that get eight of eight. the first threshold you try will flag the installer lol.
| File | Profile | Should | Verdict | Why |
|---|
The maths
The detector here is this rule and nothing more. Chi-square plays no part in it.
- With “only executables” on, a file that doesn't parse as one is skipped.
- Slide a window of w bytes along the file, half a window at a time, and measure H in each. A window belongs to the zone under its centre: the headers, a named section, a gap, or the overlay after the last section.
- A window counts only in a zone the rules look at: never headers or gaps, read-only sections unless you leave them in, the overlay only if you include it.
- Counted windows in a row with H ≥ t make a run. A run stands only if it spans at least L bytes.
- The file is flagged if any run stands.
Every setting trades one error for the other. Lower t and you catch more payloads but flag installers and archives (false positives). Raise it and the false alarms stop, while small or lightly packed blocks slip through (false negatives). Plot the share of payloads caught against the share of innocent files flagged as t sweeps from 0 to 8 and you get a ROC curve, from (1, 1) where everything is flagged to (0, 0) where nothing is. A better rule bends it towards (0, 1). Each setting is one point on it, and the gates and L move the curve itself.
Why a section beats a file average. The entropy of a mixture is at least the size-weighted average of its parts, so a 20 KB packed section in a 400 KB file moves the whole-file figure only a few per cent. A section’s own entropy ignores its surroundings, so measuring per zone keeps the payload at full strength.
In practiceReal tools pair entropy with section flags (writable and executable at once is a classic tell), raw against virtual size, the entry point, the import table, a valid signature and reputation. Entropy alone would flag your own installer.
DefenceTune against a corpus that includes the innocent high-entropy files you ship, count the false positives first, and treat a high score as a reason for a closer look.