Rename this
A protector wraps a program so that its strings, its structure and its names tell you nothing. This workshop is the analyst's side of that. Below is a fictional loader called Cinderling in a workbench modelled on a disassembler, and six jobs that take it from unreadable to a configuration you could hand to a defender: find the string decryptor, cut the junk out, set calling conventions, decode the strings, name what you found, and pull the config out of its encrypted blob.
Everything runs in this tab, and your names, comments and settings stay in this browser. Nothing here is real malware: the listing was written for this workshop, it is a page you read and not a program you run, every value is invented, and the hosts use the reserved .example domain.
Sample readYou found the decryptor, cleaned the junk, typed the calls, decoded the strings, named what they are for and pulled out the configuration. That is the whole loop an analyst runs on a protected sample. The YARA lab is the natural next step: turning what you found into a rule.
Start with the most called function
Protected code hides its strings, but the program still has to read them when it runs. So somewhere there is a small routine that turns scrambled bytes into text, and it is called from every place a string is needed. That makes it the most cross-referenced small function in the file, and finding it first is how you get a foothold: everything else hangs off it.
- In the Functions window the Xrefs column counts the places that refer to each function. Click its header to sort by it.
- Select the busiest function and press X (or the Xrefs button) to list every reference. Look at what is set up just before each call.
- Open it (Enter on a call, or click it in the list), read it, then press N and give it a name that says what it does and what it works on.
The maths
Treat the program as a directed graph: each function is a node and each call is an edge from the caller to the callee. The in-degree of a node is the number of edges that point at it, and that is what the Xrefs column counts (pushed addresses and table slots are references too, so they count as well).
in-degree(f) = the number of references to f, and ∑f in-degree(f) = the number of references in the listing
Why the decryptor tops the list. A protector hides every string, so every place that needs text must go through the one routine that reveals it. u uses of hidden strings means u edges into a single node. Ordinary helpers have a few callers, and the big functions (start-up, the beacon loop) have one or none, so they sit near the bottom. Ranking by in-degree and reading the top is a one-line query, which is why tools do it for you.
Two guards make it more than a guess. The winner should be small: a decryptor is a short loop, whereas a large function with many callers is more likely a logger or a dispatcher. And in-degree only counts references the disassembler found, so a routine reached through a table shows fewer than it really has.
In practiceTools such as FLOSS and the string plugins for IDA and Ghidra find this routine automatically, by looking for small functions with many callers. Doing it by hand once shows you what they are looking for.
DefenceA detection that keys on the decryptor itself (its constants and the shape of its loop) survives a change to every string in the sample.
Instructions that do nothing
Protectors pad code with instructions that change nothing, so that a decompiler gives up and a human tires. Try the Pseudocode tab on the decryptor now: it refuses. A few patterns cover nearly all of what you will meet, and each one answers the same question: does this change anything the program uses later?
- push then pop of the same register, with nothing between.
- inc then dec (or add then sub) of the same register: net zero.
- A jump over garbage: a short jmp that skips one or two stray data bytes.
- lea or mov of a register onto itself, adding nothing.
- An opaque predicate: a test whose answer never changes (a register xor-ed with itself, then jz), with dead code on the path that never runs.
- Click an instruction in the listing to select it (drag with Shift for a range) and press J to mark it as junk.
- Check junk tells you how many you have right and wrong. When a function is exact, press Patch: the marked instructions become nops and the decompiler accepts the function.
- Two functions to clean: the decryptor, and the one that calls it most (the loader's start-up).
The maths
An instruction is junk when removing it changes nothing the program can observe. Two ideas cover every pattern in the list above.
Dead code: a value that is never read. A write to a register is dead if no later instruction reads that register before another write replaces it, and the instruction does nothing else (no memory write, no call, no flag that the next branch uses). push eax then pop eax writes and restores the same value. inc ecx then dec ecx is +1 − 1 = 0. lea esi, [esi+0] computes esi from esi.
An opaque predicate: a test that always comes out the same way. xor edi, edi sets edi to x ⊕ x = 0 whatever it held, so the zero flag is set and jz is always taken. The path that is never taken is unreachable, so everything on it is junk however important it looks. A jump over garbage is the same idea for bytes: the bytes after an unconditional jmp are never executed, they exist only so a disassembler decodes them as instructions.
bytes removed = ∑ size(i) over the junk instructions, kept = total − removed
The decompiler needs a consistent stack depth and a valid instruction at every reachable address. Junk breaks both, so it refuses until the junk is patched to nops. Removing junk never changes what the program computes, only how long the listing is.
In practiceDeobfuscation scripts run this pass automatically once someone has recognised the patterns. The skill is spotting a new one, by asking the same question of every instruction that looks pointless.
DefenceA named pattern becomes a signature: once you can describe it you can write a rule for it, or a script that strips it from every sample the same protector makes.
Who passes what, and who tidies up
A calling convention is the agreement between a caller and the function it calls: where the arguments go and who removes them from the stack afterwards. Get it wrong and the decompiler shows the wrong arguments, or none. The evidence is in the listing, and it is the same evidence every time.
| Convention | Arguments | The function ends with | After the call |
|---|---|---|---|
| cdecl | all on the stack | retn | the caller does add esp, N |
| stdcall | all on the stack | retn N | nothing: the function already removed them |
| fastcall | the first in ecx, the second in edx, the rest on the stack | retn, or retn N for stack ones | the caller sets ecx first |
| thiscall | this in ecx, the rest on the stack | retn N | the caller loads an object pointer into ecx first |
- Select a function and press Y. Choose the convention and the number of arguments (for thiscall do not count
this). - Type these four: the string decryptor; the routine with the two 256-step loops; the function whose address is pushed just before
CreateThread; and the method that callsInternetOpenA. - Open the Pseudocode tab on a typed function. When the type is wrong, the decompiler leaves a note saying what it saw.
The maths
A calling convention answers two questions: where do the arguments go, and who puts the stack pointer back. In 32-bit code each argument is pushed as one dword, 4 bytes, so n stack arguments leave 4n bytes that exactly one side must remove, and the two sides have to agree which.
cdecl: the caller runs add esp, 4n; the function ends with a plain retn
stdcall: the function ends with retn 4n
fastcall: the first two arguments travel in ecx and edx, so it ends with retn 4(n − 2), or a plain retn when n ≤ 2
thiscall: this travels in ecx, and the function ends with retn 4n for its n other arguments
The instruction retn N is three bytes (C2 and N as 16 bits), a plain retn is one (C3). In every case the stack ends where it began: stack bytes pushed = stack bytes removed. This is also why a function that takes any number of arguments, like printf, has to be cdecl: only the caller knows how many it pushed, so only the caller can clean up.
In practiceA protected binary often has no symbols and no type information, so the convention is the first thing you set on every function you rename. Wrong ones cascade: a fastcall function typed as cdecl loses its first argument everywhere it is called.
DefenceRecovered prototypes turn a listing into something a colleague can read in minutes, and make hunting for the same routine in the next sample a matter of matching a signature, not an afternoon.
Read the constants, not the strings
The decryptor works out a starting key from the string's index, then walks the bytes: each output byte is the input byte XOR-ed with the key, and the key then changes by a fixed rule. The constants in that arithmetic are visible in the listing as immediate operands. Read them out of the decryptor and put them in the Decoder tab; the strings turn into text as you get them right.
- In the decryptor, follow
ecx(the index) through the multiply, the add and the mask. That is the starting key. - Follow
ebx(the running key) inside the loop. That is how the key changes. - When all twelve read as text, press Annotate call sites so each decrypted string shows in the listing where it is used.
- Find the string that is handed to the RC4 routine in the start-up function, and leave a comment on the way it gets there.
The maths
The decryptor is a stream cipher whose keystream comes from a small recurrence. For string i the four boxes in the Decoder tab give:
k0 = (i·sm + sa) mod 256, kj+1 = (a·kj + c) mod 256, outj = inj ⊕ kj
The and 0FFh in the listing is the “mod 256”. The key recurrence is a linear congruential generator over 256 states, so it must repeat within 256 steps; the question is whether it visits all of them first. The Hull-Dobell theorem answers it. For a modulus of 28 the period is the full 256, from every starting key, if and only if c is odd and a − 1 is divisible by 4. (An odd c shares no factor with 256; 2 is the only prime factor of 256, so a − 1 must be even, and since 4 divides 256 it must be a multiple of 4.) Otherwise the key cycles through fewer values, at worst a single repeated byte.
XOR undoes itself. (x ⊕ k) ⊕ k = x, because k ⊕ k = 0 and x ⊕ 0 = x; the same routine hides and reveals. Short periods are breakable. XOR gives the keystream to anyone who knows a plaintext byte: cipher ⊕ plain = k. If the keystream repeats every P bytes, P known plaintext bytes reveal all of it, and a constant key falls to one. Even a full period is only as secret as the four constants that generate it, which the listing prints.
In practiceOnce you know the algorithm you decode in bulk with a script: a small loop in IDAPython, or FLOSS emulating the routine. Twelve strings by hand is the same job at a small scale, and the decoded strings are your first indicators: a mutex, a user agent, a path, a log file.
DefenceEvery decoded string is a detection candidate. The mutex name and user agent go into blocklists and hunts, and a protector that hides them makes them worth more, because the sample never shows them until it runs.
A name is a note to the next reader
Name a string for what the program uses it as, not for what it says. The call sites tell you: a string handed to a mutex API as the name is a mutex name. Then look at the three tables of function pointers in the read-only data. Each is a class's vtable; the constructor stores its address into a new object, and the methods that hang off it, with the system calls they make, tell you what the class is. Rename the vtable to the class name.
- Purpose over appearance:
s_user_agent, nots_mozilla. - One convention. A prefix for strings (
s_), snake_case or camelCase for functions, PascalCase for classes. - Never a default or a filler:
sub_,str1,tmpandthingsay nothing. - Unique. Two items cannot share a name.
- Open the Strings tab, select a decoded string and press N. X lists where it is used; Enter jumps to the first use.
- For a class: find a constructor (it stores an
offsetinto[eax]), follow the offset with Enter, and press N on the table.
The maths
The rubric is a set test, not a score. It has no weights: every check passes or fails. Split a name into words at underscores, other symbols and changes of case, in lower case, and call that set W. A word matches a stem if they are equal, or if the stem has five or more letters and the word starts with it (so “decrypting” matches “decrypt”).
string or class says it ⇔ W ∩ A ≠ ∅ and W ∩ N = ∅
decryptor says it ⇔ W ∩ Verb ≠ ∅ and W ∩ Noun ≠ ∅
A holds the words that fit the item’s role, and N the words that belong to another role (a mutex name must not read like a log path). Why verb plus object. A routine’s name should carry its job and its subject: “decrypt” alone fits any decryptor and “string” alone fits any string helper, and only together do they pick this one out. A name for the mechanism (“xor”) tells the next reader how it works, not what it is for.
Why one convention. Before the rubric runs, the shape is checked in this order: not empty; at most 48 characters; letters, digits, underscores and :: only, no leading digit; not a default name such as sub_401010; not the name it already has; not made only of generic words (foo, tmp, thing); unique in the database. House style is a warning, not a failure: a prefix such as s_ for strings so they sort together, one case style for functions. Consistency is what lets a reader search and sort a database they did not build.
In practiceClass-recovery tools rebuild this from run-time type information. A protected sample strips it, so you infer each class from its constructor, its vtable and what its methods do.
DefenceA consistent database is shareable: a teammate can read yours cold, and the durable things you named (a mutex, a class layout, a config format) are what the next sample gets matched against.
The settings are the prize
Malware keeps its settings encrypted and unpacks them at start-up. Read the start-up function: it passes a blob, a length and a key to the RC4 routine, then hands the result to a parser. Decrypt the blob in the RC4 tab (the key is one of your decoded strings), read the parser to learn the layout it expects, define that layout in the Structures tab, and report what the config says.
- In the start-up function, find the call to the RC4 routine. Its arguments, pushed last to first, are the data, its length, the key and the key's length.
- Enter them in the RC4 tab. The output is plain data when they are right and noise when they are not.
- Read the parser, offset by offset (
[esi+N]), and define each field in the Structures tab: offset, type, size for text, and a name. - Type what you found below. Answers are checked against sealed digests, so there is nothing on the page to copy.
The maths
RC4 keeps a table S holding the 256 byte values in some order, and two indices. The key schedule starts from the identity and stirs the key in. The generator then produces one output byte per step.
KSA: S[i] = i; then for i = 0 to 255: j ← (j + S[i] + K[i mod |K|]) mod 256, and swap S[i], S[j]
PRGA: i ← (i + 1) mod 256; j ← (j + S[i]) mod 256; swap S[i], S[j]; output S[(S[i] + S[j]) mod 256]
The output is XOR-ed with the data, so decrypting is encrypting again. The invariant: the only thing ever done to S is swapping two of its entries, so it is a permutation of 0 to 255 at every moment. Nothing is created or lost, so the output is always a valid byte and the table never collapses.
Why it is deprecated. The output is not uniform at the start: the second byte is zero about twice as often as chance (2/256, not 1/256). Keys that differ a little give related streams, which is how WEP was broken. And there is no nonce, so a reused key reuses the keystream. Why malware still uses it. The whole cipher is a dozen lines with no library and no table of constants to give it away, and a fixed key is enough to unpack a config in memory.
In practiceConfig extraction is the payoff of the whole exercise: hosts, ports, intervals and campaign ids that link samples together. Extractors for well-known families are written exactly the way you just did it, once, and then run on every new sample.
DefenceDefang indicators before sharing (the dots in brackets stop them turning into live links), record the version and campaign id so samples can be tied together, and remember that a config format outlives any one sample.
Cinderling.exe
32-bit loader · unpacked · 17 functions · a fictional program, written for this page
↑↓ move · N rename · Y type · X xrefs · ; comment · J junk · G go to · Enter follow · Esc back
i:i × + ) & 0xFFXOR keyFields of the blob, as the parser reads them. Values come from the RC4 tab's output.
| Offset | Type | Size | Name | Value | Check | Remove |
|---|