Everything runs in this tab, and your names and comments stay in this browser. Nothing here is real malware: we wrote the listing for this workshop, every value is invented, and the hosts use the reserved .example domain.
Rename this
A protector wraps a program so its strings, structure and names tell you nothing. This workshop is the analyst's side of that: a fictional loader called Cinderling in a disassembler-style workbench, and six jobs that take it from unreadable to a config you could hand to a defender.
Sample readThat's the whole loop an analyst runs on a protected sample. If you want to turn what you found into a rule, the YARA lab is next door.
Start with the most called function
Protected code hides its strings, but the program still has to read them when it runs. So somewhere there's a small routine that turns scrambled bytes into text, and every place that needs a string calls it. That makes it the most cross-referenced small function in the file. Find it first and everything else hangs off it.
- In the Functions window the Xrefs column counts the places that refer to each function. Click its header to sort by it.
- Select the busiest function and press X (or the Xrefs button) to list every reference. Look at what each caller sets up just before the call.
- Open it (Enter on a call, or click it in the list), read it, then press N and name it for what it does and what it works on.
The maths
Treat the program as a directed graph: functions are nodes, calls are edges from caller to callee. A node's in-degree is how many edges point at it, and that's what the Xrefs column counts (pushed addresses and table slots too).
in-degree(f) = the number of references to f, and ∑f in-degree(f) = the number of references in the listing
Why the decryptor tops the list. A protector hides every string, so every place that needs text goes through the one routine that reveals it. u uses of hidden strings means u edges into a single node. Ordinary helpers have a few callers, and the big functions (start-up, the beacon loop) have one or none. Ranking by in-degree is a one-line query, which is why tools do it for you.
Two checks keep it from being a guess. The winner should be small: a decryptor is a short loop, and a large function with many callers is more likely a logger. And in-degree counts only the references the disassembler found, so a routine reached through a table shows fewer than it has.
In practiceFLOSS and the string plugins for IDA and Ghidra find this routine for you by looking for small functions with many callers. Do it by hand once and you'll know what they're looking for.
DefenceA detection that keys on the decryptor itself (its constants and the shape of its loop) still fires after every string in the sample changes.
Instructions that do nothing
Protectors pad code with instructions that change nothing, so a decompiler gives up and a human tires. Try the Pseudocode tab on the decryptor now: it refuses. A few patterns cover most of what you'll meet, and each comes down to whether the instruction changes anything the program uses later.
- push then pop of the same register, with nothing between.
- inc then dec (or add then sub) of the same register: net zero.
- A jump over garbage: a short jmp that skips one or two stray data bytes.
- lea or mov of a register onto itself, adding nothing.
- An opaque predicate: a test whose answer never changes (a register xor-ed with itself, then jz), with dead code on the path that never runs.
- Click an instruction in the listing to select it (drag with Shift for a range) and press J to mark it as junk.
- Check junk tells you how many you have right and wrong. When a function is exact, press Patch: the marked instructions become nops and the decompiler accepts the function.
- Two functions to clean: the decryptor, and the one that calls it most (the loader's start-up).
The maths
An instruction is junk when removing it changes nothing the program can observe. Two ideas cover every pattern above.
Dead code: a value that is never read. A write to a register is dead if nothing reads it before the next write replaces it, and the instruction has no other effect (no memory write, no call, no flag a branch uses). push eax then pop eax writes and restores the same value. inc ecx then dec ecx is +1 − 1 = 0. lea esi, [esi+0] computes esi from esi.
An opaque predicate: a test that always comes out the same way. xor edi, edi sets edi to x ⊕ x = 0 whatever it held, so the zero flag is set and jz is always taken. The path that is never taken is unreachable, so everything on it is junk however important it looks. A jump over garbage is the same idea for bytes: the bytes after an unconditional jmp never run, and exist only so a disassembler decodes them as instructions.
bytes removed = ∑ size(i) over the junk instructions, kept = total − removed
The decompiler needs a consistent stack depth and a valid instruction at every reachable address. Junk breaks both, so it refuses until you patch the junk to nops. Removing junk never changes what the program computes, only how long the listing is.
In practiceDeobfuscation scripts run this pass automatically once someone has recognised the patterns. The skill is spotting a new one: ask the same question of every instruction that looks pointless.
DefenceA named pattern becomes a signature: describe it once and you can write a rule for it, or a script that strips it from every sample the same protector makes.
Who passes what, and who tidies up
A calling convention is the deal between a caller and the function it calls: where the arguments go and who cleans them off the stack afterwards. Get it wrong and the decompiler shows the wrong arguments, or none. The clues are in the listing, and they're the same every time.
| Convention | Arguments | The function ends with | After the call |
|---|---|---|---|
| cdecl | all on the stack | retn | the caller does add esp, N |
| stdcall | all on the stack | retn N | nothing: the function already removed them |
| fastcall | the first in ecx, the second in edx, the rest on the stack | retn, or retn N for stack ones | the caller sets ecx first |
| thiscall | this in ecx, the rest on the stack | retn N | the caller loads an object pointer into ecx first |
- Select a function and press Y. Choose the convention and the number of arguments (for thiscall, don't count
this). - Type these four: the string decryptor, the routine with the two 256-step loops, the function whose address is pushed just before
CreateThread, and the method that callsInternetOpenA. - Open the Pseudocode tab on a typed function. When the type is wrong, the decompiler leaves a note saying what it saw.
The maths
A calling convention says where the arguments go and who puts the stack pointer back. In 32-bit code each argument is a 4-byte dword, so n stack arguments leave 4n bytes that exactly one side must remove.
cdecl: the caller runs add esp, 4n; the function ends with a plain retn
stdcall: the function ends with retn 4n
fastcall: the first two arguments travel in ecx and edx, so it ends with retn 4(n − 2), or a plain retn when n ≤ 2
thiscall: this travels in ecx, and the function ends with retn 4n for its n other arguments
The instruction retn N is three bytes (C2 and N as 16 bits), and a plain retn is one (C3). Either way the stack ends where it began. A function with a variable number of arguments, like printf, has to be cdecl, because only the caller knows how many it pushed.
In practiceA protected binary often has no symbols and no type information, so the convention is the first thing you set on every function you rename. Wrong ones cascade: type a fastcall function as cdecl and it loses its first argument at every call site.
DefenceRecovered prototypes make a listing readable to a colleague in minutes, and turn finding the same routine in the next sample into a signature match.
Read the constants out of the decryptor
The decryptor derives a starting key from the string's index, XORs each byte with the key, then updates the key by a fixed rule. The constants in that arithmetic show up in the listing as immediate operands. Read them out, put them in the Decoder tab, and the strings turn into text as you get them right.
- In the decryptor, follow
ecx(the index) through the multiply, the add and the mask. That's the starting key. - Follow
ebx(the running key) inside the loop. That's how the key changes. - When all twelve read as text, press Annotate call sites so each decrypted string shows up in the listing where it's used.
- Find the string the start-up function hands to the RC4 routine, and leave a comment on the way it gets there.
The maths
The decryptor is a stream cipher whose keystream comes from a small recurrence. For string i the four boxes in the Decoder tab give:
k0 = (i·sm + sa) mod 256, kj+1 = (a·kj + c) mod 256, outj = inj ⊕ kj
The and 0FFh in the listing is the “mod 256”. The key recurrence is a linear congruential generator over 256 states, so it has to repeat within 256 steps. What matters is whether it visits all of them first. The Hull-Dobell theorem settles that. For a modulus of 28 the period is the full 256, from every starting key, if and only if c is odd and a − 1 is divisible by 4. (An odd c shares no factor with 256. Since 2 is the only prime factor of 256 and 4 divides it, a − 1 must be a multiple of 4.) Otherwise the key cycles through fewer values, at worst a single repeated byte.
XOR undoes itself. (x ⊕ k) ⊕ k = x, because k ⊕ k = 0 and x ⊕ 0 = x, so the same routine hides and reveals. Short periods are breakable. XOR gives the keystream to anyone who knows a plaintext byte: cipher ⊕ plain = k. If the keystream repeats every P bytes, P known plaintext bytes reveal all of it, and a constant key falls to one. Even a full period is only as secret as the four constants that generate it, and the listing prints them.
In practiceOnce you know the algorithm you decode in bulk with a script, such as an IDAPython loop or FLOSS emulating the routine. The decoded strings are your first indicators: a mutex, a user agent, a path, a log file.
DefenceEvery decoded string is a detection candidate. The mutex name and user agent go into blocklists and hunts, and are worth more because the sample hides them until it runs.
Name each thing for its job
Name a string for what the program uses it as. The call sites tell you: a string passed to a mutex API as the name is a mutex name. Then look at the three tables of function pointers in the read-only data. Each is a class's vtable, and the methods in it, plus the system calls they make, tell you what the class is.
- Purpose over appearance:
s_user_agent, nots_mozilla. - One convention. A prefix for strings (
s_), snake_case or camelCase for functions, PascalCase for classes. - Never a default or a filler:
sub_,str1,tmpandthingsay nothing. - Unique. Two items can't share a name.
- Open the Strings tab, select a decoded string and press N. X lists where it is used, and Enter jumps to the first use.
- For a class: find a constructor (it stores an
offsetinto[eax]), follow the offset with Enter, and press N on the table.
The maths
The rubric is a set test with no weights: every check passes or fails. Split a name into lower-case words at underscores, other symbols and changes of case, and call that set W. A word matches a stem if they are equal, or if the stem has five or more letters and the word starts with it (so “decrypting” matches “decrypt”).
string or class says it ⇔ W ∩ A ≠ ∅ and W ∩ N = ∅
decryptor says it ⇔ W ∩ Verb ≠ ∅ and W ∩ Noun ≠ ∅
A holds the words that fit the item’s role, and N the words that belong to another role (a mutex name must not read like a log path). Why verb plus object. A routine’s name should carry its job and its subject: “decrypt” alone fits any decryptor, and only with “string” does it pick this one out. A name for the mechanism (“xor”) says how it works but not what it is for.
Why one convention. Before the rubric runs, the page checks the name's shape: not empty, at most 48 characters, letters, digits, underscores and :: only, no leading digit, not a default such as sub_401010, not generic (foo, tmp, thing), and unique. House style (a s_ prefix for strings, one case style for functions) only warns. Consistency lets someone search and sort a database they didn't build.
In practiceClass-recovery tools rebuild this from run-time type information. A protected sample strips it, so you infer each class from its constructor, its vtable and what its methods do.
DefenceA consistent database is shareable, and the durable things you named (a mutex, a class layout, a config format) are what the next sample gets matched against. the next reader is usually you in three weeks lol.
Get the settings out of the blob
Malware keeps its settings encrypted and unpacks them at start-up. The start-up function passes a blob, a length and a key to the RC4 routine, then hands the result to a parser. Decrypt the blob in the RC4 tab (the key is one of your decoded strings), define the parser's layout in the Structures tab, and report what the config says.
- In the start-up function, find the call to the RC4 routine. Its arguments, pushed last to first, are the data, its length, the key and the key's length.
- Enter them in the RC4 tab. The output is plain data when they're right and noise when they're not.
- Read the parser, offset by offset (
[esi+N]), and define each field in the Structures tab: offset, type, size for text, and a name. - Type what you found below. Answers are checked against sealed digests, so there's nothing on the page to copy.
The maths
RC4 keeps a table S holding the 256 byte values in some order, and two indices. The key schedule starts from the identity and stirs the key in. The generator then produces one output byte per step.
KSA: S[i] = i; then for i = 0 to 255: j ← (j + S[i] + K[i mod |K|]) mod 256, and swap S[i], S[j]
PRGA: i ← (i + 1) mod 256; j ← (j + S[i]) mod 256; swap S[i], S[j]; output S[(S[i] + S[j]) mod 256]
The output is XOR-ed with the data, so decrypting is encrypting again. The invariant: the only thing ever done to S is swapping two of its entries, so it's a permutation of 0 to 255 at every moment. Nothing is created or lost, so the output is always a valid byte and the table never collapses.
Why it's deprecated. The output is biased at the start: the second byte is zero about twice as often as chance (2/256). Keys that differ a little give related streams, and with no nonce a reused key reuses the keystream. Why malware still uses it. The whole cipher is a dozen lines with no library and no table of constants to give it away, and a fixed key is enough to unpack a config in memory.
In practiceConfig extraction is the payoff: hosts, ports, intervals and campaign ids that link samples together. Extractors for well-known families work the way you just did it: written once, then run on every new sample.
DefenceDefang indicators before sharing, record the version and campaign id so samples can be linked, and keep the config format: it outlives any one sample.
Cinderling.exe
32-bit loader · unpacked · 17 functions · a fictional program, written for this page
↑↓ move · N rename · Y type · X xrefs · ; comment · J junk · G go to · Enter follow · Esc back
i:i × + ) & 0xFFXOR keyFields of the blob, as the parser reads them. Values come from the RC4 tab's output.
| Offset | Type | Size | Name | Value | Check | Remove |
|---|