Cryptographic/Compression Algorithms

Cryptographic/Compression Algorithms
Support HackTricks and get benefits!
Identifying Algorithms
If you ends in a code using shift rights and lefts, xors and several arithmetic operations it's highly possible that it's the implementation of a cryptographic algorithm. Here it's going to be showed some ways to identify the algorithm that it's used without needing to reverse each step.
API functions
CryptDeriveKey
If this function is used, you can find which algorithm is being used checking the value of the second parameter:
Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-id​
RtlCompressBuffer/RtlDecompressBuffer
Compresses and decompresses a given buffer of data.
CryptAcquireContext
The CryptAcquireContext function is used to acquire a handle to a particular key container within a particular cryptographic service provider (CSP). This returned handle is used in calls to CryptoAPI functions that use the selected CSP.
CryptCreateHash
Initiates the hashing of a stream of data. If this function is used, you can find which algorithm is being used checking the value of the second parameter:

Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-id​
Code constants
Sometimes it's really easy to identify an algorithm thanks to the fact that it needs to use a special and unique value.
If you search for the first constant in Google this is what you get:
Therefore, you can assume that the decompiled function is a sha256 calculator.
You can search any of the other constants and you will obtain (probably) the same result.
data info
If the code doesn't have any significant constant it may be loading information from the .data section.
You can access that data, group the first dword and search for it in google as we have done in the section before:
In this case, if you look for 0xA56363C6 you can find that it's related to the tables of the AES algorithm.
RC4 (Symmetric Crypt)
Characteristics
It's composed of 3 main parts:
Initialization stage/: Creates a table of values from 0x00 to 0xFF (256bytes in total, 0x100). This table is commonly call Substitution Box (or SBox).
Scrambling stage: Will loop through the table crated before (loop of 0x100 iterations, again) creating modifying each value with semi-random bytes. In order to create this semi-random bytes, the RC4 key is used. RC4 keys can be between 1 and 256 bytes in length, however it is usually recommended that it is above 5 bytes. Commonly, RC4 keys are 16 bytes in length.
XOR stage: Finally, the plain-text or cyphertext is XORed with the values created before. The function to encrypt and decrypt is the same. For this, a loop through the created 256 bytes will be performed as many times as necessary. This is usually recognized in a decompiled code with a %256 (mod 256).
In order to identify a RC4 in a disassembly/decompiled code you can check for 2 loops of size 0x100 (with the use of a key) and then a XOR of the input data with the 256 values created before in the 2 loops probably using a %256 (mod 256)
Initialization stage/Substitution Box: (Note the number 256 used as counter and how a 0 is written in each place of the 256 chars)
Scrambling Stage:
XOR Stage:
AES (Symmetric Crypt)
Characteristics
Use of substitution boxes and lookup tables
It's possible to distinguish AES thanks to the use of specific lookup table values (constants). Note that the constant can be stored in the binary or created dynamically.
The encryption key must be divisible by 16 (usually 32B) and usually an IV of 16B is used.
SBox constants
Serpent (Symmetric Crypt)
Characteristics
It's rare to find some malware using it but there are examples (Ursnif)
Simple to determine if an algorithm is Serpent or not based on it's length (extremely long function)
Identifying
In the following image notice how the constant 0x9E3779B9 is used (note that this constant is also used by other crypto algorithms like TEA -Tiny Encryption Algorithm).
Also note the size of the loop (132) and the number of XOR operations in the disassembly instructions and in the code example:
As it was mentioned before, this code can be visualized inside any decompiler as a very long function as there aren't jumps inside of it. The decompiled code can look like the following:
Therefore, it's possible to identify this algorithm checking the magic number and the initial XORs, seeing a very long function and comparing some instructions of the long function with an implementation (like the shift left by 7 and the rotate left by 22).
RSA (Asymmetric Crypt)
Characteristics
More complex than symmetric algorithms
There are no constants! (custom implementation are difficult to determine)
KANAL (a crypto analyzer) fails to show hints on RSA ad it relies on constants.
Identifying by comparisons
In line 11 (left) there is a +7) >> 3 which is the same as in line 35 (right): +7) / 8
Line 12 (left) is checking if modulus_len < 0x040 and in line 36 (right) it's checking if inputLen+11 > modulusLen
MD5 & SHA (hash)
Characteristics
3 functions: Init, Update, Final
Similar initialize functions
Identify
Init
You can identify both of them checking the constants. Note that the sha_init has 1 constant that MD5 doesn't have:
MD5 Transform
Note the use of more constants
CRC (hash)
Smaller and more efficient as it's function is to find accidental changes in data
Uses lookup tables (so you can identify constants)
Identify
Check lookup table constants:
A CRC hash algorithm looks like:
APLib (Compression)
Characteristics
Not recognizable constants
You can try to write the algorithm in python and search for similar things online
Identify
The graph is quiet large:
Check 3 comparisons to recognise it:
Support HackTricks and get benefits!

Reversing & Exploiting - Previous

Windows Exploiting (Basic Guide - OSCP lvl)

Unpacking binaries

Last modified 3mo ago