HackTricks

Searchâ€¦

Linux/Unix

Windows

Mobile Apps Pentesting

Pentesting

Pentesting Web

Forensics

A.I. Exploiting

Blockchain

Courses and Certifications Reviews

Physical attacks

Reversing

Cryptography

Cryptographic/Compression Algorithms

Identifying Algorithms

If you ends in a code** using shift rights and lefts, xors and several arithmetic operations** it's highly possible that it's the implementation of a **cryptographic algorithm**. Here it's going to be showed some ways to **identify the algorithm that it's used without needing to reverse each step**.

API functions

CryptDeriveKey

If this function is used, you can find which **algorithm is being used** checking the value of the second parameter:

Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-idâ€‹

RtlCompressBuffer/RtlDecompressBuffer

Compresses and decompresses a given buffer of data.

CryptAcquireContext

The **CryptAcquireContext** function is used to acquire a handle to a particular key container within a particular cryptographic service provider (CSP). **This returned handle is used in calls to CryptoAPI **functions that use the selected CSP.

CryptCreateHash

Initiates the hashing of a stream of data. If this function is used, you can find which **algorithm is being used** checking the value of the second parameter:

Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-idâ€‹

Code constants

Sometimes it's really easy to identify an algorithm thanks to the fact that it needs to use a special and unique value.

If you search for the first constant in Google this is what you get:

Therefore, you can assume that the decompiled function is a **sha256 calculator.**
You can search any of the other constants and you will obtain (probably) the same result.

data info

If the code doesn't have any significant constant it may be **loading information from the .data section**.
You can access that data, **group the first dword** and search for it in google as we have done in the section before:

In this case, if you look for **0xA56363C6** you can find that it's related to the **tables of the AES algorithm**.

RC4 **(Symmetric Crypt)**

Characteristics

It's composed of 3 main parts:

**Initialization stage/**: Creates a**table of values from 0x00 to 0xFF**(256bytes in total, 0x100). This table is commonly call**Substitution Box**(or SBox).**Scrambling stage**: Will**loop through the table**crated before (loop of 0x100 iterations, again) creating modifying each value with**semi-random**bytes. In order to create this semi-random bytes, the RC4**key is used**. RC4**keys**can be**between 1 and 256 bytes in length**, however it is usually recommended that it is above 5 bytes. Commonly, RC4 keys are 16 bytes in length.**XOR stage**: Finally, the plain-text or cyphertext is**XORed with the values created before**. The function to encrypt and decrypt is the same. For this, a**loop through the created 256 bytes**will be performed as many times as necessary. This is usually recognized in a decompiled code with a**%256 (mod 256)**.

- Use of
**substitution boxes and lookup tables**- It's possible to
**distinguish AES thanks to the use of specific lookup table values**(constants).*Note that the**constant**can be**stored**in the binary**or created**dynamically**.*

- The
**encryption key**must be**divisible**by**16**(usually 32B) and usually an**IV**of 16B is used.

SBox constants

Serpent **(Symmetric Crypt)**

Characteristics

- It's rare to find some malware using it but there are examples (Ursnif)
- Simple to determine if an algorithm is Serpent or not based on it's length (extremely long function)

Identifying

In the following image notice how the constant **0x9E3779B9** is used (note that this constant is also used by other crypto algorithms like **TEA **-Tiny Encryption Algorithm).
Also note the **size of the loop** (**132**) and the **number of XOR operations** in the **disassembly **instructions and in the **code **example:

As it was mentioned before, this code can be visualized inside any decompiler as a **very long function** as there **aren't jumps** inside of it. The decompiled code can look like the following:

Therefore, it's possible to identify this algorithm checking the **magic number** and the** initial XORs**, seeing a **very long function** and **comparing **some **instructions **of the long function **with an implementation** (like the shift left by 7 and the rotate left by 22).

RSA **(Asymmetric Crypt)**

Characteristics

- More complex than symmetric algorithms
- There are no constants! (custom implementation are difficult to determine)
- KANAL (a crypto analyzer) fails to show hints on RSA ad it relies on constants.

Identifying by comparisons

- In line 11 (left) there is a
`+7) >> 3`

which is the same as in line 35 (right):`+7) / 8`

- Line 12 (left) is checking if
`modulus_len < 0x040`

and in line 36 (right) it's checking if`inputLen+11 > modulusLen`

MD5 & SHA (hash)

Characteristics

- 3 functions: Init, Update, Final
- Similar initialize functions

Identify

Init

You can identify both of them checking the constants. Note that the sha_init has 1 constant that MD5 doesn't have:

MD5 Transform

Note the use of more constants

CRC (hash)

- Smaller and more efficient as it's function is to find accidental changes in data
- Uses lookup tables (so you can identify constants)

Identify

Check **lookup table constants**:

â€‹

A CRC hash algorithm looks like:

â€‹

APLib (Compression)

Characteristics

- Not recognizable constants
- You can try to write the algorithm in python and search for similar things online

Identify

The graph is quiet large:

Check **3 comparisons to recognise it**:

Last modified 1yr ago

Copy link

Contents

Identifying Algorithms

API functions

Code constants

data info

RC4 (Symmetric Crypt)

Characteristics

Initialization stage/Substitution Box:
(Note the number 256 used as counter and how a 0 is written in each place of the 256 chars)

Scrambling Stage:

XOR Stage:

AES (Symmetric Crypt)

Characteristics

SBox constants

Serpent (Symmetric Crypt)

Characteristics

Identifying

RSA (Asymmetric Crypt)

Characteristics

Identifying by comparisons

MD5 & SHA (hash)

Characteristics

Identify

CRC (hash)

Identify

APLib (Compression)

Characteristics

Identify