HackTricks

Searchâ€¦

Linux/Unix

Windows

Mobile Apps Pentesting

Pentesting

Pentesting Web

Forensics

A.I. Exploiting

Blockchain

Courses and Certifications Reviews

Cloud Security

Physical attacks

Reversing

Cryptography

Powered By GitBook

Cryptographic/Compression Algorithms

Identifying Algorithms

If you ends in a code** using shift rights and lefts, xors and several arithmetic operations** it's highly possible that it's the implementation of a **cryptographic algorithm**. Here it's going to be showed some ways to **identify the algorithm that it's used without needing to reverse each step**.

API functions

CryptDeriveKey

If this function is used, you can find which **algorithm is being used** checking the value of the second parameter:

Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-idâ€‹

RtlCompressBuffer/RtlDecompressBuffer

Compresses and decompresses a given buffer of data.

CryptAcquireContext

The **CryptAcquireContext** function is used to acquire a handle to a particular key container within a particular cryptographic service provider (CSP). **This returned handle is used in calls to CryptoAPI **functions that use the selected CSP.

CryptCreateHash

Initiates the hashing of a stream of data. If this function is used, you can find which **algorithm is being used** checking the value of the second parameter:

Check here the table of possible algorithms and their assigned values: https://docs.microsoft.com/en-us/windows/win32/seccrypto/alg-idâ€‹

Code constants

Sometimes it's really easy to identify an algorithm thanks to the fact that it needs to use a special and unique value.

If you search for the first constant in Google this is what you get:

Therefore, you can assume that the decompiled function is a **sha256 calculator.**
You can search any of the other constants and you will obtain (probably) the same result.

data info

If the code doesn't have any significant constant it may be **loading information from the .data section**.
You can access that data, **group the first dword** and search for it in google as we have done in the section before:

In this case, if you look for **0xA56363C6** you can find that it's related to the **tables of the AES algorithm**.

RC4 **(Symmetric Crypt)**

Characteristics

It's composed of 3 main parts:

Use of **substitution boxes and lookup tables**

It's possible to **distinguish AES thanks to the use of specific lookup table values** (constants). *Note that the **constant **can be **stored **in the binary **or created** **dynamically**.*

The **encryption key** must be **divisible **by **16 **(usually 32B) and usually an **IV **of 16B is used.

SBox constants

Serpent **(Symmetric Crypt)**

Characteristics

It's rare to find some malware using it but there are examples (Ursnif)

Simple to determine if an algorithm is Serpent or not based on it's length (extremely long function)

Identifying

In the following image notice how the constant **0x9E3779B9** is used (note that this constant is also used by other crypto algorithms like **TEA **-Tiny Encryption Algorithm).
Also note the **size of the loop** (**132**) and the **number of XOR operations** in the **disassembly **instructions and in the **code **example:

As it was mentioned before, this code can be visualized inside any decompiler as a **very long function** as there **aren't jumps** inside of it. The decompiled code can look like the following:

Therefore, it's possible to identify this algorithm checking the **magic number** and the** initial XORs**, seeing a **very long function** and **comparing **some **instructions **of the long function **with an implementation** (like the shift left by 7 and the rotate left by 22).

RSA **(Asymmetric Crypt)**

Characteristics

More complex than symmetric algorithms

There are no constants! (custom implementation are difficult to determine)

KANAL (a crypto analyzer) fails to show hints on RSA ad it relies on constants.

Identifying by comparisons

In line 11 (left) there is a

`+7) >> 3`

which is the same as in line 35 (right): `+7) / 8`

Line 12 (left) is checking if

`modulus_len < 0x040`

and in line 36 (right) it's checking if `inputLen+11 > modulusLen`

MD5 & SHA (hash)

Characteristics

3 functions: Init, Update, Final

Similar initialize functions

Identify

Init

You can identify both of them checking the constants. Note that the sha_init has 1 constant that MD5 doesn't have:

MD5 Transform

Note the use of more constants

CRC (hash)

Smaller and more efficient as it's function is to find accidental changes in data

Uses lookup tables (so you can identify constants)

Identify

Check **lookup table constants**:

â€‹

A CRC hash algorithm looks like:

â€‹

APLib (Compression)

Characteristics

Not recognizable constants

You can try to write the algorithm in python and search for similar things online

Identify

The graph is quiet large:

Check **3 comparisons to recognise it**:

Last modified 10mo ago

Copy link

Contents

Identifying Algorithms

API functions

Code constants

data info

RC4 (Symmetric Crypt)

Characteristics

Initialization stage/Substitution Box:
(Note the number 256 used as counter and how a 0 is written in each place of the 256 chars)

Scrambling Stage:

XOR Stage:

AES (Symmetric Crypt)

Characteristics

SBox constants

Serpent (Symmetric Crypt)

Characteristics

Identifying

RSA (Asymmetric Crypt)

Characteristics

Identifying by comparisons

MD5 & SHA (hash)

Characteristics

Identify

CRC (hash)

Identify

APLib (Compression)

Characteristics

Identify