Taming Virtual Machine Based Code Protection – 2

In the last episode …

As you’ve probably guessed it, this is the second part of my journey to reverse engineer a virtual machine protected binary. If you haven’t read the first part[1], I encourage you to do so, because I will not repeat everything again here. While the first part dealt with explaining the virtual environment and giving an initial first look into the virtual machine’s custom instruction set, I will focus on disassembling the virtual machine code completely this time.

I might repeat some steps from the first part again, mostly because I felt that it was necessary to do so :-).

Into the battle

We already explained the environmental setup in the previous blog post and also identified the main loop, which is responsible for instruction execution.

Each iteration, an instruction is parsed and the final CALL in the left branch of figure 1 executes the instruction.

Critical functions

I covered the instruction parsing process in my last blog article a little bit. But since we are going to build a disassembler, I will explain the most important routines once again.

0x4013DF / ParseInstruction

This function is called each iteration in the loop from figure 1 and is responsible for parsing the byte codes.

Each loop, the Virtual Instruction Pointer/VIP is retrieved, pointing at the instruction to execute. Each instruction is parsed. This function is fully responsible for transforming the bytes into a further processable format. Let’s take a look at how the first three instructions are parsed:

If you are interested in understanding this format fully, I recommend you to jump to the disassembler code[2]. I will only cover the first instruction here.

So how do we get from 03 15 03 00 04 to the parsed format ?

The first byte is always the instruction id. 03 is the id for the PUSH instruction. The second byte is divided into its upper 6 bits and lower 2 bits, representing the instruction size and number of operands used for this instruction. The next bytes are used to represent a single operand. In the example above, the first operand config 00 03 00 00, is the configuration for USE 32 BIT OF REGISTER, SPECIFIED BY THE NEXT DWORD 04 00 00 00. The next DWORD is 04 00 00 00, which is the fourth virtual register. Now what is the fourth register here ? Let’s take a quick look at the instructions.

PUSH VR4
MOV VR4, VR7
SUB VR7, 0xB4

This looks very similar to the usual function prologue ;-). So the fourth register must be EBP!.

PUSH EBP
MOV EBP, ESP
SUB ESP, 0xB4

0x401271 / GetOpval & 0x401322 / StoreOpval

I will not cover these two functions in depth here. If you take a look at figure 3 again, you will see that I mention the operand configs. These functions are responsible for filling the operands according to these configs.

In the example above, the SUB VR7, 0xB4 instruction uses 00030000 07000000 for the first operand and 00020000 B4000000 for the second config. If you reverse engineer every single option, you will find out that the following configurations exist:

# First DWORD CONFIG
00000000 ==> LOWEST BYTE OF REG X # f.e AX
00010000 ==> SECOND LOWEST BYTE OF REG X # f.e. AH 
00020000 ==> LOWER 16 BIT OF REG X # f.e. AX
00030000 ==> 32 BIT OF REGX # f.e. EAX
01000000 ==> BYTE AT LOC
01010000 ==> BYTE AT LOC
01020000 ==> WORD AT LOC
01030000 ==> DWORD AT LOC
02000000 == BYTE FROM IMM.
02010000 ==> BYTE FROM IMM.
02020000 ==> WORD FROM IMM.
02030000 ==> DWORD FROM IMM.
# Second DWORD CONFIG, if register
00000000 ==> EAX
01000000 ==> EBX
02000000 ==> ECX
03000000 ==> EDX
04000000 ==> EBP
05000000 ==> ESI
06000000 ==> EDI
07000000 ==> ESP

Eternal Debugging

Now we can use the gained knowledge to gain an initial understanding of what is happening and to verify whether we are able to decode instructions manually.

If you take a look at the last instructions, you will see that there are some constants pushed into memory. If you google these constants, you will come to the conclusion that this must be the MD5 Init routine[3]. The next step is to build a disassembler.

Disassembling the code

I wrote this one in C++ and you can find the source code to it on my github page[4]. Writing this on Python would have been possible too … and probably a lot easier and faster, I chose C++ though for learning purposes. If my C++ is awful, forgive me. We all start somewhere ;-).

Our disassembler does have some limitations though. The disassembly was complex and I believe that some memory address offsets and register sizes are wrong. Also, I did not reverse engineer all instructions. However though, that should not be a problem, because we only need to understand what is happening here on a higher level.

Identifying the algorithm

We already spotted the variables, which we also found in the MD5.c source code(f.e. 0x2381bc0). However, the actual hashing algorithm does not match the original one. Therefore it seems to be some kind of a modified version of it. Furthermore we spot a routine, which seems to be the XTEA algorithm[5].

Final words

So that’s basically it. I don’t know when and if I will a third part covering the serial key generator. When I started this challenge, I was only interested in learning how to disassemble custom instruction sets.

If you are interested in how others solved this challenge, I recommend you to read the tutorials from wagonono and kernelj, they both completely solved this challenge[6]. Wagonono also created a disassembler and his version is better than mine.