Overcoming obfuscation in binaries has always been an interesting topic for me, especially in combination with malware. Over the last weeks I’ve been playing around with Virtualised Code Protection in order to see how well I could handle it.
I decided to download a simple crack-me challenge which is obfuscated with this technique. It takes me some time to reverse everything, so there will be atleast 2 blog articles about my little project.
Virtualised Code Protection
Each architecture has a defined instruction set. By looking up the instructions to the corresponding bytes, we are able to translate these bytes into disassembly. The unit that actually executes these bytes is the CPU.
Virtual machine based code protection emulates a processor and thus switches our usual instruction set against a custom one. So in order to really understand what a virtual machine hardened binary is doing on a low level basis, we need to reverse the virtual machine first. This means we have to understand the custom instruction set.
I want to show you a practical example of how such a custom instruction can look like and be discovered.
Practical Example
Preparing the virtual machine
The challenge demands a serial key and a username. Both of them need certain values for the serial key to be valid. After entering a username and a serial key, the length of both of them are checked first.
Next At the bottom of this routine, we can already spot 2 interesting functions and operations which push the success or failure message onto the stack.
The function InitialiseVM is where it gets interesting for us. If you just look quickly through the disassembly in the figure below, you will see that there are multiple buffers allocated and static values written into an internal structure. Furthermore it is filled with function pointers. Each one of those functions represents a custom instruction. This routine is used to allocate the virtual address space our virtual machine will use for emulation, as well as a table to select custom instructions from.
Next is the CheckSerial function, which implements the virtual machine loop that emulates the virtual processor unit.
In the block at loc_4015E5 the function sub_4013DF is executed each iteration. Afterwards the byte which the address in ESI+0x7C points to is used to calculate the dynamic call at the end of the current block we are talking about (call dword ptr [esi+eax*4+80h]). That means that the byte influencing which function to enter, is deciding which custom instruction to execute. Before we look at how some of the opcodes are actually parsed here, let’s review how the virtualised address space of this VM looks like.
Executing custom instructions
The function sub_4013DF is called each iteration and reads bytes from the buffer which contains opcodes for custom instructions. The first one has a size of 5 bytes. Each of them is used by the virtual machine for translating these opcodes into a valid operation. At the moment of writing this article, I did not fully explore this function yet. However, I am confident that the last 2 bytes of an instruction are used to influence registers.
Upon returning from this function, the program takes the first byte of the ESI+0x7C structure and uses it to determine which function from the previously allocated function table is called. The first run returns EAX=3, so we are dealing with the custom instruction with instruction id 3.
Let’s jump into our first custom instruction.
The function sub_401271 has 31 XRefs and is used in every function from the function table. Before the function is called, the pointer to ESI+7C, our 0x24 buffer holding the custom opcodes are retrieved.0xC is added, that means we are pointing at the byte at ESI+7C+0xC, the 4th DWORD in this buffer.
The routine accesses the third byte of the current opcode and is responsible for determining the instruction type. The first four bits decide wether it is an instruction utilizing 2 registers, a memory read or moving an immediate value into a register. The second 4 bits influence the size of the byte that will be moved around. These 4 bits are zero extended into bytes.
Take a look at the figure below. The result of our InstrType function is saved in ebp+0x4. Next the memory address which ESI+0x20 points at is decreased and filled with the value we just computed. Doesn’t this look familiar ? The stack is also decreased if we put data onto it.
It seems that the custom instruction we just investigated is a custom PUSH instruction. ESI+0x20 points to the virtual stack that is emulated by this virtual machine. Since the pointer at ESI+0x4C is increased here after an instruction, it might hold the virtual instruction pointer.
So far we figured out what the first 3 opcodes do and we have an idea what the last 2 ones are responsible for. In order to give a proper answer on how they are used, it is needed to look at more than just 1 virtual instruction execution.
Conclusion
So it just took me a complete blog article to really explain how to reverse a single custom instruction of a binary hardened with Virtualised Code Protection ;-). As you can see, this kind of software protection is very powerful.
I will finish this challenge for sure and will write a second blog article about how I solved it.
How do you get an ip adress out of it ?
The easiest way would be to intercept network traffic if it tries reach the ip address during execution. If you collect the network traffic with f.e. Wireshark or tcpdump, you should be able to see the ip address in the pcap file.