To complicated reverse engineering and detection, malware will sometime implement a custom virtual machine. This differs from the kind of virtual machine you might use to analyze malware, which creates an entire virtual computer and operating system. Instead, the malware might create a virtual CPU within its own process, which enables it to execute a custom machine code or scripting language.

CPUs only understand a single language: machine code, which is often referred to as native code. An x86 CPU only understands x86 machine code, and an x86_64 only understand x86_64 machine code. Every other language must either be compiled to native code, or translated to native code. For example, C++ compiles to Assembly language, which is then assembled to native code. Assembly language is a 1:1 human-readable representation of machine code. This means every assembly instruction maps to a single machine instruction.

Programming languages like C# and Java are not compiled to native code, instead, they’re compiled to custom machine code created by the language developers. These languages require a virtual machine to be installed (.NET Framework for C# and JVM for Java), which translates this custom machine code to native code.

You may be wondering what the purpose of this is. Well, the answer is code portability. Native code varies from CPU to CPU. ARM, MIPS, x86, and PowerPC are all different architecture which run difference machine code (instruction sets). Furthermore, native executable file formats differ from operating system to operating system. You can’t run ELF files on Windows, or exes on Mac.

If a developer wanted to support multiple operating systems and CPU architectures, they’d have to compile their code for every distinct platform and architecture. With virtualized languages, the language developer can instead write a virtual machine for every platform and architecture. Since the virtual machine uses the same custom language regardless of platform, the same application will work on many different platforms. This enables developers to focus on maintaining a single application, and just let the virtual machine handle the rest.

Since virtual machines allow developers to create and run code written in custom programming languages, malware developer sometimes use this to their advantage. If malware implements its own custom code, security analysts now need to figure out how it works before they can even start reverse engineering the software. While there’s plenty of tools for working with Assembly languages, C#, .NET, Python, and so on, if it’s a brand-new language, analysts may have to create brand-new tools.

Typically, malware developers will use commercial tools such as VMProtect, which are used by legitimate applications, therefore preventing security companies from simply just writing rules to detect the virtual machine itself. While some malware does implement its own custom VMs, it’s not super common. However, since tools like VMProtect are extremely complex to reverse, these challenges are built using much simpler custom virtual machines.

In this example, the code responsible for retrieving the flag is written using custom bytecode (machine code), which is translated to native code by the malware’s virtual machine (VM). Your job is to perform static analysis of the malicious executable, figure out how the VM works, extract the bytecode, then build a script to run it to retrieve the decrypted flag. Both the custom bytecode and encrypted flag is stored in the same block of memory which acts at the virtual machine’s RAM.

Recommended Environment

Host Machine: Anything with an x86_64 CPU (32-bit CPUs won’t work and ARM CPUs will require an emulator).
Virtual Machine: VMware or VirtualBox
Operating System: Windows 10 64-bit
Disassembler: Binary Ninja or Ghidra

Note: for Windows Labs the walkthrough videos will be done using Binary Ninja, with a 64-bit Windows 10 Virtual Machine running on VMWare Workstation Pro. You are free to choose your own software & hardware, just be aware that there will only be official support for the recommendations listed above, any custom setups are your own responsibility.