Inline Hooking for Programmers (Part 2: Writing a Hooking Engine)

We’ll be writing a hooking engine using trampoline based hooks as explained in the previous article (we don’t handle relative instructions as they’re very rare, but we do use atomic write operations to prevent race conditions).

First things first, we need to define the proxy functions which we will redirect the hooked functions to, these must have the same calling convention, return type, and parameters as the functions we are going to hook with them. For this example we will simply have them print out the parameters before displaying the message box.

OldMessageBox is simply a typedef that will point to 25 bytes of executable memory which the hooking function will store the trampoline into.

Now for the hooking function, we will have the following parameters:

  • name – The name of the function to hook.
  • dll – The dll the target function resides in.
  • proxy – a pointer to the proxy function (NewMessageBox).
  • original – A pointer to 25 bytes of executable memory, where we will store the trampoline.
  • length – A pointer to a variable which receives the number of bytes worth of instructions stored in the trampoline (remember we can only copy whole instructions). 

Inside the hooking function we will get the address of the target function, then use the “Hacker Dissasembler Engine (HDE32)” to dissasemble each instruction and get the length, until we have 5 or more bytes worth of whole instructions (hde32_disasm returns the length of the instruction pointed to by the first parameter).

To build the actual trampoline we first copy “TrampolineLength” of bytes from the target function to the trampoline buffer (passed to the function in the parameter “original”), then we append the copied bytes with a jump to n bytes into target function (n is TrampolineLength e.g. resume execution in the target function where the trampoline left off).

A relative jump is the distance from the end of the jump, that is: (destination – (source + 5)). The source of the jump will be the trampoline address + TrampolineLength and the destination will be the hooked function + TrampolineLength.

Before we can write the jump to the function, we need to make sure the memory is writable (it’s usually not), we do this by setting the protection to PAGE_EXECUTE_READWRITE using VirtualProtect.

To place the hook all we need to do is create a jump to jump from the target function to the proxy, then we can overwrite the first 5 bytes of the target with it. To avoid any risk of the function being called while we’re writing the jump, we must write all of it at once (atomically). Sadly atomic functions can only work with sizes of base 2 (2, 4, 8, 16, etc); our jump is 5 bytes and the closest size we can copy is 8, so we will have to make a custom function (SafeMemcpyPadded) that will pad the source buffer to 8 bytes with bytes from the destination, so that the last 3 bytes remain unchanged after the copy.

cmpxchg8b compares the 8 bytes held in edx:eax, with the destination, if they’re equal it copies the 8 bytes held in ecx:ebx, we set edx:eax to the destination bytes so that the copy always happens.

All that’s left to do now is restore the page protection. flush the instruction cache, and set the “length” parameter to TrampolineLength.

The hooking function can simply be called like so.

Unhooking is done by copying “length” bytes to the hooked function from OldMessageBox (the trampoline).

You can see my full hooking engine, including example usage, on GitHub.