Windows API - A Deep Dive

Ever wondered how VirtualAlloc "just allocates memory"? Or how WriteProcessMemory just seamlessly writes to the memory of another process? Well, I certainly have at some point, and I promise there is no black magic involved.

Just a quick disclaimer: this post is about Windows NT syscalls. Not every single Windows API function is a syscall; in fact, only about ~400 of them are. However, some functions that most of you should be familiar with, such as WriteProcessMemory, ReadProcessMemory, VirtualProtect, and so on, are all syscalls. To get the most from this post, you should have some basic C and WinAPI knowledge, as well as knowing how to use a debugger, and what hooking does.

All disassembly snippets in this article were taken from x64dbg. I strongly advise following along in a usermode debugger of your choice. We won't dive into the kernel here, but possibly in another post.

First of all, we need a program to step through. In this example, I will use this simple C program below:

#include <Windows.h>

int main(void)
{
    PVOID ptr;
    ptr = VirtualAlloc(NULL, 0x1000, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);

    if (ptr)
    {
	      VirtualFree(ptr, 0, MEM_RELEASE);
	      ptr = NULL;
    }

    return 0;
}

Once you've compiled the program, open it with a debugger and set a breakpoint on the function you want to step through. If you're unsure which library the function belongs to, check Microsoft documentation. VirtualAlloc is located in kernel32.dll.

Working down to the syscall

Once you resume the debugger, you should hit a breakpoint and see something like this:

You may be wondering, "VirtualAlloc jumps to VirtualAlloc??" Yes, that's correct. If you look at the RIP register, you should see it points to kernel32.VirtualAlloc (not all debuggers show this). However, if we step to the next instruction, you'll see more expected output:

If you look at the RIP register, you'll see we're now in kernelbase.VirtualAlloc. This routine handles 4 things:

• Executing the syscall stub.
• Setting up the correct parameters for the syscall stub (see NT Functions)
• Returning the value from the syscall stub
• Handling any errors, and setting the value returned by GetLastError().

In a successful execution path, NtAllocateVirtualMemory gets called—this is where the magic happens.

The inside of a syscall stub looks like this:

This routine transitions to and from kernel mode. Unlike the previous routine, all syscall stubs look identical with one key difference: the value moved into the eax register (here, 0x18). This is the SSN (System Service Number), serving as a unique identifier for each system call and an index into the SSDT (System Service Dispatch Table)—an array of offsets to actual kernel routines (on 32-bit, it's an array of absolute addresses).

In the past, hooking the SSDT was common for malware and EDRs alike. However, with PatchGuard/KPP (Kernel Patch Protection) introduced in Windows Vista, hooking the SSDT will bug check (BSOD) your system. There are ways to bypass PatchGuard, but they're beyond this post's scope.

NT Functions

You'll find that syscall stub names are similar yet often different than the original function. For example, WriteProcessMemory's NT version is NtWriteVirtualMemory. However, they all begin with “Nt”.

You may wonder if there's an easy way to invoke these routines yourself. Yes, there is. All these syscalls are located in ntdll.dll, which loads in every Windows process regardless of subsystem. Although ntdll.dll contains mostly syscall stubs, there are exceptions like RtlInitUnicodeString.

You may have noticed these functions don't appear when you include the windows.h header file. Although ntddk.h contains their declarations, ntddk is intended for driver development (DDK meaning Driver Development Kit).

Before calling these functions directly, understand that although the WinAPI function calls the NT function, the NT function has additional parameters as shown bellow:

LPVOID VirtualAlloc(
    [in, optional] LPVOID lpAddress,
    [in]           SIZE_T dwSize,
    [in]           DWORD  flAllocationType,
    [in]           DWORD  flProtect
);

NTSTATUS NTAPI NtAllocateVirtualMemory(
    In HANDLE ProcessHandle,
    Inout PVOID *BaseAddress,
    In ULONG_PTR ZeroBits,
    Inout PSIZE_T RegionSize,
    In ULONG AllocationType,
    In ULONG PageProtection
);

Most importantly, they all return a NTSTATUS value. A list of NTSTATUS values can be found here.

How to use NT functions

There are 4 ways to call these functions from usermode. First, and in my opinion easiest, declare the functions yourself and statically link to ntdll.lib. You can find undocumented Windows structures and NT function definitions here. This way you can call the functions directly.

The second way uses GetProcAddress. You can create a variable holding the function pointer and call it like this:

typedef NTSTATUS(NTAPI* tNtTerminateProcess) (HANDLE ProcessHandle, NTSTATUS ExitStatus);
tNtTerminateProcess pfn_NtTerminateProcess;

pfn_NtTerminateProcess = GetProcAddress(GetModuleHandleW(L”ntdll.dll”), “NtTerminateProcess”);

if (pfn_NtTerminateProcess)
{
	// -1 == calling process
	pfn_NtTerminateProcess((HANDLE)-1, 0x1337):
}

In theory, if GetProcAddress and GetModuleHandle are not hooked, this will never fail since ntdll is usually loaded inside of every process.

The main benefit of these 2 methods is the fact that even if the WinAPI counterpart is hooked, you will still bypass that hook.

However, the main shortcoming of this method is that nothing stops an anti-virus or anti-cheat from hooking the NT function itself. In fact, this is what you'll most commonly see in the real world. To circumvent this, we can create our own syscall stub.

To use this in your own project, you need to include MASM to your build dependencies, google it.

.code

NtAllocateVirtualMemory PROC
mov r10, rcx
mov eax, 018h
syscall
ret
NtAllocateVirtualMemory ENDP 

END

However, there's a glaring issue: the SSN itself. As mentioned before, the SSN isn't static and differs for every function. Additionally, it can change per Windows version. Hardcoding the SSN isn't viable if the program will run on machines other than yours. Several solutions exist for this problem, though I won't cover them in depth here. I'd advise looking into Syswhispers, Syswhispers2, Syswhispers3, and HellsGate.

The easiest, yet counter-intuitive solution is to simply read offset 0x4 from the function address, in memory all syscalls look like this:

0x4C 0x8B 0xD1 0xB8 <4 byte SSN>

If the function points to the initial 0x4C byte, if you add 4 that gives you the SSN. The main pitfall to doing things this way is the fact that if the function gets hooked, the SSN value will be overwritten as the smallest possible jmp instruction is 5 bytes, the SSN is the 5th byte in the function.

But even if you implemented a reliable way to get each function's SSN, there's still one major pitfall: the syscall instruction is located outside of ntdll. EDRs and anti-viruses can scan for that syscall instruction outside of ntdll's address space, which is incredibly suspicious if found. To bypass that check, we can make our stub jump to a legitimate syscall instruction within ntdll. Therefore, our new stub should look like this:

.data
; you can write some code to scan for a syscall instruction, the bytes are 0x0F and 0x05
syscall_address dq 0000DEADBEEF1337h

.code

NtAllocateVirtualMemory PROC
mov r10, rcx
mov eax, 018h
jmp [syscall_address]
ret
NtAllocateVirtualMemory ENDP 

END

The only thing stopping this would be a SSDT hook, which as mentioned above is much harder, but not impossible.

Anyways I hope you learned something new from reading this, this is my first attempt at writting any form of educational material so if you have any feedback or suggestions, feel free to contact me on discord @ly5ll. But with all that aside, never give up on learning! See you in the next one.