API hooking is one of the most powerful and widely abused techniques in modern Windows security, sitting at the intersection of offensive tradecraft, malware development, and defensive engineering. Whether you are a red teamer building stealthy payloads, a malware analyst reversing a rootkit, or a blue teamer tuning EDR detection logic, understanding how hooks work at every layer of the Windows architecture is not optional - it is foundational.
This guide breaks down the three primary hook types (IAT, inline, and kernel-level), shows how attackers exploit each, and walks through concrete detection and defense strategies that actually hold up in practice.
API hooking intercepts calls between software components, redirecting execution flow before, during, or after a target function runs. On Windows, this almost always means intercepting calls into the Win32 API, the Native API (ntdll.dll), or the kernel itself.
Attackers use hooks to:
Defenders and security vendors use hooks for precisely the opposite reasons: to observe, log, and block suspicious API call chains in real time.
Understanding both sides of this technique is what separates a practitioner from someone who just runs tools. If you want to go deep on Windows internals exploitation and hook-based offense, the Windows Red Teaming Extreme course at Redfox Cybersecurity Academy covers this material with hands-on labs that go far beyond theory.
Every PE (Portable Executable) file that imports functions from external DLLs has an Import Address Table. At load time, the Windows loader resolves the actual memory addresses of imported functions and writes them into the IAT. When your code calls CreateFile, it is really doing an indirect call through a pointer in the IAT.
An IAT hook replaces one of those pointers with the address of attacker-controlled code.
The process requires three steps: locating the IAT of the target process, finding the entry for the function you want to hook, and overwriting it.
Here is a C implementation targeting MessageBoxA in a remote process:
#include <windows.h>
#include <stdio.h>
// Hook replacement function
int WINAPI HookedMessageBoxA(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
printf("[HOOK] MessageBoxA intercepted: %s\n", lpText);
// Optionally call the original or silently drop it
return IDOK;
}
void PatchIAT(HMODULE hModule, const char* targetDLL, const char* targetFunc, LPVOID hookFunc) {
PIMAGE_DOS_HEADER dosHeader = (PIMAGE_DOS_HEADER)hModule;
PIMAGE_NT_HEADERS ntHeaders = (PIMAGE_NT_HEADERS)((BYTE*)hModule + dosHeader->e_lfanew);
PIMAGE_IMPORT_DESCRIPTOR importDesc = (PIMAGE_IMPORT_DESCRIPTOR)(
(BYTE*)hModule + ntHeaders->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_IMPORT].VirtualAddress
);
for (; importDesc->Name; importDesc++) {
char* dllName = (char*)((BYTE*)hModule + importDesc->Name);
if (_stricmp(dllName, targetDLL) != 0) continue;
PIMAGE_THUNK_DATA thunk = (PIMAGE_THUNK_DATA)((BYTE*)hModule + importDesc->FirstThunk);
PIMAGE_THUNK_DATA origThunk = (PIMAGE_THUNK_DATA)((BYTE*)hModule + importDesc->OriginalFirstThunk);
for (; thunk->u1.Function; thunk++, origThunk++) {
if (origThunk->u1.Ordinal & IMAGE_ORDINAL_FLAG) continue;
PIMAGE_IMPORT_BY_NAME ibn = (PIMAGE_IMPORT_BY_NAME)(
(BYTE*)hModule + origThunk->u1.AddressOfData
);
if (strcmp((char*)ibn->Name, targetFunc) != 0) continue;
DWORD oldProtect;
VirtualProtect(&thunk->u1.Function, sizeof(LPVOID), PAGE_READWRITE, &oldProtect);
thunk->u1.Function = (ULONG_PTR)hookFunc;
VirtualProtect(&thunk->u1.Function, sizeof(LPVOID), oldProtect, &oldProtect);
printf("[*] IAT hook installed for %s!%s\n", targetDLL, targetFunc);
return;
}
}
}
int main() {
HMODULE hMod = GetModuleHandle(NULL);
PatchIAT(hMod, "user32.dll", "MessageBoxA", HookedMessageBoxA);
MessageBoxA(NULL, "Test", "Test", MB_OK);
return 0;
}
[cta]
IAT hooks are relatively easy to detect because they leave a clear artifact: an IAT entry pointing outside the expected module's memory range.
Detection logic in pseudocode:
# Enumerate all loaded modules, resolve expected IAT addresses
# Compare actual IAT pointer vs. what GetProcAddress returns
for each module in process:
for each import in module.IAT:
expected_addr = GetProcAddress(import.dll, import.name)
actual_addr = module.IAT[import.name]
if actual_addr != expected_addr:
flag_hook(module, import.name, actual_addr)
[cta]
Tools like PE-sieve, Moneta, and hollows_hunter automate this kind of IAT integrity checking across all loaded modules in a live process.
Inline hooking is more surgical and more widely used by both attackers and security vendors. Instead of redirecting a pointer, the attacker overwrites the first few bytes of the target function with a jump instruction that redirects execution to a trampoline or hook handler.
The typical approach on x64:
JMP to the hook function at the start of the target#include <windows.h>
#include <stdio.h>
typedef int (WINAPI *MessageBoxA_t)(HWND, LPCSTR, LPCSTR, UINT);
BYTE originalBytes[14];
void* trampolineAddr = NULL;
MessageBoxA_t originalFunc = NULL;
// Write a 14-byte absolute jump on x64
void WriteAbsoluteJmp(BYTE* dest, void* target) {
// FF 25 00 00 00 00 JMP [rip+0]
// followed by 8-byte target address
dest[0] = 0xFF;
dest[1] = 0x25;
*(DWORD*)(dest + 2) = 0;
*(UINT64*)(dest + 6) = (UINT64)target;
}
int WINAPI HookedMessageBoxA(HWND h, LPCSTR text, LPCSTR caption, UINT type) {
printf("[HOOK] Intercepted: %s\n", text);
return originalFunc(h, text, caption, type); // call trampoline
}
void InstallInlineHook() {
FARPROC target = GetProcAddress(GetModuleHandleA("user32.dll"), "MessageBoxA");
if (!target) return;
// Allocate trampoline
trampolineAddr = VirtualAlloc(NULL, 32, MEM_COMMIT | MEM_RESERVE, PAGE_EXECUTE_READWRITE);
// Copy original bytes to trampoline
memcpy(trampolineAddr, (void*)target, 14);
// Write jump back from trampoline to original+14
WriteAbsoluteJmp((BYTE*)trampolineAddr + 14, (BYTE*)target + 14);
originalFunc = (MessageBoxA_t)trampolineAddr;
// Overwrite the original function prologue
DWORD oldProtect;
VirtualProtect((void*)target, 14, PAGE_EXECUTE_READWRITE, &oldProtect);
WriteAbsoluteJmp((BYTE*)target, HookedMessageBoxA);
VirtualProtect((void*)target, 14, oldProtect, &oldProtect);
printf("[*] Inline hook installed on MessageBoxA\n");
}
int main() {
InstallInlineHook();
MessageBoxA(NULL, "Hooked!", "Test", MB_OK);
return 0;
}
[cta]
This pattern is exactly what EDR vendors use to monitor API calls, and exactly what attackers try to bypass by unhooking ntdll.dll before executing shellcode.
Red teamers frequently bypass user-mode EDR hooks by loading a fresh, unhooked copy of ntdll.dll directly from disk and copying the clean function bytes over the hooked in-memory version. This is sometimes called "direct syscall evasion" or "ntdll unhooking."
// Conceptual: load ntdll from disk, map it, copy .text section over in-memory ntdll
HANDLE hFile = CreateFileA("C:\\Windows\\System32\\ntdll.dll",
GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
// ... map, locate .text, VirtualProtect + memcpy over live ntdll
[cta]
This is one reason why kernel-level telemetry has become so critical to modern EDR architectures. If you want hands-on training in these exact evasion and detection flows, the Windows Red Teaming Extreme course at Redfox Cybersecurity Academy walks through building and bypassing user-mode hooks step by step.
# Scan all memory regions of a target process for hook indicators
pe-sieve.exe /pid 4321 /hooks /report_path C:\reports\
# Moneta: scan for executable memory anomalies across a process
moneta64.exe -p 4321 -m ioc
# SilentMoonwalk / Hunt-Sleeping-Beacons for beacon detection in hooked processes
HuntSleepingBeacons.exe
[cta]
PE-sieve flags functions whose prologues have been modified, identifies trampolines allocated in non-module memory, and produces a detailed JSON report per process.
The System Service Descriptor Table (SSDT) is a kernel structure that maps syscall numbers to their handler functions in ntoskrnl.exe. On 32-bit Windows, patching the SSDT was trivial and wildly popular among rootkit developers. On 64-bit Windows, Kernel Patch Protection (KPP, also called PatchGuard) monitors the SSDT and will bugcheck the system if unauthorized modifications are detected.
Modern kernel hooks therefore rely on documented or semi-documented mechanisms.
Legitimate (and malicious) drivers register kernel callbacks to observe and intercept system activity:
// Register a process creation callback (documented API)
PsSetCreateProcessNotifyRoutineEx(MyProcessNotifyCallback, FALSE);
// Register an image load callback
PsSetLoadImageNotifyRoutine(MyImageLoadCallback);
// Register a registry callback
CmRegisterCallbackEx(MyRegistryCallback, &altitude, driverObject, NULL, &cookie, NULL);
// Minifilter for file I/O interception
FltRegisterFilter(driverObject, &filterRegistration, &filterHandle);
FltStartFiltering(filterHandle);
[cta]
These are the same APIs used by EDR kernel components. A malicious driver abusing these callbacks can silently filter process creation events, intercept file writes, or modify registry data before it reaches user-mode consumers.
Rather than hooking functions, DKOM manipulates kernel data structures directly to hide objects from enumeration. The classic example is unlinking a process from the PsActiveProcessHead doubly linked list:
// Conceptual: walk EPROCESS list and unlink target process
PEPROCESS target = FindProcessByPID(hiddenPID);
PLIST_ENTRY entry = (PLIST_ENTRY)((BYTE*)target + EPROCESS_ACTIVELINKS_OFFSET);
RemoveEntryList(entry);
// Process is now invisible to NtQuerySystemInformation and Task Manager
[cta]
This technique does not trigger PatchGuard because it is modifying object data rather than kernel code. Detection requires cross-referencing multiple enumeration sources, which is exactly what tools like DriverQuery, WinPmem, and Volatility3 enable.
# Acquire a live memory image
winpmem_mini_x64.exe memory.raw
# Analyze with Volatility3
python3 vol.py -f memory.raw windows.ssdt.SSDT
python3 vol.py -f memory.raw windows.callbacks.Callbacks
python3 vol.py -f memory.raw windows.pslist.PsList
python3 vol.py -f memory.raw windows.psscan.PsScan
# Compare pslist vs psscan output to detect DKOM-hidden processes
# Any PID appearing in psscan but not pslist is a red flag
[cta]
Cross-referencing pslist (which walks the active process list) against psscan (which carves EPROCESS structures from raw memory) exposes DKOM-hidden processes that have been unlinked from the visible list.
Because user-mode hooks can be removed and IAT entries can be restored by attackers, mature defensive architectures do not rely solely on user-mode API monitoring. Event Tracing for Windows (ETW) combined with kernel callbacks provides telemetry that is significantly harder to suppress without a kernel-level driver.
# Enable ETW tracing for Microsoft-Windows-Threat-Intelligence provider
# This provider surfaces syscall-level telemetry used by EDRs
$session = New-EtwTraceSession -Name "HookDetection" -LogFileMode 0x8000000
Add-EtwTraceProvider -SessionName "HookDetection" `
-Guid "{F4E1897C-BB5D-5668-F1D8-040F4D8DD344}" `
-Level 0xFF `
-MatchAnyKeyword 0xFFFFFFFF
[cta]
The Microsoft-Windows-Threat-Intelligence ETW provider (also called EtwTi) exposes process allocation events, remote memory writes, and thread creation in remote processes. It is the backbone of user-mode hook detection in Windows Defender and many commercial EDRs.
An attacker that loads ntdll.dll from disk to overwrite hooked prologues leaves detectable artifacts:
C:\Windows\System32\ntdll.dll followed by a VirtualProtect on an existing ntdll mapping within the same processNtMapViewOfSection to map a second copy of ntdll into the process address spaceDetection rule for Sigma:
title: Potential NTDLL Unhooking via Section Mapping
id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
status: experimental
description: Detects process mapping a second image of ntdll.dll into its address space,
a common technique used to bypass user-mode EDR hooks.
logsource:
category: image_load
product: windows
detection:
selection:
ImageLoaded|contains: 'ntdll.dll'
ImageLoaded|startswith: '\\Device\\'
filter_legitimate:
Image|contains:
- 'WerFault.exe'
- 'SearchIndexer.exe'
condition: selection and not filter_legitimate
falsepositives:
- Legitimate forensic or debugging tools
level: high
tags:
- attack.defense_evasion
- attack.t1562.001
[cta]
On 64-bit Windows, Driver Signature Enforcement (DSE) prevents unsigned drivers from loading. Attackers bypass this using:
Defensive countermeasures include:
# Check HVCI status
Get-CimInstance -ClassName Win32_DeviceGuard -Namespace root\Microsoft\Windows\DeviceGuard |
Select-Object VirtualizationBasedSecurityStatus, CodeIntegrityPolicyEnforcementStatus
# Apply Microsoft's vulnerable driver blocklist
Set-MpPreference -EnableControlledFolderAccess Enabled
# Deploy via WDAC policy targeting driver signing scenarios
[cta]
API hooking is not a single technique but a layered family of methods that span from user-mode PE structures all the way down to kernel data objects. The attacker's goal in each case is the same: intercept execution, modify behavior, and avoid detection.
For defenders, the core principle is that no single telemetry source is sufficient. User-mode hooks can be removed. IAT entries can be restored. Kernel callbacks can be registered by both defenders and attackers. The strongest detection postures combine ETW kernel telemetry, memory scanning with tools like PE-sieve and Volatility3, and hardware-enforced integrity guarantees like HVCI.
For red teamers and malware analysts, fluency in all three hook types, along with the ability to read and write their implementation in C against Windows internals, is what separates entry-level work from advanced tradecraft.
If you are ready to move from conceptual understanding to hands-on exploitation and detection, the Windows Red Teaming Extreme course at Redfox Cybersecurity Academy is built specifically for practitioners who want to operate at this level. The course covers hook-based injection, EDR bypass techniques, kernel exploitation fundamentals, and live lab environments where you can build and test everything covered in this guide.
Redfox Cyber Security Inc.
8 The Green, Ste. A, Dover,
Delaware 19901,
United States.
.avif)
