This blog series is the written companion to our Malware Development: Red Team Tradecraft YouTube playlist, where we walk through live demos of the evasion techniques discussed here, including payload staging, AV/EDR bypass, and in-memory, on disk, and network evasion. We recommend following the playlist along with these posts.
In Part 1, we covered the foundational concepts of processes, threads, and memory. In Part 2, we explored Windows APIs, the PE format, and injection techniques; in Part 3, we covered Shellcode Fundamentals, Cryptography in Malware, Shellcode Encryption & Obfuscation, Evasion Techniques, etc.
Disclaimer
This malware development blog series is designed for cybersecurity professionals, red teamers, and security analysts who want to understand malware development from both offensive and defensive perspectives. The knowledge shared here is meant to help you better create, detect, analyze, and defend against malicious software. Always use this information ethically and legally within authorized security assessments and controlled environments.
Introduction
Every time a Windows process calls an API, it follows a predictable path: the code looks up an address, jumps to it, and trusts the next step. API hooking interrupts this trust, allowing Red team operators to observe, modify, suppress, or redirect calls without the process knowing. This technique is central to both offensive and defensive security, used by AV & EDR products to monitor runtime activity and by red teams to intercept credentials and manipulate behavior. Understanding it is essential for serious work on either side.
This blog covers three main techniques: Detours trampoline hooks, IAT hooking, and manual inline patching. We examine how each works at the instruction and memory levels, detection methods, and trade-offs. We also demonstrate GhostHook, a tool we built for offensive trampoline hooking, which captured plaintext credentials in Password Safe and transmitted them filelessly. Lastly, we discuss the ongoing detection-and-evasion race between EDR vendors and attackers.
What is API Hooking?
API hooking is a technique used to intercept, monitor, or modify calls to Windows API functions at runtime. It sits at the intersection of offensive and defensive security: EDR agents use it to gain visibility into process behavior, while red team operators and malware use it to manipulate that behavior, harvest sensitive data, or evade detection. Understanding how the three primary hooking techniques work is essential knowledge for anyone operating on either side of that line.
Why API Hooking Matters
When a process calls a Windows API function, execution follows a predictable path: from the calling code, through the function’s resolved address, into the implementation inside a system DLL such as ntdll.dll or kernel32.dll. API hooking intercepts that path. By inserting a redirect at the right point, an Red team operator or security tool can observe the arguments being passed, modify the return value, block the call entirely, or silently log what is happening without the calling code knowing anything has changed.
For red teams, this creates opportunities to intercept credentials, suppress telemetry, or manipulate process behavior in ways that are difficult to detect. For defenders, the same mechanisms power the user-mode hooks that most EDR products rely on to monitor process activity. Understanding both sides is what makes purple team assessments effective.
The Three Main Techniques
1. Detours-Style Trampoline Hooks
Trampoline hooking is the most widely used approach in both security tooling and malware. It is the technique underlying tools such as GhostHook and is the basis of the Microsoft Detours library.
The hook works by overwriting the first few bytes of the target function with an JMP instruction that redirects execution to a custom hook function. What distinguishes a trampoline hook from simple overwriting is the trampoline stub. Before any bytes are modified, the original bytes are copied into a small region of executable memory, followed by a JMP back to the remainder of the original function. This means the hook function can call through to the real implementation at any point, and the rest of the target process continues to operate normally.
From the perspective of the calling code, nothing has changed. The API function still executes correctly and returns the expected result. The only difference is that the hook function runs first, with full access to the arguments and the ability to alter them before the real function sees them.
Trampoline hooks work on both x86 and x64, though x64 requires additional care. Because the standard 5-byte relative JMP cannot reach arbitrary 64-bit addresses, implementations typically use a 14-byte absolute JMP or a similar sequence. Instruction boundary alignment must also be handled correctly: overwriting partial instructions will crash the process, so the hook must copy enough bytes to cover complete instructions before adding its own JMP back.
Strengths: Transparent call-through to the original function, reliable, widely understood, and works across both architectures.
Weaknesses: Modifies function bytes in memory, which can be detected by integrity checks that hash or scan the first bytes of hooked functions. This is a common EDR detection mechanism and also a common red team bypass target.
2. IAT Hooking
Import Address Table (IAT) hooking takes a fundamentally different approach. Rather than touching the target function itself, it modifies the pointer to that function inside the calling module’s Import Address Table.
When Windows loads a module, it resolves all imported function addresses and writes them into the IAT. Any call the module makes to an imported API goes through this table to look up the address. By replacing one of those pointers with the address of a custom function, all calls from that module to the targeted API are silently redirected. The original function code is completely untouched.
The redirect is invisible to the called function: it simply receives a call and executes normally. Only the caller is affected, because its IAT now points somewhere different.
IAT hooking is straightforward to implement. The process involves locating the target module’s IAT, finding the entry for the function to hook, changing memory permissions on that page if necessary, and writing the new pointer. No instruction-level manipulation is required, which eliminates the risk of crashing the process due to misaligned bytes.
Strengths: The original function is never modified, simpler to implement, and has a lower risk of process instability.
Weaknesses: Scope is limited to calls made through a specific module’s IAT. Calls made directly by address, calls from other modules with their own IAT entries for the same function, and calls resolved dynamically GetProcAddress at runtime are all outside the hook’s reach. IAT hooks are also detectable by any tool that checks whether IAT pointers point to expected locations within the corresponding DLL’s loaded image.
3. Inline Patching (Manual / Hand-Rolled)
Inline patching achieves the same outcome as trampoline hooking, but without any supporting library. The developer writes the JMP bytes directly into the target function, builds the trampoline stub by hand, and manages all architecture-specific details without assistance.
This approach gives complete control over how the hook is installed, what the trampoline looks like in memory, and how the call-through is handled. That level of control is valuable when the goal is to avoid patterns associated with known libraries. Detours, for example, have well-documented byte signatures that security tools can scan for. A fully custom implementation with a non-standard trampoline layout can be significantly harder to fingerprint.
The tradeoff is that everything that Detours handles automatically must be done correctly by hand. On x64, instruction boundary alignment, relative address fixups, and the handling of RIP-relative instructions in the copied bytes all require careful attention. Thread safety during hook installation is another concern: if another thread executes the function while bytes are being overwritten, the results are unpredictable. Experienced implementers typically use WriteProcessMemory combined with page permission manipulation and may briefly suspend threads during installation.
Strengths: Maximum control over implementation details, avoids library-associated signatures, can be tailored specifically for evasion.
Weaknesses: High implementation complexity, significant risk of process instability if done incorrectly, and a deep, low-level knowledge requirement to get right on x64.
Advanced Considerations for Trampoline and Inline Hooks
While the core trampoline technique provides reliable interception with call-through support, production-grade or highly evasive implementations must address several low-level challenges, particularly on x64. These go beyond basic prologue overwriting and become critical when operating in complex environments (e.g. multi-threaded processes, EDR-instrumented targets, or when layering multiple hooks).
1. Hot Patching and Function Prologue Variations
Many modern Windows functions (especially in ntdll.dll and kernelbase.dll) are built with hot-patchable prologues. These typically start with a 2-byte MOV EDI, EDI (or similar NOP-equivalent) followed by a 5-byte instruction, allowing Microsoft to insert updates without breaking existing hooks or requiring full DLL reloads.
- Implication for hooking: A naive 5-byte JMP (E9 rel32) might land in the middle of a hot-patch region or misalign with instruction boundaries. Mature implementations (including Microsoft Detours) detect and consume enough bytes (often 5-14+) to cover complete instructions.
- Evasion/Defense Angle: EDRs may monitor for unexpected modifications in these hot-patch areas. Operators sometimes prefer to target non-hot-patched functions or to use careful unhooking and re-hooking strategies.
2. Multi-Hook Chaining and Thread Safety
When multiple components (your implant + EDR user-mode hooks + other security products) target the same function, conflicts arise.
- Chaining: A robust trampoline must attach cleanly to an already-hooked function. This often involves skipping existing JMPs (e.g., via DetourCodeFromPointer in Detours) or walking the hook chain. Poor implementations can cause infinite recursion or broken call-through.
- Thread Safety: Hook installation is inherently racy. Recommended practices include:
- Suspending threads (or using critical sections).
- Using VirtualProtect to toggle page permissions.
- Atomic writes where possible.
- Proper cleanup on unload to avoid dangling trampolines.
Failing here leads to crashes or easy detection via anomalous behavior.
3. x64-Specific Edge Cases (RIP-Relative Addressing and Relocation)
The standard 5-byte relative JMP is often insufficient on x64 due to address space. Common workarounds use 14-byte absolute jumps (e.g., FF 25 00 00 00 00 + 64-bit address via RIP-relative indirect) or register-based sequences.
The real complexity appears when copying stolen bytes to the trampoline:
- RIP-Relative Instructions: Many x64 instructions (MOV rax, [rip + offset], LEA, calls, etc.) are position-dependent. Blindly copying them breaks the relative offsets because the trampoline resides at a different address.
- Solution: A disassembler/relocator (built into Detours or custom engines) must parse instructions, detect RIP-relative operands, recalculate offsets for the new location, and rewrite them.
- Common edge cases: Functions with many short RIP-relatives, conditional jumps near the prologue, or instructions that straddle the copied region.
- Trampoline Placement: Must be executable (VirtualAlloc with PAGE_EXECUTE_READWRITE), preferably in non-backed memory for stealth, and within reach if relative jumps are used.
- Additional Mitigations: Alignment (AVX/SSE), calling convention shadow space, exception unwinding (SEH), and compatibility with Control Flow Guard (CFG) and other modern protections.
Manual inline patching provides maximum control over evasion (custom relocators can avoid library signatures), but the implementation complexity is significantly higher. This is why many operators start with Detours or MinHook before moving to fully custom solutions (often paired with direct syscalls to bypass user-mode hooks entirely).
Detection and Evasion: The Arms Race
From a defensive standpoint, each hooking technique leaves a different footprint. Trampoline hooks modify function bytes in memory, making them detectable via periodic integrity checks against a clean copy of the DLL. IAT hooks leave function bytes clean but create anomalous pointers in the IAT. Manual inline patches may avoid library signatures but still produce the characteristic JMP pattern at the start of hooked functions.
Red team operators performing unhooking operations (reading a fresh copy of a DLL from disk and restoring the original bytes) specifically target trampoline hooks, since most EDR user-mode hooks use this mechanism. Understanding which techniques an EDR uses and how it detects tampering is critical to planning an evasive engagement.
For purple team assessments, this dynamic makes API hooking an excellent test subject. Simulating hook tampering and observing whether the EDR detects the modification, raises an alert, and recovers its hooks provides direct evidence of how resilient the defensive stack is against a sophisticated operator.
GhostHook (a.k.a Pekenux): Offensive API Hooking in Practice
The three techniques covered in this blog are not theoretical. To demonstrate the practical application of trampoline hooks in the offensive space, we built GhostHook (also known as Pekenux): a fileless infostealer that flips the EDR playbook. The same E9 JMP trampoline that antivirus and EDR products use to watch process behavior can be turned offensively to extract plaintext credentials from a running process before they are ever hashed or stored.
Target and Objective
The target is Password Safe (pwsafe.exe), a widely used credential manager. When a user opens their vault, the application presents a master password dialog. At the point where the user submits that dialog, the plaintext password exists in memory for a brief window before it is processed, hashed, and used to decrypt the vault. GhostHook intercepts at exactly that point.
Mechanism
GhostHook uses Microsoft Detours to install a trampoline hook on the target function handling master password submission inside pwsafe.exe. The hook is installed in-process after the implant is loaded via reflective injection, meaning no DLL is written to disk at any stage.
The Detours installation follows the standard trampoline pattern. The first bytes of the target function are overwritten with a JMP to GhostHook’s credential capture function. The original bytes are saved into a trampoline stub with a JMP back to the remainder of the target function. From the process’s perspective, execution proceeds normally. The user sees no error, the vault opens, and Password Safe behaves as expected.
Inside the hook function, the plaintext password is read directly from memory before the original function is called through the trampoline. The credential is captured in a local buffer, the trampoline is invoked to allow Password Safe to continue normally, and the captured string is then passed to the C2 beacon.
C2 Exfiltration
Credential exfiltration uses WinHTTP directly. WinHttpOpen, WinHttpConnect, WinHttpOpenRequest, and WinHttpSendRequest are called in sequence to POST the captured plaintext to the operator’s C2 infrastructure. No third-party networking library is used. The entire exfiltration path stays within Windows native API calls, keeping the network activity consistent with legitimate application traffic patterns.
Why This Works Against Defensive Tooling
GhostHook operates entirely in user space inside a legitimate, already-running process. It does not spawn a child process, write to disk, or load a module that would appear in the target process’s module list. The hook itself is installed using the same Detours pattern that many AV/EDR user-mode agents use internally, which creates a detection challenge: a hook that looks structurally identical to a legitimate security product’s own instrumentation.
Detection requires the defensive stack to go beyond module enumeration and look at unbacked executable memory regions, unexpected function prologue modifications in pwsafe.exe‘s mapped image, or anomalous outbound WinHTTP traffic from a credential manager process. Each of these is detectable with the right telemetry, but none is caught by file-based scanning or standard module integrity checks.
This is precisely the gap that purple team assessments are designed to expose: the distance between what your AV/EDR theoretically detects and what it actually catches in your environment, when evaluated against real tooling.
Summary
| Technique | Modifies Function Code | Modifies IAT | Call-Through Supported | Implementation Complexity |
|---|---|---|---|---|
| Trampoline Hook | Yes | No | Yes | Medium |
| IAT Hook | No | Yes | Yes | Low |
| Inline Patch | Yes | No | Yes (manual) | High |
All three techniques remain relevant in modern offensive operations and defensive tooling. The right choice depends on the goal: reliability and transparency favor trampoline hooks, simplicity favors IAT hooks, and evasion of known signatures favors manual inline patching.
Resources
Training
- Sektor7 Malware Development Courses (institute.sektor7.net): Practical malware development training by reenz0h. The courses cover custom loaders, shellcode, and evasion at a level that directly maps to the implementation and understanding of API hooks.
Tools:
- SysWhispers: Syscall generation utility. Relevant when evaluating direct syscall techniques as an alternative to API hooking that bypasses user-mode hooks entirely.
- Donut: Shellcode generator that converts executables and DLLs into position-independent shellcode. Useful for delivering hook-based payloads in-memory.
- PE-bear: PE analysis tool for inspecting import tables, export tables, and section layouts. Directly useful when working with IAT hooks and understanding what a module imports.
- x64dbg: Open-source Windows debugger for x64 and x32. The go-to tool for setting breakpoints on hooked functions, inspecting the IAT at runtime, and stepping through trampoline logic.
Reading:
- The Shellcoder’s Handbook by Koziol et al.: Covers fundamentals of low-level exploitation and code injection. Provides the groundwork for understanding how function prologues and memory patching work in practice.
- Practical Malware Analysis by Sikorski and Honig: Includes coverage of API hooking as both an offensive and defensive technique. Highly recommended for anyone building or analyzing hook-based tooling.
- Windows Internals by Yosifovich, Russinovich et al.: The authoritative source on the Windows loader, the IAT, the PEB, and the kernel/user-mode boundary. Required reading for anyone working with hooks at a serious level.
Christian Ramirez (a.k.a. Polunchis) is a seasoned cybersecurity professional with nearly two decades of hands-on experience in offensive security and adversarial simulations. He specializes in web application, cloud (AWS, Azure, GCP), and infrastructure penetration testing, having led numerous high-impact engagements that strengthened organizational resilience worldwide.
His expertise spans red and purple team assessments, exploit and malware development, advanced evasion techniques, and bypassing AV/EDR defenses. Over his career, Christian Ramirez has tested diverse environments including web services, APIs (SOAP & REST), thick clients, wireless networks, SAP ERP systems, and ATMs.
Renowned for delivering realistic adversary emulation and comprehensive threat analysis, Christian Ramirez empowers organizations to detect, defend, and respond to advanced threats effectively. Blending technical precision with strategic insight, he helps teams uncover vulnerabilities and elevate overall security maturity.
When not dissecting systems or leading red team operations, Christian Ramirez shares his knowledge through research, thought leadership, and collaborative defense initiatives that push the boundaries of modern cybersecurity.