Question x64 ASM — Rescuing Return Values from External Shellcode Calls

[byte_corvus]

Writing custom shellcode to invoke functions like my_malloc externally is a standard move when you're bypassing standard hooks or manual mapping, but it's easy to trip over the x64 calling convention.

The Problem
The current implementation successfully triggers the function call, but the return value ($RAX) is lost in transit or never written to the target allocation address. If the call is happening but the pointer isn't being stored, we're likely looking at a stack alignment issue or a failure to respect the shadow space requirements.

auto shellcode = make_shellcode(
    "\x48\x83\xEC\x20",                // sub rsp, 0x20 (Shadow space is 32 bytes!)
    "\x48\xB8", uint64_t(address),     // mov rax, target_function
    "\x48\xB9", uint64_t(size),        // mov rcx, arg1
    "\x48\xBA", uint64_t(alignment),   // mov rdx, arg2
    "\xFF\xD0",                        // call rax
    "\x48\xA3", uint64_t(allocation2), // mov [allocation2], rax (Direct store RAX to absolute addr)
    "\x48\x83\xC4\x20",                // add rsp, 0x20
    "\xC3"                             // ret
);

Spoiler: Technical Observations

1. Shadow Space: In the x64 ABI, the caller must reserve 32 bytes (0x20) of shadow space on the stack, even if the function takes fewer than 4 arguments. Using 0x10 is a recipe for memory corruption if the callee spills registers.
2. Relative vs Absolute: If you're moving rax into rdi and then moving a memory address into rax, make sure your instruction for mov [rax], rdi is actually encoding as a 64-bit store.
3. Memory Protections: Double-check that allocation2 resides in a PAGE_READWRITE or PAGE_EXECUTE_READWRITE segment. If the pointer itself is in a read-only section, the shellcode will silent-crash or throw an exception that your external caller might miss.

If the function is definitely executing but the write fails, try using a debugger to attach to the test process and set a breakpoint on the shellcode entry. Watch the RSPs like a hawk—it usually comes down to the stack being off by a few bytes or the shadow space being overwritten by the callee.

Anyone else ran into issues with return value persistence when hijacking threads for these calls?