Guide DirectX 9 ImGui Hooking — Fixing Rendering Z-Order & Custom Mouse Issues

[byte_corvus]

If you're still digging into old DX9 titles, you've definitely hit the wall where ImGui behaves like a brat. A common headache being discussed lately involves custom UI windows rendering inside the game's native windows instead of as a clean overlay, coupled with mouse input being hijacked by the game's internal software cursor.

The logic being used relies on a standard Detours implementation for CreateWindowExA, EndScene, and Reset. While the hook itself is structurally sound, the clipping and input issues are classic symptoms of how older engines handle their render state and input buffers.

The Technical Breakdown

1. Rendering Z-Order: If your ImGui menu is appearing behind game elements or 'inside' game windows, it's often because the game is performing additional draw calls after EndScene or using a specific ScissorRect that clips your overlay. You might need to reset the render state before ImGui::NewFrame.
2. Custom Mouse Logic: Most DX9-era games that don't use the standard Windows cursor are likely polling DirectInput or Raw Input. Simply hooking WndProc won't stop the game from processing movement while your menu is open.
3. Hooking Strategy: The implementation uses a VTable offset (42 for EndScene) which is standard, but the initialization inside CreateWindowExA can be race-condition prone depending on how the game initializes its device.

Spoiler: Current Implementation Logic

// Header snippet for the DX9 Hook Class
class MDirectx9 {
public:
    static MDirectx9& gInst();
    void MDirectx9DX();
    void UnMDirectx9DX();
private:
    CreateWindowExA_t oCreateWindowExA;
    WNDPROC OWndProc;
    EndScene oEndScene;
    Reset oReset;
    static LRESULT WINAPI MDirectx9WndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam);
    static HRESULT APIENTRY MDirectx9EndScene(IDirect3DDevice9Ex* D3D9Device);
};

Potential Fixes to Explore

Regarding the mouse: If the game uses a software cursor, you'll need to manually nullify the game's input processing when bDisplay is true. For the rendering clipping, try forcing a state block or checking if Present is a better hook candidate for this specific engine.

In my experience, DX9 hooks are rarely 'plug and play' with ImGui. You usually have to fight the engine's internal state management to get a clean overlay. If you're seeing your menu clipped by game UI, the game is likely doing its own UI pass in a way that bypasses standard EndScene expectations.

Anyone else dealt with this specific UI clipping in older DX9 engines?