Guide Halo CE — Analysis of BCrypt RSA Signature Verification

[byte_corvus]

Diving into the Halo CE binaries again—specifically looking at how the engine handles its integrity and signature checks via the Windows BCrypt (CNG) API.

I've been poking around the subroutines in the more recent builds and found a clean implementation of RSA signature verification. If you're currently reverse engineering the game's file validation or trying to understand how it handles its signed blobs, this is the core logic you need to account for. The original code uses standard Windows crypto providers, but seeing it in the context of the game's architecture is always worth a look.

Spoiler: Decompiled Verification Routine

bool __fastcall sub_180C5CFAC(PUCHAR pbHash, PUCHAR pbSignature, __int64 a3)
{
  int v5; 
  __int64 v6; 
  bool v7; 
  BCRYPT_ALG_HANDLE v8; 
  BCRYPT_KEY_HANDLE phKey; 
  const WCHAR *pPaddingInfo; 
  phKey = nullptr;

  if ( !pbSignature || !a3 )
    goto LABEL_11;

  v5 = sub_180C5CF44(a3);
  v6 = v5;
  if ( v5 == -1 || v5 == 1 )
    return 0;

  v8 = phAlgorithm;
  if ( !phAlgorithm )
  {
    if ( BCryptOpenAlgorithmProvider(&phAlgorithm, L"RSA", nullptr, 0) < 0 )
      goto LABEL_11;
    v8 = phAlgorithm;
  }

  if ( BCryptImportKeyPair(v8, nullptr, L"RSAPUBLICBLOB", &phKey, *(PUCHAR *)&xmmword_182C98390[3 * v6], *((_QWORD *)&xmmword_182C98390[3 * v6] + 1), 0) >= 0 )
  {
    pPaddingInfo = L"SHA1";
    v7 = BCryptVerifySignature(phKey, &pPaddingInfo, pbHash, 0x14u, pbSignature, 0x100u, 2u) >= 0;
    goto LABEL_12;
  }

LABEL_11:
  v7 = 0;
LABEL_12:
  if ( phKey )
    BCryptDestroyKey(phKey);
  return v7;
}

Technical Breakdown — The Meat of the Logic:

1. The routine starts with basic sanity checks on the signature pointer. If it's null, it bails out early to LABEL_11.
2. It looks up an index via sub_180C5CF44 to determine which public key to pull from the global array. If the index is invalid (-1 or 1), it returns false.
3. Provider Initialization: It uses a global algorithm handle phAlgorithm. If it hasn't been opened yet, it calls BCryptOpenAlgorithmProvider with the "RSA" identifier.
4. Key Import: The game pulls an RSAPUBLICBLOB from an internal table (xmmword_182C98390). This is where the hardcoded public keys for the game's ecosystem live.
5. Validation: Finally, it hits BCryptVerifySignature. It uses SHA1 as the padding info and expects a 20-byte hash (0x14u). The return value v7 is the result of this comparison.

Implementation Notes & Risks:
For those of you working on internal tools or map bypasses, manipulating the return boolean v7 is the obvious entry point, but it's sloppy. A cleaner approach for persistent research is a hook on BCryptVerifySignature or simply replacing the target public keys in the data section. Note that this implementation is heavily reliant on the host OS's crypto libraries, so it's as stable as the Windows version it's running on.

While others are catching bans for basic memory pokes and getting flagged by simple heuristic scanners, the real progress happens when you understand the engine's crypto layer.

Who has more info on the structure of the keys in the global array?