[cmer81]

DHooks: Linux x86_64 support

Why

Upstream DHooks skips Linux x86_64 (AMBuilder: "Presently only x86_64 on Windows is supported"). On Linux upstream only offers it through the KHook rewrite (alliedmodders#2403, draft), which needs MM:S 2.0 and breaks every SourceHook-based extension. We need DHooks on our CS:S x64 servers now, on our pinned SourceMod (1.13.0.7427) with MM:S 1.12.

What

Based on b68a6097 (1.13.0.7427, our production pin):

• rtldg's unfinished dhookslinux64wip work (DHooks x64 Linux enabler alliedmodders/sourcemod#2292, closed). Authorship is kept on 12 commits; one was dropped because 7427 already does the same thing with int64 Address support. The work covers:
  • the System V x86_64SystemVDefault calling convention for detours;
  • stack/argument size fixes;
  • enabling the linux64 build.
• Our changes:
  • Legacy Address on x64. Plugins built before Address became int64 pass 32-bit addresses. SourceMod 1.13 has no pseudo-addresses: ISourceMod::To/FromPseudoAddress are stubs that return 0, and the core natives just sign-extend such cells. DHooks used to C-cast those values (and this pushed to callbacks) to and from raw pointers, which doesn't build with clang on Linux x86_64. It now converts them the same way as the core natives (PtrToLegacyAddress/LegacyAddressToPtr in util.h). That's lossless on x86. On x86_64 such plugins can't hold a pointer and must be rebuilt with SourceMod 1.13 (see the note for plugin authors below). An earlier revision of this PR mapped them through the pseudo-address stubs, which returned 0.
  • DHookGetParamAddress truncated the pointer for legacy plugins; it now goes through the same conversion.
  • SysV vhook thunk. The Linux x86_64 GenerateThunk was a stub that returned straight away. It now rebuilds the arg stack Callback() reads, at the offsets GetStackParamOffset() computes, from rsi, rdx, rcx, r8, r9, xmm0-7 and the caller's stack.
    • By-value objects over 16 bytes are copied inline from the stack. Smaller ones are assumed to be integer-class.
    • ODTOR objects are passed by hidden reference.
    • It calls Callback/Callback_float/Callback_vector/Callback_stringt with the delegate still in rdi. Vector results come back in xmm0/xmm1.
  • GetStackArgsSize() skipped the result-pointer slot of Vector/string_t callbacks on linux64, so their params were copied 16 bytes short.
  • Detours keep every System V argument register. The convention only saved and restored registers of declared arguments. Anything undeclared therefore reached the original function clobbered by the callbacks, for example the this of a member function declared cdecl, which is fine on x86 where it stays on the stack. LagCompensation's CEntityTouchManager::FrameUpdatePostEntityThink detour segfaulted this way. GetRegisters() now always includes rdi, rsi, rdx, rcx, r8, r9, xmm0-7.
  • Post-callback generated before the bridge. Both JIT buffers share the page allocator, and a GenBuffer that grows marks its old block (so the whole page) read/execute. The post-callback was generated in the middle of the bridge, and once bridges got bigger the next bridge write faulted.
  • plugins/testsuite/dhooks/: a CS:S runtime test (virtual hooks + detours) with its gamedata. -dhx_auto runs it unattended.

Requires the SourceHook linux64 generator from srcdslab/metamod-source#4, because virtual hooks use MakeHookMan from MM:S core.

Testing

On a CS:S x64 dedicated server (build 11003710) with bots, using plugins/testsuite/dhooks/dhooks-css-test.sp:

• Virtual hooks:
  • Teleport: params read correctly, NULL detection works, and the teleport really happens. The hook also catches the game's own respawn teleports.
  • OnTakeDamage: the pre-hook changes the damage (applies in game) and the post-hook reads the return value.
  • Weapon_ShootPosition (ReturnType_Vector): the post-hook reads the eye position, and a pre-hook supercede reaches the caller.
  • GetPlayerMaxSpeed (ReturnType_Float): supercede works.
  • AcceptInput: input name, activator, variant_t (by hidden reference) and output id are all read correctly.
• Detours:
  • CCSPlayer::SwitchTeam(int) and CCSPlayer::RoundRespawn();
  • CCSGameRules::TerminateRound(float, int): float + int params, this, and changing a param (seen by the post-hook);
  • CGameServer::SetHibernating(bool) in engine.
• No crash.

The whole SourceMod config of our ZE server (≈160 plugins, 14 of them using DHooks) also loads and runs on that bench without crashing.

Notes for plugin authors on x86_64:

• Any plugin dealing with addresses (GetEntityAddress, GameData.GetAddress, Load/StoreToAddress, SDKCall_Raw, DHooks addresses) has to be built with SourceMod 1.13 includes (64-bit Address). Legacy builds get wrong pointers, e.g. LagCompensation crashed CS:S x64 in CCollisionProperty::MarkPartitionHandleDirty.
• variant_t::fieldType is at offset 20, not 16 (the value union grows to 16 bytes because string_t is a pointer). Read it from gamedata ("linux" 16 / "linux64" 20), as the testsuite does.
• SDKCall returning a Vector by value also needs the BinTools fix (BinTools: classify every eightbyte of small objects on System V x86_64 #10) to get z.

Not covered yet: small (≤ 16 bytes) objects passed by value (the System V class split is guessed as integer), detour return overrides. Next step is running our ZE plugins that use DHooks.

Only dhooks.ext is meant to be taken from this build. SourceMod core and the other extensions stay the official 7427 package.

[WareZ57]

Thanks for this work. We run it on CS:S build 11003710 x86_64: SourceMod 1.13.0.7427 with this PR, #10 and srcdslab/metamod-source#4.

We hit a bug in the Linux x86_64 detour bridge. Every detoured function that returns a value hands its caller the address it returns to, which reads as true for a bool. It happens whatever the callbacks do, even with a post-hook that only returns MRES_Ignored.

Cause

In CHook::CreatePostCallback() (extensions/dhooks/DynamicHooks/hook.cpp):

1. Write_RestoreRegisters(jit, HOOKTYPE_POST) puts the return value back in rax.
2. The generated code then calls CHook::GetReturnAddress. That is a regular C call, so it overwrites rax, rdx, xmm0 and xmm1.
3. It ends with jit.jump(rax), so the caller resumes with that return address still in rax.

What we saw on CS:S

• A detour on CCSBot::IsFriendInTheWay froze every bot, even a post-hook that changes nothing.
• A detour on CCSBot::IsVisible left the bots unable to see any player.
• A detour on CSteam3Server::NotifyClientConnect let clients whose Steam ticket had been refused join, as STEAM_ID_PENDING.

Void detours such as CCSPlayer::AddAccount and CBaseServer::RecalculateTags were fine. The CS:S testsuite in this PR only detours void functions, so it doesn't catch this.

Fix

The patch keeps rax, rdx, xmm0 and xmm1 on the stack around the GetReturnAddress call. It passes the key as rsp + 48, so it still matches what SetReturnAddress stored, and jumps back through r11:

--- a/extensions/dhooks/DynamicHooks/hook.cpp
+++ b/extensions/dhooks/DynamicHooks/hook.cpp
@@ -419,6 +419,17 @@
 	// Restore the previously saved registers, so any changes will be applied
 	Write_RestoreRegisters(jit, HOOKTYPE_POST);
 
+	// The return value (rax, rdx, xmm0, xmm1) must reach the caller untouched: GetReturnAddress is a
+	// C call that overwrites them, and rax then held the return address, so every non-void detoured
+	// function returned it (a "true" bool) to its caller. Kept on the stack around the call (48 bytes:
+	// the key passed to GetReturnAddress, rsp as SetReturnAddress stored it, is now rsp + 48), and the
+	// jump back goes through r11 (caller-saved, never a return register).
+	jit.push(rax);
+	jit.push(rdx);
+	jit.sub(rsp, 32);
+	jit.movsd(rsp(), xmm0);
+	jit.movsd(rsp(16), xmm1);
+
 	// Get return address
 	union
 	{
@@ -437,8 +448,8 @@
 	MSVC_ONLY(jit.mov(rcx, reinterpret_cast<std::uint64_t>(this)));
 
 	// 2nd parameter (rsp)
-	GCC_ONLY(jit.lea(rsi, rsp()));
-	MSVC_ONLY(jit.lea(rdx, rsp(40)));
+	GCC_ONLY(jit.lea(rsi, rsp(48)));
+	MSVC_ONLY(jit.lea(rdx, rsp(40 + 48)));
 
 	// Call GetReturnAddress
 	jit.mov(rax, func.address);
@@ -449,9 +460,15 @@
 	// Free shadow space
 	MSVC_ONLY(jit.add(rsp, 40));
 
-	// Jump to the original return address
+	// Back the return value, then jump to the original return address
+	jit.mov(r11, rax);
+	jit.movsd(xmm0, rsp());
+	jit.movsd(xmm1, rsp(16));
+	jit.add(rsp, 32);
+	jit.pop(rdx);
+	jit.pop(rax);
 	jit.add(rsp, 8);
-	jit.jump(rax);
+	jit.jump(r11);
 }
 
 void CHook::Write_CallHandler(x64JitWriter& jit, HookType_t type)

Tested on CS:S 11003710 Linux x86_64

• A detour on CCSBot::IsAttacking (bool, ThisPointer_Ignore), with a pre-hook that supercedes with true: SDKCall returns 1, then 0 again once the detour is disabled.
• Post-hook on the same function: it reads the original value (1 for an attacking bot, 0 for an idle one), and an MRES_Override with the other value reaches the caller.
• A no-op detour on IsFriendInTheWay: the bots move normally (59 of 59).
• With the IsVisible detour enabled, the bots see and attack players again.
• Still working with the patch: void detours, virtual hooks (bool supercede on AcceptInput, bool post-override on IsValidObserverTarget, variant_t string parameter), and CTakeDamageInfo objects in RadiusDamage and OnDamagedByExplosion hooks.

Only the Linux path was tested. The MSVC_ONLY lines are adjusted the same way for consistency, but we haven't built or run them on Windows.

Binaries and sources

The DHooks, BinTools and Metamod:Source binaries we run (this PR with the patch above, #10, srcdslab/metamod-source#4), their complete sources and a Dockerfile that rebuilds them: https://github.com/WareZ57/DHooks-x64-CSS/releases/tag/v1.0.0