Error with LoadLibrary returning ERROR_STACK_OVERFLOW

[warz]

The call to LoadLibrary on storm.dll is returning ERROR_STACK_OVERFLOW. All of these files are in the running directory. The calls to LoadLibrary prior to the storm.dll call don't fail - they succeed. Anyone know why? I was thinking maybe I need to call SetDllDirectory but the first search path is the current running directory anyways, so it should work.

Code Select Expand


wchar_t starcraftw[MAX_PATH];
char starcraft[MAX_PATH];
char battle[MAX_PATH];
char storm[MAX_PATH];
char lockdown[MAX_PATH];

int main() {
wcscpy(starcraftw, L"starcraft.exe");
strcpy(starcraft, "starcraft.exe");
strcpy(battle, "battle.snp");
strcpy(storm, "storm.dll");
strcpy(lockdown, "lockdown-IX86-13.dll");

hStarcraft = LoadLibrary(starcraft);
if(!hStarcraft)
printf("Error: starcraft (%d)\n", GetLastError());

HMODULE hStorm = LoadLibrary(storm);
if(!hStorm)
printf("Error: storm (%d)\n", GetLastError());

HMODULE hBattle = LoadLibrary(battle);
if(!hBattle)
printf("Error: battle (%d)\n", GetLastError());

HMODULE hLockdown = LoadLibrary(lockdown);
if(!hLockdown)
printf("Error: lockdown (%d)\n", GetLastError());

funcptr_1 CheckRevision = (funcptr_1)GetProcAddress(hLockdown, "CheckRevision");

...
}

# ERROR_STACK_OVERFLOW
# 1001     Recursion too deep; the stack overflowed.


[Kp]

Call SetLastError(0) before you load the offending library.  You may be seeing a stale error code.  Also, consider attaching a debugger before that LoadLibrary call to see if Storm loads successfully.  Perhaps it is failing to find a dependency, or Storm's DllMain is failing.

[warz]

Ah, figured it out. I was setting global api hooks, and my GetModuleHandleA hook was interfering with LoadLibrary calls. I just set my hooks after I load all of these libaries now. Works like a charm.

[warz]

oops, another problem. my hook to getmodulehandlea will call my own version of the function, which looks like so..

Code Select Expand


HMODULE WINAPI myGetModuleHandleA(LPCTSTR lpModuleName) {
if(!lpModuleName) {
return hStarcraft;
} else {
hGetModuleHandleA->set(0);
HMODULE rtn = GetModuleHandle(lpModuleName);
hGetModuleHandleA->set(1);
return rtn;
}
}


I'm stepping through using the VC++ debugger, and my call to CheckRevision produces an error at myGetModuleHandleA function. This means that CheckRevision is calling my function, but my function definition is wrong, somehow. The error is..

Unhandled exception at 0x00412d99 in test.exe: 0xC00000FD: Stack overflow.

The debugger in VC++ points to my "HMODULE WINAPI myGetModuleHandleA(LPCTSTR lpModuleName) {" line. This is exactly how the actual GetModuleHandle is defined. Is there something wrong here? :-(

[Kp]

My first guess would be that you're not properly disarming your GetModuleHandle hook, so your call to the real function chains back to your hook, which calls the real function, which calls the hook...  What does that set call do?

You can confirm my suspicion by checking the call stack when the exception is raised.  Depending on how your function is compiled, the debugger may or may not be able to produce a sensible stack trace on its own.  If it can't, you'll have to walk the stack by hand.

[warz]

That set function disarms my hook. Hm. I'll look at the call stack.

[warz]

Ah, ok. I figured it out.

My api hook functions use GetModuleHandle. What was happening was my spoof function was trying to disable my GetModuleHandleA hook, which had to call GetModuleHandle in order to do so. Well, that just called my spoof function, and called the original one again, and so on, looping forever.

So, since GetModuleHandleA calls GetModuleHandleW, but not vice versa, I checked to see if I was unsetting a hook on GetModuleHandle, and if I was, it will use the widechar version, GetModuleHandleW, instead.

Thanks.

[Skywing]

A better solution may be to just set a flag indicating that you are in a recursive context and just call the original implementation in that case, instead of trying to unhook and rehook on the fly.

[warz]

Well, my hook method may not be the best, not sure, but what I do is write 5 bytes to the beginning of the function which i plan on hooking. So, literally, the original implementation jmp's to my implementation. So, calling the original implementation in the event of a recursive loop wouldn't help, because, that's where the loop would have been occuring in the first place.

[Skywing]

The simple way to solve that is to disassemble the target until you have enough room to write your jump, then save all of the ovewritten instructions in a stub.  Then, after all of the saved instructions in the stub, you place a jump to the first valid instruction (that you did not copy to the stuf due to a partial or complete overwrite with your jump).

This gives you a way to call the original implementation even though you  have hooked it.

There are a number of libraries out there with this capability (such as Microsoft Research's Detours) - be sure to check the license to see if it acceptable for your program, though.

[warz]

I've read up on detours, and even messed around with them when I was writing some of my brood war client manipulating libaries. I do look until I have enough room to write my jump, and save the overwritten instructions so I can remove the jmp. You have a good idea, though. I didn't think of putting a jump at the end of my stub to the original implementation like that though. That is a good idea. I will have to add that in.