Beware AfxGetInstanceHandle() call In MFC Extension dll

This crash was first in top charts in our product's latest release. It was with a mysterious call stack contains only MFC/Win32 calls and a freed(MEM_FREE) address.

 # Call Site
00 0x00000001`1cd15c70
01 user32!DispatchHookA
02 user32!CallHookWithSEH
03 user32!_fnHkINLPMOUSEHOOKSTRUCTEX
04 ntdll!KiUserCallbackDispatcherContinue
05 win32u!NtUserCallNextHookEx
06 user32!CallNextHookEx
07 user32!DispatchHookA
08 user32!CallHookWithSEH
09 user32!_fnHkINLPMOUSEHOOKSTRUCTEX
0a ntdll!KiUserCallbackDispatcherContinue
0b win32u!NtUserCallNextHookEx
0c user32!CallNextHookEx
0d ieframe!TLSMouseHookProc
0e user32!DispatchHookW
0f user32!CallHookWithSEH
10 user32!_fnHkINLPMOUSEHOOKSTRUCTEX
11 ntdll!KiUserCallbackDispatcherContinue
12 win32u!NtUserPeekMessage
13 user32!PeekMessage
14 user32!PeekMessageA
15 mfc140!CWinThread::Run
16 mfc140!AfxWinMain
17 MyApp!_security_check_cookie
18 MyApp!_security_check_cookie
19 kernel32!BaseThreadInitThunk
1a ntdll!RtlUserThreadStart 

It was pretty simple to say how it was occurred. It was just trying to call a hook procedure in a bogus memory address.

My initial hypothesis was that the hook procedure is in an unloaded dll. So i did some research and  created some sample app to verify it. The result was different from the expected. In this case, call stack must have <Unloaded_DllName> which was negative in crash scenario.

So what happened in this crash?

The reason was strange. A third party library initialization needs the instance handle of caller. It was obtained by AfxGetInstanceHandle() call. Here is the catch. The caller was a MFC extension dll. Extension DLLs do not have their own module state. They take on the state of the calling application or DLL. So it has returned instance handle of caller of extension dll.

I don't know the internal implementation of third party library. It seems, it was using that handle for installing a global hook  using SetWindowsHookExA(). The global hook procedure is actually part of  extension dll since it is linked with third party library. The  third party library tried to invoke hook procedure in caller dll since wrong instance handle is provided. Thus it has crashed.

To fix this  dll instance handle is obtained from dll main.