Thursday, March 29, 2012

Stack Buffer Overrun Killed My CLR!

So I ran into an interesting problem the last few days. And by ran into I mean made me lose more hair than I have already lost! It took quite a bit of research and I thought I'd help the Internet by sharing what I ran into.

The Problem


We've been working on a new version of our product that was originally built on .Net 3.5 but has been migrated to .Net 4.0. Part of that application does a DllImport on some unmanaged code which has worked great in .Net 3.5.



Now that we're using .Net 4.0 the method that uses that DllImport is crashing. And I mean crashing HARD. There's a try/catch around the method call but it's not getting that far. There's no minidump to be seen, but there is an entry in the Windows event log:

Faulting application name: MyAppService.exe, version: 2.0.0.2, time stamp: 0x4f738b08
Faulting module name: clr.dll, version: 4.0.30319.239, time stamp: 0x4e181a6d
Exception code: 0xc0000409
Fault offset: 0x002b60d0
Faulting process id: 0x15b8
Faulting application start time: 0x01cd0dc5e21cf6a0
Faulting application path: C:\Program Files (x86)\MyCompany\MyAppService.exe
Faulting module path: C:\Windows\Microsoft.NET\Framework\v4.0.30319\clr.dll
Report Id: 5453534c-79b9-11e1-8970-000c29dad353

Research


Wow. The freaking CLR died. No wonder no exception was caught. I looked up the exception code and it said that it was a Stack Buffer Overrun exception.

Stack buffer overrun? Can that happen in .Net? No, not really, but I was calling into unmanaged code which led me to believe that the unmanaged code was bad. Of course, if that code was bad, why did it work when .Net 3.5 was calling it?

Looking through StackOverflow.com I found a few things. Apparently stack buffer overrun protection has been in place in Windows for unmanaged code for a while but was ignored until .Net 4.0. I can't say for sure that this is why I'm running into problems but seems likely.

The Solution


Then I ran across this and this that talk about changing the calling convention from __declspec to __stdcall. At first I didn't think much of it because this was all C++ code and I was dealing with .Net. And then it hit me, that DllImport is using Cdecl isn't it? So then I changed the CallingConvention to use the StdCall and things started to magically work!



I wish I knew all the technical details about why this made things work but if this helps someone else out, I'll be happy enough.

2 comments:

  1. http://blogs.msdn.com/b/oldnewthing/archive/2004/01/08/48616.aspx

    Might help explain some of the differences. Kinda fascinating.

    ReplyDelete
  2. Thank you for this.
    I just had this exact issue with an unmanaged 3rd party library that I had written a .Net wrapper for.

    The most fun was that it ran perfectly fine out of the IDE and wouldn't crash unless I ran the application on its own.

    ReplyDelete

Note: Only a member of this blog may post a comment.