Prevent graceless crash from occurring in application calling library via P/Invoke

Question

I have a C# application App A that calls, via P/Invoke, a C++ (Qt) DLL App B. Let's assume I cannot, by any means, edit App B, but I know that App B is throwing an Out of Memory Exception under a set of conditions which I can replicate but not accurately test for before passing input from App A to App B. This exception does not pose any threat to App A and could be ignored if App B could be "reset" in some manner, but when App B crashes it calls abort() which in turn causes App A's runtime to terminate.

How can I prevent App B's inevitable, unpredictable, and generally mundane crash from impacting App A?

Notes:

• An implemented UnhandledExceptionHandler is ignored when this error is thrown.
• App B crashes due to a bug in QtWebkit which could feasibly be handled by App B via deleting the oversize object and returning null.
• App A does not report as out of memory and the machine has more than enough memory to perform the App B's operation several times over, regardless of the bug, but this memory is apparently not allocated to App B for whatever reason.

Answer 1

You create a shim process whose only purpose is to load App B and use some IPC mechanism to communicate with App A. App A spawns the shim and if the shim crashes then App A takes the appropriate action (respawns, communicates an error to the user, etc.).

Answer 2

You can handle the SIGABRT coming from the unmanaged code, but it's a little messy. It's possible it could be done completely in C#, but my method requires a C++/CLI assembly to make it work (at least it is all in one process).

Step 1: Create a C++/CLI Class Library project, in the *.h file put the following function

namespace ClassLibrary1 {

    public ref class Class1
    {
    public:
        static void callaborter();
    };
}

Step 2: in the cpp file put the following code:

// This is the main DLL file.

#include "stdafx.h"
#include <csetjmp>
#include <csignal>
#include <cstdlib>
#include <iostream>
#include "ClassLibrary1.h"
#include "..\Win32Project1\Win32Project1.h"


#pragma managed(push, off) 
jmp_buf env;

void on_sigabrt (int signum)
{
    longjmp (env, 1);
}

void run()
{
    if (setjmp (env) == 0) {
            signal(SIGABRT, &on_sigabrt);
            fnWin32Project1();
    }
    else {
            std::cout << "aborted\n";
    }
}

#pragma managed(pop)

void ClassLibrary1::Class1::callaborter()
{
    run();
}

I left the names of my classlibrary and class as default, and my aborting function is call fnWinProject1(). You will need to link the offending DLL directly to this project (just like it's in C++)

In your C# class, include a reference to the C++/CLI assembly and call the "callaborter" method.

The code in run will setup a catcher for the SIGABRT (called by abort()) and process it in on_sigabrt(), which will cause the whole function to bail. You can then clean up and try again.

I will leave it as an exercise for the OP to choose useful names.

Note: In Debug you will still get the abort dialog, just press Ignore to continue. In Release you will not see the dialog.

P.S. I should mention I built my response based on this stackoverflow question: How to Handle SIGABRT signal?