Here is a post I have been meaning to write for a while now as it was quite a big issue for me at one time:
DEP is short for Data Execution Prevention, a technology which exists to prevent the execution of code from memory pages marked as none executable. This is done to reduce the attack surface available to malicious software that is trying to hijack a process.
This is great you might say, and yes it is, however if your application interops with native binaries or exposes a plugin model to 3rd parties (shockwave ActiveX component in my case) you may say… aggggghhhh.
A Bit of Background
I first came across DEP when I decided to add facility for direct communication between my wiindows application shockwave .dcr/swf movies. In order to do this part of the application has a windows form with the relevant ActiveX controls embedded within it.
Fine so far and yes it was in VS2005, however upon my first Build in VS2008 the following error was triggered:
Unable to get the window handle for the 'AxShockwaveCtl' control. Windowless ActiveX controls are not supported
After a little Google time in reference to this error DEP reared its ugly head as a possible/likely cause. This is due to the C# compiler in Visual Studio 2008 and the .NET 3.5 Framework (csc.exe) now generating PE files with the NXCOMPAT bit set. Read..