dyadica.net

Here is a post I have been meaning to write for a while now as it was quite a big issue for me at one time:

DEP is short for Data Execution Prevention, a technology which exists to prevent the execution of code from memory pages marked as none executable. This is done to reduce the attack surface available to malicious software that is trying to hijack a process.

This is great you might say, and yes it is, however if your application interops with native binaries or exposes a plugin model to 3rd parties (shockwave ActiveX component in my case) you may say… aggggghhhh.

A Bit of Background

I first came across DEP when I decided to add  facility for direct communication between my wiindows application shockwave .dcr/swf  movies. In order to do this part of the application has a windows form with the relevant ActiveX controls embedded within it.

Fine so far and yes it was in VS2005, however upon my first Build in VS2008 the following error was triggered:

Unable to get the window handle for the 'AxShockwaveCtl' control. Windowless ActiveX controls are not supported

After a little Google time in reference to this error DEP reared its ugly head as a possible/likely cause. This is due to the C# compiler in Visual Studio 2008 and the .NET 3.5 Framework (csc.exe) now generating PE files with the NXCOMPAT bit set.

A Bit More Detail

In the header of a PE file there is a flag called IMAGE_DLLCHARACTERISTICS_NX_COMPAT. This flag affects whether or not the OS enables DEP for a process. Setting this flag tells the OS that the image is compatible with DEP. For executable images, if this flag is set, the process is run with DEP enabled unless the machine is configured with the DEP policy set to AlwaysOff.

Since the C# compiler emits PE files which are MSIL only and therefore compatible with DEP, the output binaries from the VS 2008 and .NET 3.5 C# compilers have this flag set.

This means that by turning off DEP It would be possible to test weather DEP truly was the cause of the error.

Enabling and Disabling DEP in Vista

DEP can easily be enabled and disabled in Vista by using the following command line calls:

1. Open the Start Menu.
2. In the white line (Start Search) area, type cmd
3. Right click on Cmd (at top), and click on Run as Administrator.

This will open a elevated command prompt. To enable DEP type the following at the command prompt and press Enter:

bcdedit.exe /set {current} nx OptIn

If for some reason this command does not enable DEP after restarting the computer, then repeat the process and try the following code instead:

bcdedit.exe /set {current} nx AlwaysOn

To disable DEP repeat steps 1-3 and in the command prompt, type the following and press enter:

bcdedit.exe /set {current} nx AlwaysOff

With this facility a few tests were run and as suspected DEP was identified to be causing the error. This meat that all we need to do to resolve the error is manipulate status of the NXCOMPAT bit.

Controlling the Flag in the PE Header

If you are using visual studio this can be simply accomplished via adding a switch to the postbuild event of your project to mark your compiled project as DEP non compliant. The steps for doing this are as follows:

1. Right click on project in Visual Studio
2. Properties
3. Build Events
4. Edit Post Build with your bit switch code

My post build step (4) is as follows:

REM Mark project as DEP Noncompliant
call "$(DevEnvDir)..\..\VC\bin\vcvars32.bat"
call "$(DevEnvDir)..\..\VC\bin\editbin.exe" /NXCOMPAT:NO "$(TargetPath)"


Another possible way to do this is to open up the Visual Studio command prompt, browse to your exe location and type:

editbin.exe /NXCOMPAT:NO YourProgram.exe

However I have not tested this as of yet.

Please note that if you sign the binary in Visual Studio, flipping the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag in the post build step after the binary has been signed will result in an assembly that will fail strong name validation. To work around this sign your binary as part of the post build steps.

To do this, use SN.EXE from the Windows SDK.

So there you have it, Data Execution Prevention, problem and solution and methods of manipulation.