It's time for part three in the Anti-Debugging Series. With this post we will stay in the realm of "API based" anti-debugging techniques but go a bit deeper into some techniques that are more complex and significantly more interesting. Today we will analyze one method of detecting an attached debugger, and a second method that can be used to detach a debugger from our running process.

Advanced API Based Anti-Debugging

There are a number of functions and API calls within the Windows operating system that are considered internal to the operating system and thus not documented well for the average developer. Many of these functions have undergone extensive research and reverse engineering to be able to understand how they operate and what can be achieved using them. One such poorly documented API function is the NtQueryInformationProcess function which is used to retrieve information about a target process. The function prototype looks like the following:

NTSTATUS WINAPI NtQueryInformationProcess(
    __in HANDLE ProcessHandle,
    __in PROCESSINFOCLASS ProcessInformationClass, 
    __out PVOID ProcessInformation,
    __in ULONG ProcessInformationLength,
    __out_opt PULONG ReturnLength

This function resides within the Ntdll.dll file and is not an exported function. Because of this, we must use run-time dynamic linking to gain access to the functionality. Run-time dynamic linking is the dynamic loading of a library and mapping of functions within the library to a function pointer allowing it to be called and executed. To load our function, "NtQueryInformationProcess", we first call LoadLibrary("ntdll.dll") and then execute GetProcAddress(HMODULE, "NtQueryInformationProcess") to receive a pointer to our required function.

FARPROC _NtQueryInformationProcess;
hmod = LoadLibrary(L"ntdll.dll");
_NtQueryInformationProcess = GetProcAddress(hmod, "NtQueryInformationProcess");

Note: The dynamic linking method is slightly different when using C++ due to declaration differences.

Once we have a function pointer to the NtQueryInformationProcess function, we can call the API. The function call takes five parameters, the first two of which are the most interesting to our anti-debugging efforts. The first parameter is a HANDLE to the target process that we wish to interrogate. Since we are trying to determine information about our own process, we will use a HANDLE that points to ourselves. By default, a HANDLE value of -1 will instruct the function to use the current process as the target. The second parameter is a value indicating what type of information is being requested from the target process. In the Microsoft MSDN documentation there are four documented values for this parameter ProcessBasicInformation (0), ProcessDebugPort (7), ProcessWow64Information (26), and ProcessImageFileName (27). There are other undocumented values that can be passed in, some of which allow for interesting anti-debugging techniques, however we will focus on the ProcessDebugPort (7) value. This value, when used as the second parameter in the NtQueryInformationProcess function will return a DWORD, returned in the address of the third parameter, indicating the DebugPort that is currently available for the process. If a non-zero value is returned, indicating that a DebugPort exists, we can be sure that a debugger is attached and act accordingly.

status = (_NtQueryInformationProcess) (-1, 0x07, &retVal, 4, NULL);

if (retVal != 0) {
    //Debugger Detected - Do Something Here
} else {
   //No Debugger Detected - Continue

The second anti-debugging method we will look at today also uses run-time dynamic linking of the Ntdll.dll library along with GetProcAddress() to gain access to the NtSetInformationThread function. This function's primary purpose is to modify thread specific data for a targeted thread.

NTSTATUS NTAPI NtSetInformationThread(
    __in HANDLE ThreadHandle, 
    __ in THREAD_INFORMATION_CLASS ThreadInformationClass,
    __in PVOID ThreadInformation
    __in ULONG ThreadInformationLength

For the anti-debugging use of this function we are again only interested in two particular parameters. The first parameter is an identifier to the thread we wish to target and the second parameter is the particular information we wish to modify on the target thread. To get a pointer to our current thread we will use a call to GetCurrentThread(). We will submit that as the first parameter and the enum value for ThreadHideFromDebugger, 0x11, as the second parameter. If a debugger is attached and we pass in 0x11 to NtSetInformationProcess, our process will immediately detach any attached debugger and terminate the process.

lib = LoadLibrary(L"ntdll.dll");
_NtSetInformationThread = GetProcAddress(lib, "NtSetInformationThread");

(_NtSetInformationThread) (GetCurrentThread(), 0x11, 0, 0);

MessageBox(NULL, L"Debugger Detached", L"Debugger Detached", MB_OK);

Many of the Microsoft API calls are intentionally not well documented to discourage their use/abuse. In this case we can make calls to two non-exported functions within the Ntdll.dll library to achieve our goals of detecting or detaching a debugger from our process. There are a number of other methods of API based anti-debugging, feel free to comment about them below.

Stay tuned for our next installment as we touch on process and thread block anti-debugging.

About Tyler Shields

Tyler Shields is a Senior Researcher for the Veracode Research Lab whose responsibilities include understanding and examining interesting and relevant security and attack methods for integration into the Veracode product offerings. He also keeps track of new developments from other computer science and information security researchers to ensure that Veracode technologies are always kept in line with the most recent security advancements.

Comments (5)

Adam Leyshon | January 8, 2009 9:07 am

Very informative, I wish MS would document these functions better as they can be quite handy sometimes.

A John Doe | January 8, 2009 6:55 pm

Meh, API anti-debugging techniques are fairly useless. Reverse engineers are very familiar with them and they are exceedingly easy to bypass. Might as well use them a bit anyway because they are so easy to implement, but they should be a part of a MUCH MUCH stronger defensive scheme. If they form a notable percentage of the defensive scheme, time is being wasted that could be spent making the defended asset better IMO.

HuRrIcAnE | January 30, 2009 11:14 pm

I'm one of the fan here, and I wanna said that these type of subjects are increduble .. And I agree the abover's comment that these functions are not that much difficult to bypass its. But what I'm trying to said that the Art of REC is not just for having fun on CRACKING programs and protections, It's like to learn, to improve something, even if it's useless.

what I'm agree here is that many private and public schemes will at less use this art of protection in somehow, somewhere to but it at that greatfull steps .

I hope Mr. Shields to continue your path and I'm one of your followers.


jn | February 22, 2010 10:21 pm

"If a debugger is attached and we pass in 0x11 to NtSetInformationProcess, our process will immediately detach any attached debugger and terminate the process."
This statement is not correct:
1) you are actually referring to NtSetInformationThread
2) the process is not terminated but debug events for the specified thread are not received by the debugger anymore.

Jane Doe | July 12, 2016 3:59 pm

All of these functions are exported from ntdll, just not directly available using the Win API in usermode applications.

Just a correction.

Please Post Your Comments & Reviews

Your email address will not be published. Required fields are marked *

Love to learn about Application Security?

Get all the latest news, tips and articles delivered right to your inbox.