Win32 Debug CRT Heap Internals

... by Andrew Birkett (andy@nobugs.org)

If you are lazy, skip the explanation and jump to the table at the bottom of the page

When you compile programs with DevStudio in debug mode, all of your calls to malloc() and free() use a special "debug" implementation. Rather than being blazingly fast, the debug heap concerns itself with spotting heap errors. It achieves this by surrounding your memory blocks with guard bytes (aka "no mans land", 0xFD) so that it can detect buffer overruns and underruns. It also initialises newly allocated memory to a fixed value (0xCD) to aid reproducability. Finally, it sets free()d blocks to a known value (0xDD) so that it can detect that people are writing through dangling pointers.

Mnemonics for remembering what each fill-pattern means:

• The newly allocated memory (0xCD) is Clean memory.
• The free()d memory (0xDD) is Dead memory.
• The guard bytes (0xFD) are like Fences around your memory.

The debug CRT heap defers most of the heavy work to the Win32 heap functions HeapAlloc() and HeapFree(). Therefore, you won't see any first-fit or "buddy system" code in the debug CRT. The 4Gb virtual memory space which you process has is sliced up and managed by the Win32 heap inside kernel32.dll.

If you call malloc(8), the debug CRT will request a 48 byte block from HeapAlloc(). It needs the extra 40 bytes to store information about the memory blocks - such as the file/line where malloc() was called, and space for links to the next/prev heap block. In the table below, all of the debug CRT information is colored in shades of red.

HeapAlloc() itself needs bookkeeping information. The HeapAlloc(40) call will, in turn, reserve a total of 80 bytes from your process's address space. Eight bytes of bookkeeping appear below the requested 40 bytes, and the other 32 bytes appear above it. In the table below, the Win32 heap bookkeeping is colored grey. The memory which you get back from HeapAlloc() is always initialised to the 4 byte pattern 0xBAADF00D.

(As an aside, when you request pages from the VM manager via VirtualAlloc, they are initialized to zero, so HeapAlloc is actually doing additional work to re-initialize them to this pattern).

Once the debug CRT has got it's 40 byte block, it will fill in it's book-keeping information. The first 2 words are links to the previous and next blocks on the CRT heap. The choice of names is confusing, because the "next" pointer actually takes you the block which was allocated before this one chronologically, while the "previous" pointer takes you to the one allocated subsequently. The reason for the naming is that the linked list starts at the last-allocated block, and progresses back in time as you follow "next" links. The debug CRT heap also internally maintains pointers to the first and last blocks (_pFirstBlock and _pLastBlock) to allow heap-checking code to traverse all the blocks.

If the filename and line number of the malloc() call are known, they are stored in the next 2 words. Following that, the next word tells you how many bytes were requested. The next word gives a type field. Usually this is "1" which means a normal block, allocated by malloc/new. It will be "2" for blocks allocate by the CRT for its own internal purposes, and "0" for blocks which have been freed but not released back to the win32 heap (see below for more info). The final word is a simple counter which increases everytime an allocation is made.

Surrounding the 8-byte malloc()'d memory there are areas of "no mans land". These are filled with a known value (0xFD), and when the block is free()d, the CRT will check that they still have the right value. If they've changed, then your program contains an error. Unfortunately, the corruption may have happened a long time ago. You can use Purify or Boundschecker to stop at the corruption point, or if you don't fancy spending any money, you can wait a few days until I write an article telling you how to do it using only a bit of ingenuity!

The eight bytes which were originally requested are initialised with 0xCD. If you see this pattern appearing in your variables, you have forgotten to initialise something.

When you call free() on the 8-byte block, the CRT sets the whole 48-byte block (including its bookkeeping) to 0xDDDDDDDD. This means that it can tell if the block gets subsequently altered (ie. via a dangling pointer) by checking that the pattern is still there.

At this point, two things can happen. Normally, HeapFree() will be called to return the block to the win32 heap. This causes the block to be overwritten with the win32 heap's "freed memory" pattern, which is 0xFEEEFEEE. Note that the debug CRT does not maintain any "free lists" - all of that is done within the black box of HeapFree().

However, you can also tell the debug heap to keep hold of free()d blocks. You do this by passing the _CRTDBG_DELAY_FREE_MEM_DF flag to _CrtSetDbgFlag(). In this case, the debug CRT will keep hold of the block. This is useful if you are trying to track down a dangling pointer error, since memory blocks will not be reused and you should expect them to remain filled with 0xDDDDDDDD unless someone is writing to the free()d block. You can call _CrtCheckMemory() and it will tell you if any of these values have been tampered with.

Here's an allocation I prepared earlier ...

I called malloc(8) followed by free() and stepped through the CRT calls to see how the memory was changed. Read the columns from left to right, and you will see what values appear in memory at various stages during malloc() and free(). The call to malloc(8) returned a block at address 0x00321000, and I've included offsets from that address so that you can find out the information for one of your allocations.

(I've tried to helpfully color-code things. Blue and grey is for Win32 heap stuff, and reds are for the debug crt heap stuff. If you hate the color scheme, the colors are set up using CSS at the top of this .html file - go edit them yourself!)

Any comments or feedback? Please email me.