Appendix G — After the program is dead: reading a dump

A debugger works on a live program. You stop it, look at values, step a line at a time. chapter 18 is where that lives. But the most common failure in practice does not meet you that way. The program is already dead, it died at three in the morning on somebody else’s machine, and what is left is a single file.

This appendix is about reading that file.

What a dump is#

Nothing grand. A dump is a photograph of a process’s memory, taken as it was at that instant. Register values and signal information are attached. That is all.

To know what appears in the photograph you first have to see what a process’s memory looks like. On Linux a program can see that for itself.

/* What is in a dump --- a process looks into its own memory.
   This is exactly what a core dump does: write this list and these bytes to a file. */
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int global_zero; int global_init = 0x41424344;
int main(void){
    char *heap = malloc(4096); strcpy(heap, "heap");
    char stackbuf[64]; strcpy(stackbuf, "stack");
    printf("code=%p  rodata=%p  data=%p  bss=%p  heap=%p  stack=%p\n",
           (void*)main, (void*)"ro", (void*)&global_init, (void*)&global_zero, (void*)heap, (void*)stackbuf);
    FILE *f=fopen("/proc/self/maps","r"); char line[512]; int n=0;
    puts("\n--- /proc/self/maps (first 8 lines) ---");
    while (fgets(line,sizeof line,f) && n++<8) fputs(line,stdout);
    fclose(f);
    /* read our own memory --- this is what a dump does */
    FILE *m=fopen("/proc/self/mem","rb");
    if (m){ fseek(m,(long)(size_t)&global_init,SEEK_SET); unsigned v=0;
        size_t got=fread(&v,1,4,m);
        printf("\nread through /proc/self/mem at &global_init = 0x%08X (%zu bytes)\n", v, got);
        fclose(m); }
    free(heap); return 0;
}

code=0x55a0aca901c9  rodata=0x55a0aca91008  data=0x55a0aca93060  bss=0x55a0aca93074  heap=0x55a0bd6ef2a0  stack=0x7ffcaced5a90

--- /proc/self/maps (first 8 lines) ---
55a0aca8f000-55a0aca90000 r--p 00000000 00:56 1944683                    ./apx-postmortem/whats_in_a_dump
55a0aca90000-55a0aca91000 r-xp 00001000 00:56 1944683                    ./apx-postmortem/whats_in_a_dump
55a0aca91000-55a0aca92000 r--p 00002000 00:56 1944683                    ./apx-postmortem/whats_in_a_dump
55a0aca92000-55a0aca93000 r--p 00002000 00:56 1944683                    ./apx-postmortem/whats_in_a_dump
55a0aca93000-55a0aca94000 rw-p 00003000 00:56 1944683                    ./apx-postmortem/whats_in_a_dump
55a0bd6ef000-55a0bd710000 rw-p 00000000 00:00 0                          [heap]
7fb3e515a000-7fb3e515d000 rw-p 00000000 00:00 0 
7fb3e515d000-7fb3e5185000 r--p 00000000 00:67 5715590                    /usr/lib/x86_64-linux-gnu/libc.so.6

read through /proc/self/mem at &global_init = 0x41424344 (4 bytes)

The output shows three things, in the list of pieces from /proc/self/maps and in the bytes read back.

First, memory is not one block but a list of pieces. Each piece has a start and end address and permissions. r-xp can be read and executed but not written, so it is code; rw-p can be written, so it is data. r--p is read-only — where string literals live. The layout seen as a picture in chapter 5 appears here as a list.

Second, some pieces carry a label. [heap], [stack], and the paths of mapped files (libc.so.6 and the like). This list is what decides which address belongs where, and half of reading a dump is making that decision.

Third, an address alone is enough to read the bytes there. The demonstration finds the address of global_init through /proc/self/mem and reads 0x41424344 straight out of it.

★ Making a core dump is exactly this. The kernel walks a list of pieces like the one above and writes each piece’s bytes into a file. No magic — read, then write.

What is in it, and what is not#

There is a common misunderstanding about what a dump holds.

The last row of Table 115.1 is the fundamental limit. A dump shows you what state is wrong, never how it got that way. What you need for that is record and replay, or a sanitiser — record and replay is covered in chapter 111, sanitisers in chapter 18.

The shape of the file — a Linux core is an ELF#

A Linux core dump is not a separate format but an ELF (executable and linkable format) file. Only its kind differs — where an executable is ET_EXEC or ET_DYN, a core is ET_CORE.

A Windows minidump has a different format but holds the same things — memory pieces, registers, notes. The format is not the thing to memorise; what it holds is.

Turning an address into a name#

The first clue a dump gives is usually an address — something like 0x00005584a1b2c3d4. Turning that into a name such as parse_header takes debug information.

/* Two layers that turn addresses into names --- the symbol table and debug information.
   Here a live program unwinds its own stack, but a debugger reading a dump does
   exactly the same: collect addresses, then turn them into names and line numbers. */
#define _GNU_SOURCE
#include <execinfo.h>
#include <stdio.h>
#include <stdlib.h>

static void level3(void)                 /* static --- not in the dynamic symbol table */
{
    void *frames[16];
    int n = backtrace(frames, 16);       /* unwind and collect addresses */
    char **names = backtrace_symbols(frames, n);   /* try to put names on them */

    printf("frames captured: %d\n\n", n);
    for (int i = 0; i < n && i < 5; i++)
        printf("  [%d] %s\n", i, names[i]);
    free(names);
}
static void level2(void) { level3(); }
static void level1(void) { level2(); }

int main(void) { level1(); return 0; }

frames captured: 7

  [0] ./apx-postmortem/who_called_me(+0x1188) [0x55f828cf6188]
  [1] ./apx-postmortem/who_called_me(+0x121f) [0x55f828cf621f]
  [2] ./apx-postmortem/who_called_me(+0x122b) [0x55f828cf622b]
  [3] ./apx-postmortem/who_called_me(main+0x9) [0x55f828cf6237]
  [4] /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7ff16a981ca8]

The demonstration puts two of the layers that turn an address into a name, the symbol table and the debug information, on one screen. main comes out by name while level1 through level3 come out only as raw addresses. backtrace_symbols, which supplies the names, looks only in the dynamic symbol table (.dynsym), the table searched at run time; all three are static, so they are local symbols and never reach it (main reaches it only because the program is linked with -rdynamic). Look at the file’s static symbol table (.symtab) with nm and all three are there, and the difference shows in a single letter — a lowercase t is local, an uppercase T is global.

000000000000126e t level1
0000000000001262 t level2
0000000000001179 t level3
000000000000127a T main

The debug information knows more than that. Give addr2line those addresses and out come not just a function name but a file and a line number. So “the name is missing” does not mean the information is missing; it is a question of which table you looked in. When a name fails to appear, ask first: are the symbols gone, is the debug information gone, or am I looking at the wrong binary?

★ Build the shipping binary with -g too, and keep the debug information rather than discarding it. This is not a call to turn optimisation off — -O2 -g is no contradiction at all. After a failure, without that debug-information file there is simply no way left to turn an address into a name.

How the stack is unwound#

To learn “who called this” you have to climb back up the stack. There are two ways.

1. Follow the frame pointers. If each frame holds the location of the previous one, you climb the chain. Simple and fast.
2. Read the unwind tables. When optimisation removes the frame pointer (to save one register) the chain breaks. So the compiler builds a separate table — the call frame information (CFI) in .eh_frame, which records “in this address range, the return address is at such a place on the stack”.

This is the other side of chapter 18′s remark that a debugger looks strange on an optimised build. A tangled backtrace, or a frame gone entirely, is not the debugger being poor: the information vanished from the code. With the table it can be recovered; without it, it cannot.

The order to read in#

With a dump in front of you, what do you look at first? There is an order.

1. Where did it die — the program counter and the signal. SIGSEGV and SIGABRT part company right there (the latter is usually our own code calling it).
2. What address was it touching — null means a pointer used while still null (an unchecked malloc result, say); a very small value means a field access through a null pointer (p->field with a null p); an absurdly large value means a corrupted pointer.
3. Who called it — the backtrace. A strange frame is grounds to suspect the stack was already damaged.
4. Do the values make sense — is a length negative, does a pointer aim outside its own struct?
5. Form a hypothesis — and then reproduce it on a live program.

Making sure a dump is left — and the common reasons none is#

If there is no dump after a failure, everything above is useless. What to check on Linux.

What to take from this#