Appendix I — After the program is dead: reading a dump

A debugger works on a live program. You stop it, look at values, step a line at a time. chapter 18 is where that lives. But the most common failure in practice does not meet you that way. The program is already dead, it died at three in the morning on somebody else’s machine, and what is left is a single file.

This appendix is about reading that file.

What a dump is#

Nothing grand. A dump is a photograph of a process’s memory, taken as it was at that instant. Register values and signal information are attached. That is all.

To know what appears in the photograph you first have to see what a process’s memory looks like. On Linux a program can see that for itself.

/* 덤프에 무엇이 들어 있는가 --- 프로세스가 제 기억을 스스로 들여다본다.
   코어 덤프가 하는 일이 정확히 이것이다: 이 목록과 이 바이트들을 파일에 적는 것. */
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int global_zero; int global_init = 0x41424344;
int main(void){
    char *heap = malloc(4096); strcpy(heap, "heap");
    char stackbuf[64]; strcpy(stackbuf, "stack");
    printf("code=%p  rodata=%p  data=%p  bss=%p  heap=%p  stack=%p\n",
           (void*)main, (void*)"ro", (void*)&global_init, (void*)&global_zero, (void*)heap, (void*)stackbuf);
    FILE *f=fopen("/proc/self/maps","r"); char line[512]; int n=0;
    puts("\n--- /proc/self/maps (first 8 lines) ---");
    while (fgets(line,sizeof line,f) && n++<8) fputs(line,stdout);
    fclose(f);
    /* 제 기억을 스스로 읽어 본다 --- 덤프가 하는 일이 이것이다 */
    FILE *m=fopen("/proc/self/mem","rb");
    if (m){ fseek(m,(long)(size_t)&global_init,SEEK_SET); unsigned v=0;
        size_t got=fread(&v,1,4,m);
        printf("\nread through /proc/self/mem at &global_init = 0x%08X (%zu bytes)\n", v, got);
        fclose(m); }
    free(heap); return 0;
}

code=0x55ceac3d31c9  rodata=0x55ceac3d4008  data=0x55ceac3d6060  bss=0x55ceac3d6074  heap=0x55ced506f2a0  stack=0x7fff05b38400

--- /proc/self/maps (first 8 lines) ---
55ceac3d2000-55ceac3d3000 r--p 00000000 00:57 1075839                    ./apx-postmortem/whats_in_a_dump
55ceac3d3000-55ceac3d4000 r-xp 00001000 00:57 1075839                    ./apx-postmortem/whats_in_a_dump
55ceac3d4000-55ceac3d5000 r--p 00002000 00:57 1075839                    ./apx-postmortem/whats_in_a_dump
55ceac3d5000-55ceac3d6000 r--p 00002000 00:57 1075839                    ./apx-postmortem/whats_in_a_dump
55ceac3d6000-55ceac3d7000 rw-p 00003000 00:57 1075839                    ./apx-postmortem/whats_in_a_dump
55ced506f000-55ced5090000 rw-p 00000000 00:00 0                          [heap]
7f534fe3d000-7f534fe40000 rw-p 00000000 00:00 0 
7f534fe40000-7f534fe68000 r--p 00000000 00:66 3857579                    /usr/lib/x86_64-linux-gnu/libc.so.6

read through /proc/self/mem at &global_init = 0x41424344 (4 bytes)

Three things to note.

First, memory is not one block but a list of pieces. Each piece has a start and end address and permissions. r-xp can be read and executed but not written, so it is code; rw-p can be written, so it is data. r--p is read-only — where string literals live. The layout seen as a picture in chapter 5 appears here as a list.

Second, some pieces carry a label. [heap], [stack], and the paths of mapped files (libc.so.6 and the like). This list is what decides which address belongs where, and half of reading a dump is making that decision.

Third, an address alone is enough to read the bytes there. The demonstration finds the address of global_init through /proc/self/mem and reads 0x41424344 straight out of it.

★ Making a core dump is exactly this. The kernel walks a list of pieces like the one above and writes each piece’s bytes into a file. No magic — read, then write.

What is in it, and what is not#

This is where a misunderstanding splits off.

The last row of Table 105.1 is the fundamental limit. A dump shows you what state is wrong, never how it got that way. What you need for that is record and replay, or a sanitiser — that story is in chapter 18.

The shape of the file — a Linux core is an ELF#

A Linux core dump is not a separate format but an ELF (executable and linkable format) file. Only its kind differs — where an executable is ET_EXEC or ET_DYN, a core is ET_CORE.

A Windows minidump has a different format but holds the same things — memory pieces, registers, notes. The format is not the thing to memorise; what it holds is.

Turning an address into a name#

The first clue a dump gives is usually an address — something like 0x00005584a1b2c3d4. Turning that into a name such as parse_header takes debug information.

/* 주소를 이름으로 바꾸는 두 층 --- 심볼 표와 디버그 정보.
   여기서는 살아 있는 프로그램이 제 스택을 되감지만, 덤프를 읽을 때 디버거가 하는
   일이 정확히 같다: 주소를 모아, 그것을 이름과 줄 번호로 바꾼다. */
#define _GNU_SOURCE
#include <execinfo.h>
#include <stdio.h>
#include <stdlib.h>

static void level3(void)                 /* static --- 동적 심볼 표에 오르지 않는다 */
{
    void *frames[16];
    int n = backtrace(frames, 16);       /* 되감아 주소를 모은다 */
    char **names = backtrace_symbols(frames, n);   /* 이름을 붙여 본다 */

    printf("frames captured: %d\n\n", n);
    for (int i = 0; i < n && i < 5; i++)
        printf("  [%d] %s\n", i, names[i]);
    free(names);

    puts("\nnotice: main is named, the static functions are not.");
    puts("a name here comes from the *symbol table*, and static functions are local");
    puts("symbols -- they never reach the dynamic one. the debug information (DWARF)");
    puts("still knows them, which is what addr2line reads.");
}
static void level2(void) { level3(); }
static void level1(void) { level2(); }

int main(void) { level1(); return 0; }

frames captured: 7

  [0] ./apx-postmortem/who_called_me(+0x1198) [0x55ef65393198]
  [1] ./apx-postmortem/who_called_me(+0x126b) [0x55ef6539326b]
  [2] ./apx-postmortem/who_called_me(+0x1277) [0x55ef65393277]
  [3] ./apx-postmortem/who_called_me(+0x1283) [0x55ef65393283]
  [4] /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7fe3bd52fca8]

notice: main is named, the static functions are not.
a name here comes from the *symbol table*, and static functions are local
symbols -- they never reach the dynamic one. the debug information (DWARF)
still knows them, which is what addr2line reads.

The demonstration puts both layers on one screen. main comes out by name while level1 through level3 come out only as address fragments. All three are static, so they are local symbols and never reach the table that supplies names at run time. Look at the symbol table directly and the difference shows in a single letter — a lowercase t is local, an uppercase T is global.

000000000000126e t level1
0000000000001262 t level2
0000000000001179 t level3
000000000000127a T main

But the debug information knows all three. Give addr2line those addresses and out come a file and a line number. So “the name is missing” does not mean the information is missing; it is a question of which table you looked in. When a name fails to appear, ask first: are the symbols gone, is the debug information gone, or am I looking at the wrong binary?

★ Build the shipping binary with -g too, and keep the debug information rather than discarding it. This is not a call to turn optimisation off — -O2 -g is no contradiction at all. After a failure, without that file there is simply no way left to turn an address into a name.

How the stack is unwound#

To learn “who called this” you have to climb back up the stack. There are two ways.

1. Follow the frame pointers. If each frame holds the location of the previous one, you climb the chain. Simple and fast.
2. Read the unwind tables. When optimisation removes the frame pointer (to save one register) the chain breaks. So the compiler builds a separate table — the CFI information in .eh_frame, which records “in this address range, the return address is at such a place on the stack”.

This is the other side of chapter 18′s remark that a debugger looks strange on an optimised build. A tangled backtrace, or a frame gone entirely, is not the debugger being poor: the information vanished from the code. With the table it can be recovered; without it, it cannot.

The order to read in#

With a dump in front of you, what do you look at first? There is an order.

1. Where did it die — the program counter and the signal. SIGSEGV and SIGABRT part company right there (the latter is usually our own code calling it).
2. What address was it touching — null means an uninitialised pointer; a very small value means a field access through a null pointer (p->field with a null p); an absurdly large value means a corrupted pointer.
3. Who called it — the backtrace. A strange frame is grounds to suspect the stack was already damaged.
4. Do the values make sense — is a length negative, does a pointer aim outside its own struct?
5. Form a hypothesis — and then reproduce it on a live program.

Making sure a dump is left — and the common reasons none is#

If there is no dump after a failure, everything above is useless. What to check on Linux.

What to take from this#