I've recently found out a bit more about separating debug info, and thought a consolidated reference might be handy.
The Idea
Most every distribution now provides separate debug packages which contain only the debug info, saving much space for the 99% of people who never want to start gdb.
This is achieved with objcopy and --only-keep-debug/--add-gnu-debuglink and is well explained in the man page.
What does this do?
This adds a .gnu_debuglink section to the binary with the name of debug file to look for.
$ gcc -g -shared -o libtest.so libtest.c $ objcopy --only-keep-debug libtest.so libtest.debug $ objcopy --add-gnu-debuglink=libtest.debug libtest.so $ objdump -s -j .gnu_debuglink libtest.so libtest.so: file format elf32-i386 Contents of section .gnu_debuglink: 0000 6c696274 6573742e 64656275 67000000 libtest.debug... 0010 52a7fd0a R...
The first part is the name of the file, the second part is a check-sum of debug-info file for later reference.
Build ID
Did you know that binaries also get stamped with a unique id when they are built? The ld --build-id flag stamps in a hash near the end of the link.
$ readelf --wide --sections ./libtest.so | grep build [ 1] .note.gnu.build-id NOTE 000000d4 0000d4 000024 00 A 0 0 4 $ objdump -s -j .note.gnu.build-id libtest.so libtest.so: file format elf32-i386 Contents of section .note.gnu.build-id: 00d4 04000000 14000000 03000000 474e5500 ............GNU. 00e4 a07ab0e4 7cd54f60 0f5cf66b 5799b05c .z..|.O`.\.kW..\ 00f4 2d43f456 -C.V
Incase you're wondering what the format of that is...
uint32 name_size; /* size of the name */ uint32 hash_size; /* size of the hash */ uint32 identifier; /* NT_GNU_BUILD_ID == 0x3 */ char name[name_size]; /* the name "GNU" */ char hash[hash_size]; /* the hash */
Although the actual file may change (due to prelink or similar) the hash will not be updated and remain constant.
Finding the debug info files
The last piece of the puzzle is how gdb attempts to find the debug-info files when it is run. The main variable influencing this is debug-file-directory.
(gdb) show debug-file-directory The directory where separate debug symbols are searched for is "/usr/lib/debug".
The first thing gdb does, which you can verify via an strace, is search for a file called [debug-file-directory]/.build-id/xx/yyyyyy.debug; where xx is the first two hexadecimal digits of the hash, and yyy the rest of it:
$ objdump -s -j .note.gnu.build-id /bin/ls
/bin/ls: file format elf32-i386
Contents of section .note.gnu.build-id:
8048168 04000000 14000000 03000000 474e5500 ............GNU.
8048178 c6fd8024 2a11673c 7c6a5af6 2c65b1b5 ...$*.g<|jZ.,e..
8048188 d7e13fd4 ..?.
... [running gdb /bin/ls] ...
access("/usr/lib/debug/.build-id/c6/fd80242a11673c7c6a5af62c65b1b5d7e13fd4.debug", F_OK) = -1 ENOENT (No such file or directory)
Next it moves onto the debug-link info filename. First it looks for the filename in same directory as the object being debugged. After that it looks for the file in a sub-directory called .debug/ in the same directory.
Finally, it prepends the debug-file-directory to the path of the object being inspected and looks for the debug info there. This is why the /usr/lib/debug directory looks like the root of a file-system; if you're looking for the debug-info of /usr/lib/libfoo.so it will be looked for in /usr/lib/debug/usr/lib/libfoo.so.
Interestingly, the sysroot and solib-search-path don't appear to have anything to do with these lookups. So if you change the sysroot, you also need to change the debug-file-directory to match.
However, most distributions make all this "just work", so hopefully you'll never have to worry about anyway!