No source code? No problem! That’s the motto of the binary analyst.
We at Veracode have pushed the limits of static analysis (studying a program’s behavior without running it) to automatically detect and report security vulnerabilities in our customers’ codebases. Doing binary static analysis by hand is still a worthwhile skill, however, with myriad practical uses:
- Uncovering the behavior of malware
- Patching bugs in old, unsupported programs
- Verifying a program does what it claims it does
- Looking for evidence of stolen code
- Reverse engineering protocols and file formats for product compatibility
- Realizing just how much other people can learn about your own code!
Laws concerning reverse engineering third-party programs without permission vary around the world, but in this tutorial we will use an open source program that is safe for everyone to dissect.
The simplest tool for binary analysis of C/C++ is a disassembler, which reveals the raw assembly opcodes the compiler produced. There are many free and commercial disassemblers for Intel binaries to pick from. In the past few years, however, decompiler technology (conventionally considered “impossible”) has really picked up. A C decompiler is able to reconstruct the disassembly into something approximating the original source code, minus niceties such as variable and function names, comments, macros, and anything else that is there to help programmers rather than the computer. The industrial-strength commercial decompiler solution is Hex-Rays, and this tutorial certainly applies to its use, but I am using a new tool still under active development (i.e. it’s beta) which is affordable to any hobbyist at home with access to a Mac. Hopper is a disassembler which can handle 32-bit or 64-bit programs for both Windows and OSX (other platforms, including iOS/ARM, are in development) with a basic but surprisingly functional decompiler mode. It costs $29 on the App Store or direct from the creator. With Hopper, anyone can take a look into an application’s internals without needing a detailed understanding of assembly (although you will still need to learn it if you want to get very serious).
(No-one at Veracode is affiliated with Hopper or its creator.)
This tutorial is aimed at experienced C programmers who are beginners in static analysis. The binary we examine is non-malicious and non-obfuscated, and is not run through the highest optimization settings of the compiler. We will start at line one and proceed linearly, just to get a feel for how to read decompiled code. All decompilers have strange quirks owing to the medium of disassembly they have to work with, so one needs to learn to spot and deal with these quirks that are not seen in proper source code.
We will be looking at an ancient piece of C code for Unix called metamail, which was a mimetype helper that received base64-encoded attachments from email clients and opened the appropriate viewing program. I chose it because we use it as a standard test case at Veracode owing to its extreme Unix-ness and the fact that few new analysts will have examined it before. Having reviewed it during training, I strongly recommend that you do not use it on a production system (not that there is any use for it in this millennium). The source code is yonder, but it is so old that I had to change a few functions to return 0 instead of void to get llvm to compile it for OSX. I also ran the
strip utility to remove any debug symbol names of internal functions that may be there, which would make it too easy (you will almost never see these in distributed closed-source programs). Download my metamail binary here.
When you open metamail with the “Read Executable” button in the upper left, you will be dumped here:
This is the asm prologue before we reach C’s
main. Skip down to the first
call instruction. Place your cursor over the
sub_1000018b0 symbol – the address of the first function called, which presumably is
main – and press enter to jump to it.
We are now faced with a wall of inscrutable-at-fir-st-glance assembly, but there is no need to panic if it seems overwhelming. In the upper right hand corner of Hopper is the Pseudo Code button. It will pop up a C-like reconstruction of the function.
External (i.e. dynamically linked) library functions will show their names just like in source code, and are your waypoints which make it much easier to understand what’s going on. Internal functions will be have a default name of “sub” for “subroutine” followed by their address. Local variables default to “var” followed by a number. Hopper’s pseudocode viewer does not attempt to do much type guessing, nor does it completely write out the use of registers, although it collapses as much as it can manage. (This means you can’t just copy/paste the C code and compile it into a new program without substantial editing.) In a 32-bit program you will see
eax, ebx, ecx, edx, esi, edi, ebp, esp, and in a 64-bit one you will also see
rax, rbx, rcx, rdx, rsi, rdi, rbp, rsp and some unnamed registers that look like
r<num>. In optimized code, you may also see partial registers
ax, bx, cx, dx (16 bits) and
ah/al, bh/bl, ch/cl, dh/dl (8 bits).
If you have never looked at assembly before, just know that the registers are named storage locations inside the processor itself, and compilers try to place variables into registers as much as possible because it is much faster than reading and writing RAM.
esp/rsp refer to the top of the stack,
ebp/rbp to the base of the current stack frame, and
eax/rax is normally where a function’s return value is received. In the disassembly view of a function, you will normally see it move the stack pointer. This creates the storage area for the local variables. Rewriting these from relative offsets to the stack base to a name like
var_40 is a nice thing that the decompiler view does for us.
The decompiler view does not support in-place text editing, and disassembly-view comments do not carry over to decompiliation view, so in the following screenshots I have copied the decompiled C to a code editor for annotating with explanatory comments. (You can also directly rename symbols in the disassembler view with ‘n’ when you have an idea what they’re for, but we won’t be doing that to keep the symbol name consistent across screenshots.) Let’s focus on this block for now (first screenshot is of the disassembly, second is of the decompilation):
rsi are the two arguments to C’s
main function, which you would know as
rsi is actually optimized in the assembly as only its lower half
argc is considered to be 32 bits by the compiler. The decompiler “widens” it back to full size for clarity. Hopper does not yet support auto-decoding function arguments (it currently shows “void” as a placeholder) so you need to look at what registers or stack locations the function accesses before getting down to business. On the x86-64 calling convention we are using, arguments in order are
rdi, rsi, rdx, rcx, r8, r9. (More mature, high-end decompilers already handle this for you.) The two arguments are copied to their own local variables on the stack.
var_144 is set to a function pointer. Switching back to disassembly view, we see that the
sub_100001620 pointer is first loaded into
rcx and then copied from there to
var_144 and then from there to
rsi. See how the decompiler abstracted that for us? It’s then passed as the second argument to
signal so that it will be set up as the handler for signal
0x2 (stored in
var_156), which according to the documentation is
SIGINT. Place your cursor on the
sub_100001620 reference and press enter to see what function this is. Go ahead and annotate that it’s a signal handler.
Before worrying about anything else, we see several standard functions; another call to
signal, followed by
kill on the result of
getpid, so we know already that the process is attempting to send itself a signal in its own signal handler.
Keeping in mind that the
void argument list is a placeholder, you can see by inspecting the function body that it assumes there is a value in
rdi (and we can see
rdi being populated in
main before the call). We know from signaling documentation that handlers receive the integer value of the signal they are handling. We already know that it at the very least will be processing
SIGINT, but there’s no reason this handler can’t be attached to other signals as well (spoiler alert: this will happen). Work through the rest of the function assuming
var_24 has the value of
We see that the handler calls another subroutine. Follow through in disassembler view to
sub_100009c50 with enter (backspace will take you to your previous function).
Uh-oh. Things are getting uglier.
What on earth is
*0x10000F988? It is a value with a fixed address, i.e. a global variable. If you hover over it in disassembly view, you will see that it has a default value of zero. With it highlighted, press ‘x’ to bring up the cross-reference window. Aside from the current spot, where it is being checked for zero, it is set to 1 in
sub_100009bd0, which is… well, let us not worry about it for now, as we will quickly get into a mess of function calls. We now know that
0x10000F988 is a boolean flag. If it’s set, we are doing an
ioctl with standard output and standard input. Don’t worry too much about what exactly it’s doing, as
ioctl takes arbitrary values and does arbitrary things pretty much by definition (one would have to dig into the macros for the platform it was compiled against and do some bitwise math- which isn’t a productive use of our time until we’re sure that it’s important to the application logic and not just boilerplate, which, spoilers, it is). What we should note, though, is that
ioctl is a variadic function, and as such will confuse the decompiler and create orphan variables in the disassembly view that may not even show up in the decompiler view. In this case, it loses sight of the fact that
0x10000F838 are global variables (each moved into
rdx before the call) being passed as the third arguments to
ioctl. If we cross-reference them, we see that they are set in another small function almost identical to this one except it passes a different
ioctl value… and sets the boolean flag we found earlier. It is reasonable at this point to assume it is obtaining and later pushing back at exit time some structs containing information about standard input and output, which could be used to reset the terminal state. (If we cheat and check the source code, this is exactly what it is doing, in function RestoreTtyState.)
After annotating the function with what we’ve learned, it is much easier to understand.
Press backspace until we’re back in the signal handler at
sub_100001620. We went on quite the little excursion just to determine that it’s pushing a stored terminal state. Picking up where we left off, we see that the handler takes the received signal number and reassigns it to a null handler pointer with
signal(var_24, 0x0). This will cause the next raising of that signal to go to the default handler instead of a custom one. We can see it store the return value in
var_8 and proceed to do exactly nothing with it. (This would probably disappear altogether on a high optimization setting.) Having changed the signal handler after calling the terminal reset function, it then resends the same signal to itself by calling
kill on the value of
getpid. The default handler will be raised with the propagated signal, which generally will result in process termination.
Isn’t that quite easy to understand? And we haven’t taken a single peek at any source code!
Alright, we will back up one more level back into
main. The next several lines are quite obvious: it’s just repeatedly setting up our signal handler for several different signals. Bla bla bla, skip down to the next chunk, which is more interesting:
It’s our well-known friend
getenv! The program checks for the
METAMAIL_TMPDIR variable and, if found, points the global variable
0x10000f278 to the returned buffer. Otherwise, it points it to the constant string
"/tmp". Next it checks
MM_HEADERS. It gets the
strlen of the buffer if it exists; note that in high optimization settings,
strcpy is often inlined and you will not see an explicit function call to it, just a few lines of byte-copying right in the middle of the surrounding code. It mallocs a new buffer into
var_160 that is the length of the
MM_HEADERS string plus fifteen (
0xf) extra bytes. Why? We don’t know yet.
If the malloc fails and returns zero, the program goes to
sub_100006130 passing the global pointer variable
0x10000f270. Follow the global pointer and it dumps you to another pointer; follow it again and you get the string
"Out of memory!". It is not hard to guess what this function will do.
It flips out and quits.
Backing up one – assuming the
malloc was successful, we are going to
sprintf something, and hence it becomes obvious that the extra fifteen bytes were to make room for the static string
"MM_HEADERS=" and the trailing null. You may have noticed we are actually calling a wrapper that will land you in dynamic library purgatory if you try to follow it. We are not calling the real true
sprintf; a theoretically more secure variation (
int __sprintf_chk(char * str, int flag, size_t strlen, const char * format);) has automatically been substituted by the compiler. The seemingly mangled call is actually intentional. However… it specifies a security flag of zero and a max string length of
MAXINT, so I am not sure how that is supposed to help anyone. Just ignore these flags as noise. That being said, since
sprintf is a variadic function, the variadic part (in this case, the buffer stored at
0x10000f930) fell off the end. If you are not intimately familiar with
printf format strings, it’s time to brush up- you need to understand them to correctly reconstruct calls and/or spot vulnerabilities in incorrect
We haven’t gotten very far into the program at all, but we’ve covered quite a few different C paradigms and how they appear in a disassembly. Hopefully binaries now appear less inscrutable and less magical, and you understand why reverse engineers laugh in the face of programmers who think no-one will never understand their awesome secret algorithm without the source code. Don’t count on “but it’s compiled” as a security feature!
With some practice, you can learn to read decompiled code very quickly and learn to spot boilerplate that can be skipped over. Stay tuned for a Part 2 coming very soon investigating more interesting parts of the metamail binary. Thanks for reading, and you can follow me on Twitter at @0xabad1dea.