CHAPTER

Compilers and Optimizers

For many higher-level programming languages, compilation is a vital part of

the process of converting an application from source code to machine-readable
binary code. During this process, a compiler may make minor changes to the
code to make it as fast and efficient as possible.
The process of compiling and optimizing an application can make it more
difficult to reverse engineer. This chapter describes how to find a starting point
for reversing an application and some of the common actions that compilers
take that can complicate reverse engineering.

Finding Starting Code

When code is compiled, the compiler introduces a large amount of boilerplate
that is executed before the actual application code is ever called. When reverse
engineering, one of the art forms you’ll need to master is how to skip over this
and focus on the target code, not the boilerplate setup. However, identifying
the entry point into the target code can be complex.
When reversing someone else’s code, it’s unlikely that the code will be com-
piled with debugging symbols. This means function and variable names and
other information that could provide a hint regarding the actual code’s entry
point have been stripped from the application. Figure 8.1 shows what opening
a file without debugging symbols looks like in gdb.

127
128 Chapter 8 ■ Compilers and Optimizers

Figure 8.1: Application without debugging symbols in gdb

This lack of debugging symbols creates a major challenge because applica-

tions written in higher languages rather than pure assembly include much more
overhead and compiler-generated symbols. The following is sample output from
 an info files command in gdb showing the number of different sections that
exist in a simple executable:
Entry point: 0x80483a0
0x08048154 -0x08048167 is .interp
0x08048168 -0x08048188 is .note.ABI-
tag
0x08048188 -0x080481ac is .note.gnu.build-
id
0x080481ac -0x080481cc is .gnu.hash
0x080481cc -0x0804823c is .dynsym
0x0804823c -0x080482a6 is .dynstr
0x080482a6 -0x080482b4 is .gnu.version
0x080482b4 -0x080482e4 is .gnu.version_r
0x080482e4 -0x080482ec is .rel.dyn
0x080482ec -0x08048314 is .rel.plt
0x08048314 -0x08048338 is .init
0x08048340 -0x080483a0 is .plt
0x080483a0 -0x08048648 is .text
0x08048648 -0x0804865d is .fini
0x08048660 -0x080486a9 is .rodata
0x080486ac -0x080486f0 is .eh_frame_hdr
0x080486f0 -0x080487f4 is .eh_frame
0x08049f08 -0x08049f0c is .init_array
0x08049f0c -0x08049f10 is .fini_array
0x08049f10 -0x08049f14 is .jcr
0x08049f14 -0x08049ffc is .dynamic
0x08049ffc -0x0804a000 is .got
0x0804a000 -0x0804a020 is .got.plt
0x0804a020 -0x0804a028 is .data
0x0804a028 -0x0804a02c is .bss
0xf7fdc114 -0xf7fdc138 is .note.gnu.build-
id in /lib/ld-linux.so.2
0xf7fdc138 -0xf7fdc1f4 is .hash in /lib/ld-linux.so.2
0xf7fdc1f4 -0xf7fdc2d4 is .gnu.hash in /lib/ld-linux.so.2
0xf7fdc2d4 -0xf7fdc494 is .dynsym in /lib/ld-linux.so.2
0xf7fdc494 -0xf7fdc612 is .dynstr in /lib/ld-linux.so.2
0xf7fdc612 -0xf7fdc64a is .gnu.version in /lib/ld-linux.so.2
0xf7fdc64c -0xf7fdc714 is .gnu.version_d in /lib/ld-linux.so.2
Chapter 8 ■ Compilers and Optimizers 129

0xf7fdc714 -0xf7fdc77c is .rel.dyn in /lib/ld-

linux.so.2
0xf7fdc77c -0xf7fdc7ac is .rel.plt in /lib/ld-
linux.so.2
 0xf7fdc7b0 -0xf7fdc820 is .plt in /lib/ld-
linux.so.2
0xf7fdc820 -0xf7ff4baf is .text in /lib/ld-
linux.so.2
0xf7ff4bc0 -0xf7ff8a60 is .rodata in /lib/ld-
linux.so.2
0xf7ff8a60 -0xf7ff90ec is .eh_frame_hdr in /lib/ld-
linux.so.2
0xf7ff90ec -0xf7ffb654 is .eh_frame in /lib/ld-
linux.so.2
0xf7ffccc0 -0xf7ffcf3c is .data.rel.ro in /lib/ld-
linux.so.2
0xf7ffcf3c -0xf7ffcff4 is .dynamic in /lib/ld-
linux.so.2

This list can get even longer in more complex binaries, with numerous depen-
dencies and libraries. Looking at this output, you know that the .text section
of the executable is located at address 0x080483a0. Disassembling the code at
this location can provide a hint to the entry point of the target code. Figure 8.2
shows the result of disassembling the code at this location in gdb.

Figure 8.2: .text disassembly in gdb

When searching for the entry point to the target code, this can depend on the
exact compiler and language used to build. You’ll see an example for finding
starting code in a C/C++ application, as that’s still one of the most common
languages used today. To begin with, look for a call to __libc_start_main.
The address of the target code will be passed as a parameter to this function,
and given what you know of calling conventions, you know that means we’re
looking for what’s put on the stack before the call.
In Figure 8.2, the address 0x804848c is pushed onto the stack right before the
call to __libc_start_main, making it a parameter to the function. Therefore,
the target code begins at that address. Figure 8.3 shows a disassembly of the
main function, including calls to libc.
130 Chapter 8 ■ Compilers and Optimizers
 Figure 8.3: Main function disassembly in gdb

Compilers
Compilers take code and translate it to machine code that the processor can
read. There are various things that compilers can do to affect reverse engi-
neering, both intentionally and unintentionally. This section focuses on unin-
tentional changes; intentional techniques such as obfuscation will be covered
in Chapter 12, “Defense.”

Optimization
Compilers can be configured to optimize code based on various metrics, including
speed and disk size, or not optimized at all. The code can look very different
based on whether optimizations are applied.
Consider the following code sample. This code implements a simple if state-
ment with two conditions.
int main(int argc, char* argv[])
{
if (argc >= 3 && argc <= 8)
{
printf("valid number of args\n");
}
}

Figure 8.4 shows what the code looks like in a disassembler (more on this in
Chapter 11, “Patching and Advanced Tooling,” don’t worry) when compiled
with no optimizations. Note that the checks for the two conditions comparing
the values to 2 and 8 are clearly visible in the code.
Chapter 8 ■ Compilers and Optimizers 131

Figure 8.4: Unoptimized code in a disassembler

Figure 8.5 shows the same code when optimized for speed and space.
The comparisons with the values 2 and 8 are no longer visible in the code,
and the code no longer looks like an if statement with two conditions.
Figure 8.6 shows the code optimized solely based on disk space. Again, the
two comparisons are missing.
If you examine the code, you’ll see that the code checks if (argc- 3) > 5. If
argc < 3, then subtracting 3 will cause an underflow and cause the value in
eax to be a large positive number. If argc > 8, then argc- 3 > 5. In both of these
cases, the result will be greater than 5, so the optimized statement is equivalent
to the original test. Compiler optimizations result in equivalent logic, but they
can make code much more difficult to read and reason about.
Most compilers have options for setting the level of optimization. While you’re
learning, if you’re having difficulty reversing an application you’ve written, try
disabling optimizations when compiling. On the flip side, if you want to make
your code more difficult to reverse engineer, compiler optimizations are an easy
and beneficial way to do so.
132 Chapter 8 ■ Compilers and Optimizers

Figure 8.5: Speed and space-optimized code in a disassembler

Stripping
Stripping a binary means removing all information that is not necessary for the
code to execute, including the symbol table. An unstripped binary retains its
symbol table, while a stripped one does not.
Symbols can be extremely useful for debugging an application. For example,
consider the following code:
// Declare an external function
extern double bar(double x);

// Define a public function

double foo(int count)
{
double sum = 0.0;

// Sum all the values bar(1) to bar(count)

for (int i = 1; i <= count; i++)
Chapter 8 ■ Compilers and Optimizers 133

sum += bar((double) i);

return sum;
}

Figure 8.6: Space-optimized code in a disassembler

 If this code is parsed by a compiler, it will at least contain the symbol table
entries shown in Figure 8.7. Symbols are so useful in debugging that Microsoft
allows you to download symbols for their applications in case you need to trou-
bleshoot! This additional information can be invaluable for understanding the
intent behind an application.

Figure 8.7: Application debugging symbols

134 Chapter 8 ■ Compilers and Optimizers

If a file is stripped, it will show that no debugging symbols are found when
opened in gdb, as shown in Figure 8.1. These files are much more difficult to
reverse engineer.
Symbols can be stripped from an application in a few different ways. One
option is to use compiler flags, such as gcc –fno-rtti –s. Another option is to
use post-build stripping tools, such as strip in Linux.
Symbols make it easier for an attacker to reverse engineer an application
because they can help with locating areas of interest and understanding the
intent behind certain variables. However, there are legitimate reasons to leave an
application unstripped. For example, symbols help with creating crash reports
and error logs and support legitimate debugging to fix client errors. While
learning, if you are writing your own code and compiling it to practice with, start
by making sure you’re building with symbols left in. As you progress in your
skills, then remove symbols. When reverse engineering someone else’s code, it’s
highly unlikely you will find symbols have been left in it, but it does happen!

Linking
Applications are rarely written in isolation anymore. What’s more common is
to include libraries that provide core pieces of capabilities (such as communica-
tions, logging, drawing, etc.). When compiling an application that uses libraries,
there are two options for how those get built. These libraries can be statically or
dynamically linked into the application. Each has its benefits and drawbacks
from a software cracking perspective.

Static Linking
With static linking, libraries are built into the application itself. This improves
the speed of execution because the target addresses of any calls to the library
are built into it at compile time. Also, statically linked applications are more
portable because they have fewer dependencies on the environment.
 However, static linking also has its downsides. Statically linked applications
are larger because the entire library is built into the executable, even if you use
only one function from a large library. Additionally, any updates to the library
require recompilation of the applications using them.
The file bloat caused by static linking can be significant for programs. For
example, as shown in Figure 8.8, even a simple one-line “hello world” program
will link dozens of libraries.

Dynamic Linking
Dynamic linking is the other option and the default choice for many compilers.
With dynamic linking, the required libraries are located on the system at runtime.
Chapter 8 ■ Compilers and Optimizers 135

If a library is not already loaded into system memory, the library must be found
on the system and loaded into the shared library memory; however, common
libraries are likely already loaded and available for use.

Figure 8.8: Linked libraries in “hello world” program

Dynamic linking reduces application size and eliminates the need to recom-
pile an application after a library update if the update is backward compatible.
Additionally, dynamically linked applications can be faster at load time if the
libraries that they use are already loaded into memory.
However, dynamically linked applications depend on the libraries that they
need being installed on the system and can be slower than statically linked ones
(if dependencies are not already loaded and need to be located and loaded). In
addition to the need to load any libraries not already in memory, dynamically
linked applications need to find the address of called functions at runtime. This
involves searching the shared memory space for the library and may require a
great deal of memory paging.

Security Impacts of Linking

The choice of whether to use static or dynamic linking depends on the devel-
oper or the compiler. But putting your software cracking hats on, both options
have their security implications.
Reverse engineers typically prefer that an application be statically linked. Static
linking makes it easier to determine the exact load address of shared library
functions, which is useful when crafting exploits. It means you can leverage
code in the shared libraries to perform your exploitation, and that code will be
at a predictable location inside of your binary at runtime. Leveraging a library
that is linked dynamically is possible, but it is much more difficult because of the
136 Chapter 8 ■ Compilers and Optimizers

need to search for the desired library in the shared library memory and locate
its address every single time, as it will move and be unpredictable.
Crackers, on the other hand, tend to prefer dynamically linked libraries.
Dynamic linking results in much less code to sift through, and crackers are
interested solely in an application’s custom code, not the shared library code.

Summary
The process of compiling and optimizing an application can make it much more
difficult to reverse engineer even if the compiler isn’t intentionally obfuscating
it. However, like any anti-reversing protection, this can only slow down the
process since no software is uncrackable.
CHAPTER

9
Reverse Engineering: Tools and
Strategies

Up until this point, the focus of this book has been on understanding how the
guts of computers work. This is essential to being an effective software cracker.
Now that you have the foundation, the focus shifts to the art of software
cracking. To experiment and practice cracking, you’ll work with a variety of
targets:
■■ Real software: Software taken from the real world. When analyzing real
software, you must take into account copyright law to ensure no copyright
violations.
■■ Manufactured examples: Applications written for this book to illustrate
specific concepts.
■■ crackmes: Small crackable programs written by other software crackers
to demonstrate an idea and challenge others.
crackmes like those used in this course are manufactured examples that pro-
vide a few benefits to an aspiring cracker. In general, they are designed to be
solvable, legal to crack, and safe to run in a debugger.
crackmes are also often labeled based on their focus, level of expertise, etc.
As a result, you can specifically seek out challenge problems suited to your
interests and skill level (i.e., advanced C cracker versus beginner Java cracker).

137
138 Chapter 9 ■ Reverse Engineering: Tools and Strategies

Lab: RE Bingo
This lab provides hands-on experience in reversing code that has been built
(and obfuscated) by a compiler.
Labs and all associated instructions can be found in their corresponding
folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES
For this lab, please locate Lab RE Bingo and follow the provided instructions.

Skills
This lab uses objdump to practice identifying control flow constructs and com-
piler settings when reversing. Some of the key skills being tested include the
following:
■■ Reverse engineering x86
■■ Control flow constructs
■■ Impact of compiler settings

Takeaways
Quickly identifying control flow constructs can massively speed up reverse
engineering. They provide insights into the logic of an application and make it
more readable and comprehensible.
However, compiler configuration has a significant impact on the speed of
reversing. For example, stripping and optimizing, in general, slow things down.
In larger and more complex programs, automating some reverse engineering
is often necessary. It is common to write custom tools for a specific target.
Unpacking, deobfuscating, and circumventing anti-debug checks are common
tasks for automation.

Basic REconnaissance
As a software cracker, these are the most common situation that you’ll face:
■■ You want to crack a program.
■■ You have no source code.
 ■■ You have an executable.
In this situation, you need a means of quickly assessing the target executable
and finding a starting point for your analysis. Some of the most commonly used
Chapter 9 ■ Reverse Engineering: Tools and Strategies 139

initial tools for reverse engineers are objdump, strace, ltrace, and strings.
You’ll see more advanced tools as you progress through the book, but as these
are some of the most foundational, they’re a good starting point.

objdump
Object Dump (objdump) is a Linux-based tool for dumping the disassembly of
any program. As shown in Figure 9.1, it has numerous options. The most impor-
tant ones for quick reverse engineering include the following:
■■ -d: Instructs objdump to disassemble the content of all sections
■■ -Mintel: Tells objdump to display assembly in Intel syntax (as opposed
to AT&T)
For example, to disassemble an application named appname, use the command
objdump –d –Mintel appname.

Figure 9.1: objump options

Figure 9.2 shows the output from running objdump on a sample application.
Note that objdump will display memory locations, function names, x86 machine
code, and x86 assembly.

Figure 9.2: Sample objdump output

140 Chapter 9 ■ Reverse Engineering: Tools and Strategies

strace and ltrace

strace and ltrace provide the ability to monitor library (ltrace) and system
(strace) calls. They make it possible to trace through a program and get a sense
of what other programs are doing.
If any program in any language wants to do anything useful, it will have to
make system calls. Characterization of what libraries and external functionality
 it’s using can be immensely useful when doing reconnaissance on a system.
You’ll notice that, with these tools, not only can you see what it’s using, but
you can also see who is using it (i.e., what address in the application called).
So, it can also help you to focus on useful functions. For example, you might
see which piece of code calls into cryptography libraries a lot; that’s probably
interesting from a cracking perspective.

ltrace
ltrace (library trace) is a Linux command-line utility that traces library calls.
Library calls are calls by your application into dynamically linked libraries. The
syntax of the command is ltrace <command>.
For example, if you #include <stdio.h>, that library gets dynamically linked
when your program loads. When you call printf or fopen, that is calling into
the standard C library. This construct holds true for all programming languages,
which all include a notion of including external libraries.

strace
strace (system trace) is a Linux command-line utility that traces system calls.
The syntax of the command is strace <command>.
System calls are calls by your application into the operating system, which
manages things like files and your console window. Functions like fopen and
printf eventually, in their inner workings, must make calls into the operating
system. Just like with ltrace, this holds true for all programming languages;
it’s rare for an application to exist that doesn’t utilize OS-level functionality.

strace Example: echo

Monitoring system calls provides a crude way to trace through a program.
Suppose you wrote the echo utility and wanted to watch how it was running.
echo is a Linux command that echoes the input to the output. For example,
the command echo hello! prints "hello!" to the terminal.
But what is it actually doing? Running strace echo hello! will produce
output similar to Figure 9.3.
Chapter 9 ■ Reverse Engineering: Tools and Strategies 141

Figure 9.3: strace output for echo hello!

This image is complex and can be a lot to decipher. Looking through the
result, you can see some standard system calls at the beginning that are used
to get the echo program up and running.
The following lines are the interesting output, which are found at the very end:
write(1, "hello!\n", 7hello!
 ) =7
close(1) =0

This says that echo wrote a string to stream 1, which, remember, is stdout.
The write command had a return value of 7 because seven characters were
written. Finally, echo closed stream 1, which returned 0 for success. While this
seems simple, imagine using this to track where an application wrote a piece
of configuration data. Say you change a setting and want to see how it stores
that on the system.

strace Example: Malicious Kittens

Comet Cursor was an early example of spyware on the Windows OS. It allowed
users to change the appearance of the mouse cursor and websites to use cus-
tomized cursors. The application installed itself without user permission and
secretly tracked users.
142 Chapter 9 ■ Reverse Engineering: Tools and Strategies

As shown in Figure 9.4, numerous kitten cursor applications exist in the

wild. This example uses an example cursor application that secretly calls out
to a Russian IP address.
Running the code shows no signs of the malicious functionality, as shown here:
deltaop@deltaleph-ubuntu:~$ ./kittens
Registering kitten cursor!
Done! Enjoy the kitties!
deltaop@deltaleph-ubuntu:~$

Figure 9.4: Kitten cursor applications

However, analyzing the code in strace tells a different story, as shown here:
deltaop@deltaleph-
ubuntu:~$ strace ./kittens
...
poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
send(3, "!$\1\0\0\1\0\0\0\0\0\0\7kremlin\2ru\0\0\34\0\1",
28, MSG_NOSIGNAL) = 28
poll([{fd=3, events=POLLIN}], 1, 5000) = 1 ([{fd=3, revents=POLLIN}])
ioctl(3, FIONREAD, [28]) =0
recvfrom(3, "!$\201\200\0\1\0\0\0\0\0\0\7kremlin\2ru\0\0\34\0\1", 1024,
 0, {sa_family=AF_INET, sin_port=htons(53),sin_addr=inet_
addr("192.168.1.1")}, [16]) = 28
close(3) =0
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("192.168.1.1")}, 16) = 0
...

This sample output from strace shows multiple events. To focus on events
of interest, use grep (which limits results to lines that match your search string,
in this case connect).
Chapter 9 ■ Reverse Engineering: Tools and Strategies 143

deltaop@deltaleph-
ubuntu:~$ strace -
f ./kittens 2>&1 | grep connect

connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"},

110) = -
1 ENOENT
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"},
110) = -
1 ENOENT
connect(3, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("192.168.1.1")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("192.168.1.1")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("192.168.1.1")}, 16) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("195.208.24.91")}, 16) = 0
write(2, "connected.\n", 11) = 11

The previous sample output looks for events with the word connect in them.
This includes multiple Internet connections, including one to 195.208.24.91,
which is suspicious as it’s an external IP address, and why would your cursor
need to do that?

strings
strings is a Linux utility designed to extract the printable strings used by an
application. It looks for a series of ASCII printable characters with a (configu-
rable) minimum length and prints any that it finds.
strings can be very useful in reverse engineering because it provides a
high-level understanding of the sorts of things that a program might do. Also,
once you find strings of interest, you’ll see later how you can use those strings
to easily locate the associated piece of code. For example, a string that says
"incorrect password" can be used to quickly trace where the password handling
code is. For example, the following strings provide valuable hints about an
application:
■■ "Enter password:"
■■ "open_socket"
■■ "YOUR FILES HAVE BEEN ENCRYPTED!"

The syntax of the command is strings program. While it is commonly used with
no options, the following flags are sometimes useful when reverse engineering:
■■ -a: Show all strings in the file, as opposed to only those in the loaded sec-
tions of object files. This is often useful when dealing with obfuscated,
nested, or otherwise unusual binaries.
■■ -n: Specify the minimum length of successive printable characters for a
sequence of bytes to be considered a string. The default is 4. It is often
useful to expand or limit the number of strings found by the tool.
144 Chapter 9 ■ Reverse Engineering: Tools and Strategies

Dependency Walker
Dependency walking is a technique used to quickly understand the imports
and exports of an application. Dependency Walker is one example of such a
tool. (See the “Tools” section of our repository for links.)
Dependency walking provides a valuable, high-level view into what actions
a program will perform and is often a useful first step in cracking. Most appli-
cations don’t implement all their own functions; they will use functions from
the operating system, or external libraries. Each time an application reaches
outside of its code, that will show up as an imported function. Also, often
applications will share functionality with other applications, and anytime a
function is something “available to be shared,” it will show up as an export of
the application.
Loading a program into a program like Dependency Walker shows the DLLs
that it uses and the API calls it is expected to make. Figure 9.5 shows that the
program will create several registry keys.

Figure 9.5: Examining registry modifications in Dependency Walker

 Reverse Engineering Strategy
Reverse engineering is still more of an art than a science. While great tools and
techniques are available to help, effective reversing ultimately relies heavily on
intuition and experience.
As such, it is not possible to give a prescriptive solution. However, there are
general approaches and best practices that can help.
Chapter 9 ■ Reverse Engineering: Tools and Strategies 145

Find Areas of Interest

Applications contain large volumes of code, and most of it is irrelevant or
unnecessary to reverse engineer. An important first step when reversing an
application is finding the area of the program you are after.
You’ll continue to learn lots of interesting techniques for narrowing this
down, but a few are now available to you based on just knowledge of the
previous tools:
■■ Interesting strings: Look for program strings that you are interested in
(e.g., “Incorrect Key”) and find where those strings are used (e.g., identify
calls to printf using those strings).
■■ User input: Look for where input from the user is received (e.g., scanf,
dialog boxes, etc.) and find where that input is processed.
■■ System input: Look for where input is read in from the system, such as
configuration files and registry settings.
■■ Authentication code: If possible, use a debugger to pause the program
after inputting the username/key. Then, scan the memory for the entered
values, set HW breakpoints on those locations, and rerun the application
to find where the values are read or written.

Iteratively Annotate Code

Even after you identify the code of interest, it may be difficult to understand.
One approach to understand complex code is to perform multiple passes, add-
ing information (such as comments) during each pass.
To do so, annotate the target area until you understand how it works using
the following process:
■■ Identify and mark local variables: Use calling convention rules to iden-
tify locals (e.g., [ebp-4] in cdecl). These can be labeled using something
vague at first (e.g., local1).
■■ Identify and mark function parameters: Use calling convention rules to
identify parameters (e.g., [ebp+12] in cdecl). These can also be labeled
using something vague at first (e.g., arg1).
■■ Identify API calls (e.g., atoi): Use knowledge of API parameters to further
annotate local variables. For example, API documentation indicates atoi
 is passed a string that will be converted to an integer so can rename our
parameter integer_string.
■■ Add comments to complex control flows: For example, “this code factors
the number.”
146 Chapter 9 ■ Reverse Engineering: Tools and Strategies

■■ Refine descriptions based on observed data flows: For example, local1

may become loop_counter if you see it used as the counter in a for loop.
A big part of effective reverse engineering is moving quickly. Even small
programs have too much code to analyze everything.
The vast majority of an application’s code will have no relevance to what you
are after. Knowing what to focus on is often less important than knowing what
not to focus on. Learning where to make leaps takes time.

Summary
This chapter introduced some of the core tools and techniques that you will use
as a software reverse engineer and cracker. Before moving on, take some time to
practice and get some hands-on experience using the tools. This practice time
will be invaluable later when you move on to more complex software and more
advanced RE and cracking techniques.
CHAPTER

10

Cracking: Tools and Strategies

Cracking is the art of reversing software to bypass protections or other

undesirable functionality. This chapter explores some of the key tools and
strategies used for software cracking, including the use of key generators and
patching to defeat key checkers.

Key Checkers
One of the most common practices for licensing software is through license keys.
In a goal to defeat piracy, every installation of the software requires a unique key
to complete the installation. In the case of software with multiple tiers of fea-
tures, they may have some features always freely available, while others reside
behind a license wall, or the software may not work at all without a license key.
 License keys are a common anti-piracy solution, and they have their advan-
tages. These are two of the most significant:
■■ License keys are easy to generate and verify.
■■ The ratio of valid to invalid keys is so small that random guessing is
unlikely to generate a valid key (assuming a reasonable key length).

147
148 Chapter 10 ■ Cracking: Tools and Strategies

However, like all security, if they are implemented poorly, they can be highly
susceptible to cracking, and like all security, they are not entirely infallible.
A sufficiently knowledgeable and motivated cracker could eventually defeat
or bypass them. However, they’re still one of the stronger forms of protection;
this is just a reminder that there is no such thing as 100 percent secure software.
Back in the day when offline systems were more common, license checking
and validation were often done entirely offline, meaning all of the logic to verify
the key was resident on the system. Now, with prolific connectivity, we often see
license key checks that consist of both an offline and online component, where
they reach out to a license server for additional verification. There are a few
different ways to implement key checks with varying levels of effectiveness.

The Bad Way

In the past, very popular computer games StarCraft and Half-life both used a
checksum as a license key. Recall checksums are often very simple mathematical
expressions performed on a binary blob, some as simple as adding all the num-
bers together. In the checksum used by these games, the 13th digit verified the
first 12.
This meant that a user could enter anything that they wanted for the first 12
digits and then calculate a 13th to create a valid checksum. This lapse in security
led to the infamous 1234-56789-1234 key, which was valid for these games and
used widely to pirate them.
One of the biggest problems in these cases was that the algorithm used to
calculate the checksum was too simple.
x = 3;
for(int i = 0; i < 12; i++)
{
x += (2 * x) ^ digit[i];
}
lastDigit = x % 10;
 There are two ways to approach cracking this. One is that you run the algorithm
and calculate the valid value of the last digit as shown previously.
The other is a brute-force attack. Given it was only one digit you had to figure
out, there are only 10 options for the last digit [0-9]. You can randomly select a
set of 12 digits and then just guess and check the 10 options for the last one until
you find success. The infamous 1234-56789-1234 key was so famous because it
was easy to remember, but by following either of these two approaches (calcu-
lation or brute force), you could generate any number of new keys.
Chapter 10 ■ Cracking: Tools and Strategies 149

A Reasonable Way
A brute-force attack against a license key is guaranteed to work. . .eventually.
The best that a license key can do is waste enough of a cracker’s time that it
becomes infeasible or impossible to carry out a brute-force attack.
So, how to protect against brute-force attacks? One common option in other
contexts is a cryptographic hash. For example, a license key could be imple-
mented using one of the following options:
■■ Username: SHA(username)
■■ Random value: WXYZ-SHA(WXYZ)
The use of a hash function makes a brute-force attack against this much harder.
However, it’s trivially easy for a cracker to determine how the algorithm works
after a look at the code. Depending on your mindset, if you’re an attacker, this
means leveraging the reverse engineering skills you’ve learned to this point to
find the algorithm and unravel it, and if you’re a defender, it means this is a
key piece of code that you need to protect.
An alternative is to use a custom, complex hash rather than a standard one.
While this is normally a horrible idea in security, it’s not an unheard-of choice for
this application. The goal isn’t to provide absolute protection, just to slow down
reverse engineering. For anyone in the security space whose toes are curling at
the suggestion of making your own hash, just note that this suggestion comes
with the caveat that you are able to make a decently good one. As a defender,
keep in mind there are lots of tools out there to do common hashing techniques,
so those will be all the first things an attacker will try to unroll your key.
Also, find ways to add unique complexity so a key can be used only in a
unique setting, and not proliferated. Schemes such as concatenating the product
name and version and computer name within the hashed value adds a solid
level of complexity. This way, a cracked valid key for one installation doesn’t
unlock other releases.

A Better Way
Hashes are better, and, if implemented correctly, they can be decent. But there
are even better options. A great example of this is the approach Microsoft uses
when generating license keys for its software.
Instead of hash algorithms, Windows uses public key cryptography. With
public key cryptography, a digital signature can be generated using a private
key and verified using a public one. This means that a digitally signed license
key can be verified by an application without exposing sensitive keys.
150 Chapter 10 ■ Cracking: Tools and Strategies

When generating its license keys, Windows uses a lot of information about
the software, including but not limited to:
■■ Bitness (32, 64)
■■ Type (home, professional, enterprise)
■■ Product ID
■■ Hardware features
Including all of this information helps to lock a product key to a specific
installation of the software. If you’re interested in more information on the pro-
tocol, there are lots of resources online tearing into Microsoft’s key generation.

Digitally Signed Keys

Digital signatures on license keys, like those used by Windows, make it much
more difficult to generate fake, valid keys. A valid signature must be generated
using the private key but can be validated with a nonsensitive public key.
Digital signatures prevent the straightforward generation of license keys and
present attackers with two options. The first is to leak a legitimate key, which
could be traced back to a specific user. Alternatively, an attacker can modify
the program to remove the key-checking code, which increases the time and
complexity of pirating the software.

The Best Way

The examples presented so far have focused predominately on offline verifica-
tion of license keys, meaning the entirety of the code to verify and unlock the
software is resident on the system. However, given the prolific connectedness of
systems these days, a way to add more strength is to add an online component.
This can take many forms, but one you see today is each piece of software can
be distributed with a license key in the form of a large random number distrib-
uted alongside the software. When the product is installed and registered, this
value is sent to the license server, which verifies that it is valid and has not been
used already. For digital software distribution these days, the key you’re sent
isn’t even valid until after you buy the software, meaning that if you had guessed
that key 10 minutes before you bought the software, it wouldn’t have worked.
Or you can use hybrid approaches where much of the algorithm to verify
 through hashing or public key cryptography is resident on the system, but then
there is also a step where the license server is checked to see if that key has been
used before or if the key has been revoked.

Other Suggestions
The methods introduced align with more of the industry best practices and the
most commonly used methods. But there is not a one size fits all to security,
Chapter 10 ■ Cracking: Tools and Strategies 151

and some of the following are techniques you could encounter in a cracking
scenario, or you might find them useful in a defensive scenario if you have
unique constraints.

Prefer Offline Activation

While the addition of online key servers sounds powerful from a security per-
spective, and it is, it’s worth acknowledging that technique comes with a huge
amount of manageability and infrastructure pain. Managing key servers is no
small feat, and they become a beacon for cyberattacks. So, you will still often
find that many companies aren’t able or willing to bite off that level of chaos,
so they will still favor going for stronger offline verification. Supporting offline
key verification eliminates the complexities of managing a key server and is
inclusive to users without Internet access.

Perform Partial Key Verification

In an offline mode, you have no method to perform revocation and have no
way to make some keys no longer work. To prevent a single leaked key from
working on all future versions of your software, check only some of the key.
A simplistic example would be to check only the first character of each group
in a license key such as the X, 9, B, and B in X4Z-951-B41-BR0.
If someone releases a key generator for your application, release a new ver-
sion that checks part of the remaining key. For example, switch to checking the
second character of each group (4, 5, 4, and R). This limits the potential damage
caused by a single key generator.

Encode Useful Data in the Key

Encoding useful data in the key can help to limit its applicability. For example,
a key may specify the maximum version of the application that it applies to,
limiting the impact of a compromised key.

Key Generators
If a piece of software uses a key for activation, crackers will want to build a
key generator for it. This is true regardless of which type of key activation you
did. Key generators are then distributed for people to generate a “free” key for
software.
You’ll see later how to patch software to simply remove a key check, so for
now focus on making a key generator, and assume you can’t just bypass the key
check. Key generators typically require a more in-depth analysis of the program
and a deeper understanding of the key algorithm.
152 Chapter 10 ■ Cracking: Tools and Strategies

Why Build Key Generators?

If key generators are more difficult to create, why bother building them? There
are a few different reasons.
Software can have various defenses that can make patching the more difficult
route, such as the following:
■■ Tamper proofing
■■ Dynamic checks
■■ Anti-debugging
■■ Software guards
Also, patching may require releasing a modified copy of the target software,
which may be watermarked. Watermarking is a technique to trace a piece of
software back to who originally purchased it. These watermarks can be used
to trace a cracked piece of software back to the cracker, which is obviously
something they don’t want.
The software could implement online checks to look for patched/modified
versions of programs. Alternatively, some software may decrypt itself based on
the entered key (unpacking, which will be explored in Chapter 13, “Advanced
Defensive Techniques”), and removing the key check entirely means it won’t
be able to decrypt.
Key generators are also more future-proof than patching. An application
developer can’t easily revoke valid keys.
Finally, crackers may choose key generators because they are harder. Patch-
ing in some cases is easy, while building a successful keygen is a challenge that
carries a certain amount of prestige.

The Philosophy of Key Generation

When cracking key checkers, it is useful to think of the key checker in the form
of f(u) == g(k), where:
■■ u is the username entered by the user.
■■ f is a transformation function on the username.
■■ k is the key entered by the user.
■■ g is a transformation function on the key.
 In this model, the key check is a validation that f(u) == g(k). In non-math-
speak this means that some transformation/mutation is done on the username
and then compared to some type of transformation done on the key. In this
example (and the following examples), the username is the input, but keep in
mind this can be any combination of things; they could use the version number,
Chapter 10 ■ Cracking: Tools and Strategies 153

computer name, etc. But the idea is something is going into a transformation to
come up with a result. And that result is compared to the input key, which has
also gone through some type of transformation (note this transformation could
be nothing, meaning the result is simply the key, or it could be more hashing
or mutation). With this model in mind, there are a few potential variants of
key checks.
Going back to the initial StarCraft/Half Life example, u would actually be the
first 12 digits of the key, and k is the last digit. In this setup, there is no username
entered; rather, part of the key is used to check the other part.
Another option is that u, and therefore f(u), is a constant (i.e., hard-coded
keys). In this setup, there is no username entered; rather, the key is transformed
and checked against a fixed value. For example, “the sum of all of the digits in
the key is equal to 1337.”

Cracking Different Types of Key Checks

By reasoning about key checkers in the formula f(u) == g(k), you can start to
build techniques for cracking different permutations.

Key Check Type I: Transform Just the Username

For this case, the username is transformed using some function, and that is then
compared to the key that was entered. So, in this case you can consider g()
causes no mutation to the key. This allows us to simplify our key check to just
f(u) == k. In this setup, the program transforms the username and validates
that the transformed value matches the key entered by the user.
To crack this type, locate and extract the transformation function f into a
key generation application. For example, multiply ordinals of characters in
username together and match against the key. The key generator will prompt
the user for the username they desire to use and then perform f(u)and print
out the valid key.

Key Check Type II: Transform Both

For Type II, you still have a transformation of the username, but, in addition,
g performs a mutation. The very mathematical way to look at this is that g has
an inverse. That is, g-1 exists, and g-1(g(k)) == k). The simple way to think
about this is that g will perform a mutation, and every mutation has a way of
unmutating it (i.e., do the exact opposite).
In this setup, the program transforms the username, transforms the entered
key, and validates that the two produce the same results. However, the function
g can be inverted (“reversed” or “backed out”).
154 Chapter 10 ■ Cracking: Tools and Strategies

To crack this type of key check, reverse engineer g and derive g-1. Often, this
.

is as simple as “undoing” each transformation on g in reverse order. Then, gen-

erate a key with g-1(f(u)).
For example, assume g(k) = k * 2 + 1000. If so, g-1(h) = (h – 1000) / 2.
In this case, the key generator would prompt for the desired username (as
with Type I) and perform the f(u), but then the result now is the mutated key, so
you have to do your unrolling with g-1(h). That final result is then the valid key.

Key Check Type III: Brute Forceable

In Type III, a collision on f(u) can be brute forced through g(k). This is a viable
approach if the key space is very small or you have a lot of computing power.
In this setup, the program transforms the username, transforms the entered
key, and validates that the two produce the same results (same as Type II). But
you instead are looking for a solution to f(u) == g(k) by repeatedly testing
random or pseudorandom ks.
To crack this type, determine the format of k. Then, extract g into a self-
contained key generator. Finally, generate random k s until a solution to
f(u) == g(k) is found.
For example, consider the case where g(k) = CRC32(k). If the key mutation
is using something so small such as the CRC32 algorithm, then brute force
becomes pretty trivial on a standard computer. Since CRC32 has such a small
range of possible values, it’s possible to brute force.

Defending Against Keygens

Key checks may be a combination of these types. For example, the key trans-
formation g may be both brute forcible and invertible.
Key checks generally must fall into one of these categories. Otherwise, there
would be no way to generate keys in the first place.
Key Check Type I is the weakest. The cracker needs only to extract the algorithm
from the key checker, with no need to actually RE the algorithm.
Key Check Type III is better. It requires the attacker to extract both algorithms
 and identify a way to brute force the key transformation, which is not always
obvious.
Key Check Type II is likely best but also the hardest to design well. Cracking
this requires the attacker to derive the inverse of the key transformation function.
This may necessitate a deep analysis of the transformation algorithm, slowing
the attack.
As always, there is no silver bullet. Every key checker can be cracked even-
tually, and the best that a defender can do is slow down the attacker.
Chapter 10 ■ Cracking: Tools and Strategies 155

Lab: Introductory Keygen

This lab provides experience in creating a keygen for a simple program.
Labs and all associated instructions can be found in their corresponding
folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Introductory Keygen and follow the provided
instructions.

Skills
This lab practices the use of objdump and the strings utility to generate a
keygen. Some of the key skills it tests include the following:
■■ Initial reconnaissance
■■ Reverse engineering x86
■■ Key generation

Takeaways
In addition to modifying a program, it’s often possible to crack a program just by
observing how it works. The right approach is often determined by the program
constraints, and choosing which to use is an important skill.

Procmon
In reverse engineering, you want to learn as much about how the program
works as possible. Before jumping to super-fancy debugging, start easy by just
observing software’s behavior.
Procmon is a tool distributed as part of the Sysinternals suite of tools (avail-
able at http://technet.microsoft.com/en-us/sysinternals/bb842062). This
repository contains about 60 windows utilities made and freely distributed by
Microsoft. Note these tools work only on Windows OSs.
Example: Notepad.exe
Try taking a look at what notepad.exe does when you create a new file, change
the font, and then save some content. To do so, take the following steps:
1. Open Procmon.exe.
2. Launch Notepad.
156 Chapter 10 ■ Cracking: Tools and Strategies

3. Enter some text into the Notepad document.

4. Click the Format menu and then the Font menu item.
5. In the Font window, change the font to Webdings.
6. In the Font window, change the size to 20.
7. Click the OK button.
8. Save the Notepad document as Example1.txt.
9. Close Notepad.
Stop Process Monitor capture activity by clicking the Capture button,
as shown in Figure 10.1. The icon should now show an X over the magni-
fying glass. At this point, Process Monitor has captured all File, Registry, and
Process/Thread events.

Figure 10.1: Halting Process Monitor

Process Monitor captures thousands of events a second, which results in too

many records to review manually. It’s necessary to filter the results down to
events of interest. To do so, open the Filter menu by clicking the funnel icon,
as shown in Figure 10.2.

Figure 10.2: Filtering events in Procmon

To see only events related to the process Notepad.exe, define a filter stat-
ing that the Process Name is Notepad.exe, as shown in Figure 10.3. You can
accomplish this via the following steps:
1. Select Process Name from the Column list box.
2. Select is from the Relation list box.
 3. Type Notepad.exe in the Value text box.
4. Select Include from the Action list box.
5. Click the Add button.
6. Click Apply and OK.
Chapter 10 ■ Cracking: Tools and Strategies 157

Figure 10.3: Defining a filter in Procmon

Filtering based on the process name dramatically decreases the number of

events. However, it’s still not enough.
To find events of interest, you need to define additional filters. Procmon has
several categories of events that you can filter on, including the following:
■■ Registry
■■ File
■■ Network
■■ Process thread
To start, try focusing on the Registry values that Notepad modifies. Process
Monitor has a handy button for this, as shown in Figure 10.4.

Figure 10.4: Filtering on Registry events in Procmon

If Notepad has saved values to the Registry, it will create an event entry of type
'Operation' 'RegSetValue'. By right-clicking entries in Procmon’s log, you can
choose to include or exclude certain types of events, as shown in Figure 10.5.
This enables you to further refine your results and focus on events of interest.
Figure 10.6 shows a Procmon entry that seems to be related to the changes to
the font in Notepad. To see more information, right-click the entry and select
Properties.
158 Chapter 10 ■ Cracking: Tools and Strategies

Figure 10.5: Including and excluding event categories in Procmon

 Figure 10.6: Notepad font change registry event

Figure 10.7 shows the properties of the event. In the Data field, you can see
the text “Webdings,” indicating that this is an event triggered by changing the
Notepad font to Webdings.

How Procmon Aids RE and Cracking

Procmon made it possible to see the Registry changes made by Notepad. However,
this isn’t all that it can do. Further exploration of the tool reveals a great deal
of useful information.

Call Stacks
The Properties window for an event has a few different tabs. Clicking over to
the Stack tab shows the sequence of calls used to reach this point, as shown in
Figure 10.8.
Looking further down this stack trace, it’s possible to see the point where the
program left notepad.exe, as shown in Figure 10.9. This transition point from
application to libraries might be a good starting point for reversing.

File Operations
Procmon also records events for file operations, such as opening, closing, and
editing files. Figure 10.10 shows an example of this.
Chapter 10 ■ Cracking: Tools and Strategies 159

Figure 10.7: Event properties in Procmon

Figure 10.8: Stack view in Procmon’s Properties window

These file events can provide useful information for reversing. For example,
they can help with identifying and analyzing configuration files, export functions,
and proprietary file formats.
160 Chapter 10 ■ Cracking: Tools and Strategies
 Figure 10.9: Stack trace for notepad.exe

Figure 10.10: File operations in Procmon

Registry Queries
The Notepad.exe example showed how to find the Registry operation for chang-
ing the font in Notepad. However, this isn’t the only possible use for registry
queries.
For example, Figure 10.11 shows that Notepad looked for two keys with the
word “Security” in them but couldn’t find them. You could add these keys to
your Registry and place custom values in them to change how Notepad operates.

Figure 10.11: Security Registry queries in Procmon

Resource Hacker
Resource Hacker (also known as ResHacker or ResHack) is a free extraction
utility or resource compiler for Windows. Resource Hacker can be used to add,
modify, or replace most resources within Windows binaries including strings,
images, dialogs, menus, and VersionInfo and Manifest resources. (For tool links,
visit the tools section of our GitHub site at https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES.)
Chapter 10 ■ Cracking: Tools and Strategies 161

Resource Hacker can be a useful tool for exploring the structure of a binary
prior to the cracking process. It can be used to find and understand the structure
of nag screens, key entry screens, help menus, and more.
Resource Hacker can also be used to add functionality to a program before
or after cracking. For example, it’s possible to add new icons, menus, and skins
to an existing application.
To get started, open an .exe file in ResHack to explore its strings, images,
dialogs, menus, etc., as shown in Figure 10.12. Then, click an item in ResHack
(left) to show how that item would look in the application (right).
Figure 10.12: Sample application in Resource Hacker

Example
Suppose you see the window shown in Figure 10.13 in a program. As a cracker,
you want to understand how that window would be used by the program.

Figure 10.13: Password window

To find out, open the program in ResHack. Then, use Ctrl+F to search for one
of the strings used in the dialog box, as shown in Figure 10.14.
162 Chapter 10 ■ Cracking: Tools and Strategies

Figure 10.14: String search in Resource Hacker

Resource Hacker identifies this dialog box as the “GETPASSWORD2” dialog

box, as shown in Figure 10.15. Knowing this can help to guide the process of
reversing the program.

Figure 10.15: Identifying a dialog box in Resource Hacker

Mini-Lab: Windows Calculator

To try your hand at using Resource Hacker, try rebranding the Microsoft
Calculator. As shown in Figure 10.16, the Calculator window is titled Calculator.
Try changing this value to something else.
To start, open the calc.exe executable in Resource Hacker. Then, search for
the word Calculator, as shown in Figure 10.17.
Chapter 10 ■ Cracking: Tools and Strategies 163

Figure 10.16: Microsoft Calculator

Figure 10.17: Searching for Calculator in ResHack

The main Calculator window may not be the first result. Keep on searching until
you find the code defining the Calculator dialog box, as shown in Figure 10.18.
In Figure 10.18, the CAPTION string determines the title on the application
window. Change this string to rebrand the application as your own.
After changing the CAPTION, click the green arrow button shown in
Figure 10.19. This will compile the modified Calculator application.
After the application has been compiled, the updated version of the window
should be shown in the window preview. This should include the modified
caption, as shown in Figure 10.20.
Compiling the application doesn’t automatically save the modified version.
To do so, select File ➪ Save, as shown in Figure 10.21.
164 Chapter 10 ■ Cracking: Tools and Strategies

Figure 10.18: Calculator window in Resource Hacker

Figure 10.19: Compiling the modified application

Figure 10.20: Modified window in Resource Hacker

Chapter 10 ■ Cracking: Tools and Strategies 165

Figure 10.21: Saving the modified application in ResHack

At this point, you’ve successfully rebranded Windows Calculator. For more

of a challenge, try the following:
■■ Use Resource Hacker to resize the window to accommodate your new name.
■■ Modify the available buttons.
 ■■ Modify the calculator background.
■■ Open and edit other programs in your VM.

Patching
Patching involves modifying a compiled binary to modify code affecting its
execution. Depending on the situation, sometimes the easiest thing to do is
patch an application to circumvent its security.

Patching vs. Key-Genning

In some cases, advanced integrity checks or obfuscation might make patching
difficult. For example:
■■ Patching an encrypted/packed program on disk is not feasible.
■■ Patching around dynamic integrity checks (e.g., continuously validated
checksums) may be too cumbersome.
■■ The logistics of distributing a patched executable may not be desirable.
166 Chapter 10 ■ Cracking: Tools and Strategies

In these situations, you may choose to fall back on key generators instead.
Otherwise, patching a program to remove its key checks (or any other logic you
want to avoid) is often the easier approach, when possible.

Where to Patch
Patching can be done in two different places: in memory or on disk.
Patching in memory modifies the machine code in memory. This is use-
ful for reverse engineering attempts because you may need to try dozens (or
hundreds. . .or more. . .gasp) of things before one works. In-memory patching
affects only the current execution of the application. Each time you restart the
application, any in-memory patching will be lost.
Patching on disk modifies the machine code in the compiled binary. This
is useful once you know what works and affects all future executions of the
application. It makes the modifications persistent and will be there every time
the application is launched.

NOPs
Recall the instruction nop. It is a one-byte instruction (0x90) that does nothing.
When patching applications, it is critical to not move the code. In fact, mod-
ifying the size or simply deleting code will crash the application. To remove
sections of code yet maintain the same size, fill the space with nops.
For those of you who are curious why simply deleting code doesn’t work,
 there are many factors to this, but the most important is that some x86 code is
relative and some of it is absolute references. Looking at the relative case first:
this means some code translates to relative things like “jump forward 40 bytes
from where I am now.” In cases like this, if you remove code between the jump
and its destination 40 bytes away, you’ve messed up the jump. It will continue
to jump 40 bytes ahead, except that now it may land in the middle of an opcode
or skip critical instructions, which then results in a crash. If the code you remove
is outside of that 40-byte bubble and the jump forward 40 bytes still lands in
the same spot, then it would have no effect.
Now, consider absolute references. These types of references would look like
“use the data value at address 0x1234567.” If you remove code anywhere in
the binary before that address, you’ve caused everything to shift. So, when any
absolute reference goes to grab its values or perform an absolute jump, all of the
locations will be wrong, even if all you did was remove 1 byte from the binary.
This means relative references are affected only by adding/removing bytes if
they occur in between where the reference is made and the destination. How-
ever, all absolute references are destroyed if you shift the application even by
Chapter 10 ■ Cracking: Tools and Strategies 167

1 byte. This is why it’s critical in patching to maintain the size (unless of course
causing everything to crash is your goal, in which case smash away!).
Circling back to nop, if you want to remove a piece of code, such as causing
software to skip a key checker, instead of deleting the code, you simply replace
it all with nops. This maintains the application’s byte alignment but causes
nothing to happen when it reaches the undesirable code.

Other Debuggers
For reverse engineering with dynamic analysis on Windows, there are numerous
popular choices. Here are a few:
■■ OllyDbg
■■ Immunity
■■ x64dbg
■■ WinDbg
Which of these to use depends on the situation and user preference. All of
them have similar features, and skills in one typically translate to the others
as well. You’ll dip your toes into a few different pieces of software throughout
the book; the goal is to give you a taste of many so you can get a feel for when
each is useful.

OllyDbg
OllyDbg is an immensely popular and powerful debugger. While most debug-
gers focus on debugging, Olly has extended features, including the following:
■■ Extensibility, plugins, scripting
■■ Execution tracing system
■■ Code patching features
■■ Automatic parameter descriptions for most Windows functions
■■ Emphasis on binary code analysis (i.e., not based around source debugging)
■■ Small and portable
These features make OllyDbg excellent for the following:
■■ Writing exploits
■■ Analyzing malware
■■ Reverse engineering
168 Chapter 10 ■ Cracking: Tools and Strategies

However, while OllyDbg is a powerful and popular tool, it does have its
limitations. One of these is that it works only for 32-bit executables, which
admittedly are a dying breed but not dead yet.
The other is that the OllyDbg interface often takes some getting used to and
does not feel robust or intuitive at first. However, you should definitely stick
with it, as it is a powerful dynamic analysis tool.

Immunity
Immunity is a fork of OllyDby, meaning that it has many of the same capabil-
ities. It also introduces many additional features that make it popular for exploit
developers, such as support for Python scripting.
However, like OllyDbg, Immunity can be used only to debug 32-bit execut-
ables. Also, it inherits OllyDbg’s unintuitive user interface.

x86dbg
x86dbg is a replacement for OllyDbg that supports both 32-bit (x86dbg) and
64-bit (x64dbg) applications. This wider support means that it is commonly the
tool of choice when reversing or debugging 64-bit applications.

WinDbg
WinDbg is a debugger that is universally applicable, has strong support, and
offers excellent debugging symbol support (but which is less useful with RE).
However, it has a debugging focus and lacks some features of RE-focused tools.

Debugging with Immunity

Because of time and space constraints, exploring all of these debuggers is infea-
 sible in this book. Immunity was selected because of its popularity for reverse
engineering and exploit development. However, it’s important to remember
that all of these debuggers have similar features, and skills learned in one will
often translate over to others.
Figure 10.22 shows how Immunity looks in Windows. From the top-left and
moving clockwise, the four windows show the program’s disassembly, regis-
ters, stack, and memory.

Immunity: Assembly
Figure 10.23 shows a program’s disassembly in Immunity. Note that it shows
the memory address, machine code, and x86 assembly.
Chapter 10 ■ Cracking: Tools and Strategies 169

Figure 10.22: Immunity debugger window

Figure 10.23: Assembly code in Immunity debugger

170 Chapter 10 ■ Cracking: Tools and Strategies

To select a line of code, click it. Once a line is selected, Immunity offers var-
ious keyboard shortcuts, including the following:
■■ ;: Add a comment to the selected line. This is the most important part of
reverse engineering; it helps you keep track of your work.
■■ ctrl-a: Auto-analyze the program. Immunity can do a fairly good job of
adding comments and guessing function parameters.
■■ <enter>: Navigate to the selected function. For example, if you see the
assembly call 0x1234 and want to find out what the function at 0x1234 does.
■■ -: Go back to the previous location. For example, after you’ve analyzed
function 0x1234 and want to return to where you were.
■■ +: Go to the next location (after pressing -). For example, if you returned
to the calling function with -, but then want to go back to function 0x1234.
■■ ctrl-r: Find cross-references to the selected line. For example, if you have
a string selected in the memory dump window and want to know who
uses that string; or if you have the top of a function selected in the disas-
sembly and want to find out who calls that function.
■■ Double-click address: Set a debugging breakpoint at this address.
 Immunity: Modules
In Immunity, you can load the list of executable modules by pressing the e button.
This shows all the code—including dynamically loaded libraries—that you can
debug, as shown in Figure 10.24. After opening the list, you can double-click a
module to go to that code.
When you start Immunity, see what module you are currently looking at
by checking the eip register. In nearly every case, you will want to start by
debugging the main executable, not a shared library like ntdll. You can use
the modules window to switch to the main executable.

Immunity: Strings
It is often useful to find what code is using a certain string in the executable. To
find all the strings that a program is using, right-click and select Search For ➪
All Referenced Text Strings, as shown in Figure 10.25.
In the strings window, right-click and select Search For Text to find a specific
string, as shown in Figure 10.26. Then, right-click again, and select Search For
Next to find the next reference to that string. You can double-click a string’s
address to go to the location where it is used in the disassembly.
Chapter 10 ■ Cracking: Tools and Strategies 171

Figure 10.24: Executable modules in the Immunity debugger

Figure 10.25: Strings in Immunity debugger

172 Chapter 10 ■ Cracking: Tools and Strategies

Figure 10.26: String references in Immunity debugger

Immunity: Running the Program

Click the play arrow to launch the executable under the debugger, as shown in
Figure 10.27. Execution can be stopped by clicking the X to the left of the play
arrow or can be paused using the pause button to its right. Execution can be
restarted via the button with two left-facing arrows.
 Figure 10.27: Launching an executable in Immunity debugger
Chapter 10 ■ Cracking: Tools and Strategies 173

After execution has been halted by a breakpoint or the pause button, you can
click Step Into to progress the program one instruction, as shown in Figure 10.28.
Alternatively, if you are stopped on a function call but already know or do not
care about what the function does, click overstep Over, as shown in Figure 10.29,
to continue debugging after the function returns.

Figure 10.28: Single-stepping in Immunity debugger

Immunity: Exceptions
Many applications generate exceptions as part of normal execution. For example,
a try {} except {} block will generate an exception if anything goes wrong in
the try block. As a debugger, dynamic analysis tools like Immunity typically
intercept the exception first to see if you want to do anything with it.
But for reverse engineering, you generally don’t want to interfere with normal
execution. Instead, you want to let the application handle the exception the way
it normally would. This means you almost always want to pass the exception
from the debugger to the application.
As shown in Figure 10.30, exceptions are reported at the bottom of the Immu-
nity window, but each debugger is slightly different. In Immunity, press Shift+F9
to pass the exception and continue execution.
174 Chapter 10 ■ Cracking: Tools and Strategies

Figure 10.29: Stepping over instructions in Immunity debugger

Figure 10.30: Exceptions in Immunity debugger

Immunity: REwriting the Program

 Immunity has many features to aid in the development of patches to modify
software behavior. For the purposes of software cracking, this includes making
program edits to remove key checks, nag screens, etc.
Chapter 10 ■ Cracking: Tools and Strategies 175

In your first cracks, you will use the process of “noping” out code to remove
it from the program. This involves replacing program instructions with nop
instructions.
To do so in Immunity, first select the instruction(s) that you want to remove.
Then, right-click and select Binary ➪ Fill With NOPs, as shown in Figure 10.31.
This will replace the selected instruction(s) with a series of nops, as shown
in Figure 10.32.

Figure 10.31: noping out code in Immunity debugger

Figure 10.32: noped code in Immunity debugger

After modifying the program, test the patch by rerunning the program. If you
patched the correct portion of code, you should find that the nag screen (key
check, etc.) has disappeared.
However, if the patch crashes or failed to remove your target, you can easily
revert your changes and try again. To do so, select the patch button to bring up
the patches window. Then, right-click your patch and select Restore Original
Code, as shown in Figure 10.33, to revert your patch and try again.
Once you have identified a working patch, save your changes to the executable
to make it permanent. As shown in Figure 10.34, right-click and select Copy To
Executable ➪ All Modifications. When a confirmation window appears, select
Copy All.
A modified executable window should appear, showing your changes. Close
the window, and select Yes to save your file. Give your file a new name, such
as cracked.exe.
If you are confident in your modification, you can run cracked.exe directly.
If you want to keep debugging with these new changes, you’ll need to reload
cracked.exe into Immunity.
176 Chapter 10 ■ Cracking: Tools and Strategies
 Figure 10.33: Reverting modified code in Immunity debugger

Figure 10.34: Saving a modified file in Immunity debugger

Lab: Cracking with Immunity

This lab provides hands-on experience with cracking programs using a debugger.
Labs and all associated instructions can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Lab Cracking with Immunity and follow the provided
instructions.

Skills
This lab practices reverse engineering, patching, and circumventing software
protections using Immunity and Resource Hacker. Some of the key skills tested
include the following:
■■ Reverse engineering x86
Chapter 10 ■ Cracking: Tools and Strategies 177

■■ Patching
■■ Static versus dynamic analysis

Takeaways
Software can be easily modified to add, change, or remove functionality. These
same techniques can be used to circumvent anything from trivial to advanced
protections, as long as you understand how the software works.

Summary
Key checkers are intended to protect against the distribution and use of unli-
censed and cracked copies of software, but no defense is perfect. Tools like
Procmon, Resource Hacker, and debuggers can be used to understand these
defenses and defeat them through the use of key generators or patching.
 CHAPTER

11

Patching and Advanced Tooling

The previous chapter introduced software cracking and patching. This chapter
provides a more in-depth look at patching and some of the more advanced tools
that can be used for reversing and cracking.

Patching in 010 Editor

It is often useful to be able to view and edit the hex of a file. If you’ve ever
tried to open a binary in a text editor, you saw a lot of crazy symbols and blank
space. This is because the text editor is trying to interpret everything in the file
as ASCII, which it’s not. Instead, we need an editor that will display as hex,
not ASCII. There are many different hex editors capable of doing this. One of
our favorites is 010 Editor. (Find links in the Tools section of our GitHub site
at https://github.com/DazzleCatDuo/X86-SOFTWARE-REVERSE-ENGINEERING-
CRACKING-AND-COUNTER-MEASURES).
Open any file (executable, data file, image, music, etc.) to view its hex.
Figure 11.1 shows a sample executable in 010 Editor.
Figure 11.2 shows the Inspector pane. This shows the various different pos-
sible interpretations of the data at your cursor.

179
180 Chapter 11 ■ Patching and Advanced Tooling

Figure 11.1: Viewing a file in 010 Editor

Figure 11.2: Inspector pane in 010 Editor

 If you know what you’re looking for, you can search for it, as shown in
Figure 11.3. You can search for many different types of data, including the
following:
■■ Text
■■ Hex bytes
■■ ASCII string
■■ Unicode string
■■ EBCDIC string
■■ Signed/unsigned byte
■■ Signed/unsigned short
Chapter 11 ■ Patching and Advanced Tooling 181

■■ Signed/unsigned int
■■ Signed/unsigned int64
■■ Float
■■ Double
■■ Variable name
■■ Variable value

Figure 11.3: Searching in 010 Editor

You can jump to a specific address if you know where you need to go, as
shown in Figure 11.4. This location of “where to go” can be specified as a byte,
line number, sector, or short.

Figure 11.4: Jumping to an address in 010 Editor

182 Chapter 11 ■ Patching and Advanced Tooling

In 010 Editor, you can directly modify the hex. Simply place your cursor and
start typing to overwrite.
However, 010 Editor understands how important it is to maintain file size.
When you type values, in 010 Editor it overwrites existing values at that location.
It does not insert them, which would make the file larger.
 CodeFusion Patching
After a researcher figures out how to crack a program, the next step is often
to create a patcher/cracker utility. This will allow others to crack the same
program.
CodeFusion is a popular patch generator. It creates a stand-alone execut-
able file that can be used to crack a specific application. (Find links in the
tools section of our GitHub site here: https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES).
To start creating a patcher, launch CodeFusion, and configure the information
that will appear when the patcher is launched. This information is shown in
Figure 11.5 and includes the program caption, program name, comments, icon,
etc. These can be whatever you want.

Figure 11.5: CodeFusion start screen

On the next screen, add the files to be patched, as shown in Figure 11.6. This
is the executable that you want to crack.
Chapter 11 ■ Patching and Advanced Tooling 183

Figure 11.6: Loading a file in CodeFusion

Next, add the patch information by clicking the + icon shown in Figure 11.7.
This is typically the information you learned from Immunity, Cheat Engine,
IDA, etc. It usually includes an offset to patch, and the bytes to replace. Often,
the bytes to patch with are 0x90 (nops). On the next page, click Make Win32
Executable to create an EXE file to patch the target application.

Figure 11.7: Adding patch information in CodeFusion

184 Chapter 11 ■ Patching and Advanced Tooling
 CodeFusion will add a new executable alongside the target application. As
shown in Figure 11.8, run this executable, select the target, and click Start to
apply the patch and crack the application.

Figure 11.8: Launching the patched executable in CodeFusion

This cracking executable is what a cracking group would often redistribute.

It is much smaller and more portable than the full, cracked app. Someone just
needs to have the application installed, download your small patcher, and run
it, and it will perform the patching to the genuine executable.

Cheat Engine
Cheat Engine is a popular and powerful open-source memory scanner, hex editor,
and debugger. While the tool is primarily used for cheating in computer games, it
can also often be valuable for quick dynamic analysis in software cracking. (Find
links in our tools section on our GitHub here: https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES).
Cheat Engine enables searches for values input by the user with a wide
variety of options. These allow the user to find and sort through the com-
puter’s memory.

Cheat Engine: Open a Process

Unlike other tools, reversing with Cheat Engine doesn’t start with opening an
executable. Instead, you select a running process to edit.
First, run the program that you want to crack. Then, start Cheat Engine and
click Select A Process To Open, as shown in Figure 11.9. The Process List window
appears, and you can select the process to crack and click Open.
Chapter 11 ■ Patching and Advanced Tooling 185

Figure 11.9: Opening a process in Cheat Engine

Cheat Engine: View Memory

Cheat Engine is based heavily around the idea of memory scans.
The main Cheat Engine window is primarily used for scanning memory. How-
ever, for now, focus on some simpler functionality: memory view. As shown in
Figure 11.10, click Memory View to view the process’s memory.
The memory view provides an easy and powerful way to view, scan, and
modify a process’s memory. As shown in Figure 11.11, memory view includes
the disassembly at the top of the screen, an instruction reference in the middle,
and a hex dump at the bottom.
186 Chapter 11 ■ Patching and Advanced Tooling

Figure 11.10: Viewing memory in Cheat Engine

Figure 11.11: Memory Viewer pane in Cheat Engine

Chapter 11 ■ Patching and Advanced Tooling 187

Cheat Engine: String References

As discussed, examining the strings in an executable can provide invaluable
hints regarding its functionality. To view strings in Cheat Engine, select View
➪ Referenced Strings to get a list of all of the strings used by the program.
Figure 11.12 shows the window that will pop up, where you can click on a
string to view its cross references. Double-click on a cross-reference address to
go to where the string is used in the disassembly.

Figure 11.12: String references in Cheat Engine

Cheat Engine: REwriting Programs

Recall that noping out a chunk of code is the safest and easiest way to remove
it without affecting the rest of the program. Cheat Engine makes this easy. To
bypass an instruction (such as a final conditional jump in a key check), right-
click the instruction and select Replace With Code That Does Nothing, as shown
in Figure 11.13.
Cheat Engine is highly interactive. You can immediately try your modifica-
tion in the running program! If your modification didn’t work or if you want
to undo it, right-click the modified code and select Restore With Original Code,
as shown in Figure 11.14.
188 Chapter 11 ■ Patching and Advanced Tooling

Figure 11.13: noping out instructions in Cheat Engine

Cheat Engine: Copying Bytes

Once you’ve found a working patch, the next step is to copy that patch over
into an executable file rather than a running process. As shown in Figure 11.15,
you can right-click the patch location and select Copy To Clipboard ➪ Bytes
Only to copy those bytes for use by other tools.

Cheat Engine: Getting Addresses

To make a patch, you need to know where in the file the data to patch is. Cheat
Engine is all about runtime analysis, so it does not know where in the file the
data is.
To find an address, use 010 Editor to perform a search for the machine code
you are replacing. That address is the file offset to patch for use in CodeFusion
or other patchers.
Chapter 11 ■ Patching and Advanced Tooling 189

Figure 11.14: Reverting changes in Cheat Engine

Figure 11.15: Copying bytes in Cheat Engine

190 Chapter 11 ■ Patching and Advanced Tooling

Lab: Cracking LaFarge

This lab practices using these tools to patch programs. Labs and all associated
instructions can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES
For this lab, please locate Lab LaFarge and follow the provided instructions.
 Skills
This lab provides experience using CodeFusion and Cheat Engine to practice
the following skills:
■■ Reverse engineering x86
■■ Patching and patchers

Takeaways
A variety of tools are available for reverse engineering and cracking; choosing
the “right” one depends on the challenge at hand and personal preference.
Crackmes are a (usually) safe, always legal, incredibly addictive way to prac-
tice your cracking skills.

IDA Introduction
If you’ve ever googled reverse engineering tools, IDA is guaranteed to come
up. It’s the Cadillac of reverse engineering tools.
IDA, aka the Interactive Disassembler, allows for binary visualization of dis-
assembly. It is available under a freemium model where limited features are
available for free, while some of the more powerful features (or more obscure
architectures) require a paid license.
Figure 11.16 shows the process of loading a new file in IDA. IDA automatically
recognizes many common file formats, but if it gets it wrong, you can select
the generic Binary File. IDA also offers a Processor Type drop-down menu to
change architectures.
One of IDA’s greatest strengths is its graph view, which shows a visual rep-
resentation of an executable’s x86 assembly and control flows. Figure 11.17
shows this view and some of the most useful components of it, including a
memory map of the executable, a list of functions, the logic block view, and
a graph window.
Chapter 11 ■ Patching and Advanced Tooling 191

Figure 11.16: Loading a file in IDA

Figure 11.17: IDA graph view

192 Chapter 11 ■ Patching and Advanced Tooling
 IDA: Strings
As always, strings are a good starting point when analyzing a new executable.
However, IDA doesn’t show them by default. Figure 11.18 shows how to access
the String view by clicking View ➪ Open Subviews ➪ Strings.

Figure 11.18: Opening strings view in IDA

Figure 11.19 shows the full list of strings in IDA. IDA shows the text of the
string itself, its address, and its predicted length.

Figure 11.19: Strings view in IDA

Chapter 11 ■ Patching and Advanced Tooling 193

Click a string to highlight it. Then, press X or right-click and select Jump To
Xref To Operand. This will open up a window showing all of the locations where
the string is used in the program, as shown in Figure 11.20.

Figure 11.20: String cross-references in IDA

Following one of these cross-references will show the disassembly where the
string is used. As shown in Figure 11.21, IDA understands how string references
work. When it sees one, it shows the string as a comment.

Figure 11.21: Strings in IDA code view

IDA: Basic Blocks

IDA’s graph view shows code in basic blocks. A basic block is a contiguous
sequence of instructions uninterrupted by a branching instruction or branch-
ing reference.
Consider the following simple program in pseudocode. Figure 11.22 shows
what this program looks like when disassembled in IDA.
194 Chapter 11 ■ Patching and Advanced Tooling

Figure 11.22: Basic blocks in IDA

int main(int argc, char* argv[])

{
return argc;
}

IDA: Functions and Variables

IDA understands many calling conventions, including cdecl. It will recognize
cdecl and knows the first argument always starts at ebp+8. IDA renames that
offset to arg_0 to make it easier to read. It will do this renaming with all of the
input variables (arg_X), as shown in Figure 11.23.
This understanding also extends to how local variables are handled on the
stack. For example, as shown in Figure 11.24, IDA will rename local variables
to var_X.
Knowing how IDA labels arguments and variables can greatly aid in function
analysis. For example, with the function shown in Figure 11.25, we can very
quickly tell it has one local variable and six input variables because we recog-
nize how IDA does its naming conventions.
Often, IDA has no information about the intent or context in which these var-
iables are used, so it labels them sequentially. As you learn about an argument,
variable, or function, you can rename it by pressing N or right-clicking the
variable label and selecting Rename.
Chapter 11 ■ Patching and Advanced Tooling 195

Figure 11.23: Function arguments in IDA

Figure 11.24: Local variables in IDA

Figure 11.25: Local variables and function arguments in IDA

196 Chapter 11 ■ Patching and Advanced Tooling

IDA: Comments
When reversing an application, it’s essential to be able to track what you’ve
figured out and done so far. In IDA, pressing ; opens up a box to enter com-
ments, as shown in Figure 11.26.

Figure 11.26: IDA comment window

One tip is to put an identifier like “_x” in all of your comments. This gives
you something to search for to find all comments.
To start a search for comments, select Search ➪ Text, as shown in Figure
11.27.
Then, search for “_x” while selecting Find All Occurrences to find all of the
comments that you’ve placed in the program.

Figure 11.27: Searching for comments in IDA

By using a consistent commenting style and searching for comments, it’s easy
to find places in the code that you’ve already explored. For example, as shown
in Figure 11.28, you can quickly identify locations that were marked “TODO”
for later analysis.
Chapter 11 ■ Patching and Advanced Tooling 197
Figure 11.28: Search results in IDA

IDA: Paths
IDA shows three types of paths between basic blocks:
■■ Red: Path taken if a conditional jump is not taken
■■ Green: Path taken if a conditional jump is taken
■■ Blue: Guaranteed path (no conditionals)
For example, consider the following code sample containing a simple if
statement:
int main(int argc, char* argv[])
{
if (argc > 1)
return 0;

return argc;
}

Figure 11.29 shows how this code would look in IDA. After the conditional
block, the paths diverge. The colors aren’t shown in this book, but the left path,
which is red in IDA, shows what happens if the jump is not taken. The right
path, which is green in IDA, is followed if the conditional resolves to false.

Figure 11.29: Code paths in IDA

198 Chapter 11 ■ Patching and Advanced Tooling

Below this point, several more arrows indicate transitions between basic
blocks. Since none of these involves conditionals, they will all be blue in IDA.

IDA Patching
IDA is another tool that can be used to patch executables. As an example, con-
sider the following code:
printf("please enter the password\n");
scanf("%s", user_entered_password);
if (strcmp(user_entered_password, correct_password) == 0)
{
printf("SUCCESS\n");
}
else
 {
printf("Failure\n");
}

This code implements a simple authentication system. It asks a user to enter

a password and checks the answer. If the answer is correct, it prints SUCCESS;
otherwise, it prints Failure. While it’s a simplistic example, keep in mind this
flow of checking the password and going one way if it’s wrong and one way if
it’s right is very common. In IDA, you can patch the application to defeat this
password verification.
By default, IDA does not show the machine code in graph view. Unless you’re
patching, it doesn’t serve much purpose. But when you start to desire patch-
ing, you’ll want to see it. To show machine code, select Options ➪ General to
open the window shown in Figure 11.30. Then, specify the number of opcode
bytes to show in graph view (most opcodes don’t exceed 8 bytes, so it’s a good
practice to set it to 8).
Figure 11.31 shows the application’s password-checking logic in IDA. As
shown, the left (red) path is taken if the passwords match, while the right (green)
path is taken if they don’t.
The instruction that decides which jump to take is jnz. Recall that jnz stands
for “jump not zero.”
This password check could be defeated in a couple of different ways. One
option is to try to figure out what needs to be “not zero.” This means figuring
out what two values it’s comparing so you can potentially make a valid key
or a cracker.
An easier alternative is to use your knowledge of x86 to patch the applica-
tion. As is, the application evaluates a condition and performs a jnz (0x75) if
the password is incorrect. But what if you did the exact opposite? Changing
Chapter 11 ■ Patching and Advanced Tooling 199

this jnz to a jz (0x74) will reverse the logic, causing the application to accept
only incorrect passwords. With the logic flipped, an incorrect password would
result in success and a correct one would result in failure.

Figure 11.30: Showing opcode bytes in IDA

Figure 11.31: Password-checking code in IDA

To change the instruction, highlight it and click Edit ➪ Patch Program ➪

Change Byte. Then, in the Patch Bytes window shown in Figure 11.32, change
the first value from 74 to 75.
Figure 11.33 shows how the application will look after the patch is applied.
The single bit that was changed will be highlighted in IDA, and the meaning of
the two paths after the jump will be reversed. Now, the application will work
for anything except the correct password.
200 Chapter 11 ■ Patching and Advanced Tooling

Figure 11.32: IDA Patch Bytes window

Figure 11.33: Password-checking logic in IDA after patching

Lab: IDA Logic Flows

This lab provides an introduction to using IDA for reversing. The lab files are
on the Windows VM in the ida_logic folder on the Desktop. Inside this folder
will be several binaries. Find out which of them is:
■■ if
■■ Multipart if ( i.e., if(cont1 && cond2))
■■ while loop
■■ for loop
■■ do while loop
Chapter 11 ■ Patching and Advanced Tooling 201

Skills
This lab provides practice using IDA to reverse engineer control flow graphs.
The goal is to learn to quickly identify high-level coding constructs based
on their control flow patterns.

Takeaways
Analyzing a program’s control flow can make it easier to quickly understand
what is happening inside of code. Getting good at recognizing these flows
quickly can vastly improve your reverse engineering ability.
Ghidra
Ghidra is a static analysis tool released in 2019 by the NSA. It has many simi-
larities to IDA, but unlike IDA, it is free and open source. In many situations,
Ghidra is an adequate replacement to IDA.
IDA has a much longer reputation in the space, but Ghidra is also immensely
powerful and in many cases has a lot of the same features. This example demos
IDA given its long history in the reverse engineering space, but everything
shown can also be done in Ghidra. The tools are similar enough that skills in
one will often transfer over. Try Ghidra out for some of the later, open-ended
labs in this book and your own practice.

Lab: Cracking with IDA

This lab takes a look at a more complex application in IDA. Labs and all asso-
ciated instructions can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Lab Cracking with IDA and follow the provided
instructions.

Skills
This lab practices using IDA to crack large, real-world applications. The goal is
to learn to quickly identify points of interest and to prioritize multiple cracking
approaches.
202 Chapter 11 ■ Patching and Advanced Tooling

Takeaways
Real-world programs are too large for uniform, fine-grained analysis. Triage is
critical to finding the points of interest.
Multiple opportunities are usually available to a cracker. Selecting which to
pursue can save (or cost) significant time.

Summary
This chapter explored some of the most widely used tools for reversing and
cracking. Take the time to become familiar with them. It’ll pay off in the long run!
CHAPTER
 12

Defense

How do you defend against cracking? To start, it’s essential to have a good key
check design (don’t pull a Starcraft/Half-Life). From there, you can implement
additional defensive options.
However, it’s important to remember that there is no such thing as uncrackable
software. As a defender, your job is to slow attackers down in the critical parts
of your software and make them frustrated enough they go to a different target.
Like many things in cybersecurity, you just don’t want to be the low-hanging
fruit. “When swimming in shark-infested water, you don’t have to be the fast-
est. . .just faster than the guy next to you.”

Obfuscation
Obfuscation is the practice of hiding the intended meaning of code by purpose-
fully making logic ambiguous and unclear. It can be valuable for slowing reverse
engineering to do the following:
■■ Slow cracking
■■ Slow tampering
■■ Protect intellectual property

203
204 Chapter 12 ■ Defense

Done well, obfuscation can make code essentially unreadable. For example,
the following C code (available from www.ioccc.org/1988/phillipps.c), when
compiled and run, prints out the lyrics to the entire 12 days of Christmas song.
It was one of the IOCCC winners, which is a competition to hand-obfuscate
code. Looking at it makes my brain hurt, and I can’t guess at how long I’d have
to reverse engineer the code before I figured out what it did.
#include <stdio.h>
main(t,_,a)
char
*
a;
{
 return!
0<t?
t<3?
main(-79,-13,a+
main(-87,1-_,
main(- 86, 0, a+1 )
+a)):
1,
t<_?
main(t+1, _, a )
:3,
main ( -94, -27+t, a )
&&t == 2 ?_
<13 ?
main ( 2, _+1, "%s %d %d\n" )
:9:16:
t<0?
t<-72?
main( _, t,
"@n'+,#'/*{}w+/w#cdnr/+,{}r/*de}+,/*{*+,/w{%+,/w#q#n+,/#{l,+,
/n{n+,/+#n+,/#;\
#q#n+,/+k#;*+,/'r :'d*'3,}{w+K w'K:'+}e#';dq#'l q#'+d'K#!
/+k#;\
q#'r}eKK#}w'r}eKK{nl]'/#;#q#n'){)#}w'){){nl]'/+#n';d}rw'
i;# ){nl]!/n{n#'; \
r{#w'r nc{nl]'/#{l,+'K {rw' iK{;[{nl]'/w#q#\
\
n'wk nw' iwk{KK{nl]!/w{%'l##w#' i; :{nl]'/*{q#'ld;r'}{nlwb!/*de}'c ;;\
{nl'-{}rw]'/+,}##'*}#nc,',#nw]'/+kd'+e}+;\
#'rdq#w! nr'/ ') }+}{rl#'{n' ')# }'+}##(!!/")
:
t<-50?
_==*a ?
putchar(31[a]):
main(-65,_,a+1)
:
Chapter 12 ■ Defense 205

main((*a == '/') + t, _, a + 1 )
:
0<t?
main ( 2, 2 , "%s")
:*a=='/'||
main(0,
 main(-
61,*a, "!ek;dc i@bK'(q)-
[w]*%n+r3#l,{}:\nuwloca-
O;m .vpbks,
fxntdCeghiry")
,a+1);}

The concept of obfuscation has also made its way into popular culture. The
following quotes are from a scene in one of the James Bond movies, Skyfall when
Q is attempting to get into Silva’s laptop.
■■ “There are algorithms and encryptions and asymmetrics!”
■■ “Looks like obfuscated code to conceal its true purpose. Security through
obscurity!”
Obfuscations can be applied by hand or automatically to a program at var-
ious stages of its life cycle, including the following:
■■ Source code
■■ Bytecode
■■ Object code
■■ Binary executable code

Evaluating Obfuscation
When evaluating options for obfuscation, there are a few different factors to
consider:
■■ Potency: How much obfuscation is applied to the program
■■ Resilience: How well-obfuscated code holds up to attack from reverse
engineering tools
■■ Stealth: How well-obfuscated code blends in with the rest of the program
■■ Cost: Performance penalty of an obfuscated application
In general, these factors tend to work against each other. For example, the
more potent the obfuscation is, the less stealthy it typically is.
In practice, performance cost is often the limiting factor. However, almost
all obfuscations allow some degree of scaling/tuning based on requirements.
206 Chapter 12 ■ Defense

Automated Obfuscation
Obfuscation can be performed manually. However, it’s almost always better to
use tools to obfuscate the code. Some of the common obfuscation techniques
include the following:
■■ Name mangling
■■ String encryption
■■ Control flow obfuscation
 ■■ Control flow flattening
■■ Opaque predicates
■■ Instruction substitution

Name Mangling
Name mangling involves obfuscating function and variable names. This can be
done a few different ways, including the following:
■■ Replace with gibberish (get_key -> aVJ230AM)
■■ Replace with misleading name (get_key -> draw_screen)
■■ Replace with nondescriptive name (get_key -> a)
After mangling, the purpose of functions and variables is no longer immedi-
ately apparent. For example, consider the following code sample:
public static void SelectionSort <T> (T[] data, int size)
where T: IComparable
{
for (int num1 = size – 1; num1 >= 1; num1-
-
)
{
T local1 = data[0];
int num2 = 0;
for (int num3 = 1; num3 <= num1; num3++)
{
if (data[num3].CompareTo(local1) > 0)
{
local1 = data[num3];
num2 = num3;
}
}
T local2 = data[num2];
data[num2] = data[num1];
data[num1] = local2;
}
}
Chapter 12 ■ Defense 207

After mangling, this might look something like the following:

public static void a <a> (a[] A_0, int A_1) where a:IComparable
{
int num1 = A_1 – 1;
Label_004D:
if (num1 < 1)
 {
return;
}
a local1 A_0[0];
int num2 = 0;
int num3 = 1;
while(true)
{
if (num3 <= num1)
{
if (A_0[num3].CompareTo(local1) > 0)
{
local1 = A_0[num3];
num2 = num3;
}
}
else
{
a local2 = A_0[num2];
A_0[num2] = A_0[num1];
A_0[num1] = local2;
num1-
-
;
goto Label_004D;
}
num3++;
}
}

In the original, it is relatively easy to determine that the code is a sort algorithm
even without the function name. However, doing so after mangling is much harder.

String Encryption
Another obfuscation technique is for the obfuscator to encrypt strings when the
executable is built. A decrypt function in the code will then decrypt individual
strings as needed at runtime. This renders tools like IDA’s string view unusable.
String encryption can have a dramatic effect on code readability. Consider
the following code:
public a() {
this.a = "Hi, my name is Paul."
}
208 Chapter 12 ■ Defense
public static void a() {
a a1 = new a();
Console.WriteLine("Enter password: ");
string text1 = Console.ReadLine();
if (!text1.Equals(a1.a))
{
Console.WriteLine("Incorrect password.");
}
else
{
Console.WriteLine("Correct password.");
}
Console.ReadLine();
}

After string encryption, this code might look like this:

pubic a() {
int num1 = 5;
this.a =
a("\ue6ad\u9eb1\u94b3\uc1b7\u9ab9\ud2bb\uadbf\ua7c1\ue4c
3\uafc5\ubbc7\ueac9\u9ccb\uafcd\ua5cf\ubed1\ufad3", num1;
}

public static void a()

{
int num1 = 13;
a a1 = new a();
Console.WriteLine(a("\uf3b5\ud6b7\uceb9\uccbd\ue0bf\ub2c1\ua5c3\u
b5c5\ubbc7\ubdc9\ua3cb\ubccd\ub4cf\ue8d1\uf4d3, num1));
string text1 = Console.ReadLine();
if (1text1.Equals(a1.a)) {
Console.WriteLine(a(\uffb5\ud8b7\ud3bb\uccbd\ub2bf\ua7c1
\ua7c3\ub2c5\ue8c7\ubac9\uadcb\ubdcd\ua3cf\ua5d1\ubb
d3\ua4d5\ubcd7\uf4d9", num1));
}
else
{
Console.WriteLine(a("\uf5b5\ud7b7\uc8b9\ucebb\ua3bf\ub6c1
\ue4c3\ub6c5\ua9c7\ub9c9\ubfcb\ub9cd\ubfcf\ua0d1\ub0
d3\uf8d5", num1));
}
Console.ReadLine();
}
 In the original, the strings make it easy to determine that this is authentica-
tion code (which is often very interesting to attackers). Without these strings,
the logic of the code is much more difficult to figure out. Keep in mind one of
the biggest struggles to cracking an application is finding the relevant code. In a
Chapter 12 ■ Defense 209

binary with hundreds of thousands of lines of code, only five might be related to
the key checker, and using tools like strings is a powerful way to quickly hone in
on those five lines. Taking away strings is quite painful to the reverse engineer.

Control Flow Flattening

With this obfuscation technique, the control flow of each function is “flattened.”
This includes the following steps:
1. The function is collapsed into a switch statement within an infinite loop.
2. Each basic block of the original flow is assigned a state number.
3. A switch statement selects between basic blocks, dispatching them in the
correct order.
Figure 12.1 shows how the flattening process transforms an application in
IDA. While the logic is the same, the control flow is much harder to analyze.

Figure 12.1: Control flow flattening in IDA

Opaque Predicates
Opaque predicates add junk code interleaved with real code. The junk code
never executes, while the real code always executes. However, to a reverse engi-
neer, this is a good way to distract them with useless code, making them spend
hours reverse engineering junk code that is essentially irrelevant. Figure 12.2
shows an example of this in IDA.
The path is determined by an if statement that always resolves to the same
value. However, it can take time to identify (an “opaque predicate”), slowing
analysis.
210 Chapter 12 ■ Defense

Figure 12.2: Opaque predicates in IDA

 Consider the following statement:
if ( (a<<1)%2 ) { b = a * b + a; } else { a = a + b; }

Where is the junk code here?

Instruction Substitution
Instruction substitution involves replacing easily identified instructions with
complex ones that perform the same action. For example, consider the follow-
ing code:
sub edx, 0x192A6C72
neg ecx
sub edx, ecx
add edx, 0x192A6C72

What was the original operation?

Obfuscators
Obfuscators typically provide “knobs” that allow the developer to tweak the
level of obfuscation. The reason for this is that more obfuscation is not always
better. In general, increasing obfuscation decreases execution speed and increases
file size. Also, drastically increasing obfuscation does not substantially increase
the difficulty of reverse engineering. Balancing usability and security requires
finding a middle ground.
If you manage to do that, obfuscation can be a valuable tool, especially for
code that is otherwise trivial to decompile (such as the JIT languages discussed
earlier, e.g., .NET, etc.). However, it’s also important to ensure that the tool you
are using does not also provide an easily accessible de-obfuscator.
Chapter 12 ■ Defense 211

For general-purpose obfuscation, OLLVM can be a good starting point. This tool
has a few benefits, including the fact that it works with the LLVM intermediate
representation (IR) and supports all LLVM front ends (gcc, clang) and many
source languages (C, C++, C#, Lisp, Fortran, Haskell, Python, Ruby, etc.).
The use of OLLVM is not recommended for production code. However, it
can be a good basis for custom obfuscators or simply learning/playing with
obfuscation.
In addition to OLLVM, there are numerous language-specific obfuscator tools
and tricks. Some examples include Dotfuscator for C# and Proguard for Java.
For JavaScript programs, tools such as YUICompressor and UglifyJS can be
used for obfuscation. In general, minimizers, simply as a byproduct, introduce
some reasonable level of obfuscation.
Python code can be compiled to bytecode to remove some variable names and
comments. Then, the bytecode can be obfuscated and released with a custom
interpreter. Some Python obfuscators include Tigress, BitBoost, and Opy, but
these are less popular than the ones mentioned earlier.

Defeating Obfuscators
Obfuscators are designed to protect against reverse engineering by making
it more difficult and time-consuming to perform. However, obfuscation isn’t
perfect, and as stated many times previously, motivated crackers can eventu-
ally defeat it.
Some of the ways that a reverse engineer can speed up the process of ana-
lyzing an obfuscated binary include the following:
■■ Run traces to identify real versus fake code
■■ Use symbolic analysis to simplify complexity
■■ Write custom scripts to remove obfuscations

Lab: Obfuscation
This lab explores obfuscation techniques. T Labs and all associated instructions
can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Lab Obfuscation and follow the provided instructions.

Skills
This lab provides experience in circumventing obfuscation techniques using
objdump. The goal is to understand the impact of common code defense techniques.
212 Chapter 12 ■ Defense

Takeaways
Obfuscation techniques will slow down—but not defeat—cracking. However,
remember that sometimes slowing down is enough. Advanced reverse engineers
often have tools to automatically circumvent common obfuscations.

Anti-Debugging
Debugging is often the fastest way to reverse engineer an executable. Anti-
debugging is a series of techniques to try to stop someone from having the
ability to dynamically analyze your application with a debugger. There are
many techniques in this space, but most of them are geared at trying to check
for the presence of a debugger. A few common anti-debugging checks include
 the following:
■■ Memory checks
■■ CPU checks
■■ Timing checks
■■ Exception checks
■■ Environment checks
As with most security controls, there are usability trade-offs to anti-debugging,
code size and performance being the two most painful side effects. Because of
this, anti-debugging functionality is often added only selectively, reserving its
use for the code most likely to be attacked (key checkers, sensitive IP addresses,
etc.). But as with all security there are pros and cons; if you build a bunch of anti-
debugging checks around your sensitive code, you’re also painting a bull’s-eye
telling an attacker exactly where the interesting stuff is. So, while they might not
be able to debug it, they now know exactly where to focus with static analysis
techniques. But that doesn’t mean it’s not worth doing; static analysis might
take 100x longer than dynamic, so even if you paint arrows to your sensitive
code, forcing them to do it statically can still be a powerful tool.
The main goal with anti-debugging is to identify when a debugger is attached
and take an action. The most commonly used actions include the following:
■■ Forcibly disconnecting the debugger
■■ Exiting the program
■■ Executing red herring code to waste an attacker’s time

IsDebuggerPresent()
IsDebuggerPresent is a memory check for a debugger. The function
IsDebuggerPresent, which is located in Windows.h, returns true if a program
Chapter 12 ■ Defense 213

is being run under a debugger. The following code shows an example of how
it is used to exit an application if a debugger is attached:
if (IsDebuggerPresent())
exit(1);

A check using IsDebuggerPresent can be defeated by placing a breakpoint

at the instruction right after the function returns. When the breakpoint triggers,
set the value of eax to 0, which tells the program that no debugger is attached.
Remember that eax holds the return value. Returning a 1 is undesirable, because
that means it detected the debugger, so instead make it return a 0. While that
seems trivial, keep in mind the game is to just make it harder. If your code has
100 of these checks, attackers wanting to debug have to track each of these
down and either manually breakpoint and change the return value every time
or start to get custom scripts going to do this change for them. Is that annoying
as an attacker? Yup.

Debug Registers
An application can also make use of the CPU’s debug registers to perform a
check for a debugger. Recall that the debugging section discussed software and
hardware breakpoints. A hardware breakpoint uses CPU hardware registers to
set itself.
These hardware breakpoints use debug registers (in x86: DR0, 1, 2, 3, 6, 7)
instead of memory modifications. It’s possible to detect debugging by exam-
ining these registers.
For example, consider the following code sample. It checks to see if any of
the debug registers are set, indicating a hardware breakpoint.
if (GetThreadContext(hThread, &ctx))
if ((ctx.Dr0 != 0x00) || ... || (ctx.Dr7 != 0x00))
exit(1);

The call to GetThreadContext() is crucial to this anti-debugging technique.

For those looking to bypass this technique, place a breakpoint after this call and
modify the context structure, setting the observed values of all of the debug
registers to 0x0. Again, is it doable to bypass? Yes. Is it annoying to an attacker
to have to keep doing these modifications? Yup. An annoyed attacker equals
success to a defender! Also recall we discussed that IDA 6.3 and above support
hardware breakpoints. These breakpoints don’t use the debug registers and
instead use page permissions. In other words, this type of anti-debugging check
won’t catch a hardware breakpoint.
214 Chapter 12 ■ Defense

RDTSC
RDTSC stands for the x86 instruction Read Timestamp Counter. This counter
can be used to read a timestamp from the CPU. This has lots of interesting uses,
but one of them is to perform a timing check for a debugger.
When running an application (with no debugger), the CPU is very fast, but
when a debugger is attached, it isn’t. Even if you’re not stepping and you’re just
letting the code run, it’s orders of magnitude slower than just letting the CPU
go. And it’s even slower if you’re doing something like single-stepping through
the code. With RDTSC, an application can take timestamps before and after a
block of code and measure how long the code took to execute. If the delta is
large, it’s likely that the code hit a breakpoint or was being manually stepped
through with a debugger.
The following pseudocode shows how RDTSC could be used to detect a
 debugger:
a = __rdtsc();
keycheck();
b = __rdtsc();
if (b -a > 0x10000)
exit(1);

To defeat this type of anti-debugging check, you could break on the second
call to RDTSC. You could then modify the value of either a to be closer to b or
b to be closer to a. Essentially, make the difference between the two very small
so it assumes execution went as planned. Bypassable? Yes. Annoying to have
to patch every time you debug? Yes!

Invalid CloseHandle()
The use of an invalid call to CloseHandle is an example of an exception check for
a debugger. The Windows CloseHandle function throws an exception if called
with an invalid handle while running under a debugger (and not otherwise).
An application can use this knowledge to call CloseHandle on an invalid handle
to detect the presence of a debugger.
The following code demonstrates how CloseHandle can be used to detect a
debugger:
HANDLE hInvalid = (HANDLE)0xDEADBEEF;
__try { CloseHandle(hInvalid); }
__except (EXCEPTION_EXECUTE_HANDLER) { exit(1); }

To defeat this check, set a breakpoint on CloseHandle. When the breakpoint

is triggered, modify the argument to INVALID_HANDLE_VALUE.
Chapter 12 ■ Defense 215

Directory Scanning
Directory scanning is an environment check for a debugger. It involves scanning
the file system for installations of common debuggers and cracking tools. If
these tools are found, then the application can choose to exit.
However, this is an indiscriminate search, and these tools may not be actively
debugging the application. As a result, it hurts legitimate users of these tools.
To defeat this check, set a breakpoint on the directory traversal. Then, mask
out the tool directories so that the application doesn’t see or search them.

Offensive Anti-Debugging
Anti-debugging techniques need not be passive detection of debuggers. Many
“active defense” approaches exist, including the following:
■■ NtUserBlockInput: Block keyboard input to the attached debugger.
■■ NtUserFindWindowEx: Get a handle to the debugger window.
■■ Debugger-specific attacks: For example, IDA versions older than 7.0 crash
at about 10,000 instructions without a branch.
Many more options exist. For offensive anti-debugging, first you need to
recognize the debugger is there, and then you take some type of offensive
action. Open-source plugins are available to help, including some used in the
following lab.
For defensive anti-debugging, it’s important to remember that you don’t
need to reinvent the wheel. Ready-made solutions are available, including free,
open-source Windows anti-debugger checks.

Defeating Anti-Debugging
Like other software defenses, anti-debugging code can be defeated (though if
done right, it’s painful). The first step is to find and reverse engineer the anti-
debug check. Often, this is accomplished by working backward from where
you got caught using the debugger.
Once you’ve identified the anti-debug code, you have a few different options
for defeating it, including the following:
■■ Removing the check via nops
■■ Placing a breakpoint on the check and modifying memory/registers to
mask the debugger
■■ Using built-in debugger plugins or scripts
216 Chapter 12 ■ Defense

In general, it’s stealthier to mask the debugger immediately at the anti-debug

check. For example, if an application is using IsDebuggerPresent, modify the
return value of IsDebuggerPresent rather than messing with the if statement
or exit code designed to use that value.

Lab: Anti-Debugging
This lab provides practice in defeating anti-debugging techniques. Labs and all
associated instructions can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Lab Anti-Debugging and follow the provided
instructions.
 Skills
This lab uses x64dbg to circumvent anti-debugging techniques. The goal is to
understand the impact of common defensive coding techniques.

Takeaways
Again, slowing down a reverse engineer is often enough; defenses don’t need
to be perfect. However, skilled reversers will have tools to overcome common
defensive techniques.

Summary
Developers want to defend themselves and their code against reversers and
crackers. This chapter explored some of the common methods for accomplishing
this, including obfuscation and anti-debugging protections.
CHAPTER

13

Advanced Defensive Techniques

The previous chapter presented some basic techniques for protecting an appli-
cation against reverse engineering and cracking. This chapter demonstrates
some more advanced techniques that are more difficult to defeat, including
tamper-proofing, packing, virtualization, and the use of cryptors.

Tamper-Proofing
One of the powerful cracking techniques we’ve covered is patching, both for
long-term cracking but also in the aid of reverse engineering. Tamper-proofing
is a series of techniques geared toward making software more difficult for an
attacker to modify. Some common approaches include the following:
■■ Hashing
■■ Signature
■■ Watermark
■■ Software guards
All of the following techniques have ways of being defeated, but (and I can’t
stress this enough) just because they have ways of being defeated doesn’t mean
they are not worth doing. Each of them provides a layer of defense in depth,
and even if the method for defeating them fits into a few sentences, this doesn’t
mean it’s easy in practice.
217
218 Chapter 13 ■ Advanced Defensive Techniques

Hashing
An application can use hash functions to implement tamper-proofing via the
following steps:
1. Compute a hash of the software.
2. Embed the hash in the software.
3. Have the software check its own hash before executing.
4. Any modifications to the software modify the hash.
The defense relies on the fact that changes to the application will cause the
hash check to fail. To defeat this, an attacker will need to make their changes
and then recompute the hash after modifications and changing the checked
value or removing the hash check entirely.

Signatures
Digital signatures can provide strong data integrity and authenticity protec-
tions. They use public key cryptography where a public and private key pair
is generated. To use them for tamper-proofing, follow these steps:
1. Sign the software with a private key, creating a signature.
2. Embed the signature in the software.
3. Have the software check its signature with your public key before executing.
4. Any modifications to the software make the signature invalid.
One of the key benefits of digital signatures is that it is effectively impos-
sible to generate a valid signature without knowledge of the private key. To
defeat this type of protection, an attacker would have to remove the signa-
ture check entirely or get ahold of the private key so they can regenerate a
valid signature.

Watermark
To implement watermarking, each purchaser of your software receives a unique
version of the executable, where modifications are made to the following:
■■ Instruction order
■■ Function names
■■ Parameter order
■■ Instruction substitution
■■ Etc.
Chapter 13 ■ Advanced Defensive Techniques 219
 The specific changes “watermark” that instance, allowing you to trace it back
to its owner, as well as detect modifications. Also, any modifications to the soft-
ware taint the watermark, making them obvious.
For an attacker to defeat this protection, they will need to identify water-
marked sections. Then, replace them with an alternate mark to hide the source
of the modified software.

Guards
With guards, code inside the program checks sensitive areas for modification.
For example, the code may specifically look at a critical jump to make sure it
still jumps to the intended location. Common areas to monitor with guards
include key checks, jump instructions, other guards, etc.
Any modifications to these sections are caught by the guards. The guards
will then change the software’s behavior (exit, change paths, undo modifi-
cations, etc.).
This defense relies on the fact that the guard is present and able to modify
the software as needed. If an attacker wants to defeat this technique, they will
need to remove the software guard code.

Packing
Packing is a broad term referring to techniques commonly used on executables
to compress and obfuscate their contents. Some common packing techniques
include the following:
■■ Compression/encryption of data sections
■■ Scrambling code sections
■■ Compression/encryption of code sections
■■ Anti-reverse engineering
One of the main advantages of packing is that it makes reverse engineering
harder. For example, a packer may include features that address many of the
common reverse engineering threats, including the following:
■■ Anti-debugging: Packers can conceal the use of IsDebuggerPresent,
making it more difficult to detect.
■■ Anti-virtualization: Packers can detect when an application is being
virtualized in a platform such as VMware and conceal detection code.
■■ Anti-dumping: Packers can erase headers in memory, making it difficult
to dump memory.
220 Chapter 13 ■ Advanced Defensive Techniques

■■ Anti-tampering: Anti-tampering can be implemented via checksums.

This includes both common ones (rolling checksum, CRC32, MD5, and
 SHA-1) and others (Tiger, Whirlpool, MD4, Adler).
Packers can use encryption to conceal their code. Often, this involves simple
algorithms, such as bitwise operators (XOR/ROL/...), LCG, RC4, and Tea.
However, more advanced encryption algorithms (DES, AES, Blowfish, Trivium,
IDEA, ElGamal, etc.) can also be used. If an application has been packed in
such a way that its code and data sections are encrypted, if you were to drop
it into one of the disassemblers or hex editors, you’d see only a small section
of code and a lot of nonsensical junk. The tiny section of code that is available
is the unpacker. For the code to run, it will need to unpack itself in memory at
runtime, but this means static analysis can’t see the rest of the code.
Packers can also use mutators (obfuscation), which alter code while keep-
ing the same instruction set and architecture. Some mutations that might be
used include reflowing and oligomorphism, or other obfuscation techniques
discussed in Chapter 12, “Defense.”

How Packers Work

The packer (a stand-alone tool) packs an executable (compresses, obfuscates,
etc.). Then, the packer adds an unpacker to the beginning of executable. When
the executable is run, the unpacker will be the first code that is run, and it will
unpack the original code and data into memory (and only memory).
Figure 13.1 shows what a packed executable will look like in IDA. IDA sees
the initial jump to the unpacker; however, the rest of the code looks like data.

Figure 13.1: Packed code in IDA

Is This a Strong Protection?

In the following sections, we will talk about some protection techniques and ask
the question of if they are a strong protection. The assessments are meant to, at a
very high level, bucket which areas each protection has the strongest impact. The
focus of our book is predominately offensive, but we felt it important to take a
Chapter 13 ■ Advanced Defensive Techniques 221

quick look at some of the defenses. In each section, to evaluate the effectiveness
of an anti-cracking defense, we will use something called the CIA triad (CIA
stands for confidentiality, integrity, and availability). For those not familiar with
this, it’s a common way to think about security controls, as not all security con-
trols cover all three parts of the triad, so it’s important to know which is useful
in each pillar. Integrity is the authenticity of something. Is it as it was originally
intended, or has it been modified? Confidentiality is the ability of something to be
accessible to only authorized parties. Availability is the level to which something
is available to perform its intended function. These three together are commonly
known as the CIA triad. Evaluating packers against the CIA triad:
■■ Confidentiality: Yes, aside from the unpacking portion of the code, the
rest of it is in nonreadable format.
■■ Integrity: Yes, modifications to the binary would cause corruption of the
packed sections, causing likely application failure.
■■ Availability: Packers can have a negative effect on performance, which
can affect availability. However, if configured carefully, this effect can be
minimized.

Defeating Packing
So, how can packers be defeated? Debug the program and watch for the program
to decrypt in memory. Once it is unpacked in memory, you can analyze it, but
any patching done will be viable only on the unpacked binary. Patching can’t
be saved back to the packed binary.
One natural thought that occurs to people is once it’s unpacked in memory,
can’t I just memory dump that out to a new unpacked binary? This is techni-
cally possible but difficult to do. Applications include a lot of startup code, and
getting it loading in the right spot in memory, setting up the stack, etc., doesn’t
naturally come from dumping memory and just calling it an EXE.
Another option is to see if you can unpack the program. Some of the common
packers out there have unpacking tools that can be used to reverse the protec-
tions put in place; some examples include UPX, MEW, and ASPack.
However, there may be no stand-alone unpacker, and the unpacking code exists
only in the packed executable. However, that doesn’t mean we’re stuck! There
are a number of great plugins and tools built specifically for this purpose, such
as OllyDumpEx and ImpRec, which aim to reconstruct the import table. This is a
complex but doable process, but not the focus of our book. However, if this is of
interest, there are some great blogs to be found online on import reconstruction.

PEiD
Often when approaching a file, it can be difficult to figure out what types of
manipulations were done to it. If you somehow know out of the gate it was
222 Chapter 13 ■ Advanced Defensive Techniques

packed with a certain tool, then it’s easy to start down that path. But cracking
doesn’t typically come with a handy playbook telling you what defenses are in
place. PEiD is a tool to detect most common packers, cryptors, and compilers
for portable execution files (e.g., applications). It can detect the signatures of
 more than 470 different obfuscation tools. Another more recent tool in this space
is Detect it Easy.
As we’ve mentioned, many defensive tools such as packers and cryptors
have unpackers and decryptors as well. Identifying the one used can reduce
analysis time by an order of magnitude by allowing you to strip away many of
an application’s protections.
Figure 13.2 shows an example of using PEiD. To start, select the file to check.
PEiD will then show the details of its packing, crypting, and compiling.

Figure 13.2: Identifying packers with PEiD

Lab: Detecting and Unpacking

This lab explores how to detect and defeat the use of a common packer. Labs
and all associated instructions can be found in their corresponding folder here:
https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES

For this lab, please locate Lab Detecting and Unpacking and follow the provided
instructions.

Skills
Packers are a common protection against reversing. This lab explores the use
of IDA, Cheat Engine, and PEiD to test the following skills:
■■ Detecting the presence of packers
■■ Unpacking programs with existing tools
■■ Unpacking programs with advanced debugging
Chapter 13 ■ Advanced Defensive Techniques 223

Takeaways
Off-the-shelf unpackers are available for many packers (don’t reinvent the
wheel). When unpackers are not available, the unpacked, original program can
still be manually recovered from memory.

Virtualization
Virtualization provides a form of obfuscation and packing. It translates a program
into a custom machine language and generates a virtual environment/machine
(VM) to interpret it. The VM is embedded into the application and runs when
the application is executed. Note that in this case we’re not talking about typ-
ical large virtual machines such as Windows or Linux running in a hypervi-
sor. Virtualization in this case can quite simply mean a layer of abstraction/
interpretation being added between the host (x86) and the code.
For example, consider the following simple “hello world” program:
#include <stdio.h>
int main(void)
{
printf(“hello, world!\n”);
return 0;
}

This program could then be compiled to an arbitrary machine language. For

example, this is what it looks like in Brain$#@!:
++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---
.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.#

The application is then packaged with an interpreter written in the target

architecture (i.e., x86).
#include <stdio.h>

char data[30000];
char program[30000];
int ip=0; /* instruction pointer */
int dp=0; /* data pointer */

char read_byte(void) { return getchar(); }

void write_byte(char b) { putchar(b); }

int main(void) {
224 Chapter 13 ■ Advanced Defensive Techniques

int i=0; char b;

do {
b=read_byte();
program[i]=b;
i++;
} while (b!='#’);
while (1) {
b=program[ip];
if (b==0) {
 break;
} else if (b=='>') {
dp++;
} else if (b=='<') {
dp-
-;
} else if (b=='+') {
data[dp]++;
} else if (b=='- ') {
data[dp]- -
;
} else if (b=='.') {
write_byte(data[dp]);
} else if (b==',') {
data[dp]=read_byte();
} else if (b=='[') {
if (!data[dp]) {
int c=1;
do {
ip++;
if (program[ip]=='[‘) { c++; }
else if (program[ip]==']’) { c-
-
;}
} while (c);
}
} else if (b==']') {
if (data[dp]) {
int c=1;
do {
ip-
-;
if (program[ip]=='[‘) { c-
-
; }
else if (program[ip]==']’) { c++; }
} while (c);
}
} else {
/* do nothing */
}
ip++;
}
return 0;
 }

This adds a layer of abstraction that a cracker or reverse engineer must get
through. First, reverse engineer the intermediate VM language. For those familiar
with the programming language Java, Java runs inside of a VM, called the Java
Chapter 13 ■ Advanced Defensive Techniques 225

virtual machine (JVM). While that was done for portability, not security, it does
add a layer of complexity. There are other languages that run inside of a VM,
but you can also create your own (as with the example).

How Code Virtualization Works

Unlike in the simplistic example, a good virtualizer will create a unique, arbitrary
machine language on the fly, as opposed to using a static or known language.
This makes it more difficult to develop devirtualization tools.
In this case, the program logic is translated to a custom instruction set. As a
result, reversing tools are not immediately applicable because they are unable
to recover/analyze program logic. Then, the VM is compiled to the native
architecture (i.e., x86).
Reversing the application requires both of these:
■■ Reversing the VM to decipher the custom instruction set
■■ Reversing the application logic in the new instruction set
This process is complicated and tedious because your access to debugging is
limited. You cannot debug the target program logic directly, only the VM. Some
tools that aid in accomplishing this include Themida and VMProtect.

Layered Virtualization
Virtualization protections can be layered as in the following process:
■■ Virtual machine VM0 implements the custom instruction set IS0.
■■ IS0 runs the virtual machine VM1, which implements custom instruction
set IS1.
■■ IS1 runs the original application.
An example of layered virtualization may include the following:
1. Compile the C source code to a custom language, such as DazzleZ.
2. Write the DazzleZ interpreter in a custom language such as CatCat.
3. Write the CatCat interpreter in x86.
4. Run the program on the regular x86 platform.
5. Reversing requires backing out all layers of virtualization.

Issues with Virtualization

Virtualization can be an effective tool to slow reversing and cracking. However,
it does have its downsides, including the following:
■■ AV detection: Often, malware will use virtualization to conceal itself, so
 many antivirus programs will automatically flag applications using it.
226 Chapter 13 ■ Advanced Defensive Techniques

■■ File bloat: An application using virtualization needs to have the VM built

in, which increases file sizes.
■■ Slowed execution: Virtualized applications need to run both the VM and
the virtualized code, slowing the application’s execution.
Layering multiple VMs compounds these size and speed issues exponentially.

Is This a Strong Protection?

Evaluating virtualization against the CIA triad has the following results:
■■ Confidentiality: Yes, the original code is abstracted through layers of
virtualization.
■■ Integrity: Yes, modifications to any of the layers will likely cause a ripple
effect of failures, making it difficult to patch.
■■ Availability: Each layer added into this setup has an effect on performance.
Too many layers can dramatically affect speed and availability of code,
data fetches, etc.

Defeating Virtualization
Virtualization can be an effective defense because defeating it is time-
consuming and difficult. In general, the following process can be used to
defeat virtualization:
■■ Reverse the code-dispatch scheme: VMs typically follow the familiar
fetch-decode-execute cycle of a CPU, which makes it possible to under-
stand how code is dispatched.
■■ Reduce complexity: Use pattern matching, symbolic analysis, and similar
techniques to remove unnecessary complexity.
■■ “Devirtualize” the program: Attempt to recover a representation of the
original code. However, this is not always a simple “inverse” for complex
VMs and may not be possible to recover original code, forcing you to
reverse engineer the virtualized code.
■■ Reverse the recovered code: Use traditional tools to reverse the recovered
code. You may need to rely on static analysis if a functioning program
cannot be recovered.
Virtualization can be defeated by reverse engineering the virtual machine
and then transforming the application back into x86 machine code for anal-
ysis. Some tools that aid in accomplishing this include Themida, VMProtect,
and Tigress.
Chapter 13 ■ Advanced Defensive Techniques 227
Cryptors/Decryptors
Cryptors encrypt the application code sections (a subset of the techniques
discussed in earlier section on packers), often to avoid malware detection.
Many anti-malware tools will analyze a piece of software prior to running and
block software based on API calls to suspicious operating system functions. By
encrypting the code section, the malware makes it impossible for anti-malware
programs to inspect the content of the application before execution.
In general, encrypted software must decrypt itself prior to execution. Typically,
this means the decryption key is somewhere within the software. Therefore,
reverse engineering should be able to find the key and decrypt the software.
However, there are some exceptions to this. For example, node-locked soft-
ware may derive a key from the specific system on which it resides. Alterna-
tively, malware may beacon to a server to retrieve a decryption key on the fly.

Is This a Useful Protection?

The benefits that cryptors provide include the following:
■■ Confidentiality: Yes, encryption always adds a layer of confidentiality.
Only under the right circumstances will it decrypt.
■■ Integrity: Some, most encryption algorithms add a layer of integrity pro-
tection here because modifying the encrypted data yields corruption
versus translating to modification of the end code.
■■ Availability: None; this has no effect.

Defeating Cryptors
Most encryptors have supporting decryptors, which are tools that can automat-
ically restore the original software. Often these decryptors are just the encryptor
itself with a different input flag
If you are reversing a crypted application, decrypting will get you back to
the original binary. Since this will be much easier to analyze, see if there is an
available decryptor before you begin your Reverse Engineering. Some common
cryptors to check include Yoda’s Cryptor, Morphine, and PGMP.

Summary
In looking at defense options, there is no silver bullet. Most anti-reversing tech-
niques also have downsides.
228 Chapter 13 ■ Advanced Defensive Techniques

Obfuscation incurs performance impact and complicates legitimate debug-

ging. However, at reasonable levels, it can be a good option for slowing down
 RE, especially of decompilable languages.
Anti-debugging has relatively low impact on RE time (many debuggers have
plugins to circumvent all common anti-debugging tricks) and complicates
legitimate debugging. However, it may be sufficient to thwart novice crackers.
Packers again raise the bar in level of difficulty to reverse engineering and pack,
but, that said, beware of commercial off-the-shelf (COTS) packers, which typi-
cally have corresponding, publicly available unpackers.
Cryptors and decryptors significantly complicate RE. This makes them useful
for protection of software. However, if not used carefully, they can flag common
AV as when used maliciously it can also help protect malware.
When considering if/when to use anti-reversing tools, you should weigh
the trade-offs of potency, resilience, stealth, and cost. Consider your adversary
and their goals:
■■ Competing company (IP theft)
■■ Casual cracker (low-hanging fruit)
■■ Professional cracker (big/high-value targets)
Also, consider what needs to be defended:
■■ Key check? Entire program? Most defenses can be applied to specific
functions.
■■ Consider that adding defenses can call attention to a target.
A common falsehood is that “everything is hackable and can be reverse
engineered if someone tries hard enough, so we shouldn’t bother [protecting/
obfuscating/encrypting/etc.] it.” This is a gross misunderstanding of what
defense is supposed to achieve. Often a cracker may give up hacking, reversing,
cracking, or breaking a product just because it stopped being fun.
If you can slow down reverse engineers enough, you’ve done your job. Often,
a solid design with moderate settings of a commercial-off-the-shelf (COTS)
obfuscator is the best available option. You will need to weigh each approach
based on project needs.
There is no silver bullet. Don’t let perfect be the enemy of good. Once a rea-
sonable approach is settled on, obfuscators, anti-debugging, packers, etc., can
be built into your DevOps.
CHAPTER

14

Detection and Prevention

Application developers use various mechanisms to detect and protect against

reversing and cracking. However, some of these methods are more effective
than others. This chapter explores some of the most common techniques, their
relative strengths and weaknesses, and how they can be defeated.

CRC
A cyclic redundancy check (CRC) is a mathematical calculation performed on
the bytes of the data to be protected. The result is stored as the CRC, which is
often appended to the data (i.e., data data data data data data CRC). To
verify the data, recalculate and compare.
CRC algorithms have their advantages, including the following:
■■ Fast and compact
■■ Easy to accelerate with hardware
■■ Quick to calculate and compare
■■ Numerous options available (IEEE802.3, CRC-32, etc.)
In general, CRCs are great for detecting accidental errors or modification,
such as transmission errors.

229
230 Chapter 14 ■ Detection and Prevention

However, they are a poor defense against intentional errors or modifications.

CRCs can be easily recalculated and updated by an adversary. For example,
a simplistic CRC might add all of the bytes together and save the result. If a
corruption were to occur in the file somewhere in the data, then the new sum
would not match, and action could be taken. If the corruption occurred in the
CRC portion of the file, then the sum would not match the corrupted CRC, and
action could be taken. This is great for detecting if a bit got accidentally flipped
while being downloaded, for example.
But because the CRC is so trivial to recalculate, it’s simple for an attacker to
make their modifications and simply update the CRC to include their new values.

Is This a Strong Protection?

Comparing CRCs to the CIA triad yields disappointing results:
■■ Confidentiality: None
■■ Integrity: Very little (it’s too easy for an attacker to recalculate and put
the new CRC into the file)
■■ Availability: None
This defense can easily be defeated by generating a new, valid CRC. Alterna-
tively, you can simply patch out the CRC check. CRCs are powerful for detecting
accidental corruption but are not useful for intentional corruption.
 Code Signing
Many organizations digitally sign their code before releasing it. This is because
code signing provides two main benefits:
■■ Authenticity: A digital signature can be generated only with the correct
private key. This proves that software came from its alleged creator.
■■ Integrity: Changing digitally signed data invalidates its signature. Code
signing proves that software hasn’t been modified after release.
Code signing protects against a wide range of potential attacks. However,
from a cracker’s perspective, the most significant impact is that it can prevent
patching if a program checks its signature before executing.

How to Code Sign

Code signing works using public key or asymmetric cryptography. These cryp-
tographic algorithms use a pair of public and private keys. To code sign, you
first need to generate a public/private keypair.
Chapter 14 ■ Detection and Prevention 231

Digital signatures are validated using your public key; however, you need a
way to prove that a particular public key belongs to you. This is where public
key infrastructure (PKI) comes into the picture. Using the generated public key,
you apply for a certificate from a code signing certificate authority (CA). The
CA will verify your identity and issue a digital certificate, which contains your
public key and validates your ownership of this.
With this certificate, you can now generate digital signatures. To do so, you
would generate a hash of the executable and encrypt that hash with the private
key. Then, when you distribute the executable, you would bundle the resulting
signature and your digital certificate with the executable.
While you can go through this process manually, many build tools will do
this for you. You still would need to buy a certificate and load it into your build
tool, but then you can ask the build tool to sign the application. If this is your
first exposure to PKI, know that this is intentionally just scratching the surface
of it; there are many books dedicated to just this concept.

How to Verify a Signed Application

A code signature is essentially an encrypted hash of the executable. After verifying
that the public key is valid using the associated certificate, you can decrypt the
executable’s hash. Then, you independently calculate the hash of the applica-
tion using the same hash function as the application developer. If you compare
the two hashes and they match, the application is authentic and unmodified. If
they differ, the application is fake or has been tampered with.
 Most operating systems will verify code signatures for you. The OS will also
generate a warning if the public key used to generate the code signature is
unverified, as shown in Figure 14.1. However, most people will click Run anyway.

Figure 14.1: Windows warning of unverified program

232 Chapter 14 ■ Detection and Prevention

Is Code Signing Effective?

Does code signing stop all patching attacks? No.
The reason for this is that there must be a piece of unsigned code that does
the sign checking. This includes performing a few actions:
1. Calculate the hash of the code.
2. Check if this is what it should be.
2a. If the answer is correct, run code.
2b. If the answer is incorrect, don’t run code.
This signature verification code can’t be included in a code signature because
it needs to contain (or access) the hash value to compare against. It’s impossible
to predict what this value would be without hashing the application. If you
hashed the application (which includes this value) and included the hash in the
application, then the modified application would have a new hash.
Since the signature verification code can’t be signed, there is a different loca-
tion that could be patched to bypass code signing. However, code signing is
hands down one of the best techniques for securing software integrity against
both accidental and intentional modifications.

Code Signing vs. CRC

CRCs are commonly used to detect bit errors in data sent over a network.
However, they provide protection only against accidental changes, not inten-
tional ones. CRCs are easily recalculated by an adversary.
Code signing is as strong as your protection of the private key. Without the
private key, an attacker cannot regenerate a valid signed hash.

Is This a Strong Protection?

Code signing does a lot better than CRCs when compared to the CIA triad.
■■ Confidentiality: None
■■ Integrity: Yes! Fantastic
■■ Availability: None
A more difficult approach to defeating code signing is to steal the private
 signing keys and use them to digitally sign a modified version of the application.

RASP
Runtime application self-protection (RASP) embeds security into the running
application. It does so by intercepting system calls and verifying that they are
Chapter 14 ■ Detection and Prevention 233

from an expected source. It also intercepts data manipulations and verifies that
they are coming from authorized sources.
RASP is a reactionary defense. It can be configured to “stop” attacks live. For
example, RASP can do the following:
■■ Drop/delete a call it deems malicious, such as a suspicious SQL call into
an application.
■■ End a user session.
■■ Halt execution.

Function Hooking
One technique that RASP uses is function hooking. This involves overwriting
the first few bytes of a function’s code with a jump to the RASP code.
The RASP code will include checks to verify that the call is legitimate. This
can include the following:
■■ Checking the parameters and context of the call
■■ Checking the code has not been modified (might compare a hash of the
function with a known good hash)
At the end of the RASP code, it will then execute the overwritten code before
jumping back to the original function.

Risks of RASP
If RASP detects an attack, it can stop execution. However, this may not be accept-
able depending on the use case of your software. For example, in hospitals,
manufacturing, critical infrastructure, automobiles, and similar environments,
an application suddenly halting can pose a significant risk to health and safety.
RASP can also have its downsides even in the absence of an attack. Some
effects include the following:
■■ Speed: Because of the function hooking, RASP has a nontrivial effect
on speed.
■■ Size: Function hooking and lookup tables help to assure security; how-
ever, they also bloat binaries.

Is This a Strong Protection?

RASP provides mixed results when compared against the CIA benchmark:
■■ Confidentiality: No
 ■■ Integrity: Yes (for the sections that RASP is protecting, the context checking
at runtime is a very powerful check)
■■ Availability: No (in fact can be negative)
234 Chapter 14 ■ Detection and Prevention

If RASP is correctly configured, it can be difficult to defeat. You can’t easily

patch the application if code signing is enabled, and you can’t easily reverse
the application if anti-debugging is enabled.
However, RASP does open a potential avenue for an attack against availability.
If you can identify an input that it perceives as an “attack,” you can potentially
make it shut itself down.

Allowlisting
Allowlisting, sometimes called whitelisting, is providing the execution envi-
ronment with a list of “good” things. For example, a computer may allow only
allowlisted applications to run.
There are numerous commercial products that provide allowlists. For example,
the Windows operating system has built-in software restriction policies.
From a cracking perspective, allowlisting can prevent the use of cracking
and reverse engineering tools. For example, tools such as Procmon, debuggers,
Cheat Engine, ResourceHacker, Dependency Walker, and other reversing and
cracking applications are unlikely to be included in the allowlist.
Allowlisting is difficult to get right. It can be difficult to know all of the var-
ious libraries that your application needs. When generating a whitelist, a great
deal of testing must be performed to ensure that all required applications and
libraries are included on the allowlist.

How Allowlisting Works

There are two main approaches to allowlisting. A list can be based on a pro-
cess’s name or on its hash. These lists are applied only when an application is
first launched.

Breaking Name-Based Allowlists

Allowlists keep track of the names of processes or applications allowed to exe-
cute. To bypass this type of list, name your malicious application a whitelist-
approved name. For example, you determine that solitare.exe is allowlisted,
so you name your malicious app solitare.exe.

Breaking Name and Hash-Based Allowlists

 If an allowlist uses both application names and hashes, it can’t be bypassed by
renaming an application. The hash of the malicious app wouldn’t match that
of the legitimate one.
Chapter 14 ■ Detection and Prevention 235

However, these allowlists can be defeated through process injection. Once an

allowlisted application is running, if you can get code execution, you can inject
malicious libraries. And while that’s easily said, getting code execution is not
often trivial. So, this is one of those cases where it sounds easy because it can
be said in a sentence, but in reality having a prerequisite of code execution in
a whitelisted process may be a full roadblock for a cracker.
If you have gotten the elusive code execution in an allowlisted applica-
tion, there are numerous techniques for loading into the process. In Windows,
you can use LoadLibrary() or SetWindowsHookEx(). In Linux, you can use
ptrace()/PTRACE_POKEDATA/opcodes for uselib() syscall.
An application’s hash is checked prior to application launch. Modifications
to the application after it is launched won’t be detected by the allowlist.

Example: Metasploit
Metasploit is a popular hacking tool. Its main goal is to exploit an application
and inject a meterpreter, which provides the attacker with remote access to the
infected computer. (See the “Tools” section of our repository for links.)
With Metasploit, no new applications are started; a meterpreter injects into the
hacked process. From there it can “pivot” into any other running application.

Is This a Strong Protection?

Allowlisting provides limited protection:
■■ Confidentiality: No
■■ Integrity: Yes (if paired with the name and hash; however, the integrity
checking is generally done only at application start time)
■■ Availability: No
Allowlisting can be defeated in a couple of ways. A malicious program can
impersonate a legitimate application to defeat a name-based whitelist or use
code injection to defeat an allowlist that uses both names and hashes.

Blocklisting
Blocklisting, sometimes referred to as blacklisting, is the exact opposite of allow-
listing. Instead of specifying everything that is permitted, it is a list of all the
things that are not allowed. The blocklist can be based on names, keys, or hashes.
Blocklists are easy to make but difficult to maintain. For example, consider
a blocklist including the malicious executable virus1.exe. What happens next
week when virus2.exe comes out?
236 Chapter 14 ■ Detection and Prevention

From a more cracking perspective, you might blocklist keys that you know
to be bad (i.e., cracked). Depending how your key generation works, it may be
possible to blocklist whole subsets of keys.
Alternatively, a program can also refuse to run if certain other applications
are seen. For example, the application may not run if a debugger is installed.
Many antiviruses use this approach to identify and block known malware.
They include a list of “signatures” of known bad applications. If something
matches the signature, it’s flagged as bad.

Is This a Strong Protection?

A blocklist provides less protection than a whitelist:
■■ Confidentiality: None
■■ Integrity: Some (if paired with hashes or keys)
■■ Availability: None
The means of defeating a blocklist depend on the information that it uses to
identify malicious applications. If it is name-based, change the name. If it stores
the hashes of known-bad programs, mutate it by making a small change to the
application’s code or data to change its hash.

Remote Authentication
For most anti-reversing and anti-cracking strategies, the attacker has all of the
pieces that they need to overcome the defense. With enough time, they can
reverse engineer and/or patch the application.
Remote authentication requires the application to retrieve something remotely
in order to work. For example, it might get a key from a remote server that it
uses to decrypt some crucial code.
Most attackers will reverse engineer a system “offline.” They don’t want
it reaching out to your servers because they don’t want you to have their IP
address or to know that they are running your software. Keep in mind when
attempting to crack a piece of software, you’re likely launching and running the
startup and checking code frequently. Whereas a legitimate user would likely
launch the application at max a few times a day. That type of behavior is really
easy to spot on a remote authentication server. A user who is authenticating
100 times a day is likely doing something nefarious.
Architecting the application in such a way that it can’t run without information
from a server helps prevent reverse engineering. The attacker will either need
to reverse it “online” or give up.
Chapter 14 ■ Detection and Prevention 237
Remote Authentication Example
One possible approach to implement remote authentication is to encrypt every
part of the application except for the loaders. The loader sends system information,
a hash of the software, and the activation key to the server.
The server will verify the expected hash and the activation key. If they vali-
date, it uses an algorithm to produce a decryption key, which it will send back to
the application. The loader can then decrypt the application, enabling it to run.
An attacker will not be able to “mimic” your remote server and algorithm
without access to the server-side code. The only way to research the software
will be to activate it online. The application can have decryption code be resident
in memory only. This way, each startup requires server interaction.
The main challenge of this approach is that implementing cryptography
and enterprise key management solutions is not trivial. A mistake may allow
a cracker to bypass the validation code and generate their own decryption
keys. As discussed with packed applications, once it’s unpacked in memory,
you could take a memory dump of it for future static analysis. However, that
memory dump can’t easily (or sometimes not at all) be turned into a decrypted
application capable of running. The memory dump won’t be useful for patching
or testing modifications but never discount the value of static analysis.

Is This a Strong Protection?

Remote authentication provides mixed reviews when compared against the
CIA triad:
■■ Confidentiality: Some (the application will eventually be decrypted in
memory, but the binary at rest is restricted)
■■ Integrity: Yes (the server should be doing some type of integrity checking
prior to releasing the response)
■■ Availability: Possibly negative
One possible attack against a remote server is to set up a fake server. To start,
activate the application online and capture all communication between the
application and server.
Then, stand up a fake server with appropriate responses. The application’s
code will be decrypted and can be saved to disk.
This approach does require a single online application to get the decrypted
code. However, this allows a full, decrypted binary to be created, making further
online authentication unnecessary. But note that this approach can’t always work
if the application does good due diligence in requiring certain certificates from
the server or if the challenge/response from the server is not always the same
(i.e., changes with time or date).
238 Chapter 14 ■ Detection and Prevention
 Lab: ProcMon
This lab shows that there is more than one possible way to crack a program.
Head back to the book’s GitHub page (https://github.com/DazzleCatDuo/
X86-SOFTWARE-REVERSE-ENGINEERING-CRACKING-AND-COUNTER-MEASURES)
and
locate the ProcMon lab.Skills.
This lab uses ProcMon and IDA to understand opportunities for alternative
cracking solutions. Some key skills being tested include the following:
■■ Analyzing program behavior dynamically
■■ Identifying indirect approaches to circumventing software defenses

Takeaways
Watching what a process does from the outside can be quicker/easier than
watching it from the inside (that is, debugging is not always the best approach).
There are usually many ways to crack a program; finding the best takes practice.

Summary
This chapter presented various methods of protecting against software cracking
and reversing. Some techniques are generally ineffective, while others can work
but also have some downsides.
It’s important to remember that almost any defense can be defeated given
enough time and effort. The goal is to slow an attacker down and, ideally, make
them frustrated enough to give up.
CHAPTER

15

Legal

In an “Introducton” the legal implications and considerations of reverse engi-

neering and cracking were explored at a high level. This section provides a more
in-depth discussion of relevant U.S. laws and their impacts and interpretations.

WA R N I N G As a disclaimer, we are not lawyers, and this is not legal advice. If

you need legal advice, please contact any reputable lawyer or the Electronic Frontier
Foundation at www.eff.org, which has deep specialization in the security space.
U.S. Laws Affecting Reverse Engineering
Laws regarding copyrights, hacking, etc., vary based on jurisdiction. This section
covers some applicable laws in the United States. If you are located elsewhere,
check your local laws and restrictions.

The Digital Millennium Copyright Act

Digital rights management (DRM) is a solution designed to protect intellectual
property. DRM solutions can track and control protected content after it reaches
the marketplace.
The Digital Millennium Copyright Act (DMCA) was passed by Congress in
1998. It brought the United States into compliance with international copyright
agreements.
239
240 Chapter 15 ■ Legal

Computer Fraud and Abuse Act

The Computer Fraud and Abuse Act (CFAA) was enacted in 1984. It is a fed-
eral anti-hacking statute that prohibits unauthorized access to computers and
networks.
Lawmakers wrote the law so poorly that creative prosecutors have been abus-
ing it ever since. However, in recent years, efforts have been made to protect
security researchers from being prosecuted under it. Wired.com said this about
the CFAA in a 2014 article:
A high profile case involving misuse of the statute occurred in 2008 when three
MIT students were barred from giving a presentation at the Def Con hacker
conference. The students had found flaws in the electronic ticketing system used by
the Massachusetts Bay Transportation Authority that would have allowed anyone
to obtain free rides. The MBTA sought and obtained a temporary restraining order
to bar the students from speaking about the flaws. In granting the temporary
gag order, the judge invoked the CFAA, saying that information the students
planned to present would provide others with the means to hack the system. The
judge’s words implied that simply talking about hacking was the same as actual
hacking. The ruling was publicly criticized, however, as an unconstitutional
prior restraint of speech, and when the MBTA subsequently sought a court order
to make the restraining order permanent, another judge rejected the request,
ruling in part that the CFAA does not apply to speech and therefore had no
relevance to the case.
www.wired.com/2014/11/
hacker-lexicon-computer-fraud-abuse-act
 A second high-profile, and very sad, abuse of the CFAA resulted in a high-profile
suicide. This came after a U.S. attorney used CFAA to launch a heavy-handed
prosecution against Internet activist Aaron Swartz for what many considered
a minor infraction. Swartz, who helped develop the RSS standard and was
a cofounder of the advocacy group Demand Progress, was indicted after he
gained entry to a closet at MIT and allegedly connected a laptop to the univer-
sity’s network to download millions of academic papers that were distributed
by the JSTOR subscription service. Swartz was accused of repeatedly spoofing
the MAC address of his computer to bypass a block MIT had placed on the
address he used.
Although JSTOR did not pursue a complaint, the Justice Department pushed
forward with prosecuting Swartz. U.S. Attorney Carmen Ortiz insisted that
“stealing is stealing” and that authorities were just upholding the law. Swartz,
in despair over his pending trial and the prospect of a felony conviction, com-
mitted suicide in 2013. In response to the tragedy, two lawmakers proposed a
long-overdue amendment to the law that would help prevent prosecutors from
Chapter 15 ■ Legal 241

overreaching in their use of it. The amendment, referred to as Aaron’s law, was
introduced months after Swartz’s death by Rep. Zoe Lofgren (D-Calif.) and
Sen. Ron Wyden (D-Oregon). The amendment would exclude breaches of terms
of service and user agreements from the law and also narrow the definition
of unauthorized access to make a clear distinction between criminal hacking
activity and simple acts that exceed authorized access on a minor level. Instead,
the amendment proposes to define unauthorized access as “circumventing
one or more technological measures that exclude or prevent unauthorized
individuals from obtaining or altering” information on a protected computer.
The amendment also would make it clear that the act of circumvention would
not include a user simply changing his MAC or IP address to gain access to
a system.

Copyright Act
Under the Copyright Act of 1976, a copyright for a computer program comes
into being as the source code for the computer program is being written by the
programmer. The program does not need to be complete or even functional for
copyright protection to come into being. Copyright case law treats the copyright
of the source code and object code as equivalent.
If you are not the copyright owner, it is typically not legal to perform any of
the following actions without permission:

■■ Copying a program, or parts of programs, to give or sell to someone else

■■ Preloading a program onto the hard disk of a computer being sold
 ■■ Distributing a program over the Internet
■■ Circumventing controls that prevent access to copyrighted material

However, there are many exceptions and nuances to this. The first copyright for
software was in 1964. The justification for why to begin granting protection of
software was they now viewed a computer program like a “how-to book.” The
Copyright Act of 1976 officially calls out software as copyrightable.
So, when a piece of software gets copyright protection, what exactly is copy-
righted? The copyright protects the expression of an idea, not the idea itself.
For example, if you develop the concept of a lemonade stand game, you can
copyright your implementation of it but not the idea of a lemonade stand game.
Second, the protection protects the object (executable) program, not the source
code. Lastly, it protects the screen displays produced by the program while it
executes.
The source code of software is generally kept as a trade secret and not released
under a copyright to the public.
242 Chapter 15 ■ Legal

Important Court Cases

In addition to laws, court precedent is important to determining what is and isn’t
legal in the United States. A couple of important court cases include the following:
■■ Apple Computer v. Franklin Computer: Established that object programs
are copyrightable.
■■ In the early 1980s Franklin Computer corporation started to produce
the Franklin Ace computer to compete with the Apple II. The Franklin
ACE was compatible with Apple 2 programs. To do that, the Franklin
Ace had copied some of the operating system functions directly from
the ROM on an Apple II. Apple sued Franklin Computers for copyright
infringement because they copied their object code. Apple won.
■■ Sega v. Accolate: Established that disassembling object code to determine
technical specifications is fair use.
■■ A video game maker, Accolate, wanted to make some of its games for
the Sega Genesis. However, Sega didn’t share the technical specs for
the system, so Accolade disassembled the object code of a Sega game
to determine how it worked. Sega sued Accolade for infringing on
their copyright. However, this time the court ruled in favor of Accolade,
because Accolade’s actions constituted fair use of the software.
What was gained from these two court cases was that reverse engineering was
OK as long as you didn’t infringe on the copyright. Recall Franklin Computer
infringed on the copyright by copying some of Apple’s code. Where Accolate
did not infringe the copyright, they didn’t copy any copywritten material; they
just learned from it.
One way to make sure you fall in the OK use like Accolate and not in the
 copying situation like Franklin Computers is to use something known as a clean
room software strategy. This consists of having two teams that are separated; each
do different parts of the work. The first would research the competitor system
or program and write technical specifications of how it performs. The second
team would use that specification to develop the new system.
If Franklin Computer had taken this approach, it would have had some team
members figure out how the Apple system worked and describe the function-
ality. Then, if a second team implemented it themselves, never having seen the
Apple implementation, it potentially could have changed the outcome of the
event. If they had approached the situation this way they likely would have
been fine from lawsuits because they would not have used any of Apple’s code.
The key is to avoid the unconscious copying of code. If the team that researched
the Apple II had also been the team to implement the specification, they would
likely have suffered from a predisposition to use code like that in Apple’s system
because that’s what they had seen already.
Chapter 15 ■ Legal 243

Fair Use
Sometimes it is legal to reproduce a copyrighted work without permission. In
general, courts consider four factors when evaluating whether something falls
under the fair use exceptions:
■■ Purpose and how it’s used: If the purpose is criticism, commentary, news
reporting, teaching, or research, then it is likely permissible. However,
commercial use likely isn’t.
How about for character of use? The most important consideration is
how much the work has been transformed from the original. If the new
author has added new expressions or meaning, then it’s potentially a
candidate for fair use.
■■ Nature of work: Fair use is granted more favorably to works of nonfiction
than of works of fiction.
■■ Amount of work being copied: A brief excerpt is more likely to be OK
than copying an entire book or an entire chapter.
■■ Effect on market for copyrighted work: For example, copying out-of-print
material doesn’t have the same material effect as copying a newly written
and printed work.

According to the Copyright Act, 17 U.S.C. § 107, reverse engineering falls

under “fair use” when done for “. . .purposes such as criticism, comment, news
reporting, teaching (including multiple copies for classroom use), scholarship,
or research. . . .” However, this is weighed against “the effect of the use upon
the potential market for or value of the copyrighted work.”
DMCA Research Exception
In October 2016, DMCA added a good-faith security research exception to the
law. It states that “accessing a computer program solely for purposes of good-
faith testing, . . .where such activity is carried out in a controlled environment
designed to avoid any harm to individuals or the public, . . .and is not used or
maintained in a manner that facilitates copyright infringement.”
This also can apply to reverse engineering and cracking. It states “. . .researchers
can circumvent digital access controls, reverse engineer, access, copy, and manipu-
late digital content which is protected by copyright without fear of prosecution—
within reason.”
This is not a get-out-of-jail-free card or a blank permission to go hack and
crack everything. This represents an evolution in the industry to recognize that
security research done for the right reasons is a good thing and that the law will
now protect those who are doing good faith research.
244 Chapter 15 ■ Legal

Legality
Copyright law in relation to reverse engineering and code modification heavily
emphasizes intent and effects. When proceeding on your own, consult with a
lawyer. . .or keep it to yourself. This isn’t meant in a sneaky way, but recall that
a part of fair use is the effect your work has on the market. If you’re tinkering
and cracking for education or for research and your outcomes stay with you,
they don’t really affect the potential market for the work. That’s a key factor in
fair use. The second you use your knowledge to make a keygen that you put
online that causes a vendor to lose money, then it’s no longer considered fair
use. But if you’re keeping it all to yourself in a way that doesn’t affect the market
or others, then you’ve gone a long way to fall under fair use.

WA R N I N G To repeat, we are not lawyers, this is not legal advice, and this is our
interpretation and understanding of the regulatory landscape in the United States that
affects reverse engineering.

Summary
This chapter covered some of the legal considerations of reverse engineering
and cracking, but we are not lawyers. For legal advice, we recommend contact-
ing the EFF.
CHAPTER

16
 Advanced Techniques

Up to this point, this book has covered the core tools and skills used for reverse
engineering and cracking. However, this is an evolving field, and new methods
are being developed to make it faster and easier. This section describes at a high
level some advanced techniques and tools on the cutting edge of reverse engi-
neering. Our goal with this chapter is that if at this point you’re still loving software
cracking and looking to take it even further to the next level, we want to present
you with a plethora of rabbit holes to go down. Depending on what interests
you, we hope the following will point you in the right directions to go deeper.

Timeless Debugging
Timeless debugging is also known as reverse debugging. The core idea is: “what
if we could go backwards when debugging?”
Consider the case where something went wrong while debugging. Maybe
a patch failed, you missed an anti-debug check, you don’t know how you got
here, etc.
There are a few different tools designed for timeless debugging, including
the following:
■■ Visual Studio Ultimate (.NET)
■■ rr
■■ gdb
245
246 Chapter 16 ■ Advanced Techniques

To get started, check out George Hotz @ Enigma in his 2016 USENIX Enigma
talk at www.youtube.com/watch?v=eGl6kpSajag.

Binary Instrumentation
Binary instrumentation is when you inject code to watch or modify a process
as it executes. This can be useful for finding memory leaks, tracing key checks,
performing anti-anti-debugging, etc.
Some tools for binary instrumentation include the following:
■■ PIN
■■ DynamoRIO
■■ Frida
■■ Valgrind
 ■■ QBDI

For an introduction to binary instrumentation, check out the 2015 Blackhat

USA talk “Augmenting Static Analysis Using Pintool: Ablation” at www.youtube
.com/watch?v=wHIlNRK_HiQ.

Intermediate Representations
Normally, for reversing and cracking, it’s necessary to learn and write tools for
each new architecture. The idea of intermediate representations is to translate
all assembly code for all architectures to the same language. That way, you can
learn and write tools for just that language.
There are a few different tools that can be used to work with intermediate
representations, including the following:
■■ Binary Ninja
■■ REIL
■■ VEX
■■ BNIL
■■ Ghidra PCode
■■ IDA microcode
■■ LLVM IR

To get started with intermediate representations, check out “Finding Bugs

with Binary Ninja” by Jordan Wiens from LevelUp 0x03 at www.youtube.com/
watch?v=55gClG-sjWc.
Chapter 16 ■ Advanced Techniques 247

Decompiling
The idea of decompiling is to recover original source code from advanced
automated analysis of assembly code. Some tools that offer decompilation
include the following:
■■ IDA’s Hex-Rays
■■ Ghidra
■■ Binary Ninja
■■ Snowman Decompiler

To learn more about decompiling, check out “Decompiling a Virus using IDA
Pro” at www.youtube.com/watch?v=gYkDcUO9otQ.

Automatic Structure Recovery

Automatic structure recovery involves automatically finding patterns and links
in memory to make inferences about the data types used. Some tools for this
include the following:
■■ dynStruct
■■ Cheat Engine

To learn more about automatic structure recovery, check out the dynStruct
idea and paper at https://github.com/ampotos/dynStruct.

Visualization
Code listings and text can be difficult to think and reason about. Visualization
can be used to deepen your understanding of file structure and execution.
Some reversing tools that offer useful visualizations include the following:
■■ BinWalk
■■ Hopper
■■ IDA plugins
■■ Veles
■■ ..cantor.dust..
■■ Cheat Engine

A good starting point for understanding how visualization can be used for
reversing includes the Derbycon talk “Dynamic Binary Visualization” from
Christopher Domas at www.youtube.com/watch?v=4bM3Gut1hIk.
248 Chapter 16 ■ Advanced Techniques

Deobfuscation
Obfuscation is designed to slow down reversing in an attempt to get a cracker
to give up. The idea is to use tools to automatically remove obfuscations from
programs using tools like Tigress Protection.
Check out “Lets break modern binary code obfuscation” at www.youtube
.com/watch?v=TDnAkm6ZTYw.

Theorem Provers
Theorem provers use mathematics to analyze code, including reduction, deob-
fuscation, boundaries, inputs, etc. Some theorem proving tools for reversing
include the following:
■■ Z3
■■ STP
■■ Boolector
■■ Yices
To see how theorem provers can be used, watch “Using z3 to find a password
and reverse obfuscated JavaScript” at www.youtube.com/watch?v=TpdDq56KH1I.
 Also check out the yearly SMT-COMP!, which has some really interesting
benchmarks on many unique solvers at https://smt-comp.github.io/2023.

Symbolic Analysis
The idea behind symbolic analysis is trying to find inputs that cause interesting
results. For example, what inputs could cause a crash, pass a key check, unlock
a secret, etc.
Symbolic analysis tools will trace user input through a program. At each
branch, they ask a theorem prover which user input would go down the taken
path. What user input would go down the not-taken path?
For example, consider the following code:
if (strlen(username) > 10)
if (key_1^sum(username)==key_2)
printf("key passed");

A symbolic analysis tool will automatically identify the combination of

username, key_1, and key_2 that will pass the checks and reach the “key passed”
print statement.
Chapter 16 ■ Advanced Techniques 249

Some symbolic analysis tools include the following:

■■ Angr
■■ Mayhem
■■ KLEE
■■ Triton
■■ S2E
To see an example of symbolic analysis with Angr, check out the DEF CON
23 talk by Shoshitaishvili and Wang, “Angry Hacking: The next gen of binary
analysis,” at www.youtube.com/watch?v=oznsT-ptAbk.

Summary
At this point, the best way to improve your reversing and cracking skills is
via more hands-on practice. On the Windows VM, the allthethings folder on
the Desktop contains a variety of different crackmes to practice with sorted by
difficulty level.
CHAPTER

17
 Bonus Topics

This last chapter of this book introduces software reversing and cracking. It is
primarily focused on understanding how a program works and bypassing or
modifying undesirable functionality (like key checkers).
This chapter takes this knowledge and applies it to real-world hacking. Stack
smashing and shellcoding both use an understanding of how a program and
the stack works to run malicious code within a program.

Stack Smashing
Stack smashing, also known as stack-based buffer overflows, is one of the most classic
attacks against software. It takes advantage of the fact that non-memory-safe
languages such as C/C++ have no built-in protection that prevents an applica-
tion from accessing or overwriting data in other parts of memory. For example,
C/C++ doesn’t automatically check that the data written to an array fits within
the bounds of that array. If you don’t know C, don’t worry. As long as you know
any programming language, you should be able to follow along.
Because stack smashing has been around for such a long time, there are
numerous compilers that have built-in automatic guards that are put into com-
piled code to prevent this. While it’s not as easy of an attack as it used to be,
everyone should fully understand how the attack works, because:

251
252 Chapter 17 ■ Bonus Topics

■■ Some facets of it still work.

■■ It’s the foundation of other types of attacks.
■■ Not every application has stack protections.
For any of the following C code examples, if you build them with gcc ,
you must use the flag -fno-stack-protector to turn off these protections.
Making the full command line for using gcc to build in Linux: gcc myfile
.c-fno-stack-protector.
For example, consider the following simple C program:
void function(int a, int b, int c) {
char buffer1[5];
char buffer2[10];
}
 void main() {
function(1,2,3);
}

After this application has been compiled and the object has been dumped
from memory, it results in the following assembly code:
function:
push ebp
mov esp, ebp
sub ebp, 20 (*stack shown here)
leave
ret
main:
push ebp
mov ebp, esp
push 3
push 2
push 1
call function
add esp 0xc
leave
ret

After executing the first three instructions under function , including

sub ebp, 20, the stack will look like the following table with addresses increasing
from the top of the table going down:

NAME SIZE
buffer2 10
buffer1 5
ebp 4
Chapter 17 ■ Bonus Topics 253

NAME SIZE
ret 4
a 4
b 4
c 4
ebp 4

Now, consider the following example code:

 void function(char *str) {
char buffer[16];

strcpy(buffer,str); //Copies incoming str to buffer

}

void main() {
char large_string[256];
int i;

for( i = 0; i < 255; i++)

large_string[i] = 'A’; //creates a string of 255 ‘A’s

function(large_string);
}

In this code, the main function builds a string that consists of 255 As. It then
passes a pointer to that buffer to function, and function allocates 16 bytes for
a local buffer but then copies (using strcpy) the input buffer blindly with no
length checks. This means the input buffer that was 255 As will overflow the
local buffer that was allocated only 16 bytes.
If you run the code, the result will be Segmentation fault (core dumped).
A segmentation fault occurs when an application attempts to read, write, or exe-
cute an invalid memory address. Let’s dig deeper to figure out what happened.
After assembly, the code is transformed into the following assembly code:
0804840c <function>:
804840c: 55 push ebp
804840d: 89 e5 mov ebp,esp
804840f: 83 ec 28 sub esp,0x28
8048412: 8b 45 08 mov eax,DWORD PTR [ebp+0x8]
8048415: 89 44 24 04 mov DWORD PTR [esp+0x4],eax
8048419: 8d 45 e8 lea eax,[ebp-
0x18] ;[1]
804841c: 89 04 24 mov DWORD PTR [esp],eax
804841f: e8 cc fe ff ff call 80482f0 <strcpy@plt>
8048424: c9 leave
8048425: c3 ret
254 Chapter 17 ■ Bonus Topics

Looking at this, you can see that ebp-0x18 is the address at the start of the
buffer (marked as [1] in the previous code). Looking at the function preamble,
with the stack setup, you can see that 0x28 bytes were allocated for the stack.
 Recall that ebp points to the bottom of the stack and esp the top. Therefore,
ebp = esp+0x28.
So, at the time of function setup, the start of the array, in terms relative to esp,
starts at esp+0x10. While this seems complicated, all it means is that the buffer
is 0x10 bytes away from the end of the function’s allocated stack, which makes
sense. Recall that 0x10 is 16 in base 10, and the function is allocated 16 bytes.
To see the effects of the stack smashing in action, run the application in gdb and
set a breakpoint right before the strcpy operation. At the breakpoint, printing
memory at the stack pointer should show something similar to Figure 17.1.

Figure 17.1: Function stack frame before strcpy

In this image, the allocated buffer takes up the row indicated by address
0xffffd130, and 0x10 bytes after that is the end of the function’s stack frame.
That is then followed by the saved value of the previous stacks ebp, and lastly
the return address. The value of the saved ebp (previous functions stack frame)
register is 0xffffd278, and the return address is 0x08048470.
After stepping over the strcpy operation, the same region of memory will
look like Figure 17.2. The strcpy operation overwrites the buffer, the saved ebp
register, and the return address with 0x41 (A).

Figure 17.2: Function stack after strcpy

When the application reaches the ret operation, it will pop the return address
off of the stack and attempt to continue execution at that location. However,
since 0x41414141 is an invalid address, the CPU segfaults.
This example causes the application to crash, but this is not the only possible
effect. At a high level, what you have the ability to do is control the return address
and the stack frame of the previous function. While stack frame manipulation
has its uses, it’s a lot more common to go after the return address manipula-
tion, so we’ll focus on that. In the first case, the return address was overwritten
with junk, but what if we were more tactical about what we overwrite with
Chapter 17 ■ Bonus Topics 255

the return address? The following code sample is designed to alter the return
address to control code execution. The goal is to skip over the x=1 instruction
in the following code:
#include <stdio.h>
void function(int a, int b, int c) {
//do something so we skip x=1 after a return
}
void main() {
int x;
x = 0;
function(1,2,3);
x = 1;
printf("%d\n",x);
}

In this code, the main function sets up a local variable x and gives it an initial
value of 0. It then calls function with some fixed values. Inside of function,
there is no code yet. The next step is to figure out what code is needed there to
achieve the goal of rewriting the return address.
After returning from function, the main function updates the value of x to be
1 and then proceeds to print the value of x. Can we use our knowledge of cdecl
and the stack setup to make it so the code never runs x=1 and instead prints
x=0? Yes! The challenge is to write the contents of function in such a way that
the x=1 instruction inside of the main function is skipped.
For this code, the stack inside of Function would look like the following:

NAME ADDR
ebp ebp
return address ebp+4
a ebp+8
b ebp+12
c ebp+16

This is your run-of-the-mill standard cdecl stack setup. You know you’re
going to want a buffer since this chapter is all about buffer overflows, so add
a buffer to function. You’re also going to want a way to manipulate certain
values in the buffer, so add a pointer. You could also use syntax like buffer[z],
but the pointer helps to more explicitly state memory offsets, which is helpful
for learning.
#include <stdio.h>
void function(int a, int b, int c) {
char buffer[16];
256 Chapter 17 ■ Bonus Topics

int *r;
 r = 0x99; //this is here so r is not optimized out
buffer[0] = 0x88; //this is here so buffer is not optimized out
}
void main() {
int x;
x = 0;
function(1,2,3);
x = 1;
printf("%d\n",x);
}

When assembled, this translates to the following assembly code:

0804840c <function>:
804840c: 55 push ebp
804840d: 89 e5 mov ebp,esp
804840f: 83 ec 20 sub esp,0x20
8048412: c7 45 fc 99 00 00 00 mov DWORD PTR [ebp-
0x4],0x99
8048419: c6 45 ec 88 mov BYTE PTR [ebp-
0x14],0x88
804841d: c9 leave
804841e: c3 ret

Now there are new things on the stack, the pointer and the buffer.

NAME ADDR
buffer ebp-0x14
r ebp-4
ebp ebp
return address ebp+4
a ebp+8
b ebp+12
c ebp+16

In this stack frame, the return address is at buffer+0x18. The next step is to
update function’s code to have the pointer point to this address in memory.
For those not familiar with C, & is “address of,” so the following code sets
ret to point to the address in memory where buffer+0x18 is. By drawing out
the stack, you can see that this is the saved return address. At this point, the
return address hasn’t been changed, but we have a pointer to it. The next step
is to figure out what to change it to, to skip x=1.
#include <stdio.h>
 void function(int a, int b, int c) {
Chapter 17 ■ Bonus Topics 257

char buffer[16];
int *ret;

//now we have the return value, what do we do with it?

ret = (unsigned int)&buffer+0x18;
buffer[0] = 0x88; //this is here so buffer is not optimized out
}
void main() {
int x;
x = 0;
function(1,2,3);
x = 1;
printf("%d\n",x);
}

To figure out how to manipulate the return address, take a look at the assem-
bled code for main:
0804841f <main>:
804841f: 55 push ebp
8048420: 89 e5 mov ebp,esp
8048422: 83 e4 f0 and esp,0xfffffff0
8048425: 83 ec 20 sub esp,0x20
8048428: c7 44 24 1c 00 00 00 mov DWORD PTR [esp+0x1c],0x0
8048430: c7 44 24 08 03 00 00 mov DWORD PTR [esp+0x8],0x3
8048438: c7 44 24 04 02 00 00 mov DWORD PTR [esp+0x4],0x2
8048440: c7 04 24 01 00 00 00 mov DWORD PTR [esp],0x1
8048447: e8 c0 ff ff ff call 804840c <function>
804844c: c7 44 24 1c 01 00 00 mov DWORD PTR [esp+0x1c],0x1;x=1
8048454: 8b 44 24 1c mov eax,DWORD PTR [esp+0x1c]
8048458: 89 44 24 04 mov DWORD PTR [esp+0x4],eax
804845c: c7 04 24 08 85 04 08 mov DWORD PTR [esp],0x8048508
8048463: e8 88 fe ff ff call 80482f0 <printf@plt>
8048468: c9 leave
8048469: c3 ret

Normally, the return address of the function would be 0x804844C, and, looking
at that instruction, that is the x=1 that we want to avoid! After this line, the next
instruction starts at 0x8048454.
Now, there are two options for changing the return address. One is to use
the pointer to the return address to change it to be the hard-coded 0x8048454.
The problem with this approach is that the address is a virtual address chosen
at build time by the compiler, and every time you launch it, it will be the same,
until you recompile. When you recompile, there is a chance you will get new
virtual addresses. You’d need to recompile to test this theory, so this approach
is a bit rigid.
258 Chapter 17 ■ Bonus Topics

Instead, the better approach is to note that the x=1 instruction is 8 bytes long.
That will always be consistent, so the stronger approach is to add 8 bytes to the
current return address.

N OT E When printing out assembly, gdb will often cut off the hex display, so if
you’re looking at the printout, you’ll count only 7 bytes on the x=1 line. That is simply
because it was cut off. Always do the math with the addresses to make sure you have
the right byte count.

To skip the x=1 instruction, the return address should be updated by adding
8 bytes. Adding that into the code produces the following:
#include <stdio.h>
void function(int a, int b, int c) {
char buffer[16];
int *ret;

ret = (unsigned int)buffer+0x18; //get the return value

*ret +=0x8; //increment the return value by 8
buffer[0] = 0x88; //this is here so buffer is not optimized out
}
void main() {
int x;
x = 0;
function(1,2,3);
x = 1;
printf("%d\n",x);
}

Running this code (with the compile flag -fno-stack-protector) should result
in the program printing out a value of 0. This indicates that the return address was
successfully modified and the program skips over the x=1 instruction. Victory!

Shellcode
The ability to modify return addresses provides control over code execution,
 which is powerful. One common application of this is to “pop a shell,” providing
the ability to run more powerful commands.
To pop a shell, you need to be able to run your own, arbitrary code within
the application. To do so, you need to place shellcode within the buffer that is
being overflowed and modify the return address to point to the beginning of this
code. Shellcode quite literally means code that will launch a command prompt
(shell). The shellcode can come before or after the return address depending on
the amount of buffer space you have available. The goal is to get your shellcode
into a buffer somewhere and then modify the return address to point to it.
Chapter 17 ■ Bonus Topics 259

The following code shows a very simple shellcode. It uses the execve Linux
syscall to execute /bin/sh, which is a common shell application. execve is asking
the Linux kernel to do something. In this case, passing in the shell application
asks Linux to launch the shell.
#include <stdio.h>

void main() {
char *name[2];

name[0] = "/bin/sh";
name[1] = NULL;
execve(name[0], name, NULL);
exit(0);
}

This simple shellcode assembles to the following assembly code:

0804843c <main>:
804843c: 55 push ebp
804843d: 89 e5 mov ebp,esp
804843f: 83 e4 f0 and esp,0xfffffff0
8048442: 83 ec 20 sub esp,0x20
8048445: c7 44 24 18 18 85 04 mov DWORD PTR [esp+0x18],
0x8048518
804844c: 08
804844d: c7 44 24 1c 00 00 00 mov DWORD PTR [esp+0x1c],0x0
8048454: 00
8048455: 8b 44 24 18 mov eax,DWORD PTR [esp+0x18]
8048459: c7 44 24 08 00 00 00 mov DWORD PTR [esp+0x8],0x0
8048460: 00
8048461: 8d 54 24 18 lea edx,[esp+0x18]
8048465: 89 54 24 04 mov DWORD PTR [esp+0x4],edx
8048469: 89 04 24 mov DWORD PTR [esp],eax
 804846c: e8 cf fe ff ff call 8048340 <execve@plt>
8048471: c7 04 24 00 00 00 00 mov DWORD PTR [esp],0x0
8048478: e8 a3 fe ff ff call 8048320 <exit@plt>

This code relies on standard C methods for execve and exit, which will move
around in memory, making it difficult to predict their addresses and embed
them in the code. Meaning that if you took this assembly code as is, dropped
the opcodes into a buffer, and updated the return address to point to it, when
the code reaches the call execve instruction, it would likely segfault. This is
because the address compiled into the shellcode is where execve was loaded
for that application (0x8048340), but that is not a universal address. You would
need to know where execve is loaded for the target application (if it even has
execve at all). This makes it necessary to find an alternative way of popping a
shell that doesn’t involve C libraries.
260 Chapter 17 ■ Bonus Topics

If you disassemble the execve and exit methods, you can see the underlying
implementation, as shown in the following code sample:
mov eax, 0xb
mov ebx, string_addr
lea ecx, string_addr
lea edx, null_string
int 0x80 ;sys call for exec
mov eax, 0x1
mov ebx, 0x0
int 0x80 ;sys call for exit
“:/bin/sh”\0

So that solves some of the struggle, and the C library calls distill down into
the int 0x80 syscalls covered earlier in the book. But now there is another
challenge: the values of string_addr and null_string are unknown since
you can’t predict where they will be loaded in memory. Again, the assembled
shellcode placed them in that local memory space (in this example 0x8048518
is the compiled address for /bin/sh), but when the shellcode is dropped into
the target buffer, those addresses will be wrong.
Making the shellcode work requires figuring out another way to find the
address that is relative and not hard-coded. One way to learn this value is to
take advantage of return addresses in function calls; again, apply your immense
knowledge of calling conventions and the stack! If a function call is placed right
before the string, then the address of the string will be at the top of the stack
within that function (because the string is sitting at the function’s return address).
To start, add in a few place holders to the existing shellcode.
 jmp ??
pop esi
mov [esi+0x8],esi
mov [esi+0x7],0x0
mov [esi+0xc],0x0
mov eax, 0xb
mov ebx, esi
lea ecx, [esi+0x8]
lea edx, [0xc+esi]
int 0x80
mov eax, 0x1
mov ebx, 0
int 0x80
call ??
.string \"/bin/sh\"

This code sample takes the initial shellcode and adds two instructions to
the front and two to the end. The next step is to determine the address of the
string, which is located at the end of the assembly block. Ideally, the initial jmp
instruction should jump down to the new call at the bottom.
Chapter 17 ■ Bonus Topics 261

Then, this call should call the new pop esi line. Why? When using a call
(instead of a jump) to get back up to the top of the code, the return address
(the next address after the call) will be placed on the stack. We have no inten-
tion of doing a normal cdecl stack setup; this is abusing x86 knowledge to do
naughty things.
After the call back up to pop esi, the top of the stack will have the return
address, which in this case is the shell string. This address can be popped off
the stack into esi and used in the previous shellcode.
Now, that sounds awesome, but there are currently placeholders for the jump
and call. To figure out where those are going to jump to, we have to count our
bytes. Here we count the compiled bytes to determine the correct offsets for
jmp and call:

jmp 0x26 # 2 bytes

pop esi # 1 byte
mov [esi+0x8],esi # 3 bytes
mov [esi+0x7],0x0 # 4 bytes
mov [esi+0xc],0x0 # 7 bytes
mov eax, 0xb # 5 bytes
mov ebx, esi # 2 bytes
 lea ecx, [esi+0x8] # 3 bytes
lea edx, [0xc+esi] # 3 bytes
int 0x80 # 2 bytes
mov eax, 0x1 # 5 bytes
mov ebx, 0 # 5 bytes
int 0x80 # 2 bytes
call -0x2b # 5 bytes
.string \"/bin/sh\"

This modified code solves the problem of finding the string in memory by
making it all relative (no hard-coded addresses) and uses the fundamental work-
ings of x86 to help. The final challenge is getting the code to run, which requires
placing a binary representation of the code on the stack via a buffer overflow.

Stack Smashing and Stack Protection

As mentioned, by default many compilers now build in stack protections to pre-
vent rudimentary stack attacks. As an example, gcc and g++ after gcc 4.1 have
some built-in stack protection. To practice stack smashing, it’s necessary to
build executables using the -fno-stack-protector flag. So, what does stack
protection look like? Let’s build an example and see what it adds.
The following code sample shows a program built with stack protection
enabled:
0804845c <function>:
804845c: 55 push ebp
804845d: 89 e5 mov ebp,esp
262 Chapter 17 ■ Bonus Topics

804845f: 83 ec 48 sub esp,0x48

8048462: 8b 45 08 mov eax,DWORD PTR [ebp+0x8]
8048465: 89 45 d4 mov DWORD PTR [ebp-
0x2c],eax
8048468: 65 a1 14 00 00 00 mov eax,gs:0x14
804846e: 89 45 f4 mov DWORD PTR [ebp-0xc],eax
8048471: 31 c0 xor eax,eax
8048473: 8b 45 d4 mov eax,DWORD PTR [ebp-
0x2c]
8048476: 89 44 24 04 mov DWORD PTR [esp+0x4],eax
804847a: 8d 45 e4 lea eax,[ebp-
0x1c]
804847d: 89 04 24 mov DWORD PTR [esp],eax
8048480: e8 bb fe ff ff call 8048340 <strcpy@plt>
8048485: 8b 45 f4 mov eax,DWORD PTR [ebp-0xc]
 8048488: 65 33 05 14 00 00 00 xor eax,DWORD PTR gs:0x14
804848f: 74 05 je 8048496 <function+0x3a>
8048491: e8 9a fe ff ff call 8048330 <__stack_chk
_fail@plt>
8048496: c9 leave
8048497: c3 ret

The bolded lines illustrate the things added by the compiler for stack pro-
tection. The compiler added code that will save the return address on function
entry and will verify that it is unchanged after a strcpy operation. The com-
piler knows calls like strcpy can be dangerous; this prevents the strcpy from
overwriting the return address.
There are a few options for protecting against stack smashing, including
gcc’s built-in stack protections, the use of memory-safe languages with bounds
checking, and Data Execution Prevention (DEP). However, buffer overflows are
still a threat in some cases because not all compilers will support stack protec-
tion or DEP, and as you can see, there is nuance to how it protects, not adding
stack guards around every single call. Yet protections are focused against specific
things like strcpy. And many compilers are pretty smart about which are most
dangerous and need protection.

Connecting C and x86

Any program that can be written in C (or any other language) can also be written
in assembly. In fact, higher-level languages are compiled into assembly before
they are run by the CPU. However, in some cases, it can be helpful to mix C and
assembly code. If you’re writing your own exploits/cracks, this is a powerful
combination. Some things are nuanced enough that you need assembly-level
control, and some things are just code that needs to happen, and it’s faster to
write it in C, so feel free to mix the two!
To call a function written in another language, it’s necessary to know where
that function is located in memory. The linker can provide this information
automatically.
Chapter 17 ■ Bonus Topics 263

It’s also necessary to know how to pass information to that function, i.e.,
its calling convention. For this case we will assume our C functions are using
cdecl. Recall the following, with cdecl:

■■ Arguments are passed on the stack pushed from right to left.

■■ The caller is responsible for cleaning up the stack after the call returns.
■■ The function’s return value is stored in eax.
 ■■ The eax, ecx, and edx registers are available to the callee. The caller should
save these registers’ values if needed, and the callee should save and
restore the values of any other registers that they need.
If you follow the correct calling convention, you can call C functions from
your assembly code.

Using C Functions in x86 Code

For x86 code to use C functions, the assembly code needs to know that the C
function is defined elsewhere. This is done using the extern directive in the
assembly code. For example, to call the C function, x(), in x86, use the follow-
ing instructions:
extern x
call x

The first step to using C functions in assembly is to include the

extern function_name directive at the top of the assembly file. This tells the
assembler that you intend to use this function, but you don’t know its location
(address) yet. When you write call function_name in the assembly code, it will
initially be assembled as call 0x????????. However, the program won’t be able
to run until you put it through a linker, which will fill in the appropriate address.
The next step is to call the desired function using the cdecl calling convention.
For example, when calling the C function int add(int x, int y), you’d use the
following assembly code. Remember, arguments are pushed from right to left,
and you need to clean up the stack after the call and place the return value in eax.
push [y]
push [x]
call add
add esp, 8
mov [sum], eax

After writing the assembly code, the next step is to assemble it using nasm.
Here’s an example: nasm example.asm –o example.o.
At this point, everything will be in assembly except those placeholders. If you
had no external functions, your code would be ready to run, but since it does,
264 Chapter 17 ■ Bonus Topics

you need a linker’s help. The final step is to link your assembly code to the C
function. If you’re using gcc and calling functions from the C library, gcc can
handle this automatically. For example, gcc example.o -o example will use
the linker to fill out any addresses that it knows, transforming call 0x????????
to call 0x08048320.
For example, consider the following example, which runs
printf hello world 42:

extern printf
global main

section .text
main:

push 42
push world
push hello
call printf
add esp, 12

mov eax, 1
mov ebx, 0
int 0x80

section .data
hello: db "hello %s %d", 0xa, 0
world: db "world"

This assembly code can be assembled using nasm –f elf example.asm and
linked with gcc –m32 example.o –o example.
It can be very helpful and powerful to be able to call simple things like printf
from your assembly code while you’re testing your crack/patch ideas.

Using x86 Functions in C Code

It’s also possible to call assembly functions from C code. The C program must
have a prototype for the x86 functions that it wants to use. For example, in C, to
use the assembly function f, you need the prototype int f(void);. A prototype
is a fancy way of saying that you need to declare how that function definition
would look if it was in a higher-level language (what’s its name, what argu-
ments does it take, and what does it return).
To use x86 functions in your C code, they need to be exported from your
assembly code so that the linker can find them. To export an x86 function in
your assembly file, label it with the global directive, as shown in the following
example:
global f
f:
 Chapter 17 ■ Bonus Topics 265

mov eax, 0xdabbad00

ret

Then, assemble your assembly code with nasm and compile and link the
complete program with gcc.
For example, consider the following C program:
// x.c

#include <stdio.h>

int add(int,int);

int main(void)
{
int x=add(1,2);
printf("%d\n",x);
return 0;
}

This program uses the add function, which is defined in the following assem-
bly code:
; y.asm

add:
push ebp
mov ebp, esp

mov eax, [ebp+8]

add eax, [ebp+12]

leave
ret

To link and assemble this program, run the following commands:

nasm –f elf y.asm # produces y.o object
gcc –m32 –c x.c # produces x.o object
gcc x.o y.o –o adder # produces executable adder
# run with ./adder
_start vs. main()
x86 assembly programs commonly begin with a label named _start. C pro-
grams, on the other hand, start with a main() function. What’s the difference?
266 Chapter 17 ■ Bonus Topics

Execution of a program (whether written in C, assembly, or any other lan-

guage) doesn’t really start at main. For example, consider the simplest possible
C function, as shown here:
int main()
{
return 0;
}

Compiling this with gcc simple.c –o simple translates your program to

assembly. As part of this process, the compiler adds a function called _start,
and _start calls main.
The resulting compiled main function has the following assembly:
80483b4: 55 push ebp
80483b5: 89 e5 mov ebp,esp
80483b7: 5d pop ebp
80483b8: c3 ret

The start function looks like this:

8048300: 31 ed xor ebp,ebp
8048302: 5e pop esi
8048303: 89 e1 mov ecx,esp
8048305: 83 e4 f0 and esp,0xfffffff0
8048308: 50 push eax
8048309: 54 push esp
804830a: 52 push edx
804830b: 68 30 84 04 08 push 0x8048430
8048310: 68 c0 83 04 08 push 0x80483c0
8048315: 51 push ecx
8048316: 56 push esi
8048317: 68 b4 83 04 08 push 0x80483b4
804831c: e8 cf ff ff ff call 80482f0 <__libc_start_main@plt>
8048321: f4 hlt

The start function is responsible for a few different tasks, including the
 following:
■■ Initializing the frame pointer
■■ Configuring the stack
■■ Setting up the standard arguments (parameters to main())
■■ Calling libc_start_main, which performs security checks, threading
subsystem, init, calls your main function, and finally calls exit()
When writing pure assembly code, you write everything yourself. You don’t
need all of the setup C does and can write your own _start function.
Chapter 17 ■ Bonus Topics 267

When combining assembly and C, you need gcc to step in. Often, gcc wants to
provide its own _start function and expects you to provide a main() function.
When writing an assembly program that will be linked against the standard
C library, do the following:
1. Use main instead of _start (libc_start_main will call main() for you).
2. Set up a stack frame only, not the entire stack (_start has already config-
ured your stack).
3. Finish with ret, not int 0x80 (ret will return to libc_start_main, which
will call the C exit function, which will call int 0x80 for you).
4. Set the return value in eax before ret’ing (usually 0).
For example, consider the following stand-alone assembly program, which
defines its own _start:
global _start

section .text
_start:
mov esp, stack
mov ebp, esp

...

mov esp, ebp

mov eax, 1
mov ebx, 0
int 0x80

section .data
times 128 db 0
stack equ $-
4
When linking to libc, the program should use main instead.
global main

section .text
main:
push ebp
mov ebp, esp

...

mov eax, 0
leave
ret
268 Chapter 17 ■ Bonus Topics

Standard Arguments
In C, arguments can be read from the command link with stdargs. For example,
main() is commonly defined as int main(int argc, char **argv), which pro-
vides access to these command-line arguments. Recall that argc is the number
of arguments passed in, and argv is an array that holds those arguments.
It’s also possible to access command-line arguments when writing a main
function in assembly. Your assembly version of main will be automatically called
with cdecl. Recall that the following:
■■ Arguments are passed on the stack, pushed on from right to left.
■■ Arguments are at [ebp+8], [ebp+12], etc.
■■ argc will be the last argument and is at the top of the list of arguments
on the stack, at [ebp+8].
■■ argv is the first argument pushed to the stack and will be at [ebp+12].

For example, the following assembly program will print the first command-
line argument:
extern printf
global main

main:
push ebp
mov ebp, esp

mov eax, [ebp+12] ; load argv into eax

push dword [eax+4] ; push argv[1]
call printf ; print argv[1]
add esp, 4 ; clean up stack
mov eax, 0
 leave
ret

Mixing C and Assembly

In C, it’s possible to switch seamlessly between C and assembly code. This is
called inline assembly, named for the fact that the assembly is inlined with your
source code.
Inline assembly is not part of the C specification, but most compilers will
support it via an extension. However, the syntax is unique for each compiler.
In gcc, this is the AT&T x86 syntax.
The basic form of this is __asm__ (“assembly code here”);. When com-
piling, gcc compiles the C code to assembly and pastes in the assembly code
from the __asm__ directive.
Chapter 17 ■ Bonus Topics 269

For example, consider the following C program:

int main(void)
{
// set keyboard control register

__asm__ ("mov $0x10010001, %eax");

__asm__ ("out %eax, $0x64");

return 0;
}

The extended form of inline assembly lets you set advanced “constraints.”
These constraints can include the following:
■■ Input variables: C variables that you want to manipulate using assembly.
■■ Output variables: Values produced in the inline assembly code that you
want to use in the C code.
■■ Clobbered registers: gcc translates the C to assembly and figures out
which registers to use. This list ensures that the registers used by the C
and assembly code won’t conflict.
Extended assembly can be specified as follows:
__asm__(
“assembly”
: input constraints
: output constraints
: clobber list
 );

The following code sample shows an example of using extended assembly in C:

#include <stdio.h>

int main(void)
{
// getting the return address for the current function

int x;

__asm__("\
movl 0x4(%%ebp), %%eax \n\
movl %%eax, %0 \n\
"
:"=r"(x)
:
:"%eax"
270 Chapter 17 ■ Bonus Topics

);

printf("%08x\n", x);

return 0;
}

Inline assembly is used extensively in C for the following:

■■ An operating system kernel (check out the Linux kernel source).
■■ Embedded systems.
■■ Any code that needs to work with hardware.
■■ Any code that needs to be very fast.
■■ You’ll see it from time to time if you ever work with C, and you may need
to use it yourself.
Remember that when using inline assembly, you’ll need to add a new flag
to gcc. For example, the command gcc –masm=intel myFile.c tells gcc that
you’ve written some intel assembly into your C file.

Summary
 This chapter demonstrated how to use an understanding of x86 and the stack
for hacking. By smashing the stack and inserting shellcode, a reverser can trick
a program into running the attacker’s code.
Conclusion

Wow, this has been quite a journey! We’ve covered offense to defense; high-level
languages down to assembly; registers, control flow, reverse engineering; patch-
ing, tools, techniques, and mindset. If you’ve made it this far, you have an
amazing baseline of knowledge to build from as you continue to move forward.
And as you do move forward, you will always encounter something new. At
first, it will be assembly instructions you don’t know, then defenses you’ve never
seen, then architectures you’ve never heard of, and of course the latest, greatest
tool-of-the-day or defense-of-the-year. But now that you have the basics, you’ll
find that new things become easier and easier to pick up quickly.
Now that you know mov, you can easily understand the string version movs.
You’ve worked with bit manipulations like not, so negation with neg makes
sense pretty quickly. You’ve mastered comparisons like cmp , so cmps isn’t
much of a stretch, and from there how about cmpxchg or cmpxchg16b or
lock cmpxchg8b? The gist is: now that you have the basics, it becomes increasingly
easy to understand new instructions; whether it’s ud (undefined instruction) or
gf2p8affineinvqb (Galois field affine transformation inverse), the fundamen-
tals tend to be mostly the same for everything.
But of course, learning more doesn’t end there. New instructions are great,
but if you keep on this path, you’ll soon encounter entirely new architectures.
The good news is, they also tend to follow the same basic patterns, and now
that you’ve mastered one, you’ll be able to understand new ones in no time.
x64 (64-bit x86) is easy now that you’ve done x86—just extend the registers to
64 bits (rax instead of eax, rsp instead of esp) and follow some different calling
conventions (AMD64 ABI in addition to cdecl), and you’ll be able to apply all
the same tools and techniques to 64-bit code. From there, Arm comes pretty
easily—again, it’s just new registers (r0 instead of rax), instructions (b instead

271
272 Conclusion

of jmp), and calling conventions (Arm instead of cdecl). The underlying patterns
tend to be mostly the same, so whatever your target—PowerPC, MIPS, RISC-V,
MIL-STD-1750A, etc.—you can usually learn the basics in a few hours. Expanding
to new architectures will also let you apply your skills to new devices. Whether
it’s phones, routers, cars, or satellites, the fundamentals are fairly uniform.
Naturally, as you keep advancing, you won’t just encounter new architectures;
you’ll encounter new tools as well. The good news here, too, is that they tend
to build off of the same base set of concepts. We’ve worked through a bevy of
disassemblers, hex editors, debuggers, and decompilers. Now it’s time to start
exploring new options to see what clicks with you. Ghidra, Binary Ninja, and
Cutter/radare2 are popular next steps that will build off of your experience
with IDA and offer even more ways to dissect and understand a program. As
you grow your arsenal of tools, you’ll gradually build up your own scripts,
workflows, and strategies to become increasingly proficient with more and
more difficult targets.
And, of course, if you keep at it, you’ll begin to encounter new defenses.
Whether it’s the latest anti-cheat in online gaming, a new opaque predicate
obfuscation from academia, or creative new hashing in an esoteric keychecker,
keeping up-to-date with the latest trends will help you stay sharp, whether your
passion is offense or defense. Both academic journals and cracking forums can
be fantastic resources here.
But whatever your end goals with this skill set, the singular key to moving
forward is practice. Try writing your own keychecker, and then see if you
can crack it—playing both sides at once can offer interesting insights into the
challenges and limitations of an adversary. Crackmes offer a fantastic, fun,
and (mostly) safe way to get experience in reverse engineering and software
modification on a wide variety of languages and architectures. Whenever you
have a few minutes, grab a crackme that seems in line with your experience
and skill level and see if you can defeat it; if you have a few hours, find one
that uses a language you don’t know or defenses you’ve never seen. Beyond
cracking, modifying simple programs can quickly offer new insights and expand
your skill set. Drop your favorite 90s video game into IDA and see if you can
add infinite lives; try out Ghidra on your favorite text editor and see if you
can add a secret menu. Alternatively, capture-the-flag competitions can be an
exciting way to push your reverse engineering skills to their limit, while simulta-
neously branching into new areas like binary exploitation and computer forensics.
However you proceed, stay persistent, keep practicing, and continue to push
your limits into new domains. As you do, we hope that this book has helped
you establish a broad baseline of skills and that you’ll use them to dive ever
deeper into this awesome facet of security.