A Brief Survey on Rootkit Techniques in Malicious Codes

Sungkwan Kim, Junyoung Park, Kyungroul Lee Ilsun You

Soonchunhyang University Korean Bible University
Shinchang-myun, Asan-si, Republic of Korea Seoul, Republic of Korea
{carpedm, wwkim3, apple}@sch.ac.kr isyou@bible.ac.kr

Kangbin Yim∗
Soonchunhyang University
Shinchang-myun, Asan-si, Republic of Korea
yim@sch.ac.kr

Abstract
Nowadays, malicious codes are significantly increasing, leading to serious damages to information
systems. It is worth to note that these codes generally depend on the rootkit techniques to make it
more difficult for themselves to be analyzed and detected. Therefore, it is of paramount importance
to research the rootkits to effectively defend against malicious codes. In this paper, we explore and
survey the rootkit techniques both in user-level and kernel-level. Several rootkit samples are also
utilized for the test and verification purpose.

Keywords: rootkit, malicious codes, keyboard security

1 Introduction

The superlative invention of the Internet in the 20th century has generated innovative developments in
the computer industry. While internet speed is increasing, adverse effects are also increasing. A com-
puter virus, which would have taken a long time to propagate in former days, can now spread throughout
the world currently in a few seconds or minutes. In the past, the spreading technology was very simple
and the path was limited, thus enabling an account technician sufficient time to protect against the virus.
However, malicious codes have been intelligently armed with self-deformation and concealment, cre-
ating many problems such as Distributed Denial of Services Attacks (DDoS), SPAM, and hijacking of
personal information[2]. The number of malicious codes and different types of viruses has continued to
increase in the first half of 2011[4]. Security specialists collect, disassemble, and analyze malicious codes
to deal with their security threats. Especially, by using reverse engineering, they can trace and analyze
each stage of the malicious act so they are able to determine the damage level, spreading speed, eliminat-
ing method, and effective countermeasure to prevent malicious codes from spreading. Note that reverse
engineering can be used not only by security technicians but also by malicious code developers[5]. These
developers make use of reverse engineering to detect the weak points of the malicious codes and make
these codes stronger. Furthermore, they create codes that can avoid security specialist’s analysis, us-
ing anti-debugging and rootkit. Security specialists are therefore researching ways to cope with these
evolving malicious codes. However, malicious code generation tools are in circulation through hacking
forums and closed news, making malicious codes easier to access, while techniques to evade analysis
Journal of Internet Services and Information Security (JISIS), volume: 3, number: 4, pp. 134-147
∗ Corresponding
Author: LISA Laboratory, Soonchunhyang University, 9418, Engineering Building, 646, Eupnae-ri,
Shinchang-myun, Asan-si, Chungcheongnam-do, Republic of Korea, Tel: +82-(0)415301741, Web: http://lisa.sch.ac.
kr

134
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

are progressing. Thus, it is not easy for security specialists to timely and effectively defend against ma-
licious codes. To make matters worse, human resources for analysis of malicious code are deficient and
the development speed of analysis technology is not faster than that of malicious codes. Importantly,
most malicious codes take full advantage of the rootkit techniques to hide themselves while evading the
analysis and detection. Therefore, it is vital for security specialists to study the rootkits in addition to ma-
licious codes. Motivated by this, this paper explores the rootkit techniques used in malicious codes and
studies their countermeasures. Additionally, sample rootkits are implemented and executed to analyze
the rootkit techniques in more practical and effective.

2 What is the Rootkit?

A rootkit is a software that is developed to enable its associated programs to conceal their presence from
analysis and detection while maintaining privileged access to a target system[12]. It is worth to note that
contemporary malwares tend to apply the rootkit techniques to install and activate themselves without
being detected, thus effectively doing serious damages to victim systems. Once installed, the rootkit can
modify the Operating System (OS) or the existing softwares, some of which might otherwise be utilized
for malware detection and analysis. Therefore, it is so hard to detect the rootkit and its associated
programs because it can subvert security or anti-virus programs whose goal is to find itself. As a result,
it becomes an important challenge to discover and handle the rootkit.
Normally, each OS consists of a user-mode, called ‘ring 3’ and a kernel-mode, called ‘ring 0’. In
the kernel-mode, actual codes are executed to perform the core tasks given by the OS, whereas in the
user-mode those codes are indirectly executed via an Application Programming Interface (API). In other
words, in the user-mode, the API is not used to directly process the kernel operations but to assist in
accessing the kernel-mode. A rootkit usually conceals itself in the process of accessing the user-mode
and kernel-mode. Figure 1 shows a rootkit in detail[3][1].

Figure 1: Rootkit’s position and role in a system

As depicted in Figures 2 and 3, rootkits can be divided into three types: the user-level, kernel-level,
and other rootkits. In the subsequent sections, we will focus on the user-level and kernel-level rootkits.

135
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

Figure 2: shows this classification

Figure 3: Types of rootkit

3 User-level Rootkit
The user-level rootkits exploit the API hooking to hide themselves since they have no authority to control
the kernel structure. In more detail, the API Hooking helps these rootkits to perform their concealment
by accessing the program’s memory address space and manipulating its information. However, the main
drawback of the user-level rootkits is that they are easily detected by the rootkit detector. There are 3
subsystems: Win32, POSIX, OS/2 on Windows. Each sub system is composed of APIs, through which
a process asks its OS to execute its required operations. Therefore, a user-level rootkit can manipulate
APIs, i.e. API hooking, to modify the regular operations of its OS at its will. The API hookings are
largely classified into Import Address Table (IAT) hooking and inline hooking. In the IAT hooking, a
hacker tries to replaces an origianl function address stored in IAT with the address of the rootkit. In this
way, if the original function is called, the rootkit is executed. Unlike the IAT hooking, the inline hooking
tries to modify the function code of a victim program at its will, thus being independent on the function
address.
In order to hook the API, the rootkit code should be inserted into another process’ address space.
The representative method for this is the Dynamic-Link Library (DLL) Injection

136
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

3.1 DLL Injection

The DLL injection is a method aiming to insert a rootkit code or a malicious code into another process’
address space. Once a malicious DLL is injected into a program, a hacker can amend the execution flow
of the program’s API calls. This method can be categorized into three techniques: using registry, using
hooking function, and using remote thread.

• DLL Injection using registry

A hacker can conduct the DLL injection by modifying the registry value of
HKEY_LOCAL_MACHINE\Software\Microsoft\WindowsNt\CurrentVersion\Windows\
AppInit_DLLs and adding a malicious DLL to windows NT/2000/XP/2003. In order to use this injec-
tion, rebooting is necessary because the changed registry value should be applied in the system. Once the
registry change is reflected, the system needs no more booting for the newly started processes. Clearly
it is easy to run this method, but at least one reboot is necessary. Thus, the security specialist can just
check the registry in every booting to prevent this DLL injection.

• DLL injection using window hooking function

In a windows OS, application programs are executed based on events. The SetWindowsHookEx[9]
function is defined to process an event in a special way by allowing an application to install an user-
defined hook procedure into a hook chain. As shown in the function prototype, this function is called
with an event, which will be hooked, and a hook procedure. Then, the hook procedure is called whenever
the event happens once being loaded into the virtual memory address by the function. It is necessary
to monitor the DLL injections using windows hooking or remote thread in real-time because they are
immediately operated without rebooting. For this goal, in the case of using the window hooking function,
we can use the PDESKTOPINFO pDeskInfo structure which is in the THREADINFO structure and has
the desktop information. The array of a global hooking list is registered inside the DESKTOPINFO
structure PHOOK[7] aphkstart. Since the HOOK structure has information on the hooking event type,
simply checking this field can determine whether global hooking is set. There are 15 global hooking
types. Results of global hooking before and after the injection are shown in Figure 4.

Figure 4: Method for DLL injection using window hooking function

An embodied example attack code proceeds to global hooking at the keyboard entry. We can confirm
that hooking is not established at the 4th distribution before performing hooking. After hooking, global
hooking is established on the keyboard event at the 4th distribution. Therefore, a bulwark can detect
global hooking by periodically chasing an established global hooking event.

137
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

• DLL Injection using remote thread

This method creates a remote thread for a victim process to inject the rootkit DLL in the address area
of that process. For this purpose, the CreateRemoteThread function[6] is used. In order to start this
injection, we first use the VirtualAllocEx function[10] to assign a memory to the address space of a victim
process, and use the WriteProcessMemory function[11] to put the DLL name. Then, the LoadLibrary
function is called for the DLL load, and the CreateRomoteThread function is executed to create the
thread in the destination process. This DLL injection is a more intellectual attack because it can attack
not only the global process, but every process. Thus, countering this method is very time consuming and
causes an overload of the system because all processes should be examined. This attack loads the DLL
in which the malicious code is inserted, creates the thread and executes that code, so it is detectable by
checking the numbers of the generated threads. However, in the case of the network server program,
the above method is not effective because the number of threads changes dynamically according to the
client’s request. This problem is solved by testing if the DLL loaded in the process is a certified one.
Because the malicious DLL should be loaded for attack, it can be more efficient to detect a DLL without
a permit or a DLL that is not included in the original file. This is shown in Figure 5.

Figure 5: Method for DLL injection using remote thread

4 Kernel-level Rootkit
It is difficult to detect the kernel-level rootkit because it performs hidden behaviors by hooking the native
API, and executes in the same level as anti-rootkit softwares. The most commonly used hooking methods
in the kernel mode include System Service Descriptor Table (SSDT), Interrupt Descriptor Table (IDT),
and a major function of device driver object, and so on, most of which are based on execution codes.
Because modified execution codes can be detected by the anti-rootkit software, some attackers adopt the
Direct Kernel Object Manipulation (DKOM) technique, which modifies kernel-level data .

4.1 System Service Dispatch Table (SSDT) Hooking

A Microsoft windows supports sub-systems for the Win32, POSIX, and OS/2. Hence, for these sub-
systems, a specific kernel structure is prepared called the System Service Dispatch Table (SSDT). This
table maintains the memory addresses of the related fuctions in the order of the system calls. In addition,
the System Parameter Table (SSPT) is used to manage the size of parameters used in each system service
factions, and in a Microsoft windows, it is possible to obtain various types of data by the ZwQuerySys-
temInformation function. This function provides a list of running processes; thus, a rootkit can hide

138
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

one of them by changing the address of the NwQuerySystemInformation function. Figure 6 shows an
example of a hidden process using this technique.

Figure 6: SSDT hooking result

Note that the above example is the rootkit program that deodorizes keyboard data. With the help of
the SSDT hooking, the rootkit hides itself to evade the detection. To address this attack, we can maintain
the white list including all SSDT tables to detect forge/modulation.

4.2 Interrupt Descriptor Table (IDT) Hooking

The Interrupt Descriptor Table (IDT) is a system table used to handle the interrupt. The interrupt is caused
by software or hardware, and a keyboard interrupt is one of the typical interrupts. An OS executes the
keyboard interrupt by calling the keyboard interrupt handler when a user types a key. The sidt instruction
can be used to get the information and location of an IDT. Therefore, an attacker is able to hook by
collecting the IDT information using this instruction, and Figure 7 depicts an example.

Figure 7: IDT hooking result

When the keyboard interrupt happens, the above example prints a particular message by the hooked
interrupt. Note that the actual keyboard data can be leaked by this technique. We are able to detect
forge/modulation by constructing the white list including all IDT tables.

4.3 Hooking to Inline Function

In the inline function hooking, the rootkit inserts an unconditional jump within the body of a victim
function to an area of the memory which it can control with some malicious codes. For example, the

139
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

NtDeviceIoControlFile function code[2] can be patched into a new one [4] so that it can jump to the
rootkit code. For the actual test, we use the NtDeviceIoControlFile, which sends the device control code
and related information to the device driver. Using this, we can hide or change the information.
Figure 8 shows the NoDeviceIoControlFile and patched NtDeviceIoControlFile codes while Fig-
ures 9 and 10 show the execution result of the sample rootkit.

Figure 8: The result of NoDeviceIoControlFile code patch

Figure 9: the execution result of the sample rootkit using inline function hooking

140
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

This method alters the flow of the internal practice function without changing the contents of the
table that has the address of the relevant function because the address of the function itself is not changed.
Similar to other hooking, the factor value which is delivered when the system call or function is called
can be altered by the inserting rootkit code.

Figure 10: Alteration of operation flow by detour patch

This technique can be detected by checking the beginning part of the original function because its
basic action changes the code of the destination function for hooking. To determine if it is hooked, we
can also calculate the CRC of the code. Figure 11 shows the result that determines the image coincidence
using WinDbug.

Figure 11: Alteration of operation flow by detour patch

Five errors appear and the memory area with the error is 0x8057442-0x80570446. As a result of
disassembling the memory 0x8057442, we can confirm that it diverges to
my_function_detour_ntdeviceiocontrolfile, which is the Hooking function of the sample rootkit.
To retreat the detected function back to the original function, the driver file or non-changed function code
can be made.

4.4 Jump Template Technique

In the case of the above hooking, when an attacker wants to hook more functions, they have to patch many
functions because a rootkit function should return an original function after it is executed. This procedure
is not easy and therefore a new rootkit technique called the jump template technique has been developed.
The technique first executes the rootkit function, and then calls the original function while returning to
the normal service. Because an attacker needs to return a result of the called original function for a
stealth attack, the implementation overhead is reduced due to the automatic call of the original function
without the need to patch each function. Figure 12 shows the flow chart of jump template technique

141
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

and Figure 13 illustrates the implement result of sample rootkit. This technique should let each interrupt
routine indicate its corresponding rootkit code address.

Figure 12: Flow chart of the jump template technique

Figure 13: Execution result of the sample rookit using jump template

The basic concept of the jump template is that an original code changes from an interrupt service
routine to the jump template code. Therefore, the detecting method is limited when verifying an address
of the interrupt service routine because a defender does not decide whether or not the interrupt service

142
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

routine address is correct. Hence, the defender can detect the jump template technique by checking the
first five bytes. This means that the first five bytes of the original code are not equal to the jump template
code. To prove this, we show disassemble codes of the original interrupt service routine in Figure 14.

Figure 14: Result of the original keyboard interrupt ISR disassemble

In Figure 14, the first five bytes of an original keyboard interrupt service routine consist of PUSH
ESP and PUSH EBP, while those of the jump template code different from these instructions. Hence,
we can judge that the interrupt service routine is not hooked when the first five bytes are 54, 55, 53,
56, and 57, otherwise it is hooked. Figure 14 shows an implement result of detecting the jump template
technique.

Figure 15: Jump template when hooked

In Figure 15, we can see that 0x93 (keyboard) interrupt is hooked. To treat a hooked function, a
defender generates an original function code and then modifies the hooked interrupt service routine to
the generated original function code.

4.5 Layered drivers

The layered drivers are formed into a hierarchical chain or stack to structuralize the hardware-related
drivers physically and functionally. This structure has an advantage for implementation because a de-
veloper only implements the part of an additional function when a driver is developed. This means that
the developer needs not to implement the entire driver codes and thus the cost for implementation is
reduced. Almost all the current hardware devices have driver chains to support layered drivers. In detail,
the lowest driver deals with hardware-related functions while the highest driver handles error codes and
queries, and structuralizes data to deliver a lower layer driver. Hence, an attacker can steal or hide data
which is transferred to the lower or higher layer driver and it calls the rootkit by the layered drivers. The
layered drivers use the I/O Request Packet (IRP) for communicating to each other. If a rootkit driver is
successfully inserted between the driver chains, it is possible to steal the IRP, which has a lot of important
information such as USB data, keyboard data and so on. In the case of a keyboard driver, a request to
read the inputted keyboard scan code is generated when a user inputs the key while the IRP is generated.

143
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

The generated IRP is transferred to the i8042prt driver, which is the lowest driver in the keyboard driver
stacks. In this process, if an attacker inserts the rootkit driver at the higher driver of the i8042prt driver,
he or she can modify or steal the IRP. This is because the I8042prt driver reads the keyboard data from a
keyboard output buffer and the driver then transfers the IRP to the higher driver after storing the keyboard
scan code in the IRP. It means that the IRP transferred to the rootkit driver is stored in the keyboard scan
code and the attacker is thus able to steal or modify the keyboard scan code. We build a sample rootkit
that deodorizes keyboard data through this technique. Figure 16 is shown sample rootkit stack inserted
in PS/2 keyboard device stack. Figure 16 is shown the result of program execution.

Figure 16: sample rootkit driver inserted in ps/2 keyboard device stack

Figure 17: sample rootkit results using layer driver

In the case of the approach of the layered drivers, the rootkit driver needs to link the device chains.
By using this, we can detect this rootkit. That is, if an unknown driver connects to the device chains, the
defender can decide that the driver is the rootkit driver. For this, the defender must know all the device
chains of all the drivers. In order to prove this detection method, we implement a sample rootkit driver

144
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

that is attached to the i8042prt driver. Figure 19 shows the detection result of the rootkit by the layered
drivers.

Figure 18: The detection result of the keyboard rootkit driver by layered drivers

In the case of an original device stack, there is no driver higher than the \Device\Keyboardclass0,
although when inserting the rootkit driver, we know that the \Device\Samplerootkit driver exists as
shown in Figure 19. Therefore, we can decide that the \Device\Samplerootkit driver is the rootkit
driver. In order to treat the rootkit driver, a defender inserts an anti-rootkit driver at the lower stack of
the inserted rootkit driver. Using this countermeasure, if the anti-rootkit driver can erase the keyboard
scan code in the IRP, the rootkit driver cannot steal and modify the keyboard scan code. Nevertheless,
there is a competition problem with this countermeasure between the attacker and the defender because
the rootkit driver is able to insert at the lower stack of the anti-rootkit driver. Hence, this countermeasure
can treat the rootkit driver in a short period of time. The other countermeasure involves detaching the
detected driver from the device stacks by the IoDetachDevice function [8]. Figure 19 shows the treatment
result of the sample anti-rootkit driver. Figure 19, a keyboard scan code is exposed before loading the

Figure 19: The treatment result of the sample anti-rookit driver

anti-rootkit driver. Otherwise, after loading the anti-rootkit driver, the keyboard scan code is not exposed

145
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

to anyone.

5 Conclusion

This paper has briefly surveyed the rootkit techniques while discussing the countermeasures. Such a
survey is powered by the sample rootkits, which are specially implemented to show how they operate.
As a future work, it is necessary to study the latest rookit technologies in newly emerging computing
platforms such as smart phones, cloud computing, and so forth.

References
[1] H. G. and B. J. Rootkits: Subverting the Windows Kernel. Pearson Education, Inc., 2005.
[2] J. Kim, J. Lee, E. Park, E. Jang, and H. Kim. Research on Modeling and Simulation Technique for Cost
Analysis on DDoS Attack Damage Scale and Dealing Method. In Proc. of the Korea Simulation Association
(KSS’09), Korea, pages 39–47, December 2009.
[3] Y. Kwon. Rootkit, Reveal Your Identity: The Story of spears and shields. Microsoftware, 2008.
[4] A. Lab). Monthly Security Report. Ahn lab, September 2011.
[5] D. Lee, S. Lee, and G. Lee. Research on early detection technique on cyber threats based on honey net. In
Proc. of the Korea Information Assurance Society (KIAS’05), Korea, pages 67–72, December 2005.
[6] msdn. CreateRemoteThread function. MSDN(Microsoft), September 2012. http://msdn.microsoft.
com/en-us/library/ms682437(VS.85).aspx.
[7] msdn. Hooks Overview. MSDN(Microsoft), September 2012. http://msdn.microsoft.com/en-us/
library/ms644959(VS.85).aspx.
[8] msdn. IoDetachDevice routine. MSDN(Microsoft), September 2012. http://msdn.microsoft.com/
en-us/library/ff549087(VS.85).aspx.
[9] msdn. SetWindowsHookEx function. MSDN(Microsoft), September 2012. http://msdn.microsoft.
com/en-us/library/ms644990(VS.85).aspx.
[10] msdn. VirtualAllocEx function. MSDN(Microsoft), September 2012. http://msdn.microsoft.com/
en-us/site/aa366890.
[11] msdn. WriteProcessMemory function. MSDN(Microsoft), September 2012. http://msdn.microsoft.
com/en-us/library/ms681674(VS.85).aspx.
[12] Wikipedia. Rootkit. http://en.wikipedia.org/wiki/Rootkit.

146
A Brief Survey on Rootkit Techniques in Malicious Codes Lee, Kim, Park, You, and Yim

Sungkwan Kim is currently working towards his B.S. in Information Security Engi-
neering at Soonchunhyang University, Republic of Korea. Also, he is a member of
the Lab. of Information Systems Security Assurance (LISA) leaded by Prof. Kang-
bin Yim. His research interests include vulnerability analysis, obfuscation, systems
security, access control and insider threats.

Junyoung Park is currently working towards his B.S. in Information Security Engi-
neering at Soonchunhyang University, Republic of Korea. Also, he is a member of
the Lab. of Information Systems Security Assurance (LISA) leaded by Prof. Kang-
bin Yim. His research interests include vulnerability analysis, kernel mode root kit,
systems security, access control and insider threats.

Kyungroul Lee received his B.S and M.S. degrees from Soonchunhyang University,
Republic of Korea in 2008 and 2010 respectively. He is currently working towards his
Ph.D. degree in the same university. His research interests include vulnerability anal-
ysis, kernel mode root kit, obfuscation, systems security, access control and insider
threats.

Ilsun You received his M.S. and Ph.D. degrees in Computer Science from Dankook
University, Seoul, Korea in 1997 and 2002, respectively. Since March 2005, he has
been an Assistant Professor in the School of InformationScience at the Korean Bible
University, South Korea. Dr. You served or is currently serving on the organizing or
program committees of international conferences and workshops including IMIS’07-
13, MIST’09-12, MobiWorld’08-12, and so forth. Also, he has served as a guest editor
for more than 10 international journals. He is on the editorial boards of International
Journal of Ad Hoc and Ubiquitous Computing (IJAHUC), Computing and Informatics (CAI), and Journal
of Korean Society for Internet Information (KSII). His main research interests include mobile Internet
security and formal security veri?cation.

Kangbin Yim received his B.S., M.S., and Ph.D. from Ajou University, Suwon, Korea
in 1992, 1994 and 2001, respectively. He is currently an associate professor in the
Department of Information Security Engineering, Soonchunhyang University. He has
served as an executive board member of Korea Institute of Information Security and
Cryptology, Korean Society for Internet Information and The Institute of Electronics
Engineers of Korea. He also has served as a committee chair of the international
conferences and workshops and the guest editor of the journals such as JIT, MIS and
JoWUA. His research interests include vulnerability assessment, code obfuscation, malware analysis,
leakage prevention, secure platform architecture and mobile security. Related to these topics, he has
worked on more than forty research projects and published more than ninety research papers.

147