Analyzing Malicious Windows Programs

Malicious Windows Programs

 Windows API: a broad set of functionality -> governs the way
malware interacts with Microsoft libraries.
 Handles: items opened or created by the OS (e.g. window,
process, module, menu, file)
 a value of some opaque type that identifies an object (the type of the
handle is unrelated to the element referenced)
 Can be implemented by a pointer or other than a pointer
 Can be thought of a “handle’’ to a door (a door has only one handle
for it to open).
 E.g. CreateWindowEx function returns HWND -> handle to a
window; whenever want to do anything with that window, need to
use that handle.
File System Functions
 Important: file activity can hint what the malware does.
 CreateFile, ReadFile, WriteFile
 CreateFileMapping -> loads a file from disk into memory
 This function is used to create a handle to a file mapping that loads a file into memory and makes it
accessible via memory addresses. Launchers, loaders, and injectors use this function to read and
modify PE files.
 MapViewofFile -> returns a pointer to the base address of the mapping.
Windows Registry
 Store configuration information (settings and options –
networking, driver, startup, user account)
 Malware often hits registry
 Root Keys (HKLM, HKCU, etc)
 HKEY_LOCAL_MACHINE (HKLM) – Settings global to the
machine
 HKEY_CURRENT_USER (HKCU) – Settings for current user
 Regedit tool for examining values
 HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\
CurrentVersion\Run -> determines the executables to start when
a user logs in
 Registry functions
 Common registry functions (malware use them to modify registry)
 RegOpenKeyEx: open a registry for editing and querying
 RegSetValueEx: adds a new value and set data
 RegGetValue: return data for value entry.

Once clicked, it will do

Networking APIs
 Berkeley sockets API
 socket, bind, listen, accept, connect, recv, send

 WinINet API
 Higher-level API (stored in Wininet.dll)
 InternetOpen: initiate connection to the Internet
 InternetOpenURL: connect to a URL
 InternetReadFile: read data from a downloaded file
Dynamic link libraries (DLL)
 DLL
 use libraries to share code among multiple applications (code re-use
among apps including malware)
 Memory shared among running processes (static libs are loaded n
times for n programs, n times of memory)
 Minimize software distributions
 Malware:
 hides malicious code in DLL rather than .exe
 uses standard Windows DLLs to interact with OS
 Uses third-party DLLs (e.g. Firefox DLL) to avoid re-implementing
functions
 No real difference between DLL and .exe (except a flag)
Processes

 Windows uses processes as containers to manage

resources/resource sharing.
 Processes can have the same memory address as 0x004000, but
physical memory stores the data should be different.(street
address without zip code)
 Malware can execute code outside of current process as part of
another process.
 Create a process to execute its malicious code (bypass firewall);
create an instance of Internet Explorer and use that program to
access malicious code
Threads
 A process contains one or more threads
 Share the same memory space
 Each thread has its own registers and stack
 Complete control of the CPU (core), other threads cannot affect.
 Before switching between threads, all values are saved in structure
called thread context. (Possibly other threads are modifying the register)

If another thread runs in between

1. Stores EDX in thread context
2. After the other thread finishes,
restore EDX from thread context
Threads
 Malware can
 Use CreateThread to load a new malicious DLL into a process
(CreateThread with address of LoadLibrary as start
address)
 Also used to remotely control a process
 Two threads created (input and output)
 One takes network input and sends to process stdin (via WriteFile)
 One takes process stdout (via ReadFile) and sends to network
 Send all information to a single socket to communicate seamlessly with the
running application
 Mutex
 Mutexes: Only one thread can own a mutex at a time -> used to
coordinate multiple processes and threads
 Malware: often use hard-coded names
 Mutex name must be consistent if used by two processes
 Thread gains access of mutex WaitForSingleObject
 Subsequent threads must wait
 A thread finishes: ReleaseMutex
 CreateMutex, OpenMutex (get a handle to another process’s
mutex)
 Malware will commonly create a mutex and open an existing
mutex with the same name to ensure only one version of the
malware is running at a time
Mutex Example (Listing 7-9)
Mutex Name

Try to OpenMutex an existing name HGL345

If NULL -> Create a Mutex Name HGL345

If NOT NULL -> Exit

Services
 Malware installing as services
 Processes run in the background
 Scheduled and run by Windows service manager without user input
 Starts automatically with OS, may not even show up in Task Manager
as process (Malware isn’t running a separate process)
• OpenSCManager(returns a handle to the service control
manager), CreateService, StartService
• Service Types:
 WIN32_SHARE_PROCESS = allows multiple processes to share service (e.g.
svchost.exe) or dll.
 WIN32_OWN_PROCESS = independent process
 KERNEL_DRIVER = loads code into kernel
 Kernel mode/User Mode – Kernel mode direct access to I/O,
hardware, all memory address, interrupt
Microsoft Component Object Model
(COM)
• Interface standard that allows software components to call each other
• Implemented as a client/server framework. Clients: programs making
use of COM objects; server: resusable software components.
• If uses COM, thread must call OleInitialize,
CoInitializeEx (need more identifiers)
• COM objects access via class identifiers (CLSIDs) and interface
identifers (IIDs).
• Common function used by malware is Navigate in
IWebBrowser2 interface (launch IE and access a web address)
 IWebBrowser2 implemented by Internet Explorer
 Details see Listing 7-11, 7-12 (Read p.155) –
IWebBrowser2/IE have their unique identifiers.
Com Server
 Microsoft Component Object Model
 Malware implemented as COM server within web browser via Browser Helper Objects (3rd
party plug-ins)
 Used to monitor traffic going through browser without creating a separate process that
can be detected (run inside IE)
 Can be detected via export functions (features)
 DllCanUnloadNow, DllGetClassObject, DllInstall,
DllRegisterServer, DllUnregisterServer
 Exceptions
• Allow program to handle exceptional conditions during
program execution
• Transfer to a special routine to resolve the exception
– Windows Structured Exception Handling
• Exception handling information stored on stack before function
invocation
• Not all handlers respond to all exceptions
• Thrown to caller's frame if not handled
• If none responds, top-level handler crashes
– Used by malware to hijack execution
• pointer to exception is stored on stack
• attacker can overwrite the pointer in stack overflow
• Attacker then triggers exception to gain execution
Kernel Mode
 Two privilege levels: kernel mode and user mode
 Nearly all code (applications) runs in user mode, except drivers
 Use Windows API to manipulate kernel structures
 Kernel mode
 Shared resources/memory addresses
 Fewer security checks
 Invalid -> blue screen
 Antivirus/firewall->kernel mode
 Only sophisticaed malware runs in kernel
Native API
 Popular among malware – bypass normal Windows API

 Ntdll.dll: manages interactions

between user space and kernel
 Processor then switches to kernel
mode
 Ntdll uses the Native API

 Separation allows Microsoft to change

kernel without affecting applications
Native API
Malware often call ntdll
directly to avoid detection of
security programs between
kernel32.dll and
ntdll.dll
Malware not calling Windows
API (ReadFile, WriteFile)
but calling Native API
(NtReadFile,
NtWriteFile) instead

Well-designed security program will monitor calls at

all levels including the kernel