Hunting with Splunk Lab 2

Hunting with Splunk (Lab 2)

Scenario

The IT Security manager has provided you with a Splunk instance that has ingested the Boss Of The SOC
v2 dataset. He tasked you with getting familiar with Splunk through the given hunting tasks, so that you
can create more complex Splunk hunting searches in the future.

All hunts of this lab build mostly on hypotheses derived from the MITRE ATT&CK framework.

Specifically, you will hunt for:

• PowerShell

• Exfiltration Over FTP

• Exfiltration Over DNS

• Adversary Infrastructure

• Lateral Movement

More info can be found on the following resource.

https://www.lockheedmartin.com/content/dam/lockheed-martin/rms/documents/cyber/LM-White-
Paper-Intel-Driven-Defense.pdf

Full credit goes to Ryan Kovar, David Herrald, James Brodsky, John Stoner, Jim Apger, and David
Veuve for sharing the Splunk detection tips this lab covers with the public.

That being said, the provided hunt solutions/searches in this lab are slightly different.

Goals

The learning objective of this lab is the rule generation ability and becoming more familiar with Splunk.

What you will learn

The learning objective of this lab is to not only get familiar with Splunk's architecture and detection
capabilities but also to learn effective Splunk search writing.

Specifically, you will learn how to use Splunk's capabilities in order to:

• Have better visibility over a network

• Respond to incidents timely and effectively

• Proactively hunt for threats

Hunting with Splunk Lab 2

Activity Status: unstarted

Report an issue

overview

tasks

solutions

SOLUTIONS

Below, you can find solutions for each task. Remember though, that you can follow your own strategy,
which may be different from the one explained in the following lab.

Kali Machine
Task 1. Hunt for PowerShell Empire

When looking at https://github.com/EmpireProject/Empire/blob/master/setup/cert.sh, we can see the

default key that is created uses C=US as the subject of the certificate. Let's hunt for that! A query that
can perform this kind of hunt can be found below.

index=botsv2 sourcetype=stream:tcp ssl_issuer="C = US"

Once the query is completed, we can look at the src. The result should be similar to the one below.

The SSL subject/issuer value of "C = US" was seen on four internal systems
Bonus (for advanced hunting): You may have noticed that the SPL search above took some time to
complete. We could have accelerated our search by using the tstats command as follows.

index=botsv2 sourcetype=stream:tcp ssl_issuer="C = US" | top src_ip dest_ip

| sort - count

The above search utilizes Splunk Datamodels and provides us with the same result but in considerably
less time.

Note the firstTime and lastTime entries as well as the common destination 45.77.65.211.

A data model is a hierarchically structured search-time mapping of semantic knowledge about one or
more datasets. It encodes the domain knowledge necessary to build a variety of specialized searches of
those datasets. These specialized searches are used by Splunk software to generate reports for Pivot
users.

Goals of a Datamodel:

• Make it easy to share/reuse domain knowledge [Admins/power users build data models]

• Acceleration of large datasets to make searching more efficient

• Provide an interface for 'non-technical users' interact with data via pivot UI [note: non-technical
== analyst with less "domain" knowledge]

To list all available datamodels in your environment, execute the search below.

| datamodel

| spath displayName

| stats values(displayName)
Information on the Certificates data model can be found below.

https://docs.splunk.com/Documentation/CIM/4.15.0/User/Certificates

Back to our hunt, let's see what data we have for the common destination we noticed, 45.77.65.211

index=botsv2 45.77.65.211 | stats count by sourcetype | sort - count

We notice that we have web and Sysmon logs containing that indicator/IP. This means that this IP isn't
being blocked at the network level and that it has reached the system level.
The available firewall logs can indicate the workstations/servers that communicated with that IP the
most, as follows.

index=botsv2 45.77.65.211 sourcetype=pan:traffic | stats count by src_ip dest_ip | sort - count

We obtain a similar view of the situation through Suricata logs, as follows.

index=botsv2 45.77.65.211 sourcetype=suricata | stats count by src_ip dest_ip | sort - count

Let's check Suricata's inbound alerts, as follows.

index=botsv2 src=45.77.65.211 sourcetype=suricata

Once the search is completed click on 32 more fields and select the alert.signature field.
Unfortunately, if we click that (signature) newly-added field no interesting information appears.

Let's focus on another available sourcetype stream:http, to identify communication flows with the IP
indicator (45.77.65.211).

index=botsv2 45.77.65.211 sourcetype=stream:http | stats count by src_ip dest_ip | sort - count

Let's investigate the identified IPs above one by one.

1. 172.31.4.249

The indicator IP (45.77.65.211) has a tremendous amount of events generated with a destination of
172.31.4.249. Note that 172.31.4.249 is a Linux server belonging to our organization.

Let's drill into the events where the source address is our indicator and the destination is our web server
172.31.4.249.

index=botsv2 sourcetype=stream:http src_ip=45.77.65.211 dest_ip=172.31.4.249

It is not uncommon to come across strange-looking user-agents during an attack or penetration test. If
we click on the http_user_agent field, we will see the below.

By searching online for w3af.org, we see that it is an open source web application security scanner. It
looks like the IP indicator has also been scanning a web server of ours in addition to being a destination
for some of our workstations/servers. Note that our web server that was scanned didn't appear in the
outbound communications.

1. 71.39.18.125

index=botsv2 45.77.65.211 sourcetype=stream:http 71.39.18.125

71.39.18.125 is the IP address is the external interface of our firewall. We may want to investigate this
further later. Let's move to the last IP we identified 10.0.2.109.

1. 10.0.2.109

index=botsv2 45.77.65.211 sourcetype=stream:http src_ip=10.0.2.109

Nothing that can advance our hypothesis about PowerShell Empire being executed comes up. We may
come back to investigate this further later.

It is about time we move our attention to host centric data through Microsoft Sysmon logs, as follows.

index=botsv2 45.77.65.211 sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" |

stats count by src_ip dest_ip | sort - count
We see a large amount of events coming from a single server, Venus (10.0.1.101) communicating to our
indicator.

Let's use a wildcard on dest to ensure we get all of the values we saw in the previous search

index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
dest=45.77.65.211*

Once the search is completed, we notice that all returned events are running PowerShell. We also notice
a lot of Network Connect Sysmon events. It is not common to see PowerShell performing network
connections. Finally, some progress...

Let's analyze these Network Connection Sysmon events related with powershell.exe further, as follows.

index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
dest=45.77.65.211* | stats values(dest_port) as dest_port values(host) as host values(src_ip) as src_ip
values(src_port) as src_port by process,dest,user
The user executing these events are either the SYSTEM account or a frothly domain account called
service3.

By using the timechart command we can see that this network activity took place between 8/23 and
8/26. We are focusing on the service3 user.

index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
dest_ip=45.77.65.211* user=FROTHLY\\service3 | timechart count by src_ip

Let's dig deeper into what has been executing through the service3 account, as follows.

index="botsv2" sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
user=FROTHLY\\service3 | stats values(CommandLine) by Computer,process,ParentImage
Not only we can see malicious PowerShell commands being executed but we also notice whoami.exe
and ftp.exe being executed by PowerShell(which is suspicious). These three suspicious processes appear
in both venus.frothly.local and wrk-klagerf.frothly.local.

How we know that these PowerShell commands are malicious you may ask. Let's Base64 decode them.
The output looks certainly malicious.

The output also contains /admin/get.php. Asking for this file is characteristic of PowerShell Empire!
Other strings may also exist that can indicate the existence of PowerShell Empire in our environment)

https://github.com/EmpireProject/Empire/blob/master/data/agent/agent.py

For the extra mile try to identify similar PowerShell commands on other workstations/servers. We
focused on the service3 account only. Maybe another account executed similar commands.

Task 2. Hunt for FTP exfiltration

Let's start with the available sourcetypes for this hunt focusing on August 2017 (data obtained from the
previous Task).

index=botsv2 ftp | stats count by sourcetype | sort - count

Let's now try to identify an IP/indicator.

index=botsv2 ftp sourcetype=suricata | stats count by src_ip dest_ip | sort - count

or

index=botsv2 ftp sourcetype=stream:ftp | stats count by src_ip dest_ip | sort - count

We notice that two internal systems speaking to the same external destination address, 160.153.91.7.

If we look at Palo Alto data, we notice we have two sourcetypes, pan:traffic and pan:threat. The traffic
sourcetype has by far the greater volume of traffic, but interestingly, we see the vast majority of traffic
heading to the external interface of our firewall. So additional inspection of this data to learn more is
warranted.

index=botsv2 ftp sourcetype=pan:traffic | stats count by src_ip dest_ip | sort - count

Let's step away from the network level for a bit and focus on the endpoint level. First, Sysmon events
related to FTP by host.

index=botsv2 ftp sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" | stats count by

host | sort - count

Then, Sysmon events related to FTP by CommandLine

index=botsv2 ftp sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" | stats count by

CommandLine

We notice some curious-looking ftp activity! We will get back to this activity and investigate further.

Back to the network now, let's try to see inside the FTP traffic.
index=botsv2 ftp sourcetype=stream:ftp src_ip=* dest_ip=160.153.91.7

The interesting fields to browse are:

• flow_id

• reply_content

• method

• method_parameter

• filename

So, let's refine our search as follows

index=botsv2 sourcetype=stream:ftp src_ip=* dest_ip=160.153.91.7 method!=PORT method!=TYPE

method!=NLST | table _time src_ip filename method method_parameter reply_content | sort + _time
Looking at the results you will notice the exact same sequence of actions taking place at both 10.0.2.107
and 10.0.2.109. We can see that we have 3 executables, a DLL, a MSI installer for python, a python script
and a hwp file extension. If we aren't sure what a hwp file is, we can Google for the extension and we
find out that it is a Korean word processing application.

Make also sure you give reply_content a look for interesting information.

It is worth looking into each separate transmission. You can do that through the query below, clicking
into each green bar of the histogram and then pressing +Zoom to Selection.

index=botsv2 sourcetype=stream:ftp src_ip=* dest_ip=160.153.91.7

Now you can view which files were uploaded/exfiltrated (do the below for each transmission) as follows.

index=botsv2 sourcetype=stream:ftp src_ip=* dest_ip=160.153.91.7 | stats count by filename

Finally, we should focus our attention on the DLL files we spotted. To identify the sourcetypes that
include information about these DLLs execute the search below.

index=botsv2 (singlefile.dll OR winsys32.dll) | reverse

Let's leverage both to investigate further, as follows.

index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" (singlefile.dll OR
winsys32.dll) | table _time host user CommandLine ParentCommandLine | reverse

index=botsv2 sourcetype=wineventlog (singlefile.dll OR winsys32.dll) | stats count by host

Let's also not forget to search for the other files that were downloaded.

index=botsv2 sourcetype!=stream:ftp (dns.py OR nc.exe OR psexec.exe OR python-2.7.6.amd64.msi OR

wget64.exe OR winsys64.dll OR *.hwp) | stats count by host
To look for events related to these files on, for example, the venus host/server, use the search below.

index=botsv2 sourcetype!=stream:ftp (dns.py OR nc.exe OR psexec.exe OR python-2.7.6.amd64.msi OR

wget64.exe OR winsys64.dll OR *.hwp) | reverse | search host="venus"

We have investigated FTP activity thoroughly enough. You should now have a good idea of how to hunt
for FTP exfiltration.

Task 3. Hunt for DNS exfiltration

With the IP address of 160.153.91.7, we can leverage this indicator to determine if any exfiltration was
being attempted using DNS. stream:dns provides us with all DNS query and responses occurring within
Frothly (our domain).

index=botsv2 sourcetype=stream:dns 160.153.91.7 | stats count by src_ip

Let's start investigating 10.0.2.107, as follows.

index=botsv2 sourcetype=stream:dns 160.153.91.7 src_ip=10.0.2.107

If we click the name field we notice a curious-looking domain.

Let's investigate further as follows.

index=botsv2 sourcetype=stream:dns hildegardsfarm.com | stats count by dest_ip | sort - count

We notice almost 70K events for this domain! Let's look into the queries (since we are hunting for DNS
exfiltration).

index=botsv2 sourcetype=stream:dns hildegardsfarm.com "query{}"="*"

| table _time query{} src_ip dest_ip

Let's leverage URL Toolbox to refine our search. We will need to reference the mozilla list (part of URL
Toolbox) and then run the ut_parse_extended macro against the query field. This will chop the query
into finer pieces to analyze. We can then calculate the Shannon entropy score of the subdomain that
was extracted and then table the output.

As we review at our results, we can see that all of these subdomains have a high entropy score. Entropy
scores greater than 3 are considered high and can often be used to determine if domains or urls are
algorithmically generated. Entropy is not perfect, just look at an AWS subdomain name, but it is another
tool in the tool chest that can help indicate a DGA and that additional interrogation is needed.

index=botsv2 sourcetype=stream:dns hildegardsfarm.com "query{}"="*" query *.hildegardsfarm.com

| eval query{}=mvdedup(query)

| eval list="mozilla"

| `ut_parse_extended(query{},list)`

| `ut_shannon(ut_subdomain)`

| table src_ip dest_ip query{} ut_subdomain ut_shannon

We can further evolve our previous search by further dissecting these subdomains and their associated
queries as a larger data set. If we use our eval statement after we calculate the Shannon Entropy score,
we can calculate the length of each of the subdomains. Using the stats command, we can determine the
average Shannon Entropy score, average length of the subdomain and the standard deviation of the
length of the subdomain by the domain.

Generally, a high average entropy, high subdomain length and low standard deviation of the length,
coupled with a high event count could be indicative of a DNS exfiltration attempt and we seem to have
met those metrics here.

index=botsv2 sourcetype=stream:dns hildegardsfarm.com "query{}"="*" query *.hildegardsfarm.com

| eval query{}=mvdedup(query)

| eval list="mozilla"

| `ut_parse_extended(query{},list)`

| `ut_shannon(ut_subdomain)`

| eval sublen = length(ut_subdomain)

| table ut_domain ut_subdomain ut_shannon sublen

| stats count avg(ut_shannon) as avg_entropy avg(sublen) as avg_sublen stdev(sublen) as stdev_sublen

by ut_domain
You should now have a better idea of how DNS exfiltration looks like and how it can be proactively
detected in your environment.

Task 4. Hunt for adversary infrastructure

We already know that our 10.0.2.107 endpoint communicated with the 45.77.65.211 IP indicator. We
also know that the communication was based on a SSL certificate of specific configuration. Let's extract
the certificate's sha256 hash and hunt for it in the wild as follows.

index=botsv2 sourcetype=stream:tcp dest_ip=45.77.65.211 src_ip=10.0.2.107 | stats count by

ssl_cert_sha256

Now, if we submit this hash to Censys.io, we recieve no results.

However, let's submit the IP Indicator to virustotal.com and see what the results are. The initial results
show as clean, but lets look at relations. You can see under Communicating Files that there are three
malicious files associated to the IP Address. By clicking on one or more of these files you now have an
opportunity to expand you indicators of compromise list and have a look at other assocaited IP
addresses and domains.

Task 5. Hunt for lateral movement through WMI

Based on the guidance provided in the JPCERT doc, we can craft a search using Sysmon and we find 11
events (between 8/23/2017 and 8/24/2017), all associated with wrk-klagef. Recall that we need escape
characters for \.
index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=1
ParentImage="C:\\Windows\\System32\\svchost.exe" CurrentDirectory="C:\\Windows\\system32\\"
CommandLine="C:\\Windows\\system32\\wbem\\wmiprvse.exe -secured -Embedding"
ParentCommandLine="C:\\Windows\\system32\\svchost.exe -k DcomLaunch" User="NT
AUTHORITY\\NETWORK SERVICE" Image="C:\\Windows\\System32\\wbem\\WmiPrvSE.exe" | stats
count by host

index=botsv2 sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational" EventCode=3

Image="C:\\Windows\\System32\\svchost.exe" User="NT AUTHORITY\\NETWORK SERVICE"

We didn't have any luck with the Network Connection event in Sysmon

The below is what we should see in Remote Execution via WMI at the destination side, but we don't
have all the needed data in our Sysmon logs.

Let's focus on Teymur Kheirklabarov's advice on hunting for remote execution via WMI (systems with
Network 4624 Event followed by Sysmon Process Creations).
index=botsv2 (sourcetype=wineventlog (EventCode=4624 Logon_Type=3)) OR
(sourcetype="xmlwineventlog:microsoft-windows-sysmon/operational"
ParentCommandLine!="*\\svchost.exe" EventCode=1)

| eval login=mvindex(Logon_ID,1)

| eval user_id=mvindex(Security_ID,1)

| eval session=lower(coalesce(login,LogonId))

| transaction session startswith=(EventCode=4624) mvlist=ParentImage

| search eventcount>1

| eval Parent_Process=mvindex(ParentImage, 1)

| table _time dest_ip session host user_id Parent_Process Image CommandLine

To better see all the events that occurred and all processes that were running during the identified
sessions (LogonId) above on the venus and wrk-klagerf hosts, use the searches below.

index=botsv2 ((Logon_ID=0x171491a OR LogonId=0x171491a) host=venus) | table _time EventCode

TaskCategory Account_Name Security_ID Process_Command_Line CommandLine ParentCommandLine |
reverse
index=botsv2 ((Logon_ID=0xf9b47f OR LogonId=0xf9b47f) host=wrk-klagerf) | table _time EventCode
TaskCategory Account_Name Security_ID Process_Command_Line CommandLine ParentCommandLine |
reverse

You should now have a better idea of how to hunt for remote execution through WMI.

Not running

For the best experience, choose the region closest to you. Then, start the lab to begin.

Select the region closest to you

US-East
Keyboard layout:🇺🇸 English (US)