ArcSight & Splunk CEF Integration

“Make new friends, but keep the old, one is silver, the other is gold.”

White Paper & Technical Guide

V9 – 07/02/2018
 Contents
Contents................................................................................................................................................................
Document Revisions..............................................................................................................................................
Disclaimer and Caveats..........................................................................................................................................
Description............................................................................................................................................................
Benefits of ArcSight Enriched Data within Splunk.................................................................................................
Prerequisites.........................................................................................................................................................
ArcSight Integrator TA Installation........................................................................................................................
Splunk Configuration for SmartConnector CEF Syslog Ingestion...........................................................................
Splunk Add-on for Kafka Configuration with Event Broker or Kafka....................................................................
ArcSight SmartConnector CEF Syslog Destination Configuration.........................................................................
ArcSight SmartConnector Event Broker / Kafka Destination Configuration.........................................................
ArcSight SmartConnector Aggregation Configuration.........................................................................................
ArcSight Integrator App Installation....................................................................................................................
Prerequisites.......................................................................................................................................................
Installation...........................................................................................................................................................
ArcSight Integrator Features...............................................................................................................................
Summary.............................................................................................................................................................
Deployment Considerations................................................................................................................................
Appendix A: Troubleshooting..............................................................................................................................
Appendix B: Acknowledgment & Dedications.....................................................................................................
Appendix C: ArcSight Integrator Aspirations & Road Map...................................................................................

Document Revisions
Version Date Author Summary
1 1-Apr-18 P. Titov Genesis
2 19-Apr-18 P. Titov Admin queries, revisions, and grammar correction
 3 7-May-18  P. Titov  Added *. conf files for simple implementation
4 7-May-18  P. Titov  Updated for general consumption.
5 14-May-18 P. Titov Updated for additional consumption.
6 23-May-18 P. Titov Updated for enhanced ease of use via app & additional testing results
7 04-Jun-18 P. Titov Updated for TA & SA configuration and aggregation metrics
8 09-Jun-18 P. Titov Grammar corrections, and process update for removed “arcsight” index.
9 02-Jul-18 P. Titov Added Disclaimer & Caveat section.

2
Disclaimer and Caveats
This process is only supported with an ADP ArcSight license, while technically possible without the license it is
operating outside of legacy ArcSight license parameters. Please contact your ArcSight Sales Representative if
you have any questions or concerns. Micro Focus will support ArcSight products which are only under support
contracts; implementation of Kafka (vanilla, Confluent, etc..) is not supported as the Kafka instance is not the
Micro Focus Event Broker.

Description
This guide shows how to integrate full CEF mapping with Splunk via ingesting events from an
ArcSight SmarConnector. The purpose of this document is to improve the user experience with
Splunk with a fully standardized schema, enhance operations efficiency by lowering Splunk
licensing costs, increase Splunk data retention via aggregated events, reduce Splunk hardware
resource utilization, all while simultaneously improving operations’ capabilities. This cross-
pollination of features synthesizes the best of ArcSight and Splunk to provide users with a hybrid
solution that is as flexible and powerful as possible.

This process is built upon the Splunk TA (Technology Add-on) for ArcSight CEF data inputs; it is
available on Splunkbase ( https://splunkbase.splunk.com/app/3694/ ) or via GitHub (
https://github.com/SecretVisons/Splunk_ArcSight/ ).

Please note: Instructions for leveraging Event Broker utilize a deprecated method (as defined by
Splunk). The reason for doing so was to continue to keep this process as simple as possible. The
“deprecated” method is fully configured within the Splunk GUI; whereas the most recent Splunk
Kafka ingestion method relies on the user to make changes via command line. The process to
configure this updated Kafka ingestion method will be captured in later releases of this document.

Benefits of ArcSight Enriched Data within Splunk

Below are two examples, the first is data ingested into Splunk without aggregation via the “eb-other”
topic, the second is with aggregation via the “eb-cef” topic; both are normalized and aligned to the
ArcSight schema. As you will see, by implementing aggregation at our SmartConnector, we are able to
reduce storage & licensing significantly, while increasing efficiency and interoperability.

3
 Aggregated events via the “eb-cef” topic.

Un-aggregated events via the “eb-other” topic.

Without aggregation, our Splunk license is heavily utilized and reaching capacity. On the third day,
you’ll notice that our Splunk license usage was approximately 12.38 GB/day.

4
With aggregation, our Splunk license has a far less impact which empowers more data to be processed
into Splunk. On the third day with this Splunk instance, you’ll notice that our Splunk license usage was
approximately 2.07 GB/day.

Thanks to aggregation via our MicroFocus ArcSight SmartConnectors, our Splunk license reduction was
approximately 83.27%.

5
Another example is produced via Elastic to showcase what a reduction in data aggregation may have
on storage and operations efficiency; from approximately 10:15pm through 9:20am unaggregated
events were ingested then reverted back to aggregated events.

Prior to un-aggregated data being ingested, the event count was approximately 10,000 events at any
given time:

While the un-aggregated event count on the lower end of the spectrum was approximately 180,000
events to 290,000 events at any given time:

6
Prerequisites
• Splunk Version 7.X (Tested on Splunk 7.0 & 7.1, and survived upgrade.)
• An ADP ArcSight environment with at least a single SmartConnector.
• If using Event Broker or Kafka, the Splunk Add-on for Kafka.

o ( https://splunkbase.splunk.com/app/2935/ )

• Admin/root access to both Splunk and ADP ArcSight environments.

7
 ArcSight Integrator TA Installation

o Choose which Technology Add-on is applicable for your environment, if necessary install both.
o Install the ArcSight Integrator TA via the Splunk GUI by click the sprocket next to “Apps” on the
left hand side of your screen.

o Select “Install app from file” and browse to the folder which contains the
Splunk_TA_ArcSight_Integrator_*.zip file downloaded from the Splunk App Store.

o Once complete, restart Splunk.

8
 Splunk Configuration for SmartConnector CEF Syslog Ingestion
o Select Add Data from the settings menu, then select Monitor.

9
o Select TCP / UDP from the input option, ensure TCP is selected, input which port number you will
use, then click NEXT>.

o Select will be enabled by default, start to type in the search field “Smart” which will cause the
“SmartConnector” source type to appear, keep the default App Context “Search & Reporting
(search)” selected and ensure that the applicable index is selected (please create an “arcsight”
index here to leverage pre-built content quickly or segregate your newly onboarded ArcSight
data.); finally, click Review>.
*PLEASE NOTE: This example simulates a pre-existing Splunk environment by creating an
“arcsight” index; while not mandatory, this step is taken to segregate data that may already exist
within Splunk and is not normalized or enriched via an ArcSight SmartConnector.

10
 The ArcSight Integrator app will also leverage the term index=”arcsight” within queries; please
reference Appendix A: Troubleshooting section ArcSight Integrator Content Update for the process
to match your environment if necessary.

(Use of the IP method for hostname resolution is for lowering overall network utilization; this may
not be necessary for your environment.)

o Review the settings and click Submit> to commit the changes.

11
o Congratulations, you have configured Splunk to ingest CEF for the Arcsight Integrator Technology
Add-on.

Splunk Add-on for Kafka Configuration with Event Broker or Kafka

https://splunkbase.splunk.com/app/2935/

o Download the app from the link above & install the app via the Splunk GUI; once installed Splunk
will require a restart.
o Upon restarting Splunk and logging back in, select “Set up now”.

12
o Select “Add Kafka Cluster”, then the “Add Kafka Cluster” window appears, complete the minimal fields here
to save and continue.

o Confirm the Kafka input Global Settings are correct and click “Save” to continue.

13
o From the settings drop down menu, select Add Data button on the top left.

o Select “Monitor”.

o Select “Splunk Add-on for Kafka”.

14
o Complete these necessary fields, toggle “More settings” and ensure that the index is correct (if
you need to create the “arcsight” index, please select “Indexes” from the “Settings” menu and
create it before continuing); then click “Next” to continue.

ArcSight SmartConnector CEF Syslog Destination Configuration

o Add a new destination as CEF Syslog. (In this example, I am using TCP, port 515, CEF 1.0, & not a
forwarder).

15
 ArcSight SmartConnector Event Broker / Kafka Destination Configuration

o Add a new destination as Event Broker. (In this example, I am using the “eb-cef” topic which my
Splunk instance was configured to receive, my content type is compatible with IPv6, and my
Acknowledgment mode is set to “leader”.)

ArcSight SmartConnector Aggregation Configuration

o Implement field-based aggregation by modifying the Connector Parameters of the destination.

16
o Ensure that you are using the sole Default or optionally, modify the parameters for a specific time,
by clicking the pencil next to the “X”.

o Within the “Network” section, disable all name resolution, and ensure the Zone Population Mode
is set to No Zoning (clear). OPTIONAL: Set applicable URI fields to match your environment.

o Scroll down until you see the bolded section “Field Based Aggregation”, set the time interval to
“30 sec”, set the event threshold to “300 events”, field names should be the default, fields to sum
should read “bytesIn,bytesOut”, finally preserve common fields should be set to “Yes”.

17
o Next, move on to the “Processing” section and set Turbo mode to “Faster”.
o Finally, the “Zone Population Mode” should be set to “No Zoning (clear)”.

o Click “Save” at the bottom to commit your changes.

ArcSight Integrator App Installation

Prerequisites
• The ArcSight Integrator TA installed and configured.
• Splunk Version 7.X (Tested on Splunk 7.0 & 7.1, and survived upgrade.)
• An ADP ArcSight environment with at least a single SmartConnector.
• Admin/root access to both Splunk and ADP ArcSight environments.

• The Sankey Diagram - Custom Visualization app installed; this for visualizations which are
contained within the ArcSight Integrator app.

o ( https://splunkbase.splunk.com/app/3112/ )

• The Wordcloud Custom Visualization app installed; this for visualizations which are
contained within the ArcSight Integrator app.

18
 o ( https://splunkbase.splunk.com/app/3212/ )

Installation
• Install the ArcSight Integrator app via the Splunk GUI by click the sprocket next to “Apps”
on the left hand side of your screen.

• Select “Install app from file” and browse to the folder which contains the
ArcSight_Integrator.zip file downloaded from the Splunk App Store.

• Once complete, restart Splunk.

ArcSight Integrator Features

Currently the ArcSight Integrator includes a series of dashboards inspired from Elastic’s ArcSight X-
Pack. There is the additional dashboard for ESM generated events for organizations that wish to
integrate a correlation engine with their Splunk deployment. Current content is quite limited, but
there will be more added as ArcSight content is ported into Splunk.

19
 Summary
Enriched ArcSight data not only saves on storage costs, but also simplifies the analytical process by
standardizing data inputs. The implementation is quite simple and can be accomplished with few
steps. It should be noted that to take advantage of this, you must be on ADP (ArcSight Data
Platform). With ADP, ArcSight enriched data may be shared with any platform.

Higher Fidelity Content

With structured & enriched data now in Splunk, the content which was created in ArcSight can now be
translated into the Splunk Processing Language to leverage a common language. Basic searches and
some pipe functions can even be copied and pasted between Splunk & ArcSight! With the benefits of
normalization and categorization, organizations can now write vendor-agnostic content, which future-
proofs their content library. Combining the strengths of Splunk, with the standardization of ArcSight,
empowers Cyber-Security teams to truly discover a new Micro Focus on their data.

Deployment Considerations
Your current environment will greatly impact how this app is deployed. The following scenarios will
detail suggestions on how to best approach implementation for production environments. In some
cases, it may be beneficial to separate the data streams (e.g. an “arcsight” index); however, this is
ultimately decided by what is best for you, the user.

Appendix A: Troubleshooting

ArcSight Integrator Content Update

o Modify ArcSight Integrator App index contents to reflect your desired index. The files which will
be modified are located:

Linux Installations:
$INSTALL_LOCATION /opt/splunk/etc/apps/ArcSight_Integrator/local/data/ui/views
Microsoft Windows Installations:
$INSTALL_LOCATION\splunk\etc\apps\ArcSight_Integrator\local\data\ui\views

20
o Using your preferred file editor, open each *.xml file and search for:
index=”arcsight”
o Replace the field with your desired index and save; now restart Splunk:

ArcSight Integrator App Update

o When deploying an updated version of the ArcSight Integrator app, simply upload the app like you
have done previous, with the caveat of ensuring the Upgrade app option box is checked.

21
Appendix B: Acknowledgment & Dedications
Some may argue that the iron triad of any cybersecurity endeavor are its people, processes, and technology.
The process and technology is as only as good as the people that use it; which is why if there is an area that YOU
the user would like to see improved, please do not hesitate to contact me at Peter@ArcSight.Me.

Appendix C: ArcSight Integrator Aspirations & Road Map

Information listed here is subject to change without notice.

 Incorporate the ArcSight Activate suite; this could be bundled in an all-inclusive app or
separated components akin to the current model.


22