Effectively

Benchmarking
your Phishing
Awareness
Program
When orchestrating any
security awareness program,
it is only natural to try to gauge an organization’s overall security position against
a jury of one’s peers, so to speak – that is, measure success against comparable
organizations and programs. The growing need to benchmark these programs
has created a bit of a flurry over benchmarking methodologies of late within
the phishing arena, particularly given the absence of industry standards on
everything from program maturity to awareness tools and phishing templates.

This eBook addresses benchmarking and examines what to keep in mind when
using external or internal benchmarks to assess your phishing program. We
will cover which variables to consider while creating benchmarks, and what
to look for when comparing your results across industries. We’ll also cover
global and internal benchmarking as well as concepts such as tiering.
01 What Is
Benchmarking?
What is a benchmark, anyway? And how many variables
need to be similar for the benchmark to be valid, or
even make sense? According to Bernard Marr & Co.,
benchmarks are reference points that can be used to
compare your performance against the performance
of others, typically across internal business entities or
external competitors. Business processes, procedures, and
performance analytics are compared taking into account the
best practices and statistics from similar organizations.

While there are several types of benchmarks, phishing

falls under performance benchmarking, which uses
performance metrics. This type of benchmarking involves
comparing not only against like companies or competitors
in similar industries, but also making global comparisons
regardless of industry. It can even involve internal
benchmarking across your own company, such as comparing
different departments, regions, or business units.
Phishing simulations are used by many Some variables to consider when comparing the analytical
companies across all industries as a key details of phishing benchmarking include:
cyber training tactic to teach people how
• The representative sample size of the workforce
to better identify and stop phishing attacks
where adversaries use deception to gather • The history and length of the phishing program
sensitive and personal information. Phishing
• The difficulty of the simulation and how many indicators are present
simulations generally measure the undesired
• The experience of the simulation (link/attachment/credential request)
action rate (click rate) and the report
rate (number of reports generated). • The topic and how relative it might be to all participants

• When the phishing simulation was sent

Most Security Awareness teams are interested
(day/time based on location)
in how their phishing program metrics
compare to similar businesses, and it can • The ease of reporting and reporting options
be tempting to place too much emphasis on
• The availability and diversity of training
performance benchmarking. While identifying
and awareness materials
areas of improvement based on comparable
• “Time in band/onboard” of employees
industries is important, it is also important
(for example, a heavy influx of new
to track multiple variables using phishing
employees might be impactful)
simulations and results as a primary driver.
• Other relative demographics of the workforce

• The overall maturity of the Security

Awareness Program or effort
Program Maturity
When benchmarking, it is important to consider your phishing program in relation to your larger security awareness efforts and
the maturity of your program as a whole. For example, if your phishing program lands on the higher end of the SANS Security
Awareness Maturity Model® – that is, you have previously promoted cyber awareness tactics and experienced long-term sustainment
and culture change – then your phishing assessment results, even with a similar phishing simulation, might be very different from
those of a similar organization with a less mature program still in the compliance-focused phase of the model. The workforce
under the compliance-focused phase will have had less time to use teachable moments and safe cyber awareness tactics.

For more information about this maturity model, visit https://go.sans.org/lp-ebook-maturity-model.

Non-existent Compliance- Promoting Awareness Long-term Sustainment Strategic Metrics

focused & Behavior Change & Culture Change Framework
Simulation
Difficulty
Another variable to consider is the difficulty of the
simulations. If the simulations across each business or
organization are identical or very comparable, the statistics
will be more valid. If the simulations you are using for
benchmarking vary in difficulty, the benchmark metric most
likely will be skewed. This is why more mature phishing
programs often use a concept called tiering, which digs deeper
into the nature of the simulations and indicators used.

While benchmarking has its place and can be used to gain

support from leadership, care must be taken to ensure
that the benchmarking is of “apples to apples” wherever
possible. Benchmarking is a worthwhile assessment tool
only if you have the capacity to consider all the factors and
variables. Otherwise too much emphasis on benchmark
statistics – instead of on more impactful metrics such as
the report and click rates – can drive the wrong behavior.

Together, program maturity and strategic tiering will

provide the baseline measurements necessary to
perform meaningful benchmarking. The next chapter
in this eBook will explore tiering in more detail.
02 How to Effectively
Use Phishing
Benchmarks
to Assess Your
Security Awareness
Program: The SANS
Tiering Model
Chapter 1 introduced the concept of benchmarking in your phishing
programs, the importance of selecting the correct metrics to track,
and how those metrics relate to your progress on your organization’s
maturity model. This chapter will address tiering, which allows for
a more valuable assessment of your simulation data analysis, or an
“apples to apples” comparison. The tiering concept identifies the
difficulty of your simulations, how many indicators are involved, and
how hard it is to identify and recognize these indicators as a phish.
The SANS tiering model has five levels. Each tier has three familiar. This could be a non-business-related shipping notice in the
specific areas: audience, experience, and simulation keys. While Tier 2 block, or a Zoom business-related simulation within Tier 3.
the audience and experience might vary from tier to tier,
Tier 4 includes personalization and some trust and prior
the simulation keys are what truly separate the tiers.
knowledge, such as program status. Tier 5 simulations become
As shown in the diagram below, a Tier 1 simulation would be very very personalized and highly targeted to specific individuals and
“spammy” in nature, easy to recognize as a phish, and impersonal. executive leaders. Assessing these simulations requires additional
As you move up in the tiering, the indicators are more difficult to resources, and such assessments typically are separate events
spot, with brands or memo styling with which the workforce may be outside most simulation schedules and benchmark data.

TIER 01 TIER 02 TIER 03 TIER 04 TIER 05

Audience Audience Audience Audience Audience
Targets All Users Targets All Users Targets All Users Specific Targets High Profile People

Researched Targets

Experience Experience Experience Experience Experience

Drive-by/Link Drive-by/Link Drive-by/Link Drive-by/Link Drive-by/Link
USB Drop USB Drop USB Drop USB Drop USB Drop

Attachment Attachment Attachment Attachment Attachment

Data Entry Data Entry Data Entry Data Entry Data Entry
Reply-to

Simulation Key Simulation Key Simulation Key Simulation Key Simulation Key
Impersonal Recognizable Branding Recognizable Style Personalized Highly Personalized

Misspelled Disjointed Domains Conveys Some Trust Conveys Trust Customized

Poor Grammar Not Business-related Business-related Conveys Prior Knowledge Cloned Email or Spoofing

No Signs of Prior Knowledge Some Personalization Conveys Personal Trust

As can be seen in the sample Tier 1 simulation below, the indicators are easy to pinpoint, and should be quickly identifiable by a well-trained workforce.

There is no trust, or even

New Message
2 knowledge of, sender

A Friend Sent You an Ecard!

ET Ecard Team <eCard@daily-winner.net>

There is no specific
To: Ecard User 1 addressee in the “To:” line

A friend just sent you an ecard from e-hugs-online.com

Spelling or grammatical
You can view it by clicking heer: 3 errors are present
This was so funny!

Using our new tracking feature, you can now view all the ecards received by you in the last 30 days.

Your ecard is going to be with us for the next 30 days.

Lacks recognizable logo or

4 internal company format
Tiering is especially important when benchmarking historical simulation assessments and select those
with external entities. Benchmarking is often easier that fit the selected tier. When each organization
said than done, since so many variables impact in an industry shares metrics within the given
the validity of the results. Many of these variables, tier, the metrics are far more impactful than
such as the demographics of the workforce or metrics across all tiers, or all difficulty levels.
the simulation schedule, are easy to identify.
The ideal situation involves collaboration with
However, identifying the level of difficulty of the
similar industries and cybersecurity programs
phish is harder to determine because it can be
– that is, sending the identical simulation to
dependent on vendor tools and how simulations
a represented sample size. If the workforce
are labeled based on difficulty, if identified at all.
demographics are comparable, then the undesired
With such a wide range of organizations now using action rate, along with the rate of employees
phishing assessments as an awareness tool, it has reporting the phish, will provide a trustworthy
become easier to identify similar industries within a and actionable benchmark if indicated.
given vertical. It can be tempting to further narrow
Communication and cooperation across the external
this down by size and even workforce maturity,
entities is crucial, but it may be difficult based on
though, typically, specifics and similarities with
the typical workload of the security awareness
regard to the actual simulation data are hard
officer and operational teams. Carving out the time
to compare. Using a tiering model provides the
and resources to execute phishing simulations,
ability to better “bucket” simulations, ensuring that
and establishing strategic priorities, could drive
the benchmark data are as viable as possible.
the team to rely on historical analytics with tiering
For example, if an organization requires the considerations. In this situation, the team could
assessment of its overall security position via- rely on bucketing the simulations in a like tier,
á-vis comparable organizations, it could look at reducing the resource and scheduling impact.
03 Internal
Benchmarking
The final chapter in this eBook focuses on internal
benchmarking, which involves comparing different entities
within your company or enterprise. Typically, the goal of
internal benchmarking is to gain a better understanding
of how entities in your company are doing. With phishing
simulations, you can analyze the results to determine which
entities are responding well to phishing simulations, and
which may need additional work in terms of awareness.

Internal benchmarking will be highly dependent on

your company structure, the layout of your organization,
departments, business units, regions, and teleworking, access
to awareness and training materials, and even “time in band,”
that is, how long employees have been in their current role.
At first, internal benchmarking may seem However, other factors and tactical aspects
easier to tackle than external comparisons. may require thought, strategic planning, and
It uses a common set of characteristics that historical trending (if available), including:
most likely are similar across the company
• How relative the simulation
or can be planned accordingly, including:
might be to participants
• Overall position within the
• The organization and distribution of
SANS Maturity Model®
departments and roles across the business
• Representative sample size
• The availability, variety, and distribution
of the workforce
of training and awareness materials
• History and length of the
• Leadership, management, and
phishing program
stakeholder engagement
• When the simulation was sent
As with external benchmarking against
(day/time based on location)
other organizations, comparing “apples
• Difficulty and experience of the
to apples” is important when validating
simulation, and how many indicators
the results of internal benchmarking. The
are present (providing a lower
previous chapter addressed tiering in order
tier and not based on roles)
to ensure the validity of benchmarks. When
• Ease of reporting and reporting options using benchmarks as an analytic tool to
evaluate your phishing program, the analytics
should look at the difficulty of the phish,
how many indicators present themselves,
and how hard it is to distinguish, isolate,
and identify something as a phish.
Based on the tiering concept, we can see that while
one simulation might fit all employees within the
company, a more advanced simulation may not.
A Tier 1 or 2 simulation would be appropriate for
the entire workforce, but typically not targeted
to a specific business, role, or skill. Tier 3 or 4
simulations may not be appropriate for the entire
Subject: [EXTERNAL] Office 365 Error: Mandatory Software Update
workforce. These more sophisticated options
may only apply to specific groups or roles across
From: IT Department <itsupport@secure-monitor.com>
the organization and are more business-related,
while often conveying trust or prior knowledge.
IT Date: Monday, November 8, at 5:12 PM

To: "King, Tonya" <tking@examplecompany.org>

The example below is appropriate for employees
with company-issued laptops and could result We attempted to update your Office 365 Application suite but encountered active toolsets.
Ensure your company issued laptop is powered down by COB tonight, since the upgrade
in a higher undesired action rate than those
will improve many of our business solutions. Please become familiar with those new to
employees without company-issued devices. It you, with helpful information within the overview below.
would also hopefully result in a higher report rate. We have included a personal license for the first 100 families that submit the agreement
you may find here.

Kind Regards,
Application Admin
In all cases, it is imperative for the workforce
to have had exposure to security awareness
and training materials. This would not be
appropriate if there were many new employees
who were just issued a new device and had
yet to receive awareness materials.

While the prior example was based on a situation, Subject: Authentication Confirmation

the phish shown below is aimed at a specific

From: O365 Elevated Privileges <O365 Authentication@msupdate.net>
role. The employees targeted here hold elevated
and advanced credentials or authorizations. EP Date: Sunday, April 25, at 7:17 AM

To: "King, Tonya" <tking@examplecompany.org>

As a key player with elevated privileges, the O365 Security / Email Team now requires
quarterly confirmation of your authentication details. Please access your account information
as soon as possible, you will receive verification once endorsed.

Confirm Your Identity with your User ID

Name: {fname} {lname} User ID: {email}
Sign in to comfirm authentication details
Thank you, The Microsft O365 Security Team
Another factor to consider when measuring a phishing program is the Internal benchmarking is a valuable tool: comparing metrics
diversification and distribution of awareness and training materials. If allows you to leverage those findings to entice the workforce
you have a wide range of demographics, ensure that your teachable into being more aware while developing the skills to recognize
moments are meaningful and understandable across your workforce. and report a phish. Some options to compare include:
We all learn and consume information in different ways – some by
• Corporate-wide simulations that apply to everyone, Tier 1 or 2
print, others through videos and interactive options. Also consider,
if applicable, the different physical locations and cultures of your • Similar roles across business entities, including IT
benchmark groups. On-site availability of awareness and training administration, administrative assistants, developers, human
resources might be more diverse than for a totally remote workforce resource business partners , and even new employees
or even a hybrid setup. Your benchmark data will be more valid if
• Comparable departments such as Finance and Accounting,
you’ve incorporated diverse awareness and training methodologies.
Research and Development, and Compliance

How your stakeholder, management, and leadership teams handle • Parallel management levels, such as directors or associates
security awareness is also a factor to consider when assessing internal
Your security teams can also provide suggestions on target groups to
benchmarking data. The hope is that all levels of managers and
benchmark, as those teams often have the resources to pinpoint the
stakeholders – including Human Resources, Communications, or IT –
most vulnerable employees. In addition, your historical data derived
recognize the importance of improving and enhancing cybersecurity and
from your phishing program can uncover weak departments, teams, and
social engineering skills. However, the background of leaders, organizational
other areas that would benefit from additional training opportunities.
priorities, and limited resources can impact that focus. A remote field
supervisor might feel differently about proper cyber behavior than an
IT leader, and dedicated program managers faced with tight schedule
deadlines may not promote security awareness as much as they should.
CONCLUSION
How to Benchmark
Your Security
Awareness Program
Benchmarking is an important undertaking for many facets of an
organization. Your cybersecurity awareness program is no exception,
as this eBook has shown. This is especially true if your program
is not delivering expected results. Benchmarks allow you to more
easily identify areas to reduce human risk to manageable levels.
While it is tempting to quickly compare your program to those of
competitor organizations based on intuition or limited research, the
pragmatic, step-by-step approach outlined in this eBook will drive
more meaningful results and a more successful phishing program.

For more information, please visit sans.org/awareness.