White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

White Paper

Third Party Software Security Working Group

Appropriate Software Security Control

Types for Third Party Service and
Product Providers

Third Party Software Security Working Group 1

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

2 Third Party Software Security Working Group

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Contents
Working Group Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Mandate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Controls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Summary of Controls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Control Type 1: vBSIMM Process Maturity Assessment . . . . . . . . . . . . . . . . . . . . 6

Practice Area 1: Threat Modeling or Applying Risk Controls
in the Design of Applications and Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Practice Area 2: Code Review. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
Practice Area 3: Security Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Practice Area 4: Manual Penetration Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Practice Area 5: Configuration Management and Incident Response . . . . . . . . . . 10
vBSIMM Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Control Type 2: Binary Static Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Security Policy Selection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Artifacts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Veracode vs. Fortify On Demand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Control Type 3: Policy Management and Enforcement

for Consumption of Open Source Libraries and Components . . . . . . . . . . . . . 15

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Appendix 1: Control Type 1 Through a vBSIMM Case Study . . . . . . . . . . . . . . . 18

Appendix 2: Sample vBSIMM Questionnaire. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

TPRM Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
QA/UAT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Penetration Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Appendix 3: Working Group Member Biographies . . . . . . . . . . . . . . . . . . . . . . . . 34

Third Party Software Security Working Group 1

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Working Group Model

The FS-ISAC Product & Services Committee is responsible for defining, developing and delivering products and
services that solve critical business problems for FS-ISAC members. A common practice used for developing new
products and services is to form a working group of members to focus on the definition and rationale on behalf of
member firms.

According to PWC, detected security incidents have increased 25% this year, while the average financial costs of
incidents are up 18%.1 It is increasingly critical that organizations understand the risk associated with sharing data
with third parties, however few organizations take this step. In fact, only 20% of organizations evaluate the security
of third parties with which they share data or network access more than once a year.2 This trend of ignoring the risk
posed by third parties cannot continue.

For this reason, the FS-ISAC Product & Services Committee asked several member firms to form the Third Party
Software Security Working Group to determine what additional software security control types would be appropri-
ate to add to vendor governance programs. The Third Party Software Security Working Group was established with
a mandate to analyze control options and develop specific recommendations on control types for member firms
to consider adding to their vendor governance programs.

The members that participated included:

Steering Committee Members Working Group Members

1. J
 erry Brady, Morgan Stanley 1. David Smith, Fidelity
2. Mark Connelly, Thomson Reuters 2. Don Elkins, Morgan Stanley
3. Mahi Dontamasetti, DTCC 3. Matt Levine, Goldman Sachs
4. Paul Fulton, Citi 4. David Hubley, Capital One
5. Keith Gordon, Capital One 5. Tim Mathias, Thomson Reuters
6. Royal Hansen, Goldman Sachs 6. Rishikesh Pande, Citi
7. Chauncey Holden, RBS Citizens Bank
8. Rich Jones, JP Morgan Chase
9. Ben Miron, GE Capital
10. Jim Routh, Aetna

PWC Global State of Information Security Survey 2013 – October, 2013

1

2
PWC Global State of Information Security Survey 2013 – October, 2013

2 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Executive Summary
Third party software is the new perimeter for every financial institution. According to Gartner, “since enterprises
are getting better at defending perimeters, attackers are targeting IT supply chains.”3 Further, recent breach
reports such as Verizon’s Data Breach Investigations Report underscore the vulnerability of the application layer,
including third party software. This new perimeter of third party software must be addressed.

Fortunately, the majority of financial services firms and many technology vendors are investing in improving
software security control practices within the lifecycle of software development to provide products and capabili-
ties that are more resilient to attack. Pushing innovation in the marketplace while protecting information assets
exposed in emerging technologies (like mobile computing or cloud services) is a continual challenge and dilemma
for financial services firms. The financial services industry has historically provided leadership in the development
of effective vendor management practices to reduce the risk of exposure of customer and employee information.
Financial institutions have led the implementation of effective governance models for third parties providing IT
products and services for over a decade. Many IT vendors have incorporated prudent risk management controls
into their product development processes as a result.

Evolving vendor governance practices have helped to improve information risk management for firms applying
these standards and for vendors that incorporate these standards into their product development. Codified practices
like the Shared Assessments Program have had a positive effect in encouraging vendors and industry firms to work
collaboratively with the goal of improving information risk management practices and better serving customers and
employees. However, as the financial services industry increases their reliance on third party software services and
products, we are seeing an increase in the number of breaches stemming from software vulnerabilities. This trend
necessitates improved controls operating in concert with vendor management practices to advance the relationship
between security and third party software service providers and commercial off-the-shelf software (COTS) vendors.

Mandate
The mandate of the Third Party Software Security Working Group is to identify control types to incorporate
with vendor governance programs in order to improve information protection capabilities when using third party
services and products in the supply chain for financial institutions’ customers and employees.

Selecting controls to add to a vendor governance program requires collaboration between third party suppliers
and financial institutions. Many software service and product vendors have adopted industry leading practices for
software security embedded in their product development processes such as BSIMM (www.bsimm.com). Several
leading commercial software providers have codified software security practices in an effort to encourage other
providers to adopt leading software security practices through the SAFECode organization (www.safecode.org).

Gartner, “Maverick*Research: Living in a World Without Trust: When IT’s Supply Chain Integrity and Online Infrastructure Get Pwned” October 2012
3

Third Party Software Security Working Group 3

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

It is the responsibility of the financial services industry to make

software security requirements explicit rather than implicit.
This means that a clear set of control requirements needs to be It is the responsibility of the
consistently communicated and applied to specified services and financial services industry
products. By aligning on these control types as an industry, financial
to make software security
institutions can improve the adoption rate for vendors, and ultimately
can promote software security from an outlier request to a standard- requirements explicit rather
ized norm. The objective is to have clearly defined control types than implicit.
that are consistently applied to enable sharing of artifacts between
financial institutions based on vendor acceptance for release of the
artifacts. Software security controls are an integral part of building
high quality software.

If a vendor controls the development and build process, then they

also are responsible for applying appropriate security controls. If a vendor controls the
This is the premise upon which the Working Group built the controls development and build
for addressing third party risk. The Working Group selected three
process, then they also are
control types for adoption by financial services and vendors,
two of which apply directly to third party vendors while the third responsible for applying
applies to the supply chain that financial institutions rely on for appropriate security controls.
software development.

Controls
Each control type was evaluated based on practices that were adopted and implemented by one or more financial
institutions and the institutions’ experiences with the control type. In all three controls, several of the Working Group
members had practical experience with the implementation of the control type for third party vendors. For the
second and third control types there are currently two preferred vendors that offer solutions satisfying the control
requirements defined by the Working Group. The specific vendor solutions were discussed in depth by Working
Group members. During these conversations members shared implementation experiences and evaluated of the
effectiveness of the control type. The Working Group chose not to identify a single vendor solution for each control
type but agreed to share experiences with vendor products with other financial institution members.

4 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Control Type Description

1 vBSIMM process maturity A derivative of BSIMM that specifies selected practice areas of BSIMM specific
assessment to vendor supplied software and uses the vBSIMM activities to determine
process maturity of the product development function of the vendor.

2 Binary static analysis A determination of software vulnerability density for a specific version of
software at a point in time provided through a third party administered process.
This analysis is done against the software’s binaries not the source code.

3 Policy management and This control type identifies consumable open source libraries for a given
enforcement for consumption Financial Institution, identifies the security vulnerabilities by open source
of open source libraries and component and enables the Financial Institution to apply controls or
components governance over the acquisition and use of open source libraries.

Summary of Controls
The Working Group recommends adoption of the three control types for each financial services member firm. The
financial institution will determine how best to implement the control type into its vendor governance and software
acquisition process and which vendor services or products to use.

The first control type, vBSIMM process maturity assessment, does not require member organizations to
implement any vendor product or solution, however the Working Group recommends that financial institutions
participate in education and certification of software security assessment professionals by using education,
training and certification services provided through FS-ISAC. The second control, binary static analysis, and third
control, policy management and enforcement for consumption of open source libraries and components, both
require the use of vendor services or products.

The Working Group does not endorse any of the vendors or products within the control types. The Working
Group believes that each of the three control types are required for financial institution member firms to achieve
third party software security, and will share information on implementation experience to help other financial
institutions with adoption and implementation.

Each of the control types is covered in more depth in the subsequent pages of this paper based on the evaluation and
assessment work completed by the Working Group members. In some cases, vendor management professionals will
need help from software security assessment professionals to effectively introduce the control types to the vendor
management program in their respective financial institutions. In the third control type (open source supply chain
governance) the implementation of the control type will not likely require the involvement of vendor management
professionals. The implementation is more likely to be administered and enforced by application development leads,
software quality assurance professionals or architects with the application development function.

Third Party Software Security Working Group 5

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Control Type 1: vBSIMM Process Maturity

Assessment
Assessing the maturity of controls integrated with the product development process is the primary focus of the
first control type. This is to overall security being dependent on the aggregate assessment of controls working
together as opposed to a single control (e.g. static scanning). The maturity of the security controls is a leading
indicator of the potential for security vulnerabilities in a specific release of software and subsequent releases
of software by the vendor. Much has been published on common practices for improving software quality and
specifically security. Twenty financial intuitions use BSIMM (Build Security in Maturity Model) to benchmark
the maturity of the controls within a development process or program. This is a model of observed activities
or practices that number over one hundred within twelve practice areas. Financial intuitions may use different
technology solutions and practices to improve software security and BSIMM provides for a way of measuring
maturity across different enterprises.

For this reason, BSIMM was considered as a potential source of measuring the software security maturity of vendors
providing products and services to financial institution member firms. Vendor BSIMM (vBSIMM) was derived from
BSIMM activities and practices that apply to third party software development by several financial firms represented
on the Working Group. vBSIMM provides a method for measuring software security maturity across vendors that
use different technical tools, methods and techniques to improve software security maturity. The key is that multiple
controls or “touchpoints” are required in the software development process to achieve maturity.

1
Software security process
maturity assessment

3 2
Security Open Source Binary Dynamic Penetration Web Application
Architecture Review Security Validation Static Scan Scan Testing Firewall

Requirements Development Q/A Production

& Design

Development Q/A
AGILE

Development Q/A

Figure 1: vBSIMM Framework

6 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Practice Area 1: Threat Modeling or Applying Risk Controls in the

Design of Applications and Products
A common misconception among those involved in software
development is that all security vulnerabilities are introduced once
the code is developed. This practice area addresses techniques Threat modeling involves taking
for approaching the risk of building functionality in the design of a data flow diagram of proposed
the application or product. Often significant security flaws in a
functionality and identifying
web or mobile application have their root in a specific design
decision as opposed to errors in coding practices leading to an the methods/techniques that
implementation defect, such as cross-site scripting. Design flaws a malicious attacker (often
often have a far more significant impact than coding flaws in a
referred to as a threat actor)
given application in both severities of the flaw and the associated
cost of remediation. A common technique used to identify and can use to break functionality
address security vulnerabilities in application or product design or compromise credentials
is called threat modeling. Threat modeling involves taking a
(called the attack surface).
data flow diagram of proposed functionality and identifying the
methods/techniques that a malicious attacker (often referred
to as a threat actor) can use to break functionality or
compromise credentials (called the attack surface).

This information can be used to consider adjustments to the application/product design to make the application
more resilient to common attack vectors or methods. Threat modeling is difficult to teach to application designers
and often requires an apprenticeship where one professional learns techniques and methods from another more
experienced professional over time. For vendor products, this type of analysis during design may end up providing
information that may drive changes to product functionality (adding security features to the product).

Decisions on where to store sensitive data and how to protect

them in web and mobile applications offer an excellent example For example, when Apple
of how critical these types of design decisions are from a security developed its mobile map
and privacy perspective. A Working Group member bank noticed functionality it made a
that formal threat modeling was only used in approximately 15% conscious choice to give
of the software vendors and service providers participating in its users control over the use
initial threat modeling program (see Appendix 1 for full case study). of their location information.
However, other techniques for considering security features in the Users must use the provided
application and for determining how to best protect information dialog box to permit their
were applied by vendors and represented a higher maturity for location information to be
this specific practice area. It was common for COTS (commercial accessed. This functionality,
off-the-shelf software) vendors to apply less formal techniques which mitigates the security
for assessing the risk during design discussions, without creating liability of obtaining user
formal artifacts or threat models. The bank’s assessor had to location information, probably
determine which of these activities were consistent with BSIMM evolved from a design discus-
activities and identify ways to evaluate the effectiveness of these sion and ultimately a decision
techniques to assess the practice maturity score. to convert this mitigation into
user experience feature.

Third Party Software Security Working Group 7

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Practice Area 2: Code Review

Code review as a practice is one of the most commonly misinterpreted
practices by COTS vendors. Code review is a well-established process
or technique for improving software quality. Code reviewers will review Code review as a practice is
code looking for the enforcement of development standards, proper one of the most commonly
syntax, techniques applied to comments, the use of specific state-
misinterpreted practices by
ments and input/output validation approaches. The methods of
applying code reviews certainly improve quality but unless the reviewer COTS vendors.
understands software security, these reviews do nothing to improve
the risk caused by security defects. Code resiliency can be improved
through code reviews as long as the reviewer has the requisite
software security skills and the number of developers is small
enough to make this method scalable.

If a development team has fewer than six developers then a manual code review by a security expert is a funda-
mentally sound approach to addressing security defects in the code. If there are 60 developers, then approxi-
mately 10 reviewers with the right level of security skill are required to spot security defects. Finding the required
number of software security professionals to do manual code review gets more difficult the larger the development
team. In practice, development teams over 50 developers need to consider other alternatives to improve security
in the coding process. The use of commercial products to complete automated code reviews (code scanning) for
both quality and security is increasing but it is more prevalent with financial institutions than with COTS software
vendors. The Working Group developed a “rule of thumb” for product development teams over 50 developers: the
vendor needed to acquire and use a commercial product to do code scanning in development to achieve a high
level of maturity for this practice. Using different commercial products to do code scanning across development
teams does not influence control maturity as long as a method for enforcement was in place in each case.

In practice, development teams over 50 developers

need to consider other alternatives to improve
security in the coding process.

8 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Practice Area 3: Security Testing

Methods and techniques for applying security testing varied significantly across vendors. In reviewing an example
of applied vBSIMM, the Working Group saw that almost all of the vendors had some methods or approach in place
for system security testing at a minimum of a level 1 maturity.4 A few of the COTS software vendors demonstrated
a high level of rigor and discipline (i.e.: level 3 maturity) in their security testing process that applied to functional
requirements and security vulnerabilities. Both software development and COTS vendors’ demonstrated matu-
rity in the controls they had in place. In some cases a separate Quality Assurance team handled responsibility for
security testing while in other instances the developers applied testing scripts and evaluated the results with Q/A
reviewers who did additional testing and analysis. Several vendors developed complex testing scripts for testing
access and entitlements for their product. A handful of firms hired other third party firms to do testing and provide
detailed reports on the results. As with other practice areas the financial institution must distinguish security
testing from quality testing techniques in the assessment process.

Dynamic testing tools for security are not a requirement for system testing but they do make it easier to implement
testing procedures for larger development teams. The Working Group found that many of the vendor firms in the
example of applied vBSIMM would capture security vulnerabilities from the testing processes and prioritize the
results based on risk and then assign responsibility for remediation in the bug tracking process.

Many COTS vendors have developed sophisticated processes for defect tracking and management, and some
even feed security testing results into this process. Some vendors do not allow applications with security vulner-
abilities to be promoted into production without approval from a senior officer (e.g. CTO). A few software vendors
include manual penetration testing as part of their security testing process, however most software vendors only
used manual penetration testing services upon client request.

Practice Area 4: Manual Penetration Testing

This was the practice area that was the most familiar to all of the software vendors participating in the example
execution of vBSIMM reviewed by the Working Group.5 Manual Penetration Testing (or pen testing) has become
more of a commodity service with common attributes that do not differentiate the service from one provider to
another. A few vendors with less experience in software security have misinterpreted this practice as a network
pen test performed annually vs. an application pen test conducted in Q/A.

Several vendors had their own ethical hacking teams internally that conducted pen testing of their products and
these teams were always separate from the development teams. Results were often shared with the CTO or head
of development to assign responsibility for remediation. The remediated products were then retested before
release. Often vendors applied pen testing to an entire release prior to production as a standard process.

4
To read the full example, see Appendix 1
To read the full example, see Appendix 1
5

Third Party Software Security Working Group 9

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Practice Area 5: Configuration Management and Incident Response

This practice includes vulnerability management tailored to how the vendor harvests findings about security
identified through one or more of the other practice areas. The vendor prioritizes the remediation effort, assigns
the work and tracks progress. Automation of this process is not required for maturity but having a process that is
applied consistently is required for level 1-3 maturity. COTS suppliers tend to have more mature processes in place
for vulnerability management throughout the lifecycle when compared to the software service providers.

One of the most significant lessons learned in the initial vBSIMM implementations6 was how differently the
contract terms and conditions were interpreted and implemented by various software service providers. For
example, each provider that participating in a sample program (Appendix 1) spent many weeks negotiating a
contract that included a notification clause with the bank; however the interpretation of how to enforce the clause
varied greatly. To understand these differences, the enterprise running this sample program asked participating
vendors many questions about interpreting the requirement to notify customers within a reasonable time frame
of any security incident.

When asked about their definition of what a “reasonable

time period” was for disclosure to a customer of a data
When asked about their definition breach, the most common response was that a customer
of what a “reasonable time period” would only be contacted after a third party cyber security
was for disclosure to a customer forensics vendor was hired to conduct an assessment and
determined that the customer’s data was actually impact-
of a data breach, the most common ed. The implication is that for a vulnerability discovered
response was that a customer would in production of a hosted application, the customer was
only be contacted after a third party not likely to receive notification. In addition, in a significant
incident, the customer would only be notified weeks after
cyber security forensics vendor was the event when a forensics vendor confirmed that the
hired to conduct an assessment and breach impacted data related to the customer. In other
determined that the customer’s data words, the customer would not have visibility into the risk
and therefore would not be able to manage that risk.
was actually impacted.

6
To read the full example, see Appendix 1

10 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

vBSIMM Results
The results of vBSIMM are
delivered as three documents:

• A spreadsheet that is used

to capture the questionnaire
responses

• A spreadsheet used to
document the assessment
results

• Assessment artifacts
provided by the vendor.

An example of a vBSIMM
assessment is provided
to the right. Figure 2: Sample vBSIMM Result

The spreadsheet documenting the assessment results has a column for recording the maturity score of each
practice area. Practice area observations are recorded in another column. The action items are also captured in
the spreadsheet and they are tracked in the vendor management system for follow up in the next review or sooner
when specified.

The highest maturity score is 15 out of 15 (5 practice areas x 3). However one or more practice areas may not apply
to some types of vendors. To normalize the scores across all vendors, the maturity score is divided by the number
of practice areas. The result is a percentage used to measure results across all vendors in different categories.

The Working Group recommends that the completed assessment results spreadsheets7 be shared across all
financial institutions through an information sharing process established through the FS-ISAC portal. This recom-
mendation was made to the FS-ISAC Product & Services Committee and is under consideration. The Product &
Services Committee is preparing a preliminary design of the information sharing process. One design goal is to
include an explicit approval by vendors to share assessment artifacts with all FS-ISAC members. Both the Product
& Services Committee and the Working Group recognize that the ability to share one vBSIMM assessment with
multiple financial institution clients is advantageous for the vendor since the vendor does not have to complete
multiple vBSIMM assessments for multiple financial institutions.

To see a sample assessment spreadsheet, see Appendix 2

7

Third Party Software Security Working Group 11

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

The Working Group also recommends The education would be offered several
that an education and certification program times a year and would be applicable for
be administered by the FS-ISAC to certify vendor governance as well as information
vBSIMM assessors for consistency and security staff from member firms.
quality of the assessment process. The The Product & Services Committee will
FS-ISAC is considering accepting bids from review the need for certification with the
third party vendors to conduct the education potential information sharing capability and
and certification process for financial insti- make a final recommendation to the FS-ISAC
tution members. These vendors must have Board for approval of this control type and
previous knowledge and experience related support requirements.
conducting vBSIMM assessments.

The vBSIMM approach satisfies the need for assessing the process maturity of third party software vendors and
services providers. The challenges with this control are ensuring consistent quality of assessment results and
training vendor governance professionals to understand software development practices well enough to complete
assessments. Each financial institution will manage its own unique vendor assessment practices, and must
determine the appropriate integration of vBSIMM assessments with its existing vendor governance practices.

Control Type 2: Binary Static Analysis

The second control type addresses the need for understanding the vulnerability density of every version of every
product being supplied by third party vendors. Static analysis tools used for software security testing are common
in financial institutions, with several products available for in-house software development. The difference between
those products implemented in the development processes and binary static analysis is that the latter does not
require access to source code. Sharing access to third party source code is problematic for software suppliers as
they need to ensure proper protection of their intellectual property. Binary static scanning provides a method for
determining security vulnerabilities without the need to access to source code, a significant benefit for vendors.

The Working Group considered two leading providers of binary static scanning services—Veracode and HP
Fortify on Demand. The majority of Working Group members are using one or both services today. One vendor
is not recommended over the other. However, a summary of the differences based on Working Group member
experience is provided at the end of this section.

Similar to the detective control of using static analysis in the development process, binary static analysis uses a set
of tests to identify software vulnerabilities, optimizes the results to reduce false/positives, and uses a set of rules to
represent the policy that is enforced.

12 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Security Policy Selection

Choosing the right rule set or policy is the responsibility of the financial institution applying this control. For
example, an enterprise may want to consider specific vulnerabilities that must be detected (i.e.: OWASP top 10)
or a specific regulatory requirements to meet (i.e.: PCI DSS) or may wish to identify a specific vulnerability that
is common in their vendors (custom rule) that they wish to apply. The ability to modify and customize a rule set
or policy is available in both vendor solutions although it is a bit easier to implement using Veracode. The Working
Group recommends using a combination of OWASP Top 10 and PCI compliance as a baseline policy. Additional
rules can be added on top of this baseline as needed.

Artifacts
The Working Group recommends that a number of artifacts be prepared in advance of engaging in static binary
analysis to help the third party software vendors understand and comply with the vendor governance requirement
for binary static scanning including:

1. Letter from the head of the vendor governance function (consider asking the CIO to sign the letter as well).

2. An overview of the binary static scanning process.

3. A defined set of responsibilities for the third party software supplier, the security analysis provider (the
Working Group reviewed Veracode and HP Fortify On Demand) and the enterprise throughout the process.

4. An artifact describing the risk classification definition and the recommended remediation time based
on the severity of the vulnerability.

5. A sample artifact that the third party software supplier receives from the security analysis provider
and a sample summary of what the enterprise receives from the security analysis provider.

6. An acknowledgement of several items:

• That this level of product information (i.e. software vulnerabilities) was not shared in the past.

• A recognition that the vendor’s remediation prioritization must balance the level of risk posed by
discovered vulnerabilities and the resources required to fix defects and create new functionality.

• The enterprise and the vendor need to come to consensus on the right level of urgency for
remediation priorities.

Exposing security vulnerabilities for a specific version of software at a specific point in time provides an opportunity
for a meaningful dialog with the third party software provider regarding remediation priorities. The security analysis
provider managing the static binary analysis will deliver detailed information about defects and vulnerabilities to the
third party software supplier. Together, the security analysis provider and third party software supplier will determine
if there are any additional mitigating controls that may change the risk profile of the vulnerability.

The third party software provider chooses when to release the summary results to the enterprise within a few
weeks. Some third parties choose to remediate vulnerabilities and submit a new release of software after the
original scan. The summary results from the latest scan are then shared with the enterprise.

Third Party Software Security Working Group 13

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

The vendors of binary static analysis make it easy to score the vulnerability results either using a letter grade
score based on the selected policy or with score based on five categories of risk. The most important aspect of this
process is to discuss the findings with the third party software supplier in order to agree on remediation priorities.
This discussion often provides insight into how the third party software vendor deals with software security risk
and remediation prioritization.

Financial Institution Manages Program

Financial institution Financial Third party Third party Security analysis Third party
introduces security institution submits software vendor software vendor provider publishes software vendor
analysis provider new third party accepts scanning uploads software results to third publishes summary
to the third party software vendor requirement for testing party software results to financial
software vendor request form vendor institution

Security analysis
Security analysis provider facilitates On-going support for the third party software vendor
provider creates
process for third party software scanning is provided by the security analysis provider
application profile

Financial institution and third

party software vendor work
ITRC remediation Plan based on
security policy

Figure 3: Process for collecting static binary analysis artifact

Veracode vs. Fortify On Demand

The two security analysis vendors that provide binary static scanning services are similar in their approach and
both capabilities will identify security vulnerabilities in the software that is scanned. The Working Group observed
the following strengths for each security analysis provider based on their track record providing the services over
time to the respective member firms:

Veracode Fortify On Demand

Historical strength in static binary analysis Level of support for systems integrators

Mature cloud-based service offering Access to broad portfolio of application security

testing products and services under HP’s umbrella

Program management service dedicated to building and Scalability to handle large third parties
executing a vendor application security testing program

Approach to vendor application security testing which shifts

the cost of analysis to the third party software provider

14 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Choosing one vendor over the other is up to each

financial institution that applies this control. What is The Working Group was interested in
important to the financial institution should be the the ability to share scan result artifacts
ability to encourage the firm’s third party software across the financial institution commu-
vendors to accept responsibility for applying effec- nity. This implies the hosting of a site
tive controls in the development process that improve to store the scanning artifacts and
resiliency and security of the products and/or systems. provide the third party vendors with
Veracode offers a unique approach to this with their a mechanism for releasing results to
Vendor Application Security Testing (VAST) program. specific financial institutions. Neither
Veracode’s VAST program manages the process of Veracode nor Fortify on Demand has
collecting binary static analysis artifacts, while working the mechanics to support this capability
with software vendors to embed software security in today but could develop this capability
the development process. Additionally, the VAST in the future. This would provide a
program incorporates a shift of responsibility and significant advantage for third party
cost burden onto the third party software vendors over software providers since they deliver
time while also increasing the amount of software in the assessment results to any financial
scope for this control type for the financial intuition. institution at any time based on their
needs without having to scan code for
each respective financial institution.
Migration of responsibility for security assessment from
Also a single remediation prioritization
the financial institution to the third party software vendor
effort and roadmap from the third party
is clearly moving in the right direction for the industry.
software vendor could satisfy the needs
Enabling third party software vendors to share scan
of the entire community.
artifacts with many financial institutions makes adoption
more compelling for the third party software provider.

Control Type 3: Policy Management and

Enforcement for Consumption of Open Source
Libraries and Components
Control Type 3 does not apply to third party software development or COTS vendors. It is included as a control
because it represents how the supply chain is feeding internal software development processes within financial
institutions today. The majority of internal software created by financial services involves acquiring open source
components and libraries to augment custom developed software. The Central Repository (formerly Maven
Repository) is one of the largest open source code repositories. Sonatype’s analysis of this repository estimates
that about 90% of all software development requires the downloading of components.8 Open source code is
available freely and reviewed by many independent developers, but this does not translate into software
components and libraries free from security vulnerabilities.

From Sonatype Press Release dated Sept 9 2013, “The composition of today’s applications is often as high as 90% open source components and
8

10% custom source code (Based on an analysis of the Central Repository and 1000+ Repository and Application Healthcheck Risk Assessments)”

Third Party Software Security Working Group 15

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Aspect Security, a software security consulting vendor, estimates that about 26% of the most common open source
components have high risk vulnerabilities in them.9 The more these open source components are shared, the more
widespread the vulnerabilities become. Therefore, it is essential to have a control to protect the flow of open source
components into the development process.

8,000

7,000

6,000

8 Billion
Requests in Millions

5,000
2012 Open Source Component Requests

4,000

3,000

2,000

1,000

0
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Figure 4: The growth of open source component usage

When application developers seek to build new functionality to meet business needs, they turn to open source
libraries for access to components that dramatically improve the time to market of their delivery. The most
appropriate type of control for addressing the security vulnerabilities in open source, including older versions of
the open source, is one that addresses vulnerabilities before the code is deployed—i.e. by applying policy controls
in the acquisition and use of open source libraries by developers. Therefore a combination of using controlled
internal repositories to provision open source components and blocking the ability to download components
directly from the internet is necessary for managing risk. In fact, Gartner recommends that “if open source is
used, ensure that the frameworks and libraries used are legitimate and up-to-date, and that the compiler used
hasn’t been compromised.”10

There are several technology solutions that address part or most of the needed features to apply lifecycle
management controls for open source components. The Working Group has experience with three solutions that
offer partial functionality. Two of these vendor solutions have been available on the market for more than 5 years
(Palamida and Black Duck) and provide for legal liability as well as security of open source libraries once acquired.

They both have an ability to tag code components and libraries used within an application portfolio. Thus when
new vulnerabilities are discovered, the financial institution can more easily identify the impact of remediation by
understanding where all of the components exist within the application portfolio. This also applies to determining
any legal liability for the use of open source libraries.

9
Aspect Software “The Unfortunate Reality of Insecure Libraries” March 2012
10
Gartner, “Maverick*Research: Living in a World Without Trust: When IT’s Supply Chain Integrity and Online Infrastructure Get Pwned” October 2012

16 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

A new approach in the market is Component Lifecycle Management (CLM) which offers the ability to enforce
policies in the development process. For example, if a development team inadvertently downloads obsolete soft-
ware versions, CLM can apply a method of breaking the build when that library is submitted, enforcing the use of a
more current version. CLM informs the developers and security staff which components have risky vulnerabilities
and which ones do not. The benefits of this approach include:

• Enabling application architects to control versions of software.

• Accelerating the development process by encouraging the consumption of open source libraries
that are resilient.

• Reduce operating costs since the cost of ripping out obsolete components from existing applications
is high assuming the older versions can be identified in the first place.

Financial institutions should consider options in this control type to apply policies to the consumption of open
source components and to specify methods for creating and managing an inventory of open source libraries in
use within the application portfolio. There are manual options and automated options that should be considered
to improve the resiliency of the most commonly used open source components. The controls applied to the con-
sumption of open source are less expensive to implement than fixing defects after they are deployed in production
throughout the application portfolio for the financial institution. An analogy that may apply is the delivery of pure
water through our water systems, regardless of geography, is easier to implement when purification is applied at
the reservoir rather than the downstream canals, pipes and distribution method.

The screenshot to the right

comes from Sonatype CLM
and provides a clear indicator
of risk and impact for an
application portfolio.

The supply chain for software

offers a great deal of choice
and options for improving
time to market for software
development today. Financial
institutions should consider
their options for influencing the
consumption of open source
libraries and provide scanning
capability to identify the most
Figure 5: Dashboard from a CLM product offered by Sonatype
resilient choices.

Firms should also encourage use of mature versions of software that are patched and not yet obsolete by applying
policies and enforcing them using the best methods available. The large consumption rate of open source libraries
for web and mobile applications offers compelling evidence of how time to market has been realized.

Third Party Software Security Working Group 17

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

It is time to apply resiliency controls to the consumption process that will reduce the requirements to fix old
versions with vulnerabilities after they have been deployed. Controls should encourage deployment of current
versions that have been determined to be resilient. Providing more information to architects and developers is
the responsibility of the information security staff. The information should improve the understanding that policy
management applied early in the lifecycle will both cost less effort and speed up time to market in the long run.

Conclusion
Financial institutions must determine their own path for addressing third party software security. While imple-
menting all three recommended controls or even just one will significantly improve the resiliency of the application
portfolio, these controls must be incorporated within existing vendor governance programs in order to achieve
the maximum level of efficacy. When executed correctly, these recommended approaches will increase the
effectiveness of the risk management practices and enable avoidance of expensive remediation post-production.

As member firms better understand the risks associated with sharing critical data and systems with third parties,
the FS-ISAC Product & Services Committee will continue to refine third party software security control types either
through the Third Party Software Security Working Group or another effort.

Appendix 1: Control Type 1 Through a vBSIMM

Case Study
The Working Group looked closely at an example from a leading multinational bank that implemented the vBSIMM
process for vendors. The bank performed vBSIMM assessments for over 18 months and had completed close to
50 vBSIMM artifacts. The firm evolved its implementation approach by incorporating vBSIMM questions into the
vendor questionnaire that was already in use and educating its vendors on how to provide information in the five
distinct practice areas within the vBSIMM. The results from the questionnaires were scored based on the answers
to make it easier for vendor management staff to administer.

The Working Group noted the need to engage information

security professionals with experience conducting software
Financial institutions that have security assessments in addition to the vendor manage-
successfully implemented software ment staff, who could interpret the results from both the
security programs agree that questionnaire responses and the artifacts shared by the
vendor during the assessment process. The bank published
integrating effective controls into a set of principles to apply to the assessment process.
the development process is an Financial institutions that have successfully implemented
iterative process; onboarding a software security programs agree that integrating effective
controls into the development process is an iterative pro-
software vendor into the collabora- cess; onboarding a software vendor into the collaborative
tive process initially outweighed a process initially outweighed a rigorous compliance check.
rigorous compliance check.

18 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

This bank implemented the vBSIMM for the five largest providers of off-shore development services as part of
an initial project. The five firms were asked to volunteer to participate in vBSIMM assessments. Four of the firms
agreed to allow the bank to conduct a vBSIMM assessment. The fifth one decided that the services provided by
their firm to the bank did not qualify for a vBSIMM assessment since they were not responsible for determining the
software development process and instead followed the bank’s software development lifecycle (SDLC). The four
vendors acknowledged that they were aware of software security practices but the practices were not included in
their current projects. Their perception was that the bank would view the practices as additional work and would be
unwilling to pay for the increase in labor to implement the security controls. The vendors agreed to review each of
the five practices and consider methods for implementing controls within each practice area within a few months
and then ask the bank to begin the vBSIMM assessment. The bank approved of this approach and conducted the
vBSIMM assessments by introducing candidate activities for each practice area with each vendor so they under-
stood the activities available based on the BSIMM framework. The vendors then worked on implementation of the
activities they selected.

The bank conducted the assessment

sessions reviewing information prepared The four vendors acknowledged that they were
by each vendor that included a description
aware of software security practices but the
of the activities and selected artifacts from
the activities. The review session took an practices were not included in their current
average of 60 minutes to complete and projects. Their perception was that the bank
the bank provided the risk score (maturity
would view the practices as additional work
level of activities in each practice area) to
the vendor by practice area. The vendors and would be unwilling to pay for the increase
provided the artifacts prior to the session in labor to implement the security controls.
so the assessor was able to review
the artifacts.

The sessions themselves were often conducted as presentations of the activities in each respective practice area
with the assessor asking questions of the vendor. The vendor selected product development or development leaders
and architects to participate in the vBSIMM assessment. In a few cases the vendor assigned an information security
officer to the project to oversee the implementation of the controls and/or review the assessment results. Each
vendor scored at least the minimum score of level 1 for each practice area that applied. A few scored level 2 and
level 3 for select practice areas. All of the vendors identified the appropriate controls by practice area and implement-
ed the controls on current development projects for the bank by the time the initial vBSIMM project was completed.
The vendors found they could implement the controls without changing their billing or rates of the current projects
and they made a commitment to the bank to incorporate these practices in all future assignments for the bank.

The bank included additional types of vendors to the initial project work to evaluate the effectiveness of the
vBSIMM and use the lessons learned to determine if modifications in the approach or techniques was necessary.
The bank selected vendors from two more categories of vendors, commercial off the shelf software (COTS)
vendors and providers that host Software as a Service (SaaS) offerings. They requested volunteers from both
COTS and SaaS vendors and included approximately ten vendors from each category.

Third Party Software Security Working Group 19

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

The results from the COTS category of vendors were quite diverse. This category included vendors that were famil-
iar with and had adopted the BSIMM model so their knowledge of software security was significantly more mature.
However, several larger software vendors chose not to provide assessment artifacts based on advice from their
legal departments. In addition, several smaller COTS vendors were introduced to the BSIMM model through this
initial project and the assessment results indicated very low maturity.

The SaaS providers also demonstrated diverse results with several measuring at very high maturity in the vBSIMM
assessment while others had little or no security controls in their development process and limited understanding
of the activities in the each of the respective practices. Several of these providers found that participation in this
process provided additional value as they learned a great deal about techniques, tools and practices related to
developing secure software.

The bank’s information security and vendor management leaders

reviewed the results of the initial projects very carefully. Each category
of vendors offered interesting learning opportunities for considering Product development
full rollout of the vBSIMM process. The bank recognized that additional or application development
training for the assessors was essential and it was clear vendor gover-
nance staff would likely struggle with the responsibility of conducting
leaders are essential in
the assessment. In all of the initial projects a highly experienced soft- making the vBSIMM
ware security professional led the vBSIMM process for the vendors with assessment process
vendor governance staff observing. One of the lessons that came out of
the initial projects was the need to engage the right resources from the
work effectively.
vendors. Product development or application development leaders are
essential in making the vBSIMM assessment process work effectively.
The vendor account manager can play an important role in facilitating
the introductions to the right technical resources but are not able to
provide useful information to propel the assessment process forward.

The next most significant learning was the need to gather more information from the vendor prior to the assess-
ment session and to make it easier for vendor governance teams to both collect and interpret the information
provided by the vendors. The bank decided to add specific vBSIMM questions by practice area into the vendor
questionnaire11 (often referred to as a Standard Information Gathering tool or SIG) and to score the results from
the responses making it easier for vendor governance professionals to understand how to assess maturity. The
bank anticipated having to include information security professionals that understood software security in the
process but wanted to rely on the vendor governance staff for most of the support work required.

A sample of the questionnaire used by the bank is available in the Appendix 2. For an Excel version of this questionnaire,
10

please contact a Working Group member.

20 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

The bank has implemented the vBSIMM process for selected vendors based on the types of services they offer and
the application risk. For example they apply the vBSIMM for all hosting vendors, for selected COTS products based
on an application risk classification and for service providers that manage the development process following their
own SDLC. The bank provides specific guidance to vendor governance professionals on how to determine the most
appropriate way to apply the vBSIMM when a vendor has different SDLCs by product or where there are different
development teams.

Vendor
Single
Assess-
ment

Recent
U.S. Based Overseas
Acquisition
Development Development
Acquired
Group Group Development

vBSIMM vBSIMM vBSIMM

1 2 3

Figure 6: Diagram of approach using vBSIMM assessment for a software

supplier with multiple products, development regions, and M&A activity.

Some COTS vendors will acquire other products and over time will migrate the development processes for the
acquired company to the acquirer’s core development process. Often the original product development process
is in place for a year or longer post acquisition creating a need for vBSIMM assessment for each product unless
the product development process is identical (same phases, techniques, controls, etc.). Also a services vendor
may use different development centers around the world taking advantage of the market for technical talent (the
same way a bank would) and therefore follow different development practices. In this case a vBSIMM would be
done for each geographic development center since the development practices vary by center.

The lessons learned from the bank’s initial projects were significant in influencing adjustments to the implementa-
tion approach going forward. This included an acknowledgement that a gulf existed between software vendors’
interpretation their responsibilities and the bank’s. The bank has continued the iterative process of refining the
vBSIMM assessment process within their organization and as it applies to their software suppliers.

Third Party Software Security Working Group 21

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Appendix 2: Sample vBSIMM Questionnaire

TPRM Summary
Application Development
Overview (vBSIMM)
Name of the application
What does the application
do for our firm?
Application inventory ID
Risk classification (H, M, L)
Type of application (Web,
Mobile, Client/Server, etc.)
Application development
language(s) (Java, .NET,
iOS, etc.)

Summary Min Score Maturity Score

Architecture
Development
QA/UAT
Penetration Testing
Production

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

Architecture
How do you identify A formalized process is more
the most critical consistent than an arbitrary approach.
applications/products Validate the approach to ensure that
for identifying risk? high risk apps are identified using
sound methodology (are there high
risk apps not being identified?)

Do you perform a security Security evaluations for every major

feature review (authentica- release demonstrates a high level of
tion, access controls, use maturity. Combined with additional
of cryptography, etc.)? security monitoring may be effective
at mitigating risk.

22 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

Architecture
Do perform a secure Security evaluations for every major
architecture design review release demonstrates a high level of
for high risk applications? maturity. Combined with additional
security monitoring may be effective
at mitigating risk.

Do you incorporate threat Failure to assess an application from a

modeling into the business security perspective should not be an
requirements/design acceptable approach.
process of your SDLC?

Less than 6 is poor; higher than 10 is an indication of maturity.

Development
Do you have a list of An affirmative response is indicative of
the most common some level of maturity. Ask questions
vulnerabilities/bugs that to understand what they do and assign
need to be eliminated? a level of maturity.

Do you perform secure Security evaluations for every major

code reviews against the release demonstrates a high level of
entire code base in the maturity. Combined with additional
development phase? security monitoring may be effective
at mitigating risk.

Is there a security expert An affirmative response is indicative of

who performs the review? some level of maturity. Ask questions
(Describe who conducts to understand what they do and assign
the code review.) a level of maturity.

Do you use automated An affirmative response is indicative of

code review tools? some level of maturity. Ask questions
to understand what they do and assign
a level of maturity.

Do you remediate the This demonstrates a risk based

findings? approach by focusing on the highest
risk issues. It assumes that the tools
are robust enough to identify the
critical security defects. Verify and ask
questions related to updates for the
process of identifying security issues.

Do all developers receive Formal secure development training

formal software security is an indication of some level of
training? maturity. Ask for details about the
program; i.e. what coursework is
required vs. optional and the
frequency for this training.

Do you have security An affirmative response is indicative of

experts that work with some level of maturity. Ask questions
developers for every to understand what they do and assign
application? a level of maturity.

Third Party Software Security Working Group 23

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

Development
How many applications do Commercially available code review
you perform secure code analyzers or a 3rd party evaluation
review for annually? service should be used as part of a
comprehensive software security
practice. A dedicated software security
group should be considered to drive/
manage the process. Understand the
process for re-evaluation once initially
identified issues are remediated. Man-
ual code reviews are not sustainable
for a portfolio this size. Low developer
counts (less than 100) could indicate
outsourced development.

Do you outsource any This reduces the risk of external

development? (Provide the development introducing
name of the company and vulnerabilities.
geographic location.)

How many developers Large development shops should use

follow the SDLC under commercially available code review
this review? analyzers or a third party evaluation
service as part of a comprehensive
software security practice. A dedicated
software security group should be
considered to drive/manage the
process. Understand the process for
re-evaluation once initially identified
issues are remediated. Manual code
reviews are not sustainable for a
portfolio this size.

Are the security defects An affirmative response is indicative of

identified being shared with some level of maturity. Ask questions
the developers to prevent to understand what they do and assign
reoccurrence? a level of maturity.

Less than 6 is poor; higher than 14 is an indication of maturity.

QA/UAT
Does your QA function A negative response is indicative
execute edge/boundary of a control gap. Ask questions to
value condition testing? understand what level the vendor
does do in this space and create
an RP if necessary.

Are testing procedures An affirmative response is indicative of

in place to determine some level of maturity. Ask questions
whether security to understand what they do and assign
features are effective? a level of maturity.

24 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

QA/UAT
Do you use dynamic Security evaluations for every major
scanning against web apps release demonstrates a high level of
while in the QA phase? maturity. Combined with additional
security monitoring may be effective
at mitigating risk.

If no, is there any form of A negative response is indicative

black box testing or are of a control gap. Ask questions to
there scripts specific to understand what level the vendor
abuse cases that are used? does do in this space and create
an RP if necessary.

Do you remediate security This demonstrates a risk based

vulnerabilities identified? approach by focusing on the highest
risk issues. It assumes that the tools
are robust enough to identify the
critical security defects. Verify and ask
questions related to updates for the
process of identifying security issues.

Does your QA process An affirmative response is indicative of

involve fuzz testing (small some level of maturity. Ask questions
numbers large numbers, to understand what they do and assign
negative values, binary a level of maturity.
sequences, command line
inputs random values, etc.) ?

How many releases of the Applications should be evaluated

this application occur in a every release. The fewer the releases
calendar year? the less likelihood of new security
issues being introduced.

Less than 6 is poor; higher than 9 is an indication of maturity.

Penetration Testing
How often do you perform Security evaluations for every major
pen testing of applications release demonstrates a high level of
(not perimeter pen testing)? maturity. Combined with additional
security monitoring may be effective
at mitigating risk.

Who performs the pen tests? Using an external vendor is acceptable

assuming the vendor is reputable. Focus
on ensuring the approach the vendor
uses (uses commercial tools combined
with skilled pen testers). External firms
typically keep their testers more current
on emerging threats.

Third Party Software Security Working Group 25

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

Penetration Testing
If internal, are the pen A negative response is indicative
testers part of the of a control gap. Ask questions to
development group? understand what level the vendor
does do in this space and create
an RP if necessary.

Do you use the same An affirmative response is indicative of

approach (tools, methods, some level of maturity. Ask questions to
time spent, etc.) on each understand what they do and assign a
application pen test? level of maturity.

Which applications receive This demonstrates a risk based

penetration testing? approach by focusing on the highest
risk apps. Ignoring medium and
low apps could introduce risk as
variations in the risk ranking
methodology may exist.

Are pen testing results A negative response is indicative

managed through a defect of a control gap. Ask questions to
or vulnerability management understand what level the vendor
system where results are does do in this space and create
assigned for remediation? an RP if necessary.

Do you test the complete An affirmative response is indicative of

production version of the some level of maturity. Ask questions to
application (not just understand what they do and assign a
certain components)? level of maturity.

Do you currently have any Verify/ensure that the production

unremediated Pen Test version was tested and that no
issues in the application vulnerabilities exist.
under review?

Do you pen test applications An affirmative response is indicative of

while authenticated? some level of maturity. Ask questions to
understand what they do and assign a
level of maturity.

Is the pen testing An affirmative response is indicative of

environment production some level of maturity. Ask questions to
or production like? understand what they do and assign a
level of maturity.

Less than 12 is poor; higher than 15 is an indication of maturity.

26 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Results Interpretation Initial Highest Assessor

Response Score Possible Comments
Score

Production
Is vulnerability/security An affirmative response is indicative of
information found in some level of maturity. Ask questions to
operations or production understand what they do and assign a
shared with developers? level of maturity.
Describe how.

If a security breach occurs, Operations will be focused on cor-

which groups have man- recting the issue from an operational
dated involvement? perspective and while helpful from an
RTO perspective, it is important for
Incident response and AppSec be
involved to manage remediation as
an incident and prevent the issue
from reoccurring.

Does the incident response An affirmative response is indicative of

process include steps to some level of maturity. Ask questions to
identify root/cause and understand what they do and assign a
prevent reoccurrence? level of maturity.
Describe what.

If you host applications A negative response is indicative

for JPMC, is there a of a control gap. Ask questions to
service in place to monitor understand what level the vendor
production applications does do in this space and create
for vulnerabilities? an RP if necessary.

Less than 6 is poor; higher than 7 is an indication of maturity.

Earlier in the lifecycle, preventative controls are most effective. As any application migrates to production, detective
controls become more important. Some vendors may rely entirely on Pen Testing as a SDLC control. While this can be
effective in the detection of vulnerabilities, it does nothing to prevent issues from being reintroduced (unless shared with
the developers who introduced the issue). Understanding who performs the pen test is important (are they qualified?).
A higher number of releases amplify the need for detective controls. Mature programs will contain a mixture of preventa-
tive and detective controls for every release ensuring that developer education is addressed. Vendor attestation is never
enough; always verify artifacts that support vendor responses. Lack of artifacts or practices may require a “point in time
assessment” to measure the security posture of a given application. Architecture focuses primarily on preventative
controls. Applications from those vendors that score less than the min mature score in development, QA/UAT, and
Pen Testing may require RPs and/or point in time assessments (such as Binary, Dynamic or a Pen Test).

A Word About AGILE Development

AGILE development focuses on short “sprints” of development which usually last several weeks. In this method
of development, engraining detective controls becomes more difficult (but not impossible). Preventative controls
are of greater importance when using this method. Ensure that detective software security controls are in place
and are used prior to the application migrating toward development. For example, not all “sprints” are released
to production immediately. In this case static analysis can be integrated into the development cycle and dynamic
may be used in the last release before production. AGILE requires more flexibility but understand that the use
of AGILE is not an excuse not to apply controls.

Third Party Software Security Working Group 27

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Architecture
AA1.1 Activity Response IRM
Comment
Threat modeling allows you to systematically Architecture
identify and rate the threats that are most Analysis Activity
likely to affect your system. By identifying and
Perform security
rating threats based on a solid understanding
design/architecture/
of the architecture and implementation of
feature review
your application, you can address threats
with appropriate countermeasures in a logical
order, starting with the threats that present
the greatest risk.
Threat modeling has a structured approach
that is far more cost efficient and effective
than applying security features in a haphaz-
ard manner without knowing precisely what
threats each feature is supposed to address.

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Architecture
How do you identify the most critical
applications/products for identifying risk?

Do you perform a security feature review

(authentication, access controls, use of
cryptography, etc.)?

Do perform a secure architecture design

review for high risk applications?

Do you incorporate threat modeling into the

business requirements/design process of
your SDLC?

28 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Development
CR1.4 Activity Response IRM
Comment
Source code review is one of the critical Code Review Activity
controls. Security code reviews focus on
Use automated
identifying insecure coding techniques
tools along with
and vulnerabilities that could lead to security
manual review
issues. The cost and effort of fixing security
flaws at development time is far less
than fixing them later in the product
deployment cycle.
The use of an automated tool demonstrates
maturity in the practice since the tools are
much more mature today and make the
review process more consistent. Managing
false/positives from a source code tool is
necessary for large scale development work
and requires expertise and effective practices.
For example, using a process or function to
interpret vulnerability information or reducing
the number of rules in the baseline rule set are
both techniques for managing false/positives.
Using a manual code review process for a
small team may be effective as long as there is
an experienced software security professional
conducting the review. Manual code review is
required for platforms not covered through
source code static analysis tools.

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Development
Do you have a list of the most common
vulnerabilities/bugs that need to be
eliminated?

Do you perform secure code reviews

against the entire code base in the
development phase?

Is there a security expert who performs

the review? (Describe who conducts the
code review.)

Do you use automated code review tools?

Do you remediate the findings?
Do all developers receive formal software
security training?

Do you have security experts that work with

developers for every application?

Third Party Software Security Working Group 29

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Development
How many applications do you perform
secure code review for annually?
Do you outsource any development?
(Provide the name of the company and
geographic location.)

How many developers follow the SDLC

under this review?
Are the security defects identified being
shared with the developers to prevent
reoccurrence?

QA/UAT
ST1.1 Activity Response IRM
Comment
The QA team goes beyond functional testing Security Testing
to perform basic adversarial tests. They Activity
probe simple edge cases and boundary
Ensure QA supports
conditions and no attacker skills required to
edge/boundary
do this. A minimalistic practice is to conduct
value condition
specific tests designed to uncover potential
testing.
input/output vulnerabilities in an application.
Test scripts used or output of tests designed
to do edge/boundary condition testing may
be considered.

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

QA/UAT
Does your QA function execute edge/
boundary value condition testing?

Are testing procedures in place to determine

whether security features are effective?

If yes, are the procedures derived by obtaining

a list of security features implemented by the
architecture group?

Do you use dynamic scanning against web

apps while in the QA phase?

30 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

QA/UAT
If no, is there any form of black box testing or
are there scripts specific to abuse cases that
are used?

Do you remediate security vulnerabilities

identified?
5. Does your QA process involve fuzz testing
(small numbers large numbers, negative
values, binary sequences, command line
inputs random values, etc.)?

How many releases of the application occur

in a calendar year?

Penetration Testing
PT1.1 Activity Response IRM
Comment
Penetration Testing is a conventional security Penetration Testing
control and the one most widely used by Activity
software vendors.
Use penetration
testers to find
problems.

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Penetration Testing
How often do you perform pen testing of
applications (not perimeter pen testing)?
Who performs the pen tests?
If internal, are the pen testers part of the
development group?
Do you use the same approach (tools,
methods, time spent, etc.) on each
application pen test?

Which applications receive penetration testing?

Are pen testing results managed through a
defect or vulnerability management system
where results are assigned for remediation?

Third Party Software Security Working Group 31

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Penetration Testing
Do you test the complete production version of
the application (not just certain components)?
Do you currently have any unremediated Pen
Test issues in the application under review?
Do you pen test applications while
authenticated?
Is the pen testing environment production
or production like?

Production
CMVM1.1 Activity Response IRM
Comment
This is often an initial point of identification Configuration
of software vulnerabilities for less mature Management—
software security programs. When an incident Incident Response/
is identified, what process is used to address Vulnerability
the incident and what is the notification Management
process with clients. Does the incident
response process drive prevention activities?

Vendor Describe the If your process Additional

Response procedure, is not in the Comments
process and list, describe
tools used what you do

Production
Is vulnerability/security information found
in operations or production shared with
developers? (Describe how.)

If a security breach occurs, which groups

have mandated involvement?

Does the incident response process include

steps to identify root/cause and prevent
reoccurrence? (Describe what.)

If you host applications for JPMC, is there

a service in place to monitor production
applications for vulnerabilities?

32 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Conclusion
Application Development
Overview (vBSIMM)
Name of the application
What does the application
do for our firm?
ID
Risk Ranking (H, M, L)
Type of application (Web,
Mobile, Client/Server, etc.)
Application development
language(s) (Java, .NET,
iOS, etc.)

Assessment Date/Location

Enterprise Assessor(s)

Conclusion

Artifacts Reviewed

RPs to be Created

Third Party Software Security Working Group 33

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Appendix 3: Working Group Member Biographies

Mark Connelly Mahi Dontamsetti

Thomson Reuters Global Head of IT Risk & Application Security, DTCC

Mark Connelly is Chief Information Security Officer Mahi Dontamsetti is Global head of IT Risk and
for Thomson Reuters, reporting to James Powell, Application Security at Depository Trust and Clearing
the company’s CTO. Mark joined the company on Corporation (DTCC). Mahi joined DTCC from Barclays
February 15, and is responsible for establishing Capital, where he was the Global Head of Access Man-
and maintaining a corporate- wide information risk agement & Entitlements. He has served on the board
management program; identifying, evaluating and of OWASP NY/NJ chapter and is author of several
reporting on information security risks that meet books on Wireless and Information Security. He has a
compliance and regulatory requirements; and M.S. in Computer Science and Telecommunications.
working with the business units to implement
practices that meet defined policies and standards
for information security.
Paul Fulton, CISSP
Citigroup
Mark most recently served as Chief Information
Security Officer for ITT Corporation. Prior to that, Paul Fulton is head of Information Security Core
he was Managing Director, Information Technology Services at Citi. He oversees several global security
Risk and Security for Credit Suisse and Vice President programs including the Data Protection, Secure
and Chief Information Security Officer for Sun Micro- Development Lifecycle, Third Party Assessment
systems. He began his career in systems engineering, and Application Security Management functions.
progressing through a variety of management roles in Previously Paul has held management positions
technical operations and information technology. in information security at UBS, Deutsche Bank,
and JP Morgan. His IT career started in application
Mark earned his Master of Science degree in Electrical development for trading floor systems and he
Engineering from Washington University in St. Louis, has held positions managing security architecture,
Missouri. He earned his Master of Arts degree in Micro- security infrastructure deployment and as lead
biology from the University of Missouri. His Bachelor information security officer for major business units.
of Arts degree is also from Washington University. As a
Certified Information Security Manager, he also holds
certifications in Governance of Enterprise IT and in
Risk and Information Systems Control.

He resides in Connecticut with his wife and two sons.

34 Third Party Software Security Working Group

 White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Keith Gordon Richard Jones

Vice President, Information Security & Risk JP Morgan Chase
Management, Capital One
Richard Jones is currently an Executive Director
Keith Gordon is VP of Information Security, with JP Morgan Chase where he has been employed
Customer Protection & Risk Management for Capital for over 15 years and is currently responsible for
One Financial Corporation. In this role, he is respon- managing third party risk for the consumer lines of
sible for directing a diverse team that manages and businesses. Collectively, Richard has over 23 years of
executes against our Enterprise Information Security experience in a variety of disciplines including finance,
and Customer Protection Strategy while balancing program management, analytics, software security,
and managing Risk Management for Information and technology. Richard is also a Certified Public
Technology. Accountant in the state of Pennsylvania.

A diversified Fortune 200 company headquartered During his tenure with JP Morgan Chase, Richard
in McLean, Virginia, Capital One is the ninth largest has had significant participation in several strategic
bank in the United States, based on deposits, and initiatives including the deployment of a global
one of the most widely recognized brands in America. application security program. Also noteworthy is
his participation in deploying one of the first online
Before joining Capital One, Mr. Gordon was an Ex- banking solutions (Wingspanbank.com), and the
ecutive with Bank of America leading the Security, design of a contactless payments solution focused
Authentication, Identity and Fraud team. Prior to Bank on the transit industry, which led to a U.S. patent
of America, Mr. Gordon held several technology and being granted for one of his designs.
eCommerce related positions with both Fortune 500
and small firms. Richard is active in a variety of civic and cultural
organizations including Habitat for Humanity and
Mr. Gordon has been a resident of Charlotte, participation on several community boards. Outside
North Carolina for over 10 years but commutes of professional interests, he enjoys restoring classic
to Richmond, Virginia for his role with Capital One. automobiles and scuba diving.
A native of Indianapolis, Indiana, he completed his
undergraduate degrees in Marketing and Mathemat-
ics at Anderson University in Anderson, Indiana.

Third Party Software Security Working Group 35

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Tim Mathias Jim is the winner of the 2009 BITS Leadership

Vice President, Information Security Architecture Award for outstanding leadership of the Supply Chain
& Engineering Working Group sponsored by the financial industry
in collaboration with NIST and the Department of
Tim has over 25 year’s experience in Information Treasury. He was the 2007 Information Security
Technology with the past 10 focusing on Information Executive of the Year for the Northeast and is a
Security. He has been at Thomson Reuters over 15 widely recognized expert in security program
years and has held a broad range of positions including implementation. Jim and Ellen have three sons
Network Manager, Wide Area Network Architect, and reside in New Jersey.
Data Center Architect and CISO of the Thomson
Financial Division of the former Thomson Corporation.
Tim conceived and built the Application Assurance
David Smith
program at Thomson Reuters from the ground up
VP Fidelity Investments, Application Security,
which provides code scanning and application security
Fidelity Investments
training for over 7000 technologists at Thomson
Reuters. His team provides Application Security David Smith is Vice President of Application Security,
consulting and Security Architecture oversight for all High Risk Asset Protection and DDoS Testing. He is
Thomson Reuters products. Prior to Thomson Reuters, responsible for enterprise wide application security
Tim has held positions at the Nuclear Regulatory programs including Secure Code Review, Penetra-
Commission, Sprint, and Laventhal & Horwath a CPA tion Testing, Application Security Tool management,
firm specializing in providing consulting services to Application Security Architecture, Developing Secure
the US Federal Government. Application training, Malicious Code Detection,
Security Solutions consulting and Emerging Vulner-
ability management. David has been with Fidelity
Jim Routh, CISM, CSSLP for 17 years and has been working in the Information
CISO, Aetna Security field for 23 years, with previous experience
at US Army INFOSEC Division, NSA, Booz Allen &
Jim Routh is the Chief Information Security Officer Hamilton, and FTP Soft.
and leads the Global Information Security function
for Aetna. He is the Chairman of the FS-ISAC
Products & Services Committee. He is currently a
board member of the National Health-ISAC. He was
formerly the Global Head of Application & Mobile
Security for JP Morgan Chase. Prior to that he was
the CISO for KPMG, DTCC and American Express
and has over 20 years of experience in information
technology and information security as a practitioner,
management consultant and leader of technology,
analytic and information security functions for
global financial service firms.

36 Third Party Software Security Working Group

White Paper Appropriate Software Security Control Types for Third Party Service and Product Providers

Third Party Software Security Working Group 37

www.fsisac.com