Revision: 6.

0 Document No: OCFO-SID-0002

Release Date: May 23, 2017 Page 1 of 59
Title: SRB Programmatic Assessment Process

National Aeronautics and SOPI 6.0

Space Administration Headquarters
Office of the Chief Financial Officer (OCFO) RELEASE DATE
Strategic Investment Division (SID) May 23, 2017

STANDARD OPERATING PROCEDURE INSTRUCTION

SOPI 6.0

SRB PROGRAMMATIC ASSESSMENT PROCESS

Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 2 of 59
Title: SRB Programmatic Assessment Process
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 3 of 59
Title: SRB Programmatic Assessment Process

TABLE OF CONTENTS
1.0 PURPOSE..............................................................................................................................4
1.1 Prerequisites .................................................................................................................... 4
2.0 REFERENCEs..........................................................................................................................4
3.0 SCOPE ..................................................................................................................................5
4.0 SRB Programmatic Team Structure...........................................................................................6
4.1 SRB Programmatic Team Lead............................................................................................. 7
4.2 Schedule Analyst ............................................................................................................... 7
4.3 Cost Analyst ..................................................................................................................... 7
5.0 SRB Programmatic Team Role..................................................................................................8
6.0 Independent Programmatic Assessment LCR Workflow ...............................................................8
6.1 Life Cycle Review (LCR) Planning.......................................................................................... 9
6.2 LCR Analysis and Feedback ................................................................................................11
7.0 Data Drops ......................................................................................................................... 17
8.0 Tailoring ............................................................................................................................. 18
9.0 LCR ToR.............................................................................................................................. 18
10.0 Dissenting Opinions............................................................................................................ 18
11.0 Requirements Tailoring....................................................................................................... 19
11.1 Program Project Management Board (PPMB) .....................................................................19
11.2 Compliance Matrix .........................................................................................................19
12.0 Cost Assessment ................................................................................................................ 19
12.1 Cost Requirements Review Process ...................................................................................20
12.2 Cost Assessment Process .................................................................................................21
13.0 Schedule Assessment ......................................................................................................... 29
13.1 Schedule Requirements Review Process ............................................................................29
13.2 Schedule Assessment Process ..........................................................................................31
14.0 Confidence Level Requirements Review Process ..................................................................... 40
14.1 Cost and Schedule Range Estimate Assessments (KDP-0/KDP-B) ............................................41
14.2 Joint Confidence Level Assessment ...................................................................................45
Appendix A: SRB Programmatic Team Aides and Product Templates ................................................ 51
Example BOE Assessment Criteria ............................................................................................51
Appendix B: SRB Programmatic Team Planning Schedule ................................................................ 53
Appendix C: Acronyms............................................................................................................... 57
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 4 of 59
Title: SRB Programmatic Assessment Process

1.0 PURPOSE
NASA’s Office of the Chief Financial Officers (OCFO) is responsible for the functional oversight of the
independent programmatic assessments and this Standard Operating Procedure Instruction (SOPI) for
Standing Review Board (SRB) Independent Programmatic Assessment Processes. This SOPI documents
OCFO best practices for conducting an independent programmatic assessment within the SRB construct.
The SOPI's purpose is to document the SRB Programmatic Team processes for supporting the completion
of an independent assessment of a project throughout the program/project life cycle, per NASA
Procedural Requirement (NPR) 7120.5E. It is the expectation that the following processes will be followed
as part of any programmatic support to an SRB 1 .

Note that this instruction uses the word “independence” in broad terms, and it encompasses the term
“independent” that is used extensively in NASA policy and requirements documents.

1.1 Prerequisites
Qualified programmatic analysts on a SRB should possess knowledge and/or prior experience in one or
more of the following subject areas2 :

 NASA cost estimating

 NASA schedule management
 Risk management
 Joint confidence level (JCL) and schedule analysis
 Resource management
 Earned value management (EVM)
 Standing Review Board
 Program Planning and Control (PP&C)

2.0 REFERENCES
NPR 7120.5E, NASA Space Flight Program and Project Management Requirements

NPR 7123.1B, NASA Systems Engineering Processes and Requirements

NPR 8000.4A, Agency Risk Management Procedural Requirements

NASA Cost Estimating Handbook, Version 4.0

NASA/SP-2010-3403, NASA Schedule Management Handbook

NASA Project Planning and Control Handbook
NASA/SP-2014-3706, NASA Standing Review Board Handbook

1See section 8.0 for guidance on tailoring the review programmatic review process.
2It is recognized that many programmatic analysts will have expertise and experience in a subset of the areas listed
here. It is also recognized that the importance of each area with respect to the review is subject which life-cycle is
being reviewed and the scope of that review. As such, the OCFO will work with programmatic analysts, Mission
Directorates, and SRB’s to ensure that the SRB Programmatic Team is has a collection of requisite skill sets.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 5 of 59
Title: SRB Programmatic Assessment Process

NASA/SP-2011-3422, NASA Risk Management Handbook

NASA Policy Directive (NPD) 1000.5B, Policy for NASA Acquisition

3.0 SCOPE
This SOPI applies to all Independent programmatic assessment activities conducted by the SRB
Programmatic Team. The Independent programmatic assessment consists of three phases: Life Cycle
Review (LCR) Planning; LCR Analysis & Feedback; and Final LCR, Report Presentations, and Closeout. These
phases cover the independent programmatic assessment from planning the review through the LCR which
culminates with the final report and presentation of materials to the governing Program Management
Council (PMC). It is the SRB Programmatic Team lead analyst’s responsibility to ensure that government
and contractor personnel supporting the independent programmatic assessment adhere to all the SOPI
requirements.

The independent programmatic assessment includes a review of strategic goal alignment, development
of project control plans, requirements management, scheduling, workforce planning, resource
management, budgeting, cost estimating, acquisition strategy planning, contract management, risk
management, performance tracking, and performing the project programmatic functions: planning,
execution, tracking, assessment, and reporting out. The Agency does not have a required standard
organizational structure that dictates where these programmatic functions reside. They could reside in
the Business Management Division, Program Planning and Control Office, a technical organization like
Systems Engineering and Integration, or the Office of the Chief Engineer. Wherever they reside, their
processes and products are related and should be using the same requirements, work breakdown
structure (WBS), and planning assumptions, while adhering to NASA policies and directives.
The SRB Programmatic Team is not only assessing how each functional area performs, but also how the
project coordinates and interacts across each programmatic function to ensure that, for example, both
the budget and scheduling products are using consistent assumptions for planning and analysis purposes.

The SRB Programmatic Team should coordinate with both the project and the SRB to conduct the
independent LCR process through a parallel approach. The parallel approach is for the SRB Programmatic
Team to maximize the use of a project’s existing products and to engage within the project LCR meetings,
boards, and/or products development cycle to minimize the impact to the project while balancing the
requirements of the SRB LCR. All products and information requested by the SRB Programmatic Team are
in accordance with NPR 7120.5E requirements and therefore should be readily available.

The SRB Programmatic Team will develop an Independent Programmatic Assessment Plan (IPAP) to
conduct the independent LCR. The IPAP contains what programmatic assessments will be conducted,
project life cycle product delivery dates, and reporting out requirements for the SRB Programmatic Team
assessment.
The LCRs are essential elements of conducting, managing, evaluating, and approving spaceflight programs
and projects. The program manager is responsible for planning and supporting the LCRs. The SOPI focuses
on the unique programmatic requirements that are defined in NPR 7120.5E, however it should be
recognized that the purpose of reviewing those products is to support the Standing Review Board’s
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 6 of 59
Title: SRB Programmatic Assessment Process

assessment of the following six criteria (as identified in NPR 7120.5E). Results of any assessment should
focus on the information that senior management needs to make forward decisions.

 Alignment with, and contribution to, Agency strategic goals and the adequacy of requirements
flow down from those strategic goals.
o The scope of this criterion includes alignment of program/project requirements and
designs with Agency strategic goals, constraints, mission needs, and success criteria;
allocation of program requirements to projects; and proactive management of changes
in program and project scope and shortfalls.
 Adequacy of management approach.
o The scope of this criterion includes program and project authorization, management
framework and plans, acquisition strategies, and internal and external agreements.
 Adequacy of technical approach, as defined by NPR 7123.1B entrance and success criteria.
o The scope of this criterion includes flow down of project requirements to
systems/subsystems; architecture and design; and operations concepts that respond to
and satisfy imposed requirements and mission needs.
 Adequacy of the cost and schedule estimates and funding strategy in accordance with NPD
1000.5B.
o The scope of this criterion includes cost and schedule control plans; cost and schedule
baselines that are consistent with the program and project requirements, assumptions,
risks, and margins; Basis of Estimate (BOE); Range Estimate and Joint Confidence Level
(JCL) (when required); and alignment with planned budgets.
 Adequacy and availability of resources other than budget.
o The scope of this criterion includes planning, availability, competency and stability of
staffing, infrastructure, and the industrial base/supplier chain requirements, for example,
thermal vacuum chamber availability.
 Adequacy of the risk management approach and risk identification and mitigation per NPR
8000.4A.
o The scope of this criterion includes risk management control plans, open, and accepted
risks, risk assessments, risk mitigation plans, and resources for managing or mitigating
risks.

4.0 SRB Programmatic Team Structure

The SRB Programmatic Team consists of a Lead, a Schedule Analyst, and a Cost Analyst. The Lead serves
as a full voting member of the SRB, while the Schedule and Cost Analysts are considered consultants to
the SRB. The Lead, Cost, and Schedule Analysts are not limited to these specific roles and can support all
programmatic assessment areas during the review. The SRB Programmatic Team members report to the
SRB Chair and will coordinate with the Review Manager, who administers the independent reviews
required per the NASA governance model, through the LCR process. The SRB Chair may assign SRB
Programmatic Team members additional tasks, analysis, and reports, in addition to the SOPI
requirements. For more details on the roles of the SRB Chair and Review Manager, refer to the NASA
Standing Review Board Handbook.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 7 of 59
Title: SRB Programmatic Assessment Process

Depending on milestone and category of mission and life cycle the size and make-up of the SRB
Programmatic Team is tailorable and may be less than three analysts, with the assessment responsibilities
distributed appropriately. Regardless of the size of the SRB Programmatic team, the underlying function
of lead, cost, and schedule are still required. As reference, Table 1: SRB Programmatic Team Size
provides guidance for team size.
Table 1: SRB Programmatic Team Size

SRR PDR CDR SIR/ORR

Category I Mission 3 analysts 3 analysts 2-3 analysts 2 analysts
Category II Mission 2-3 analysts 2-3 analysts 2 analysts 1-2 analysts
Category III Mission 1-2 analysts 1-2 analysts 1-2 analysts 1 analyst

4.1 SRB Programmatic Team Lead

The SRB Programmatic Team Lead is responsible for the planning and execution of the independent
programmatic assessment, identifying the team’s roles and responsibilities, and is the primary contact
between the SRB Programmatic Team, SRB Chair, and Review Manager.

In addition to leading the programmatic analyst team, the SRB Programmatic Team Lead tailors the IPAP,
serves as the primary point of contact for interfacing with the project’s programmatic points of contact,
develops the SRB planning schedule for the independent programmatic assessment, tracks the SRB
Programmatic Team assessment progress, ensures completion of the final report or briefings, and archives
assessment information at the end of the review. The Lead should work closely with the SRB Chair and
the RM throughout the assessment.
The Lead is also responsible for coordinating with the OCFO if additional resources are required to
adequately assess the project life cycle products and to conduct the review. OCFO will coordinate with
Mission Directorate if additional resources are required.

4.2 Schedule Analyst

The Schedule Analyst is the focal point for assessing the Integrated Master Schedule (IMS) health and
schedule BOEs, providing expert opinion of the schedule performance, conducting Schedule Risk Analysis
(SRA), Joint Confidence Level (JCL) analysis, and identifying schedule findings to the SRB. The Schedule
Analyst will work closely with the Cost analyst throughout the entire SRB evaluation period to ensure that
schedule and cost analysis results are consistent and complementary.

4.3 Cost Analyst

The Cost Analyst is the focal point for assessing the cost BOEs, analyzing the cost estimate and budget,
conducting cost estimating, providing expert opinion of the cost performance, conducting cost risk
analysis, supporting JCL analysis and identifying cost findings to the SRB. The Cost Analyst will work closely
with the schedule analyst throughout the entire SRB evaluation period to ensure that cost and schedule
analysis results are consistent and complementary.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 8 of 59
Title: SRB Programmatic Assessment Process

5.0 SRB PROGRAMMATIC TEAM ROLE

The SRB Programmatic Team provides programmatic expertise to conduct an independent programmatic
analysis that supports the overall SRB assessment of the project at the various LCRs. The SRB
Programmatic Team assesses the health of the project programmatic products through independent
assessment. The SRB Programmatic Team will focus on, in coordination with the SRB, the identification,
assessment, and analysis of the potential of significant risks affecting project-planned execution, to
increase the likelihood of the project being successful within cost and schedule objectives. The IPAP is an
SRB assessment plan for the project life cycle programmatic requirements, products, and processes.
Therefore, the SRB Programmatic Team should work closely with the SRB Chair, Review Manager, and the
other SRB members when planning for and developing the IPAP. Please refer to Appendix A: SRB
Programmatic Team Aids and Product Templates for Independent Programmatic Assessment (IPA)
templates that can be tailored and links to the OCFO Max site for other SRB programmatic aides and past
review document repository.
The IPAP should consider the following different life cycle groups for the independent assessment
approach:

 Pre-Phase A and Phase A: LCRs for projects in the formulation phase, such as standing up a project,
developing requirements, governance control plans, and preliminary cost and schedule estimates
(e.g., System Requirements Review [SRR], System Definition Review [SDR], Key Decision Point B
[KDP-B])
 Phase B: LCR that approve a project baseline for cost and schedule (e.g., Preliminary Design
Review [PDR], Key Decision Point C [KDP-C], Rebaseline Review)
 Phase C/D/E: LCRs for projects in the implementation phase and can be measured by performance
to baseline (e.g., Critical Design Review [CDR], System Integration Review [SIR], Key Decision Point
D [KDP-D], Operational Readiness Review [ORR])
The SRB Programmatic Team is officially released from its independent programmatic assessment review
duties once the appropriate PMC closes with no further documented actions for the SRB Programmatic
Team to support.

6.0 INDEPENDENT PROGRAMMATIC ASSESSMENT LCR WORKFLOW

NASA projects are unique and often tailor LCR criteria to meet the success of the mission. The RM will,
with input from the SPT, coordinate with the project to develop a review plan for the Terms of Reference
(ToR) and conduct the LCR per the agreements in the ToR. The ToR is the agreement among the SRB,
Convening Authorities (CA), and program or project that specifies the nature, scope, schedule, and ground
rules for the conduct of the LCR by the SRB. 3
The SRB Programmatic Team should develop a planning schedule for completing the independent LCR,
beginning with the planning stages and extending from SRB’s presentation to the governing PMC to the
close out of the LCR. Section 6.1 Life Cycle Review (LCR) Planning provides guidance steps for SRB
Programmatic Team members conducting the independent programmatic assessment. These are best

3 For more information on the ToR refer to the NASA Standing Review Board Handbook, Appendix H.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 9 of 59
Title: SRB Programmatic Assessment Process

practices to complete the LCR process and are tailorable to conduct the independent programmatic
assessment. The Review Manager is responsible for the schedule of the SRB, and the guidance below may
need to be adjusted to align with the SRB planning schedule. These steps do not include additional analysis
or actions the SRB Chair or Review Manager may request the SRB Programmatic Team to perform.
The independent programmatic assessment process contains three phases during the LCR and is detailed
in the NASA Standing Review Board Handbook:

 LCR Planning: For a new SRB Programmatic Team and/or SRB, the early planning stage includes
standing up the SRB, complete training, begin communication with the project, begin developing
an IPAP, and formulate a life cycle review plan for the ToR. For a SRB Programmatic Team already
assigned to an existing SRB in a follow-on review, the scope of the early planning includes
interfacing with the project to develop/adjust a review plan for the next milestone review as well
as developing the IPAP and starting the IPA.
 LCR Analysis & Feedback: This phase begins after project’s first LCR data drop and the SRB
Programmatic Team begins assessing the project products and provides feedback to the project.
The SRB Programmatic Team integrates the programmatic assessment in preparation for the SRB
discussions for identifying the project strengths and weaknesses. The goal of this phase is open
and continuous communication with the SRB and project to have a successful LCR.
 Final LCR Report, Presentations, & Closeout: This final phase is the process for the SRB to develop
the final out briefs to the project and governing PMCs in preparation of the KDP. The Agency
collects the SRB assessment information for archiving and lessons learned from the LCR.

6.1 Life Cycle Review (LCR) Planning

Below are the planning steps to stand up the SRB, complete training, begin communication with the
project, and formulate an independent LCR plan.

6.1.1 Complete SRB Programmatic Team Training

SRB Programmatic Team participants are required to complete the SRB programmatic independent
assessment training provided by the OCFO and the mission directorate. The training focuses on the role
of the SRB Programmatic Team and independent programmatic assessment expectations. Training must
be completed within two weeks after being selected for the SRB. For training information, contact the
Strategic Investments Division within the OCFO.

6.1.2 SRB Programmatic Team Coordination with the SRB Chair and Review Manager
The purpose of this formal meeting between SRB Programmatic Team, SRB Chair, and Review Manager is
to discuss the team’s role and the required SRB technical team inputs to complete the independent
programmatic assessment. The goal is to ensure roles and responsibilities are defined at the beginning
the independent assessment and that the independent assessment aligns with the expectations of the
Agency and SRB Chair. The OCFO is available to facilitate communication.

6.1.3 Develop Preliminary IPAP

The SRB Programmatic Team Lead should develop an IPAP that defines the team’s roles, programmatic
assessment plans, project data drop delivery dates, and the team’s independent assessment planning
schedule to complete the LCR. Appendix B: SRB Programmatic Team Planning Schedule for Independent
Programmatic Assessment contains an example LCR schedule.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 10 of 59
Title: SRB Programmatic Assessment Process

6.1.4 Initial Communication and Data Access with the Project

Continuous communication ensures both the project and the SRB are transparent on LCR planning and
execution. The SRB Programmatic Team Lead should establish a project contact, preferably a Program,
Planning, and Control representative or Business Manager, for developing and/or delivering
programmatic products for the LCR. The best practice is to plan for recurring telecom meetings, either
weekly or biweekly, throughout the LCR process. Refer to Section 7.0 Data Drops for more information.
Coordinate with the project and RM on how the SRB Programmatic Team will access the project LCR data. 4
Data access should include, but not be limited to, products that assist the team in understanding the
project, including the project plan, WBS, and latest monthly project status briefing.

6.1.5 Coordinate with the Project on Data Drops, LCR Timeline Flow, and Feedback Loop
The discussion meetings between the SRB and project establish the framework for the LCR to determine
when the project will have the appropriate LCR products available and how the team will conduct the
independent programmatic assessment. The meeting can be face-to-face, telecom, or via email. The SRB
Programmatic Team should discuss the data drops, review timeline with the project, and make changes
as needed.
These meetings create a basis with the project for determining LCR data drops (refer to 6.2.1 Data Drop
1) to finalize the IPAP, planned SRB Programmatic Team independent assessment schedule, and work out
any disconnects between the SRB and the project. Appendix B: SRB Programmatic Team Planning
Schedule contains an example of a team planning schedule for independent programmatic assessment. If
a JCL is required for the LCR, then a JCL agreement may be developed to detail the type of JCL model, data
to be included in the model, and planned delivery or revision dates. These agreements will facilitate
writing the ToR.

6.1.6 Coordinate with the Review Manager on Life Cycle IPAP for ToR
The SRB Programmatic Team should discuss the results of the meetings between the team and project
with the SRB Chair and Review Manager. This should include:

 The SRB Programmatic Team and project agreed plan for programmatic LCR data drops products
and delivery dates
 Continuous communication plan with project (e.g., weekly or bi-weekly meetings)
 IPAP tailoring of the programmatic LCR data drop products or the SRB Programmatic Team
independent programmatic assessments conducted that need to be reflected in the ToR
 The Review Manager inputs (e.g., SRB caucus, travel, schedule conflicts) into the SRB Program
Team independent assessment planning schedule
 Communication of the IPAP with the OCFO. Per NPR 7120.5E, ToR concurrence includes the
OCFO.

6.1.7 Brief SRB on required Programmatic Inputs

Life cycle requirements vary per life cycle milestone and it is necessary to brief the SRB during this phase
on required programmatic inputs to complete the IPA. Inform the SRB of any technical assessments or

4
Please note that consultant contractors supporting the SRB may need additional time to set up data access because
some repositories may require a NASA email (.gov) account.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 11 of 59
Title: SRB Programmatic Assessment Process

inputs needed to complete the IPA, such as technical risk identification and assessment, or uncertainty
boundary analysis. This will ensure the Review Manager has scheduled SRB technical members to provide
the appropriate assessments and feedback to the SRB Programmatic Team to complete the IPA. The Best
practice is to conduct a preliminary risk assessment meeting with the SRB prior to the life cycle review site
review.

6.2 LCR Analysis and Feedback

This phase begins after the first project LCR data drop and the SRB Programmatic Team begins the
independent assessment of the project products and providing feedback to the project. The SRB
Programmatic Team integrates the programmatic assessment in preparation of the SRB discussions for
identifying the projects strengths and weaknesses. The goal is for open and continuous communication
to have a successful LCR. The following sections concerning data drops are consistent with guidance in the
NASA Standing Review Board Handbook.

6.2.1 Data Drop 1

The first project data drop provides the SRB Programmatic Team with initial data to begin assessing the
project. Anything missing from the SRB’s initial data request shall be noted and communicated back to
the project. If the first set of data is not delivered in reasonable time per the schedule agreed upon in the
ToR, then it should be reported to the SRB Chair and the Review Manager. Refer to Section 7.0 Data Drops
for more information.

6.2.2 Review Data Drop 1

6.2.2.1 Perform sufficiency review on Data Drop 1 to meet LCR requirements
It is important that the SRB programmatic analysts review the data drop products as soon as they become
available, and provide comments and feedback to the project. This review consists of the SRB
Programmatic Team evaluating the project products and processes for reasonableness, completeness,
and consistency to meet the intent of the LCR requirement. The SRB Programmatic Team can seek
clarification, as needed. It is important that communication be continuous and the team is flexible for the
project to revise products to meet the intent of the LCR.

6.2.2.2 Provide Data Drop 1 sufficiency review feedback to project

Inform project on sufficiency review results.

6.2.2.3 Perform Preliminary assessment on Data Drop 1 products

Perform Data Drop 1 independent assessment as defined in the IPAP and the SRB Programmatic Team
independent assessment planning schedule.

6.2.2.4 Provide preliminary assessment feedback on Data Drop1 to project

Discuss preliminary assessment feedback with the project. Feedback should be continuous throughout
the independent assessment of life cycle review. The goal is to communicate concerns or issues to the
project to facilitate discussions and mitigate disconnects between SRB and project assessment
perspectives.

6.2.2.5 Status SRB Chair on Data Drop 1 preliminary assessmen t

The SRB Programmatic Team should provide the project and SRB Chair a status of products for Data Drop
1, independent programmatic assessments to date and discuss concerns and issues.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 12 of 59
Title: SRB Programmatic Assessment Process

6.2.3 Review Data Drop 2

This is the final data drop and all products should be available for the SRB Programmatic Team to review.
Data Drop 2 should include updated Data Drop 1 products and include technical content, cost data,
integrated master schedule, BOEs, risks, and if applicable, confidence level statistical model (JCL or range
estimates) delivery. Any new schedule performance data and/or programmatic data should also be
provided. If the project is developing a JCL, the final model should be delivered. The project should present
their JCL model, key assumptions, and provide supporting JCL data to the SRB Programmatic Team. This
is the snapshot in time that the IPA will be based on. If the data is not delivered in reasonable time per
the schedule agreed upon in the ToR, then it should be reported to the SRB Chair and the Review Manager.
Refer to Section 7.0 Data Drops for more information.

6.2.3.1 Perform sufficiency review on Data Drop 2 to meet LCR requirements

It is important that the SRB programmatic analysts review the data drop products as soon as they become
available, and provide comments and feedback to the project. This review consists of the SRB
Programmatic Team evaluating the project products and processes for reasonableness, completeness,
and consistency to meet the intent of the LCR requirement. The SRB Programmatic Team can seek
clarification, as needed. It is important that communication be continuous and the team is flexible for the
project to revise products to meet the intent of the LCR.

6.2.3.2 Provide Data Drop 2 sufficiency review feedback to the project

Inform project on sufficiency review results.

6.2.3.3 Perform Preliminary assessment on Data Drop 2 products

Perform Data Drop 2 independent assessment as defined in the IPAP and the SRB Programmatic Team
independent assessment schedule.

6.2.3.4 Provide preliminary assessment feedback on Data Drop 2 to the project

Discuss preliminary assessment feedback with the project. Feedback should be continuous throughout
independent assessment of life cycle review. The goal is to communicate concerns or issues to the project
to facilitate discussions and to mitigate disconnects between SRB and project assessment perspectives for
the final SRB findings presentations to the governing program management councils.

6.2.3.5 Status SRB Chair on Data Drop 2 preliminary assessment

The SRB Programmatic Team should provide the project and SRB Chair a status of products for Data Drop
2, independent programmatic assessments completed to date and discuss concerns and issues.

6.2.4 Integrated Assessment

The SRB Programmatic Team Lead should integrate the cost and schedule assessments into the IPA and
develop life cycle review charts for the SRB. The cost and schedule analysts should provide the SRB
Programmatic Team Lead with updated charts (such as schedule health check, critical path analysis, and
quantitative risk assessment) of the independent programmatic assessment to date, including the
independent assessment of cost and schedule plans to date, and JCL model (if applicable) or S-curves (if
applicable) concerning technical content and risks. The draft LCR charts should be available to brief the
SRB.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 13 of 59
Title: SRB Programmatic Assessment Process

6.2.5 Complete Initial SRB Risk and Uncertainty Assessment

The SRB Programmatic Team should brief the SRB of the draft programmatic LCR charts, facilitate the SRB
risk assessment discussion with the whole SRB and work with the appropriate technical SME SRB members
to analyze each risk as well as identify new risks. The SRB should deliver their subject matter expert
assessment of the project risks (e.g., identify new risks or adjust existing risk likelihood and consequence
distribution) and uncertainty boundaries (if applicable). The SRB Programmatic Team uses the SRB risk
and uncertainty assessment to update the IPA. It is important that the SRB Programmatic Team and SRB
communicate to ensure all parties understand the risk assessment clearly concerning potential cost and
schedule impacts. For KDP-B and KDP-C milestones, the SRB should provide SRB uncertainty boundaries
(cost and schedule), risk ratings (likelihood and consequence) for each project and SRB proposed
additional risks. This is required for KDP-B range estimates and JCL analysis.

6.2.6 Develop Final JCL or Cost/Schedule Risk Analysis Model Updates with SRB Inputs (If
Applicable)
Update the IPA for any new risks or changes to independent risk assessment or cost and schedule
uncertainty boundaries. If applicable, the SRB Programmatic Team should adjust the JCL or cost/schedule
risk analysis models and provide model output reports to SRB for review. Prepare the IPA for the OCFO
Checkpoint Review. Please refer to 14.0 Confidence Level Requirements Review Process for range
estimates and JCL assessment details.

6.2.7 Conduct OCFO Checkpoint

The goal of the OCFO checkpoint is to ensure that the independent programmatic assessment is well
documented and consistent with Agency programmatic assessment expectations. The specific checkpoint
timing will be coordinated by the SRB Programmatic Team Lead and the OCFO SID Advanced
Programmatic Analysis and Research Capability (APARC) representative. Information reviewed at the
checkpoint include project programmatic data delivered to date, project data quality assessment,
additional SRB analysis, and timeline through KDP.

6.3 Final Review Report, Presentations, & Closeout

This is the beginning of the process for the SRB to develop the final out-brief packages to the program and
governing Program Management Councils for KDP, and for the Agency to collect the SRB assessment and
lessons learned from the LCR process. SRB Programmatic Team develop the programmatic content for the
SRB final out-brief.

6.3.1 Support SRB Caucus Plan

The Review Manager facilitates the caucus meetings for all SRB members to capture risks, issues,
concerns, observations, and identified requests for action to integrate into SRB findings report. The SRB
programmatic analysts are responsible for providing the SRB Chair the programmatic Requests for Action
(RFA) if a significant SRB programmatic finding requires a mitigation plan. RFA’s are submitted to the
project as they are generated and coordinated by the Review Manager. SRB caucuses are typically
scheduled in the evenings during the life cycle site review, the full day after the life cycle site review or
the following week. The SRB Programmatic Team Lead should coordinate with the Review Manager to
ensure time is available for the team to discuss and finalize any inputs needed from SRB members to
finalize the IPA. Additional or delayed caucuses can be conducted later, depending on the availability of
the SRB Chair and members.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 14 of 59
Title: SRB Programmatic Assessment Process

6.3.2 Support Life Cycle Site Review

The SRB Programmatic Team should attend the project LCR as members of the SRB. This is the projects
review to demonstrate meeting the LCR success criteria.

6.3.3 Support Development of Management Council Snapshot Report

The SRB is required to report out a one-page summary of preliminary LCR findings within 48 hours of the
LCR site review. This report includes independent programmatic findings and if a Range Estimate or JCL is
required, then the SRB results for the confidence level for Range Estimates and the 70 percent joint cost
and schedule confidence level, respectively.

6.3.4 Receive Final Inputs from SRB

Receive any new SRB additions or revisions to existing risks and uncertainty boundary to update the IPA.
The SRB Programmatic Team should be completing the final adjustments to the IPA, SRB presentation
briefings to the project and governing program management councils.

6.3.5 Finalize Independent Assessment

Incorporate any new SRB revisions into the IPA. The SRB Programmatic Team cost and schedule analysts
should inform the SRB Programmatic Team Lead of any updates to the IPA.

6.3.6 Final Independent Assessment Findings to SRB Chair and the SRB
The SRB Programmatic Team briefs the results of the IPA and associated presentations to the SRB Chair
and the SRB. These products typically form the programmatic assessment findings portion of the of SRB
presentation materials.

6.3.7 Independent Programmatic Assessment Report Completed

The final IPA report is finished and archived by the MD and OCFO. The report is the IPA presentation or
document capturing the assessment activities completed during the review and results. Links to example
templates of IPA reports and presentations can be found in Appendix A: SRB Programmatic Team Aids
and Product Templates.

6.3.8 LCR or KDP Management Council

This step includes the SRB presentation out-brief to the governing management council and is generally
conducted by the SRB Chair. The goal is for the governing program management council to occur no later
than 30 days after the Snapshot Report. The SRB Programmatic Team may be required to present
programmatic assessment sections or answer questions during the governing program management
council.

6.3.9 Closeout & Knowledge Management Capture

The SRB Programmatic Lead should provide the OCFO SID APARC and the mission directorate contact with
the final life cycle SRB report briefing package, IPA report (if developed), IPAP, supporting analysis
materials, and lessons learned. These products will be archived in the SRB repository site maintained by
OCFO SID. 5 The intent of the archive is not to include every iteration of analysis or model run, rather the
final iteration or run leading to the final SRB findings. All supporting data for the results should be
archived. Supporting data includes project and SRB programmatic information. This information can

5
Strategic Investments Division SRB website
(https://community.max.gov/display/NASA/Standing+Review+Board+%28SRB%29+Repository).
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 15 of 59
Title: SRB Programmatic Assessment Process

include, but is not limited to, BOEs, uncertainty, risks, parametric models, model assumptions, cost and
schedule benchmarks, and review plans.

6.3.10 OCFO Closeout Checkpoint

The SRB Programmatic Team should coordinate with the OCFO SID APARC group to conduct a meeting
with the specific goals of capturing review lessons learned and mitigate product archiving risks.

Refer to Appendix B: SRB Programmatic Team Planning Schedule for Independent Programmatic
Assessment for the detailed timeline of the team LCR process mapped to appropriate workflow processes.

Analysts should use the file naming and archiving file structure listed is below. For each folder and
individual file name, it is recommended to include the mission directorate, program or project, and review
type as the standard prefix during the execution of the LCR. This is important for archiving for follow on
LCRs and analogous missions research of future programs and projects SRB assessments and analysis. For
each LCR, the OCFO establish a secure website to allow collaboration and file storage for the assigned SRB
Programmatic Team. Figure 1: SRB Programmatic Analysis Archive File Structure shows example of file
structure.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 16 of 59
Title: SRB Programmatic Assessment Process

Figure 1: SRB Programmatic Analysis Archive File Structure (with example files)
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 17 of 59
Title: SRB Programmatic Assessment Process

7.0 DATA DROPS

Data drops are the project LCR programmatic products that are available at the appropriate time to
provide the SRB Programmatic Team sufficient time to perform an independent programmatic
assessment prior to the project LCR milestone to proceed to the governing program management council.
The ToR defines scheduling and content for the data drops and is negotiated between the Project,
Program Office and SRB prior to the 100 day delivery.
To ensure adequate time for the SRB Programmatic Team to assess the project, the data drops are
required to occur before the project life cycle board/site review to proceed to the governing program
management council. Three programmatic data drop milestones ensure the SRB Programmatic Team has
sufficient time to perform the independent programmatic assessment:

 Data Access: Project provides access to required repositories for the LCR and overview
documentation (e.g., project plan, WBS dictionary, latest monthly status briefing) to assist the SRB
Programmatic Team in understanding the project prior to the beginning of the LCR
 Data Drop 1: Project provides preliminary required programmatic LCR products
 Data Drop 2: Project provides final required programmatic LCR products

NPR 7120.5 defines project data drop deliverables. Data drop deliverables dates should be included in the
IPAP and programmatic section of the ToR. The NASA Standing Review Board Handbook provides
recommendation for timelines for data drops6 .

6
All timelines should be documented in the ToR and agreed to between the SRB and the project. Timelines are often
negotiated to accommodate project and Center processes.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 18 of 59
Title: SRB Programmatic Assessment Process

Table 2: LCR Data Deliveries

Item Content* Timeline

Existing program and project management documentation (ref. NPR

7120.5E, Tables I-(2-7)), including working technical baseline
description; project risk list, matrix, and mitigation plans; WBS, WBS 100 calendar days
Data Access dictionary; master equipment (MEL) list; equipment power prior to LCR**
consumption list; software lines of code, integrated master schedule;
cost estimate and planning budget by year and phase; staffing
requirements and plans; and infrastructure requirements.

Preliminary delivery of data formally required for the review,

including BOEs for cost and schedule, a cost and schedule range
estimate or functional JCL model and analysis schedule (if required for 60 calendar days
Data Drop 1
LCR) and supporting data (as applicable), and/or any updates that prior to LCR**
have been made to the risk list, matrix, cost estimate, budget, and
schedule.

Final range estimate or JCL model and analysis schedule (if range or 20 calendar days
Data Drop 2 JCL required) and/or any updates to the risk list, matrix, cost estimate, prior to LCR**
budget, schedule, and project documents.

Single project programs, loosely coupled projects, uncoupled projects, or tightly coupled programs
* The list of programmatic cost and schedule data for each independent LCR is found in the NASA
Standing Review Board Handbook.
**For two-step LCR. The timeline is with respect to the second step of the independent LCR.

8.0 TAILORING
The criteria documented in NPR 7120.5E provides the emphasis and depth of analysis required. Whenever
possible the general review process for each LCR should be followed. However, in certain cases, the
amount of programmatic data for review and depth of analysis may be less or more than standard project
or tightly coupled programs and, thus, the analysis and reporting can be tailored appropriately. Tailoring
should be captured by the ToR.

9.0 LCR TOR

The LCR Deliverables to the SRB section of the ToR captures the required programmatic LCR products and
planned delivery dates for the data drops. The ToR should identify any project tailored programmatic
requirements and any tailoring to the IPAP. The OCFO Strategic Investments Division should be informed
of any programmatic tailoring. See Appendix A: SRB Programmatic Team Aides and Product Templates
for links to a ToR template.

10.0 DISSENTING OPINIONS

Dissenting Opinion is a disagreement with a decision or action that is based on a sound rationale (not on
unyielding opposition) that an individual judges is of sufficient importance that it warrants a specific
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 19 of 59
Title: SRB Programmatic Assessment Process

review and decision by higher level management, and the individual specifically requests that the dissent
be recorded and resolved by the Dissenting Opinion process. For details regarding the dissenting opinion
process, please refer to NASA Governance and Strategic Management Handbook7 ,NASA Space Flight and
Project Management Requirements8 , Section 3.4 Process for Handling Dissenting Options, and the NASA
Standing Review Board Handbook9 .

11.0 REQUIREMENTS TAILORING

There are three mechanisms for tailoring the requirements in 7120.5E. Requests for tailoring may be
submitted in the form of the Compliance Matrix10 , by using a waiver request (see the NASA Space Flight
Program and Project Management Handbook) individually or in groups, or via the Program Project
Management Board (PPMB).

11.1 Program Project Management Board (PPMB)

The Program and Project Management Board, run by the Office of the Chief Engineer, can assist Program
and Project Managers with tailoring guidance. It serves as a forum for adjudicating issues (e.g., SRB project
LCR planning issues) as Program and Project Managers work through the Agency process for tailoring
waivers, and deviations for program and project management policy. The Program and Project
Management Board also serves as a recommendation board to the APMC for project tailoring guidance.

The Office of the Chief Engineering chairs the Program and Project Management Board with members
from the mission directorates, the OCFO, Centers, and the Jet Propulsion Laboratory (JPL).

11.2 Compliance Matrix

The Compliance Matrix is provided to streamline the waiver and deviation process described in paragraph
3.5 in NPR 7120.5E. If the Compliance Matrix is completed in accordance with NPR 7120.5E Appendix C
instructions, it meets the requirements for requesting tailoring and serves as a group submittal for waivers
to NPR 7120.5E. If the compliance Matrix changes or if compliance is phased for existing programs or
projects, updated version of the Compliance Matrix are incorporated into an approved Formulation
Agreement or Program or Project Plan revision. For a complete conversation on the NPR 7120.5E
Compliance Matrix please refer to Appendix C within NPR 7120.5E.

12.0 COST ASSESSMENT

Cost assessments are performed as part of the SRB’s IPA. This section is intended to provide guidance to
SRB programmatic analysis team members and consultants, as well as providing some things to consider
during a SRB LCR.

While some steps of the cost assessment process are mechanistic, often assessment and especially
estimation is a predictive process for which judgment and experience add value. Effective assessment and

7
NPD 1000.0B
8
NPR 7120.5E
9 NASA/SP-2016-3706 RevB
10 NPR 7120.5E, Appendix C. Compliance Matrix
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 20 of 59
Title: SRB Programmatic Assessment Process

estimation requires an understanding of the technical work to be performed. Please note that
programmatic analysts are intended to perform an assessment of the project programmatic processes
and products, including life cycle cost estimates. An independent cost estimate (ICE)11 is not required.
However, there may be instances where benchmark12 estimates are required as part of the LCR by the
SRB Programmatic Team, see Section 12.2.2 Assessment of Reasonableness. For guidance on the process
and methodologies for developing benchmark estimates, please refer to the NASA Cost Estimating
Handbook.
The intent of the cost assessment is to show the level of confidence that the Agency can commit to
externally to accomplish its technical goals while executing its plan on schedule and within budget.

12.1 Cost Requirements Review Process

This section is intended to provide guidance to the SRB programmatic analysis members and consultants
concerning the cost estimate requirement within NPR 7120.5E. Specifically, this section provides guidance
to following requirement:
Table 3: NPR 7120.5E Cost Requirements

Paragraph Requirement Statement General Taxonomy

Reference
2.4.2 All programs and projects develop cost estimates and planned schedules for Cost/Schedule
the work to be performed in the current and following life cycle phases (see
7120.5 Appendix I tables). As part of developing these estimates, the
program or project shall document the basis of estimate (BOE in retrievable
program or project records).

This section only addresses the cost portion of the above requirement. The schedule section is in 13.0
Schedule Assessment.

12.1.1 Scope
The scope of cost assessment should include the entire life cycle of the project, or as defined in the ToR.
This typically includes Phase A through E. The SRB assesses the Agency’s commitment to the project, so
there could be items that the project manager is not actively managing that still need to be part of
assessment. 13 Cost contributions to the mission (e.g., foreign contributions) do not need to be directly
assessed for JCL purposes; however, costs associated with contribution risks such as fallback options and
delivery schedule risk and uncertainties that could affect the project should be noted in the analysis.

11Independent Cost Estimate: A quantitative assessment and estimate, performed independently from the SRB
Programmatic Assessment, resulting in an independent cost estimate of project’s lifecycle cost.
12 Benchmarking: Comparative analysis of the project’s cost and schedule plan to determine reasonableness used to
inform the SRB Programmatic Assessment. Benchmarking can be performed using any of the methodologies
specified in the NASA Cost Estimating Handbook (including historical, analogy and parametric estimating).
13
Launch vehicle procurement is an example for Science Mission Directorate missions. Typically launch vehicle
procurement is managed by Launch Services Program and the mission directorate.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 21 of 59
Title: SRB Programmatic Assessment Process

12.1.2 Basis of Estimate

The cost assessment should start with the documentation, this includes BOE capture and review, estimate
summary and detail information. The purpose is to ensure that the estimate is presented in an
understandable manner that it is clear, complete, consistent, repeatable, traceable, and defendable.

BOEs should include the scope, technical description, cost phasing, estimating methodology, ground rules
and assumptions, exclusions, and risks.
The SRB Programmatic Team should be addressing the following attributes for each BOE:

 Task(s) Description
o Is there a detailed explanation of how the work will be accomplished?
o Does the BOE have any unique ground rules and assumptions to consider?
 Rationale and Methodology
o Is the estimating methodology (e.g., parametric, analogous, grassroots, cost estimating
relationships [CERs]) appropriate for the given milestone?
o Are adjustments and assumptions (e.g., complexity factors, learning curve) adequately
explained?
 Source Data
o Does the data come from a credible source and is it representative of the work being
estimated?
o Can the assessor verify and/or access the data upon request?
o Is the supporting data current, accurate, and complete?
 Accurate
o Are any supporting equations documented (e.g.., CERs, rates, factors, etc.)?
o Are the BOE calculations correct (i.e., has a check been done to ensure it is free of errors)?

12.1.2.1 Ground Rules and Assumptions

BOEs typically provide a detailed description of the ground rules and assumptions used to develop
estimates. The ground rules and assumptions help provide insight into what is included, and often, more
importantly, excluded from the estimate and scope. It is critical that all assumptions are clearly
documented. Any costs excluded from the estimate must be clearly documented. Examples are items
covered by other programs or projects, costs covered by the Center, other government agencies, or sunk
costs. Indicate any primary trades included in the estimate. The SRB Programmatic Team should review
the ground rules and assumptions and assess validity.

12.1.2.2 Basis of Estimate Assessment Criteria

Please refer to Appendix A: SRB Programmatic Team Aids and Product Templates for example BOE
assessment criteria.

12.2 Cost Assessment Process

The cost assessment has six main steps: compile data, review, analyze/validate, document assessment,
discuss/brief results; and perform any required iterations.

 Compile Data
o Request project data
o Compile historical data
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 22 of 59
Title: SRB Programmatic Assessment Process

 Review
o Read and comprehend the project scope, assumptions, liens, threats, risks, or any
exclusions.
o Determine if the estimates add up or contain errors.
 Analysis/Validation
o Determine if the estimates make sense, and if any of the excluded items are required.
o Ascertain if the project is within family of comparable historical projects
 Document
o Write up any questions and or concerns.
o Notate which comparable historical project information was used in assessment.
 Discuss/Brief
o Talk to the project frequently; ask questions; share concerns.
o Talk to SRB about findings, especially issues, concerns, and observations.
 Iterate
o If/as required

Compiling data is discussed in Section

Figure 1: SRB Programmatic Analysis Archive File Structure (with example files)
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 23 of 59
Title: SRB Programmatic Assessment Process
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 24 of 59
Title: SRB Programmatic Assessment Process

7.0 Data Drops. Reviewing estimates is largely covered above in Section 12.1 Cost Requirements Review
Process. The estimate “review” is typically qualitative in nature, focused on ensuring the estimate meets
requirements and best practices. 14 This section will specifically cover the Analysis/Validation step
discussed above. The Analysis/Validation step is broken into two sections:

 Project Cost Plan Assessment

 Assessment of Reasonableness

12.2.1 Project Cost Plan Assessment

The SRB Programmatic Team will assess the project’s cost estimates to validate that they support the
project plan.
The SRB Programmatic Team should review all major elements of cost and schedule in the WBS, such as
spacecraft, payload systems and instruments, integration and test, level of effort management and
oversight, ground systems development and test, and mission control and operations. The team
assessment should be able to:

 Explain how the estimate for each element was determined (e.g., grassroots and bottom up,
parametric, analogy, fixed-price vendor quote, pass-through from another organization).
 Explain why that estimating methodology was chosen and how the estimate was developed.
o For grassroots estimates, identify the data sources used that provide an accurate estimate
of the schedule and cost required to complete the project.
o For parametric estimates, identify the model(s) used, the major assumptions that went
into the models, and the rationale for those assumptions.
o For analogy-based estimates, identify the missions/systems used and explain why each is
an applicable analog. If the project estimate is out-of-family, explain why.
For fixed-price quotes, the SRB Programmatic Team should assess the level of maturity of the hardware
to be delivered as well as the vendors’ history in delivering that type of hardware on time and for the
promised cost. Analysts should work with the projects to obtain the required information to perform this
type of vendor assessment.

The SRB Programmatic Team should examine how workforce estimates were created (e.g., by cost or
resource-loading the schedule or by some other method), and assess the assumptions behind the
workforce ramp-up and ramp-down and the outcome of workforce sensitivity analysis.

To independently assess this information, the programmatic analyst(s) should determine if the project’s
estimate is documented, traceable, complete, reasonable, and consistent with analogous missions or
systems. The estimate should follow the project lead Center guidance (e.g., GSFC Gold Rules) and any
other requirements (e.g., Announcement of Opportunity). The SRB Programmatic Team should consider
the experience level of the project team and identify areas where the project and reviewers agree and
disagree.

14
Best practices include those covered in the NASA Cost Estimating Handbook and by JPL, the Goddard Space Flight
Center (GSFC), and Johns Hopkins Applied Physics Laboratory (APL) guidance principles.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 25 of 59
Title: SRB Programmatic Assessment Process

When assessing grassroots estimates, assessors should consider the following questions as part of the
assessment:

• Have the project personnel provided sufficient information about the planned work, their
experience in doing or managing similar work, and how they developed their cost and schedule
estimates to provide high confidence in the accuracy of those estimates?
• Is the entire mission content covered in the project’s estimate? If not, what is missing and what
is the rationale for excluding it?
• Are the technical requirements stable?
o Are there potential changes to requirements, whether within a single project element or
handed across an interface from one element to another, that have not been accounted
for in the estimate but could drive cost and schedule changes and cause the grassroots
estimate to be inaccurate?
• Are the hardware/software requirements and designs mature enough to enable an accurate
estimate of the resources required to do the planned work? How mature are the technologies
and/or technical approaches the project plans to use?
o Is there any hardware or software that has not been built and/or flown in space over the
past five years? If so, have viable alternatives been identified?
o Is there a plan regarding how and when a decision to use alternative designs or
technologies will be made?
o Does the project’s estimate fund this plan, including the cost of carrying both alternatives
until the decision is made?

When assessing parametric estimates, assessors should consider the following questions as part of the
assessment:
• Is the modeling approach appropriate to the project’s point in the life cycle?
• Is the model’s database sufficiently analogous to the project or to the individual project element
being estimated that the model can produce a reliable estimate?
• Has the estimator identified all the model inputs (i.e., assumptions and parameters)? Are these
assumptions and parameters reasonable?
• Does the model include the entire project’s content?
• Does the project have external dependencies (e.g., international partners)?
• What assumptions or inputs had the greatest impact on the model’s output?
• Did the estimator do a sensitivity analysis by varying model inputs or using multiple models? If so,
what were the results of the sensitivity analysis?
When assessing analogy-based estimates, assessors should consider the following questions as part of
assessment:

• Which analogs were used? Was sufficient information on each chosen analog provided to
determine that it is an appropriate analog?
o Is the analog applicable at the top level or at the detailed subsystem level? How might
that change the estimate?
o Are any of the analogs a poor choice for developing the project’s estimate? Are there
more appropriate analogs that should have been used?
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 26 of 59
Title: SRB Programmatic Assessment Process

• If information on the chosen analogs was not provided, did the estimator provide a rationale for
analog selection?
• Where does this project fit within the overall envelope of costs and schedule durations, in total
and at lower levels?
• Does the project have external dependencies (e.g., international partners)?
• Is the project at or near an edge of the envelope in key areas of the project (e.g., where there are
known issues and risks)?
When assessing performance-based estimates, assessors should consider the following questions (often
used for later LCRs, CDRs, Systems Integration Reviews [SIRs] and Operations Readiness Review [ORRs]):

 Has the technical baseline changed since the last LCR or major planning milestone (e.g., as the
result of the annual planning, programming, budgeting, and execution (PPBE) cycle)?
 Does the project track earned value management (EVM)?
o How is the project EVM performance?
 Does the project have external dependencies (e.g., international partners)?
 Is there a launch window that could drive resource allocation?
 What is the nature of the prime contracts (e.g., firm-fixed-price, cost-plus-fixed-fee)?
 If performance is not to plan, what are the causes (e.g., realized risks, incorrect estimates)?
 What are the project risks, threats, and opportunities? Are they captured in the plan?
 Are the project’s sensitivity analyses appropriate based on past performance?
The SRB Programmatic Team should assess whether the planned funding profile adequately supports the
project. The goal of the assessment is to determine if the project’s funding is available when needed.
Including unallocated future expenses (UFE).
One specific area the SRB Programmatic Team should analyze is the annual cost phasing and budget/ New
Obligation Authority (NOA) by fiscal year. This should show how it supports the project's proposed
schedule and deliverables. The SRB Programmatic Team should also assess how the phasing plan was
developed, including the assumptions and strategies used, particularly as they relate to the BOE, historical
analogs, the project’s proposed schedule and deliverables and the SRB’s assessment. The SRB
Programmatic Team should address whether the proposed phasing matches the project ability to support
the project. Lastly, the SRB Programmatic Team should address how much cost carryover is assumed each
year of the project, both in absolute dollars and weeks or months of work.

12.2.2 Assessment of Reasonableness

NASA policy does not require an independent cost estimate (ICE) be performed at any SRB milestone.
NASA’s independent programmatic function is required to support the SRB by providing an independent
cost (and schedule) assessment of the project’s provided products.
Though a traditional ICE is not required, it is recognized that performing an adequate assessment may
require the analyst to do benchmarking activities to make sure estimates are reasonable and to help
facilitate conversation about assessing the project’s programmatic products. The term benchmarking will
be used throughout this document to represent the practice of performing separate analysis, as required,
to help inform the SRB of a project’s programmatic product input realisms.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 27 of 59
Title: SRB Programmatic Assessment Process

Benchmarking is a cost or schedule analysis conducted to determine the reasonableness of the project’s
submitted estimate (cost or schedule) or to assess a specific input from of the estimate. Benchmarking
can be performed using any of the cost methodologies specified in the NASA Cost Estimating Handbook,
Appendix C.
When and where benchmarking is required is left to the SRB Programmatic Team discretion. However, it
is recommended that some benchmarking is done for the following project product attributes:

 High risk subsystems/elements that are significant drivers of cost and schedule

 Elements of the project’s BOE which do not pass BOE assessment criteria

 Elements of the project where the SRB requests further analysis to fully understand estimate
and/or risk posture

 Elements of work for which only preliminary or ROM cost and schedule estimates exist (i.e., Phase
E/F estimates at early review gates)
If all, or a majority, of a project’s estimate have incomplete BOEs or perceived unrealistic optimism,
benchmarking of all activities may be warranted. Further, if an element within a project has a very
defendable and traceable BOE, a benchmarking activity may not be warranted.
Benchmarking rationale and results should be communicated to the SRB and the project.

Table 4: Cost summarizes expectations and responsibilities, by phase, with regards to cost for the SRB
Programmatic Team.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 28 of 59
Title: SRB Programmatic Assessment Process

Table 4: Cost Review Expectation

Cost Review Expectation Source or Concept &Technology Preliminary Design & Final Design & Fabrication
Responsibility Development Technology Completion (CDR)
(SRR) (PDR)

Compile Project Estimates Project Exist Exist, Detailed Estimate Exist, contract award data
& BOE15 for the phase the Project & detailed estimates for
is entering in to, self-performed work
derivation of out phases
will be at a higher level
Project Summary Project Exist Exist, aligned with Exist, aligned with
Schedule integrated master integrated master
schedule schedule
Historical Data ONCE Project & SRB Project & SRB Project & SRB
Programmatic Team Programmatic Team Programmatic Team
compile compile compile
Review Project Estimates SRB Programmatic Understand scope, content, & layout
& BOE Team

Project Summary SRB Programmatic

Schedule Team

Historical Data SRB Programmatic What is the range of timelines & budgets, which mission or missions are most like
Team the project?

Analyze & Project Estimates SRB Programmatic Entire scope covered Entire scope covered & Entire scope covered &
Validate Team & documented documented & documented &
performance to date. Are performance to date. Are
any corrections realistic? any corrections realistic?
Project Summary SRB Programmatic Major tasks aligned to Major tasks aligned to Major tasks aligned to
Schedule Team funding funding, performance to funding, performance to
date date
Historical Data SRB Programmatic Timelines & budgets, Timelines & budgets, Timelines & budgets,
Team phasing within family? phasing within Family? phasing within Family?

Document Questions SRB Programmatic If SRB Programmatic Team doesn’t understand, IG/GAO, etc., probably won’t
Team either. Ask questions to help project tell story.

Findings SRB Programmatic Identify any apparent disconnects. Explained to project so they can
Team validate/understand them.

Historical SRB Programmatic Show comparisons to all similar missions, any outliers, and identify which
Comparisons Team mission(s) are most like current project

Discuss & Project Team SRB Programmatic Talk to project team first, ask questions, determine if disconnects are disconnects
Brief Team, & project or just lack of clarity

SRB SRB Programmatic Talk to SRB, see if they have any technical concerns that may drive costs. If so, add
Programmatic Team & SRB to findings, and identify to project team.
Team & SRB
Management SRB, SRB Identify any unresolved disconnects & show historical comparisons
Programmatic
Team, & project

15
BOE maturity may not be homogeneous in detail. For example, at SRR BOE detail for phase B should have more
detail than BOE’s in phase D.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 29 of 59
Title: SRB Programmatic Assessment Process

13.0 SCHEDULE ASSESSMENT

Schedule assessments are performed as part of the SRB’s IPA. This section is intended to provide guidance
to SRB programmatic analysis team members and consultant, as well as providing some things to consider
during a SRB LCR.

The schedule assessment helps to determine whether the project has implemented scheduling best
practices and is in accordance with Agency requirements. The schedule assessment should validate that:

 The schedule control plan aligns with stakeholder objectives, and best practices are being used
to manage the project schedule

 The schedule is aligned with the technical goals of the project

 The schedule has been integrated with the budgeting/funding strategy

 The availability of resources other than budget has been considered and appropriate resources
have been incorporated,

 Risks have been identified and are being actively managed, consistently risk informing the
schedule so that the project can make informed management decisions.
Another intent of the schedule assessment is to quantify the level of confidence that the Agency can
commit to externally for date of project completion, and that the project will be able to accomplish its
technical goals while executing its schedule.

13.1 Schedule Requirements Review Process

This section is intended to provide guidance to the SRB programmatic analysis members and consultants
with regards to the planned schedule requirement within NPR 7120.5E. Specifically, this section provides
guidance to following requirement:
Table 5: NPR 7120.5E Schedule Requirements

Paragraph Requirement Statement General Taxonomy

Reference
2.4.2 All programs and projects develop cost estimates and planned Cost/Schedule
schedules for the work to be performed in the current and following
life cycle phases (see 7120.5 Appendix I tables). As part of developing
these estimates, the program or project shall document the basis of
estimate (BOE in retrievable program or project records).

This section only addresses the schedule portion of the abovementioned requirement. The cost section is
covered in Section 12.0 Cost Assessment.

13.1.1 Scope
Scope of schedule assessment should include entire life cycle of project or as defined in the ToR. This
typically includes phases A through E. The foundation of schedule assessment is the IMS.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 30 of 59
Title: SRB Programmatic Assessment Process

13.1.2 Basis of Estimates

The schedule assessment should start with the documentation, this includes a review of the project’s
integrated master schedule and the schedule BOE, schedule summary, and lower level detailed
information. The purpose of this review is to ensure that the schedule was developed and is presented in
an understandable manner that is clear, complete, consistent, repeatable, traceable, and defendable.

Schedule BOEs should include documented rationale for project task durations. They may take on a variety
of forms and may not be fully contained in one data product. The SRB Programmatic Team should verify
that all project constraints and assumptions along with other supporting historical/ana logous data
sources, and mappings to cost BOE of the same WBS element are identified within the schedule BOE.

The SRB Programmatic Team should be addressing the following attributes for each BOE:

 Task(s) Description
o Are the project schedule BOEs formally documented?
o Does the BOE have any unique ground rules and assumptions to consider?
o Is there a clear trace from the schedule to the costs?
 Rationale/Methodology
o Are the sources for deriving estimates identified: established standards, expert judgment,
analogous comparisons, time estimates based upon historical data from past/related
projects, parametric analysis, team brainstorming, and extrapolations from known data
and trends?
o Is the estimating methodology appropriate for the given milestone?
o Are adjustments and assumptions adequately explained?
o Are changes from previous estimates tracked?
 Source Data
o Is the schedule basis sound, realistic, and executable, such that activity durations are
based upon normal work schedules and calendars and do not contain padding or buffer?
o Are judgments or rationale well justified, analogies appropriate, and schedule estimating
relationships applied?
 Accurate
o Is there any evidence of bias?
o Are activity durations based upon the effort required, available resources, and resource
efficiency?
o Are duration time units (i.e., work days) consistent throughout the schedule?

13.1.2.1 Ground Rules and Assumptions

The BOEs typically provide a detailed description of the ground rules and assumptions used to develop
schedules. Estimated durations and schedule logic support the deliverables within the schedule.

13.1.2.2 Basis of Estimate Assessment Criteria

Please refer to Appendix A: SRB Programmatic Team Aids and Product Templates for an example BOE
assessment criteria.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 31 of 59
Title: SRB Programmatic Assessment Process

13.2 Schedule Assessment Process

Like the cost assessment process, the schedule assessment has five main steps: compile data, review,
analysis/validate; document, discuss/brief, and any required iterations. The steps and example high-level
questions to answer during each process:

 Compile
o Request project data
o Compile historical data
 Review
o Read and comprehend the project scope, assumptions, liens, threats, risks, and any
exclusions
 Analysis/Validation
o Check if schedule captures project scope
o Check for schedule reasonableness
o Ensure that project duration is within family of analogous projects
o Understand how level of effort activities are captured in the schedule and treated by
project analysis
 Document
o Document questions and concerns
o Compare to historical project information
 Discuss/Brief
o Talk to project frequently; ask questions; share concerns
o Talk to SRB about findings, specially issues, concerns, and observations

Compiling data is discussed in

Figure 1: SRB Programmatic Analysis Archive File Structure (with example files)
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 32 of 59
Title: SRB Programmatic Assessment Process
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 33 of 59
Title: SRB Programmatic Assessment Process

7.0 Data Drops. Reviewing the schedule estimating methodology is covered in 13.1.2 Basis of Estimates.
The schedule review focuses on ensuring estimate meets requirements and best practices16 . This section
will specifically cover the Analysis/Validation step discussed above. The Analysis/Validation step is
covered in two sections:

 13.2.1 Project Schedule Assessment

 13.2.2 Schedule Assessment of Reasonableness
The SRB Programmatic Team should verify that objective-driven requirements clearly flow down through
the WBS, as the WBS provides the project structure and serves as a framework for the schedule
development (and financial management). For projects, the integrated master schedule provides the
management vehicle, which enables integration of the approved project work scope reflected in the WBS,
budget, and certain project risks. The integrated master schedule will reflect both the project approved
time-phased baseline plan, including all subsequent approved changes, and the time-phased plan with its
current task progress, sequence, and forecasts. the SRB Programmatic Team should be able trace that the
project schedule reflects the total scope of the program and the baseline mission design with the status
of schedule data. 13.2.1 Project Schedule Assessment

When assessing project schedules, the SRB Programmatic Team should consider the following areas and
questions as part of the assessment:

The SRB Programmatic Team should review the project plan.

 Is the project plan complete and consistent with other project documents and supported by the
project’s scheduling approach?

The SRB Programmatic Team should assess whether a stand-alone plan or part of either the project Plan
or combined Technical, Schedule, Cost Control Plan, or the Schedule Management Plan captures content
per the key areas in the Schedule Management Plan Template (Schedule Management Handbook,
Appendix F), 17 as well as for its general alignment with scheduling processes and best practices as detailed
in the NASA Schedule Management Handbook. The SRB Programmatic Team should review the
documented approach, techniques, and methods the project intends to use in implementing the schedule
management process.

 Is schedule management, including tracking and control, being performed in accordance with and
integrated with the institutional EVM processes and methodologies on projects?
 Are there any efficiencies or deficiencies in the project’s processes or any issues with the project’s
ability to follow its processes as the project moves through its life cycle?
 Are appropriate analytical tools, reports, and information provided to managers to make
informed decisions?

The SRB Programmatic Team should understand the project’s major acquisitions in relation to the project
WBS and the integrated master schedule.

16
NASA Schedule Management Handbook, NASA/SP-2010-3403.
17
NASA Schedule Management Handbook, NASA/SP-2010-3403. Appendix F. March 2011.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 34 of 59
Title: SRB Programmatic Assessment Process

 Are major deliverable items accurately included in the integrated master schedule?
For projects in phases C and D that are required to perform EVM, 18 the SRB Programmatic Team should
be aware of which contracts require EVM, as these contracts should be delivering Integrated Performance
Management Reports (IPMRs), a WBS, and an integrated master schedule that will inform the project. The
SRB Programmatic Team should also be familiar with any findings coming out of the Integrated Baseline
Review (IBR), which is held in preparation for KDP-C.
It is a NASA best practice for all reporting to trace from a single integrated master schedule dataset and
not from separate schedule sources.

 Does the integrated master schedule accurately reflect accomplished work and planned work?
 How is the project performing integrated master schedule updates, analyzing schedule impacts,
and resolving issues to provide updated schedule reporting to project management and necessary
customers?
The SRB Programmatic Team should review the project schedule reports; these reports are sometimes
contained in Monthly or Quarterly Status Reports (Monthly Status Reports [MSRs] or Quarterly Status
Reports [QSRs]). Schedule reports may include a management summary schedule, logic reports, critical
path reports, total slack reports, schedule risk reports, schedule margin metrics, and performance trends.

 Is the project generating these reports using sound methodologies?

 Is the content of the reports adequate for decision making?
o For instance, the reports should provide management with a realistic understanding of
the status of the project including warning signs of potential problems (i.e., risks), as well
as a critical path assessment, plan versus actual status, milestone trends, float/slack
utilization, and reserve status?
 Is project schedule reporting being conducted consistent with overall Agency requirements (i.e.,
7120.5, 7120.7, 7120.8, etc...)?
The management and reporting requirements for applicable procurements may be contained in the
contract Statement of Work, Contract Data Requirements List, and/or Data Requirements Document. To
effectively integrate contractor schedule data into the project integrated master schedule it is imperative
that a clear understanding exists between the government and contractors about such details as schedule
content, level of detail, formats, reporting frequency, tools, thresholds, responsibilities, and controls.
While the SRB Programmatic Team is not expected to review these products, the project’s process of
obtaining and incorporating the schedule information necessary to manage the integrated master
schedule and enable informed decision making should be understood.
For partnerships between a project and other NASA Centers, research institutions, international partners,
or other business arrangements not involving contracts or procurements, schedule reporting

18“Projects in phases C and D with a life cycle cost estimated to be greater than $20 million and Phase E project
modifications, enhancements, or upgrades with an estimated development cost greater than $20 million are
required to perform earned value management (EVM) with an EVM system that complies with the guidelines in
ANSI/EIA-748, Standard for Earned Value Management Systems…. EVM system requirements shall be applied to
appropriate suppliers in accordance with the NASA Federal Acquisition Regulation (FAR) Supplement, and to in-
house work elements.” NPR 7120.5E. Chapter 2. Section 2.2.8.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 35 of 59
Title: SRB Programmatic Assessment Process

requirements should be incorporated into a Memorandum of Understanding, Space Act Agreement,

Letter of Agreement, Statement of Work, or other appropriate documents. This will enable the integrated
master schedule to fulfill its intended function as an effective and efficient integrated project
management tool.

 Do these agreements detail the level of reporting information the project expects to receive and
ensure that the project is incorporating the appropriate level of detail into the integrated master
schedule?

The SRB Programmatic Team should understand the project’s UFE posture as it relates to the integrated
master schedule, milestones, and schedule risk impacts.

 Is the schedule margin funded through the budget baseline?

o Is there enough UFE available to accommodate the schedule risk impacts identified by the
project and SRB?

The SRB Programmatic Team should verify that the integrated master schedule captures the total scope
of work at an appropriate level of detail. The SRB Programmatic Team should also review the technical
progress against schedule performance, including meeting technology readiness levels (TRLs).

If the project has experienced any technical performance issues, the SRB Programmatic Team should
communicate these with the SRB, as they may indicate additional risk to meeting schedule objectives and
may require further analysis.

 Are there changes in scope that were not part of the baseline plan?
o How are these changes incorporated into the schedule?
o Are there technical drivers driving schedule performance?

Schedule credibility is determined by monitoring key indicators within the integrated master schedule
that reflect both good and poor characteristics of schedule structure and maintenance and support
scheduling best practices.

 Examples of key indicators within the logic network that should be monitored include, but are not
limited to: missing predecessors and successors, invalid task constraints, omission of task status,
improper status on future tasks, logic ties to and from summary tasks, inaccurate logic ties, and
improperly reflecting tasks as milestones.
The SRB Programmatic Team should use a health and quality check to determine whether the project
schedule has been developed using NASA standard best practices per the NASA Schedule Management
Handbook. Typically, these metrics will have defined thresholds that should be considered as guidelines,
which serve as trigger points for additional analysis.

 Did the project conduct a schedule health check?

 What are the project’s schedule health metrics and associated thresholds?
o The SRB Programmatic Team should discuss with the project any instances where tasks
do not meet the threshold measurements of the selected metrics, and especially if the
project has decided not to adhere to these metrics.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 36 of 59
Title: SRB Programmatic Assessment Process

o Regardless of whether the project provides the SRB with a health and quality check, the
SRB Programmatic Team should perform an independent health/quality check such that
all necessary metrics are reviewed19 (i.e., analysis that focuses on additional metrics may
be necessary to reveal deficiencies not uncovered by the project).
Once the health and quality check of the schedule is performed, the SRB Programmatic Team should work
with the project to resolve issues within the schedule network. The objective is to resolve as many of the
health check issues as possible so that the schedule used for the schedule risk analysis is complete and
well vetted. All health check issues may not be resolved so the analysts should understand their impact
on future schedule risk analysis.
The SRB Programmatic Team should assess the current critical path as calculated in the integrated master
schedule. The analysis should be performed in several ways, first analyzing the critical path in the project’s
native schedule as it is calculated by the scheduling software. They should examine the critical path for
these characteristics:

 Does the schedule critical path(s) start at time of assessment and proceed as a continuous path
to project completion?
 Does the scheduling software tool generate the same critical path that the project is reporting?
 Does the schedule consists of tasks and milestones linked together with network logic using
appropriate relationship-types in a sequence that is programmatically feasible or otherwise
makes sense from a workflow standpoint?
 Does the schedule have any unexplainable lags or leads or constraints that cause unimportant
activities to drive a milestone?
 The schedule critical path(s) contains no level-of-effort activities, summary activities, or other
unusually long activities.
 The schedule has no gaps in time between tasks that cannot be explained.
 The integrated master schedule is derived from the integration of lower-level detailed schedules,
not by preselected activities that management has deemed critical.

Total float is fundamental to the critical path method (CPM) of scheduling. If the task/milestone that
represents the completion of the project has a hard constraint date assigned to it, then there would be a
possibility that the critical path could have a positive or negative total float value instead of zero. The SRB
Programmatic Team should examine the total float calculations for critical path activities and other critical
path analysis.

 Does the schedule have hard constraints?

o Are they justified?

19 Several of these software tools are available for download. The Schedule Test and Assessment Tool (STAT) software may be
requested via the NASA software catalog at https://software.nasa.gov/software/MFS-33362-1. Other software applications are
also available via the One NASA Cost Engineering (ONCE) database at
https://oncedata.msfc.nasa.gov/%28S%28bma3ciyzi0pxghnmv21igowf%29%29/default.aspx
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 37 of 59
Title: SRB Programmatic Assessment Process

 Is the logic automatically updating? Are manual entries and constraints misrepresenting the
critical path and float calculations? What are the results of the Defense Contract Management
Agency critical path test?
 What is the project integrated master schedule critical path length index?

Through each iteration of critical path analysis, the SRB Programmatic Team should monitor the near -
critical paths, also referred to as secondary or tertiary paths, to understand the sensitivity of the schedule,
as well as where the schedule has flexibility and where it does not. Tasks with total float within a narrow
range of the critical path total float are near critical. They should also review past schedule reports,
including MSRs and QSRs.

 What has the project identified as the critical path and near-critical paths over the course of
project execution and what is the rationale for any changes in these paths over time?
 Does the project IMS critical path match the SRB programmatic team identified IMS critical
path?
 If a SRB analysis schedule is created, does the SRB critical path match the project identified
critical path?

The SRB Programmatic Team should examine the metrics that the project is tracking and discuss the
results of those metrics with the SRB with respect to schedule performance. The team should also assess
whether the program or project is tracking performance at least monthly in the LCR window of 120 days
prior to the site review.
o Are there issues or past problems that should be incorporated in the threat, lien, or risk
lists?

The SRB Programmatic Team should perform margin analysis, 20 determining whether the project has
identified enough margin to account for schedule risk impacts and other unknown unknowns
(uncertainty) that may threaten the project completion. The results of the schedule risk analysis help
determine the adequacy and appropriate placement of the schedule margin in the integrated master
schedule.

 Is the margin consistently identifiable within the integrated master schedule? Is it hidden within
the duration of other tasks?
 Does the margin meet the Center’s margin guidelines?
 Are the schedule margins realistic? Have they been validated (i.e., schedule risk analysis)?
 Are there adequate management reserves to cover unfunded schedule margin?
 Are the schedule margins consistent with similar missions?
 Are margin and slack being tracked synonymously? (Note: Slack/float is not the same as margin)
 Is there methodology to how the margins were derived (e.g., expert judgment, rules of thumb,
insight from schedule risk analysis)?
 How does the project manage margin? Is there a burn-down plan? Who controls it?

20 Further information about margin assessment can be found in the NASA Schedule Management Handbook.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 38 of 59
Title: SRB Programmatic Assessment Process

 Is there a process to guide how margin will be used to offset of scope change, schedule growth,
and potential risks?
 Is the schedule margin funded?

Workforce planning is heavily dependent on the integrated master schedule for time phasing. The SRB
Programmatic Team should examine whether the project has considered potential equipment and facility
conflicts.

 Is the appropriate workforce (skillset and number) available with regards to the planned work?
 Do facility conflicts exist (e.g., for testing in thermal vacuum chambers)? Are there mitigations to
accommodate potential conflicts?
If the appropriate workforce or facilities are not available at the appropriate time, there may be additional
risk to the schedule. Furthermore, augmentation to staffing plans may be needed to cover threats, liens,
and/or possible impacts from schedule risks such as late deliveries. The SRB Programmatic Team needs to
understand the schedule and cost impacts of such threats, liens, and risks as they may reduce UFE and
schedule margin.

13.2.2 Schedule Assessment of Reasonableness

The appropriateness of the schedule should be considered by the SRB Programmatic Team. Is there
perceived unrealistic optimism of schedule duration or lack of schedule detail that may indicate lack of
understanding of the work and sequence required to develop and mature the project through the life
cycle phases?
As with cost estimates, benchmarking is conducted to determine the reasonableness of the project’s
estimate (cost and schedule) or to assess the specific input to the schedule estimate.
When and how schedule benchmarking is required and performed is left to the programmatic analyst’s
discretion. An example benchmarking practice is to compare milestone and total development schedule
life cycles of analogous missions.
Schedule assessments occur at different points in a project life cycle (i.e., maturity). To assess the level of
maturity of schedules at each LCR, refer to the NASA Schedule Management Handbook, Section 5.3.2. 21
Table 6: Schedule Assessment outlines project schedule expectations, by phase.
Table 6: Schedule Assessment Expectation

Schedule Expectations Concept & Technology Preliminary Design & Final Design &
Completion Technology Fabrication, System
(SRR, SDR) Completion (PDR) Assembly, Integration
& Test, Launch &
Checkout
(CDR, SIR, ORR)
Scheduling Tool Team possesses tool Intermediate level Intermediate level Expert level capable of
Considerations expertise capable of Schedule capable of complex developing what ifs,
Development critical path analysis schedule crashing
scenarios, etc.

21
NASA Schedule Management Handbook, NASA-SP-2010-3402 (https://www.nasa.gov/pdf/420297main_NASA-SP-
2010-3403.pdf)
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 39 of 59
Title: SRB Programmatic Assessment Process

Tool(s) selected produce

adequate summary
  
schedules, status views, and
critical path views
Tool selection appropriate for
  
Project complexity
Schedule WBS and Organizational System Level Subsystem Level At lowest level of the
Development Breakdown Structure mapped WBS
to the integrated master
schedule
Schedule Management Plan Draft Baseline Baseline and revisions
Task / activity definition / Sufficient detail to allow Activity duration Activity duration
duration accurate logic should follow GAO should follow GAO best
best practice of 44 practice of 44 working
working days or less, days or less, with
with agreed to agreed to exceptions
exceptions
Schedule margin / reserve Allocated at program Allocated at project Allocated at project
level level to safeguard level to safeguard
critical path tasks critical path tasks, and
tasks with high
likelihood / high
consequence risks
Schedule logic All tasks have All tasks have All tasks have
predecessors and predecessors and predecessors and
successors, with successors, with successors, with
exceptions for outside exceptions for outside exceptions for outside
deliveries deliveries, and all tasks deliveries, and all tasks
logic eventually leads logic eventually leads
to project completion to project completion
milestone milestone
Integrated master schedule
 
baseline established
Calculated critical path Single tool created Primary, secondary, Primary, secondary,
critical path terminating and tertiary critical and tertiary critical
at project completion path's terminating at path's terminating at
milestone project completion project completion
milestone milestone
Schedule Data backup and archive   
Maintenance Schedule updates and Daily Weekly / monthly Weekly / monthly
evidence of continuous
updates
Status update accounting  
Schedule Schedule logic measurement Less than 10% missing Less than 5% missing Less than 5% missing
Assessment and logic logic logic
Performance Schedule logic effectiveness All tasks lead to a All tasks lead to the All tasks lead to the
Monitoring critical program program completion program completion
milestone milestone milestone
Critical path analysis Zero float path Zero float path Zero float path
terminating at Critical terminating at project terminating at project
project milestone termination milestone termination milestone
Schedule margin assessment Sufficient margin exists Sufficient margin Sufficient margin exists
to cover program risks exists to cover to cover program risks,
program risks, and and usage trends
usage trends support support this
this
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 40 of 59
Title: SRB Programmatic Assessment Process

Constraint usage Minimal hard Minimal hard Minimal hard

constraints constraints, soft constraints, soft
constraints only used constraints only used
for external deliveries for external deliveries
Schedule Control Baseline control process Baseline control Baseline control
process is robust and process is robust and
adhered to adhered to
Schedule forecast Control Schedule forecast is Schedule forecast is
managed to safeguard managed to safeguard
critical program critical program
milestones milestones
Schedule exceptions Documented examples Documented examples
reporting of schedule baseline of schedule baseline
and / or schedule and / or schedule
forecast exceptions forecast exceptions
reporting to reporting to
management management

14.0 CONFIDENCE LEVEL REQUIR EMENTS REVIEW PROCESS

This section is intended to provide guidance to the confidence level requirements within NPR 7120.5E.
Specifically, this section provides guidance to following requirements:
Table 7: NPR 7120.5E Confidence Level Requirements

Paragraph Requirement Statement General

Reference Taxonomy
2.2.4 Each program and project shall perform the LCRs identified in its respective Risk
figure in accordance with NPR 7123.1, applicable Center practices, and the
requirements of this document. These reviews provide a periodic assessment
of the program's or project's technical and programmatic status and health at
key points in the life cycle using six criteria: alignment with and contribution to
Agency strategic goals, adequacy of management approach, adequacy of
technical approach, adequacy of the integrated cost and schedule estimates
and funding strategy, adequacy and availability of resources other than budget,
and adequacy of the risk management approach. (See NPR 8000.4 Agency Risk
Management Procedural Requirements and NASA/SP-2011-3422 NASA Risk
Management Handbook for further requirements and guidance on risk
management and the NASA Space Flight Program and Project Management
Handbook for further guidance on addressing the expected maturity for each
of these criteria.) A life-cycle review is complete when the governing Program
Management Council (PMC) and Decision Authority (DA) complete their
assessment and sign the Decision Memorandum (see paragraph 2.4.1).
2.4.2 All programs and projects develop cost estimates and planned schedules for Cost/Schedule
the work to be performed in the current and following life-cycle phases (see
7120.5 Appendix I tables). As part of developing these estimates, the program
or project shall document the basis of estimate (BOE in retrievable program or
project records).
2.4.3 Tightly coupled and single-project programs (regardless of life-cycle cost) and Programmatic
projects (with an estimated life-cycle cost greater than $250 million) shall Estimating
develop probabilistic analyses of cost and schedule estimates to obtain a
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 41 of 59
Title: SRB Programmatic Assessment Process

Paragraph Requirement Statement General

Reference Taxonomy
quantitative measure of the likelihood that the estimate will be met in
accordance with the following requirements.
2.4.3.1 Tightly coupled and single-project programs (regardless of life-cycle cost) and Range Estimate
projects (with an estimated life-cycle cost greater than $250 million) shall
provide a range of cost and a range for schedule at KDP 0/KDP B, each range
(with confidence levels identified for the low and high values of the range)
established by a probabilistic analysis and based on identified resources and
associated uncertainties by fiscal year.
2.4.3.2 At KDP I/KDP C, tightly coupled and single-project programs (regardless of life- JCL
cycle cost) and projects (with an estimated life-cycle cost greater than $250
million) shall develop a resource-loaded schedule and perform a risk-informed
probabilistic analysis that produces a JCL.
2.4.4.3 When a tightly coupled program, single-project program, or project with an JCL
estimated life-cycle cost greater than $250M is re-baselined, the JCL should be
recalculated and approved as part of the re-baselining approval process.

This section is divided into two major sections to cover the requirements above. The first section will
provide both cost and schedule guidance on performing an assessment on the project’s cost and schedule
range estimate requirement. The second section will provide guidance on assessing the Agency’s JCL
requirements.

14.1 Cost and Schedule Range Estimate Assessments (KDP-0/KDP-B)

The KDP-0/KDP-B requirement for range estimates focuses on the project providing a high and low cost
and schedule estimate for the project to be successful and meet objectives. The SRB Programmatic Team
assessment of cost and schedule range estimate requirements is intended to focus on the reasonableness
of the project generated cost and schedule range estimate by analyzing the validity of the range estimate
input parameters, assumptions, and overall quality of the product. NASA guidance does not direct how a
project generates their range estimates for cost and schedule but a few underlying principles should apply
to the products. Range estimates should be based on the project preliminary plan and technical scope;
range estimates should encompass project’s unique risks and uncertainty. Per the NASA Cost Estimating
Handbook and the Schedule Management Handbook, there are several acceptable methodologies for
generating a KDP-0/KDP-B cost and schedule range estimate.

The SRB Programmatic Team assessment of the range estimate will include aspects covered in 12.0 Cost
Assessment and 13.0 Schedule Assessment; but will also include additional factors such as assessing the
project uncertainty and risks in the plans are properly accounted for in the project range estimates. The
project models used to generate the range estimates will be taken and adjusted with SRB technical subject
matter expert inputs and evaluated to identify any significant impacts to the project plan.

Though the general process for evaluation of both products can be similar, the mechanics of evaluating
the products can vary depending on how the project generates the range estimates.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 42 of 59
Title: SRB Programmatic Assessment Process

14.1.1 Basis of Estimate

For all inputs to support the range estimate there should be a BOE for any risk analyses conducted (e.g.,
Monte Carlo simulation, identification of risk mitigation strategies) including probability distribut ion
assumptions and how their results were used to create the probabilistic cost and schedule estimates.

14.1.1.1 Scope
Please refer to 12.1.1 Scope and 13.1.1 Scope for a general conversation with regards to scope. The
project’s range estimates scope should include any risks, opportunities, and uncertainties that the project
controls and items that the project doesn’t manage but could affect the project’s baseline. For example,
as specified in the NASA Cost Estimating Handbook, with regards to international/inter-Agency
contributions, inter-project/program risks, and launch vehicle selection; the project is tasked to include
the programmatic risk of cost and schedule impacts to the project stemming from those systems. The
project should coordinate with the international, inter-Agency, inter-project/program, and appropriate
launch services entities when available; as well as coordinate with its mission directorate, to determine
the adjudication and communication of the risks (ownership). Further work should be performed to
determine how those risks will be incorporated and communicated in the range calculations. The SRB
Programmatic Team will have the responsibility to evaluate all aspects of the range, including
international/inter-Agency and inter-project/program relationships.

14.1.1.2 Ground Rules and Assumptions

The BOE should provide a detailed description of the ground rules and assumptions used to develop the
range estimates. The ground rules and assumptions help provide insight into what risks, opportunities,
and uncertainties are included and excluded from the estimate and scope. For example, the project could
make assumptions that certain risks will not happen, such as a hurricane striking an integration center.
These exclusions may be fine but need to be understood

14.1.2 Assessing the Cost and Schedule Range Estimates

The SRB Programmatic Team should evaluate three aspects of the cost and schedule range estimates:

 Credibility of project inputs to the range estimate

 Adequacy of the risk management plan (RMP), process, risk identification, and mitigation plans
 Reasonableness of project cost and schedule estimate

14.1.2.1 Assessment of Project Cost and Schedule Range Estimate Model

The SRB Programmatic Team will assess the project cost and schedule range estimates.

If the range estimates are generated using parametric or analogous data, then the SRB Programmatic
Team should assess the following:22

 Applicability of the project input data and the model (e.g., tools) being used to support range
estimate calculations. Is the basis for the range estimates analogous to the project? If the project
is using analogous data to support range estimates than SRB Programmatic Team should obtain
inputs from the SRB if the analogies are reasonable. The team should use inputs from the technical

22
It is important to note that the SRB Programmatic Team analysts may be required to generate SRB range estimates
in the event the project does not satisfy range estimate requirements or at the request of the SRB Chair or convening
authorities.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 43 of 59
Title: SRB Programmatic Assessment Process

SRB subject matter experts (approved by the SRB Chair) to adjust the project range estimates and
provide results to the SRB.
 Assessment that the project input data and model (e.g., tools) are being utilized properly. For
example, if the project uses parametric tools, are both inputs of uncertainty and CER uncertainty
being properly utilized? SRB technical subject matter input can be solicited, but the SRB
Programmatic Team will most likely be the SRB subject matter experts on CER best practices. 23
 Assess the project input data and model (e.g., tools). Are the project inputs to the cost and
schedule range estimates reasonable?
 Assess if the project risks are captured in the input data or model.
If the range estimates are generated via a schedule risk analysis, then refer to 13.0 Schedule Assessment.
It is important to note that a JCL is not required to fulfill a KDP-0/KDP-B range estimate requirement. The
assessment of a grassroots input looks at the schedule (schedule risk analysis) and cost range estimates
separately.

14.1.2.2 Risk Management Assessment of Cost and Schedule Range Estimate

The SRB Programmatic Team will coordinate with the SRB to assess the project risk management approach
beginning with an evaluation of the Risk Management Plan. 24 The SRB Programmatic Team should
evaluate if the Risk Management Plan summarizes how the program implements risk-informed decision
making (RIDM) and continuous risk management (CRM) in accordance with NPR 8000.4A.

The project risk list provided for the LCR should include all risks and appropriate actions to mitigate each
risk. Project’s with international or other U.S. Government Agency contributions must plan for, assess,
and report on risks due to international or other government partners and plan for contingencies.
The SRB Programmatic Team will conduct an independent risk assessment to determine if the project
meets RIDM and CRM processes in accordance with NPR 8000.4A requirements regarding programmatic
risk. The SRB technical risk SME will conduct an independent risk assessment to determine if the project
meets RIDM and CRM processes regarding safety and mission assurance risk. As examples of areas to
assess, the project:

 Documents risk acceptability criteria and thresholds, as well as elevation protocols (the specific
conditions under which a risk management decision must be elevated through management to
the next higher level).
 Establishes risk communication protocols between management levels, including the frequency
and content of reporting, as well as identification of entities that will receive risk-tracking data
from the unit's risk management activity.
 Conducts CRM process.
o Identify: Identify contributors to risk.
o Analyze: Estimate the probability and consequence components of the risk through
analysis, including uncertainty in the probabilities and consequences and, as appropriate,
estimate aggregate risks.

23 NASA Cost Estimating Handbook, v.4.0, February 2015, 2.3.3; NASA Schedule Management Handbook, NASA/SP-
20910-3403, March 2011, 7.9
24 As identified by NPR 7120.5E; Tables I -1, I-3, I-7 Program Plan Control Plans, and Appendix G section 3.3 for

Programs, Table I-5 Project Plan Control Plans and Appendix H for projects.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 44 of 59
Title: SRB Programmatic Assessment Process

o Plan: Decide on risk disposition and handling, develop and execute mitigation plans, and
decide what will be tracked.
o Track: Track observables relating to performance measures (e.g., technical performance
data, schedule variances), as well as the cumulative effects of risk disposition (handling)
decisions.
o Control: Control risk by evaluating tracking data to verify effectiveness of mitigation
plans, adjusting the plans as necessary, and executing control measures.
o Communicate and Document: Communicate and document the above activities
throughout the process.

14.1.2.3 Uncertainty Assessment of Cost and Schedule Range Estimates

Like Cost and Schedule BOEs, modeled uncertainty for range estimates need to have a basis of estimate.
Refer to 14.2.2.4 Uncertainty Assessment of JCL for additional details.

14.1.2.4 Assessment of Cost and Schedule Range Estimate Reasonableness

NASA’s independent assessment model requires the SRB to provide an independent cost and schedule
assessment of the project programmatic LCR products. For example, when a project provides a JCL to the
SRB to assess, the expectation is not for the SRB programmatic analysts to perform a separate
independent JCL analysis but assess the project inputs to the project cost and schedule estimates. This
applies to cost and schedule range estimate and for risk-informed and schedule adjusted baseline cost
estimate updates.

NASA policy does not require a SRB ICE for any LCR milestone.
Though an ICE is not required, it is recognized that performing an adequate independent assessment may
require the SRB Programmatic Team to generate a benchmarking analysis (e.g., performing separate
analysis, as required, to help inform the SRB of the reasonableness of the project estimates) to ensure
cost and schedule estimates are reasonable and to facilitate conversations with regards to assessing the
project’s programmatic products.

14.1.2.5 Cost and Schedule Range Estimate Confidence Levels

NASA policy does direct what the associated confidence levels for the cost and schedule range should be
at KDP-0/KDP-B. The policy states the associated high and low confidence levels need to be documented
to support the range estimates and provide a high and low confidence level for both schedule and cost
range estimates. The SRB Programmatic Team, with SRB subject matter expert inputs, should assess
whether the recommended ranges are reasonable.

14.1.3 Key attributes to a successful Cost and Schedule range estimate SRB assessment
The SRB Programmatic Team supports the SRB to determine if the project has identified and accurately
quantified all the known risks and if the uncertainty boundaries in the cost and schedule estimates are
appropriate. When assessing the project’s risks and confidence level calculation, the SRB should be able
to answer the following questions25 :

25
It should be noted that answers to these questions provides a solid basis to communicate the results of the SRB
programmatic assessment to NASA decision makers.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 45 of 59
Title: SRB Programmatic Assessment Process

 Has the SRB identified new risks for the range estimates or adjusted risk likelihoods, consequences
or impacts different from the project? The SRB should provide the technical rationale for the
differences from the project inputs in the risk assessment.
 Do the SRB’s uncertainty boundaries differ from the projects? If so provide the technical rationale
for the differences from the project inputs in the risk assessment.
 When the project range estimate models are updated using the SRB SME adjustments to risk and
uncertainty distribution, how do the results compare to the project’s proposed budget and
schedule?
 After evaluating risk and uncertainty drivers, what changes to the risk list and uncertainty
distribution have the biggest impact compared to the project’s proposed cost and schedule?
 How does the ratio of the assessor-identified risks to uncertainty differ from the project’s inputs?
 If appropriate, have the risks and impact of missing a launch window been included in the project’s
risk list?
 Have risks associated with partner/international contributions been included in the project’s risk
list? Has the project identified the impact if partner/international contributions are not provided
or are provided later than in the project’s plan, and have assessed alternatives if needed?

14.2 Joint Confidence Level Assessment

The SRB Programmatic Team should assess the project JCL for KDP-C or a Rebaseline Review. This
assessment is to determine the reasonableness of the project’s generated cost and schedule range
submissions in support of KDP-C and external stakeholder commitments such as the Agency Baseline
Commitment (ABC).

The SRB is responsible for analyzing the project JCL to determine the validity of the JCL inputs (e.g., cost,
schedule, risk, uncertainty) and the reasonableness of the assumptions. The assessment will include all
aspects covered in the cost and schedule estimate sections (see 12.0 Cost Assessment and 13.0 Schedule
Assessment), but will also include additional factors that are JCL modeling specific. The project JCL model
is adjusted with SRB inputs via subject matter expert analysis and evaluated to identify any significant
impacts to the project current resource plan.

14.2.1 Basis of Estimate

A JCL is based on the project’s current cost and schedule resource plan with probabilistic attributes (e.g.,
adding risks and uncertainties). Like the expectations in both the cost and schedule sections, all inputs to
support the JCL should have a BOE. Specifically, the JCL analysis adds the necessity for BOEs to support a
project’s risk and uncertainty inputs, as well as on a project’s time independent or time dependent
assumptions. 26

14.2.1.1 JCL Scope

The scope of the JCL is typically project formulation through the end of Phase D focusing on the
development costs. Please note per NPR 7120.5E this is not the total life cycle costs or operational life of
a project.

26
NASA Cost Estimating Handbook, v.4.0, February 2015, Section 3.1.3.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 46 of 59
Title: SRB Programmatic Assessment Process

The content of what is included in the costs and schedule of the JCL to meet this Phase-D requirement is
agreed upon between the project and convening authorities as part of the ToR or from the PPMB if the
project uses the board to adjudicate differences. As discussed in 14.1 Cost and Schedule Range Estimate
Assessments (KDP-0/KDP-B), the JCL should include any risks, opportunities, and uncertainties that the
project manages and items that the project does not control (e.g., contributions or other funds lines) that
could affect the project’s mission.

Refer to the NASA Cost Estimating Handbook, with regards to international/inter-Agency contributions,
inter-project/program risks, and launch vehicle selection; the project is tasked to include the
programmatic risk of cost and schedule impacts to the project stemming from those systems. The SRB
Programmatic Team should evaluate all aspects of the JCL, including international/inter-Agency
contributions and inter-project/program relationships.

14.2.1.2 JCL Ground Rules and Assumptions

The project should provide detailed description of the ground rules and assumptions used to develop the
JCL. The ground rules and assumptions help provide insight into what risks, opportunities, and
uncertainties are included and excluded from the JCL estimate and scope.

14.2.2 Assessing the JCL

Guidance on how to conduct a JCL is documented in the NASA Cost Estimating Handbook. This section will
follow this process for JCL generation and discuss how to assess each section: build a JCL schedule/logic
network, load cost onto the schedule activities, incorporate a risk list, conduct uncertainty analysis, and
analyze results. Generally, the goal of the assessment is to determine if the project adequately captured
all the risks and uncertainties with regards to cost and schedule.

14.2.2.1 Schedule Logic Network

The schedule logic network is the backbone to creating a JCL. A project may utilize their integrated master
schedule as the JCL analysis schedule (refer to 13.0 Schedule Assessment with regards to how to assess
the deterministic integrated master schedule). The project may also develop an analysis, or summary,
schedule for the JCL analysis27 when the integrated master schedule exceeds a manageable number of
activities to cost and risk load. If the project provides an analysis schedule, then the SRB Programmatic
Team needs to perform schedule health checks, per 13.2.1 Project Schedule Assessment, on both
schedules (integrated master schedule and analysis) and to ensure the analysis schedule accurately
represents the more detailed integrated master schedule (e.g., critical paths are similar). The analysis
schedule should have all schedule margin removed and they should evaluate hard constraints within the
schedule, as theses hard constraints are not recommended for JCL analysis.

14.2.2.2 JCL Cost Loading

The JCL cost inputs should be evaluated per 12.0 Cost Assessment. Additionally, the SRB Programmatic
Team needs to evaluate if the project adequately mapped the costs to the schedule. This mapping should
be clearly documented in the JCL model and include all cost scope per the ToR.

27
For pros and cons associated with the integrated master schedule versus analysis schedule for JCL, please refer to
NASA Cost Estimating Handbook, Section 3.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 47 of 59
Title: SRB Programmatic Assessment Process

The SRB Programmatic Team should assess the adequacy of the project cost inputs to determine if the
inputs are reasonable to be Time Dependent or Time Independent. JCL model adjustments by SRB should
be communicated to the project. Refer to the NASA Cost Estimating Handbook for more clarification of
time dependent and time independent for JCL modeling.

14.2.2.3 Risk Management of Assessment for JCL

The SRB Programmatic Team should assess all risks modeled in the JCL. There are several attributes of the
risk list that need to be assessed:

 Did the project identify all the risks that could affect the project? Cross-reference the risks
modeled in the JCL to the project risk list to ensure that all risks that could influence cost and
schedule are modeled. Furthermore, the SRB Programmatic Team should review the risks with
the SRB for subject matter expert inputs to determine if additional risks should be added or
adjustments to existing risks.
 Are the risks properly linked to the JCL schedule logic? The SRB Programmatic Team should
evaluate and, as appropriate, review with the SRB to verify that the risks are properly linked to
the schedule and adjust as appropriate per SRB subject matter expert inputs.
 Are the risks properly quantified with regards to likelihood and impact? The SRB Programmatic
Team should review project risk likelihood and impact for all JCL-modeled risks and adjust per SRB
subject matter expert inputs as needed.
The JCL model should model each risk against the current cost and schedule plan. Certain risks can be
quantified as pre-mitigated or post-mitigated in the JCL model. If a risk in the JCL model is a post-mitigated
consequence, then the SRB Programmatic Team should make sure that the mitigation strategy is clearly
baselined in the project plan with a funded mitigation strategy. (Refer NASA Cost Estimating Handbook,
Section J.4.1.3, for more discussion on pre-and post-mitigation in JCL models).

The SRB Programmatic Team will work with the SRB to assess the risk management approach starting with
an evaluation of the project Risk Management Plan. 28

14.2.2.4 Uncertainty Assessment of JCL

As defined in the NASA Cost Estimating Handbook, Appendix J, uncertainty is the indefiniteness about a
projects baseline plan. It represents our fundamental inability to perfectly predict the outcome of a future
event. The NASA Cost Estimating Handbook and the NASA Project Planning and Control Handbook provide
excellent dialog on how uncertainty and risks are related. NASA does not dictate at what level uncertainty
is to be applied but it should be clearly documented within the JCL model. Uncertainty should be applied
to activity durations, cost loading (time dependent and time independent), and risk consequences. 29 How
to assess uncertainty will be dependent on how the program/project generates and defends their inputs
with BOEs, but in general the SRB Programmatic Team should ask the following questions when doing the
assessment.

 Data-driven uncertainty:

28 As identified by NPR 7120.5E; Tables I -1, I-3, I-7 Program Plan Control Plans, and Appendix G section 3.3 for
Programs, Table I-5 Project Plan Control Plans and Appendix H for projects.
29 It’s important to note there will be instances where there will not be uncertainty. For each instance of uncertainty,

the SRB Programmatic Team should assess the adequacy of input.

Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 48 of 59
Title: SRB Programmatic Assessment Process

o Was the data normalized? If so, how? If the data was not normalized, some simple
normalization may be warranted (e.g., inflation). For normalized data, oftentimes outlier
events will be “normalized” out. Efforts to understand what the data constitutes is very
important.
o What level is the data, and is the data compatible with the JCL model? Uncertainty metrics
are not always easily transferable from one WBS level to another.
o Is the data relevant to what the project is estimating? Ensure the data is homogeneous to
what is being estimated.
o Is there enough data to support the analysis? Sample size matters. Small samples could
introduce statistical bias in the estimate of population range parameters. This bias should
be considered and accounted for.
 Performance-based uncertainty:
o Is past performance relevant to work forward? For example, financial mutual funds, past
performance may not be a good indicator of the future.
o At what level was the past performance data collected? Level of performance-based
metrics collected for BOE should be the same general fidelity as the JCL model.
 Subject matter expert-based uncertainty:
o Where did the subject matter expert input come from? Ensure the right subject matter
expert provided inputs. For example, a person may be quite the expert in a technical field
but may not have a good handle on the cost and schedule uncertainties of that field;
whereas a recent project manager, or Center cost estimator, may not be as competent in
the technical area but have a better feel for cost and schedule impacts.
o Is there confirmation bias? Confirmation bias is the tendency to search for or interpret
information in a way that confirms one's beliefs or hypotheses. For example, a project
may underestimate the negative uncertainty because they want the project to succeed.
o Is there framing bias? Framing bias can lead to using a too-narrow approach and
description of the situation or issue.
o Is there hindsight bias? Hindsight bias is the inclination to see past events as being
predictable.

14.2.2.5 Analyze JCL Results

All JCL analysis results and inputs for both the project JCL model and SRB-adjusted model should be
communicated with the SRB Chair. The SRB Programmatic Team should provide both the 50 and 70
percent joint cost and schedule confidence levels for both the project and SRB model at a minimum. All
differences between the project JCL and SRB model should be clearly documented and drivers identified.
The team should demonstrate that the cost and schedule deltas between the 50 and 70 percent JCLs are
reasonable, based on major risk drivers, uncertainty distributions, and historical analogous data.
If the current project plan differs from the 50 and 70 percentiles (either the project’s or SRB -adjusted
JCLs), the SRB Programmatic Team should lead a discussion with the SRB on drivers of variation for
accuracy and documentation in preparation for the presentation to support the convening authority’s
project life cycle management councils.

In addition to providing the above results, the following questions should be answered by the SRB
Programmatic Team analysis:
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 49 of 59
Title: SRB Programmatic Assessment Process

 What are the schedule activities and critical paths that may be impacted by risks?
 Has the project identified all probable critical schedule activities? Secondary? Tertiary?
 What is the probability that the schedule milestones dates will be completed on time?
 What are the risk drivers (e.g., risk tornado chart)? Which risks impact the cost and schedule the
most?
 What is the impact of uncertainty inputs?
 How much margin and UFE is required to achieve specific confidence levels (e.g. , to achieve 70
percent confidence level)?
 How much margin and/or UFE does each risk require for mitigation?
 Is the project carrying sufficient margin and/or UFE in appropriate places within the schedule?
 Does the ratio of time independent costs to time dependent costs seem reasonable?
 Is the project using reasonable correlations values in the JCL analysis?

14.2.3 Key attributes to a successful JCL assessment

The SRB Programmatic Team supports the SRB in determining whether the project has accurately
identified and quantified all the known risks to the success of the project. They also advise on whether
the generic uncertainty in the cost and schedule estimates is appropriate. When assessing the project’s
risk list and confidence calculation, the SRB should be able to answer the following questions30 :

 Does the analysis schedule have a logic network that has minimal constraints and is linked to
major milestones?
 Is the schedule cost-loaded? Are the fixed and variable costs within the schedule properly
identified?
 Is the project risk list properly linked to the schedule activities with likelihood and cost/schedule
impacts quantified? Does the SRB have adjustment to what risks could affect the project, where
the risks occur, and the likelihood and impact of each risk?
 Have the SRB members identified different risks than the project or ranked the risks differently?
The SRB should provide the technical rationale for the differences in risk assessment and
quantified likelihoods, consequences and expected values, and should include a BOE for the
likelihoods, consequences and expected values of the added or changed risks.
 Does the SRB’s uncertainty distribution(s) differ from the projects? If so, why?
 When the probabilistic estimating models are run using the SRB’s risk list and uncertainty
distribution, how different are the results compared to the project’s proposed budget and
schedule?
 What changes to the risk list and uncertainty distribution have the biggest impact compared to
the project’s proposed cost and schedule?
 How does the ratio of the assessor-identified risks to uncertainty differ from the projects?
 Have the risks and impact of missing a launch window been included in the project’s risk list?

30
It should be noted that answers to these questions provides a solid basis to communicate the results of the SRB
programmatic assessment to NASA decision makers.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 50 of 59
Title: SRB Programmatic Assessment Process
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 51 of 59
Title: SRB Programmatic Assessment Process

APPENDIX A: SRB PROGRAMMATIC TEAM AIDES AND PRODUCT TEMPLATES

This appendix contains links to and examples of presentations, tools, reports templates and BOE grading
criteria scorecard that can be used by the SRB Programmatic Team to support LCRs. The OCFO SID group
has set up the MAX web portal to provide a shared location for IPA templates, an archive SRB document
repository of programmatic documents created during earlier LCRs, and a current SRB repository for
common file storage of project and SRB products used and developed during a SRB review.

MAX Site Link:

https://community.max.gov/display/NASA/Standing+Review+Board+%28SRB%29+Repository
Table 8: Available Templates

Template Name
IPA Briefing
IPA Report
Review Kickoff
Checkpoint Review
SRB Risk and Uncertainty
SRB Snapshot
Knowledge Management and Capture
ToR Project LCR Programmatic Section

Example BOE Assessment Criteria

Table 9: Example BOE Assessment Criteria

Criteria Green Yellow Red

Well Documented
Cost / budget / schedule are
Existence of Existence of formal BOE that No formal BOE but data No formal BOE or data
documented with a formal
Formal BOE is “current” exists or “outdated” BOE delivered
BOE
BOE tracks to budget /
Estimate Cost & schedule estimates BOE tracks to budget / BOE does not track to
estimate within acceptable
Traceable trace from BOE to budget. estimate within 10% budget / estimate
rounding errors
Methodology & rationale
provided & information is Fully traceable–tools,
presented in traceable techniques, estimating
Partially traceable–
Basis manner containing all methodology, and Not traceable–no basis for
Methodology provided but
Traceable supporting source supporting data used to calculations provided
lacks supporting details
documentation & technical develop estimate is
data. Changes from previous provided
estimates are tracked.
Cost / budget / schedule
estimates can be replicated Estimate can be replicated Not enough information is
Part of estimate can be
Repeatable by personnel from outside within acceptable rounding provided to replicate
replicated
Agency or replacement errors estimate
personnel
Comprehensive
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 52 of 59
Title: SRB Programmatic Assessment Process

Criteria Green Yellow Red

Partially defined technical

baseline / scope, lacks
details such as quantities
Fully defined technical Undefined technical
Scope Well defined scope/baseline and specific descriptions
baseline/scope of effort baseline / scope
that include mass,
dimensions, & power
requirements as appropriate
All ground rules, All ground rules, Partial ground rules, No ground rules,
Complete assumptions, rationale, and assumptions, rationale, assumptions, rationale, assumptions, rationale,
exclusions are included exclusions provided exclusions provided exclusions provided
Credible
Significant calculation errors
Minor calculation error(s)
Calculations are totally error exist or not enough
Error Free Calculations are error free without impacting outcome
free information provided to
/ plan
evaluate
Improperly applied statistics
Improperly applied
Estimating methodology / methodology impacting
Estimating Estimating methodology / estimating methodology /
and/or statistics are applied outcome or not enough
Methodology statistics applied properly statistics without impacting
correctly information provided to
outcome
evaluate
Estimate is phased, rationale
Limited rationale for cost
Estimate phasing explained for cost phasing provided,
phasing and/or inconsistent No rationale for cost
Cost Phasing and consistent with and phasing is consistent
with integrated master phasing
integrated master schedule with integrated master
schedule
schedule

Sound/realistic/ executable– Methodology & rationale

Methodology and rationale Methodology and rationale
judgments or rationale well not justified and / or not
well justified and source are partially justified.
justified. Appropriate consistent with historical
data supports estimate. UFE Estimating methodology
Realism analogies, CERs and experience. No supporting
/ reserves are expected to and/or source data lacks
schedule estimating source data provided. UFE
cover all known and applicability. UFE / reserves
relationships applied, calculations not performed
unknown risks. appear low.
Realistic assumptions. or extremely low.

Risk list is quantitatively

Discrete risk analysis of assessed (likelihood x
Portion of risk list is
project's risk list is used to consequence, expected Risk list is not applied or
assessed and applied or
Discrete Risks inform level of UFE / value, or simulation) and linked to level of UFE /
discrete risks are not linked
contingency/ reserves / linked to level of UFE / reserves
to level of UFE / reserves
margin reserve / contingency /
margin

Uncertainty is quantitatively
Estimating uncertainty used assessed (expected value, Uncertainty is partially
Uncertainty not applied or
Estimating to inform level of UFE / simulation, S-curve) and applied to some elements
linked to level of UFE /
Uncertainty contingency / reserves / linked to level of UFE / and/or not linked to level of
reserves
margin reserve / contingency / UFE
margin
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 53 of 59
Title: SRB Programmatic Assessment Process

APPENDIX B: SRB PROGRAMMATIC TEAM PLANNING SCHEDULE

This appendix contains a typical planning schedule for Independent Programmatic Assessment.
Table 10: Example Planning Schedule

Workflow
Activity/Schedule Reference Date Description
SOPI

The Review Manager works the chair and

team acceptance with the CAs. The
approval letter is one vehicle for the official
SRB Approval Letter acceptance of the team. The ToR is the
6.0 SR Start - 125 days
Signed other. OCI/PCI clearance is determined
prior to the members’ approval. No
member can work on SRB tasks until this
letter is signed.
Complete SRB
Independent The training focuses on the role of the
6.1.1 SR Start – 120 days
Programmatic analyst and IPA expectations
Assessment Training
The SRB Review Plan is developed as a joint
effort between the Review Manager and
SRB Review Plan Signed 6.1.2 SR Start - 120 days cost & schedule analysts. It is an integrated
SRB Programmatic Team and Review
Manager plan with a common schedule.
The ToR defines the agreement between
Terms of Reference 6.1.5-6, the Cost Analysts, SRB, and project. It
SR Start - 110 days
(ToR) Signed 9.0 defines the deliverables and evaluation
criteria.
The Independent Programmatic
Assessment Plan (IPAP) defines the SRB
IPAP Developed 6.1.3 SR Start – 100 days Programmatic Team roles, programmatic
assessment plans, data drop delivery dates,
and the schedule to complete the LCR.
All available data will be collected from the
project for preliminary analysis. Anything
missing from the SRB’s initial data request
(Step 2, -115 days) shall be noted and
Data Access 6.1.4-5 SR Start - 100 days delivered as soon as the Project has the
data available. If the first set of data is not
delivered on time, it will be reported to the
SRB Chair and the SRB Programmatic Team
Lead.
The Schedule Analyst will provide the
Project feedback about the schedule health
Review Project Schedule
6.2.2.3 SR Start - 80 days check. Allow the project one week to fix
and Provide Feedback
any errors and provide an updated
schedule.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 54 of 59
Title: SRB Programmatic Assessment Process

Workflow
Activity/Schedule Reference Date Description
SOPI

This is the second data drop from the

project, which should include updated
technical, cost, and schedule data drop,
BOEs, risks, model delivery. Any new
schedule performance data and/or
Data Drop (1)–Initial JCL
6.2.1 SR Start - 60 days programmatic data should be provided. If
Model Delivery
the project is doing a JCL, the initial model
should be delivered. The project should
present their BOE to the SRB, and provide
preliminary data package supporting the
JCL model.
IPA Overview IPA overview presentation given to the SRB
6.1.7 SR Start - 60 days
Presentation to SRB at SRB kick-off meeting.
The SRB Chair gives the SRB team specific
directions and task assignments. The
schedule for future events and deliverables
is presented. The Cost Analysts, Program
Kick-Off Meeting 6.2 SR Start - (30 to 90) days
Executive, mission directorate contact, TA
and others provide the team with
background information to anchor them in
process and project status.
The SRB Chair gathers data from the
project and compares it to the review’s
entrance criteria and expected maturity.
Readiness to Proceed 6.2.3.1 SR Start - (30 to 90) days
The Chair makes an individual assessment
of the readiness of the project to enter the
review.
The Programmatic Analysts will meet with
Touch-base with Project the project cost/schedule/risk team to
Cost/Schedule/Risk 6.2.3.4 SR Start - 60 days discuss project risks, how they map to both
Team cost and schedule and how they potentially
impact both cost and schedule.
Initial telecom between all SRB members to
discuss the project risks and any SRB-
identified risks. This is the initial interaction
of the SRB to discuss the risks so that each
SRB Risk Assessment
6.2.5 SR Start - 40 days member can appropriately score each risk
Telecon
and provide the Programmatic Analysts
with cost and schedule input for the
integrated cost and schedule risk
assessment.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 55 of 59
Title: SRB Programmatic Assessment Process

Workflow
Activity/Schedule Reference Date Description
SOPI

An initial independent risk assessment by

the SRB is due to the Cost and Schedule
Analysts. The SRB needs to identify and
Initial SRB Risk score (likelihood and consequence
6.2.5 SR Start - 30 days
Assessment distribution) the risks, including SRB-
identified risks, as well as project-identified
risks. The SRB should also consider areas of
cost and schedule uncertainty.
Final data drop from the project. If the
project is doing a JCL, deliver the
Data Drop (2)–Final JCL
completed model. The project should
Model & Programmatic 6.2.3 SR Start - 20 days
present their final BOE to the SRB and
Updates Delivery
provide a complete data package
supporting the JCL model.
The SRB needs to identify and score
(likelihood and consequence distribution)
Updated SRB Risk the risks, including SRB- and Project-
6.2.6 SR Start - 20 days
Assessment identified risks. The SRB should also
consider areas of cost and schedule
uncertainty.
SRB Programmatic Team will conduct peer
review of IPA results to-date with OCFO.
Provide SRB analysts suggestions with
OCFO Checkpoint–Peer regards to the assessment (process driven,
6.2.7 SR Start - 7 days
Review not product driven) and provide OCFO
insight on current best practices, on-going
programmatic issues, and programmatic
review needs.
A status briefing of IPA charts showing
SRB Programmatic Team
6.3 SR Start - 5 days results to-date will be provided to the SRB
Status Briefing to SRB
by the SRB Programmatic Team.
Site Review Start 6.3.2 SR Start The Site Review begins.

Site Review End 6.3.2 SR End The Site Review is completed.

Obtain any new risk information from the
Present Final Analysis to
6.3.6 SR End SRB learned at the Site Review (last day of
SRB
the Site Review).
The SRB Chair briefs, typically by telecon,
Snapshot Report 6.3.3 SR End + 2 days
the Cost Analysts on the SRB results.
Incorporate any new risk information from
Finalize IPA 6.3.5 SR End + 5 days
the SRB learned at the Site Review.
Inform SRB Inform the SRB Programmatic Team Lead of
Programmatic Team 6.3.5 SR End + 7 days any changes to the IPA from the Site
Lead of Any Updates Review and SRB assessment.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 56 of 59
Title: SRB Programmatic Assessment Process

Workflow
Activity/Schedule Reference Date Description
SOPI

Final IPA Findings to SRB Final IPA findings are delivered to the SRB
6.3.5 SR End + 10 days
Chair and Chair.
IPA Report Completed 6.3.7 SR End + 30 days The IPA SRB final report is published.
Brief the designated Center on the SRB
CMC Briefing 6.3.8 Pre MDPMC
results.
Brief the sponsoring mission directorate on
MDPMC Briefing 6.3.8 Pre APMC
the SRB results.
Deliver the final and fully annotated
Final Briefing Package At least 7 days prior to
6.3.8 briefing package, with cover letter, to the
due to APMC Executive APMC
APMC Executive.
APMC Briefing 6.3.8 SR End < + 30 days Brief the Agency PMC on the SRB results.
SRB Programmatic Team document
analysis lessons-learned, issues, and
OCFO Checkpoint– successes. Goal is to inform OCFO on how
Knowledge Management 6.3.9 SR End + 30 days to prioritize programmatic improvement
and Closeout efforts. Lastly, conduct a completeness
review on final products for archival
purposes.
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 57 of 59
Title: SRB Programmatic Assessment Process

APPENDIX C: ACRONYMS

ABC Agency Baseline Commitment

APARC Advanced Programmatic Analysis and Research Capability

APL Advanced Physics Laboratory

APMC Agency Program Management Council

BOE Basis of Estimate

CA Convening Authority

CADRe Cost Analysis Data Requirement

CDR Critical Design Review

CEH Cost Estimating Handbook

CMC Center Management Council

CPM Critical Path Method

CRM Continuous Risk Management

DA Decision Authority
DPMC Division Program Management Council

EVM Earned Value Management

FAR Federal Acquisition Regulation

GSFC Goddard Space Flight Center

ICE Independent Cost Estimate

IMIR Individual Member Independent Report

IMS Integrated Master Schedule

IPA Independent Programmatic Assessment

IPAP Independent Programmatic Assessment Plan

JCL Joint Confidence Level

JPL Jet Propulsion Laboratory

KDP Key Decision Point

LaRC Langley Research Center

Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 58 of 59
Title: SRB Programmatic Assessment Process

LCC Life Cycle Cost

LCCE Life Cycle Cost Estimate

LCR LCR

MD Mission Directorate
MDPMC Mission Directorate Management Council

MDR Mission Definition Review

MEL Master Equipment List

MRR Mission Readiness Review

NASA National Aeronautics and Space Administration

NPD NASA Policy Directive

NPR NASA Procedural Requirement

OCE Office of the Chief Engineer

OCFO Office of the Chief Financial Officer

ORR Operational Readiness Review

PDR Preliminary Design Review

PE Program Executive

PEL Power Estimate List

PIR Program Implementation Review

PM Program or Project Manager

PMC Program Management Council

POC Point of Contact
PPBE Planning, Programming, Budgeting, and Execution
PP&C Program/project Planning and Control
RFA Request for Action
RIDM Risk Informed Decision Making

RMP Risk Management Plan

S&MA Safety & Mission Assurance

SBU Sensitive But Unclassified
Revision: 6.0 Document No: OCFO-SID-0002
Release Date: May 23, 2017 Page 59 of 59
Title: SRB Programmatic Assessment Process

SDR System Definition Review

SID Strategic Investments Division

SIR System Integration Review

SMH Schedule Management Handbook

SPT Standing Review Board (SRB) Programmatic Team

SRA Schedule Risk Analysis

SRB Standing Review Board

SRR System Requirements Review

STAT Schedule Test and Assessment Tool

ToR Terms of Reference

TRL Technology Readiness Level

UFE Unallocated Future Expenses

WBS Work Breakdown Structure