Software Verification and Validation (VIMMD052)

Verification of the
Requirements Specification
Istvan Majzik
majzik@mit.bme.hu

Budapest University of Technology and Economics

Dept. of Artificial Intelligence and Systems Engineering

Budapest University of Technology and Economics

Dept. of Artificial Intelligence and Systems Engineering 1
 Recap: Software specification lifecycle phase

Requirement
Task V&V criteria V&V technique
analysis Defining functional - Completeness - Reviews
Software
and non-functional - Consistency - Static analysis
specification requirements - Verifiability - Simulation
Architecture - Feasibility
design

Component
design
Analysis
Component Reality
implementation
Design Implementation
System space
integration Modeling 3 Implementation
- structuring
System Designing space
delivery - abstraction
- decomposition
Operation,
maintenance

3
 Overview
 Inputs and outputs of the phase
 Preparing the specification
o Formal languages
o Semi-formal and structured methods
o Example: SysML
 Verification tasks
o General aspects and verification techniques
o Verifying completeness and consistency
 Managing requirements
o Traceability
o Basic tasks and tool support
4
Inputs and outputs of the phase

Inputs and outputs

Related: Software Quality Assurance Plan
and Software Verification Plan

5
 Inputs and outputs of the phase

System requirements Software requirements

specification specification

System architecture Specifying software Software requirements

design requirements verification report

Software quality “Local” Software requirements

assurance plan verification test specification

Specification
for validation
testing

6
 Software Quality Assurance Plan
 Goals:
o Planning how to prevent systematic faults and control residual faults
o Planning the required technical and control activities
 Main aspects: QA during development
o Activities, their input and output criteria in the lifecycle  Verification Plan
o Quantitative quality expectations (e.g., ISO/IEC 9126)
o Specification of its own review and maintenance
 Methods for checking external suppliers
o Compliance of the QA Plan of the supplier
o Verification of external software components
 Issue tracking
o Documentation and feedback mechanisms
o Analysis of issues (root causes)
o Diagnosis and maintenance/repair activities and techniques
o Verification and validation of corrections
o Fault avoidance

7
 Software Verification Plan
 Often a separate plan (especially in safety-critical systems)
 Planning the verification activities
o Planning the techniques and measures (using the development standard)
o Determining acceptance criteria
 Overall aspects of verification:
o “Local” checking of the development steps: Completeness, consistency
o Conformance checking: W.r.t. the output of previous phases, specification
 Details:
o Participants roles and responsibilities
o Tools (e.g., test equipment)
o Evaluation of verification results (acceptance criteria)
• Checking the required test coverage
• Evaluation of quality requirements

8
Preparing the specification

Formal languages
Semi-formal and structured methods
Example: SysML

9
 Software requirements specification - Terminology
 Requirement
o Incoming need, vision, expectation
• From the future users
• From stakeholders (management, operator, authority, ...)
o Basis for validation
 Requirement specification
o Requirements in “converted” form, for the designers
• Result of requirement analysis
• Abstraction, structuring, filtering applied
o Several types of requirements
• Property specification, behavior specification, …
• Later: architecture design, component design, …
o Basis for verification
10
 Approaches for specifying requirements
 Contents of the requirements specification
o Functional requirements + Extra-functional requirements
 Typical: Natural language based specifications
o Problems with unambiguity, verifiability
 Possible solutions:
o Using strict specification language (e.g., formal, or semi-formal)
o Using verified “specification patterns” (e.g., for safe behavior)
o Systematic verification after the requirement specification phase
 Example: Solutions proposed by EN 50128
o Formal methods (VDM, Z, B, TL, PN, ...)
o Semi-formal methods (diagram based techniques, SysML)
o Structured methods (JSD, SADT, SSADM, …)
o Natural language based description (explanation) is mandatory

11
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages (ADT, OBJ, …)
 Process description languages (CSP, CCS, …)
 Logic languages (HOL, CTL*, …)
 Constructive languages (NUPRL, …)
 Hybrid or wide spectrum languages
(CPN, E-LOTOS, …)

12
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages Mathematical (ADT,
model:OBJ, …)
• Elements in the system: set-theoretic
 Process description languages structures(CSP,
like sets,CCS, …)relations
subsets,
• Functions, operations, events: with
 Logic languages (HOL, CTL*,
pre- and post-conditions, …)
invariants

 Example:
Constructive languages (NUPRL, …)
Specification of an access control system (in Event-B):

 Persons:
Hybrid
Buildings:
or
prswide 0, pspectrum
bld  0, b  bld
 prs (set) languages
(set)
Authorization: aut  prs  bld (binary relation) (CPN, E-LOTOS, …)
Situation: sit  prs  bld (complete function)
Invariant: sit  aut (relation to be checked)
An abstract event (change of situation):
pass = ANY p,b WHERE (p,b)aut  sit(p)b
THEN sit(p):=b END

13
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages (ADT, OBJ, …)
Abstract
Process description languages
data types: sorts (set of values), (CSP,
AbstractCCS,
algebra…)and
category theory
operations, properties as equations
Logic languages (HOL,
• AbstractCTL*, …)values,
data types:
Type Boolean is operations, properties
 Constructive
sorts Bool
opns
languages (NUPRL, …)
• First order logic is typical

 Hybrid false, true : -> Bool

not :orBool
wide spectrum languages
-> Bool
and : Bool, Bool -> Bool
eqns (CPN, E-LOTOS, …)
forall x, y: Bool
ofsort Bool
not(true) = false;
not(false) = true;
x and true = x;

14
 Overview of the types of formal languages
• Processes: Execution of statements
 Model-oriented languages (VDM,
(typically send Z,interactions)
/ receive B, …)
• Operations among the processes
 Algebraic languages (ADT,communication)
(synchronization, OBJ, …)
 Process description languages (CSP, CCS, …)
 Logic languages (HOL,
Example: Process algebra language (CCS) CTL*,
events …)
and operators
 Constructive
Sender languages
= msg.ack.Sender (NUPRL, …)
Receiver = msg.ack.Receiver
 Hybrid Chan
or wide spectrum languages
= msgin.msgout.Chan + ackin.ackout.Chan
System = Sender[msgin/msg,ackout/ack] | Chan |
(CPN, E-LOTOS, …)
Receiver[msgout/msg, ackin/ack]
Sender Receiver
msg ack msg ack

msgin ackout Chan msgout ackin

15
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages (ADT, OBJ, …)
 Process description languages (CSP, CCS, …)
 Logic languages (HOL, CTL*, …)
 Constructive languages (NUPRL,
• Formal mathematical logic …)
(first order or higher order logic)
 Hybrid or wide spectrum languages
• Temporal logics
(with temporal(CPN,
operatorsE-LOTOS,
like “next …)
time”, “eventually”, “until”, “before”)

16
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages Constructive logic (ADT, OBJ,
systems: Proof…)
of a
property of a function at the same time
 Process description languages (CSP, CCS,
provides a construction …)
(implementation)
 computable functions
 Logic languages (HOL, CTL*, …)
 Constructive languages (NUPRL, …)
 HybridExample
or wide spectrum languages
for a non-constructive proof (in mathematics)
• The existence of an artifact with a given property can be proven
(CPN, E-LOTOS, …)
without giving exactly what is that artifact
• Example: Proof that there exist a,b  Q such that ab  Q
• Properties with non-constructive proof are not feasible for
software specification, this way restrictions are needed that
guarantee the synthesis of functions
17
 Overview of the types of formal languages
 Model-oriented languages (VDM, Z, B, …)
 Algebraic languages (ADT, OBJ, …)
 Process description languages (CSP, CCS, …)
 Logic languages (HOL, CTL*, …)
 Constructive languages (NUPRL, …)
 Hybrid or wide spectrum languages
(CPN, E-LOTOS, …)
• Properties and advantages of different
formalisms are combined, e.g.,
• LOTOS: process description + algebraic lang.
• CPN: Petri-nets + data manipulation (in ML)

18
 Semi-formal languages: Examples
 Description of the structure:
o (Functional) block diagrams
 Description of data flow:
o Data flow diagrams, data flow networks
o (Message) sequence diagrams
 Description of the control flow:
o Control flow diagram, state machine, statechart
 Description of logic conditions:
o Truth tables
o Constraint languages (e.g., OCL with structure)
19
 Structured methodologies: Historical examples
 Jackson System Development (JSD)
o Entity structure: Entities + actions (ordering) + processes
o Network: Communicating sequential processes
 Real-time Yourdon (Ward-Mellor)
o Basic: Environment (input events) + behavior (response)
o Construction: Processes (+ processors)
 SSADM
o Data model (entity relationship diagram)
o Data flow diagram (processes, data storage)
o Entity diagram (life history)
o Entity effects
 Structured Analysis and Design Technique (SADT)
o Activity-factor diagram: tasks + relations;
input, control, resource, output
 ROOM: Real-Time Object-Oriented Modeling

20
 Semi-formal requirements specification: SysML
 Systems Modeling Language 1.0
o UML subset and extensions for system modeling
o Novelties: Requirement and Parametric diagram

21
 Requirement diagram
 Requirements (textual) with identifier are model elements
o <<requirement>> stereotype
o Id (identifier) and text (description) fields
o User-specified attributes: e.g., type, source, risk, ...
o Tabular form can be used to represent requirements
 Requirements can be grouped into hierarchic packages
o Functional, performance, etc. categories
 Refinement among requirements (~ subclass), composition
 Relations can be used (e.g., inserted as structured comments):
o Copy: between requirements (master – slave)
o Trace: between requirements (client – supplier)
o DeriveReqt: between requirements (source – derived)
o Refine: between requirements and design elements
o Satisfy: between requirements and design or implementation elements
o Verify: between requirements and test elements

22
Example requirements diagram: Structure

23
Example requirements diagram: Relations

25
 Block diagram
 Block: Element of the structure (black / white box)
o Component (not only software)
o In SysML: Based on UML 2.0 classes
 Block definition diagram: Types of blocks
 Internal block diagram: Concrete roles of block types

26
 Parametric diagram
 Goal: Verifiable quantitative requirements
(constraints) expressed using attributes
o Extra-functional requirements
o Supporting analysis (e.g., performance, reliability)
 ConstraintBlock: Specifying interrelations
o Formal (e.g., MathML, OCL), or informal (textual)
o Adapted to analysis tool (not SysML specific)
 Parametric diagram: Concrete application
o Application of Constraint blocks in a given context
o Binding between values
27
 Parametric diagram: Example

Concrete interrelations and

substitution of parameters in
constraint blocks (equations)

28
Illustration of the relations among diagrams

29
 Verification tasks

General aspects and verification techniques

Verifying completeness and consistency

30
31
To be verified: Generic criteria for a good specification
 Complete
o Specified functions, references, tools, …
 Consistent
o Internal and external consistency
o Traceability
 Verifiable
o Specific
o Unambiguous
o Quantifiable (if possible)
 Feasible
o Resources
o Usability
o Maintainability
o Risks: budget, technical, environmental

32
 Example: Good specification on the basis of IEEE 830-1998
Correct
• Every requirement stated therein is one that the software shall meet
• Consistent with external sources (e.g. standards)
Unambiguous
• Every requirement has only one interpretation
• Formal or semi-formal specification languages can help
Complete
• For every (valid, invalid) input there is specified behavior
• TBD only possible resolution
Consistent
• No internal contradiction, well-defined terminology

Ranked for importance and/or stability

• Necessity of requirements

Verifiable
• Can be checked whether the requirement is met

Modifiable
• Not redundant, structured

Traceable
• Source is clear, effect can be referenced

33
Example: Good specification on the basis of IEEE 29148-2011
Necessary
• If it is removed or deleted, a deficiency will exist, which cannot be fulfilled by other capabilities

Implementation-free
• Avoids placing unnecessary constraints on the design

Unambiguous
• It can be interpreted in only one way; is simple and easy to understand

Consistent
• Is free of conflicts with other requirements

Complete
• Needs no further amplification (measurable and sufficiently describes the capability)

Singular
• Includes only one requirement with no use of conjunctions

Feasible
• Technically achievable, fits within system constraints (cost, schedule, regulatory…)

Traceable
• Upwards traceable to the stakeholder statements; downwards traceable to other documents

Verifiable
• Has the means to prove that the system satisfies the specified requirement

34
 Verifying completeness and consistency
Incompleteness or inconsistency: major source of failures
 Statistics of faults found during the system testing of Voyager and
Galileo spacecraft:
78% (149/192) faults resulting from specification problem
o 23%: missing state transitions (stuck in dangerous state)
o 16%: missing time constraints for data validity
o 12%: missing reaction to external event
o 10%: missing assertions to check input values
 60-70% of IT project failures can be traced back to insufficient
requirements – Meta Group (2003)
 “Significantly more defects were found per page at the earlier
phases of the software life cycle.”
o Inspection of 203 documents
o An analysis of defect densities found during software inspections (JSS, DOI:
10.1016/0164-1212(92)90089-3)

35
 Techniques for verification
 Static analysis
o Checking documents, models or other artifacts
o Without execution
 Basis for static analysis: Checklists
o Examples: Generic criteria for good specification;
criteria for reactive controllers (see details later)
o Completeness of the checklist is always questionable
 Implementation of static analysis
o Manual review (all aspects)
o Tool-support (esp. for checking consistency)

36
 Manual review: Terminology and steps
Types of review: Steps of a review:
 Informal review 1. Planning
o Defining review criteria
o No formal process
o Allocating roles
o Peer or technical lead reviewing
2. Kick-off
 Walkthrough o Distributing documents
o Meeting led by author o Explaining objectives
o May be quite informal 3. Individual preparation
o Reviewing artifacts
 Technical review
o Collecting defects, questions
o Review meeting with experts
4. Review meeting
o Pre-meeting preparations for o Discussing and logging results
reviewers o Making decisions
 Inspection 5. Rework
o Formal (well-documented) process o Fixing defects
o Led by a trained moderator o Recording updated status
6. Follow-up
o Checking fixes
o Checking exit criteria

37
 Tool support for verification of the specification
 Natural languages
o No precise syntax and semantics
o Static analysis by manual review
 Semi-formal languages
o Precise syntax, but informal semantics
o Automated checking of syntax and well-formedness
(missing or contradictory elements)
 Formal languages
o Mathematically precise syntax and semantics
o Automated checking of syntax / well-formedness
o Automated checking of behavior
• Reachable states of computation
(e.g., model checking, equivalence/refinement checking)
• Properties of computation
(e.g., theorem proving for invariants, post-conditions)

38
 Tool support: Checking state machines
Yakindu Statechart Tools IAR visualSTATE

39
Managing requirements

Traceability
Basic tasks and tool support

40
The role of traceability and its relation to verification
 Traceability of requirements: Managing links among requirements
and design artifacts
o Among various levels of requirements: User -> System -> Component
o Among requirements and design / test artifacts:
Req. specification -> Architecture design -> Component design ->
-> Source code -> Test -> Test result
 Analysis possibilities based on traceability links
o Impact analysis: Handling the changes
• What is affected by a changed requirement?
o Derivation analysis: Handling utility and rewards
• Why is this artifact here? What is the related requirement?
o Coverage analysis: Handling the status of development
• What requirements are refined / implemented / tested?

41
 Typical tasks of requirement management tools
Storing the requirements: Hierarchic grouping
Handling the lifecycle and changes Using versions, attributes, timestamps,
of requirements: showing timeline of changes
Storing the relations: Several types: Composition, derivation,
refinement, implementation, ..
Support traceability: Requirements – Design (models) –
– Source code – Test – Test results
Navigation on relations: Forward: e.g., impact analysis;
Backward: e.g., derivation analysis
Generation of coverage lists: Identify uncovered requirements or
extra functionality
Handling authorization: Defining roles and allowed activities
Sending notifications: Messages in case of changes
Assuring integrity: Detecting unintentional changes

42
 Requirement management tools

https://www.youtube.com/
watch?v=qYK7_g4Fy44

https://www.youtube.com/
watch?v=YC_NrseqWcc
43
 Example: IBM Rational DOORS

Attributes

Req.
object Textual
identifier object
Change
mark Header
object

Hierarchy

44
 Example: IBM Rational DOORS

User level requirement Indicator for incoming relations

The Instructor shall be able to Lower level requirements for satisfying

take control of any Student PC. the user level requirement

satisfies The system shall provide a facility for the Instructor

station to monitor a student PC

satisfies The system shall, when a student PC is being

monitored, provide a facility for the Instructor
station to take control of the selected student PC

satisfies The system shall disable student PC input when

control is taken by the instructor station

45
 Requirement based verification tool-chains
 Assigning verification activities to requirements
o Checking satisfaction of the requirement, collecting evidences
o Standard-based techniques and measures (e.g., for safety case)
 Verification tool-chains (typically external)
o Analysis: Generating analysis model, performing analysis, post-
processing or visualization of results
o Testing: (Model based) test case generation, test execution, providing
test verdict
o Measuring: Configuring measurements, executing measurements,
data analysis
 Verification tool-chains can be started from the requirement
management tool
o Scripts with triggers (verifiable requirement)
 Registering the status of verification
o Successfully verified requirement + repository of evidences
46
Example: Starting verification tool chain from DOORS

Tool-chain
workflow
(V&V tool spec.)

Tool-chain Data &

manager Documents
Triggered from DOORS Repository

V&V-
V&V-
Tools
Tool

Example tools:
– ITEM (Hazard and risk analysis)
– RACER (Formal verification)
– SCADE MTC (Simulation)
– LDRA (Testing)
– PROPANE (Fault injection)
– EMI Test Bench

47
 Summary
 Inputs and outputs of the phase
 Preparing the requirements specification
o Formal languages
o Semi-formal and structured methods
 Verification tasks
o General aspects and verification techniques
o Verifying completeness and consistency
 Managing requirements
o Traceability
o Basic tasks and tool support
48
Example: Checking the specification of
reactive controllers
Completeness
Consistency

49
 Review criteria for reactive controllers
Groups of criteria (developed by N. Leveson, Safeware)
 State definition
 Inputs (events)
 Outputs
 Outputs and triggers
 Transitions
 Human-machine interface

Controller
Controlled
Operator
systems

50
 Example: Review criteria for reactive controllers
 State definition • Initial state is safe
• In case of missing input
 Inputs (events) there is a timeout,
 Outputs and no action is allowed

 Outputs and triggers

 Transitions
 Human-machine interface

Controller
Controlled
Operator
systems

51
 Example: Review criteria for reactive controllers
 State definition • For every input in every
state there is a specified
 Inputs (events) behavior
• Reactions are unambiguous
 Outputs (deterministic)
 Outputs and triggers • Input is checked (value,
timeliness)
 Transitions • Handling of invalid inputs is
 Human-machine interface specified
• Rate of interrupts is limited

Controller
Controlled
Operator
systems

52
 Example: Review criteria for reactive controllers
 State definition • Credibility checks are
specified
 Inputs (events) • There is no unused output
 Outputs • Processing capability of the
environment is respected
 Outputs and triggers
 Transitions
 Human-machine interface

Controller
Controlled
Operator
systems

53
 Example: Review criteria for reactive controllers
 State definition
 Inputs (events)
 Outputs • Effect of outputs is checked
through the inputs
 Outputs and triggers • Control loop is stable
 Transitions
 Human-machine interface

Controller
Controlled
Operator
systems

54
 Example: Review criteria for reactive controllers
 State definition • Every state is reachable
 Inputs (events) statically (incoming path)
• Transitions are reversible
 Outputs (there is a way back)
• More than one transitions
 Outputs and triggers from dangerous to safe states
 Transitions • Confirmed transitions from
safe to dangerous states
 Human-machine interface

Controller
Controlled
Operator
systems

55
 Example: Review criteria for reactive controllers
 State definition
• Priority of events to the
 Inputs (events) operator is defined
• Update rate is defined
 Outputs • Processing capability of the
 Outputs and triggers operator is respected

 Transitions
 Human-machine interface

Controller
Controlled
Operator
systems

56