Analysis of Software Artifacts

Departamento de Engenharia Informática, FCTUC

Analysis of Software Artifacts (ASA)

Henrique Madeira,
Departamento de Engenharia Informática
Faculdade de Ciências e Tecnologia da Universidade de Coimbra
2022/2023

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 1

Fundamental concepts
of software quality
and software dependability

(this is just a teaser)

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 2

Henrique Madeira, 2022/2023 1

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Concepts and terminology on dependability and

quality in action…

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 3

Dependability: an integrative concept

• Dependability: ”delivery of service that can justifiably be trusted, thus

avoidance of failures that are unacceptably frequent or severe” (J.-C. Laprie)

Security
Attributes

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 4

Henrique Madeira, 2022/2023 2

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Dependability: an integrative concept

• Dependability: ”delivery of service that can justifiably be trusted, thus

avoidance of failures that are unacceptably frequent or severe” (J.-C. Laprie)

Key non-functional attributes

of software (and systems)

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 5

Dependability: an integrative concept

• Dependability: ”delivery of service that can justifiably be trusted, thus

avoidance of failures that are unacceptably frequent or severe” (J.-C. Laprie)

The problems that may

damage dependability

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 6

Henrique Madeira, 2022/2023 3

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Dependability: an integrative concept

• Dependability: ”delivery of service that can justifiably be trusted, thus

avoidance of failures that are unacceptably frequent or severe” (J.-C. Laprie)

Different means/techniques to solve

or mitigate the effect of the threats

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 7

Dependability: an integrative concept

• Dependability: ”delivery of service that can justifiably be trusted, thus

avoidance of failures that are unacceptably frequent or severe” (J.-C. Laprie)

Causality chain
Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 8

Henrique Madeira, 2022/2023 4

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Threats: causality view

System point of view

Fault may cause Error may cause Failure

• Failure: Incorrect component and/or system response

• Error: Erroneous change in the state of the system
• Fault: Root cause

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 9

Threats: causality view

Software point of view

Error may cause Fault may cause Failure

(human) (defect/bug)

• Failure: External behaviour is incorrect

• Fault: Discrepancy in code that causes a failure.
• Error: Human mistake that caused fault

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 10

10

Henrique Madeira, 2022/2023 5

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Two views of software systems

• Functional view
– What the software system does
– Quality is related to the match between the functionalities and the user
needs/expectations

• Non-Functional view
– How the software system does it (features such as performance, security,
reliability, availability, usability, maintainability, and many, many, more)
– Typically known as Quality Attributes of a software system
– Most of them cannot be measured directly
– The biggest technical challenges are in these non-functional attributes

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 11

11

Functional and non-functional requirements

In software engineering the functional vs non-functional views starts with

the requirements elicitation (i.e., at the very beginning of the process)

• Functional requirements
– Describes what a software system should do
– Function points is a usual metric to characterize and assess the size of the
software

• Non-functional requirements
– Define constraints (or goals) on how the system will do so
– Include basically everything that is not related to the functional aspects of the
software system

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 12

12

Henrique Madeira, 2022/2023 6

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Robustness
(more on concepts & terminology)

• Robustness: “a software system can be said to be robust if it retains its

ability to deliver service in conditions which are beyond its normal domain
of operation” (Laprie)

• Robustness is used very often to test software interfaces such as system

calls, APIs, web services, etc. This is called robustness testing:
– In this context, robustness is defined as “the degree to which a system or
component can function correctly in the presence of invalid inputs
[IEEE90]”
– Experimental studies (Phil Koopman) show that approximately 15% of the OS
system calls (Linux, Unix, Windows) crashes when called with invalid input
parameters.

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 13

13

Resilience
(more on concepts & terminology)

• Resilience ≈ dependability + robustness

Resilience: the persistence of service delivery that can justifiably be

trusted, when facing changes (Laprie)

• Resilience considers changes in lato senso. That is, changes include all sort
of upsets:
– Hardware and software faults
– Malicious attacks
– Configuration changes
– Software and hardware upgrades
– Etc…
Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 14

14

Henrique Madeira, 2022/2023 7

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Dependability (and Resilience)

Attributes, Means, and Threats

• Hardware faults
• Software faults
• Environment faults
• Human faults
• …
Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 15

15

What is a software fault?

Residual(?) software faults (bugs), originated from defects in design or
implementation of software components and its integration in a system, that escape
testing and other fault avoidance methods

Software development process (in theory...)

Requirements
Specification
Design
Code development
Test
Deployment

Correctness from the

end user point of view

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 16

16

Henrique Madeira, 2022/2023 8

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

What is a software fault?

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 17

17

Different types of software faults

• In complex systems, the failures caused by software bugs may appear in
different way, defining a very first big types of software faults (bugs):
• Bohrbugs
• Bugs that cause failures deterministically
• Easiest to find during testing
• Fault tolerance à design diversity and redundancy
• Mandelbugs
• Re-execution after a failure caused by a Mandelbug will generally not cause another
failure
• Very difficult to find and correct during testing
• Fault tolerance à simple retries and recovery-oriented computing using checkpointing
• Aging-related
• Bugs tend to be activated and cause failures after long periods of system run-time
• Difficult to find during testing (but static code analysis is effective for some of them)
• Fault tolerance à software rejuvenation

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 18

18

Henrique Madeira, 2022/2023 9

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Software faults: a persistent problem

• Software reliability is mainly based on fault avoidance using good

software engineering methodologies

• In real systems (i.e., not toys) à fault avoidance not successful à

Fault-tolerance is needed, unless the impact of failures is
acceptable.

• Rule of thumb for fault density in software (Rome labs, USA)

– 10-50 faults per 1,000 lines of code à for good software
– 1-5 faults per 1,000 lines of code à for critical applications using highly
mature software development methods and having intensive testing

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 19

19

Software faults: a persistent problem

• Software reliability is mainly based on fault avoidance using good

software engineering methodologies

• In real systems (i.e., not toys) à fault avoidance not successful à

Fault-tolerance is needed, unless the impact of failures is
acceptable.

• Rule of thumb for fault density in software (Rome labs, USA)

– 10-50 faults per 1,000 lines of code à for good software
– 1-5 faults per 1,000 lines of code à for critical applications using highly
mature software development methods and having intensive testing

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 20

20

Henrique Madeira, 2022/2023 10

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Software faults: a persistent problem

• Software reliability is mainly based on fault avoidance using good

software engineering methodologies
• SW development methodologies
• In real systems (i.e., not toys) à fault avoidance not successful à
• Static analysis tools
Fault-tolerance is needed, unless the impact of failures is
• Software inspections
acceptable.
• Model checking
• Rule of thumb for• fault density
Testing, in software
testing, testing (Rome labs, USA)
– 10-50 faults per 1,000 lines of code à for good software
• Verification and validation
– 1-5 faults per 1,000 lines of code à for critical applications using highly
• …
mature software development methods and having intensive testing

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 21

21

Classification of faults
• Caused by what?
– Physical faults
– Human-Made faults
• Why?
– Accidental faults
– Intentional non malicious faults / Intentional malicious faults
• When?
– Development faults: design, coding, configuration, upgrading
– Operational faults: in use or maintenance (operation faults, interaction faults,
configuration faults,..)
• Where (with respect to the system)?
– Internal faults
– External faults
• How long?
– Permanent faults
– Transient faults
Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 22

22

Henrique Madeira, 2022/2023 11

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Classification of faults (more detailed view)

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 23

23

Dependability (and Resilience)

Attributes, Means, and Threats

Components/systems may fail

arbitrarily
Failures such as clean crashes
(i.e., stop sending outputs)
are relatively rare

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 24

24

Henrique Madeira, 2022/2023 12

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Failures classification

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 25

25

Dependability (and Resilience)

Attributes, Means, and Threats

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 26

26

Henrique Madeira, 2022/2023 13

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Dependability means
• Fault Prevention techniques: prevent the occurrence ofTwo
faults
different
– Improve development process to avoid/minimize faults
– Use selected technologies (better components, certified softwareperspectives
tools, etc. ) with
strong technical
• Fault Tolerance techniques: to provide correct serviceimplications
in presence of faults
– Triple modular redundancy, N-Version programming, check pointing and recovery, etc.

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 27

27

Dependability means
• Fault Prevention techniques: prevent the occurrence of faults
– Improve development process to avoid/minimize faults
– Use selected technologies (better components, certified software tools, etc. )

• Fault Tolerance techniques: to provide correct service in presence of faults

– Triple modular redundancy, N-Version programming, check pointing and recovery, etc.

• Fault Removal techniques: specific techniques to reduce the presence of

faults (number, seriousness, ...)
– Development: regression and non-regression testing, static and dynamic verification, etc.
– Operation: preventive maintenance such as patches, updates, SW rejuvenation, etc.

• Fault Forecasting techniques: to estimate the present number, the future

incidence, and the consequences of faults
– Probabilistic assessment, modeling, operational evaluation,…

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 28

28

Henrique Madeira, 2022/2023 14

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Dependability means diagram (Laprie)

Error masking

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 29

29

Dependability means diagram (Laprie)

V model

Error masking

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 30

30

Henrique Madeira, 2022/2023 15

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Fault tolerant techniques diagram

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 31

31

Fault tolerance techniques

Fault Error Failure

Estimated using fault

forecasting techniques

Fault tolerant mechanisms

Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 32

32

Henrique Madeira, 2022/2023 16

 Analysis of Software Artifacts
Departamento de Engenharia Informática, FCTUC

Fundamental concepts
of software quality
and software dependability

For a more complete view:

• Detailed slides provided
• Recommended papers
• Other (reliable) sources available in the Internet

1
Henrique M adeira Analysis of Software Artifacts, DEI-FCTUC, 2022/2023 33

33

Henrique Madeira, 2022/2023 17