Software Quality Management:

Managing the quality of the software process and products

December 04th 2009

ILAYARAJA.S
Faculty Management Studies
RMD ENGINEERING COLLEGE. Ilasu_jai@yahoo.com

INDEX

Introduction to software quality

software product, software development process, product quality,
product quality attributes, product quality factors, quality of service, process quality, quality-related activities

Quality assurance and standards

Quality planning and control Software testing

Software inspections and reviews

Software measurement and metrics The role of formal methods

Conclusions

PRODUCT AND PROCESS

Business System
goals

Demand Costumer or Market Product or Service

Is a Is a

Business Process

resources

software product

software development process

What is a software product?

Software product = computer programs (sources and executables) + associated documentation Software products may be
Custom - developed for a particular customer, according to its specifications Generic (package) - developed for a general market, to be sold to a range of different customers

Types of software products

Business support software
- Includes software engineering tools in the software engineering business

Personal productivity software

- spreadsheets, word processing tools,

Embedded software....

WHAT IS A SOFTWARE DEVELOPMENT PROCESS?

Is the definition of a set of activities whose goal is the development or evolution of a software product
To be followed/instantiated in individual software development projects

Its the main business process in a software development business Generic activities in all software processes are:
Specification - what the system should do and its development constraints Development - production of the software system Validation - checking that the software is what the customer wants Evolution - changing the software in response to changing demands

New or changed requirements (problem)

Software Development Process

New or changed software product (solution)

THE IMPORTANCE OF SOFTWARE

The economies of ALL developed nations are dependent on software More and more systems are software controlled
Including an increasing number of safety-critical and mission-critical systems, with high demands on dependability

More and more businesses depend on software for their success

Software and Information Systems are critical success factors in an increasing number of businesses and organizations

Software engineering expenditure (in the development and maintenance of software products) represents a significant fraction of GNP (Gross National Product) in all developed countries

What is product quality?

Quality, simplistically, means that a product should meet its specification

The software product should deliver the required functionality (functional requirements) with the required quality attributes (non functional requirements)

This is problematical for software systems

Tension between customer quality requirements (efficiency, reliability, ...) and developer quality requirements (maintainability, reusability, ...) Some quality requirements are difficult to specify in an unambiguous way Software specifications are usually incomplete and often inconsistent
The quality compromise: we cannot wait for specifications to improve before paying attention to quality, and procedures must be put into place to improve quality in spite of imperfect specification

Quality attributes are frequently conflicting and increase development costs, so there is a need for weighting and balancing Software engineering is concerned with the cost-effective development of good software

Product quality attributes (1)

Attributes of good software (beyond delivering the required functionality): Efficiency

Software should not make wasteful use of system resources (disk and memory space, CPU time, etc.) and should present appropriate response times

Usability (ease of use)

Software must be usable by the users for which it was designed

Dependability (reliability, availability, security, safety,)

Software must be trustworthy

Maintainability (ease of maintenance)

Software must evolve to meet changing needs Software costs more to maintain than it does to develop. For systems with a long life, maintenance costs may be several times development costs

Product quality attributes (2)

Other quality attributes:

Resilience (Flexibility) Robustness Understandability Testability Adaptability Modularity Simplicity Portability Reusability Learnability

Main dimensions of dependability

Reliability - The probability of failure-free system operation over a specified time in a given environment for a given purpose Availability - The probability that a system, at a point in time, will be operational and able to deliver the requested services
Its possibly to have high availability with low reliability if failures are repaired quickly

Safety - The systems ability to operate, normally or abnormally, without danger of causing human injury or death and without damage to the systems environment Security The systems ability to protect itself from accidental or deliberate external attack

Dependability and critical systems

For critical systems, it is usually the case that the most important system property is the dependability of the system Types of critical systems:
Safetycritical system a system whose failure may result in injury, loss of life or major environment damage
- e.g. an insulin delivery system

Mission-critical system a system whose failure may result in the failure of some goal-directed activity
- e.g. a navigational system for a space aircraft

Business-critical system a system whose failure may result in the failure of the business using the system
- e.g. a customer account system in a bank

Principal product quality factors (1)

Development technology
Software development process

Process quality

Product quality

People quality

Cost, time and Budget and Schedule schedule

Principal product quality factors (2)

Process quality
A good process is usually required to produce a good product For manufactured goods, process is the principal quality determinant For design-based activity (like software development), other factors are also involved especially the capabilities of the designers For large projects with average capabilities, the development process determines product quality

People quality
For small projects, the capabilities of the developers is the main determinant Corollary: you need lower quality people (and higher quality process) in larger projects? Project Size x People Quality = Constant ?

Development technology
Is particularly significant for small projects

Budget and schedule

In all projects, if an unrealistic schedule is imposed then product quality will suffer

Process quality attributes

Process Description characteristic Understandability To what extent is the process explicitly defined and how easy is it to understand the process definition? Visibility Do the process activities culminate in clear results so that the progress of the process is externally visible? Supportability To what extent can the process activities be supported by CASE tools? Acceptability Is the defined process acceptable to and usable by the engineers responsible for producing the software product? Reliability Is the process designed in such a way that process errors are avoided or trapped before they result in product errors? Robustness Can the process continue in spite of unexpected problems? Maintainability Can the process evolve to reflect changing organisational requirements or identified process improvements? Rapidity How fast can the process of delivering a system from a given specification be completed?

Quality of service

Some product-related services and their quality attributes

User Training User Help
- Quick and useful response (avoid Help does not Help)

Product repair and new versions deployment

- Quick and effective repair - Conservation qualities: - Things that worked well in the old version, continue to work well in the new version (regression tests are very important here), and dont require new user training - Installation of the new version doesnt cause loss of user data (backward compatibility) - Installation of the new version doesnt require system down for too much time - Progress qualities: - Things that worked wrong or didnt work at all in the old version, now work well in the new version, or new useful features have been added

Not focused in this presentation (more focused on product than service)

Quality-related activities (1)

Software Verification and Validation (V & V)

Goals:
- Establish the existence of defects in a product - Assess whether or not the product is usable in an operational situation

Verification
- Ensure that we are building the product right, i.e., according to its specification

Validation
- Ensure that we are building the right product, i.e., according to user needs

V & V are integral part of the development process Concerned directly with product quality

Quality-related activities (2)

Software Quality Management (SQM)

Goals: Ensure that the required level of quality is achieved in software products, namely, that defined standards and procedures are followed SQM should aim to develop a quality culture where quality is seen as everyones responsibility Sub-activities:
- (Organizationwide) Quality assurance - (Projectwide) Quality planning
- Establish organisational procedures and standards for quality in a quality manual

- (Projectwide) Quality control (QC)

- Select applicable procedures and standards for a particular project and modify these as required. Produce a quality plan.
- Ensure that procedures and standards are followed by the software development team. Produce quality review reports

Quality management should be separate from project management to ensure independence of budget and schedule pressures Concerned directly with process quality and, indirectly, product quality

Quality management and software development

deliverables

Software development process

D1

D2

D3

D4

D5

Quality management process

Standards and procedures

Quality plan

Quality review reports

Main approaches for V&V and QC

Tests
Dynamic technique, concerned with exercising and observing product behaviour to discover defects The system is executed with test data (defined test cases) and its operational behaviour is observed to discover defects (differences between observed and expected) Used mainly for V & V GUI testing difficult to automate; API testing easier to automate

Inspections and reviews

Static technique - concerned with the analysis of the static system representation (source code, documentation, ) to discover problems May be supplement by tool-based document and code analysis

Measurements
The value of defined metrics is automatically measured on selected components of the product, for prediction or control purposes Used mainly for CQ

All involve planning, execution and result analysis and reporting

Quality assurance and standards

Standards are the key to effective quality management They may be international, national, organizational or project standards

Product standards define characteristics that all components should exhibit e.g. a common programming style
Process standards define how the software process should be enacted

Importance of standards

Encapsulation of best practice - avoids repetition of past mistakes Framework for quality assurance process it involves checking standard compliance Provide continuity - new staff can understand the organisation by understand the standards applied

Problems with standards

Not seen as relevant and up-to-date by software engineers

Practitioners should be involved in development. Engineers should understand the rationale underlying a standard Standards and their usage should be reviewed regularly. Standards can quickly become outdated and this reduces their credibility amongst practitioners

Involve too much bureaucratic form filling Unsupported by software tools so tedious manual work is involved to maintain standards
Detailed standards should have associated tool support. Excessive clerical work is the most significant complaint against standards

Documentation standards

Particularly important - documents are the tangible manifestation of the software Documentation process standards
How documents should be developed, validated and maintained

Document standards
Concerned with document identification, structure, presentation, changes highlighting, etc.

Document interchange standards

How documents are stored and interchanged between different documentation systems XML is an emerging standard for document interchange which will be widely supported in future

Development of process standards

Care must be taken not to impose inappropriate process standards

Define process

De velop product

Assess product quality

Improve process

No

Quality OK

Yes

Standar dize process

ISO 9000

International set of standards for quality management (ISO 9000:2000, ISO 9001:2000, ISO 9004:2000, etc.) Applicable to a range of organisations from manufacturing to service industries ISO 9001:2000 specifies requirements for a quality management system for any organization that needs to demonstrate its ability to consistently provide product that meets customer and applicable regulatory requirements and aims to enhance customer satisfaction, in all business sectors
Integrates previous standards ISO 9001, ISO 9002 and ISO 9003 ISO 9001 is a generic model that must be instantiated for each organisation

ISO 9004:2000 provides guidance for continual improvement of a quality management system to benefit all parties (employees, owners, suppliers, society in general,) through sustained customer satisfaction. It should be used to extend the benefits obtained from ISO 9001:2000 to all parties that are interested in or affected by the business operations.

ISO 9000 certification

Quality standards and procedures should be documented in an organisational quality manual External body may certify that an organisations quality manual conforms to ISO 9000 standards (namely ISO 9001) Customers are, increasingly, demanding that suppliers are ISO 9000 certified

ISO 9000 and quality management

ISO 9000 quality models instantiated as documents

Organization quality manual is used to develop

Organiza tion quality process instantiated as

Project 1 quality plan

Project 2 quality plan

Project 3 quality plan

Project quality mana gement

Supports

The Software Engineering Institute (SEI) Capability Maturity Model for Software (CMM)
Is a model for judging the maturity of the software processes of an organization identifying the key practices that are required to increase the maturity of these processes
Level 5 Optimizing

Le vel 4 Managed Level 3 Defined Le vel 2 Repeatable Level 1 Initial

CMM maturity levels

1) Initial. The software process is characterized as ad hoc, and occasionally even chaotic. Few processes are defined, and success depends on individual effort and heroics. 2) Repeatable. Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications. 3) Defined. The software process for both management and engineering activities is documented, standardized, and integrated into a standard software process for the organization. All projects use an approved, tailored version of the organization's standard software process for developing and maintaining software. 4) Managed. Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled. 5) Optimizing. Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.

CMM key process areas

Optimizing Process change management Technology change management Defect prevention Managed Software quality management Quantitative process management

Defined Peer reviews Intergroup coordination Software product engineering Integrated software management Training programme Organization process definition Organization process focus Repeatable Software configuration management Software quality assurance Software subcontract management Software project tracking and oversight Software project planning Requirements management

Initial

The CMM and ISO 9000

There is a clear correlation between the key processes in the CMM and the quality management processes in ISO 9000 The CMM is more detailed and prescriptive and includes a more detailed framework for improvement Organisations rated as level 2 in the CMM are likely to be ISO 9000 compliant

Index

Introduction Quality assurance and standards

Quality planning and control

Software testing Software inspections and reviews

Software measurement and metrics

The role of formal methods Conclusions

Quality planning

A quality plan sets out (within a particular project) the desired product qualities and how these are assessed and define the most significant quality attributes It should define the quality assessment process It should set out which organisational standards should be applied and, if necessary, define new standards

Quality plans should be short, succinct documents

If they are too long, no-one will read them

Quality control

Checking the software development process (within a particular project) to ensure that procedures and standards, as defined in the quality plan, are being followed Two approaches to quality control
(Manual) Quality reviews main approach (Automated) Quality measurement

Index

Introduction Quality assurance and standards Quality planning and control

Software testing
Software inspections and reviews

Software measurement and metrics

The role of formal methods Conclusions

Black-box and white-box tests

Black-box testing
An approach to testing where the program is considered as a black-box The program test cases are based on the system specification Test planning can begin early in the software process

White-box testing
Sometime called structural testing Derivation of test cases according to program structure. Knowledge of the program is used to identify additional test cases Objective is to exercise all program statements (not all path combinations)
- Test coverage measures ensure that all statements have been executed at least once

Component and integration testing

Component testing
Testing of individual program components Usually the responsibility of the component developer (except sometimes for critical systems) Tests are derived from the developers experience

Integration testing
Testing of groups of components integrated to create a system or subsystem The responsibility of an independent testing team Tests are based on a system specification (black-box)
Component testing Software developer Integration testing Independent testing team

The defect testing process

inputs and expected results

Test cases

Test data

Test results

Test reports

Design test cases

Prepare test data

Run program with test data

Compare results to test cases

Index

Introduction Quality assurance and standards Quality planning and control Software testing

Software inspections and reviews

Software measurement and metrics
The role of formal methods Conclusions

Types of review
Review type Design or inspections Principal purpose program To detect detailed errors in the design or code and to check whether standards have been followed. The review should be driven Part of software verification and validation by a checklist of possible errors. Progress reviews To provide information for management about the overall progress of the project. This is both a process and a product review Part of software and is concerned with costs, plans and project management schedules. Quality reviews To carry out a technical analysis of product components or documentation to find faults or mismatches between the specification and the design, code or documentation. It Part of software quality management may also be concerned with broader quality issues such as adherence to standards and other quality attributes.

Review results

Comments made during the review should be classified:

No action. No change to the software or documentation is required. Refer for repair. Designer or programmer should correct an identified fault. Reconsider overall design. The problem identified in the review impacts other parts of the design. Some overall judgement must be made about the most cost-effective way of solving the problem.

Requirements and specification errors may have to be referred to the client.

Index

Introduction Quality assurance and standards Quality planning and control Software testing Software inspections and reviews

Software measurement and metrics

The role of formal methods Conclusions

Software metric

A software metric is a property of a software product, process or related documentation that takes a numeric value that can be measured
Lines of code in a program, number of person-days required to develop a component, etc.

We are interested here in measuring (quantifying) the quality of a software product Main difficulty: distance between what we want to know (usually an external quality attribute) and what we are able to measure (usually an internal attribute)
higher with static metrics see next

Although some companies have introduced measurement programmes, the systematic use of measurement is still uncommon and there are few standards in this area

Types of product metrics

Dynamic metrics
Are collected by measurements made of a program in execution Are closely related to software quality attributes, such as efficiency and reliability It is relatively easy to measure the response time of a system (performance attribute) or the number of failures (reliability attribute)

Static metrics
Are collected by measurements made of the system representations (source files, documentation, etc.) Have an indirect (and difficult to establish) relationship with quality attributes, such as complexity, understandability and maintainability

Relationship between static metrics and quality attributes

Internal attribute External attribute (quality attribute) (static metric)

Number of procedur e par ameters

Maintainability Cyclomatic complexity Reliability

Portability

Program size in lines of code Number of error messages

Usability Length of user manual

Product measurement process

Choose measurements to be made Analyse anomalous components Identify anomalous measurements Measure component char acteristics

select metrics

Select components to be assessed

collected data

Measurement analysis

It is not always obvious what data means

Analysing collected data is very difficult

Data analysis must take local circumstances into account

Example of measurement surprises:

Reducing the number of faults in a program leads to an increased number of help desk calls
- The program is now thought of as more reliable and so has a wider more diverse market. The percentage of users who call the help desk may have decreased but the total may increase - A more reliable system is used in a different way from a system where users work around the faults. This leads to more help desk calls

Index

Introduction Quality assurance and standards Quality planning and control Software testing Software inspections and reviews

Software measurement and metrics

The role of formal methods

Conclusions

Formal methods

Collection of techniques based on mathematical representation and analysis of software Formal methods include
Formal specification Specification analysis and proof Transformational development Program verification

Formal specification languages

The system is specified in terms of its operations and their relationships

Algebraic

Model-based

Sequential Larch (Guttag, Horning et al., 1985; Guttag, Horning et al., 1993), OBJ (Futatsugi, Goguen et al., 1985) Z (Spivey, 1992 ) VDM (Jones, 1980) B (Wordsworth, 1996 )

Concurrent Lotos (Bolognesi and Brinksma, 1987),

CSP (Hoare, 1985) Petri Nets (Peterson, 1981)

The system is specified in terms of a state model that is constructed using mathematical constructs such as sets and sequences. Operations are defined by modifications to the systems state

Development costs with formal specification

Cost Validation Design and Implementation

Validation Design and Implementation Specification

Specification

Without formal specification

With formal specification

Index

Introduction Quality assurance and standards Quality planning and control Software testing Software inspections and reviews

Software measurement and metrics

The role of formal methods

Conclusions

Key points

Software quality management is concerned with ensuring that software meets its required standards Quality assurance procedures should be documented in an organisational quality manual Software standards are an encapsulation of best practice Reviews are the most widely used approach for assessing software quality

Documentation process

Software quality attributes

Safety Security Reliability Resilience Robustness

Understandability Testability Adaptability Modularity Complexity

Portability Usability Reusability Efficiency Learnability