21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

UNIT IV

ADVANCED TESTING CONCEPTS

Performance Testing: Load Testing, Stress Testing,
Volume Testing, Fail-Over Testing, Recovery Testing,
Configuration Testing, Compatibility Testing, Usability
Testing, Testing the Documentation, Security testing,
Testing in the Agile Environment, Testing Web and Mobile
Applications.
4.1 PERFORMANCE TESTING
Performance Testing is a type of software testing that ensures
software applications perform properly under their expected workload. It is
a testing technique carried out to determine system performance in terms of
sensitivity, reactivity, and stability under a particular workload.
Performance Testing is the process of analysing the quality and
capability of a product. It is a testing method performed to determine the
system’s performance in terms of speed, reliability, and stability under
varying workloads. Performance testing is also known as Perf Testing.

Performance Testing Attributes:

Speed: It determines whether the software product responds rapidly.
Scalability: It determines the amount of load the software product can
handle at a time.
Stability: It determines whether the software product is stable in
case of varying workloads.
Reliability: It determines whether the software product is secure or not.

Objective of Performance Testing:

The objective of performance testing is to eliminate performance
congestion.
It uncovers what needs to be improved before the product is launched in
the market.
The objective of performance testing is to make software rapid.
The objective of performance testing is to make software stable and
reliable.
The objective of performance testing is to evaluate the performance
and scalability of a system or application under various loads and
conditions. It helps identify bottlenecks, measure system
performance, and ensure that the system can handle the expected
number of users or transactions. It also helps to ensure that the
system is reliable, stable, and can handle the expected load in a
production environment.

Types of Performance Testing:

Load testing: It checks the product’s ability to perform under anticipated

user loads. The objective is to identify performance congestion before the
software product is launched in the market.

Panimalar Engineering College 1

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Stress testing: It involves testing a product under extreme workloads to see

whether it handles high traffic or not. The objective is to identify the
breaking point of a software product.

Spike testing: It is a type of load testing that tests the system’s ability to
handle sudden spikes in traffic. It helps identify any issues that may occur
when the system is suddenly hit with a high number of requests.

Soak testing: Soak testing is a type of load testing that tests the system’s
ability to handle a sustained load over a prolonged period. It helps identify
any issues that may occur after prolonged usage of the system.

Endurance testing: This type of testing is similar to soak testing, but it

focuses on the long- term behaviour of the system under a constant load.

Volume testing: In volume testing, large number of data is saved in a

database and the overall software system’s behaviour is observed. The
objective is to check the product’s performance under varying database
volumes.

4.2 LOAD TESTING

Load testing determines the behavior of the application when multiple
users use it at the same time. It is the response of the system measured
under varying load conditions.
The load testing is carried out for normal and extreme load conditions.
Load testing is a type of performance testing that simulates a real-
world load on a system or application to see how it performs under
stress.
The goal of load testing is to identify bottlenecks and determine the
maximum number of users or transactions the system can handle.
It is an important aspect of software testing as it helps ensure that the
system can handle the expected usage levels and identify any potential
issues before the system is deployed to production.
During load testing, various scenarios are simulated to test the system’s
behavior under different load conditions. This can include simulating a high
number of concurrent users, simulating numerous requests, and simulating
heavy network traffic. The system’s performance is then measured and
analyzed to identify any bottlenecks or issues that may occur.

Load Testing Techniques:

Stress testing: Testing the system’s ability to handle a high load above normal
usage levels.
Spike testing: Testing the system’s ability to handle sudden spikes in traffic.
Soak testing: Testing the system’s ability to handle a sustained load over a
prolonged period of time.
Tools for Performance Testing: Make use of specialized load testing tools
like Locust, Gatling, JMeter, LoadRunner, and Apache Benchmark. These
tools assist in gathering performance measurements and simulating a large
number of users.
Specify the Test Objectives: Clearly state what your load test’s goals are.
Recognize the required response times, transaction volumes and expected
user behavior.

Panimalar Engineering College 2

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Determine Crucial Situations: Determine the essential user scenarios that

correspond to common usage patterns. A variety of actions, including user
logins, searches, form submissions and other significant interactions,
should be covered by these scenarios.

Objectives of Load Testing:

Evaluation of Scalability: Assess the system’s ability to handle growing
user and transaction demands. Find the point at which the system begins to
function badly.
Planning for Capacity: Describe the system’s ability to accommodate
anticipated future increases in the number of users, transactions and
volume of data. Making well-informed decisions regarding infrastructure
upgrades is made easier by this.
Determine bottlenecks: Identify and localize bottlenecks in the application
or infrastructure’s performance. Finding the places where the system’s
performance can suffer under load is part of this.
Analysis of Response Time: For crucial transactions and user interactions,
track and evaluate response times. Make that the system responds to
changes in load with reasonable response times.
Finding Memory Leaks: Find and fix memory leaks that may eventually
cause a decline in performance. Make sure the programme doesn’t use up
too many resources when it’s running.

Load Testing Process:

1. Test Environment Setup: Firstly create a dedicated test
environment setup for performing the load testing. It ensures that
testing would be done in a proper way.
2. Load Test Scenario: In second step load test scenarios are created.
Then load testing transactions are determined for an application and
data is prepared for each transaction.
3. Test Scenario Execution: Load test scenarios that were created in
previous step are now executed. Different measurements and metrices
are gathered to collect the information.
4. Test Result Analysis: Results of the testing performed is analyzed
and various recommendations are made.
5. Re-test: If the test is failed then the test is performed again in order
to get the result in correct way.

Panimalar Engineering College 3

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Metrics of Load Testing:

Metrics are used in knowing the performance of load testing under
different circumstances. It tells how accurately the load testing is working
under different test cases. It is usually carried out after the preparation of
load test scripts/cases. There are many metrics to evaluate the load testing.
Some of them are listed below.

1. Average Response Time

It tells the average time taken to respond to the request generated by
the clients or customers or users. It also shows the speed of the application
depending upon the time taken to respond to the all requests generated.

2. Error Rate
The Error Rate is mentioned in terms of percentage denotes the
number of errors occurred during the requests to the total number of
requests. These errors are usually raised when the application is no longer
handling the request at the given time or for some other technical problems.
It makes the application less efficient when the error rate keeps on
increasing.

3. Throughput
This metric is used in knowing the range of bandwidth consumed during
the load scripts or tests and it is also used in knowing the amount of data
which is being used for checking the request that flows between the user
server and application main server. It is measured in kilobytes per second.

4. Requests Per Second

It tells that how many requests are being generated to the application server
per second. The requests could be anything like requesting of images,
documents, web pages, articles or any other resources.

5. Concurrent Users
This metric is used to take the count of the users who are actively present at
the particular time or at any time. It just keeps track of count those who are
visiting the application at any time without raising any request in the
application. From this, we can easily know that at which time the high
number of users are visiting the application or website.

6. Peak Response Time

Peak Response Time measures the time taken to handle the request. It also
helps in finding the duration of the peak time(longest time) at which the
request and response cycle is handled and finding that which resource is
taking longer time to respond the request.

Load Testing Tools: Apache Jmeter, WebLoad, NeoLoad, LoadNinja, HP

Performance, Tester, LoadUI Pro, LoadView.

4.3 STRESS TESTING

Stress Testing is a software testing technique that determines the
robustness of software by testing beyond the limits of normal operation.
Stress testing is particularly important for critical software but is used for all
types of software. Stress testing emphasizes robustness, availability, and
error handling under a heavy load rather than what is correct behavior

Panimalar Engineering College 4

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

under normal situations.

Stress testing is defined as a type of software testing that verifies the
stability and reliability of the system. This test particularly determines the
system on its robustness and error handling under extremely heavy load
conditions. It even tests beyond the normal operating point and analyses
how the system works under extreme conditions. Stress testing is performed
to ensure that the system would not crash under crunch situations. Stress
testing is also known as Endurance Testing or Torture Testing.

Characteristics of Stress Testing:

1. Identification of Risk: Stress testing’s main objective is to locate and
evaluate a system’s possible hazards and weaknesses.
2. Quantitative and Qualitative Analysis: While numerical data are
crucial, it’s also critical to comprehend the qualitative characteristics
of the system’s response and potential weak points.
3. Variable Parameters: Stress testing include changing variables
including interest rates, market conditions, transaction volumes and
outside influences that could have an impact on the system.
4. Cross-Functional Involvement: Many departments within an
organization must work together and participate in stress testing. This
cross-functional strategy makes sure that the stress testing procedure
benefits from a variety of viewpoints and specialties.
5. Open and Honest Communication: Stress testing necessitates open
and honest communication regarding the goal, approach, and
outcomes of the testing procedure.

Need for Stress Testing:

To accommodate the sudden surges in traffic: It is important to
perform stress testing to accommodate abnormal traffic spikes. For
example, when there is a sale announcement on the e-commerce
website there is a sudden increase in traffic. Failure to accommodate
such needs may lead to a loss of revenue and reputation.
Display error messages in stress conditions: Stress testing is
important to check whether the system is capable to display
appropriate error messages when the system is under stress
conditions.
The system works under abnormal conditions: Stress testing
checks whether the system can continue to function in abnormal
conditions.
Prepared for stress conditions: Stress testing helps to make sure
there are sufficient contingency plans in case of sudden failure due to
stress conditions. It is better to be prepared for extreme conditions by
executing stress testing.

Purpose of Stress Testing:

Analyze the behavior of the application after failure: The purpose
of stress testing is to analyze the behavior of the application after
failure and the software should display the appropriate error
messages while it is under extreme conditions.
System recovers after failure: Stress testing aims to make sure that
there are plans for recovering the system to the working state so that
the system recovers after failure.

Panimalar Engineering College 5

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Uncover Hardware issues: Stress testing helps to uncover hardware

issues and data corruption issues.
Uncover Security Weakness: Stress testing helps to uncover the
security vulnerabilities that may enter into the system during the
constant peak load and compromise the system.
Ensures data integrity: Stress testing helps to determine the
application’s data integrity throughout the extreme load, which means
that the data should be in a dependable state even after a failure.

Stress Testing Process:

The stress testing process is divided into 5 steps:

1. Planning the stress test: This step involves gathering the system
data, analyzing the system, and defining the stress test goals.
2. Create Automation Scripts: This step involves creating the stress
testing automation scripts and generating the test data for the stress
test scenarios.
3. Script Execution: This step involves running the stress test
automation scripts and storing the stress test results.
4. Result Analysis: This phase involves analyzing stress test results
and identifying the bottlenecks.
5. Tweaking and Optimization: This step involves fine-tuning the
system and optimizing the code with the goal meet the desired
benchmarks.

Types of Stress Testing:

1. Server-client Stress Testing: Server-client stress testing also known
as distributed stress testing is carried out across all clients from the
server.
2. Product Stress Testing: Product stress testing concentrates on
discovering defects related to data locking and blocking, network
issues, and performance congestion in a software product.
3. Transactional Stress Testing: Transaction stress testing is
performed on one or more transactions between two or more
applications. It is carried out for fine-tuning and optimizing the
system.
4. Systematic Stress Testing: Systematic stress testing is integrated
testing that is used to perform tests across multiple systems running
on the same server. It is used to discover defects where one
application data blocks another application.
5. Analytical Stress Testing: Analytical or exploratory stress testing is
performed to test the system with abnormal parameters or conditions
Panimalar Engineering College 6
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

that are unlikely to happen in a real scenario. It is carried out to find

defects in unusual scenarios like a large number of users logged at
the same time or a database going offline when it is accessed from a
website.
6. Application Stress Testing: Application stress testing also known as
product stress
testing is focused on identifying the performance bottleneck, and
network issues in a software product.

Stress Testing Tools:

1. Jmeter: Apache JMeter is a stress testing tool is an open-source,
pure Java-based software that is used to stress test websites. It is an
Apache project and can be used for load testing for analyzing and
measuring the performance of a variety of services.
2. LoadNinja: LoadNinja is a stress testing tool developed by SmartBear
that enables users to develop codeless load tests, substitutes load
emulators with actual browsers, and helps to achieve high speed and
efficiency with browser-based metrics.
3. WebLoad: WebLoad is a stress testing tool that combines
performance, stability, and integrity as a single process for the
verification of mobile and web applications.
4. Neoload: Neoload is a powerful performance testing tool that
simulates large numbers of users and analyzes the server’s behavior.
It is designed for both mobile and web applications. Neoload supports
API testing and integrates with different CI/ CD applications.
5. SmartMeter: SmartMeter is a user-friendly tool that helps to create
simple tests without coding. It has a graphical user interface and has
no necessary plugins. This tool automatically generates advanced test
reports with complete and detailed test results.

Metrics of Stress Testing:

Metrics are used to evaluate the performance of the stress and it is usually
carried out at the end of the stress scripts or tests. Some of the metrics are
given below.
1. Pages Per Second: Number of pages requested per second and
number of pages loaded per second.
2. Pages Retrieved: Average time is taken to retrieve all information
from a particular page.
3. Byte Retrieved: Average time is taken to retrieve the first byte of
information from the page.
4. Transaction Response Time: Average time is taken to load or
perform transactions between the applications.
5. Transactions per Second: It takes count of the number of
transactions loaded per second successfully and it also counts the
number of failures that occurred.
6. Failure of Connection: It takes count of the number of times that
the client faced connection failure in their system.
7. Failure of System Attempts: It takes count of the number of failed
attempts in the system.
8. Rounds: It takes count of the number of test or script conditions
executed by the clients successfully and it keeps track of the number
of rounds failed.

Panimalar Engineering College 7

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

4.4 VOLUME TESTING

Volume Testing is a type of software testing which is carried out to
test a software application with a certain amount of data. The amount
used in volume testing could be a database size or it could also be the size
of an interface file that is the subject of volume testing.
While testing the application with a specific database size, database is
extended to that size and after that the performance of the application is
tested. When an application needs interaction with an interface file this
could be either reading or writing the file or same from the file. A sample file
of the size required is created and then functionality of the application is
tested with that file in order to test the performance.
In volume testing a huge volume of data is acted upon the software. It
is basically performed to analyze the performance of the system by
increasing the volume of data in the database. Volume testing is performed
to study the impact on response time and behavior of the system when the
volume of data is increased in the database.
Volume Testing is also known as Flood Testing.

Characteristics of Volume Testing

Following are the characteristics of the Volume Testing:
 Performance of the software decline as passing of the time as there is
huge amount of data overtime.
 Basically the test data is created by test data generator.
 Only small amount of data is tested during development phase.
 The test data need to be logically correct.
 The test data is used to assess the performance of the system.

Objectives of Volume Testing

The objectives of volume testing is:
 To recognize the problems that may be created with large amount of data.
 To check The system’s performance by increasing the volume of data in
the database.
 To find the point at which the stability of the system reduces.
 To identify the capacity of the system or application.
Volume Testing Attributes
Following are the important attributes that are checked during the
volume testing:
 System’s Response Time: During the volume testing, the response
time of the system or the application is tested. It is also tested whether
the system responses within the finite time or not. If the response
time is large then the system is redesigned.
 Data Loss: During the volume testing, it is also tested that there is no
data loss. If there is data loss some key information might be missing.
 Data Storage: During the volume testing, it is also tested that the
data is stored correctly or not. If the data is not stored correctly then
it is restored accordingly in proper place.
 Data Overwriting: In volume testing, it is tested that whether the
data is overwritten without giving prior information to the developer. If
it so then developer is notified.

Advantages of Volume Testing

 Volume testing is helpful in saving maintenance cost that will be
spent on application maintenance.
Panimalar Engineering College 8
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

 Volume testing is also helpful in a rapid start for scalability plans.

 Volume testing also helps in early identification of bottlenecks.
 Volume testing ensures that the system is capable of real world usage.

Disadvantages of Volume Testing

 More number of skilled resources are needed to carry out this testing.
 It is sometimes difficult to prepare test cases with respect to the
number of volume of data to be tested.
 It is a time consuming technique since it requires lot of time to
decide the number of volume of data and test scenarios.
 It is a bit costly as compared to another testing technique.
 It is not possible to have the exact break down of memory used in
the real world application.

4.5 FAIL-OVER TESTING

Software products/services are tested multiple times before delivery to
ensure that it is providing the required service. Testing before delivery
doesn’t guarantee that no problem will occur in the future. Even sometimes
the software application fails due to some unwanted event due to network
issues or due to server-related problems. Failover testing aims to respond to
these types of failures.
Suppose that the PC gets off due to some technical issue, and on restarting
we open the browser, then a pop-up is shown saying Do you want to restore
all pages? On clicking restore, all tabs are restored. The process of ensuring
such restorations is known as FAILOVER TESTING.

What is Failover Testing :

Failover testing is a technique that validates if a system can allocate
extra resources and back up all the information and operations when a
system fails abruptly due to some reason. This test determines the ability of
a system to handle critical failures and handle extra servers. So, the testing
is independent of the physical hardware component of a server.
It is preferred that testing should be performed by servers. Active-
active and active- passive standby are the two most common
configurations. Both the techniques achieve failover in a very different
manner but both of them are performed to improve the server’s reliability.
For example, if we have three servers, one of them fails due to heavy
load, and then two situations occur. Either that failed server will restart on
its own or in another situation when the failed server cannot be restarted,
the remaining servers will handle the load. Such situations are tested
during this test.

Considerable Factors Before Performing Failover Testing:

1. The budget has to be the first thing to be taken into consideration
before thinking about performing the Failover test.
2. The budget is connected to the frameworks that might crash or
break down under pressure/load.
3. Always keep in mind how much time it will take to fix all the issues
caused by the failure of the system.
4. Note down the most likely failures and organize the outcomes
according to how much harm is caused by the failure.

Considerable Factors While Performing Failover Testing:

Panimalar Engineering College 9
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

1. Keep a plan of measures to be taken after performing a test.

2. Focus on the execution of the test plan.
3. Set up a benchmark so that performance requirements can be achieved.
4. Prepare a report concerning issue requirements and/or requirements of
the asset.

Working of Failover testing:

1. Consider the factors: Before performing failover testing like budget,

time, team, technology, etc.
2. Analysis on failover reasons and design solutions: Determine probable
failure situations that the system might experience. Examine the
causes of failover, including software bugs, hardware malfunctions,
network problems, etc. It provides fixes for any flaws or vulnerabilities
found in the failover procedure.
3. Testing failover scenarios: It develops extensive test cases to replicate
various failover scenarios. This covers both unplanned failovers
(system or component failures) and scheduled failovers (maintenance).
Test cases ought to address many facets of failover, such as load
balancing, user impact, network rerouting, and data synchronization.
4. Executing the test plan: To reduce the impact on production systems,
carry out the failover test plan in a controlled setting. Keep an eye on
how the system behaves during failover to make sure it satisfies the
recovery point and recovery time objectives (RPO and RTO,
respectively).
5. Detailed report on failover: Keep a record of the failover testing
findings, including any problems you ran across, how long it took to
failover and how it affected customers or services. Assess problems
according to their severity and offer suggestions for improvements.
6. Necessary actions based on the report: Distribute the report on the
failover test to all pertinent parties, such as project managers,
developers, and system administrators. Determine what needs to be
done and prioritize it based on the report’s conclusions. This might
involve fixing found flaws in the system, updating failover setups or
improving the documentation.

Benefits of Failover Testing:

1. Determines Vulnerabilities and Weaknesses: Failover testing helps in
locating holes and vulnerabilities in the system by simulating

Panimalar Engineering College 10

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

different failure scenarios. Organizations can address any problems

before they have an impact on production settings by taking a
proactive approach.
2. Verifies Redundancy Procedures: Redundancy mechanisms are
frequently incorporated into systems to improve reliability. In order to
make sure that these systems such as redundant servers, databases,
or network paths, work flawlessly in the event of a breakdown, failover
testing verifies their efficacy.
3. Improving the User Experience: Consumers anticipate consistent and
dependable service availability. By reducing interruptions and
ensuring service availability even in the face of unanticipated
circumstances, failover testing contributes to the provision of a positive
user experience.
4. Encourages Compliance: Failover testing supports organizations in
demonstrating compliance with these rules by ensuring that systems
can recover within prescribed times.
5. Encourages Continuous Improvement: Organizations that regularly
test their systems can learn from each test cycle, fix problems found
and upgrade their failover procedures to meet changing business
needs and technological advancements.

Examples of Failover Testing:

1. Banking and Financial applications
2. Telecom applications
3. Visa applications
4. Trading applications
5. Emergency service business applications
6. Government applications
7. Defense service-related applications
4.6 RECOVERY TESTING
Recovery testing is a type of system testing which aims at testing
whether a system can recover from failures or not. The technique involves
failing the system and then verifying that the system recovery is performed
properly.
To ensure that a system is fault-tolerant and can recover well from
failures, recovery testing is important to perform. A system is expected to
recover from faults and resume its work within a pre-specified time period.
Recovery testing is essential for any mission-critical system, for example, the
defense systems, medical devices, etc. In such systems, there is a strict
protocol that is imposed on how and within what time period the system
should recover from failure and how the system should behave during the
failure.
A system or software should be recovery tested for failures like:
Power supply failure
The external server is unreachable
Wireless network signal loss
Physical conditions
The external device not responding
The external device is not responding as expected, etc.

Steps to be performed before executing a Recovery Test

A tester must ensure that the following steps are performed before
carrying out the Recovery testing procedure :
Panimalar Engineering College 11
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

1. Recovery Analysis – It is important to analyze the system’s ability to

allocate extra resources like servers or additional CPUs. This would
help to better understand the recovery-related changes that can
impact the working of the system. Also, each of the possible failures,
their possible impact, their severity, and how to perform them should
be studied.
2. Test Plan preparation – Designing the test cases keeping in mind the
environment and results obtained in recovery analysis.
3. Test environment preparation – Designing the test environment
according to the recovery analysis results.
4. Maintaining Back-up – Information related to the software, like
various states of the software and database should be backed up.
Also, if the data is important, then the backing up of the data at
multiple locations is important.
5. Recovery personnel Allocation – For the recovery testing process, it
is important to allocate recovery personnel who are aware and
educated enough for the recovery testing being conducted.
6. Documentation – This step emphasizes on documenting all the steps
performed before and during the recovery testing so that the system
can be analyzed for its performance in case of a failure.

Example of Recovery Testing

When a system is receiving some data over a network for processing
purposes, we can stimulate software failure by unplugging the system
power. After a while, we can plug in the system again and test its
ability to recover and continue receiving the data from where it
stopped.
Another example could be when a browser is working on multiple
sessions, we can stimulate software failure by restarting the system.
After restarting the system, we can check if it recovers from the failure
and reloads all the sessions it was previously working on.
While downloading a movie over a Wifi network, if we move to a place
where there is no network, then the downloading process will be
interrupted. Now to check if the process recovers from the interruption
and continues working as before, we move back to a place where there
is a Wifi network. If the downloading resumes, then the software
has a good recovery rate.

Types of Recovery Testing

Database Recovery Testing: Evaluate the system’s capacity to
recover from corrupted or malfunctioning databases. In order to test
how well the system can restore the database to a consistent and
useful condition, it involves intentionally destroying or damaging it.

Load and Stress Recovery Testing: Determine how effectively the

system bounces back from variables that affect performance,
including heavy loads or stressful situations. It helps in determining
if the system is capable of handling higher loads and in the event that
it cannot, how soon it will resume normal operation after the load is
dropped.

Panimalar Engineering College 12

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Crash Recovery Testing: Determine how well the system bounces

back from a hardware or software failure. To make sure the system
can resume regular operations without losing data, it can involve
unexpected shutdowns, abrupt power failures or a sudden halt of
services.
Security Recovery Testing: Examine the system’s resilience to
security lapses, illegal access, and other security-related events by
conducting security recovery testing. It guarantees that the system
can recover from security breaches and helps discover loopholes in
the security procedures, reducing the impact of any unauthorized
access.
Data Recovery Testing: Evaluate the system’s capacity to restore
data following an unplanned disruption or failure. To make sure that
data backups, restoration procedures and recovery mechanisms are
efficient and dependable, this might involve planned data loss
scenarios.
Environment Recovery Testing: Examine the software’s ability to
adjust to changes in dependencies or configurations in the
environment. It guarantees that in the event of modifications to the
underlying structure or environmental circumstances, the system can
recover and go on operating as anticipated.

Advantages of Recovery Testing

Improves the quality of the system by eliminating the potential
flaws in the system so that the system works as expected.
Recovery testing is also referred to as Disaster Recovery Testing. A
lot of companies have disaster recovery centers to make sure that if
any of the systems is damaged or fails due to some reason, then there
is back up to recover from the failure.
Risk elimination is possible as the potential flaws are detected and
removed from the system.
Improved performance as faults are removed, and the system
becomes more reliable and performs better in case a failure occurs.

Disadvantages of Recovery testing

Recovery testing is a time-consuming process as it involves multiple
steps and preparations before and during the process.
The recovery personnel must be trained as the process of recovery
testing takes place under his supervision. So, the tester needs to be
trained to ensure that recovery testing is performed in the proper way.
For performing recovery testing, he should have enough data and back
up files to perform recovery testing.
The potential flaws or issues are unpredictable in a few cases. It is
difficult to point out the exact reason for the same, however, since the
quality of the software must be maintained, so random test cases are
created and executed to ensure such potential flaws are removed.

Panimalar Engineering College 13

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

4.7 CONFIGURATION TESTING

Configuration Testing is the type of Software Testing that verifies the
performance of the system under development against various combinations
of software and hardware to find out the best configuration under which the
system can work without any flaws or issues while matching its functional
requirements.

What is Configuration Testing?

Configuration Testing is the process of testing the system under each
configuration of the supported software and hardware. Here, the different
configurations of hardware and software mean the multiple operating system
versions, various browsers, various supported drivers, distinct memory
sizes, different hard drive types, various types of CPU, etc.

The various configurations are Win XP, Win 7 32/64 bit, Win 8 32/64 bit, Win
10, etc.
1. Database Configuration: Oracle, DB2, MySQL, MSSQL Server, Sybase
etc.
2. Browser Configuration: IE 8, IE 9, FF 16.0, Chrome, Microsoft Edge etc.

Objectives of Configuration Testing:

1. Adaptability to Different Configurations: Check that the program’s
basic features work consistently and dependably in all configurations.
Testing the behavior of the program with different setups and settings
is part of this process.
2. Evaluation of Stability: Examine the software’s stability under
various configurations. Find and fix any configuration-specific
problems that might be causing crashes, unstable systems or strange
behavior.
3. Testing the User Experience: Assess the value and consistency of the
user experience across various setups. Make that the graphical user
interface (GUI) of the software adjusts to various screen sizes,
resolutions and display settings.
4. Security Throughout Configurations: To make sure that sensitive
data is kept safe, test the software’s security features in various
setups. Determine and fix any vulnerabilities that might be
configuration-specific.
5. Compatibility of Networks: Examine the software’s behavior with
various network setups. Evaluate its compatibility with various
network types, speeds and latency.
6. Data Compatibility: Check if the programme can manage a range of
data configurations, such as those from diverse sources, databases
and file formats. Verify the consistency and integrity of the data
across various setups.

Panimalar Engineering College 14

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Configuration Testing Process:

Types of Configuration Testing:

Configuration testing is of 2 types:
1. Software Configuration Testing: Software configuration testing is
done over the Application Under Test with various operating system
versions and various browser versions etc. It is a time-consuming
testing as it takes long time to install and uninstall the various
software which are to be used for testing. When the build is
released, software configuration begins after passing through the unit
test and integration test.

2. Hardware Configuration Testing: Hardware configuration testing is

typically performed in labs where physical machines are used with
various hardware connected to them. When a build is released, the
software is installed in all the physical machines to which the
hardware is attached and the test is carried out on each and every
machine to confirm that the application is working fine. While doing
hardware configuration test, the kind of hardware to be tested is
spelled out and there are several computer hardware and peripherals
which make it next to impossible to execute all the tests.

Design the Test Cases to Run on Each Configuration:

Let consider the steps required to test each configuration. This can be as simple
as
1. Select and set up the next test configuration from the list.
2. Start the software.
3. Load in the file configtest.doc.
4. Confirm that the file is displayed correctly.
5. Print the document.
6. Confirm that there are no error messages and that the printed
document matches the standard.
7. Log any discrepancies as a bug.

Execute the Tests on Each Configuration:

You need to run the test cases and carefully log and report your
results to your team and to the hardware manufacturers if necessary. It’s
often difficult and time-consuming to identify the specific source of
configuration problems. You’ll need to work closely with the programmers
and white-box testers to isolate the cause and decide if the bugs you find are
due to your software or to the hardware.
If the bug is specific to the hardware, consult the manufacturer’s
website for information on reporting problems to them. Be sure to identify
yourself as a software tester and what company you work for. Many
Panimalar Engineering College 15
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

companies have separate staff set up to assist software companies writing

software to work with their hardware. They may ask you to send copies of
your test software, your test cases, and supporting details to help them
isolate the problem.

Rerun the Tests Until the Results Satisfy Your Team:

It’s not uncommon for configuration testing to run the entire course of
a project. Initially a few configurations might be tried, then a full test pass,
then smaller and smaller sets to confirm bug fixes. Eventually you will get to
a point where there are no known bugs or to where the bugs that still exist
are in uncommon or unlikely test configurations. At that point, you can call
your configuration testing complete.

4.8 COMPATIBILITY TESTING

Compatibility testing is software testing which comes under the
non-functional testing category, and it is performed on an application to
check its compatibility (running capability) on different
platform/environments. This testing is done only when the application
becomes stable. Means simply this compatibility test aims to check the
developed software application functionality on various software, hardware
platforms, network and browser etc. This compatibility testing is very
important in product production and implementation point of view as it is
performed to avoid future issues regarding compatibility.

Types of Compatibility Testing:

Several examples of compatibility testing are given below.

1. Software:
 Testing the compatibility of an application with an Operating System
like Linux, Mac, Windows
 Testing compatibility on Database like Oracle SQL server, MongoDB
server.
 Testing compatibility on different devices like in mobile phones,
computers.

Types based on Version Testing:

There are two types of compatibility testing based on version testing
1. Forward compatibility testing: When the behavior and compatibility
of a software or hardware is checked with its newer version then it is
called as forward compatibility testing.
2. Backward compatibility testing: When the behavior and
compatibility of a software or hardware is checked with its older
version then it is called as backward compatibility testing.

2. Hardware:
Checking compatibility with a particular size of
 RAM
 ROM
 Hard Disk
 Memory Cards
 Processor
 Graphics Card

Panimalar Engineering College 16

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

3. Smartphones:
Checking compatibility with different mobile platforms like android, iOS etc.
4. Network:
Checking compatibility with different:
 Bandwidth
 Operating speed
 Capacity

Along with this there are other types of compatibility testing are also
performed such as browser compatibility to check software compatibility
with different browsers like Google Chrome, Internet Explorer etc. device
compatibility, version of the software and others.

How to perform Compatibility testing?

Testing the application in a same environment but having different
versions. For example, to test compatibility of Facebook application in your
android mobile. First check for the compatibility with Android 9.0 and then
with Android 10.0 for the same version of Facebook App.
Testing the application in a same version but having different
environment. For example, to test compatibility of Facebook application in
your android mobile. First check for the compatibility with a Facebook
application of lower version with a Android 10.0(or your choice) and then
with a Facebook application of higher version with a same version of
Android.

Why compatibility testing is important?

1. It ensures complete customer satisfaction.
2. It provides service across multiple platforms.
3. Identifying bugs during development process.

Compatibility testing defects:

1. Variety of user interface.
2. Changes with respect to font size.
3. Alignment issues.
4. Issues related to existence of broken frames.
5. Issues related to overlapping of content.

4.9 USABILITY TESTING

Usability Testing in software testing is a type of testing, that is
done from an end user’s perspective to determine if the system is easily
usable. Usability testing is generally the practice of testing how easy a design
is to use on a group of representative users.
A very common mistake in usability testing is conducting a study too
late in the design process. If you wait until right before your product is
released, you won’t have the time or money to fix any issues – and you’ll
have wasted a lot of effort developing your product the wrong way.
This testing has a cycle wherein when:
1. the product is ready,
2. customers are asked to test it,
3. If any further changes,
4. product (software) is returned to the development team with
feedback to update the changes,
5. again the software had to run usability testing,
Panimalar Engineering College 17
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

6. if there are no more changes required,

7. the software is launched in the market.
This whole process from 1 to 5 is repeated unless the software is completely
ready and there are no further changes required. This process helps you to
meet customers’ needs and identify the problems faced by customers during
the usage of the software. Usability testing is also referred to as User
Experience.

Phases of Usability Testing:

There are five phases in usability testing which are followed by the system
when usability testing is performed. These are given below:

1. Prepare your product or design to test: The first phase of usability

testing is choosing a product and then making it ready for usability
testing. For usability testing, more functions and operations are
required than this phase provided that type of requirement. Hence,
this is one of the most significant phases in usability testing.
2. Find your participants: The second phase of usability testing is
finding an employee who is helping you with performing usability
testing. Generally, the number of participants that you need is based
on several case studies. Mostly, five participants can find almost as
many usability problems as you’d find using many more test
participants.
3. Write a test plan: This is the third phase of usability testing. The
plan is one of the first steps in each round of usability testing is to
develop a plan for the test. The main purpose of the plan is to
document what you are going to do, how you are going to conduct the
test, what metrics you are going to find, the number of participants
you are going to test, and what scenarios you will use.
4. Take on the role of the moderator: This is the fourth phase of
usability testing and here the moderator plays a vital role that involves
building a partnership with the participant. Most of the research findings
are derived by observing the participant’s actions and gathering verbal
feedback to be an effective moderator, you need to be able to make instant
decisions while simultaneously overseeing various aspects of the research
session.
5. Present your findings/ final report: This phase generally involves
combining your results into an overall score and presenting it
meaningfully to your audience. An easy method to do this is to
compare each data point to a target goal and represent this as one
single metric based on the percentage of users who achieved this goal.

Panimalar Engineering College 18

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Why we Used Usability Testing?

When software is ready, it is important to make sure that the user
experience with the product should be seamless. It should be easy to
navigate and all the functions should be working properly, the competitor’s
website will win the race. Therefore, usability testing is performed. The
objective of usability testing is to understand customers’ needs and
requirements and also how users interact with the product (software). With
the test, all the features, functions, and purposes of the software are
checked.

The primary goals of usability testing are – discovering problems

(hidden issues) and opportunities, comparing benchmarks, and
comparison against other websites. The parameters tested during
usability testing are efficiency, effectiveness, and satisfaction. It should
be performed before any new design is made. This test should be iterated
unless all the necessary changes have been made. Improving the site
consistently by performing usability testing enhances its performance which
in return makes it the best website.

Factors Affecting Cost of Usability Testing:

The testing cost will depend on the following factors:
1. No. of participants for testing.
2. Number of Days which you need for testing.
3. which type of testing.
4. the size of the team used for testing.
Remember to budget for the usability testing, making usability testing into a
product or any website is an iterative process and the elements that are needed
are as follows:

Time: Time is an important factor with considering usability testing,

it will use the specialist of usability and the team to evolve the site
and also need to test the test scenarios. Be sure to budget in time for
this test preparation as well as for running test cases, report writing,
analysing the data, and presenting the findings.
Rental cost: If you are not considering the equipment, you will need
to ensure the budget cost for all other equipment, and also need to
allot the location for the testing purpose. For example the rental room
like a conference room which is used to perform all operations.
Recruiting Costs: Consider how and where you have recruited your
participants. You will need to allow the staff to engage a recruiting
team to schedule participants based on requirements.
Participants Compensation based on: You will need to compensate
the participants for their time and travel purposes that also important
to finding the testing budget.

Techniques and Methods of Usability Testing:

There are various types of usability testing that when performed lead to
efficient software. But few of them which are the most widely used have been
discussed here.
1. Guerilla Testing
It is a type of testing where testers wander to public places and ask random
users about the prototype. Also, a thank gift is offered to the users as a
gesture of token. It is the best way to perform usability testing during the
Panimalar Engineering College 19
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

early phases of the product development process. Users mostly spare 5–10
minutes and give instant feedback on the product. Also, the cost is
comparatively low as you don’t need to hire participants. It is also known as
corridor or hallway testing.

2. Usability Lab
Usability lab testing is conducted in a lab environment where moderators
(who ask for feedback on the product) hire participants and ask them to
take a survey on the product. This test is performed on a tablet/desktop.
The participant count can be 8-10 which is a bit costlier than guerrilla
testing as you need to hire participants, arrange a place, and conduct
testing.

3. Screen or Video Recording

Screen or video recording kind of testing is in which a screen is recorded as
per the user’s action (navigation and usage of the product). This testing
describes how the user’s mind runs while using a product. This kind of
testing involves the participation of almost 10 users for 15 minutes. It helps
in describing the issues users may face while interacting with the product.

Generally, there are two studies in usability testing –

1. Moderated – the Moderator guides the participant for the changes
required in the product (software)
2. Unmoderated – There’s no moderator (no human guidance),
participants gets a set of questions on which he/she has to work.
While performing usability testing, all kinds of biases (be it friendly bias,
social bias, etc.) by the participants are avoided to have honest feedback on
the product so as to improve its durability.

Advantages and Disadvantages of Usability Testing

Usability testing is preferred to evaluate a product or service by testing it
with the proper users. In Usability testing, the development and design
teams will use to identify issues before coding and the result will be earlier
issues will be solved. During a Usability test, you can,
 Learn if participants will be able to complete the specific task completely.
 identify how long it will take to complete the specific task.
 Gives excellent features and functionalities to the product
 Improves user satisfaction and fulfils requirements based on user’s
feedback
 The product becomes more efficient and effective
The biggest cons of usability testing are the cost and time. The more
usability testing is performed, the more cost and time is being used.

4.10 TESTING THE DOCUMENTATION

As a software tester, you typically aren’t constrained to just testing
the software. Your responsibility will likely cover all the parts that make up
the entire software product. Assuring that the documentation is correct is
your job, too.

4.10.1 Type of Software Documentation:

Here’s a list of software components that can be classified as
documentation. Obviously, not all software will have all the components,
but it’s possible:
Panimalar Engineering College 20
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

• Packaging text and graphics. This includes the box, carton, wrapping,
and so on. The documentation might contain screen shots from the
software, lists of features, system requirements, and copyright information.
• Marketing material, ads, and other inserts. These are all the pieces of
paper you usually throw away, but they are important tools used to promote
the sale of related software, add-on content, service contracts, and so on.
The information for them must be correct for a customer to take them
seriously.
Warranty/registration. This is the card that the customer fills out and sends
in to register the software. It can also be part of the software, being displayed
onscreen for the user to read, acknowledge, and complete online.

• EULA. Pronounced “you-la,” it stands for End User License Agreement.

This is the legal document that the customer agrees to that says, among
other things, that he won’t copy the software nor sue the manufacturer if
he’s harmed by a bug. The EULA is sometimes printed on the envelope
containing the media—the floppy or CD. It also may pop up onscreen during
the software’s installation.
• Labels and stickers. These may appear on the media, on the box, or on the
printed material. There may also be serial number stickers and labels that
seal the EULA envelope. Figure 4.10 shows an example of a disk label and all
the information that needs to be checked.

Figure 4.10 There’s lots of documentation on this disk label for the software
tester to check.

Panimalar Engineering College 21

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

• Installation and setup instructions. Sometimes this information is

printed directly on the discs, but it also can be included on the CD sleeve or
as a CD jewel box insert. If it’s complex software, there could be an entire
installation manual.
• User’s manual. The usefulness and flexibility of online manuals has made
printed manuals much less common than they once were. Most software
now comes with a small, concise “getting started”–type manual with the
detailed information moved to online format. The online manuals can be
distributed on the software’s media, on a website, or a combination of both.
• Online help. Online help often gets intertwined with the user’s manual,
sometimes even replacing it. Online help is indexed and searchable, making
it much easier for users to find the information they’re looking for. Many
online help systems allow natural language queries so users can type Tell
me how to copy text from one program to another and receive an
appropriate response.
• Tutorials, wizards, and CBT (Computer Based Training). These tools blend
programming code and written documentation. They’re often a mixture of both
content and high-level, macro-like programming and are often tied in with the
online help system. A user can ask a question and the software then guides
him through the steps to complete the task.

Panimalar Engineering College 22

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

• Samples, examples, and templates. An example of these would be a word

processor with forms or samples that a user can simply fill in to quickly
create professional-looking results. A compiler could have snippets of code
that demonstrate how to use certain aspects of the language.
• Error messages.

4.10.2 What to Look for When Reviewing Documentation:

Table 4.1 is a simple checklist to use as a basis for building your
documentation test cases.
Table 4.1 A Documentation Testing Checklist
4.10.3 The Importance of Documentation Testing:
Good software documentation contributes to the product’s overall quality in
three ways:
• It improves usability.
• It improves reliability.
• It lowers support costs. The reason is that users who are confused or run
into unexpected problems will call the company for help, which is expensive.
Good documentation can prevent these calls by adequately explaining and
leading users through difficult areas.

4.11 SECURITY TESTING

Security testing is an integral part of software testing, which is used
to discover the weaknesses, risks, or threats in the software application and
also help us to stop the nasty attack from the outsiders and make sure the
security of our software applications.
The primary objective of security testing is to find all the potential
ambiguities and vulnerabilities of the application so that the software does
not stop working. If we perform security testing, then it helps us to identify
all the possible security threats and also help the programmer to fix those
errors.
It is a testing procedure, which is used to define that the data will be
safe and also continue the working process of the software.

4.11.1 Principle of Security testing:

Availability: In this, the data must be retained by an official person, and

they also guarantee that the data and statement services will be ready to
use whenever we need it.
Integrity: In this, we will secure those data which have been changed by the
unofficial person. The primary objective of integrity is to permit the receiver

Panimalar Engineering College 23

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

to control the data that is given by the system.

The integrity systems regularly use some of the similar fundamental
approaches as confidentiality structures. Still, they generally include the
data for the communication to create the source of an algorithmic check
rather than encrypting all of the communication. And also verify that correct
data is conveyed from one application to another.
Authorization: It is the process of defining that a client is permitted to
perform an action and also receive the services. The example of

authorization is Access control.

Confidentiality: It is a security process that protracts the leak of the data
from the outsider's because it is the only way where we can make sure the
security of our data.
Authentication: The authentication process comprises confirming the
individuality of a person, tracing the source of a product that is necessary to
allow access to the private information or the system.

Non-repudiation: It is used as a reference to the digital security, and it a

way of assurance that the sender of a message cannot disagree with
having sent the message and that the recipient cannot repudiate having
received the message.
The non-repudiation is used to ensure that a conveyed message has
been sent and received by the person who claims to have sent and received
the message.

4.11.2 Key Areas in Security Testing:

While performing the security testing on the web application, we need to
concentrate on the following areas to test the application:

System software security: In this, we will evaluate the vulnerabilities of the

application based on different software such as Operating system, Database
Panimalar Engineering College 24
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

system, etc.
Network security: In this, we will check the weakness of the network
structure, such as policies and resources.
Server-side application security: We will do the server-side application
security to ensure that the server encryption and its tools are sufficient to
protect the software from any disturbance.
Client-side application security: In this, we will make sure that any
intruders cannot operate on any browser or any tool which is used by
customers.

4.11.3 Types of Security testing:

As per Open Source Security Testing techniques, we have different
types of security testing which as follows:

Security Scanning: Security scanning can be done for both automation

testing and manual testing. This scanning will be used to find the
vulnerability or unwanted file modification in a web-based application,
websites, network, or the file system. After that, it will deliver the results
which help us to decrease those threats. Security scanning is needed for
those systems, which depends on the structure they use.

Risk Assessment: To moderate the risk of an application, we will go for risk

assessment. In this, we will explore the security risk, which can be detected
in the association. The risk can be further divided into three parts, and
those are high, medium, and low. The primary purpose of the risk
assessment process is to assess the vulnerabilities and control the
significant threat.

Vulnerability Scanning: It is an application that is used to determine and

generates a list of all the systems which contain the desktops, servers,
laptops, virtual machines, printers, switches, and firewalls related to a
network. The vulnerability scanning can be performed over the automated
application and also identifies those software and systems which have
acknowledged the security vulnerabilities.

Penetration testing: Penetration testing is a security implementation where

a cyber-security professional tries to identify and exploit the weakness in
the computer system. The primary objective of this testing is to simulate
outbreaks and also finds the loophole in the system and similarly save from
the intruders who can take the benefits.

Panimalar Engineering College 25

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Security Auditing: Security auditing is a structured method for evaluating

the security measures of the organization. In this, we will do the inside
review of the application and the control system for the security faults.

Ethical hacking: Ethical hacking is used to discover the weakness in the

system and also helps the organization to fix those security loopholes before
the nasty hacker exposes them. The ethical hacking will help us to increase
the security position of the association because sometimes the ethical
hackers use the same tricks, tools, and techniques that nasty hackers will
use, but with the approval of the official person.
The objective of ethical hacking is to enhance security and to protect
the systems from malicious users' attacks.

Posture Assessment: It is a combination of ethical hacking, risk

assessments, and security scanning, which helps us to display the complete
security posture of an organization.

4.11.4 Security testing tools:

We have various security testing tools available in the market, which
are as follows: SonarQube, ZAP, Netsparker,
Arachni, IronWASP

4.12 TESTING IN THE AGILE ENVIRONMENT

Agile Testing is a type of software testing that follows the principles of
agile software development to test the software application.
All members of the project team along with the special experts and
testers are involved in agile testing.
Agile testing is not a separate phase and it is carried out with all the
development phases
i.e. requirements, design and coding, and test case generation. Agile
testing takes place simultaneously throughout the Development Life
Cycle.
Agile testers participate in the entire development life cycle along
with development team members and the testers help in building the
software according to the customer
requirements and with better design and thus code becomes possible.
Agile Testing has shorter time frames called iterations or loops. This
methodology is also called the delivery-driven approach because it
provides a better prediction on the workable products in less duration
time.

4.12.1 Features of Agile Testing

Some of the key features of agile software testing are:
Simplistic approach: In agile testing, testers perform only the necessary
tests but at the same time do not leave behind any essential tests. This
approach delivers a product that is simple and provides value.
Continuous improvement: In agile testing, agile testers depend mainly on
feedback and self- learning for improvement and they perform their activities
efficiently continuously.
Self-organized: Agile testers are highly efficient and tend to solve problems
by bringing teams together to resolve them.
Testers enjoy work: In agile testing, testers enjoy their work and thus will
be able to deliver a product with the greatest value to the consumer.
Panimalar Engineering College 26
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Encourage Constant communication: In agile testing, efficient

communication channels are set up with all the stakeholders of the project
to reduce errors and miscommunications.
Constant feedback: Agile testers need to constantly provide feedback to the
developers if necessary.

4.12.2 Agile Testing Principles

Shortening feedback iteration: In Agile Testing, the testing team gets to
know the product development and its quality for each and every iteration.
Thus continuous feedback minimizes the feedback response time and the
fixing cost is also reduced.
Testing is performed alongside: Agile testing is not a different phase. It is
performed alongside the development phase. It ensures that the features
implemented during that iteration are actually done. Testing is not kept
pending for a later phase.
Involvement of all members: Agile testing involves each and every member
of the development team and the testing team. It includes various
developers and experts.
Documentation is weightless: In place of global test documentation, agile
testers use reusable checklists to suggest tests and focus on the essence of
the test rather than the incidental details. Lightweight documentation tools
are used.
Clean code: The defects that are detected are fixed within the same
iteration. This ensures clean code at any stage of development.
Constant response: Agile testing helps to deliver responses or feedback on
an ongoing basis. Thus, the product can meet the business needs.
Customer satisfaction: In agile testing, customers are exposed to the
product throughout the development process. Throughout the development
process, the customer can modify the requirements, and update the
requirements and the tests can also be changed as per the changed
requirements.
Test-driven: In agile testing, the testing needs to be conducted alongside
the development process to shorten the development time. But testing is
implemented after the implementation or when the software is developed in
the traditional process.

4.12.3 Agile Testing Life Cycle

The agile testing life cycle has 5 different phases:
1. Impact Assessment: This is the first phase of the agile testing life cycle
also known as the feedback phase where the inputs and responses are
collected from the users and stakeholders. This phase supports the test
engineers to set the objective for the next phase in the cycle.
2. Agile Testing Planning: In this phase, the developers, customers, test
engineers, and stakeholders team up to plan the testing process schedules,
regular meetings, and deliverables.
3. Release Readiness: This is the third phase in the agile testing lifecycle
where the test

Panimalar Engineering College 27

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

engineers review the features which have been created entirely and test if the
features are ready to go live or not and the features that need to be sent
again to the previous development phase.
4. Daily Scrums: This phase involves the daily morning meetings to check
on testing and determine the objectives for the day. The goals are set daily
to enable test engineers to understand the status of testing.
5. Test Agility Review: This is the last phase of the agile testing lifecycle that
includes weekly meetings with the stakeholders to evaluate and assess the
progress against the goals.

4.12.3 Agile Testing Strategies:

1. Iteration 0
It is the first stage of the testing process and the initial setup is performed in
this stage. The testing environment is set in this iteration.
 This stage involves executing the preliminary setup tasks such as
finding people for testing, preparing the usability testing lab,
preparing resources, etc.
 The business case for the project, boundary situations, and project scope
are verified.
 Important requirements and use cases are summarized.
 Initial project and cost valuation are planned.
 Risks are identified.
 Outline one or more candidate designs for the project.

2. Construction Iteration
It is the second phase of the testing process. It is the major phase of the
testing and most of the work is performed in this phase. It is a set of
iterations to build an increment of the solution. This process is divided into
Panimalar Engineering College 28
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

two types of testing:

Confirmatory testing: This type of testing concentrates on verifying that

the system meets the stakeholder’s requirements as described to the team
to date and is performed by the team. It is further divided into 2 types of
testing:
Agile acceptance testing: It is the combination of acceptance testing
and functional testing. It can be executed by the development team
and the stakeholders.
Developer testing: It is the combination of unit testing and
integration testing and verifies both the application code and
database schema.
Investigative testing: Investigative testing detects the problems that are
skipped or ignored during confirmatory testing. In this type of testing, the
tester determines the potential problems in the form of defect stories. It
focuses on issues like integration testing, load testing, security testing, and
stress testing.

3. Release End Game

This phase is also known as the transition phase. This phase includes the
full system testing and the acceptance testing. To finish the testing stage,
the product is tested more relentlessly while it is in construction iterations.
In this phase, testers work on the defect stories. This phase involves activities
like:
 Training end-users.
 Support people and operational people.
 Marketing of the product release.
 Back-up and restoration.
 Finalization of the system and user documentation.

4. Production
It is the last phase of agile testing. The product is finalized in this stage after
the removal of all defects and issues raised.

4.12.4 Agile Testing Quadrants

The whole agile testing process is divided into four quadrants:

Panimalar Engineering College 29

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

1. Quadrant 1 (Automated)
The first agile quadrat focuses on the internal quality of code which contains
the test cases and test components that are executed by the test engineers.
All test cases are technology-driven and used for automation testing. All
through the agile first quadrant of testing, the following testing can be
executed:
 Unit testing.
 Component testing.

2. Quadrant 2 (Manual and Automated)

The second agile quadrant focuses on the customer requirements that are
provided to the testing team before and throughout the testing process. The
test cases in this quadrant are business-driven and are used for manual
and automated functional testing. The following testing will be executed in
this quadrant:
 Pair testing.
 Testing scenarios and workflow.
 Testing user stories and experiences like prototypes.

3. Quadrant 3 (Manual)
The third agile quadrant provides feedback to the first and the second
quadrant. This quadrant involves executing many iterations of testing, these
reviews and responses are then used to strengthen the code. The test cases
in this quadrant are developed to implement automation testing. The testing
that can be carried out in this quadrant are:
 Usability testing.
 Collaborative testing.
 User acceptance testing.
 Pair testing with customers.

4. Quadrant 4 (Tools)
The fourth agile quadrant focuses on the non-functional requirements of the
product like performance, security, stability, etc. Various types of testing are
performed in this quadrant to deliver non-functional qualities and the
expected value. The testing activities that can be performed in this quadrant
are:
 Non-functional testing such as stress testing, load testing, performance
testing, etc.
 Security testing.
 Scalability testing.
 Infrastructure testing.
 Data migration testing.

4.12.5 Challenges During Agile Testing:

Below are some of the challenges that are faced during agile testing:
Changing requirements: Sometimes during product development changes
in the requirements or the specifications occur but when they occur near
the end of the sprint, the changes are moved to the next sprint and thus
become the overhead for developers and testers. Inadequate test coverage:
In agile testing, testers sometimes miss critical test cases because of the
continuously changing requirements and continuous integration. This

Panimalar Engineering College 30

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

problem can be solved by keeping track of test coverage by analyzing the

agile test metrics.
Tester’s availability: Sometimes the testers don’t have adequate skills to
perform API and Integration testing, which results in missing important test
cases. One solution to this problem is to provide training for the testers so
that they can carry out essential tests effectively.
Less Documentation: In agile testing, there is less or no documentation
which makes the task of the QA team more tedious.
Performance Bottlenecks: Sometimes developer builds products without
understanding the end-user requirements and following only the
specification requirements, resulting in performance issues in the product.
Using load testing tools performance bottlenecks can be identified and fixed.
Early detection of defects: In agile testing, defects are detected at the
production stage or at the testing stage, which makes it very difficult to fix
them.
Skipping essential tests: In agile testing, sometimes agile testers due to
time constraints and the complexity of the test cases put some of the non-
functional tests on hold. This may cause some bugs later that may be
difficult to fix.

4.13 TESTING WEB AND MOBILE APPLICATIONS

4.13.1 Introduction:
Web Applications
Web applications refer to computer programs that run in a web browser.
Commonly built with the help of HTML5, CSS and JavaScript, web
applications offer more interactivity than websites and can be accessed via a
desktop or laptop. The classic examples of web applications include webmail,
online stores and web banking.

Mobile Applications
A mobile application is a program that was built to be used on mobile
devices (smartphones, tablets and various wearables). Mobile apps are not
as straightforward as desktop web apps and fall into three varieties: mobile
web, native and hybrid apps.
Mobile web applications
A mobile web application is a program that can be accessed via a mobile
browser, meaning that you don’t have to download them to your device to
start using them. Like web apps, mobile web applications are usually built
using JavaScript, CSS and HTML5; however, there is no standard software
kit. Contrary to other mobile applications, web apps for mobile use are easier
to build and test, but they’re usually much more primitive in terms of
functionality.
Native applications
Fast and more advanced when it comes to features, native mobile
applications run on the device itself, so you have to download them before
using them. Since they are platform- specific, native mobile apps are built
using specific languages and integrated development environments (IDEs).
For example, Android native applications are developed using Java or Kotlin
and Android Studio or Eclipse IDE. At the same time, to build an app for an
Apple device, you’ll need to use Objective-C or Swift and the XCode IDE.
Native apps are secure, integrate with the hardware perfectly and have the
best UI/UX experience.
Hybrid applications
Panimalar Engineering College 31
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Hybrid apps combine the characteristics of native and mobile web apps.
Built with the help of the “standard web” stack (JavaScript, CSS and
HTML5), they are then wrapped in a native environment, so you can use the
same code for different platforms. While running on your mobile browser,
hybrid applications are downloadable and have access to your camera, GPS,
contact list, etc. Though such applications are easier to build and maintain,
they are slower and offer less advanced functionality than their native
counterparts.

4.13.2 Types of Mobile App Testing vs. Web App Testing

Whether it comes to testing web or mobile applications, the aim is to ensure
that an app is user-friendly and functions properly under different
circumstances, even if it’s already in production. Furthermore, both
application testing varieties include common types:
 Functional testing
 User-interface testing
 Usability testing
 Configuration and Compatibility testing
 Security testing
 Performance testing
 Database testing

1. Functional Testing
Functional testing involves checking of the specified functionality of a
web application. Functional test cases for web applications may be generated
using boundary value analysis, equivalence class testing, decision table
testing and many other techniques.
Example:
Let us consider the eCommerce application sells products such as
computers, mobile phones, cameras, electronics, etc. The home page of this
web application is given in Figure 4.13.

Figure 4.13 Homepage of online shopping web application

Panimalar Engineering College 32

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Table 4.13 presents some sample functional test cases of the order
process form of an online shopping website.

Table 4.13. Sample functional test cases of order process of an

online shopping web application
2. User-interface Testing
User interface testing tests that the user interaction features work
correctly. These features include hyperlinks, tables, forms, frames and user
interface items such as text fields, radio buttons, check boxes, list boxes,
combo boxes, command buttons and dialog boxes.
User interface testing ensures that the application handles mouse
and keyboard events correctly and displays hyperlinks, tables, frames,
buttons, menus, dialog boxes, error message boxes, and toolbars properly.

2.1. Navigation Testing

Navigation testing investigates the proper functioning of all the internal and
external links. Navigation testing must ensure that websites provide
consistent, well-organized links and should also provide alternative
navigation schemes such as search options and site maps. The placement of
navigation links on each page must be checked. Search based navigation
facility must also be thoroughly tested and search items should be
consistent across one page to another. All the combinations of keywords
and search criteria must be verified in navigation testing. Table 4.14
presents test cases for navigation testing for an online shopping website as
given in Figure 4.13.
Table 4.14 Navigation testing test cases for online shopping
website
Manual checking of hyperlinks can be very time consuming. There are
various online tools available for checking broken links, accuracy and
availability of links and obtaining advice on search engines. Some tools for
navigation testing include Performance Technologies’ TestLink, W3C’s Link
checker, Xenu’s LinkSleuth, Dead Links’ Dead Links, LinkTiger’s LinkTiger,
Viable Software Alternative’s LinkRunner, Elsop’s LinkScan, REl Software’s
Panimalar Engineering College 33
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Link Validator, UCI’s MQMspider and Illumit’s WebLight.

2.2 Form Based Testing

Websites that include forms need to ensure that all the fields in the form are
working properly. Form-based testing involves the following issues:
1. Proper navigation from one field of the form to another using the tab key.
2. Ensures that the data entered in the form is in a valid format.
3. Checks that all the mandatory fields are entered in the form.
Consider the registration form of an online shopping website (web page is
given in Figure 4.13). Its inputs include login id, password, repeat password,
name, last name, email address, phone number, shipping details (address,
city, state, zip code) and billing details (address, city, state, zip code). Its form

based test cases are shown in Table 4.15.

Table 4.15 Test cases of registration form of an online shopping web
application

3. Usability Testing
Usability testing refers to the procedure employed to evaluate the degree to
which the software satisfies the specified usability criteria.
4. Configuration and Compatibility Testing
One of the significant challenges of web testing is that it must ensure
the proper functioning of a web application on all the supported platforms
and suitable environments.
Configuration testing determines the behaviour of the software with
respect to various configurations whereas compatibility testing determines
whether the web application behaves as expected with respect to various
supported configurations.
5. Security Testing
Security is the procedure used to protect information from various
threats. It is very important to protect sensitive and critical information and
data while communicating over the network. The user wants implementation
of a safeguard to protect personal, sensitive and financial information. We
want data to be accurate, reliable and protected against unauthorized access.
Panimalar Engineering College 34
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Security involves various threats such as unauthorized users,

malicious users, message sent to an unintended user, etc. The primary
requirement of security includes:
i. Authentication: Is the information sent from an authenticated
user?
ii. Access Control: Is data protected from unauthorized users?
iii. Integrity: Does the user receive exactly what is sent?
iv. Delivery: Is the information delivered to the intended user?
v. Reliability: What is the frequency of a failure? How much time
does the network take to recover from a failure? What measures
are taken to counter catastrophic failure?
vi. Non-repudiation: Is the receiver able to prove that the data
received came from a specific sender?
6. Performance Testing
The goal of performance testing is to evaluate the application’s
performance with respect to real world scenarios. The following issues must
be addressed during performance testing:
i. Performance of the system during peak hours (response time,
reliability and availability).
ii. Points at which the system performance degrades or system fails.
iii. Impact of the degraded performance on the customer loyalty, sales and
profits.
6.1 Load Testing
Load testing involves testing the web application under real world
scenarios by simulating numerous users accessing the web application
simultaneously. It tests the web application by providing it maximum load.
6.2 Stress Testing
Stress testing involves execution of a web application with more than
maximum and varying loads for long periods. Unlike performance and load
testing, stress testing evaluates the response of the system when the system
is given a load beyond its specified limits. The behaviour of the system is
monitored to determine when the system under stress test fails and how
does it recover from the failure.

7. Database Testing
In web applications, many applications are database driven, for
example, e-commerce related websites or business-to-business applications.
It is important for these applications to work properly and provide security
to the user’s sensitive data such as personal details and credit card
information. Testing data-centric web applications is important to ensure
their error-free operation and increased customer satisfaction.
For example, consider the example for purchasing items from an
online store. If the user performs a search based on some keywords and
price preferences, a database query is created by the database server.
Suppose due to some programming fault in the query, the query does not
consider the price preferences given by the customer, this will produce

Panimalar Engineering College 35

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

erroneous results. These kinds of faults must be tested and removed during
database testing. Table 4.16 shows sample test cases based on a user
operation in an online shopping website.

4.13.3 Difference between Mobile App and Web App Testing:

The divergence between mobile app testing and web app testing is
more discernible in terms of user interaction, Internet connection and
compatibility.
User Interaction
From the viewpoint of interaction with users, most desktop browser-based
applications are limited to a mouse and a keyboard, and all the operations
can be performed by clicking or hitting a particular key. Mobile apps, on the
other hand, offer a wider scope of options in this respect. While all the
tapping, swiping, pulling and pinching is not as challenging to test, there are
voice assistants and move commanders to be included in your testing
checklist.

Internet Connection
The majority of web applications don’t function offline, and the same rule
applies to mobile web applications. To this end, when dealing with this type
of apps, you should test if they perform well under different speeds of the
Internet connection. When it comes to native and some hybrid mobile apps,
it is critical to verify if an app works properly in the offline mode, how it
reacts to interrupted connection and how it functions with 3/4/5 G or WI-
FI.

Compatibility
Web applications should be tested against different browsers, which also
applies to their mobile web and hybrid counterparts. From the device
compatibility viewpoint, web-based applications are usually more simple to
test due to the functionality of a desktop. With mobile apps, the testing
procedure is much more complicated because of the wide variety of mobile
devices, which expand their functionality at an exponential rate. That is
why, when checking mobile applications for compatibility, it is critical to
pay attention to the technical characteristics of each mobile device and how
they affect the behavior of your app. The most important of these
characteristics are:
Capacity limitations
Note that RAM and SSD are absolutely different for web and mobile. While
some mobile devices can ship 3-4GB of RAM, the capacity of even the best
smartphone falls behind that of an average desktop. To this effect, if your
app isn’t properly optimized for mobile use, it will function much slower
than its desktop fellow. Furthermore, since mobile apps are updated more
frequently than their web counterparts, it is critical to test how your app
affects memory and whether it resumes proper work after a number of
updates and installations.
Battery life
Desktop computers don’t have problems like this, but a smartphone battery
that’s running low has become an actual nightmare for thousands of people.
Even when you’re using a laptop, having a dead computer is not as bad as
having a dead mobile phone. An application that drains battery life will be
deleted soon. So, it is paramount for your QA team to check the
performance of a mobile application on a fully-charged and a low-charged

Panimalar Engineering College 36

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

device.
Screen size
While desktops are limited to a few screen dimensions, mobile devices offer
much greater variety in this respect. It is important to examine specific
mobile screen dimensions and make sure that all the app fields are visible
and easy to access.

4.13.4 Different Tools for Mobile vs. Web App Testing

The reasons for choosing the right tool for test automation are rather
obvious: a higher level of test coverage and parallel testing, better reliability
and faster test execution.

Tools for Web App Testing

To be able to work without a glitch under continuous, high load and
growing market
expectations, web apps need to undergo a plethora of tests to ensure
compliance with the UI standards, compatibility and usability.

Certainly, there’re a lot more commercial and open-source tools, but

Selenium is still the most popular open-source tool for automated testing
against web apps and websites.

Selenium
Selenium is a powerful open-source automated testing framework consists
of Selenium IDE, Selenium WebDriver and Selenium Grid. Selenium
supports multiple programming languages for script creation, allows users to
record and re-run saved scripts and works well on different browsers and
operating systems. While it’s not particularly self-sufficient, Selenium can
be easily integrated with other tools for test management, image-based
testing, performing load tests and efficient functioning in an agile working
environment.

To execute Selenium WebDriver scripts, you need to ensure that you have
the initial configuration done. The WebDriver consists of three separate
parts, take Chrome as an example, the browser itself (“i.e. Chrome“), the
language bindings provided (“the Driver”) and an executable ChromeDriver
downloaded from the Chromium project which acts as a bridge between
“chrome” and the “driver”. All of them is discussed here.

Tools for Mobile App Testing

While the tooling world for web app testing looks quite simple, as Selenium
is widely used by web developers and testers, the tools used for mobile app
testing is not as straightforward.

First of all, not all web application QA tools are applicable to mobile app testing.
Secondly, when choosing a tool or framework for your mobile app, consider if
it is cross- platform or designed for a particular OS.
Finally, not every mobile app testing tool is best for all application types:
native, web mobile and hybrid.
But don’t let the number of tools and frameworks scare you away from
mobile app test automation. We’ve gathered the best of them right here.

Appium
Panimalar Engineering College 37
 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

If you need a black-box mobile app testing tool, look no further than
Appium. Based on Selenium, it is an open-source tool for testing hybrid, web
and native Android and iOS mobile applications. Offering the Selenium set of
features, Appium runs tests on real devices, emulators and simulators.

It allows you to create test cases on a variety of programming languages,

including Java, C#, Python, PHP, etc., using the same API. Still, you’ll need
to create separate iOS and Android scripts to run UI tests. Like Selenium,
Appium is a flexible tool and can be easily integrated with other
frameworks. Learn all tips and tricks about Appium here.

Espresso
Espresso is a UI quality assurance framework designed by Google for white
box testing. Since it was created to test Android native applications,
Espresso tests can be written in Java and Kotlin, the programming
languages used to develop Android applications.

Due to the simple API, it easily adapts to your needs without the complexity
of multi-threaded testing. But the most outstanding feature of Espresso is
that it provides automatic synchronization between tests and applications
(the test doesn’t continue until the application is ready).
XCUITest / XCTest
For iOS functional and UI testing, XCUITest and XCTest are getting their
popularity among iOS developers. The reason being, they run tests inside
the XCode IDE and testers can write tests in either Objective-C and Swift.

XCUITest and XCTest can execute iOS tests much faster than other cross-
platform frameworks. Their tests are usually more reliable and trustable.
However, it is because they are native to iOS development and testing that
keeps away many mobile developers that are more looking into the flexibility
of the used tools.

Calabash
Although the team behind it has stopped developing it, Calabash was once
an open-source automated QA tool for running functional and acceptance
tests of native and hybrid applications.

Users can run simultaneous tests against multiple devices in parallel with
Calabash, and it provides an API for testing native applications on touch
screen devices. The only minor setback is that the test scripts have to be
written in Ruby only.

Selenium Appium Espresso XCUITes Calabash

t XCTest
Platfor Deskto Both Android Android iOS Both
m type p & iOS Android &
browse iOS
rs
App Web Native, web Native, Native, Native
type and hybrid web and web and and
hybrid hybrid hybrid

Panimalar Engineering College 38

 21IT1904 UNIT-IV SOFTWARE TESTING AND AUTOMATION

Area Function Functional, UI UI Functional,

s to al, regression, UI
test regressio UI acceptance
n
Scripti Java, C#, Java, C#, Java, Kotlin Swift, Ruby
ng Perl, Python, PHP, Objective-
languag Python, Ruby, C
e JavaScript JavaScript
, Ruby,
PHP
CI Popular Popular CI Popular Popular Popular
CI tools tools (e.g., CI tools CI tools CI tools
(e.g., Jenkins, (e.g., (e.g., (e.g.,
Jenkins, Teamcity) Jenkins, Jenkins, Jenkins,
Teamcity Teamcity Teamcity Teamcity
) ) ) )
Licen Open- Open-source Open-source Open- Open-source
se source source
type

Panimalar Engineering College 39