Last Updated:April 16, 2020

Sigla Asignatura IEY4478 Nombre de la Asignatura Calidad de software Tiempo 3h

Experiencia de Aprendizaje N° 1 Clasificación de requerimientos
Actividad N° 1.3 Aplicando pruebas NO funcionales

Nombre del Recurso Didáctico 1.3.2 Manual Complete Non-Functional Testing Guide

A Complete Non-Functional Testing Guide For Beginners

A Complete Guide to Non-Functional Testing: Its Purpose, Types, Tool, Test Cases with Examples
What is Non-Functional Testing?

Non-functional testing is done to verify the non-functional requirement of the application like Performance,
Usability, etc.

It verifies if the behavior of the system is as per the requirement or not. It covers all the aspects which are not
covered in functional testing. In our day to day testing, a lot of attention is given to functional testing and
functional requirements.

The clients are also interested in fulfilling the functional requirements which are directly related to the
functionality of an application. But in the actual phase i.e. when you’re functionally tested, the software comes
into the market and is used by the real end-users, and there are chances for it to face some issues related to the
performance.

These issues are not related to the functionality of the system, but they can affect the user experience in a
negative manner. Hence it is important for the software or application to be tested for Non-Functional
requirements as well in order to avoid negative customer experience.

Testing is broadly classified into two types:

 Functional Testing
 Non-Functional Testing

Importance

This testing was missing due attention considering that it's not affecting the functionality of the system.

The Non Functional requirements were also not given proper attention in the earlier test cycles. However, this
has changed now. Non-functional tests are now most important as they consider all the application performance
and security issues these days.

This testing has a greater impact on applications when it comes to the performance of the application under high
user traffic. This testing ensures that your application is stable and is able to handle load under extreme
conditions.
Last Updated:April 16, 2020

As the name itself depicts, this testing concentrates on the non-functional aspect of the application. So what are
the non-functional aspects? Or should I say what are the features which are not related to the functionality of the
application?

Well, here are the answers to those:

 How does the application perform under normal circumstances?

 How does the application behave when too many users login concurrently?
 Can the application handle stress?
 How secure is the application?
 Can the application recover from any disaster?
 Can the application behave in the same way in a different environment or OS?
 How easy is to port the application in a different system?
 Are the documents/user manual provided with the application easy to understand?

The list keeps going on. But the point here is that – are not these features contributed towards the quality of the
application? The answer is YES. These features are equally important.

Imagine that an application meets all the user requirements perfectly, but some unauthorized user easily goes
and cracks the data entered by the user in the application, or the application dies when more than 5BB of any file
is uploaded. So would you say that the application is of good quality? Obviously not right!!

Purpose

The sole purpose of this type of testing is to ensure that the non-functional aspects of the application are tested
and the application works well in context to the same.

The purpose is to cover the testing of all the characteristics of the application which help to provide an application
that meets the business expectation.

Example

This is an important testing type.

Functional testing tests the application's functionality and ensures that it works as expected but the non-
functional testing ensures that the application works good enough to meet the business expectations.

In order to understand its importance let’s take a simple Example:

An application is developed and is completely tested for functionality, but non- functional testing is not performed
on the same.

Meanwhile, when the application goes live, it might result in critical or major issues like when the load is increased
on the application, it becomes too slow and takes a lot of time to open.

Response time might increase or when the load is increased to an extent, the application might crash. This shows
how important it is to test an application's non-functional aspects.
Last Updated:April 16, 2020

Advantages
Given below some of the advantages of a non-functional test:

 It covers the testing which cannot be covered in functional testing.

 It ensures that the application runs efficiently and is reliable enough.
 It ensures the security of the application.

How to Capture Non-Functional Requirements?

While we perform testing, the focus is mainly on functional testing which tests the functionality of the product.
But non-functional testing is as important as functional testing and its requirement should be taken into
consideration right from the inception of the product.

Non-Functional requirements are used to perform Non-Functional testing. These requirements include the
performance output that is expected from the application or the software under test. This basically includes the
time taken by the software to operate a particular system.

Non-Functional requirements also capture the behavior when a large number of people are using the software at
the same time. Most of the time it is experienced that the servers are busy or unavailable due to heavy load (i.e.
more people are using it at the same time). Booking online railway tickets can be the best example of such a
situation.

Hence documenting the Non-Functional requirement properly and performing the testing correctly will ensure
high satisfaction in terms of usability by the potential customers.

Though this testing does not have a direct business impact on the functionality of the system, it can increase the
user experience and user-friendliness to a higher extent which in turn will have a greater impact on the quality of
the software.

Example:
Last Updated:April 16, 2020

Consider the same Facebook login page example. In this case, the scope of Non-Functional testing is to note the
time required by the system to log in to Facebook after entering the valid credentials.

Also, it can be tested as when (let’s say 100) the users log in at the same time, how much time does it take to log
in the user on Facebook.

This ensures that the system can handle load and traffic which in turn has a good user experience.

In agile, the non-functional requirement should be captured using inputs.

A non-functional requirement should be captured as:

 User /Technical Stories

 In Acceptance criteria
 In Artifact

1) User /Technical Stories

A non-functional requirement can be captured using user stories or technical stories. Capturing Non-functional
requirements as a user story is same as that of capturing any other requirement. The only difference in the user
and a technical story is that the user story requires discussion and has visibility.

2) Acceptance Criteria

Acceptance Criteria is the point that is defined for accepting the product by the customer i.e. to get the product
accepted to the defined points should be in pass state.

A non-functional requirement should be included in the acceptance criteria but sometimes it’s not possible to test
the non-functional requirements with every story i.e. with every iteration. Hence, the requirements should be
added or tested with the relevant iteration only.

3) In Artifacts

A separate artifact should be prepared for the non-functional requirements, this, in turn, would help to have a
better idea of what needs to be tested and how it can be done in iterations.
Last Updated:April 16, 2020

Difference in Functional & Non Functional Requirements

There are several differences between the functional and non-functional requirements and few of them are
stated below:

S.No. Functional Requirement Non functional Requirement

1 Functional requirement is Customer Non Functional Requirement is based on developers and
based. technical knowledge of the team.
2 Functional Requirement specifies which Non functional requirements specify how it needs to be
functionality to be taken into tested.
consideration i.e. what needs to be
tested.
3 Functional testing is performed before Non functional requirements includes the maintenance
the application goes live. testing, Documentation testing which are not required
while execution is going on but one the application has
gone live.
4 It is known as functional requirement Also known as Quality requirements.
only.
5 The implementation plan for functional The implementation plan for non functional requirement
requirement is defined in system design is defined in system architecture.
document.
6 Functional requirement includes testing Non functional requirement includes qualities like security
of technical functionality of the system. ,usability etc.
Further reading = > Differences between Functional and Non-Functional Testing

Is This Black Box or White Box Testing?

The non-functional test comes under a black box testing technique.

This technique is not limited to test the functionalities only but can also be used for testing the non-functional
requirements as well as the Performance, usability, etc. Black box testing technique does not require any
knowledge of the internal system i.e. it does not require the knowledge of code to the tester.

Non-Functional Test Cases Checklist

A checklist is used to ensure that no important aspect is left without testing.

A checklist is generally used when there is no time for documentation and the product has to be tested or when
there is a time constraint, a checklist can be used to ensure that all the important aspects have been covered.
Last Updated:April 16, 2020

Let’s see an example of Performance, Security & Documentation testing checklist.

Checklist for Performance Testing
 The response time of the application should be verified i.e. how long does it take to load the application,
any input given to the application provides the output in how much time, refreshing the browser, etc.
 Throughput should be verified for the number of transactions completed during a load test.
 Environment set up should be the same as the live environment or else the results would not be the same.
 Process time – Process activities like import & export of excel, any calculations in the application should
be tested.
 Interoperability should be verified i.e. a software should be able to inter-operate with the other software’s
or systems.
 ETL time should be verified i.e. the time taken in extracting, transforming and loading the data from one
database to another.
 Increasing Load on the application should be verified.

Checklist for Security testing

 Authentication: Only an authentic user should be able to Log in.
 Authorized: User should be able to log into those modules only for which he is authorized or for which
the user has been provided access to.
 Password: Password requirement should be verified i.e. password should be as per how the requirement
defines i.e. length, special characters, numbers, etc.
 Timeout: If the application is inactive, then it should timeout in a specified time.
 Data Backup: Data backup should be taken at a specified time and should be copied to a secured location.
 Internal links to the web application should not be accessible if placed directly in the browser.
 All communication should be encrypted.

Checklist for Documentation Testing

 User & System documentation.
 Documents for training purpose.

Approach Document
Develop a specific approach document for the Performance Test stage by refining the overall Test strategy. This
Test approach guides in the planning and execution of all the Performance Test tasks.

 Test Scope
 Test Metrics
 Test Tools
 Key Dates and Deliverables
Last Updated:April 16, 2020

Test Scope
Conduct Performance Testing from different perspectives, such as user performance, business processes, system
stability, resource consumption, and so on. Types of Performance Testing to execute are discussed in the above
section of the article (like Load test, Stress test, etc.)

Test Metrics
The Test approach refines the metrics to measure and report during Testing, such as:

 Response time (online)

 Batch window (batch)
 Throughput (For Example, the number of transactions per unit of time)
 Utilization (For Example, the percentage of resources utilized)

Test Tools
Mostly Performance Testing requires the usage of appropriate tools:

 Load generation tools

 Performance monitoring tools
 Performance analysis tools
 Application profiling tools
 Base-lining tools.

Key Dates and Deliverables

The Performance Test Approach Document should describe the following:

 Date and time of each Performance Test conduct.

 Types of tests and functionality mix to be included in each Performance Test conduct.
 Performance Test completion dates.

Non-Functional Testing Types

The following image depicts the types of Non-Functional testing:
Last Updated:April 16, 2020

Performance Testing:
Evaluates the overall performance of the system.

Key elements are as follows:

 Validates that the system meets the expected response time.

 Evaluates that the significant elements of the application meet the desired response time.
 It can also be conducted as a part of integration testing and system testing.

Load Testing:
Evaluates whether the system’s performance is as expected under normal and expected conditions.

Key points are:

 Validates that the system performs as expected when concurrent users access the application and get the
expected response time.
 This test is repeated with multiple users to get the response time and throughput.
 At the time of testing, the database should be realistic.
 The test should be conducted on a dedicated server which stimulates the actual environment.

Stress Testing:
Evaluates whether the system’s performance is as expected when it is low on resources.

Key points are:

 Test on low memory or low disc space on clients/servers that reveal the defects which cannot be found
under normal conditions.
 Multiple users perform the same transactions on the same data.
 Multiple clients are connected to the servers with different workloads.
 Reduce the Think Time to “Zero” to stress the servers to their maximum stress.

Think Time: Just like the time interval between typing your user and password.

Volume Testing:
Evaluates the behavior of the software when a large volume of data is involved.

Key points are:

 When the software is subject to large amounts of data, checks the limit where the software fails.
 Maximum database size is created and multiple clients query the database or create a larger report.
 Example– If the application is processing the database to create a report, a volume test would be to use
a large result set and check if the report is printed correctly.

Usability Testing:
Evaluates the system for human use or checks if it is fit for use.
Last Updated:April 16, 2020

Key points are:

 Is the output correct and meaningful and is it the same as which was expected as per the business?
 Are the errors diagnosed correctly?
 Is the GUI correct and consistent with the standard?
 Is the application easy for use?

User Interface Testing:

Evaluates the GUI.

Key points are:

 GUI should provide help and tooltips to make it easy for use.
 Consistent for its look?
 Data is traversed correctly from one page to another?
 GUI should not annoy the user or get difficult to understand.

Compatibility Testing:
Evaluates that the application is compatible with other hardware /software with minimum and maximum
configuration.

Key points are:

 Test each hardware with minimum and maximum configuration.

 Test with different browsers.
 Test cases are the same as those that were executed during functional testing.
 In case the number of hardware and software are too many, then we can use OATS techniques to arrive
at the test cases to have maximum coverage.

Recovery Testing:
Evaluates that the application terminates gracefully in case of any failure and the data is recovered appropriately
from any hardware and software failures.

The tests are not limited to the below points:

 Power interruption, to the client while doing CURD activities.

 Invalid database-pointers and keys.
 Database process is aborted or prematurely terminated.
 Database pointers, fields and keys are corrupted manually and directly within the database.
 Physically disconnect the communication, power turn off, turn down the routers and network servers.
Last Updated:April 16, 2020

Instability Testing:
Evaluates and confirms if the software installs and uninstalls correctly.

Key points are:

 Validates that the system components are installed correctly on the designated hardware.
 Validates that the navigation on the new machine updates the existing installation and older versions.
 Validates that with insufficient disc space, there is no unacceptable behavior.

Documentation Testing:
Evaluates the documents and other user manuals.

Key points include:

 Validates that the stated documents are available in the product.

 Validates all the user guides, set up instructions, read me files, release notes and online help.

Failover Testing:
Failover testing is done in order to verify that in case of a system failure the system is capable enough to handle
extra resources like servers.

In order to prevent such a situation, backup testing plays a big role. Creating a backup system is what the process
is all about. If the backup is available, then it helps to get the system back.

Security Testing:
Security testing is done to ensure that the application has no loopholes which could lead to any data loss or
threats. It is one of the important aspects of non-functional testing and if not performed properly, it can lead to
security threats.

It includes testing authentication, authorization, integrity, and availability.

Scalability Testing:
Scalability testing is done to verify if the application is capable enough to handle increased traffic, number of
transactions, data volume, etc. The system should work as expected when the volume of data or change in the
size of data is done.

Compliance Testing:
Compliance testing is done to verify if the standards defined are being followed or not. Audits are done to verify
the same.

For Example, Audits are done to verify the process of creating test cases/test plans and placing them in the shared
location with the standard name that is being done or not. In QC, while naming the test cases the standard test
case name is being followed or not. Documentation is complete and approved or not.

These are the few pointers that are covered while auditing.
Last Updated:April 16, 2020

Endurance Testing:
Endurance Testing is done to verify the system's behavior when a load is increased to an extent for a long time.

It is also called as Soak testing & Capacity testing. It helps to verify if there are any memory leaks in the system.
Endurance testing is a subset of load testing.

Localization Testing:
Localization testing is done to verify the application in different languages i.e. different locales. The application
should be verified for a particular culture or locale. The main focus is to test the content, GUI of the application.

Internationalization Testing:
Internationalization testing is also known as i18n testing.

I18n represents I –eighteen letters- N. It is done to verify if the application works as expected across all the
language settings. It verifies that any functionality or application itself does not break i.e. the application should
be capable enough to handle all the international settings.

It also verifies that the application gets installed without any issues.

Reliability Testing:
Reliability testing is done to verify if the application is reliable and is tested for a specific period of time in the
defined environment. An application should give the same output as expected every time, only then it can be
considered as reliable.

Portability Testing:
Portability testing is done to verify if in case a software/application is installed on a different system or on a
different platform it should be able to run as expected i.e. no functionality should be affected because of a change
in the environment.

While testing, it is also required to test the change with the hardware configuration such as the hard disk space,
Processor and also with different operating systems to ensure that the applications correct behavior and expected
functionality are intact.

Baseline Testing:
Baseline testing is also known as benchmark testing as it creates a base for any new application to be tested.
For Example: In the first iteration, the response time for an application was 3 seconds. Now, this has been set as
a benchmark for the next iteration and in the next iteration, the response time changes to 2 seconds. It is basically
a validation document which is used as a base for future references.

Efficiency Testing:
Efficiency testing is done to verify if the application works efficiently and the number of resources required, tools
required, complexity, customer requirement, environment required, time, what kind of project it is, etc.
Last Updated:April 16, 2020

These are some of the pointers that would help to define how efficiently an application would work if all the
considered parameters work as expected.

Disaster Recovery Testing:

This testing is done to verify the success rate of recovery of an application or system if any critical failure happens
and whether the system is able to restore the data and application or the system could cope up easily to return
the way it was working earlier i.e. from the operational front.

Maintainability Testing:
Once the application/Product goes live, then there are chances for an issue to come up in the live environment or
the customer may want an enhancement for the application which is already live.

In this case, maintenance testing team is available to test the above scenarios mentioned. Once the application
goes live it still needs maintenance for which the maintenance testing team works.

Non-Functional Test Tools

There are several tools available in the market for Performance (Load & Stress) testing.

Few of them are listed below:

 JMeter
 Loadster
 Loadrunner
 Loadstorm
 Neoload
 Forecast
 Load Complete
 Webserver Stress Tool
 WebLoad Professional
 Loadtracer
 vPerformer

Is Non-functional Testing Always Carried out without Documentation and Test Cases? Why?

“We are always taught how to write functional test cases. Why is that? Is ‘non-functional testing' carried out
without documentation (in other words, on an ad-hoc basis) or is that a separate process that is much more
difficult to understand? How are test cases written for different kinds of testing that happen on an application?”

This is one of the most original, distinctive and out-of-the-box questions that I have been asked in recent times.
Let's find the answer.

How come we never get to see and practice on writing non-functional test cases?

Let’s start with what we know and as always a practical scenario.

Example: The following are the steps to be performed on an Online Banking application in order to perform a
transfer. Let us use that as our test for reference.
Last Updated:April 16, 2020

1. Login to the site.

2. Choose the bank account.
3. Choose the payee (this payee could belong to the same bank or a different one- this depends on your
data choice to execute this step. In any case, choose one. Also, we are going to assume that the payee is
already added.).
4. Enter the amount to be transferred (positive value, within the limit, correct format, etc.).
5. Click Transfer and check if confirmation is received, account balance has been updated and all that.

This is the functional test case, correct?

On the same application, on the same transfers’ page, let us say we are performing Performance, Security and
Usability Testing. These are non-functional types, correct?

How would we write the test cases?

#1) Usability Testing Test cases

Usability testing is a genre of software testing that deals with user experience. These are some of the questions
that we try to answer.

 How easy is the application to use?

 How satisfying is the experience of using the system?
 If not that familiar right away, how easy is to learn?

More info on it is here: Usability Testing Guide

How would a user determine the answers to the above questions in the context of usability testing?

The user would perform the exact same steps to make as in the functional test case. Am I right?

#2) Performance Testing Test cases

There are several variations to performance testing, but at its core, it is used to get statistics about the system, its
resource utilization, response time, network consumption, etc. at various load points.
Check out our Performance Testing Tutorials to know more about it.
Now, If I were to test the performance of the transfers’ transaction, I would have 10, 20, 30, 100…1000…etc users
perform the transfer operation simultaneously or incrementally depending on what I want to target and gather
data about.
What steps would each user perform in order to use the transfer while the performance test is in progress?
The same exact steps as the functional test, correct?
#3) Security Testing test cases
Security Testing is a branch of QA that helps to make the software systems hack-proof. It identifies vulnerabilities
(potential problem areas in the software system), exploits them through penetration or white-hat testing
technique and when loop-holes are found, they are worked on.
Last Updated:April 16, 2020

When do I want to check if the transfers are hack-proof and are directed correctly to the intended recipients and
that there are no black spots in the entire process? I would perform the transfer while the monitoring process for
security leaks goes on in parallel.
Therefore, in effect, I am carrying out the exact same steps that I would normally do in case of a functional test
case.
I guess, we have enough to establish that the steps in all the situations are the same. The method and the intention
behind the process are what is different.
Let’s take a Comparative Look:
Type of Testing Who? Why? Intention
Accuracy
Functional testing QA testers Efficiency
Business applicability
Ease of use
Usability QA testers or real time users Ease of learning
Efficiency
Performance testers via a tool Response time
that treats the operation as a
transaction performed by a
Performance
certain number of concurrent Network usage etc
users while the tester is
analyzing all the logistics
Scanning tools and other Hack safe
Security monitoring system by Payee and the payer identity
specialized security experts protection etc.

What is interesting to note is that no matter what form of testing we want to do, all the steps are the same.

The real difference is that:

1. Who performs these steps?

2. What is the intention, or in other words what am I trying to achieve via this test?
3. The tools and techniques used.

Coming back to our question, why do we never learn to write non-functional test cases with all the detailed steps
that are there to it?

It’s because, at their very core, test steps for a variation on test types on a certain function are all the same,
functional or not. It is the intention that makes a difference and maybe the method.
Last Updated:April 16, 2020

Conclusion
Before performing non-functional testing, it is essential to plan the testing strategy correctly to ensure proper
testing. There are different tools that are available in the market to perform this type of tests like Load Runner,
RPT, etc.

This testing plays a major role in the success of an application and to build up good customer relationship and
hence it should not be neglected. This is one of the important parts of Software testing and testing cannot be
considered complete without this.

We can include non-functional testing details in the test plan or can create a separate strategy for it. In either
case, the goal is to have proper coverage of non-functional aspects of the software.

We hope that this process of delving deep into this topic has been as much fun to you as it has been presented to
you all. We would love to hear your feedback and thoughts on this subject.

How do you handle non-functional testing in your teams? And as always, let us know if you agree or disagree
or have anything to add to what we have going on here.