International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)

Volume-3 Number-3 Issue-12 September-2013

Assessing the Quality of M-Learning Systems using ISO/IEC 25010

Anal Acharya1, Devadatta Sinha2

Abstract In this work, we first develop a M-learning

framework which can be adapted to the design
Mobile learning offers several advantages over requirements and factors that are critical to the
other forms of learning like ubiquity and idle time success of a M-learning application. However, for all
utilization. However for these advantages to be the advantages of M-learning to be addressed
properly addressed there should be a check on the successfully, the quality of the system developed
system quality. Poor quality systems will invalidate should be high. To this end, we derive a set of
these benefits. Quality estimation in M-learning metrics which measure the quality of M-learning
systems can be broadly classified into two systems. For this we extend the characteristics and
categories: software system quality and learning sub characteristics of ISO/IEC 25010 Software
characteristics quality. In this work, a M-Learning engineering - Software product Quality Requirements
frame work is first developed. Software System and Evaluation (SQuaRE) model [13] that are
quality is then evaluated following the ISO/IEC relevant to M-learning. The derived metrics are then
25010 Software Quality model by proposing a set of measured using a set of relevant attributes and
metrics which measure the characteristics of a M- suitably interpreted. The use of these metrics were
Learning systems. The applications of these metrics then illustrated numerically.
were then illustrated numerically.
2. A framework for M-Learning
Keywords
Mobile devices along with wireless connections are
M-Learning, ISO/IEC 25010, Quality Characteristic, central to the concept of M-Learning system giving
Requirements, Measurement, Framework, Mobile. access to the learning environment (Table 1). This
environment in turn generates a set of learning
1. Introduction characteristics. The involvement of the system
developers, tutors and learners should validate these
There are an estimated 6 billion mobile phones in the characteristics.
world today. This is about six times the number of
Personal Computers (PC) available [15]. In fact the 2.1 Devices and infrastructure
processing power that is available in certain high The first and foremost advantage of M-learning over
range mobile phones is similar to those of low end traditional class room learning and E-learning is
PCs available in today’s market. These facts make us anytime anyplace learning. Thus a busy company
speculate that in near future top range mobile devices executive who needs to travel a lot can utilize this
will replace the PCs in most applications. Coupled idle time by giving a test or taking a course; this is
with this, there has been a lot of advancement in the not possible in traditional learning. This can be
field of mobile communication. M-learning enables achieved in several ways. Tutors may mail the course
learning independently of place and time, is materials to the learners. Learners in turn may SMS
ubiquitous through the use of wireless networks and test answers to the tutors. The location of the learner
mobile devices, such as personal digital assistants during the learning phase also affects the learning
(PDA), cellular phones, smart-phones, and mp-3 process. If the learner is in remote location network
devices [6]. These changes make us believe that M- connectivity may be poor due to obstacles. Thus
learning could signal the end of classroom learning in bandwidth availability is an important factor
the near future [1]. influencing the success of M-learning. Gafani[4] has
identified a set of properties that are typical to
handheld devices. These devices in general have a
Anal Acharya, Computer Science Department, St Xavier’s poor user interface, small memory size, low
College, Kolkata, India. processing power, short battery life, small screens.
Devadatta Sinha, Computer Science and Engineering
Georgieva [9] in her comparative study has identified
Department, Calcutta University, Kolkata, India.

67
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

two types of M-learning systems: offline and online. not depend on network connectivity. Online systems
In offline systems domain data is loaded in the device can access the learning environment at any instant of
memory before the system is used. These systems do time using web services.
Table 1: A framework for M-Learning
Components Involved Implementation Desirable Outputs
Requirements

Mobile Device and Connectivity PDA, Cellular phones, Bandwidth Availability, Ubiquitous Learning
Smart phones, Tablets, Device Quality, Secure
MP3 Devices Network

Learning Characteristics Learning Contents, Reusable Learning Objects, SMS, Alerts, Instant Message,
Learning Methods, Personalized and Schedule Calendars, Chat
Learning Impact Collaborative learning tools forums, Discussion Boards
Development and Use Learners, Tutors and Software Management Course materials, Tests,
Developers Assignments, Feedback

M-Learning System

Pocatilu [3] has identified several security concerns learners prefer a course length less than six hours
in M-learning applications. Online exams, justifies this. His survey also concludes that 8-12 am
assessments and user management have high security is the best time for M-learning. Designing and
requirement whereas content management, quizzes identifying the necessary learning materials is
have low security requirement. The importance of the another important goal of M-learning. Courses and
mobile device itself cannot be overemphasized: the Learning Objects can be designed specifically
device may be lost or stolen. There are various types keeping the weak students in mind. Greg [10] has
of devices which can be used for m-learning: suggested two methods for doing this. Tutors can
Personal Digital Assistants (PDA), Mobile phones, assemble learning materials themselves keeping the
MP3 devices, tablets along with a host of other needs of their learners in mind. Also a vast resource
devices. In our work we concentrate on mobile of learning materials is available in the web. Tutors
phones. can use these materials instead of creating new ones.
Thus learning object reusability helps to optimize the
2.2 Learning Characteristics: study material development time. Finally the impact
BenMoussa[19] has identified several advantages of created by M-learning courses on the learners should
M-learning in addition to those discussed in the also be evaluated.
previous section. Filtering information according to
personal needs is personalized or individualized 2.3 System development, use and support
learning. Instant interactivity is achieved by Three major players are involved in the development
collaborative learning. Motiwala[14] suggests use of and use of M-learning environment [15]. The learner
alerts and scheduling calendars for personalized has to attend courses, take online tests, send
learning and Instant Messaging , Discussion boards assignments before due date and send course
and chat forums for collaborative learning. According feedback. The tutors have to create and identify
to Henry[11], there should be continuous assessment course contents, assess the exams and assignments,
of learning. This could be in the form of unit tests, send suitable feedback to the students. The system
submission of assignments and term end exams. The developer has to define and implement the databases
evaluative reports of these should be sent to the required for storing M-learning contents, details
learners for self correction. Also learning should be relating to learner and tutor management, messages
in short bursts. Liu’s[2] finding in a survey that most received and sent to learners. They also need to

68
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

identify suitable web services and servers to interface Some of the factors pertaining to hardware
mobile devices with the database. Finally, the learner requirement like low processing power, small keypad
may not be very conversant with the use of mobile are already discussed in Section 2. Factors relating to
devices for learning purpose. Suitable training should infrastructure include high cost of implementation(
be imparted to them to avoid disillusionment with due to the use of mobile devices, suitable learning
such type of learning [15]. object preparation and internet connectivity), diverse
types of devices and networks, locational diversity
3. M-Learning Systems quality leading to varied communication support. Finally
measurement factors relating to learning characteristics include
small units of information transfer between the tutors
The above framework illustrates several advantages and the learners, poorly organized contents, small
of M-learning application, namely ubiquity, course lengths. We summarize these problems below
personalized and collaborative learning, enhanced for use in section 4:
student satisfaction. However, as discussed earlier a
substandard system developed will nullify these i. Small Keyboard.
advantages. The framework proposed in the previous ii. Small memory size.
section suggests that quality in M-learning systems iii. Low processing power.
can be measured at two ways. One way is to measure iv. Low battery power.
software and system quality from the technical point v. High cost of implementation.
of view. Another way is to measure the quality of vi. Diverse devices and networks.
learning characteristics. We focus on the first aspect. vii. Locational diversity.
In this section we first specify the factors that affect viii. Small units of information transfer.
quality of M-Learning. There are several models ix. Poorly organized learning contents.
proposed for quality definition such as Boehm’s x. Small course lengths.
Model[18] and McCall’s Model[17]. The most recent xi. Small screen
of these is ISO/IEC 25010 Software engineering -
Software product Quality Requirements and
Evaluation (SQuaRE)model [13] which was
developed in 2008.We can then evaluate a M-
learning system with the parameters developed from
this model. The research methodology used is
illustrated in Fig 1.
3.1 Factors affecting quality in M-learning
environment

69
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

Fig 1: Research Methodology used in this paper

3.2 ISO/IEC 25010 model M-Learning system on the basis of software product
We have argued in the previous section that quality model as it should be used when setting
developing high quality system is of prime quality requirements for software products [13].
importance to realize all the benefits of M-learning. Below we show the diagram (Fig 2) of Software
This can be achieved by defining appropriate quality product quality model and discuss only those
characteristics which are suitable to evaluate the M- characteristics and sub characteristics that are
learning system. We use ISO/IEC 25010 Software relevant to M-learning environment.
engineering - Software product Quality Requirements
and Evaluation (SQuaRE) model [13] for this
purpose. This standard consists of two parts: A
software product quality model composed of eight
characteristics, which are further subdivided into sub
characteristics and a system quality in use model
composed of three characteristics, which are further
subdivided into sub characteristics. We evaluate the

70
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

Fig 2: Software Product Quality Model

Function suitability includes appropriateness, Compatibility includes repacability, co-existance,

accuracy and functional suitability compliance. interoperability, compatibility compliance. Co-
Appropriateness here means that the M-learning existence is important because the learner should be
system so developed should meet all the needs of the able to execute other applications on the mobile
learner and the tutor. device while executing a learning application.

Reliability includes availability, fault tolerance, Maintainability includes modularity, reusability,

recoverability, and reliability compliance. By fault analyzability, changeability, modification, stability,
tolerant we mean that the M-learning system should testability, maintainability compliance. We have
be able to maintain a certain level of performance in already discussed about the need of reusability of
spite of poor network coverage, device malfunction learning objects in the previous sections.
and other factors. Recoverability is required because
the system should be able to return to the previous Transferability includes portability, adaptability,
state even if there is a network failure. installability, transferability compliance.
Installibility is important because the application
Performance efficiency includes time behavior, should run on a wide range of devices, operating
resource utilization, and performance efficiency systems and mobile devices.
compliance. The communication time between the
server and the client in M-learning system should be Several researchers have concentrated on quality
as less as possible for fast downloads and hence the estimation in M-Learning applications.
need of time behavoiur. Spriestersbach et al[5] in their work has developed an
“adjusted” ISO 9126 model that can ensure the
Operability includes appropriateness recognizability, quality of mobile web application. They also discuss
learnability, ease of use, helpfulness, attractiveness, how various challenges in mobile web application
technical accessibility, operability compliance. Ease development may be solved using this model.
of use means that the learners and the tutors should Gafani[4] has designed a set of metrics form the
be able to master the system without any external characteristics of ISO/IEC 9126. These metrics were
help. The system also should be attractive to use for measured by a set of formula derived from the
the users in spite of poor interface, small screen and relevant attributes and an interpretation of these was
keypad. developed. These interpretations were then
empirically and mathematically validated using the
Security includes confidentiality, integrity, non “PDA Experiment”.
repudiation, accountability, authenticity and security
compliance. M-learning database contains a lot of 4. Quality metrics in M-learning
information about learners which should not be systems derived from ISO/IEC
accessed by all users. Thus suitable authentication
mechanism is required for the students and tutors 25010
alike. Confidentiality is also needed in case of
evaluations. In this section we develop the metrics that determine
the quality of M-learning. The metrics are first
defined and then measured using related attributes.
71
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

Finally an interpretation of these was developed. We factors that affect the corresponding quality
follow the notation adopted by Kitchenham [20]. characteristic. These factors were specified in section
3.1.
Table 1 lists the metrics derived from the
characteristics of ISO/IEC 25010. It also shows the
Table 2: Metrics derived from quality characteristics and the factors affecting them

Corresponding metric in Factors (defined in Section

Characteristic Sub Characteristic
M-learning context 3.1) that affecting quality

Function Suitability Appropriateness Feature completeness (v)

Average prob. of retrieving
Fault Tolerance information from device (ii),(iii),(iv)
Reliability memory
Average resume time after
Recoverability (vi),(vii)
disconnection
Average server response
Performance Efficiency Time behavior (vi),(vii)
time
Ease of use Friendly user interface (i),(xi)
Operability
Attractiveness Message compaction (i),(ii),(iv),(xi)
Authentication,
Security System Security (vii)
Confidentiality
Multiple application co-
Compatibility Application co-existence (ii),(iii),(iv),(v),(vi)
existence
Maintability Reusability Learning content reusability (viii),(ix)

Transferability Installability Installation Success (ii),(iii),(iv),(vi)

We now define these parameters and measure them Number of features actually implemented=I
by stating the relevant attributes and interpret these Computation: X=I/P.
results. Interpretation: Values closer to 1 indicates most of
the proposed features were actually implemented.
(A) Average probability of retrieving
information from device memory (C) Average resume time after disconnection
Definition: The degree to which certain data Definition: The degree to which the M-learning
generated during learning process that could not be system can reestablish connectivity in the event of
transmitted due to network problems still exists in network failure due to poor bandwidth.
device memory. Attributes Measured: Time interval for which
mobile device remains disconnected for task i
Attributes Measured: =ti
Number of disconnections occurred during Total number of disconnections=n
transmission =I Serial number of the disconnection=i
Number of times information available in device i=n
memory=A Computation: X= (∑ti )/n.
Computation: X=A/I. i=1
Interpretation: Values closer to 1 indicates most Interpretation: Lower Value indicates lesser
of the data lost during transmission still exists in disconnection time.
device memory. (D) Average server response time
Definition: The degree to which the M-learning
(B) Feature Completeness system provides appropriate response to the
Definition: The degree to which all the features learner’s queries irrespective of location and time.
proposed in design can actually be implemented. Attributes Measured: Time taken to get response
Attributes Measured: Number of features proposed to the query from the server
in design=P for task i =ti

72
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

Total number of tasks=n Maximum number of applications that can be

Serial number of the task=i executed simultaneously while running a M-
i=n Learning application=M
Computation: X= (∑ti )/n. Total number of applications=A
i=1 Computation: X=M/A.
Interpretation: Lower value indicates lesser Interpretation: Value closer to 1 indicates more
response time. applications can be executed while running a M-
Learning application.
(E) Friendly user interface
Definition: The degree to which the constraints of (H) Learning content reusability
inputting data like small screen and keypad can be Definition: The degree to which a learning object
overcome. can be used by more than one learner.
Attributes Measured: Number of data inputs with Attributes Measured: Number of times LO i is
radio buttons=Radioflds accessed=LOcounti
Number of data inputs with check boxes=Chekbox Total number of learning objects=n
Number of data inputs with list boxes=Listbox Serial number of the LO=i
Total number of fields=Totflds i=n
Computation: X= (Radioflds+ Chekbox+ Computation: X= (∑LOcounti )/n.
Listbox)/ Totflds i=1
Interpretation: Value closer to 1 indicate most Interpretation: Higher values indicate greater
field are input with radio buttons, check and list degree of LO reuse.
boxes and hence have friendly interface.
(I) Installation Success
(F) Message Compaction Definition: The degree to which the M-learning
Definition: The degree to which the M-learning software can be successfully installed or uninstalled
system enables easy exchange of SMS among across various networks and device types.
learners and tutors irrespective of location and time. Attributes measured: Number of device
Purpose: The learner needs to send test answers, types=InstalDev
assignments to the tutor in the form of SMS. Number of network types=InstalNet
Similarly the tutor also needs to send schedules and Number of device types where application
feedback to the learner. Due to the lack of successfully installed=IDev
bandwidth these SMS should be as compact as Number of network types where application
possible. successfully installed=INet
Attributes Measured: Computation:
Length of message in characters to perform task i= X=(IDev+INet)/(InstalDev+InstalNet)
MassLi Interpretation: Values closer to 1 indicate greater
Maximum message size supported by the degree of installability .
editor=MesssM
Total number of messages used for computation=n (J) System Security
i=n Definition: The degree to which authentication,
Computation: X= (∑MassLi )/n* MesssM. encryption and confidentiality mechanism exist
i=1 in the system.
Interpretation: Lower value indicate shorter
messages and hence better. Attributes measured:

(G) Application co-existence , if authentication, encryption

Definition: The degree to which several other 1 and confidentiality check
applications like attending a phone call, listening to mechanism exist respectively.
music etc can be executed in a mobile device while A,E,C=
executing an M-learning application without any 0 , otherwise.
detrimental impacts.
Attributes Measured: Computation: X=(A+E+C)/3.

73
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

Interpretation: A value of 1 indicates all the three each metric. Based upon the formula proposed
features are present in the system. above we compute the values of X. In order to scale
these in [0,1] we apply the transformation Y=1/X.
In some cases when this transformation yields
5. An Example values that are too small we apply the
transformation Y=(MAX-X)/MAX. For simplicity
The above section provides a set of metrics which we call these values X.
could be used as an indicator to measure the quality
of M-Learning. In this section we propose to We note the nature of the transformations indicate
illustrate these numerically. We assume two M- higher value of X enhances quality. The sum totals
Learning systems namely A and B. We further of XA and XB indicate that system A is marginally
assume a set of realistic values for the attributes of better than system B qualitatively.
The details are shown in the table below.

Table 3: Mathematical validation of the derived metrics

Attributes of A Attributes of B Transformation applied XA XB

I=50, A=27 I=50, A=22 None 0.54 0.44

Feature
completeness
Average prob. of P=15, I=12 P=13,I=9 None 0.80 0.69
retrieving
information from
device memory
Average resume ∑ti =540, n=15 ∑ti =712, n=18 Y=(MAX-X)/MAX 0.28 0.20
time after
disconnection
Average server ∑ti =377, n=15 ∑ti =412, n=15 Y=(MAX-X)/MAX 0.25 0.27
response time
Radioflds=12 Radioflds=09 None 0.63 0.53
Friendly user Listflds=10 Listflds=11
interface Checkflds=11 Checkflds=13
Totflds=52 Totflds=62
∑MessLi=1876 ∑MessLi=2173 Y=(MAX-X)/MAX 0.67 0.50
Message MessM=140 MessM=140
compaction n=20 n=31

A=1 A=1 None 0.67 1.00

Security E=0 E=1
C=1 C=1

Multiple M=7 M=6 None 0.64 0.60

application co- A=11 A=10
existence
∑LOcounti=73 ∑LOcounti=82 Y=1/X 0.20 0.16
Learning content N=15 N=13
reusability

InstallDev=5 InstallDev=6 None 0.73 0.64

InstallNet=6 InstallNet=5
Installation Success
Idev=3 Idev=3
Inet=5 Inet=4
Total Score 5.41 5.38

74
 International Journal of Advanced Computer Research (ISSN (print): 2249-7277 ISSN (online): 2277-7970)
Volume-3 Number-3 Issue-12 September-2013

6. Conclusions and Future Work [13] CD 25010.2, Software engineering-Software

product Quality Requirements and Evaluation
(SQuaRE)Quality model
This work evaluates a M-learning framework with [14] L Motiwalla, “Mobile learning: A framework and
the help of ISO/IEC 25010 model. The output is a set evaluation”, Computers & Education, Vol 49,pg
of metrics which were illustrated numerically. 581–596,2007.
However this evaluation was done purely from the [15] J Atewell, “Mobile Technologies and Learning
technical point of view. In our view ISO/IEC 25010 Report” Blackmore Ltd., Shaftesbury, England.
model alone is insufficient to measure M-Learning [16] M Virvou, A Ellapis, “Mobile educational
quality because it measures the software and system features in authoring tools for personalised
quality only. It does not measure the learning tutoring”,Computers & Education, Vol 44,pg
characteristics like the effectiveness of the learning 53–68, 2005.
[17] J A McCall, P K Richards, G Walters, “Factors
objects in learner’s context, personal and in software quality”. RADC TR-77-369, Vols I,
collaborative learning and the learning outcome. II, III',1977.
[18] B W Boehm,J R Brown, J R Kaspar, M Lipow ,
References & G McCleod, G. ,”Characteristics of software
quality”. Amsterdam: North Holland,1978.
[1] E Cherian., P Williams, “Mobile Learning: The [19] C BenMoussa, C,”Workers on the move: new
Beginning of the End of Classroom Learning, opportunities through mobile commerce”.
WCECS 2008. Presented at the Stockholm Mobility Roundtable,
[2] Y Liu, F Hu, H Li,”Understanding Learners’ May, 22–23,2003.
Perspectives on M-Learning:Results from a [20] B Kitchenham, S L Pfleeger, N Fenton,
Survey”, ACM,2009. “Towards a framework for software measurement
[3] Paul Pocatilu,”Developing Mobile Learning validation”. IEEE Transactions on Software
Applications for Android using Web Services”, Engineering, Vol 21(12), pg 929-94,1995.
Informatica Economică, 2009.
[4] Ruti Gafni, “Quality Metrics for PDA-based M-
Learning Information Systems”, Interdisciplinary
Journal of E-Learning and Learning Objects,
2009.
[5] A Spriestersbach, T Springer, “Quality Attributes
in mobile Web Application Development”, Anal Acharya is currently Assistant
LNCS 3009, Proceedings of PROFES, Berlin: Profesoser in Computer Science
Springer-Verlag, 120-130, 2004. Department in St Xavier’s College,
[6] Ruti Gafni ,”Framework for Quality Metrics in Kolkata. His research interests include
Mobile-Wireless Information Systems”, Distributed Learning Systems
Interdisciplinary Journal of Information,
Knowledge, and Management,2008.
[7] D Parsons, H Ryu, “A Framework for Assessing
the Quality of Mobile Learning”. Author’s Photo
Devadatta Sinha is currently
[8] D Parsons, H Ryu,”A Design Requirement Profesoser in Computer Science And
Framework for Mobile Learning Systems, Engineering Department in University
Journal of Computers”, 2007. of Calcutta. His research interests
[9] E Georgieva,,” A Comparison Analysis of include Distributed Systems and Data
Mobile Learning Systems”, International Mining Algorithms.
Conference on Computer Systems and
Technologies,2006.
[10] D Greg,”E-Learning Agents, The Learning
Organization”, 2007. Author’s Photo
[11] L Henry,S Sankaranarayanan,”Intelligent Agent
based Mobile Learning System”, International
Journal of Computer Information,2007
[12] M Shanmugapriya, A Tamilarasi, “Designing
An M-Learning Application for A Ubitiquios
Learning Environment in the Android based
Mobile Devices using Web Services” Indian
Journal of Computer Science and Engineering
(IJCSE), 2011.
75