Choosing a Learning Management System

Advanced Distributed Learning (ADL) Initiative

Peter Berking and Shane Gallagher

peter.berking.ctr@adlnet.gov
shane.gallagher.ctr@adlnet.gov

16 December 2016
Version 8.3

http://creativecommons.org/licenses/by-nc-sa/4.0/
Choosing a Learning Management System ADL Instructional Design Team

Table of Contents
1. Purpose and scope of this paper.............................................................................7

2. Overview......................................................................................................................7
2.1 What is an LMS?..............................................................................................................7
2.2 How widely are LMSs used?..........................................................................................10
2.3 What are the benefits of using an LMS?........................................................................10
2.4 Types of general learning goals managed by LMSs.......................................................12
2.5 Who uses LMSs and why?.............................................................................................13
2.6 The importance of choosing the right LMS.....................................................................15

3. Categories of systems to deliver and manage learning......................................15

3.1 General-purpose LMSs..................................................................................................16
3.2 Specialized LMSs........................................................................................................... 16
3.3 Learning content management systems (LCMSs)..........................................................17
3.4 Course management systems (CrMSs).........................................................................19
3.5 Virtual learning environments (VLEs).............................................................................21
3.6 Mobile learning LMSs.....................................................................................................22

4. Special features and issues to consider................................................................26

4.1 Enterprise system integration.........................................................................................26
4.2 LMS skins and templates...............................................................................................27
4.3 Programming language and platform dependencies......................................................27
4.4 Pricing models............................................................................................................... 28
4.5 Return on investment (ROI)...........................................................................................29
4.6 Open-source or freeware solutions................................................................................30
4.7 Government off-the-shelf (GOTS) solutions...................................................................33
4.8 Offline content provisioning and player capability...........................................................33
4.9 Security considerations for LMSs...................................................................................35
4.10 Hosting options.............................................................................................................. 36
4.11 Multiple LMS providers...................................................................................................39
4.12 The cross domain issue.................................................................................................39
4.13 Special requirements for U.S. DoD................................................................................40
4.14 System environments.....................................................................................................41
4.15 Standards support.......................................................................................................... 41
4.15.1 SCORM............................................................................................................................... 41

Choosing an LMS.docx page 3 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.15.2 Section 508......................................................................................................................... 44

4.15.3 Aviation Industry CBT Consortium (AICC)...........................................................................44
4.15.4 Standards for metadata....................................................................................................... 44
4.15.5 Learning Tools Interoperability™ (LTI)................................................................................45
4.15.6 Common Cartridge®............................................................................................................. 45
4.15.7 QTI®..................................................................................................................................... 45
4.15.8 ADL Total Learning Architecture (TLA), including xAPI.......................................................45
4.16 Internal assessment authoring.......................................................................................46
4.17 Internationalization......................................................................................................... 47
4.18 Instantiation for individual business units.......................................................................47
4.19 LMS sharing................................................................................................................... 47
4.20 The path of least resistance...........................................................................................48
4.21 Aligning staff and processes to system capabilities........................................................48
4.22 Planning for operation and governance of your LMS.....................................................49
4.23 Data migration................................................................................................................ 50
4.24 System load and bandwidth to the users........................................................................51

5. List of possible requirements for an LMS.............................................................51

5.1 Registration and enrollment functions and workflow......................................................52
5.2 Notifications, messages, and annotations......................................................................53
5.3 Batch administration workflow........................................................................................54
5.4 Prerequisite handling.....................................................................................................54
5.5 Content importing and configuration...............................................................................54
5.6 System access and security...........................................................................................55
5.7 Permissions and roles....................................................................................................57
5.8 System performance......................................................................................................58
5.9 Course catalog database...............................................................................................59
5.10 Interface customization..................................................................................................60
5.11 Standards and language support...................................................................................60
5.12 Training infrastructure and performance analysis...........................................................61
5.13 Learning object management.........................................................................................61
5.14 Delivery architecture......................................................................................................62
5.15 Cost............................................................................................................................... 63
5.16 Assessment authoring and delivery................................................................................64
5.17 Mentoring, coaching, and other developmental scenarios..............................................66
5.18 Collaboration and communication options......................................................................67
5.19 Competency management and development/learning plans..........................................69

Choosing an LMS.docx page 4 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.20 Learner tracking............................................................................................................. 70

5.21 Certificates, forms, polls, and surveys............................................................................71
5.22 Interfaces with external systems and applications..........................................................72
5.23 Metadata support........................................................................................................... 73
5.24 User profiles................................................................................................................... 73
5.25 Resource management..................................................................................................74
5.26 Analytics and reports......................................................................................................74
5.27 Ease of use for administrators........................................................................................75
5.28 Ease of use for learners.................................................................................................76
5.29 Transcripts and other documentation.............................................................................77
5.30 Scalability....................................................................................................................... 77
5.31 Vendor characteristics....................................................................................................77
5.32 User training, technical support, and documentation......................................................78
5.33 Media and content support.............................................................................................79

6. Popularity of features and capabilities..................................................................81

7. Emerging trends in LMSs........................................................................................82

7.1 Experience API (xAPI) adoption.....................................................................................82
7.2 Support and optimization for virtual immersive environments (VIEs)..............................83
7.3 Support and optimization for informal/social/collaborative learning................................84
7.4 Massive Open Online Course (MOOC) support.............................................................91
7.5 Open architectures.........................................................................................................91
7.6 Adding authoring capabilities.........................................................................................92
7.7 Extended enterprise learning.........................................................................................92
7.8 Adding knowledge management architecture and capabilities.......................................93
7.9 Support for team-based learning....................................................................................93
7.10 “Gadget”- based interface..............................................................................................94
7.11 Adding talent management architecture and capabilities...............................................94
7.12 Adding competency analysis tools.................................................................................95
7.13 Component-based architecture......................................................................................95
7.14 Multimedia LMS (MLMS)................................................................................................96
7.15 Learning Experience Manager.......................................................................................96
7.16 Video conferencing integration.......................................................................................97
7.17 Search-based learning...................................................................................................98
7.18 Content delivery networks (CDNs).................................................................................98

Choosing an LMS.docx page 5 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.19 Integration with digital libraries.......................................................................................98

7.20 Gamification of learning.................................................................................................99
7.21 Digital badges.............................................................................................................. 100
7.22 Adaptive learning systems...........................................................................................101
7.23 Content brokering systems...........................................................................................103
7.24 Affective computing......................................................................................................104
7.25 Support for performance support tools.........................................................................105
7.26 Is the traditional LMS dead?.........................................................................................105
7.27 Personal Data Lockers (PDLs).....................................................................................108
7.28 Web-based client systems...........................................................................................109
7.29 The Flipped Classroom................................................................................................109
7.30 Microlearning............................................................................................................... 109
7.31 Data analytics.............................................................................................................. 112

8. Process for choosing an LMS...............................................................................115

9. For more information about LMSs.......................................................................119

10. References cited in this paper..............................................................................120

Appendix.......................................................................................................................126
A. Sample System Requirements Matrix..........................................................................127
B. Sample System Features Rating Matrix.......................................................................128
C. Security Considerations for DoD LMSs........................................................................130
D. Sources of Possible Requirements for U.S. DoD LMS Acquisitions and Installations. .132
E. Additional requirements for LCMSs..............................................................................133
F. Examples of products...................................................................................................135

Choosing an LMS.docx page 6 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

NOTE: Vendor citations or descriptions in this paper are for illustrative purposes and do not
constitute an endorsement by ADL. All listings of vendors and products are in alphabetical order
unless otherwise noted.

1. Purpose and scope of this paper

The purpose of this paper is to help those involved in the process of
choosing a learning management system (LMS) to make an informed
decision. This applies to choosing an LMS for the first time, where
none was already in place, and replacing an existing LMS. The paper
presents a range of considerations for choosing a system; it does not
contain a comprehensive survey of all available systems on the
market, nor does it contain a comparative rating or evaluation of
products, and should not be construed as such. For more in-depth
information about systems and their features, see the references in 10
References cited in this paper or consult the vendors. ADL presents
this paper merely as a guide to the issues, opportunities, and
processes that should be considered in choosing a system.

Although this paper is focused on LMSs, we give some consideration

to the broader scope of learning-related systems: course
management systems (CrMSs), talent management systems (TMSs),
learning content management systems (LCMSs), and virtual learning
environments (VLEs). You must account for these in the process of
choosing a system to manage and deliver your learning, since you
must first determine the high-level, basic functionality you need; if
you need your system to manage instructor-led training classes, or
include authoring capabilities, one of these systems might be a better
choice than an LMS. (We must issue an important caveat here: the
capabilities of these systems are merging into each other and with
LMSs; the category labels and distinctions are becoming less
meaningful in many cases.) In an effort to include the total decision
process and options available, we present high-level descriptions of
these categories of systems in this paper, although we are mainly
focused on systems that are designed to deliver and manage
asynchronous eLearning (this is the traditional scope of LMSs).

In line with our mission to promote reusability and interoperability in

eLearning, ADL recommends systems with built-in features that allow
managing and delivering SCORM®-conformant eLearning. Acquiring a
system that does not support eLearning that is interoperable or

Choosing an LMS.docx page 7 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

reusable can be a significant business risk. You can find SCORM

considerations for LMSs in 4.15.1 SCORM.

2. Overview
2.1 What is an LMS?
The Learning Systems Architecture Lab at Carnegie Mellon states
that, “A Learning Management System (LMS) is a software package
used to administer one or more courses to one or more learners. An
LMS is typically a web-based system that allows learners to
authenticate themselves, register for courses, complete courses and
take assessments” (LSAL, 2004 in Gallagher, 2007).

Expanding on this definition, LMSs can be described as enterprise

level, server-based software systems used to manage and deliver
(through a web browser) learning of many types, particularly
asynchronous eLearning. They generally also include the capability of
tracking and managing many kinds of learner data, especially that of
learner performance. Many training organizations rely on their LMS as
a single point of access for all their eLearning content and learner
records. They are a key enabling technology for “anytime, anywhere”
access to learning content and administration. Some systems
(LCMSs) combine the above capabilities with authoring and content
repository functions. In some LCMSs, you can decouple the authoring
tool or content repository components and use them as separate
applications without relying on the LMS component of the system.
See 3.2 Specialized LMSs for more details. Authoring tool
functionality is covered in a separate ADL paper Choosing Authoring
Tools (available at
http://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAuthoringTools
.docx).

Other categories of systems that are related to LMSs are covered in 3.

Categories of systems to deliver and manage learning. As described
in 1. Purpose and scope of this paper, it is important to understand
the functions these provide in order to be able to precisely address
your requirements; an LMS (defined in the sense we define it here)
might not actually be what you need.

Choosing an LMS.docx page 8 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

You need to be careful also about the term “LMS.” Industry

professionals sometimes use it loosely to describe the other
categories of systems described in 3. Categories of systems to
deliver and manage learning. While it is true that the lines between
these systems are becoming more and more blurred as these other
categories of systems add LMS functionalities and vice versa, it is
important to determine and use the most appropriate label for a
system according to its primary use and market, in order to avoid
confusion. These other categories of systems are designed for
different sets of learning functions than a standard LMS.

Also, it is important to understand the difference between an LMS and

an LRS. A Learning Record Store (LRS) is a specialized system used
for storing Experience API (xAPI) data only. LRSs do not purport to
cover the myriad of functions that an LMS covers. An LMS can
actually include a built-in LRS, and an LRS product could include
some LMS functions, but these systems are fundamentally not the
same. LRSs are covered in a separate ADL paper Choosing an LRS
(available at
https://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAnLRS.docx).

The following general functions are normally provided by an LMS:

 Structure – centralization and organization of all learning-related

functions into one system, enabling efficient access to these
functions via layered interface navigation functions.

 Security – protection from unauthorized access to learning

content, learner records, and administrative functions.

 Registration – finding and selecting or assigning courses,

curricula, etc. by learners and their supervisors. This may
include instructor-led training classes.

 Delivery – on-demand delivery of learning content and learning

experiences to learners.

 Interaction – learner interaction with the content and

communication between learners, instructors, and course
administrators, as well as communication between the content
and the LMS (e.g., SCORM content).

Choosing an LMS.docx page 9 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Assessment – administering assessments and the collection,

tracking, and storing of assessment results data, with further
actions taken (possibly in other systems) based on the results of
assessment. Many LMSs include the ability to create
assessments as well.

 Tracking – tracking of learner data including progress on a

predefined set of training goals and requirements, and tracking
of courses for usage, especially in relation to required
deployment of mandated training (for example, compliance
training).

 Reporting – extraction and presentation of information by

administrators and stakeholders about learners and courses,
including the information that is tracked as described above.

 Record keeping – storage and maintenance of data about

learners. This includes both demographic info that profiles
learners and the training progress and accomplishments of
learners. This is especially critical when an LMS is deployed as
the official “system of record” for an organization.

 Facilitating Reuse – searching and recombining courses and

possibly parts of courses for delivery in different curricula and
learning tracks (this is a much more prominent feature of
LCMSs, but is often included in an LMS).

 Personalization – configuration of LMS functions, interfaces,

and features by learners and administrators to match personal
preferences, organizational needs, etc. In advanced systems,
“personalization” can also include tailoring of content or
learning experiences to address the learner’s assessed or
detected learning needs.

 Integration – exchange of data with external systems to facilitate

enterprise-wide tracking of learner performance and transfer of
user data, and to exploit external content and learning resources
(i.e. content management systems).

 Administration – centralized management all of the functions in

this list.

Choosing an LMS.docx page 10 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Added to the above general functions, in the most comprehensive of

LMSs (especially ones that incorporate functions of other categories
of systems–see 3. Categories of systems to deliver and manage
learning), one may find tools such as competency management,
skills-gap analysis (Gilhooly, 2001), succession planning,
certifications, virtual live classes, and resource allocation (venues,
rooms, textbooks, and instructors).

Bailey in Watson & Watson (2007) presents general characteristics of

LMSs in education that include:
 Tying instructional objectives to individual lessons
 Incorporating lessons into the standardized curriculum
 Extending courseware several grade levels consistently
 Providing a management system
 Collecting the results of learner performance
 Providing lessons based on the individual learner’s learning progress.
Further functionality is defined by the American Society of Training and Development (ASTD, 2008) as:
 Enabling integration with the human resources system or student information system (SIS)
 Incorporating tools to manage registrations, curricula, certifications, budgeting, and scheduling
 Providing access to content delivery
 Enabling content development, including authoring, managing and storing
 Integrating content with third-party courseware
 Assessing learners’ competency gaps
 Supporting assessment authoring
 Adhering to standards
 Supporting configuration to function with existing systems and processes
 Providing data security

LMSs are typically designed for multiple publishers and content

providers and usually do not include their own authoring capabilities
(that qualifies a system as an LCMS – see 3.2 Specialized LMSs).
Their main focus instead is on managing and delivering content
created from a variety of sources (Hall, 2002).

Choosing an LMS.docx page 11 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

2.2 How widely are LMSs used?

As an overall measure of the popularity of LMSs, Bersin (2014) has
reported that the LMS market is well over $2.5 billion and grew by
over 21% in 2014. Hougan (2015) cites research from Markets and
Markets indicating that the market will grow to nearly $8 billion by
2018. Upside Learning (2016) supports this figure by predicting that
“…between the years 2017 and 2018, the LMS market will grow by
about 23.17%, with an estimate of growth from $2.65 billion in 2013 to
$7.8 billion in 2018, which is roughly an annual growth rate of 25.2%.”
A large contributor to this growth, they say, is the uptake of LMSs by
small and medium businesses.

In higher education, Brown et al (2015) report that “Estimates of

institutions running an LMS are almost always near 99%. According
to the first ECAR survey of faculty and IT, 85% of faculty use an LMS
(with 56% using it on a daily basis), and 74% say it is a useful tool to
enhance teaching. Among students, 83% use an LMS, and 56% say
they use it in most or all courses.” (p. 2)

According to a survey conducted by Elearning! magazine (Roche &

Upton, 2013), 89% of the corporate sector are using an LMS, and 63%
use an LCMS. Brandon-Hall (2012) reports that over 78% of
organizations in their 2012 survey reported using an LMS.

2.3 What are the benefits of using an LMS?

If eLearning is already being used in your organization, though not
delivered and managed through an LMS, the benefits of using an LMS
derive mostly from the automation and centralization of functions that
accrue from adopting almost any enterprise system. LMSs have been
attributed as having the following benefits as well:

 Reducing costs through decreased training redundancy and

reduced operational errors and down-time

 Maximizing efficiency through the integration of content

delivery, reducing complexity and costs of auditing

 Leveraging existing resources by including established policies

and procedures; utilizing existing training material and links to

Choosing an LMS.docx page 12 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

“off-the-shelf” commercial computer-based courseware

(Szabo & Flesher, 2002)

If an LMS is being used to enable a switch from classroom instructor-

led training (ILT) to eLearning in your organization, the following
benefits can be expected from this switch to eLearning (delivered and
managed by an LMS):

 Time and location flexibility

• ELearning can be delivered to a student at any time and any

location worldwide (where there is a computer on a network).

 Language flexibility

• ELearning can be designed to support multiple languages.

 Scale of delivery

• Thousands of students can take a single eLearning course at

the same time.

 Cost savings

• “Rule of Thirds” (Fletcher and Foster, 2002): Elearning can

reduce costs by one-third and additionally either reduce the
time to achieve instructional objectives by one-third or
increase achievement (holding time constant) by one-third.

• No instructor is needed.

• No classroom facilities are needed.

• No printed materials need to be created or shipped out.

 Flexibility in updating, adapting, and configuring eLearning

software

• eLearning content can be updated and deployed on an LMS

immediately.

Choosing an LMS.docx page 13 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

• Policies can be established to control access to content. LMS

administrators can then control who sees what, using
permissions settings.

• Different content can be delivered to individual students

according to characteristics in their user profile (their job
role, for instance).

 Training standards and uniformity

• ELearning can enforce standards and uniformity in training. It

ensures that everyone gets the same information in the same
way. There are no differences due to differences in instructor
styles, knowledge, or preferences.

• Elearning can be designed so that information can be

updated in multiple content objects from one source. In other
words, course authors can make a change to information in a
content object or database, and that change is automatically
made everywhere that information appears (in multiple
objects).

 Self-paced and just-for-me learning

• Content can adapt dynamically to the student’s preferences

and learning needs (using SCORM sequencing, for instance).
Assessments can evaluate the student’s level of knowledge
and learning progress and change the learning experience so
it is not too hard and not too easy. Pre-tests can allow
students to skip parts of content that they already know.

• ELearning develops students into better learners. It can allow

students to select learning activities that best fit their own
background, interest, and career at that moment, rather than
forcing them to be a passive receptor of information chosen
or prescribed by others. Research shows that eLearning
leads to more active student participation than classroom
training (Beam & Cameron, 1998)

• eLearning never loses patience with learners.

 Assessment and evaluation

Choosing an LMS.docx page 14 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

• The process of student assessment can be highly automated.

Assessments can be delivered, scored, recorded, and
analyzed without human intervention.

• Reports and data analytics can be generated in real time to

show weaknesses in the performance of students and
training materials.

 Learning achievement

• “Rule of Thirds” (see above): (… increase achievement

(holding time constant) by one-third...)

 Time to learn

• “Rule of Thirds” (see above): (… reduce the time to achieve

instructional objectives by one-third or increase achievement
(holding time constant) by one-third…)

 Collaborative learning environment

• Physically separated students and experts can be linked

together to form an online collaborative learning community.

• ELearning can be designed to allow students to ask

questions that they may not be able to ask in conventional
classrooms due to instructor pacing and style.

• ELearning allows students to safely express their personal

opinions without fear of being judged, and to share ideas with
each other more easily through online forums.

 Unlimited use of learning materials

• ELearning allows unlimited access and retrieval of electronic

learning materials. Information and knowledge are available
to students 24 hours a day from any location. Students can
review current or past information/knowledge stored in
courses they have taken or content repositories over and
over again.

 Enabling implementation of training policy

Choosing an LMS.docx page 15 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

• In the U.S. military, eLearning courses provide the glue or

linkage that supports some of the new emerging training
requirements, like Human Dimension, Army “Living Doctrine”,
ePublishing, and Mobile Learning.

2.4 Types of general learning goals managed by LMSs

In an instructional sense, LMSs are generally designed to account for
five basic categories of learning:

 Initial learning – acquiring skills and knowledge for the first time

 Continued learning – extending skills and knowledge in a

particular domain

 Remedial learning – refreshing skills and knowledge for learners

whose knowledge has decayed

 Upgrade learning – moving to a higher level of competence in

skills and knowledge already acquired

 Transfer learning – transfer of skills and knowledge learned in

one particular domain or context and transferring them to a
different one. For example, a trained and experienced Flash ®
developer who is now working in Captivate ® (which uses the
same Flash framework but with a different interface)

Most LMSs support all of these goals, although the system functions
that support them are not always the same. For example, the LMS
needs to ensure that remedial learning learners are not tracked the
same way as initial learning learners. Remedial learning learners
should have access to all parts of a course (suspending all forced
sequencing), without forcing them to take assessments and be
graded as in initial learning.

In a logistical sense, the categories of learning delivery that LMSs can

account for are:

 Learner-led – asynchronous, on-demand eLearning.

 Instructor-led – live presentation of content and supervision of

learning experiences by an instructor.

Choosing an LMS.docx page 16 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Co-located – instructor and learners co-located, usually in

a classroom setting.

o Virtual – instructor delivery of learning to learners at a

distance, using a technology such as video conferencing
hardware or web conferencing software. Learners may be
co-located in a distance learning classroom (without the
teacher, who is remotely located). A recording of such a
session for asynchronous playback to learners then moves
into the Learner-led category described above.

 Facilitated – so-called blended or hybrid learning. It combines

elements of learner-led and instructor-led (see above).
Facilitated learning is often used in situations where 100%
instructor-led training is impractical or costly, and some parts of
the content (but not all) can be delivered effectively through
asynchronous eLearning.

 Embedded – job and task performance support through just-in-

time Help and performance support systems.

It is important to note that LMSs do not often include support for all of
these categories; in fact, these are the main differentiators for the
categories of systems to manage and deliver learning (see 3.
Categories of systems to deliver and manage learning).

2.5 Who uses LMSs and why?

Some aspects of LMSs can be handled by simply putting files on an
intranet file server and using a spreadsheet or simple database for
tracking purposes. However, most enterprises’ needs go way beyond
the capabilities of such a home-grown system. And developing a
more robust system in-house can easily turn into a major software
system development project, beyond the capabilities and budgets of
many organizations. Thus, it is generally most cost-effective to
acquire a commercial system, or customize an open source system.

LMSs base much of their value proposition on their optimization for

ease of use by administrators and learners, and their automation of
time-consuming tasks. They offer streamlined and efficient

Choosing an LMS.docx page 17 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

administrative workflows, which can be very time-consuming without

carefully designed tools.

LMSs can be free and open source, but commercial versions can be
cost-prohibitive for small organizations. They also are technically
complex and require an administrative and maintenance
infrastructure and resources that also can be prohibitive for small
organizations. Thus, commercial systems generally make the most
sense for an enterprise with hundreds if not thousands of users,
where some level of centralized, automated control and record
keeping needs to be exerted over the process of learning. This being
said, “pay per use” pricing models for vendor-hosted LMSs can make
an LMS cost-effective and practical, even for very small organizations.

Generally, an LMS is not needed where:

 There are only a small number of users (in this case, a system
may be useful, but it would not be cost-effective).

 Learning can be delivered simply by sending learners a URL to a

file located on a file server on the enterprise intranet and they
are free to take the module(s) without any performance tracking
requirements (or they can self-report their status).

 Learning is delivered through classrooms and/or hard copy

medium such as paper documents, CD, or DVD, and little or no
systematic, centralized tracking is required.

One important rationale for needing an LMS is the fact that small and
mid-size companies or institutions do not have enough staff
resources to maintain a full-service HR capability in which employee
training needs are treated robustly and systematically. An LMS can fill
some of this need by automating and streamlining processes.

LMSs are commonly used by a variety of groups, mainly including

content developers, training administrators, course managers,
system administrators, instructors, and learners. These roles are
often accounted for in the default account categories/permission
levels available in many LMS products. Each group uses different
functional areas or a particular functional area for different purposes,
since each plays a different role in the learning delivery and

Choosing an LMS.docx page 18 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

management process. Figure 1 provides an example of how each

group uses various LMS functional areas:

Functional Content Training Course

area Developers Admins Managers System Admins Instructors Learners

Learning NA Define learning Ensure that Maintain system Assign Choose

tracks and tracks, course is integrity of learners to curriculum
curricula curricula, and correctly learning tracks, learning or view
target groups positioning curricula, and tracks and assigned
associated with within target groups curricula per curriculum
them learning track training
or curriculum needs

Course Test Review courses Review Import and Review NA

delivery developed in curriculum individual configure courses courses to
preparation courses to for content course for prepare for
ensure errors content errors providing
proper and delivery instructional
functioning problems support

Course NA Monitor to Monitor to Monitor and NA Find,

delivery ensure ensure course allocate course register for,
curriculum is is delivered delivery per and take
delivered as correctly bandwidth and courses
intended in server constraints
training plan

Operation of NA Review Review course Configure LMS to Provide Use course

course curriculum to to ensure enable proper course features as
ensure operation as operation of instructional intended
operation as intended courses support (via
intended LMS features
that enable
contact with
instructors)

Course NA Report Report and Maintain tracking Monitor Gauge

progress and progress analyze database and learner progress
completion across courses progress on generate reports progress
course and
Bill for course completion
usage, if Bill for and assign
applicable individual learners to
course usage, additional
if applicable courses as
necessary

Assessment Program Review Review Configure LMS to Review Take

assessments assessments assessments enable proper assessment assess-
(in LMS, if throughout in course operation of ment
applicable) curriculum to assessments Review
ensure assessment
compliance results and
with training take
plan appropriate
remedial

Choosing an LMS.docx page 19 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Functional Content Training Course

area Developers Admins Managers System Admins Instructors Learners

action

Learner NA Assess, Analyze Generate Monitor Monitor

performance document, and course usage statistics and learner perfor-
analyze and learner reports per needs performance mance
performance performance of training and assign
to evaluate administrator and learners to
course course manager additional
effectiveness courses as
necessary

Competencies NA Define Ensure Import and export Take further Receive

competencies competencies competencies actions competenc
are integrated to/from external based on y
into course systems learner certificatio
competencie ns from
s reported LMS
by LMS

Figure 1: LMS functional areas and roles for each

2.6 The importance of choosing the right LMS

Choosing a system to manage and deliver your learning is one of the
most crucial decisions any training organization can make. Though
most of these systems contain the same basic collection of functional
elements described in the previous section, they are optimized for
different types of learning goals, learners, and organizations.
Differences can be major in these respects. If your organization
chooses a system that is not optimized for your needs, you could end
up wasting your organization’s money and wasting time for your
learners and administrators, or worse, predisposing learners against
learning opportunities that may be important but are difficult to
access and take.

Another critical factor in choosing these systems is durability. This

relates to whether the system will have longevity in the marketplace
such that it continues to be available and supported with periodic
maintenance and upgrades. This is important, at least to account for
evolutionary changes in the IT environment (both hardware and
software) within which it operates. It also relates to whether the
system will, in the future, support delivery of new file formats,
possibly incorporating revisions to standards like SCORM.

Choosing an LMS.docx page 20 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

As with all enterprise systems, LMSs should also be chosen with

consideration for extensibility, scalability, and, generally, how they
will fit and integrate within the overall enterprise architecture of the
organization. Extensibility considerations tend to take into account
the modularity of the system and how services can be customized or
increased to meet changing user needs. When thinking about
scalability, the growth patterns and projections of the organization
are important in evaluating whether or not an LMS can meet the
potential volume demands through growth. Fit tends to consider the
organization’s other non-learning specific business needs and how
the LMS will integrate and support other business-related systems. To
this end, it is very important to involve appropriate non-learning
stakeholders as well as IT department staff in all discussions from the
very beginning.

Although this paper is primarily predicated on first-time acquirers of

an LMS, most of this information is also applicable to those switching
LMSs. There can be many drivers for this decision, but it usually
comes down to cost and technology affordances. LMSs are gradually
coming down in cost, and new LMSs are being built from the ground
up to seamlessly incorporate new technologies. These two factors
have been motivating training stakeholders to consider switching;
according to a survey conducted by Bersin & Associates, one-quarter
of customers were dissatisfied with their current LMS and said they
were likely to switch vendors within the next year (Upside Learning,
2014).

3. Categories of systems to deliver and manage learning

Systems to deliver and manage learning run a wide gamut. This
section describes the major categories of available systems. These
categories are key to choosing a system, since they set the stage for
allowing you to align your major requirements to the type of system
you need. It is important to note that these categories are not
mutually exclusive. Some systems have core elements that qualify
them for two or more categories. However, in these lists, systems are
assigned to one category as their primary intended use or design
architecture.

Choosing an LMS.docx page 21 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

As described earlier, the labels for categories are used loosely by

vendors and others. Many vendors simply categorize their system
based on where they perceive the bigger market lies for their
collection of capabilities, not based on their system’s primary
functionality. The other factor that plays into the categorization and
labeling of systems is user community usage patterns. Many users in
the higher education community will call whatever system they use a
course management system (CrMS), whether it strictly fits that
definition or not.

Some argue that the primary differentiating categorization scheme of

these systems should be whether they are geared for corporate
(including government) or academic users. The differences are
usually apparent in the terms used within the product, for example
“curriculum” for academically-oriented products vs “training track”
for corporate-oriented products. The default when “LMS” is
mentioned in public forums usually presumes a corporate LMS. These
are indeed the predominant type of widely used LMS in use (67%).
However, academic LMSs make up a large and evolving segment of
the LMS marketplace (30%) (Vipond and Clarey, 2016).

However, the authors feel that it is more meaningful to categorize

systems more in terms of pure functionality rather than the market
within which they operate. Within each functional category of system
(LMSs, LCMSs, CrMSs, VLEs) there are differences in the way the
systems are designed and marketed, for the academic vs corporate
market (and within academic, there are differences between K-12 and
higher ed systems). However, we feel that focusing on functionality is
more meaningful for those in the process of choosing an LMS,
especially since “academic LMSs” are often used in corporate
settings (especially in the case of corporate universities) and vice
versa.

Although six categories of systems to deliver and manage learning

are discussed in this section, this paper focuses mainly on systems
that we categorize as “General-purpose LMSs”.

3.1 General-purpose LMSs

LMS are optimized for delivery of learner-led and embedded learning,
explained in 2.4 Types of general learning goals managed by LMSs.

Choosing an LMS.docx page 22 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

They can include support for facilitated and instructor-led training

and education, but that is usually not their primary focus. They are
primarily designed to manage and deliver asynchronous eLearning,
although many of the large LMSs for corporate or academic use offer
tools for administering traditional classroom-based, instructor-led
training events (this is the primary function of CrMSs described in 3.4
Course management systems (CrMSs)).

This category is termed “general purpose” because it covers the

systems with feature sets that are most universally applicable to a
wide variety of use cases and users, accounting for most of the
requirements that are common to all enterprise learning
environments.

LMSs are used primarily in the business and government training

community (although a growing number are oriented towards the
academic community). This is ingrained into the minds of users to the
point where many systems that technically fit into one of the other
categories are often termed an LMS. Thus, the LMS label is used
loosely, but for purposes of differentiating and categorizing systems
in this paper, “LMS” includes only the systems that are primarily
designed to manage and deliver asynchronous eLearning, as
described above.

See Appendix F. Examples of products for a list of example systems

in this category.

3.2 Specialized LMSs

The LMSs in this list are explicitly geared and marketed for a particular use case, learning paradigm, or
type of user. Many of them are delivery platforms only, and do not handle other functions of an LMS like
managing learning records. They differ from LMSs in the “General-purpose LMS” category in that the
features in that category are generally applicable to most organizations, cases, etc., whereas “Specialized
LMSs” are more “boutique” in nature. NOTE: for information about LMSs that specialize in mobile
learning, see 3.6 Mobile learning LMSs.

See Appendix F. Examples of products for a list of example systems

in this category.

3.3 Learning content management systems (LCMSs)

LCMSs are closely related to LMSs, providing much of the same
functionality with the addition of content authoring and robust

Choosing an LMS.docx page 23 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

content repository features. Although LMSs necessarily include a

content repository as part of their architecture, LCMSs provide more
flexible access to the content repository component and more
management options for it.

The focus of an LCMS is the instructional content—its creation, reuse,

management, and delivery. This contrasts with the logistics of
managing learners, managing learning activities, and competency
mapping provided by an LMS (Oakes, 2002). In other words, an LCMS
focuses on the management of learning objects (LOs) while an LMS
manages the learning process as a whole (Watson & Watson, 2007).
This focus belies tremendous overlap in actual practice; Hall (2007)
states that 74% of LCMSs in their LCMS research report include
robust LMS functionality. And both systems manage and deliver
instructional content (often at the LO level), with an LMS being the
more comprehensive of the two in terms of system functions.

As in the case of LMSs, LCMSs are optimized for delivery of learner-

led and embedded (especially performance support) learning
strategies, explained in 2.4 Types of general learning goals managed
by LMSs. Like LMSs, they can include support for facilitated and
instructor-led training and education, but that is usually not their
primary focus. LCMSs, like LMS, are used primarily in the business
and government training community.

In its simplest form, as stated above, an LCMS is an LMS integrated

with authoring tool and advanced content repository functions. LCMS
content repositories are usually designed to manage many different
types of content objects, not just eLearning (or even training-related),
and generally include the following features that are not usually found
in an LMS:

 Versioning of files and/or content objects

 Authoring of eLearning (through web-based tools on the LCMS

server, not through software on the user’s system)

 Dynamic assembly of LOs into learning experience sequences

(at runtime, usually)

Choosing an LMS.docx page 24 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Ability to manage diverse and complex content object types.

This includes providing navigation controls, look and feel, and a
table of contents for a wide range of content object types.

 Web interface directly to the content files in the LCMS’s

repository

 User roles and privileges to manipulate content

 Cataloguing (through metadata tagging) and search to enable

discovery of content objects and/or files

Note that the term LCMS is sometimes simply used to refer to an LMS
that has bolted on authoring capability, without meeting the spirit of
the functionality described here for a true LCMS (i.e., dynamic
assembly of LOs at runtime).

Be careful if you already have established an eLearning development

capability, and staff in your organization are already using preferred
standalone authoring tools. There may be significant resistance (and
licensing issues) to changing authoring tools to an LCMS midstream.
In many cases, however, they can continue to use their preferred
tools, and files produced by these tools can be imported into the
LCMS.

As of 2013, LCMSs seem to be on the decline; according to a 2013

survey (Roche & Upton (2013), 63% of the corporate sector still used
an LCMS, but Vipond and Clarey (2016) report that LCMS ownership is
now down to 29%. There is speculation that the decline is due to the
fact that content generation is becoming ubiquitous and crowd-
sourced (user generated). For example, the ease of creating and
sharing instructional videos on smartphones may outweigh any
advantages of being able to author and manipulate video-based LOs
in your LCMS.

The primary advantage of LCMSs over LMSs is that LCMSs enable

assembly of courses (usually dynamically) from a variety of smaller
source content objects. Thus, if your environment requires output of
a variety of materials from a variety of source objects (for example,
producing an eLearning course from instructor-led training manuals
and vice versa), this is probably a good choice of a system.

Choosing an LMS.docx page 25 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

LCMSs have the following advantages over LMSs:

 Learning modules can be automatically assembled for delivery

by the system to the individual learner according to the learner’s
organization, role, language preference, learning needs, regional
differences, etc. Many LMSs can do this, but LCMSs allow this
process to occur on a much more granular and dynamic scale,
i.e., learning modules that are delivered to the learner are the
result of on-the-fly mixing and matching of smaller pre-defined
learning objects within the system. In LMSs, this automatic
selection and delivery process usually only happens on the level
of whole courses.

 Learners do not have to spend as much time looking for relevant

materials, since they are delivered automatically based on the
learner’s profile. LMSs often do this, but again, the tailoring of
appropriate material happens on a much more granular scale in
LCMSs.

 LCMSs can assemble different types of learning products (for

example, references, ILT courses, and eLearning courses) from
master objects dynamically. This means that edits can be made
in a master object, and changes immediately ripple through all
output products. This “single source, multiple outputs”
paradigm can enable much greater efficiencies in content
maintenance, especially where learning information is volatile.

 Content publishing to the delivery side of the equation, the LMS

component, is easier, since the authoring tool and LMS
component are integrated into one system.

 All of the advantages of web-based (as opposed to desktop

software based) authoring tools accrue since the built-in LCMS
authoring tool capability is always web-based. Possibly the
biggest advantage here is that access to content for editing
purposes by users other than a single course author (e.g.,
SMEs, clients, project managers, and multiple course authors) is
easier. This can result in a significant boost in speed of
authoring and updating content.

Choosing an LMS.docx page 26 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 The fact that learning modules are combinations of smaller

learning objects tends to make it harder for instructors to create
unauthorized personal “flavors” of courses, since there is less
chance that they would have authoring permissions access to
all of the components that make up a learning module.

 LCMSs include an integrated authoring tool, which usually

allows you to import and edit existing content with export to
multiple formats conformant to multiple standards or standards
versions.

 Individual assets and learning objects (including screens) can

be managed, not just courses. This can extend to objects used
by the system in navigation screens, permitting scenarios where
an organization logo that is used for branding purposes on
multiple LCMS system screens can be reused within content
screens (and updated in one place).

 Individual assets can be version controlled and configuration

managed via content repository functions.

 LCMSs are better optimized for delivering performance support

modules, because of their object-based architecture, i.e.,
learning objects can be reassembled dynamically in a format
that is better suited to the needs of users looking for just-in-time
information. Performance support objects can be automatically
selected for delivery by the system according to user media
preference, user device (i.e., mobile vs desktop computer) or
particular problem that needs to be solved.

 Competencies and objectives can be mapped explicitly to any

level of course organization and to learner progress, in some
cases assembling individual courses (not just curricula) for
learners dynamically based on their training needs.

 Learning objects and assets can be easily reused (within the

system).

LCMSs have the following disadvantages compared to LMSs:

Choosing an LMS.docx page 27 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 The learner management functions tend to be less robust, since

the system concentrates more on the authoring, assembly, and
delivery of content.

 Their capabilities are usually predicated on doing everything

within the LCMS system. They may not interoperate well with
other systems (for example, an external authoring tool).

 Navigation controls for courses usually are provided by the

LCMS, not the content (this is especially true where the content
is assembled dynamically).

 The level of effort (LOE) for administration is higher given the

greater number of individual learning objects that need to be
configured, version controlled, etc.

 The concept of an LCMS may be very attractive, but

implementation requires more of a commitment (than an LMS) to
reengineering your organizational culture to leverage its
reusable learning object and “single source, multiple outputs”
paradigm. If you do not make use of this LCMS capability, you
could end up wasting money (over the cost of a comparable
LMS).

See Appendix F. Examples of products for a list of example systems

in this category.

3.4 Course management systems (CrMSs)

CrMSs are most commonly used in higher education rather than
enterprise training environments. They are sometimes called
Education Learning Management Systems (ELMSs). The primary
focus of CrMSs is to manage all aspects of live instructor-led
classroom training, according to the categories of learning delivery
explained in 2.4 Types of general learning goals managed by LMSs.
They may include the ability to deliver learner-led courses, but these
are usually supplementary or ancillary to the instructor-led courses
they manage. As described earlier, many LMSs incorporate some
level of CrMS functionality and vice versa, since many enterprises
want to manage their instructor-led learning and eLearning in one
system.

Choosing an LMS.docx page 28 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Do not confuse course management system (CrMS) with content

management system (CMS) or competency management system (also
CMS). The acronyms are differentiated here so as not to confuse, but
often the same acronym (CMS) is used for all three. Content
management systems, by contrast with course management systems
(CrMS) are designed to manage work flow needed to collaboratively
create, edit, review, index, search, publish and archive many kinds of
content, mostly related to document publication. Competency
management systems usually include an LMS and are more
comprehensive in their approach to human resource development.
They include tools such as skills-gap analysis and succession
planning. They are closely aligned with, and often the same as talent
management systems (TMSs). See 7.11 Adding talent management
architecture and capabilities for more details.

As stated earlier, CrMSs are used primarily in the academic

community. That is their primary market target. CrMSs are sometimes
labeled as LMSs within the user community, but they are distinctly
different in the sense that they do not deliver the core learning
experiences—those are provided live in classrooms. However, a CrMS
vendor that has added eLearning delivery capability may term
themselves an LMS (though the preponderance of their functionality
really qualifies them as a CrMS).

CrMSs are predicated on the idea that instructors need to use the
system to build content, manage their courses, and contact learners.
LMSs are not usually designed from the ground up for instructor use
as a core function. They are generally more optimized for standalone
eLearning, with little or no instructor intervention.

The core features of a CrMS are:

 Instructors can store and post information and materials on the

web relating to their classes, and can put these materials under
version control and configuration management (including
stages of production).

 Instructors and administrators can manage and schedule a

variety of class-related resources, including classrooms and
instructors, and ancillary instructional materials such as

Choosing an LMS.docx page 29 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

references and required readings. Conflict handling is often a

part of these systems.

 Instructors can organize a class into groups and provide a

group work space for working on group-specific tasks and
projects.

 Instructors can mark and evaluate learner work while online.

 Instructors have grade book functions for recording learner

performance.

 A portfolio space can be created for learners where they can

showcase their work in a course, display contact information
and a photo.

 Instructors can organize appropriate support materials including

exercises, reference materials, labs, tests, etc. by course, class
section, curriculum, etc.

 Learners can collaborate (at least among learners in the same

class) using threaded discussion, chat and other
communication or social media tools.

Some CrMSs also include functions related to tracking the analysis,

design, development, and evaluation of training, and an audit trail for
tasks, learning objectives, instructional activities, lessons, courses,
and media.

In some cases, the decision whether to acquire an LMS vs LCMS may

be difficult, as there is significant overlap. However, CrMSs are
different from both of these in the sense that they generally do not
provide the capability to deliver the core learning; they are mainly
designed to manage the supporting infrastructure for live instructor-
led classroom training. Thus, it is irrelevant to discuss advantages
and disadvantages of CrMSs over LMSs, since the former are
designed more for education than for training.

Keep in mind that because of the overlap in functions and features,

differing product marketing strategies, and lack of universally
accepted labels, CrMSs are often termed “LMSs”.

Choosing an LMS.docx page 30 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Specialized CrMSs are emerging for Massive Open Online Courses

(MOOCs). MOOC CrMSs dispense with many of the standard CrMSs
features related to managing classroom courses, since MOOC
courses are by definition entirely virtual. In some cases, such as
Coursera, MOOC courses are offered and delivered exclusively
through a particular MOOC system; in others, the system can be used
merely as the delivery platform, independently from courses.

See Appendix F. Examples of products for a list of example systems

in this category.

3.5 Virtual learning environments (VLEs)

There are many software tools designed for general web-based virtual
business meetings and collaboration. VLEs (sometimes called virtual
classrooms) are a subset of these. They are often marketed as
webinar software, although webinar software tends to deal only with
one specific learning event (usually a business presentation) at a
time, not managing whole courses that might involve a collection of
webinars (in addition to other elements).

The primary focus of VLEs is to deliver, and to some degree, manage

virtual instructor-led learning (either synchronous or asynchronous).
VLEs support management of learning to varying degrees, and to this
extent they resemble CrMSs, but they focus mainly on providing the
delivery capability. They can also, in classic LMS style, include
support for asynchronous learner-led eLearning, but that is not their
primary focus. VLEs are generally more targeted towards the formal
educational environment rather than corporate training. VLEs are also
used widely for MOOCs, although VLE sessions are generally
contained within specialized MOOC CrMSs (see 8.3 Massive Open
Online Course (MOOC) support).

Note: Virtual immersive environments (VIEs) such as simulations,

virtual worlds, serious games, and augmented reality are sometimes
termed virtual learning environments (VLEs).

As in the case of CrMSs, it is moot to discuss advantages and

disadvantages of VLEs as compared to LMSs, since they are
designed for a different kind of learning experience. However,
because they overlap much more nowadays with LMSs in terms of

Choosing an LMS.docx page 31 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

sharing the ability to deliver eLearning, it is worth pointing out some

of the differences in the way these two systems function.

VLEs have the following fundamental differences from an LMS:

 They support the collaboration needs of virtual courses, thus

emphasizing constructivist learning approaches much more.

 They are learning event-driven instead of course-driven.

 Assessments are usually tracked as separate activities, not as

part of a specific unit of the course.

 Learner performance is tracked using instructor grade book

data, for instance, narrative summaries of learner
accomplishments and needs, test scores, and assignments
completed.

Many VLE products are meant to be sold with other companion LMS
products in order to attain full LMS capability. This applies especially
to tracking functions, which VLEs alone often do not handle. Apart
from mere attendance in VLE learning events, the tracking function
includes standalone assessments and asynchronous content (with
embedded assessments). However, some VLEs also dynamically
track “engagement” on an individual and group basis via parameters
such as:

 Whether the user has participated in chats

 How many questions posed by the instructor the user has

answered

 Interactions with other users within breakout meeting rooms

 Status updates

Some VLE products are also designed to work with eLearning

authoring tools, in order to provide content objects that can be used
asynchronously before or after the live session, or content that is
designed to be displayed by the instructor during the live session. For
example, Adobe Connect® is designed to be used with content

Choosing an LMS.docx page 32 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

developed using Adobe Presenter®, which creates elearning from

PowerPoint presentations.

Most VLEs incorporate videoconferencing capabilities, used typically

to allow the instructor to broadcast a “talking head” of him/ herself to
learners while slides or other media is displayed, in standard webinar
fashion. This capability often includes the ability to display (one at a
time or simultaneously) other talking heads (of other instructors
and/or learners). See 7.16 Video conferencing integration for more
information.

Some VLEs (for example, iCohere Unified Learning ®) include features

that allow them to be multipurposed for holding not just training
sessions but virtual conferences, webinars, and collaborative
meetings.

If you are considering acquiring a VLE, do not underestimate the

retraining it may take for your classroom instructors to become
effective VLE instructors. VLEs have different instructional
constraints and opportunities, and potentially many more technical
distractions than a live classroom. For instance, instructors should
poll students (using features built into the VLE) frequently to gauge
understanding and thereby adjust pacing. Instructors should also
ensure that students know how to use the audio functions to mute
themselves when not speaking to minimize distracting background
noise.

VLEs are currently experiencing an upsurge in popularity. Roche and

Upton (2013) report that 53% of the respondents in their 2013
Elearning! survey are now engaged in virtual events or virtual
classrooms. Shank (2013) reports that 64% of the respondents in their
survey use “Virtual classroom/conferencing tools”.

In addition to the basic ability to screen share media (especially

slides) on the instructor’s computer and stream audio and video of
the instructor(s), VLEs (especially ones that support MOOCs) usually
offer the following features (Poiry and Wilson, 2013):

 Whiteboard

 Chat and Q&A

Choosing an LMS.docx page 33 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Polling

 Hand-raising and instructor feedback

 Passing presentation rights and application sharing

 Breakout rooms

 Social networking backchannels

 Content annotation

 File sharing

See Appendix F. Examples of products for a list of example systems

in this category.

3.6 Mobile learning LMSs

Some vendors provide mobile-optimized versions of their LMS
environments, or systems built from the ground up for mobile. These
systems have interfaces that are optimized for mobile devices and
facilitate delivery of eLearning and documents in mobile formats.
They are usually quite robust in terms of social media functions
related to eLearning such as creating discussion threads and posts,
creating content items, commenting on blogs and journals, etc.

Some statistics are telling as to the need for management and

delivery of mobile learning:

 Smith (2015) reports that 40% of the world’s population has

smartphones, and 30% of smartphone users have taken a class
or accessed instructional content on a phone.

 Shank (2013) reports in a survey of learning technology use that:

 31% use mobile phones/smartphones
 30% use tablets
 17% use mobile web pages

As for popularity of mobile learning, other statistics support

continued growth in this learning technology approach:

Choosing an LMS.docx page 34 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Brandon-Hall (2016) reports that mobile tools are cited as the #2

focus of technology exploration (41% of respondents) in their
survey, just behind social/collaborative tools (43%).

 Vipond and Clarey (2016) report in their 2015 survey of corporate

LMS owners that nearly one-half (46%) said smartphone support
was an “extremely important” LMS feature, and more than one-
half (55%) identified tablet support as “extremely important” for
mLearning.

There is general agreement among eLearning professionals that the

mobile platform is not suitable for delivering standard eLearning, i.e.,
complete eLearning courses created for desktop delivery that have
been reformatted for mobile delivery. This has mostly to do with the
duration of courses; mobile users generally do not have the patience,
proper context, and an uninterrupted time frame (while not at their
desk) to consume training information on small screens for long
periods. Thus, mobile delivery is often cited as being most suitable
for small chunks of learning, sometimes called “microlearning” (see
7.30 Microlearning) and short duration learning-related functions such
as:
 Tests
 Surveys
 Short videos
 Just-in-time performance support, ranging from highly
interactive coaches and decision support modules to static
job aids
 Social media tools that provide access to experts, mentors,
and communities of practice
 Drill exercises (e.g., electronic flash cards)
 Learning-optimized references (e.g. infobases)
 Collaboration platforms for informal learning
 Sharing of files (e.g., photos taken on the mobile device),
usually via integration with a 3rd party service such as
DropBox.
 Grading (for instructors)

One figure that sets a realistic maximum duration for any learning
experience on a mobile phone is 26 minutes. That is the average
length of a commute in the U.S. (Barry, 2016). Commuting is

Choosing an LMS.docx page 35 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

obviously a convenient time used by many learners to consume

(usually audio-based) mobile learning.

Smartphones are particularly well-suited to location-based, contextual

learning because of their built-in GPS capability. Games for learning
can work on the mobile platform, but generally only if they are
simplified. If you are acquiring an LMS for mobile learning, you will
need to consider carefully what kinds of learning experiences will be
delivered, based on what is really appropriate for the mobile platform
(given the above examples).

Supporting mobile learning can be a complicated proposition given

the overwhelming trend of “BYOD” (bring your own device) in the
workplace. Employees want to take learning on their own mobile
device(s), and many companies do not have the resources or
inclination to supply standard devices anyway. Vipond (2016) reports
on research that revealed that 74 percent of organizations already
allow employees to bring their own devices to work, or are at least
planning for this to happen.

The advantages to BYOD (especially to Millenials) are compelling to

most end users and supported by survey research (as reported in
Vipond, 2016):

 Better productivity through being able to work at times that

would normally be “downtime”

 More flexibility through combining work and personal lives

 Better access to work content

 More functionality. Applications on a personal mobile device

may actually provide greater functionality to get work tasks
done better/faster/cheaper.

 More familiarity. Employees may be more comfortable and

skilled with the way to get things done on their personal mobile
device. This can translate to an advantage for managers as well:
reduced help desk costs.

Accounting for the differences in mobile platforms requires the LMS

to detect the device and adapt its delivery configuration accordingly

Choosing an LMS.docx page 36 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

(called “responsive design”). This is not completely new to LMSs, in

that differences between mobile and desktop browsers in the early
2000 required this adaptive content approach. In spite of the many
technical issues for LMSs in this regard, LMSs are quickly rising to
the occasion. Responsive design as part of the standard feature set
for authoring tools has become a major help in filling the gap.

A significant issue with the BYOD paradigm is that users do not want
to use their data plan to download work-related content. This content
can eat up their data plan quickly if it includes media files like high
definition video. To avoid this, content provisioning and tracking data
synchronization features are incorporated in mobile LMSs, in which
users can download content and send assessment data while they are
on a wireless network, and consume the content later while offline
(with tracking data cached until back online).

Despite the cost, enterprise-supplied mobile devices, especially

tablets, are gaining popularity, particularly under term lease
agreements with suppliers (i.e., during the designated training
period). However, enterprise-supplied devices are generally not worth
the cost and maintenance burden (and the burden on the user of
carrying around another device) unless the enterprise takes full
advantage of the control over these devices that ownership of them
affords. Systems that implement this control are called mobile device
management (MDM) or Enterprise Mobility Management (EMM)
systems. MDM/EMM can apply to both BYOD and non-BYOD
environments, but they are obviously quite limited in the case of the
former, in terms of the amount of control users are willing to allow
over their personal devices (for example, auditing and wiping phone
data if a security risk is detected).

MDM/EMM is now being offered in ways that do not require an agent

or profile to be installed on the BYOD device to mandate corporate
mobile policies. For instance, with G Suite’s Basic Mobile
Management option, admins can:

 Enforce a screen lock

 Wipe a corporate account (but not the entire device)

 View, search, and manage their device inventory

Choosing an LMS.docx page 37 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Another approach, called mobile application management (MAM) has

the ability to solve some security and control issues, but it has not
gained much traction in the marketplace, as reported by Vipond, 2016.

MDM/EMM systems can take care of most of the security issues that
would normally need to be handled by a mobile LMS, covering five
categories:

 Content security

 Device security

 Application security

 Transaction security

 Network access security

Either through the mobile LMS or an overarching MDM/EMM system,

these issues need to be addressed in a mobile learning environment.
Application security is probably the most vulnerable of these, and the
one that MDM/EMM systems typically focus on. Application security
issues for learning applications includes:

 Authenticity of the app

 Secure access to content and other enterprise systems

 Storage of content on the local device either for offline use, or

cached through normal application processes

 Forcing users to upgrade apps to fix newly identified

vulnerabilities

 Encryption of data

In choosing a mobile LMS, it is important to distinguish two kinds of

mobile LMSs: “mobile friendly” and “mobile responsive” (the
categories are mutually exclusive). Mobile friendly simply means that
features that would preclude using a mobile device to access the LMS
are stripped out (e.g., navigation drop-down menus, rollovers). No

Choosing an LMS.docx page 38 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Flash animation is used, and usability is more or less maintained,

regardless of the device on which the system is being viewed.

“Mobile responsive” describes an LMS that changes the screen size,

layout, and other interface features due to the constraints of the
device, based on detection. For instance, text and images may
change from a three-column layout to a single-column display. Fonts
are enlarged, and some images may be hidden. The interface may
change dramatically between a desktop view vs a smartphone view.
Mobile responsive is recommended to ensure a more optimal user
experience for mobile users; in fact, offering a mobile app option for
the LMS rather than a web site based on mobile LMS users is
becoming more and more common, since it allows the greatest
control over responsive features.

Udell & Woodill (2015) describe a spectrum of categories for LMSs to

implement mobile learning optimization, as follows:

 Level 0—LMSs not ready for mobile learning

 Level 1—LMSs graphically redesigned for mobile devices

 Level 2—mobile extensions (“plugins”) for existing learning

management systems; the extension only works in conjunction
with a non-mobile LMS

 Level 3—standalone, self-sufficient mobile learning management

systems

 Level 4—innovative mobile learning management systems that

use some of the new affordances of mobile devices, such as
location detection or cloud computing.

An interesting possibility that supports the idea of “spaced learning”

(see Thalheimer, 2007) is the ability to send SMS text messages to
learners containing actual learning content as reinforcement for prior
learning. An LMS that has a built-in SMS engine can also be used to
push learning tasks out to learners, asking them to try applying
something they learned in previous content in their present work
situation. These learning tasks can be quite effective in developing
new work habits. Apps that pop up alert messages with content

Choosing an LMS.docx page 39 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

reminders can also be effective. Some apps like Mobile Coach ® have
AI “bots” (i.e., coaches) with which learners can engage in short
learning conversations.

Eventually, learner context detection (location, speed, etc.) will

advance to the point where appropriate learning objects can be
automatically be pushed to learners at the time of need or availability.
This capability may ultimately not be part of the LMS, but delivered by
personal assistant for learning (PAL) software (possibly in
combination with augmented reality hardware); ADL is currently
pursuing development of PAL prototypes.

See Appendix F. Examples of products for a list of example systems

in this category.

For more information on mobile learning and the ways that LMSs can
support it, see the Mobile Learning section of ADL’s Web site at
http://www.adlnet.gov/mobile-learning.

4. Special features and issues to consider

4.1 Enterprise system integration
In 2011 (Brandon-Hall, 2012a), enterprise system integration was the
most important requirement enterprises have for a new LMS that is to
replace an existing one. In 2015, it was still third (of nine) from the top
of the list, according to a newer report (Brandon-Hall, 2015). This
report outlines the distinction between learning tools integration and
business system integration, where “Learning related tools include
support tools that are often separate but need to work in conjunction
with a Learning Management System, such as assessment builders,
virtual classrooms, or an enterprise social network.”

Business system integration, it goes on to say, “… goes beyond the

traditional employee payroll system, and now includes requirements
to integrate with enterprise resource planning systems, e-mail tools,
sales platforms, marketing platforms, single sign on systems and
HRIS platforms.” (Brandon-Hall, 2015, p. 8).

There is also a strong trend of integration with social platforms,

especially YouTube, Facebook, and Twitter. Often, an LMS vendor will

Choosing an LMS.docx page 40 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

try offering integration with an external service first, then developing

and including the same or similar functionality within its own closed
system architecture. See 7.. Emerging trends in LMSs for details on
many functions that are trending towards being integrated into the
LMS.

For LMSs used in a corporate setting, probably the most important

system to integrate with is the human resources management system
(HRMS); for LMSs used in an academic setting, the key system
integration hurdle is to integrate with student information systems
(SISs), to handle transcripts, manage credits, etc.; in associations
(professional and trade organizations), integration with member
management systems (MMSs) would be important.

The main driver for an LMS to integrate with other learning tools is
essentially to avoid giving learners a reason to leave the LMS
environment for some key part of the learning experience, which
would dilute the “one stop shop” value model and convenience factor
for the LMS. It also generally facilitates control and tracking of the
learning experiences managed by the LMS.

The main driver for business system integration with the LMS is to
avoid data redundancy and version control issues, and automate
migration of data through the enterprise in a seamless process. A
very common system integration requirement is to have the LMS pull
employee profile data from an HR system, so that lists of mandated or
eligible learners do not have to maintained separately in both
systems, and meeting of training qualifications and certifications via
activities learners do in an LMS can easily trigger HR system events
such as meeting compliance requirements and eligibility for
promotions.

Some regulatory agencies are starting to offer system integration with

LMSs so that learners can immediately receive certifications directly
from those agencies, rather than “second hand” through the LMS
owner.

There are many considerations in planning for system integration.

See Brandon-Hall (2015) for a list.

Choosing an LMS.docx page 41 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.2 LMS skins and templates

LMS skins are generally style sheets that globally control the
appearance and format of the LMS interface (differentiated from skins
used in the context of authoring tools, which refers to the ability to
globally apply a look and feel to the content itself). They usually
include banner graphics, logos, color schemes, etc.

Skins can enable local variations on parent LMS interfaces, providing

each organization within the enterprise with its organizational
branding. This can ease the barrier of sharing an LMS across an
enterprise, potentially saving a lot of money. Skins can be determined
in advance, or can be dynamically applied depending on the user’s
demographic information (such as organization they belong to, or the
country in which they reside) in their profile. Skins can include
changing the language of the interface.

Do not confuse the concept of an LMS template with the concept of

templates in authoring tools, where a starting point for building a
screen or content object is predetermined, gaining production
efficiencies. The concept of LMS templates (some other term may be
used for this concept within the LMS product) refers to any saved set
of parameters that can quickly be applied to any content or functional
object. These parameters can govern such things as workflows,
course configuration, learning tracks, and permissions. For example,
templates can be sets of permissions or roles that can be repeatedly
assigned to individuals who are members of certain groups or
functioning in certain roles, or whole groups. Thus, a whole set of
permissions can be applied all at once, to more than one user.
Templates can also refer to screen templates for building
assessments that can be created and used within an LMS.

4.3 Programming language and platform dependencies

Some LMSs are built using programming languages such as Java that
require software runtime interpreters to be installed on client
computers. This can be a problem for users who do not have the
minimum required version of the interpreter, perhaps due to
enterprise IT policy not approving it or restricting users from doing
their own software updates. Even if they are allowed to install the
runtime interpreter, it is not user-friendly to require them to do so in

Choosing an LMS.docx page 42 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

order to access the LMS, and it complicates IT support for the LMS.
Some LMSs in recent years whose codebase was Java are now
switching to other codebases for these reasons.

Other LMSs are built on database platforms such as Oracle ® or

content management systems such as SharePoint ®. These require
purchase (and maintaining a current license) of the underlying
platform, which can be costly, unless you already have an enterprise
license for that platform.

It is important to determine what these programming language and

platform dependency requirements are for an LMS you are
considering purchasing, since it can have a substantial impact on
cost and deployability. This also relates to customization of the
product, since the programming language may be one that your
programmers are not familiar with, making customization difficult.

Some vendors (e.g., ShareKnowledge®) offer business models

whereby they customize enterprise platforms (e.g., Microsoft
SharePoint, WordPress) to look, act, and feel like LMSs. This can be
an attractive option, if you want to host your LMS behind your
firewall. The advantages to this model for acquiring an LMS are:

 Lower cost if the enterprise already owns a license for the base
platform.

 Leverages the existing IT infrastructure that supports the

enterprise’s base platform.

 Lower training costs since users presumably already are

familiar with how to use the base platform.

 Much of the content is probably already in the system, therefore

there is no need to transfer it from one system to another.

 Leverages existing security, content management, user profiles,

and other business rules and workflows already in place for the
base platform.

 The enterprise will be able to take advantage of new features

added to the base platform that often have relevance to learning
or LMS functioning (especially social media).

Choosing an LMS.docx page 43 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

It is important in these cases to ensure that you are truly able to

leverage your base platform technical support staff and
administrators. If the product is so heavily customized that it requires
additional administrators and support staff, the cost advantages
could be seriously diluted.

4.4 Pricing models

As with many complex, large software systems, vendors will have
their own particular pricing model that they feel positions them best
in the marketplace and suits their needs; this makes it difficult to
compare prices between vendors. However, there are certain basic
categories of pricing models, as follows:

 Perpetual license –this model involves paying a one-time fee

that allows you to “own” the software. This may be a cost saving
model for you depending on what the fee is and how much you
plan to use the system. This model assumes that you will either
host the system behind your firewall or on a public cloud service
(see 4.10 Hosting options for more information), since it is yours
to do whatever you want with. This model is increasingly rare.

 Subscription-based license – this model is by far the most

standard (for most large software systems of any kind, in fact).
You are charged a (usually monthly) fee based on number of
users, usage, storage, etc. (see list below). This usually
presumes that the system is hosted behind the vendor’s firewall
(see 4.10 Hosting options for more information), and can thus be
monitored by the vendor for usage patterns, which determine
the price you are charged. Under this type of license, there are a
number of usage patterns that can be used to determine price,
as follows:

o Seat-based – this model uses the number of employees in

the enterprise, or possibly the number of employees who
will ever need training in the enterprise, as a basis for
pricing. These “seats” are a maximum number of people
who may end up logging in to the system over its life cycle.
There are usually tiers of seats (for example, up to 10,000
vs up to 20,000 users). Do not confuse seats as a pricing
model with system capacity seats. The latter is the number

Choosing an LMS.docx page 44 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

of concurrent users that can safely use the system without

overburdening it.

Seat-based pricing can run into problems with “extranet”

users. If partners, customers, and others outside of the
enterprise (i.e., other than employees) need to use the
system, accounting for their numbers accurately may be
complicated, and may need to be based on unreliable
estimates.

o Usage-based – this model is based on the number of

learners who actually use the system. It is not based on the
potential number of users who could use the system, as in
seat-based pricing; it is based on either the number who
actually do use it to take training (sometimes called “pay
per active user” model) or indicate by registrations that
they intend to use it (“pay per user” model). The time
period needs to be accounted for in the price as well; 1000
users registering for a course and only using it for one day
can be a different pricing scenario than those 1000 users
using that course for 6 months.

You should ask the vendor to define what constitutes

“use” of the system in the case of this model. It could be
defined as merely registering for an entire curriculum,
starting a single course within a curriculum, accessing
resources such as videos, etc.

This model is particularly attractive in the case of

anticipated LMS usage surges, due to such events as new
product releases and seasonal cycles. In these cases,
where there may be little usage of the system except in
certain short periods, paying for a baseline of seats may be
less economical than paying per use, for time used.

Usage-based models can make a hosted solution (see 4.10

Hosting options) especially attractive, since your
organization does not need to permanently maintain a full
complement of server, bandwidth, and support resources
to handle the highest load times. That is the responsibility

Choosing an LMS.docx page 45 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

of the vendor, and you will only be charged for the

(potentially) short time that usage peaks.

o Storage-based – this model is based on the size of the files

that are stored in the content repository section of the
LMS; 10 Gb of files will cost more than 5 Gb of files, no
matter what the usage patterns of either.

o Flat fee – for this model, it doesn’t matter how many users
you have or how much bandwidth or file space you need.
The fee is the same in all cases. This is similar to the
perpetual license already mentioned, except that here you
pay a periodic subscription fee rather than a one-time fee,
and it is vendor hosted. An example of this type of LMS is
Topyx®.

o Free – This only applies to open source LMSs, where you

download the free software and host it yourself (otherwise
it is not free since you are paying for hosting). Of course,
there are many ancillary costs to maintain and customize
such a system, so it is not, technically speaking, “free”.
For more information on open source LMSs, see 4.6 Open-
source or freeware solutions

On top of these pricing models, some vendors offer additional tiers of

features that can be unlocked for a fee. For instance, you can have
streaming video or integration with a web conferencing tool added to
your baseline product. There are also tiers of support that you can
buy above the basic level. Tiers of features or support could save you
money, since you are not paying for what you don’t intend to use.

4.5 Return on investment (ROI)

If you are currently switching from ILT to eLearning as your standard
delivery mode of training, your ROI calculation for acquiring an LMS
to support this switch should take into account cost items such as:

 Number of people you train

 Number of course hours per year these people will need to be in

training

Choosing an LMS.docx page 46 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Number of courses

 Cost to develop eLearning courses (if you do not already own

content)

 Cost of travel and accommodation for people to travel to ILT

courses

 Per hour compensation rates of employees for time in training

(for both ILT and eLearning)

 The annual cost of the LMS, given the most likely pricing model
that you will adopt.

 Cost of maintaining and administering the LMS, or cost of

hosting service. There can be many different elements included
in this cost – you may need a brainstorming session among
managers to create an accurate list. Elements could include:

o Engineering & technical support

o Security infrastructure setup and maintenance

o LMS version releases and administration thereof

o COTS SW - updates, integration, testing, troubleshooting,

etc.

o Database administration

o Content delivery network administration

o Configuration management

o Maintenance agreements on all COTS hardware & software

o Help Desk

o Courseware testing

o Acquisition support for customization and enhancements

Choosing an LMS.docx page 47 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Perhaps the most important factor in determining ROI is how much of

a difference the training delivered by the LMS will make in the
performance of individual employees (Kirpatrick Level 3) and the
overall performance of the organization (Kirpatrick Level 4). These are
notoriously difficult factors to assess and quantify, whether an LMS is
being used or not.

One approach to this problem is simply to accept the “Rule of Thirds”

(Fletcher and Foster, 2002), as follows: “Data drawn from many
evaluations of technology-based education and training indicate
overall that these systems can reduce costs by one-third and that
they can additionally either reduce the time to achieve instructional
objectives by one-third or increase achievement (holding time
constant) by one-third.” (p. ES-1)

Another data point relates to the fact that, with an LMS, students do
not need to travel to the training site and have housing provided for
them. The cost savings from this are considerable. In 2014, the U.S.
Army calculated that it saved $84,000,000 by using eLearning, as
opposed to training that required students to travel to the classroom
facility (U.S. Army, 2015).

Cohn and Fletcher (2010) present a method of calculating ROI (for a

training program, not an LMS specifically) that assesses the long-
term impact of potential training capabilities, combined with a wider
set of metrics that account for research and development
investments, equipment purchases, equipment maintenance costs,
learner and instructor time, follow-on sustainment, and on the job
training. This ROI method can be used to compare specific training
applications using eLearning vs classroom training methods. No
actual comparisons were done in this study, but the evidence from
other research (cited in this document) implies that eLearning should
show a substantially better ROI than classroom training.

4.6 Open-source or freeware solutions

Open source options are attractive due to the absence of any
licensing cost. However, it is important to be aware of the pros and
cons of acquiring an open source solution, as the cost could, over the
life of the system, equal or exceed a commercial system. It’s easy to
be over-enamored of the free license aspect and ignore the required

Choosing an LMS.docx page 48 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

(possibly extensive) customization and support that may be

necessary.

It is also easy to overlook the potential advantage of open source

systems in that the product can be completely tailored to the
particular requirements of the organization. If managed properly, this
advantage can make an open source solution cheaper, not just
because the license is free, but because the development and
customization efforts can be focused solely on the needs of the
organization and nothing more. Contrast this with a commercial
product with lots of features that your organization may not need (but
you are paying for them nonetheless). The business model for a
standard commercial system is to build to the widest set of possible
requirements to attract the widest client base. Your organization may
not have all or even most of these requirements.

All of the above being said, acquiring an open source LMS usually
does save money. For instance, the manager of a large U.S.
government agency’s eLearning initiative reported to the authors that
switching to an open source enterprise LMS is costing them 60% of
the ongoing costs of the commercial system they had been using.

Open source systems are indicated in the lists of systems in 3.

Categories of systems to deliver and manage learning. Descriptions
of popular open source systems can be found at
http://barrysampson.com/2009/04/open-source-lms-10-alternatives-to-
moodle/

On October 16, 2009, U.S. DoD issued new guidance on open source
software (see http://dodcio.defense.gov/Portals/0/Documents/FOSS/2009OSS.pdf). The
guidance establishes open source software as having equal weight as
proprietary software during acquisition evaluations. It is a break from
the past, when open source software was deprecated for use in DoD
due to security and quality concerns.

The benefits of open source software are described in this guidance

document as follows (open source is referred to as OSS):

 The continuous and broad peer-review enabled by publicly

available source code supports software reliability and security
efforts through the identification and elimination of defects that

Choosing an LMS.docx page 49 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

might otherwise go unrecognized by a more limited core

development team.

 The unrestricted ability to modify software source code enables

the Department to respond more rapidly to changing situations,
missions, and future threats.

 Reliance on a particular software developer or vendor due to

proprietary restrictions may be reduced by the use of OSS,
which can be operated and maintained by multiple vendors, thus
reducing barriers to entry and exit.

 Since OSS typically does not have a per-seat licensing cost, it

can provide a cost advantage in situations where many copies
of the software may be required, and can mitigate risk of cost
growth due to licensing in situations where the total number of
users may not be known in advance.

 Open source licenses do not restrict who can use the software
or the fields of endeavor in which the software can be used.
Therefore, OSS provides a net-centric licensing model that
enables rapid provisioning of both known and unanticipated
users.

 By sharing the responsibility for maintenance of OSS with other

users, the Department can benefit by reducing the total cost of
ownership for software particularly compared with software for
which the Department has sole responsibility for maintenance
(e.g., GOTS).

 OSS is particularly suitable for rapid prototyping and

experimentation, where the ability to “test drive” the software
with minimal costs and administrative delays can be important.

(Department of Defense Office of the CIO , 2009)

Furthermore, on September 8, 2016, U.S. Office of Management and

Budget (OMB) issued new guidance on open source software (see
https://sourcecode.cio.gov/). This memorandum is aimed at ensuring
that new custom-developed Federal source code be made broadly
available for reuse across the Federal Government. As part of this

Choosing an LMS.docx page 50 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

policy, it includes stipulations that require agencies, when

commissioning new custom software, to release at least 20 percent of
new custom-developed code to the public as OSS for three years, and
collect additional data concerning new custom software to inform
metrics to gauge the performance of this policy. (Scott, T. and Rung,
A., 2016). They cite the benefits as follows:

“While the benefits of enhanced Federal custom-developed code

reuse are significant, additional benefits can accrue when source
code is also made available to the public as OSS. Making source code
available as OSS can enable continual improvement of Federal
custom-developed code projects as a result of a broader user
community implementing the code for its own purposes and
publishing improvements. This collaborative atmosphere can make it
easier to conduct software peer review and security testing, to reuse
existing solutions, and to share technical knowledge. Furthermore,
vendors participating in or competing for future maintenance or
enhancement can do so with full knowledge of the underlying source
code. A number of private sector companies have already shifted
some of their software development projects to an OSS model, in
which the source code of the software is made broadly available to
the public for inspection, improvement, and reuse.” (Scott, T. and
Rung, A., 2016, p. 2).

What is important to understand about open source software is the

relationship it behooves you to build with the open source community
that has arisen for the open source product you are acquiring.
Staying in touch with the community in order to be able to discover
and use already developed modules of functionality that you need
(that are not part of the product baseline) can decrease your
customization costs enormously. Open source communities often
remind you that deploying open source means you are a responsible
member of their community. There is an expectation that you
contribute, as well as receive code, training, and documentation from
the community. The cost of staying active in the community and both
researching and acquiring as well as sharing your products and
solutions must be factored into the level of effort for acquiring an
open source tool. Open source LMSs are often backed by non-profit
organizations, such as the.LRN consortium, Claroline Consortium,
and the Sakai Foundation.

Choosing an LMS.docx page 51 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

It is also important to evaluate the strength and size of the open

source community for the open source product you are acquiring, as
well as the longevity of the product. This can mitigate obvious
concerns that major sponsors of open source software can stop
development at any time, or that communities can atrophy. Another
possible concern is that a tool can grow so quickly in its popularity
that documentation takes a back seat to development and has not
caught up to the current release of the software; especially in the
case of open source software, where you have no vendor who is
obligated to support you, a lack of adequate documentation can make
a product difficult to install, use, maintain, and troubleshoot.

Finally, the baseline versions of some open source products are

usually very basic; some level of customization is often needed to
make the software not only meet your special requirements but also
meet a modest level of universally recognized functionality for the
type of product. It may be risky to assume that an open source
product will be usable straight out of the box. If you have no
development resources ready and willing to augment the product’s
functionality right after you acquire it, you may not be able to use it
for some time. Some companies (such as Lambda Solutions) have
built their business model on selling customization and support
services for open source LMSs such as Moodle or Totara.

Due to the above concerns, despite their economic advantages and

often robust functionality, open source systems have not yet taken up
a substantial share of the LMS market.

Freeware may or may not also be open source. Freeware may have
restrictions on copying, distributing, and making derivative works of
it, where open source software does not. And freeware does not
necessarily make source code available. Freeware may be restricted
to personal use, non-profit use, non-commercial use, etc. Freeware
that is not open source is a risky investment, since you cannot easily
customize it.

There may be special restrictions on use of open source or freeware

within your organization. For U.S. DoD, see
http://dodcio.defense.gov/OpenSourceSoftwareFAQ.aspx

Choosing an LMS.docx page 52 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.7 Government off-the-shelf (GOTS) solutions

This section only applies to government entities. GOTS software can
be created either by the technical staff of a government agency or by
a commercial vendor (usually the latter). GOTS systems usually have
the following characteristics:

 The government has direct control over most aspects of the

product, including the source code.

 The vendor or creator has given a license to the government

entity who paid for it to freely use and share it within the
government. The license does not permit the government to give
or sell it to outside entities. In most cases, however, the
software can be sold by the vendor to an outside entity.

Many of the same considerations described in 4.6 Open-source or

freeware solutions apply to GOTS solutions as well.

A popular model for GOTS installations is to have regular meetings

where representatives from organizations that use the system
throughout government discuss new requirements and possible new
features. At these meetings, agreements are made between the
representatives as to sharing the cost for adding these features
(which, after they are developed, are available to all users).

The original vendor/developer is usually the preferred entity for doing

the customizations, since their developers were directly involved in
creating it and have the most knowledge about working with the code
base. This pre-existing experience and expertise can substantially
reduce the cost of further development and customization. A GOTS
license does not stipulate that the original vendor has to do the
customization, however.

4.8 Offline content provisioning and player capability

This concept involves allowing LMS content to be played and LMS
functions to be performed in environments where there is no,
intermittent, or limited bandwidth or connectivity to the LMS. It often
refers to mobile devices, where user network access is less stable
and reliable (and can be expensive). Content that is to be consumed

Choosing an LMS.docx page 53 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

must be provisioned to the device at a time and place when there is

stable and cheap connectivity, such as on a free wireless network (as
opposed to cellular data network), then used offline. The local device
must have a player capability (a web browser might suffice) in order
to play the content.

Some LMSs are appearing that have extensive pick lists of specific
LMS functions and content from which the user indicates what they
would like downloaded in advance and stored locally vs on the server
so that they can successfully work offline. They can thus
micromanage the connectivity requirements and bandwidth load
based on their specific situation.

Disconnected use scenarios can involve getting the content from a

CD or from a download at a time and place where bandwidth and
connectivity is available. He or she runs the course(s) on their local
system with an offline player, which could be on their own computer
or a shared community computer. Then, he or she uploads
performance data to the LMS at a later time when there is sufficient
bandwidth/ connectivity.

Sometimes offline players are needed when there is a diversity of

content, and all of it cannot be delivered through the LMS due to file
format incompatibilities between the content and the LMS. In this
case, the LMS can be used to author and deliver assessments and
store tracking data only. The delivery capability can be handled
through an offline player (possibly provided by another vendor) that
can display the content. Learners can take courses using the offline
player and then log in to the LMS to take the assessment and have
their performance tracked.

There are special concerns for SCORM content delivered from a

SCORM LMS, since SCORM normally requires “session data” to be
written as the learner progresses through a course within a session.
This is not such a problem for SCORM 1.2, where data can be
packaged and sent after the user has consumed the content, but for
SCORM 2004 with sequencing, the LMS needs to know in real time
what the user is doing so that it can sequence next events properly.
Workarounds can be created, but SCORM does not generally work
very well with offline players. Due to a large demand for them, they do
exist in the marketplace, however. Vipond and Clarey (2016) report

Choosing an LMS.docx page 54 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

that 70% of their LMS survey respondents said an offline SCORM

player is “extremely important” or “very important”.

The following are examples of offline player scenarios:

 If learners have intermittent connectivity (for example, Navy

personnel on ships who have connectivity at their base but not
on their ship):

1. While they have connectivity (ideally through a free

wireless network), learners log in to the LMS.

2. Learners check out (i.e., download, which registers their

intent to consume) required/desired course(s). This locks
the course(s) down so learners can only take them in
offline mode (otherwise there could be data conflicts with
taking the same course in both online and offline mode).

3. Learners take the course(s) in offline mode through their

offline player (often a mobile web browser) or pseudo LMS.

4. When learners return to a location that has connectivity,

they log in to the LMS

5. The courses are automatically checked back in (or, in some

cases, the user needs to manually check them in).

6. The offline player (or pseudo LMS) updates all of the

learner’s records in the LMS to reflect offline courses
taken, test scores, etc.

 If learners have no connectivity (for example, learners don’t

have their own computer and/or are in a remote area with no
internet connection):

1. Someone in the learner’s organization (for example, an

administrator at an HQ site) connects to the LMS.

2. The administrator checks out the learner database for a

selected group of learners, along with the catalog of
courses needed for those learners. This database contains

Choosing an LMS.docx page 55 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

all of the records of learner performance, etc. He or she

downloads this as a .zip file and puts it on a USB drive.

3. This drive is sent to the learner location.

4. At the learner location, the USB drive is loaded on a

community computer(s) that everyone shares, or
individuals’ computers.

5. Each learner takes their required or desired course(s)

using the offline player. As courses are taken, data is
written to the learner database about learner performance,
courses completed, etc.

6. After everyone has taken their courses, a .zip file is created

from the learner database on the USB drive, using a utility.
If individual learners have taken courses on their personal
computers, the files need to be collected onto one
computer, and this utility consolidates them into a
single .zip file.

7. This .zip file is sent back to the site that has connectivity.

8. The administrator at this site uploads the learner database

to the LMS using a web service.

Some LMSs are starting to use a scenario where content is

downloaded and stored on a local device (usually a tablet) as a
default rather than delivered in real time at the point of use. Content is
synchronized and updated at regular intervals whenever there is
connectivity (via a high speed wireless network, for a mobile device).
See 3.6 Mobile learning for more information.

These and other offline content scenarios can be handled by ADL’s

xAPI standard (with or without an LMS). See 4.15.8 ADL Total
Learning Architecture (TLA).

4.9 Security considerations for LMSs

Like any other enterprise system, LMSs must meet the security needs
of the organization. This is especially true in the current era, where
LMS functionality is largely delivered via the Internet, not enterprise

Choosing an LMS.docx page 56 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

intranets or extranets (the driver for this migration is mostly to allow

greater access to learning).

For commercial installations, LMS security amounts to:

 Vulnerability to malicious penetration attacks and malware,

resulting in loss, theft, or corruption of data. This security
concern is no different from any other enterprise system, and
the LMS would presumably be no more vulnerable than any
other IT system.

 Protecting against unauthorized login. This is primarily not so

much a function of the LMS, whose login functionality relies on
universal web standards, but rather the placement of the system
within the corporate intranet environment and the inherent
security features of that parent architecture. Commercial entities
are of course concerned about other organizations gaining
competitive advantage by seeing the training of competing
companies, and government has obvious security concerns, so
access to the system is a primary concern.

 Privacy policies and access to Personally Identifiable

Information (PII) may be an issue depending on how public
access to your LMS is and what kind of information you store on
it about your users. EU Internet privacy rules, Canada’s Freedom
of Information and Protection of Privacy Act, Family Educational
Rights and Privacy Act (FERPA), and the US Patriot Act may be
a consideration. Maintaining PII standards usually involves
some level of encryption of the data stored in the LMS.

 Locking users out of capabilities that are not permitted in their

user profile, in other words, keeping users from doing particular
things, once in the system, that they are not authorized to do. All
LMSs include levels of permission based on roles, but beyond
this, they vary widely in terms of the types and number of roles
and permissions that can be assigned. These permissions need
to be segmented so that they map to the levels and specific
kinds of permission that your organization requires. The
question here is, if the system forces you to use a
permission/roles assignment template, how applicable is it to
your environment, and can templates be tailored to meet your

Choosing an LMS.docx page 57 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

needs? Is there an override that permits assignment of

individual permissions on a function-by-function basis?

For DoD organizations, there are specific considerations relating to

the possible harmful effects to national security and individuals’ life
and limb due to unauthorized access to the system and particular
courses that may be classified, etc. There are a number of issues that
need to be considered in this regard. See the Appendix E: Security
Considerations for DoD LMSs for a detailed list.

It is important to find out what programming language and third party

OEM components were used to build the LMSs you are considering
acquiring. There are innate security considerations for some
programming languages.

A number of security concerns come into play for hosted solutions,

since in that case both your content and the LMS system reside
outside of your firewall. These concerns generally are the same for
cloud computing, which has become indispensable and ubiquitous
throughout all aspects of learning technology, playing a vital role in
providing the services people and employees use in their everyday
life. But as cloud computing has risen in use and mission-critical
importance, concerns related to privacy, data security, and even
sovereignty have emerged. One partial solution is to use a “private
cloud” with VPN access for those outside of the enterprise network.
This however, may not work in an environment where public access
to your LMS is required. Custom-designed hybrid cloud solutions are
becoming more and more common to meet specialized security needs
that a standard cloud cannot.

4.10 Hosting options

There are three options for hosting most enterprise learning systems,
including an LMS:

 Behind your firewall

 Vendor-hosted

 Public cloud-hosted

Choosing an LMS.docx page 58 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Most LMS vendors offer the first two options; a few are now offering
the last. A vendor-hosted LMS is installed and managed on the
vendor’s server by their staff, rather than behind your enterprise
firewall by your staff (the “behind your firewall” option). Public cloud-
hosted solutions refer to hosting the LMS not behind your own
enterprise firewall but on a public cloud service such as Amazon Web
Services (AWS). This could be arranged and managed by your staff,
or the vendor’s. In some cases, vendors do not host on their servers
but only do so on a public cloud service; in this case, “vendor-
hosted” means effectively that also that it is public cloud-hosted.

If you (not the vendor) use a public cloud hosting arrangement for
your LMS, you still have full configuration and management control
over the LMS; the only difference is that the software is hosted on
rented server infrastructure outside of your firewall. However, public
cloud hosting often requires a different approach than either of the
other two options because the server configuration options are
limited; they are controlled by the cloud service vendor. Often the
LMS vendor must make alterations to their LMS to conform to the
cloud infrastructure requirements.

The advantages of your using a public cloud hosted LMS solution in

this way are the same as with any public cloud hosted system; you do
not have to acquire and maintain the server infrastructure yourself
(which could be significant for a large LMS installation), and there is
less load on your network. However, since it is not hosted by the
vendor, your staff will still need to do updates and maintenance on
the system.

The fact of being hosted on a server outside of your firewall can raise
security concerns. These are gradually being eased. For instance,
DoD now allows Amazon Web Services hosting for some DoD
systems. On the flip side, it may be a plus that it is not behind your
firewall, if you need users outside of your enterprise network to be
able to access the LMS, and you don’t want to worry about security
breaches by outsiders coming into your network.

Vendor-hosted solutions are often termed “SaaS” (software as a

service) or “cloud” solutions, although this use of terminology can be
confusing. SaaS or cloud can be used to refer to a disaggregated,
Internet-based collection of software services or components that

Choosing an LMS.docx page 59 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

make up an entire system such as an LMS. In practice, these services

are almost always hosted on the vendor’s server, but they could be
installed within your intranet as a private cloud or custom-designed
hybrid cloud (see section 4.9 Security considerations for LMSs).

Some of the advantages of a vendor-hosted platform are:

 Eliminates the cost of hardware and network infrastructure

needed to support a local installation of the system

 Lowers your staff costs for administration and maintenance

 Puts less bandwidth load on the corporate network

 Content and feature updates can be accomplished without

intervention by your staff

 Guarantees that system upgrades and patches are applied on a

timely basis; most vendors upgrade their hosted installations on
a monthly basis. Installation of updates on your server can lag
significantly behind the vendor making them available, for a host
of reasons.

 Having the vendor take responsibility for upgrades and patches

avoids the headaches of reestablishing your integrations, etc.

 Enables faster implementation. This can be dramatic, for

instance, 3 weeks for a vendor-hosted solution vs 6 months for a
behind-the-firewall solution.

 Requires little or no internal technical support or development

 Provides incentives and guarantees for maintaining uptime (via

financial penalties assessed against vendor)

 Potentially better bandwidth to remote locations, depending on

the vendor’s server architecture and network (e.g., “edge
servers” that are positioned at network nodes that are closer to
remote areas)

 Facilitates data center compliance (esp. in regards to data

centers in foreign countries) since this is handled by the vendor

Choosing an LMS.docx page 60 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Scales more easily to account for temporary surges in usage

(due to new product releases, seasonal events, etc.), due to the
typical centralized system architecture usually implemented by
hosting vendors, with loads dynamically shared and balanced
across customer implementations.

 By virtue of the vendor taking responsibility for scaling, it

eliminates the need for you to commit to purchasing and
maintaining additional servers and bandwidth that may be
unnecessary to support normal load during non-surge times.

 Is often associated with a usage-based pricing model (see 4.4

Pricing models), which may be more economical

 Contractually, it can be easier to switch to another vendor or

end a vendor relationship

 Eliminates problem of version differences of software platforms

that the LMS may be dependent on. For instance, a situation
where your company has not upgraded their Oracle license but
your LMS vendor’s new release depends on an upgraded
version of it to run.

One of the main disadvantages of a vendor-hosted solution is that it

restricts opportunities and scope for local customization (although
this can be mitigated to some degree with skins. See 4.2 LMS skins
and templates). Also, a vendor-hosted solution may not provide the
level of security required by your organization, although vendor-
hosted solutions are increasingly more secure.

The security issue relates not just to unauthorized access, but also
the fact that you may be placing trade secrets and other intellectual
property in your content outside of your firewall on the vendor’s
server, outside of your control. If your organization’s policy prohibits
this, a vendor-hosted solution will not be right for you. And a vendor-
hosted solution is summarily ruled out if there is classified data
stored in the content. See section 4.9 Security considerations for
LMSs for more information on security issues.

Finally, for government entities, a vendor-hosted solution may not be

an option since government rules tend to mandate outright ownership

Choosing an LMS.docx page 61 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

and control of systems, rather than an arrangement like a vendor-

hosted solution that resembles leasing.

Most vendor-hosted solution scenarios involve a single instance of

the vendor’s software that is engineered to support multiple
customers, rather than establishing a separate instance of the
software for each customer. This enables efficiencies for the vendor
whereby they can apply patches and version upgrades for many
customers at the same time. This lowers the operational LOE for the
vendor and allows them to focus more on developing their product.
Vendor-hosted systems are vendor-maintained and managed with
minimal intervention required by the customer, so much of the
headache of deployment planning relating to upgrades of the
software can be avoided.

Vendors who offer hosted solutions commit themselves to providing

a robust hosting and networking infrastructure with uninterrupted
access 24 /7 basis from any location. The system that they host must
be scalable and have redundant backup and security. These are items
for due diligence verification during the acquisition process, if you
decide to buy a vendor-hosted solution. Guarantees of average
percentage of uptime are often written into the LMS service-level
contract. You may want to independently verify uptime using a Web
monitoring service. These services monitor access from multiple
global endpoints. If an issue arises, your mobile phone is texted.
Some monitoring services are quite sophisticated. They can actually
periodically read data-driven Web-page elements to validate site
availability in addition to the back-end functionality.

Vendor-hosted solutions are generally more expensive (roughly 20%)

because they require the vendor to assume responsibility for
maintenance and administration instead of the customer.

Hosted solutions are becoming more popular, as evidenced by a

survey of corporate LMSs (Vipond and Clary, 2016). This survey
determined that more than one-half (57%) of organizations in 2016 use
hosted LMSs compared with 50% in 2013. This report also said that
one-third (31%) of organizations still locate their LMS on premises,
maintained by their own IT staff. The report concludes that this is
evidence of the continued dominance of hosted LMSs over on-
premises LMSs for the last three to five years. Other sources such as

Choosing an LMS.docx page 62 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

UpsideLMS (2016) put the proportion of hosted solutions even higher,

at 87%.

You might want to use a “try before you buy” approach by using a
vendor-hosted solution for a while before you decide to buy the
system. Also, consider a vendor-hosted solution that is metered (pay-
for-use price) rather than flat license for a maximum number of users.

When considering a vendor-hosted solution, ensure that your

organization isn’t already licensing ancillary software that is needed
to run your LMS, such as an Oracle database. The vendor-hosted
solution provider will pass through their cost for this license, which
duplicates your existing (usually significant) investment. This may
make a hosted solution not cost efficient for you.

On 10/13/11, Pearson Inc. announced a free vendor-hosted LMS for

higher education called “OpenClass”. In order to support such a free
service, Pearson hoped to be able to sell more of their content (that is
optimized for the OpenClass delivery ecosystem). However, this
model of free vendor-hosted solutions (similar to other cloud-based
software business models like Google) does not seem to be viable,
judging by the fact that Pearson announced that they will remove the
system on 1/1/2018.

You may want to consider outside hosting of particular types of

content, especially video. For instance, YouTube ® and Vimeo® allow
you to upload videos, which you can then drop code into an LMS to
launch. SoundCloud® can do the same thing for audio. Note that some
corporate networks do not allow employees to access these sites,
however. Also, even if you do not use these sites, there may be
considerations regarding the video player, such as level of
permissions for user groups, level of user authentication, and
security of delivery of the video stream. This applies particularly with
employee-generated video sharing environments. If there are
concerns about privacy, security, and ownership of videos, video
streaming is an important requirement; it prevents videos from being
downloaded into the viewer’s computer.

Be careful about free cloud-based LMS-like service providers such as

Facebook®. Read the terms of service carefully. There will
undoubtedly be advertising, and you will need to determine how

Choosing an LMS.docx page 63 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

much is too much. Hidden features could include the provider selling
your user lists to headhunters or spammers. Content may transfer
ownership to the provider once uploaded. If you are charging your
users for access, then you will probably need a commercial license.

Note: Vendor-hosted solutions are sometimes called “ASP”

(application service provider) solutions. Do not confuse ASP with
Active Server Pages, a web programming script.

4.11 Multiple LMS providers

Given the high cost of LMSs, and their claim to be “one-stop shops”
and “one size fits all” vehicles for enterprise learning needs, it may
seem strange that some organizations have more than one LMS
product from more than one provider. In some cases this is simply
due to a long migration period from one product/vendor to another,
with subsequent overlap in ownership, or some user group refusing
to let go of their familiar LMS (with funding to support it) while it has
been replaced for most other users. However, this is more and more
indicative of a “learning ecosystem” approach, where the
organization acquires multiple specialized LMSs that act in concert to
provide particular needed capabilities that a single “one size fits all”
LMS (which spreads its capabilities thinly thus not very robustly)
cannot provide.

Vipond and Clary (2016) report that even though most organizations
(62%) rely upon a single LMS provider, one-fifth (20%) use two
providers. This has not really changed since their 2013 survey,
however, they conclude that, “The multiplicity of LMS providers is
accelerating, and the marketplace is becoming more fluid as LMS
providers operate across more than one market. This makes things
more confusing for organizations that are seeking to upgrade their
core learning management capabilities and survive the sometimes
long period of transition from an older legacy LMS to a feature-rich
LMS now entering the marketplace. Having numerous different LMS
providers also impacts customer loyalty and satisfaction.” (p.3)

4.12 The cross domain issue

For some time, browsers have incorporated a security feature that
prohibits a server with which it is communicating to connect directly

Choosing an LMS.docx page 64 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

to a server on another domain. When users point their browser to a

server on a particular domain, there is a presumption of trust, and
explicit choice to pull in content based on that trust. If that server
unilaterally and unbeknownst to the user gets content (especially
client-side scripts) from another server on another domain (that is not
necessarily trusted), a hacker who has co-opted the second server
can send harmful code to users by passing it through the primary
trusted server. See http://en.wikipedia.org/wiki/Cross-site_scripting
for more details on this issue.

Barring using a workaround like those listed below, the cross domain
issue requires you to have your content stored on the same domain
(i.e., server, usually) as the LMS. In other words, if the LMS is at
www.myLMS.com, the content cannot be at www.myContent.com; the
content must also come from www.myLMS.com.

This problem comes up most often in LMS implementations where

there is a separate content repository server at another location from
the LMS; this content repository server might be another enterprise
server, or it could be a repository of commercial content behind
another commercial entity’s firewall. The cross domain security
feature prohibits accessing the content that is on a separate domain
from the LMS.

Even if it were not a problem for the user to be able to access and
launch the content directly from its content repository on the other
domain (perhaps through a separate LMS), the problem remains of
communicating tracking information (e.g., course completion status,
assessment scores, etc.) to the initial LMS.

There are several workarounds to the cross domain issue:

 Allowing the primary server to serve as a proxy to the server on

the other domain
(see http://developer.yahoo.com/javascript/howto-proxy.html).

 JSONP (see http://en.wikipedia.org/wiki/JSONP#JSONP).

 Cookie security (see http://en.wikipedia.org/wiki/Cross-

site_scripting#Mitigation).

Choosing an LMS.docx page 65 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.13 Special requirements for U.S. DoD

The DoD Information Assurance Certification and Accreditation
Process (DIACAP) is the DoD process to ensure that risk management
is applied on information systems. It certifies and accredits a DoD
information system to maintain the proper information assurance (IA)
posture throughout the system's life cycle. If you are acquiring an
LMS for a U.S. DoD organization, it is important that you check the
DIACAP certification status of any LMS you seek to acquire. Also,
DIACAP may require you to have your LMS hosted at the Defense
Information Systems Agency (DISA) facility, not at your facility (and
not hosted by the vendor, either).

You may be subject to Service-specific requirements. These

requirements speak to the “fit” of the system to the enterprise
architecture of the organization (in this case DoD). These cover
requirements such as:

 Security

 IT environment

 Specific use case testing

 Training gap/training needs analysis capability

Each Service often has their own training records system that the
LMS needs to integrate with. For instance, the Navy often requires the
LMS to integrate with NTMPS (Navy Training Management and
Planning System) for personnel information and training records.

One requirement that is fairly consistent across the Services is that

the LMS must interface with DEERS (Defense Enrollment Eligibility
Reporting System) for user verification and registration information.
Another requirement across Services that may apply is that any
competency management module must be compliant with Common
Human Resource Information Standards (CHRIS).

There may be particular implementation issues when installing an

LMS in U.S. DoD or government, such as:

 Requirements for conducting site or pre-installation surveys

Choosing an LMS.docx page 66 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Constraints on who can host the LMS

 Hardware, software, and firewall requirements

 Particular government contracting rules regarding setup, startup

costs, vendor support, and annual maintenance agreements

See Appendix D: Sources of Possible Requirements for U.S. DoD LMS

Acquisitions and Installations for a list of possible sources of
requirements for U.S. DoD.

4.14 System environments

It is important that you institute at least two and possibly three
staging environments for your LMS, possibly on separate networks.
When acquiring an LMS, you should take this into account.
Consideration of multiple environments (i.e., instances of the
software installation) is often an oversight until after procurement (at
which point there are financial barriers to implementing it). The
environments are:

 Development – for content developers to upload, configure, and

test their content, and for administrators to perform “what if”
scenarios for major changes to the system.

 Test (also termed Stage) – for content and major configuration

changes made in the Development environment to be verified
and finally approved before being migrated to the Production
environment. This instance of the system should exactly match
the Production system in all respects. This environment could
be the same as the Development environment. However, you
may want it to be different (i.e., Test as described here) so that
you can more flexibly make configuration changes to the
Development environment to accommodate the needs of testers
and developers.

 Production – The live system that learners and administrators

use.

These environments do not have to be separate installations. Isolated

areas or instances of one installed system can be just as effective;

Choosing an LMS.docx page 67 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

however, firewall restrictions and different access needs for the user
groups associated with each of these environments may prohibit this,
requiring separate installations.

Acquisition of these environments in addition to your production

environment will probably affect pricing and your infrastructure
requirements. Licensing can be complicated if external entities such
as content development vendors need to use the additional instances.
Special licenses may be required for them. Some LMS vendors sell
packages that include these staging environments pre-configured
(“sandboxes”).

You need to be careful about allowing testing of new LMS

versions/features/customization and testing content on the same
environment or instance of it. This situation can lead to problems,
where, for instance, content works well in the Test environment, but
not in the Production system because they are not precisely the
same.

4.15 Standards support

4.15.1 SCORM

4.15.1.1 Overview

ADL has identified the following high-level attributes for all

distributed learning environments.

 Interoperability: the ability to take instructional components

developed in one system and use them in another system.

 Accessibility: the ability to locate and access instructional

components from multiple locations and deliver them to other
locations.

 Reusability: the ability to use instructional components in

multiple applications, courses and contexts.

 Durability: the ability to withstand technology changes over time

without costly redesign, reconfiguration or recoding.

Choosing an LMS.docx page 68 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

To achieve these attributes in distributed learning environments, ADL

promotes the use of the Sharable Content Object Reference Model
(SCORM®). SCORM defines the interrelationship of course
components, data models, and protocols so that learning content
“objects” are sharable across systems that conform with the same
model. To support interoperability, SCORM standardizes the means of
communication from the sharable content objects (SCOs) to the LMS,
through an Application Programming Interface (API) and prescribed
data model elements.

For more information on SCORM, see https://www.adlnet.gov/scorm/

It is important to understand that SCORM neither dictates nor

precludes any instructional, performance support, or evaluation
strategy. SCORM does enable object-based approaches to the
development and presentation of eLearning. This is enabled by
aggregating learning content composed from relatively small,
reusable content objects to form meaningful units of instruction.
Individual content objects can thus be designed for reuse in multiple
contexts, and aggregated variously to assemble new components and
programs of instruction.

This object-based approach, intended to support reuse, means that

content objects must not determine by themselves how to
sequence/navigate aggregations that represent parcels of instruction.
Doing so would require content objects to contain information about
other content objects, which would inhibit their reusability. ADL
addressed this requirement by standardizing a set of behaviors that
that all SCORM-2004 compliant LMSs must support. Thus, the LMS,
rather than the content, controls the movement of learners from SCO
to SCO.

To support reuse, SCORM uses metadata to enable content objects to

be discoverable through and across enterprises, within distributed
content repositories.

NOTE: LMSs for U.S. DoD installations must be SCORM-conformant

(to the “current version”) according to DoD Instruction 1322.26 (June
16, 2006 – this DoDI, as of this writing, is being revised, although it is
likely that this requirement will still remain in effect). See

Choosing an LMS.docx page 69 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

http://www.dtic.mil/whs/directives/corres/pdf/132226p.pdf for more

details.

4.15.1.2 Requirements for SCORM support

For an LMS to robustly support SCORM, it must:

 Support SCORM-conformant learning delivery

 Support all SCORM data model elements (SCORM 2004)

 Import SCORM course packages

 Support SCORM metadata

 Support sequencing and navigation rules for the course

organization (SCORM 2004)

If you expect to deliver legacy SCORM 1.2 content, you should ensure
that the system supports it; SCORM 2004 is not backwards
compatible with SCORM 1.2, so the LMS needs to include separate
functionalities for importing, configuring, and delivering these two
standards (generally, when you import SCORM content into the LMS,
there will be separate options for SCORM 1.2 and 2004).

Before you evaluate the LMSs in terms of SCORM compliance, you

should determine the target SCORM compliance level (for example,
SCORM 2004 4th Edition). This depends on the compliance level of
your legacy courseware, and courseware you plan to develop. LMSs
can lag several versions behind the current level, and since SCORM
levels are not all backward compatible, it is important to determine
the level of compliance needed for your course delivery system.

SCORM comes in five versions:

 SCORM 1.1

 SCORM 1.2

 SCORM 2004 2nd Edition

 SCORM 2004 3rd Edition

Choosing an LMS.docx page 70 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 SCORM 2004 4th Edition (the current version)

Conformance with SCORM 1.2 is broken down into three levels, LMS-
RTE1, LMS-RTE2 and LMS-RTE3. The levels indicate how much of the
SCORM run-time data model the LMS supports. LMS-RTE3 indicates
full support.

ADL highly recommends that you acquire a sample SCORM-

conformant eLearning course produced by the authoring tool you
use, and test it on the LMS you are evaluating for purchase. LMSs
implement the same SCORM compliance level differently in some
cases; the interaction of the particular implementation of SCORM in
the LMS and the particular implementation of SCORM in your SCORM
course package, even if both are at the same level of compliance, may
uncover issues. This may impact your decision to purchase a
particular LMS.

4.15.1.3 SCORM Conformance vs Certification

An LMS that is SCORM conformant has been tested in the ADL

SCORM Conformance Test Suite to ensure that it performs as
specified by the SCORM standard. This test applies to a specific
version of SCORM only. The ADL SCORM Conformance Test Suite is
available at
http://adlnet.gov/wp-content/uploads/2011/07/SCORM.2004.4ED.TS_.v
1.1.1.zip).

An LMS that is SCORM certified means that a qualified, neutral third

party has conducted a formal evaluation using the ADL SCORM
Conformance Test Suite using a rigorous, accurate, reliable, validated
methodology. Certified products display the ADL certified product
logo. There is a list of SCORM-certified LMSs on the ADL web site at
http://adlnet.gov/wp-content/uploads/2013/09/SCORMCertifiedProduct
sLocked-5.xlsx.

If you are considering products that claim to be SCORM conformant

but are not SCORM certified, you should ask for an ADL SCORM
Conformance Test Suite test log from the vendor verifying their
SCORM conformance. Alternatively, you can run this test yourself if
you have access to a version of their LMS. A product that cannot pass
the full set of conformance tests is not SCORM conformant.

Choosing an LMS.docx page 71 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

ADL recommends that you write into your contract or acquisition

language that the vendor will maintain SCORM
conformance/certification throughout the life of the contract. You do
not want to be stuck in a situation where the vendor issues a patch,
upgrade, or new release that interferes with the ability of the LMS to
deliver SCORM, and your SCORM content suddenly does not run
properly (with no recourse to force the vendor to fix the problem).

Be aware that, in the past, there have been loopholes in the

certification process whereby vendors can maintain their SCORM
certified status, even though their LMS has undergone version
upgrades, patches, etc. that inadvertently affect their SCORM engine,
with the result that SCORM content no longer works properly in their
LMS. ADL is revising the rules for the certification program to address
this loophole.

Dig deep into claims of SCORM compliance. If the LMS has an internal
authoring tool, it may mean that the product can import a SCORM
package into the authoring tool, but the authoring tool converts it into
the LMS’s proprietary format in order for it to work in the LMS. In
other words, you may not be able to import SCORM content directly
into the LMS and have the content function natively (using SCORM
affordances) within the LMS. This is not true interoperability in the
spirit of SCORM.

You may want to ask the vendor whether they participate in the
process of evolving the SCORM standards, and if so, how. ADL has a
variety of community outreach avenues that enable vendors to share
suggestions and keep abreast of SCORM developments. This is a
good indicator of the vendor’s commitment to support for the SCORM
standard.

4.15.2 Section 508

The U.S. Rehabilitation Act of 1973 was amended in 1998 to add

Section 508, which establishes rules, principles, and guidelines to
make it easier for people with disabilities to access electronic and
information technology media. The law applies to all Federal agencies
when they develop, procure, maintain, or use electronic and
information technology.

Choosing an LMS.docx page 72 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

If your organization requires Section 508 compliance for eLearning

systems, it is critical that you include this as a decision parameter in
your choice of an LMS. Do not confuse Section 508 compliance for
the LMS with Section 508 compliance for the content; 508 compliance
for the LMS means that the interface and navigation through the LMS
is accessible to those with disabilities (especially visual impairments).
Where 508 compliance is required, the content must also be 508
compliant, but LMS compliance does not affect or control this.

You should verify 508 compliance by testing the LMS with screen
reader software used by those with visual impairments and/or using
an independent accessibility checker.

For references and other information on Section 508 compliance, see

http://www.section508.gov/

4.15.3 Aviation Industry CBT Consortium (AICC)

Support for this popular legacy standard is fairly common among

LMSs. Note that the standard is used by many more organizations
than the aviation industry. One reason for the popularity of the AICC
standard for content is that it avoids the cross domain scripting
problem (see 4.12 The cross domain issue). There are several
different implementations of AICC:

 File-based

 HTTP (Web)-based

 ECMAScript-based (browser-based)

In most cases, the term “AICC” refers to the HTTP-based

implementation known as HACP (HTTP AICC Communication
Protocol). If AICC support is important to you, you may want to
ensure that it supports HACP to ensure broad content compatibility.

NOTE: AICC as an organization dissolved in 2014and transferred its

CMI-5 standard and document archive to ADL. The CMI-5 standard
has adopted ADL’s xAPI specification. For details on CMI-5, see
https://github.com/AICC/CMI-5_Spec_Current

Choosing an LMS.docx page 73 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.15.4 Standards for metadata

Some of the standards that are used specifically for metadata in

eLearning are the following:

 IEEE Learning Object Metadata (LOM)

http://www.imsglobal.org/specifications.html

 Dublin Core
http://www.dublincore.org/

Support for a particular metadata standard in an LMS is not needed

unless the standard has been fully adopted by your organization. If
the metadata standard has been adopted, LMS support for it will
facilitate search, discovery, and cataloging of your eLearning and
other content objects in your LMS. In a large enterprise with many
learning objects, this may represent a significant savings of time and
effort. Metadata normally resides within the content itself and is
imported into the LMS database when the course files are imported.

Note that SCORM does not prescribe use of metadata, or any

particular metadata standard. However, LOM is most commonly used.

4.15.5 Learning Tools Interoperability™ (LTI)

The IMS Global Learning Consortium developed LTI as a standard

way of integrating rich learning applications with LMSs and other
educational environments. In LTI these learning applications are
called Tools (delivered by Tool Providers) and the LMS, or platforms,
are called Tool Consumers. For more information, see
http://www.imsglobal.org/toolsinteroperability2.cfm.

4.15.6 Common Cartridge®

IMS Global Learning Consortium developed Common Cartridge® as a standard way to package a course
for importing to an LMS. It has many of the same advantages as the SCORM packaging specification
(Content Aggregation Model). If you are importing and delivering courses that are packaged using this
specification, you need your LMS to support it. See http://www.imsglobal.org/cc/index.html for details on
this standard.

4.15.7 QTI®
The IMS Question and Test Interoperability specification (QTI) is an interoperability specification that
specifically relates to online tests. It is concerned with the structure and display of test items as well as

Choosing an LMS.docx page 74 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

results. It allows passing of data between authoring systems, content, and delivery systems, including
LMSs. See https://www.imsglobal.org/question/index.html for details on this standard.

4.15.8 ADL Total Learning Architecture (TLA), including xAPI

ADL has termed the next generation of SCORM as the Total Learning
Architecture (TLA). All current and planned future ADL technical
projects, specifications and standards efforts fall within the scope of
the TLA, an umbrella term that covers projects designed to create a
rich ecosystem for connected training and learning. Phase I of the
TLA is focused on experience tracking that includes these four areas:
 A new runtime API
 A new data model
 A new data model format/syntax
 A new transport/communication method

The overall TLA vision also includes concepts for learner profiles,
competencies, and intelligent content brokering to meet the needs for
individualized learning content and systems. The TLA is not intended
to replace SCORM, but SCORM, and multiple other types of content
formats, will work in the TLA. The four major areas of innovation of
the TLA are:
 Experience tracking
 Learner profile
 Content brokering
 Competency infrastructure

The Experience API or xAPI (formerly known as the ‘Tin Can API’), the
“experience tracking” component described above, is the farthest
along in development currently (version 1.0 was released 4/26/13, and
the spec is now at version 1.03). The xAPI tracks both formal and
informal learning via ‘streams’ of learning experiences, similar to
social media streams such as Twitter and Facebook. By capturing
learning experiences via streams, learning can be mashed up with
other activity data to fully analyze how it ties to performance. The new
xAPI (see http://adlnet.gov/adl-research/performance-tracking-
analysis/experience-api/) enables the use of mobile devices, games,
social networks, virtual worlds, and simulations in learning and
training environments with the ability to track learning experiences
consistently across devices and platforms. You could report that

Choosing an LMS.docx page 75 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

‘David watched a video,’ ‘David rated a video,’ ‘David tweeted a video,’

and ‘Jane retweeted David’s video.’

Learning can also be tracked in real life situations and reported the
same way. For example, ‘John produced an audio track for a video,’
‘Steven edited a video,’ ‘Ralph posted a video,’ and ‘Mary earned an
Academy Award for a video.’ This is why this is described as
“connected” learning, because even “real life” situations can be
connected in more ways than just how people interact with computers
on the Internet.

The xAPI is a specification that describes an interface and the storage

/ retrieval rules that developers can implement to create a learning
experience tracking service. The service works by allowing
statements of experience (typically learning experiences, but could be
any experience) to be delivered to and stored securely in a Learning
Record Store (LRS). Widespread adoption of the xAPI may drive LMSs
to include an LRS component that can handle xAPI statements.

One major advantage of the xAPI over SCORM is that it does not
require launching content from an LMS; in fact, it does not even
require Internet connectivity while the user is engaged in the learning
experience. Learners can connect after the fact to allow the xAPI to
synch the records of their learning experiences. This has obvious
implications for the future of LMSs; to accommodate learning that is
developed for use outside of the LMS environment, or disconnected
use, LMSs may need to separate their function that handles tracking
of learner experiences into a single cloud-based service (in xAPI
terms, an LRS) that is easily accessible from a variety of content and
can dynamically capture xAPI statements describing learning
experiences. See 7.26 Is the traditional LMS dead? for more
information on the implications of xAPI on the LMS.

For information on the xAPI standard, see http://adlnet.gov/adl-

research/performance-tracking-analysis/experience-api/. There is a
list of current adopters of the xAPI at
http://adlnet.gov/adl-research/performance-tracking-analysis/experien
ce-api/xapi-adopters/. A white paper similar to the one you are reading
has been produced by ADL, called Choosing an LRS (Berking, 2015b),
available at
https://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAnLRS.docx.

Choosing an LMS.docx page 76 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Learning Record Stores (LRSs) are the system necessary for

receiving xAPI statements (that are now being integrated into or with
LMSs).

Also see 7.1 Experience API (xAPI) adoption and 7.26 Is the traditional
LMS dead? for more details on how the xAPI is affecting LMSs.

4.16 Internal assessment authoring

ADL recommends that you create assessments within the content so
that they are portable and interoperable; however, in some cases, you
may want to be able to create assessments through tools offered
within the LMS. The vast majority of LMSs offer internal assessment
creation and delivery capabilities. The downside to using this internal
authoring function for assessments is that these assessments are
often permanently resident in the LMS and cannot be exported for use
in another system or with other content.

Assessment authoring within the LMS may be attractive because

assessments must interwork closely with the LMS tracking database.
For instance, an assessment created in the LMS may allow a greater
range of reports. It is often quicker and easier for LMS instructors and
administrators to use an internal LMS function rather than create
external assessments with the appropriate data calls. Also,
assessment interactions can be more difficult to program than
presentation content, so it avoids this technical burden on the
customer as well.

Use of internal assessment authoring is particularly common in cases

where learning activities are conducted offline and cannot be
assessed and tracked by the LMS while the learner completes them.
Thus, an LMS-delivered assessment capability or assessment
management system such as Questionmark® is the only way to verify
and store the learner’s progress against outcomes, and it is easier to
author these assessments internally in the LMS. The standard types
of eLearning assessments that are offered are:
 Multiple choice (both single and multiple answer)
 Fill in the blank
 Matching
 Drag and drop

Choosing an LMS.docx page 77 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Ranking/Ordering
 Image selection
 Essay or Short answer (this usually requires instructor
intervention to score answers)

Some LMSs import and export assessments that adhere to the QTI ®
specification (described in 4.15.7 QTI®), which allows portability of the
assessments between systems.

4.17 Internationalization
If your learners include international audiences (especially including
foreign language speakers), you will need to consider features of the
LMS that will support it, as well as plan your LMS implementation
accordingly. There are many factors you may need to consider, in
addition to language, such as:
 U.S. export laws governing dissemination of information in
areas of technology that is deemed of strategic importance to
national security (this applies to information that is not
classified or marked as FOUO)
 Local government rules and regulations that may lead to non-
compliance of content
 Government requirements for training record storage
 Accreditation differences
 Cultural norms
 Local and country-wide IT environment
 Currency and currency exchange, and financial market
operation, if charging money for training.

4.18 Instantiation for individual business units

Many LMSs are adding the ability of a single enterprise LMS to create
a customized interface and some degree of local control of
administrative functions for different business units, divisions,
workgroups, etc. within the enterprise. This has the effect of allowing
these groups to have their own interface look and feel, course
catalogs, etc., giving the appearance that they have their own LMS. It
also may allow them to customize the administration of their
“storefront” to some degree. This instantiation is usually
implemented via login profiles, such that when a user from a certain

Choosing an LMS.docx page 78 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

organization logs in, the interface, available courses, global functions,

etc. are delivered as configured for that organization. See 4.2 LMS
skins and templates for more details.

This instantiation is normally used in cases of LMS sharing (see 4.19

LMS sharing).

4.19 LMS sharing

The driver for the popularity of instantiation of an LMS for individual
business units (see 4.18 Instantiation for individual business units) is
mostly economic; it allows an enterprise (especially a large one, like
the Federal government) to save money through sharing of the same
system, rather than each unit, agency, etc. having to purchase a
separate system. There are two ways LMS sharing can be achieved:
one is where an organization is paying for a greater capacity on an
LMS than they are using, and another organization can fill that
capacity up to (but not over) that maximum. This can result in no
extra cost to either organization except for the maintenance and
administration associated with using that larger capacity. This
arrangement is enabled by a seat-based pricing model (see 4.4
Pricing models) and a license with the LMS vendor that allows the
purchasing organization to share with other organizations.

The other arrangement is necessary where there is not enough

unused capacity. In this case, the LMS purchaser organization buys
more seats or a higher capacity tier on an LMS, with the organization
using the higher capacity covering the cost. This can be highly
economically advantageous for the organization using the higher
capacity, since it is almost always a net savings in cost to share an
existing LMS in this way rather than buy a separate product or
hosting package from the vendor. This arrangement can be used
when the pricing model is either seat-based or usage-based (see 4.4
Pricing models) and is also subject to licensing rules.

An example of an LMS sharing plan offered for government

organizations is the USALearning Knowledge Portal, run by the Office
of Personnel Management (OPM). See https://usalearning.gov/ for
more information.

Choosing an LMS.docx page 79 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

4.20 The path of least resistance

It is important to remember the simple fact that most users, in many
cases regardless of their skill set, will follow the path of least
resistance in using an LMS, as with any other software. In other
words, users will gravitate towards the most heavily optimized system
features—those that are prominently available in the interface and
easiest to manipulate. The system may include many advanced
capabilities, or even easy workarounds or hacks that are possible to
accomplish highly time-saving tasks, but most users will ignore these
if they are not designed to follow the path of least resistance.

So the question in evaluating an LMS is not necessarily, “What can

the system do?” but, “What can the system do in a right-out-of-the-
box, plug-and-play, easiest/most-obvious-path use case scenario?”
Just because a vendor is able to make a technical case that their
system has a particular capability doesn’t mean that it is implemented
in a way that is easy for users to see, understand, and use.

4.21 Aligning staff and processes to system capabilities

As with most software, systems that are easier to learn and use
generally have fewer capabilities, and vice versa. Sophisticated
capabilities will generally require a system that is harder to learn
and/or require specialized professional expertise. It is important to
determine the skill sets within your pool of LMS administrator staff,
so that you know what you are prepared for and/or what you might
have to acquire in terms of staffing or training. You can engineer your
staff expertise and roles to match the out-of-the-box system, but it is
usually not cost-efficient to engineer the system to match staff
expertise.

This also applies to task flow; you will almost invariably need to
decide whether you want to change your internal processes to match
the built-in LMS task flow, or vice versa (i.e., reengineer the LMS to
match how your organization does things). This is a complex issue,
and there are some strong proponents on the side of choosing an
LMS that, out of the box or perhaps with customization, supports your
existing processes, but this may be easier said than done. It is likely
that you will have to do some of both. Above all, do not underestimate
the financial pressure you may find yourself under to tailor your

Choosing an LMS.docx page 80 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

organizational policies and processes to make it easiest to work with

the system out of the box. Customization of LMSs, whether open
source or commercial products, can be quite expensive, and it should
not be undertaken lightly even if not expensive; if you start making
changes to the guts of the system, you may quickly reach a point of
no return where system updates from the vendor are incompatible
with your customized system, thus your system cannot be upgraded
(it behooves the vendor to warn you about this).

The LMS system design and “path of least resistance” workflows can
imply changes to your existing processes and infrastructure in the
following areas:
 IT infrastructure
 Administrative procedures and policies
 Workplace cultural attitudes and ingrained practices
 Training paradigms

4.22 Planning for operation and governance of your LMS

Before acquiring your LMS, you should have a preliminary plan for
how you will ensure smooth operation of it so that it will be used to its
full potential and will address the performance gap that led to your
decision to acquire it. You also want to be clear on who will maintain
both the system and the content it contains, to avoid confusion and
institutional obstacles that could affect the ability of the system to
realize its intended mission throughout its lifecycle. Without this
preliminary plan, you may face skepticism from management
approvers of the acquisition.

This preliminary plan will probably change once your system is fully
installed, after you gain some familiarity with it and better understand
how to leverage the system features to best express your business
needs, processes, and policies. The reverse applies as well as well;
you may determine that it is easier to change your processes and
policies to match the system’s standard features and workflows, as
described in 4.21 Aligning staff and processes to system capabilities.

To ensure a smooth implementation, you need to start this planning

during the acquisition phase, not wait until after acquisition. Some

Choosing an LMS.docx page 81 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

aspects of your plan may impact your choice of LMS and vendor,
especially if the vendor will act as implementation consultant.

According to Foreman (2013), the following areas should be

addressed in your LMS Operation and Governance Plan:

 Standards

o Policies
 Content inclusion policy
 Content ownership policy
 Content lifecycle policy
 Training information retention policy
o Procedures
o Guidelines
o Conventions
o Standards for course properties
o Standards for course structures
 Taxonomy
 Configuration Management
 Housekeeping
 Governance
o Governing board
o LMS steering team
o LMS working groups
o LMS operations
 LMS operations management
 Content owners
 LMS administrators
 Technical support

4.23 Data migration

If you are switching LMSs or moving data into a new LMS from an HR
system, you will need to plan carefully for data migration. The goal
should be no loss or corruption of data in the process. You may not

Choosing an LMS.docx page 82 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

need to migrate all data; some of it can be archived and accessed

only if and when needed. Be careful in making this decision, however.
Some of the data that you need to keep may have dependencies on
data that you might otherwise archive, for example, course
prerequisites.

In many cases there is no way to avoid some degree of manual data

recreation; automated tools would be prohibitively expensive to
develop. There are four categories of data that you may have to
migrate:
 Content. An analysis of how the new LMS stores and delivers
content differently might uncover discrepancies that have to be
addressed by modifying content or even converting it to another
format, which can be costly. Data mapping, data cleaning, con-
tent ownership, and content portability are important issues that
may need to be addressed.
 Logical entities. This includes logical entities stored in the
system like learning paths and certifications. The mapping from
the old system to the new could be complicated due to different
databases, definitions, dependencies, and rules. Moving these
to the new system will probably involve recreating them in the
new system.
 Training records. These are the records (i.e., transcripts) of all
training sessions. It includes not only past records, but in
progress training as well.
 User-generated content. There may be types of content related
to peer to peer sharing in the old system that need to be
migrated. This includes discussion forums, chat rooms, expert
exchanges, etc., as well as documents and media files generated
by users stored in content repositories.

An important factor to consider in data migration is standards. Your

migration may go smoother and you will have more flexible data if
you convert your legacy data to standards such as SCORM or LTI and
then have the target LMS ingest it according to the way it handles
data formatted to these standards, rather than do a straight database
to database exchange. For SCORM courses, Foreman (2013)
describes the following process:

Choosing an LMS.docx page 83 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Categorize the SCORM courses by the authoring tool or vendor

that produced them
 Then install and thoroughly test one course from each category
 Test course launch, player compatibility, bookmarking,
navigation, audio, video, animations, graphics, and embedded
links, as well as test score, and module and page tracking
 Make sure the course shows up properly in the transcript of the
new LMS
 If you run into problems, make any needed adjustments to the
course and the manifest, reinstall and retest
 Replicate your adjustments to all other courses in the category;
install and test them all.
 Manage an exception list of any courses that don’t work
properly
 If you no longer have the source code for the course, you may
need to redevelop the course using a more compatible authoring
tool
In many cases, the vendor of the new LMS will have dealt with a range
of migration issues within their customer base and have solutions
developed that can save tremendous amounts of time and headache.
Talk to your vendor to explore options that are available.

4.24 System load and bandwidth to the users

For many organizations, load on the LMS system itself may not be an
issue except perhaps during peak usage times. There are various
possible solutions to this problem outside of simply buying increased
system capacity; a simple one is to stagger course completion
deadlines between courses or groups within a course such that the
users are not piling up on the server all at once. Content delivery
networks (CDNs) can be helpful in these cases as well (see 7.18
Content delivery networks (CDNs) ).
However, network bandwidth to the users to and from the LMS server
could be a problem, especially if they are in remote areas or using
BYOD data plans on mobile devices. There could even be a problem
within retail outlets, where employees taking training onsite at the
outlet are sharing bandwidth with point-of-sale (POS) systems,
security systems, customer Wi-fi, etc. With greater and greater use of

Choosing an LMS.docx page 84 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

video (especially high definition) for training, there could be a

significant slowdown for all users within the store. This could cause
problems for not only trainees who experience latency in their
training videos, but for customers trying to complete purchases, etc.
in the store.
Provisioning content to the local site or end users in advance could
be a solution to bandwidth problems, so that the download of content
media is done at an optimal time and place and is stored for use when
needed later. See 4.8 Offline content provisioning and player
capability for more information.
Your IT department needs to look carefully at the impact of delivering
different kinds of content (esp. video) from the LMS, the logistics of
how and where training is taken by users, and a host of other issues
so that your LMS initiative is not doomed from the start by low
bandwidth to the trainees, or the corporate repercussions of slowing
down operations.

5. List of possible requirements for an LMS

This section contains a comprehensive list of possible requirements
for an LMS you are acquiring. It could also be used to assess the
quality and suitability of an LMS. The applicability of items in this list
to your situation will probably vary widely; some items may be
mission-critical for your organization and some may not be pertinent
at all. You need to carefully weigh the importance of each in
evaluating LMSs. If you rate your list of LMS candidates simply by
items in the list without weighting each item for its importance to you,
it could skew the results, which could lead to a poor final choice for
your system. A table in Appendix B. Sample System Features Rating
Matrix allows you to set weightings for your requirements and
calculates a score based on them.

There is also the issue of the degree of support and robust

implementation for a certain LMS feature. Very few of the features
listed below are a strict either/or proposition for being present or not
in a given LMS. They are implemented in various degrees of power,
flexibility, usability, scale, etc. The devil is in the details of a given
LMS feature in terms of nuances and variations in the way different

Choosing an LMS.docx page 85 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

systems implement it. These can make the difference between a

feature being truly usable or not.

Because LCMSs incorporates additional content authoring and

repository functions, other requirements are applicable for LCMSs in
addition to the criteria presented in this section. See Appendix G:
Additional requirements for LCMSs. Further, if you are looking at an
LCMS solution, you will need to focus on the quality of content
authoring features. For a list of quality criteria related to authoring
capabilities, see ADL’s Choosing Authoring Tools paper at
http://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAuthoringTools
.docx.

The following is a list of possible requirements, which can also be

used as a standard reference for assessing the quality of an LMS.
Some items on the list below are stated in a way that can be
subjective. Meeting these requirements requires a qualitative
assessment of how well specific features in a given product address
the stated requirement, with a rating assigned. However, most of the
items in the list are specific and can be basically determined as being
present or not, in “yes” or “no” form, in a given product (though still
subject to some degree to quality of implementation details, as
discussed above).

Requirements are grouped at the highest level under major functional

LMS areas; note that these functional areas are not mutually
exclusive and sometimes overlap. Generally, however, we avoided
listing requirements in more than one functional area, so that the
overall list contains no repeated items.

5.1 Registration and enrollment functions and workflow

 Uses a straightforward, simple process for administrators to
manage registrations and enrollment

 Incorporates navigation and search options within course

catalogs and learning tracks to find and register for courses

 Allows secure self-registration process for external users (who

are not automatically registered by virtue of being an employee)

Choosing an LMS.docx page 86 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows administrators to easily combine and un-combine course

sections

 Allows learners to auto-enroll for courses

 Automatically places learners in wait lists if courses are full,

with automatic notification to affected stakeholders

 Automates tuition assistance requests and allow for supervisor

and other administrative approvals of these in the system. This
includes automating tuition assistance verifications after
courses are completed.

 Provides the ability to restore learner artifacts and records in a

course if a learner who has withdrawn subsequently re-enrolls

 Manages registration and enrollment not only by individual, but

by group and cohort

 Provides the ability to print a variety of enrollment-related items,

including class schedules, seat vacancies, and class rosters

 Allows a variety of billing options: credit card, corporate

purchase orders, departmental account numbers, etc.

 As a configuration option (where courses are not mandatory),

allows learners to select, register and remove courses or
curricula from their course listing/learning track on their own
with no supervisory or administrator intervention

 Displays visual interface options such as map and tree

metaphors to enable clear understanding of the organization of
curricula and easy course selection within them

 Provides the ability to “crosslist” courses. “Crosslisting” is a

process for manually sharing courses or learning objects within
them without creating multiple instances of the same course.
For example, a course might have two names, since learners can
get credit for it while in two different curriculum tracks (e.g.,
anthropology and sociology); enrollments will be different, tied
to the different curriculum tracks, but learners will be taking the
same course.

Choosing an LMS.docx page 87 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows registration and enrollment based on multiple

memberships. For instance, a user is a member of the HR
division but also a Level 2 supervisor; they are assigned
courses based on both of these memberships.

 Manages recurring training such that learners are auto-enrolled

at the appropriate intervals

 Allows training managers and instructors to enroll and/or

approve enrollments for learners. Approvals should have due
dates associated with them.

 Saves of sets of configurations as templates that can be applied

to future courses

 Routes enrollment requests to and from appropriate parties with

notification to others: administrators, instructors, students, and
stakeholders

 Allows automated identification, disabling, and archiving of

learner accounts and course sections that have been inactive for
a set amount of time

 Interfaces effectively with HR systems to provide user lists, to

automatically add new users and deactivate users who have left
the organization. Manual maintenance of user lists can be labor-
intensive and complex.

 Provides links to and/or enrolls learners in appropriate courses

or curriculum automatically based on organizational
requirement settings

 Allows guest (i.e., non-credit) access to courses

 Can flexibly store and report course credits in different number

formats, from whole numbers down to decimal places. Some
organizations that license and certify professionals offer
courseware that needs to be tracked at the level of two decimal
places, for example, 2.75 credits.

 Allows setting course allotments and prioritizing learner

enrollments to courses based on them

Choosing an LMS.docx page 88 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Includes instructor cadre management. This includes managing

instructor qualifications, classes authorized to teach, and
resource alerts to prevent over-booking in scheduling.

 Allows administrators to easily override settings made for

groups to account for particular training needs of individuals

5.2 Notifications, messages, and annotations

 Notifies users when actions are taken in the system that affect
them, both through internal system notification functions and by
email. Notifications can be triggered by such actions as:
o Change in user profile status
o Change in course status
o Confirmation of enrollment
o Class cancellation
o Being wait listed for a course
o Learner dropped from class
o Periodic reminders to attend or finish courses
o Reminders to complete a survey

 “Canned” notification email messages can be customized using

a combination of free text and system variables. Fields that may
be customizable include:
o Recipients
o Subject
o Body
o Signature
o Course title
o Course dates
o Course location

o A notification history function is included.

 Notifications and calendar events for learning events and

deadlines can be inserted into web-based e-mail calendars, like
Gmail and Hotmail..

Choosing an LMS.docx page 89 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows pushing of general notifications out to only specific

groups

 Sends reminders to students and stakeholders about

certifications that are about to expire (so that they can take
actions to renew)

 Provides the ability to annotate and communicate actions taken,

approvals, errors, etc. in regards to administrative actions, for
future reference or for other administrators

 Has internal email or message editor with features resembling

standalone email programs (such as Outlook)

5.3 Batch administration workflow

 Offers batch options for tasks involving groups of system
objects

 Allows setting up sequences of individual system processes to

enable complex batch operations

 Allows administrators to batch register groups of learners

 Allows administrators to create backups in batch

 Allows administrators to batch set permissions and roles for

users

 Allows administrators to batch configure courses, learning

tracks, and curricula

 (for CrMSs) Allows instructors to batch grade students

 Allows time shifting of batch processes of database or

processor-intensive tasks to minimize performance disruption
during peak usage times

Choosing an LMS.docx page 90 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.4 Prerequisite handling

 Allows administrators to set prerequisites so that learners are
evaluated for meeting prerequisites before being able to enroll in
a course

 Can be configured to deliver pre-assessments to allow learners

to “test out” if they demonstrate mastery of the material for a
course

 Includes options for waiving course/curriculum requirements

based on demographic attributes other than course completion
or pre-assessments

 Establishes equivalencies so that learners can receive credit for

courses and/or waive the requirement to take courses that cover
the same material as a course already taken

5.5 Content importing and configuration

 Provides the ability to internally create and/or configure ancillary
learning objects like glossaries that can interwork with courses
and apply globally to more than one course within the LMS

 Is interoperable with 3rd party content (if applicable). If you are

delivering courses provided by a commercial provider (for
example, Skillsoft®), you will need to ensure that the content
operates effectively within the LMS you are acquiring. This
characteristic is supported through the use of standards and
specifications such as SCORM and Common Cartridge
(mentioned in 4.15 Standards support).

 If the LMS is interoperable with 3rd party content (see above), it

offers flexible, secure, and user-friendly payment options such
as PayPal®

 Imports course packages of unlimited size (especially important

if your eLearning contains rich media, or courses are very long)

 Allows elements of a course to be updated without creating a

new version of the course (for instance, swapping out the

Choosing an LMS.docx page 91 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

SCORM manifest file without having to upload an entire

replacement course package)

 Presents options to automatically move learners to a new

version of a course when a new version of it is created, or allows
them to continue on the old version. This has implications for
progress data; you do not want learners to lose existing
progress data if they are half way through the course. For minor
changes to the course, learners should be able to seamlessly
experience the updated content with no interruption in their
learning flow. However, for major version updates, it can be very
hard or inappropriate for an LMS to move users to the updated
content while maintaining their progress information.

 Includes a guided course setup feature that walks the

administrator through the process of creating a new course,
reminding him or her about features he or she might want to add
(e.g., discussion forum, blog, etc.)

 Allows creation of custom categories and folders of content, for

display to users as well as for internal authoring and
management

 Allows assignment of custom thumbnails and icons for content

objects displayed in the library or catalog

 Allows course administrators to duplicate or roll course

objects/records forward to new sessions without requiring re-
entry of data

 Can set activation and expiration dates for content that is time
sensitive, or where the license for is only for a certain period

 Features intelligent automated extraction of metadata details

(title, etc.) of web-based references and other learning materials
added to the LMSs content library

 Has a content organizer feature that allows administrators to

easily design and reconfigure a course by dragging and
dropping course elements and reordering them

Choosing an LMS.docx page 92 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows enforcement of metadata tagging and compliance with

metadata standards such that content cannot be uploaded and
published without it (especially for end user generated content)

5.6 System access and security

 Uses robust security architecture to maintain system access

 Allows learners to self-register for an LMS account using a

unique email address and follow-up email requiring a validation
response, to avoid duplicate accounts, incomplete user profiles,
etc.

 Has the ability to require at least some fields in the user profile
to be mandated to fill out when users initially register or first log
on

 Incorporates e-signatures

 Conforms to secure application infrastructure standards such as

ISO/IEC 27001

 Allows encryption of sensitive data (i.e., passwords) and

session activity (i.e., LMS-related network traffic)

 (for government organizations) Conforms to applicable security

regulations such as 21 CFR Part 11, EU GMP Annex 11

 Offers Service Organization Control (SOC) level 2 compliance

 Affords a high level of password security features, for instance:

o Allows the administrator to require users to us strong
passwords
o Limits the use of old passwords
o Defines parameters for strong passwords for users
o Requires users to change the password on first login
o Locks users out after a certain number of failed login
attempts
o Requires users to change passwords regularly (using
notifications)

Choosing an LMS.docx page 93 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Sets limits on periods of inactivity

o Only users can change their password
o Encrypts stored passwords

 Is able to handle digital signatures. In a government installation,

this could require compliance with federal regulations like 21
CFR Part 11.

 Provides a single sign-on, so that users who have logged in to

the enterprise intranet (through a portal, etc.) can get into the
LMS without additional login

 Allows login to the LMS to transfer to other enterprise systems

(especially HR)

 Requires user logon only once per LMS session

 Requires each user to be uniquely identifiable (e.g., user name

or user ID)

 Runs all user requests through a common security checkpoint

in the system architecture

 Was developed by a single company (the vendor), to avoid risks

associated with exposure of code to external organizations

 (for high security government organizations) No foreign

nationals in the vendor organization (including subcontractor
organizations) contributed to the codebase or are involved in
hosting (if a vendor-hosted solution).

 Provides audit trails for changes to data in the system such that
the organization can quickly determine the source of
unauthorized activity that could be the source of security
breaches. These changes could include everything from
uploading learning objects to running reports.

 Supports industry-standard authentication using such

standards as:
o LDAP
o CAS

Choosing an LMS.docx page 94 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Shibboleth
o Kerberos
o SSO SAML
o Social logins (Facebook, Gmail, etc.)

 (For high-security government installations) Allows Common

Access Card (CAC) access

 Incorporates appropriate security certifications and standards,

and features (see 4.9 Security considerations for LMSs and 4.13
Special requirements for U.S. DoD). Other security standards
you may need include SSL, PKI, and FIPS – 140-1.

 Allows configuration for the management of PII in accordance

with enterprise and government policy (such as FERPA)

 (for DoD) Contains multiple security access levels with ready

access to unclassified learning material and more stringent
security requirements for FOUO and classified information

 (for hosted solutions) The provider:

o conforms to the ISO 27001 information security management
system (ISMS) standard.
o has Intrusion Detection/Prevention services
o monitors individual LMS instances for suspicious activity, in
real time
o regularly audits the security of its servers

 (for mobile LMSs. From Towards Maturity, 2014). The LMS

includes:
o Automated renewal of non-compliant devices
o Remote locking, wiping and revoke credentials
o Built in encryption – e.g. on device closing
o Malicious code detection and prevention
o App management system
o Standardized administration policies

Choosing an LMS.docx page 95 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Access control – for example job role-based rules and assign

privileges and restriction for handling sensitive data
o Monitoring of device usage and automating event/incident
reporting
o Secure document distribution (e.g. Secure Content Locker)
o Support for legacy applications
o Virus and malware blocking

5.7 Permissions and roles

 Defines a wide variety of permission and role levels that are
applicable to a range of organizational structures and use case
scenarios for the system

 Restricts course enrollment to pre-authorized learners

 Incorporates permission levels and supporting features that

allow input of SME review comments that are tagged to screens,
with search and filtering capability. (this is usually only found on
an LCMS)

 Allows “extranet” user accounts. For example, international

military personnel (e.g., NATO officers).

 Uses templates to easily set group permissions

 Restricts access to functions for individual courses based on

membership on teams associated with that course

 Allows delegating permissions for users at a lower level of

permission than what one is logged in as

 Allows creation of subgroups that inherit permissions of parent

groups

 Can be set so learners are anonymous to each other,

instructors, and administrators

 Offers “organization aware” features that allow administration

based on external data feeds concerning organization roles and
permissions

Choosing an LMS.docx page 96 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows content-level permissions that specify conditions to be

met for learners to be able to view content, such as:
o For military organizations:
 Grade
 MOS
 Service status
 Service
o Payment
o Added to learner’s “My Learning” track or catalog

 Supports mirroring an organization’s structure in the database

to manage learners, supervisors and approvers based on where
they exist within the organizational structure

 Features levels of permission corresponding to clearly defined

levels of administrative responsibility. For example:
o Level 1. Overall responsibility for the total system
o Level 2. Overall responsibility for a specific “store front”
instance of the system
o Level 3. Database Administration
o Level 4. Maintenance administration of the system. This
permission may be segmented to allow users only to perform
particular maintenance tasks or for particular user groups.
o Level 5. Curriculum administration. This permission may be
segmented to allow users only to perform particular tasks for
certain curricula and/or with certain groups of learner.
o Level 6. User administration. This permission allows adding
and deleting users and setting up groups.
o Level 7. Content administration. This permission may be
segmented to allow users only to perform particular tasks on
particular courses. This permission may be subdivided to
account for content adding or creation privileges but not
content reviewing.
o Level 8. Authoring capabilities (for LMSs that have this
function)
o Level 9. Learner

Choosing an LMS.docx page 97 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 (for VLEs only) Allows instructors or administrators to configure

the system to require approvals of users entering a virtual
classroom (rather than automatically allowing them to enter
upon login).

5.8 System performance

 Performs with minimal latency under a variety of use case
scenarios and load conditions

 Handles large numbers of concurrent users

 (for behind-the-firewall solutions) Takes minimal time to actually

deploy. As a baseline expectation, Vipond and Clary (2016)
found in a survey of corporate LMS managers that nearly one-
half (43%) of organizations took more than three months after
initial installation to begin enrolling their learners.

 Handles user load efficiently, provisioning and scaling

resources to smoothly accommodate fluctuations (especially
spikes) in numbers of concurrent users

 Works equally well (all functions, including course delivery) on

all standard Internet browsers, including a reasonable span of
legacy versions of those browsers (backward compatibility with
2 year-old versions is often used as a rule of thumb)

 Has reasonable system requirements that are attainable within

your organization

 Uses normalized architectures for hardware and software

implementations

 Can be load balanced across multiple servers, with the LMS

application distributed across multiple application servers and
the database itself located elsewhere.

 Can be clustered

 Has robust mechanisms for coping with machine failure

Choosing an LMS.docx page 98 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.9 Course catalog database

 Provides a single, integrated or multiple course catalogs whose
overall and internal organization can be flexibly defined by a
variety of characteristics

 Does not arbitrarily limit the number of levels, items, or sizes of

items included in the catalog of courses delivered or imported

 Contains a course catalog including many details of courses,

especially:
o Objectives
o Credits
o Intended audience
o Course #
o Cost
o Associated career track(s)
o Associated competencies
o Delivery method (e.g., classroom, asynchronous eLearning)
o Prerequisites
o Originating delivery platform (e.g., Blackboard, SumTotal)
o Functional area
o Location (if synchronous)
o Job skill
o Product line
o Subject
o Associated resources
o Seat time
o Keywords

 Has global search function for learners that searches on all of

the above plus text within content and allows grouping of search
results.

 Offers the following advanced search capability (Foreman,

2013): “…the user can take advantage of the superior search

Choosing an LMS.docx page 99 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

experience to which they have become accustomed with tools

like Google, Bing, and Yahoo. Like those tools, enterprise
search platforms offer more advanced search methods such as
proximity ranking (most to least relevant), controlled vocabulary
(synonyms and acronyms), all forms of a word, and spelling
correction (“Did you mean...”).

 Can be linked dynamically to external catalogs (for instance,

from COTS content providers)

 Provides the ability to organize content into flexible content

ontologies and hierarchies, and label them appropriately. For
example:

o Catalog

 Curriculum

 Learning Path (possibly tied to a certification)

o Course

 Lesson/ Class session

 Learning object

o Asset

 Can be updated with release updates and additional courses

from external sources

 Provides version control and other management functions for

course updates

 Can be configured such that different versions of the course that

are treated functionally the same for training administration
purposes (for example, the same course delivered in different
languages) use the same reference ID in the LMS database. In
other words, the LMS tracks and reports that learners have
taken the same course, even if there are different language
versions.

Choosing an LMS.docx page 100 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.10 Interface customization

 Allows visual branding of the interface for all users in the
enterprise

 Allows use of skins to visually brand the LMS dynamically

according to the role, organizational membership, or other
parameters of the individual user who is logged in. See 4.18
Instantiation for individual business units. This can also be
important for “extended enterprise” LMs, where branding for
different categories of external users (e.g., customers, suppliers,
etc.) is important.

 Allows toggling display of the LMS Table of Contents (TOC) for a

course, to account for courses with no need for this (for
example, courses with just one SCO) vs SCORM courses with
many SCOs (and thus a need for good inter-SCO navigation)

 Provides the ability to customize and filter individual tables (vs

whole screens) such as lists of courses in progress

5.11 Standards and language support

 Supports the current and all required legacy versions of relevant
standards such as SCORM, AICC, IMS Common Cartridge, etc.
See 4.15 Standards support. For details on what is required for
full SCORM support, see www.ADLnet.gov.

 For SCORM content:

o Is certified at the level of your content, or has been tested for
conformance. Do not rely merely on vendor advertisements of
their conformance.
o Retains visibility for the TOC when a SCO has been launched
o Shows both “attempted” status as well as “completion”
o Is not overly proprietary in its implementation and handling of
SCORM calls other than “attempted” and “completion.”

 Complies with Section 508 requirements for system interfaces

Choosing an LMS.docx page 101 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 (For DoD installations) The Standard Data Elements used by the

product comply with the DoD Data Architecture (DDA) and
Defense Data Dictionary System(DDDS)

 Supports multi-byte (Unicode) fonts (esp. Asian language

characters) and right-to-left languages. This requires that the
LMS interface text is stored as data, separate from source code.

 Offers flavors of the interface in foreign languages for global

reach

 Supports the ADL Training and Learning Architecture (TLA). See

4.15.8 ADL Total Learning Architecture (TLA).

5.12 Training infrastructure and performance analysis

 Includes the ability to enter and capture such items as course
development costs

 Provides support for learner surveys and training needs

surveys, with options for free text as well as closed ended
questions.

 Includes training budget/cost tracking and projecting features

that stores and reports (by learner, course, organization, year,
etc.) such items as:
o Budget authorizations
o Funds allocated
o Funds still available

5.13 Learning object management

 Allows attaching, associating, and consolidating diverse content
pieces into a single course (for example, core course content
delivered as eLearning with auxiliary PDF and video resources
included separately)

 Includes ways to link content and assignments in blended

learning courses so that it is clear that the components are part
of a single course and can be assessed and tracked as such

Choosing an LMS.docx page 102 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Provides a way to centrally manage links so that changes can

easily be made to external URLs associated with learning
objects.

 Is optimized for reusability in general (not just measured by

SCORM support). Some LMSs have their own internal content
repository that allows internal mixing and matching of objects in
designing a course, curriculum, or learning track.

 Allows associations (opaque to learners) between test items,

external references, and internal content repository items to
specific content objects, for update and audit purposes

5.14 Delivery architecture

 Supports a wide variety of delivery architectures. For instance,
an eLearning architecture involving a content repository that
may be on a different server than the LMS and is supplied by
another vendor.

 Can deliver and/or manage as large a variety of learning objects

and activities as possible based on such fundamentally different
approaches as:
o Informal learning
o Constructivist learning environments (problem-centered,
discovery-based, game-based, inductive, etc.)
o Collaborative/social learning
o Blended learning
o Game-based learning

 Can be configured (via proxy server, etc.) to avoid the cross

domain scripting issue so that courses not residing in the LMS
domain can be launched. This includes launching courses from
content repositories in different locations within the corporate
intranet, as well as on the Internet.

 Can provide an audit trail for required deployments of mandated

training (for example, compliance training)

Choosing an LMS.docx page 103 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Provides integration with social networking services (e.g.,

Facebook, Twitter)

 Allows delivery of a wide variety of content in diverse file

formats and authoring tools for delivery to learners as either
embedded into eLearning modules or separate learning objects

 Has offline player capabilities (see 4.8 Offline content

provisioning and player capability). Content should look and act
the same in both online or offline modes.

 Offers a browse mode whereby testing requirements are

suspended (for learners who have already taken the course).
This may be handled through content functions or standards
like SCORM.

 Allows quick and easy access and launch of short, just-in-time

performance support modules. The process of finding and
launching these should be easier than normal eLearning since
users will often need to launch these while performing a job
task; they should not be demotivated to do so by a cumbersome
process. These are normally handled differently from regular
eLearning courses for this reason, and because they do not
normally include assessments.

 Launches courses cleanly and easily, regardless of their source

(COTS or Gov’t developed, LMS server or other server)

 Includes configuration management and version control

features for content. This includes, for example, checking files in
and out to prevent accidental overwriting, and revision tracking
to audit changes and roll back to earlier versions.

 Supports delivery to mobile devices (see section 3.6 Mobile

learning), including responsive design approach to the LMS
interface.

 Permits bookmarking locations in courses and other content as

well as storing commonly accessed screens in the LMS as
favorites

Choosing an LMS.docx page 104 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Supports e-commerce charging for content (if the LMS will

deliver content on a paid basis to “extended enterprise”
customers)

 Includes a user system requirement checker that tests learner

systems for appropriate plugins (and versions) before courses
are launched. The LMS should not allow you to launch content
unless it passes the test.

 Opens a minimum of windows to deliver courses. Some LMSs

open chains of 3 or 4 browser windows just to deliver a simple
course. If the user inadvertently closes one of these windows, it
may cause the course to stop functioning.

 (for VLEs) Allows the instructor to pull whiteboard contents from

each breakout room into the main virtual classroom.

 (for VLEs) Allows more than one whiteboard at a time to be

displayed.

 (for VLEs) Allows creating learning objects either on the fly

during the live virtual session or in stored in advance (and
stored persistently within the classroom).

 (for VLEs) Allows more than one instructor to present at once

(via talking head videos and screen controls).

 (for VLEs) Allows pausing the video (but not audio) portion of a
live session in order to conserve bandwidth.

 (for VLEs) Has the ability to quickly flip back and forth between
showing a webcam of the presenter only, show content only, or
both simultaneously.

 (for VLEs) Provides APIs that allow embedding of elements (esp.

instructor video) into interfaces for other systems (e.g., lmss,
learning portals)

 (for VLEs) Facilitates storing and delivering recorded videos

repurposed from recordings of live virtual classroom sessions,
or created as asynchronous content

Choosing an LMS.docx page 105 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 (for VLEs) Can easily work with elearning authoring tools, in

order to provide content objects that can be delivered
asynchronously before or after the live session, or that are
designed to be displayed by the instructor during the live VLE
session.

 (for VLEs) Allow not just local screen sharing in order to project
content, but more bandwidth-friendly server side delivery of
content, such as running a PowerPoint file from the server
rather than the instructor’s computer.

 (for VLEs) Has a robust set of whiteboard tools, to allow the

instructor to draw complex diagrams, etc. on the spot.

5.15 Cost
 Costs less for the base application license compared to the cost
of other similar systems with similar capabilities and feature
sets. This includes all TCO (total cost of ownership) costs.

 Has a licensing agreement that is flexible and easily scalable to

reflect changing numbers of learners and administrators. This is
especially important if you project substantial growth in your
organization, or have “extended enterprise” users (see 7.7
Extended enterprise learning for more details)

 Allows you to meter usage of the system by individual business

units, so that you can spread the cost fairly

 Costs less for recurring and ongoing support compared to the

cost of other similar systems

 Costs less for the database (if included separately) compared to

the cost of other similar systems

 Is projected to cost less for required customizations compared

to the cost of customizations for other similar systems

 Costs less for add-ons such as APIs to external applications

compared to the cost of other similar systems

Choosing an LMS.docx page 106 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Offers hosted (also termed SaaS or cloud) and/or component-

based architecture solutions to take advantage of these
potentially cost-saving options (see 4.10 Hosting options and
7.13 Component-based architecture for details)

 Costs minimally extra for separate test, staging, etc. instances

of the product (see 4.14 System environments)

 Uses or can use open source components (e.g., MySQL) that

can significantly reduce costs

 Has a vendor who is open to cost sharing arrangements. If you

are planning to make extensive customizations, discuss with the
vendor possible partnering on the development and/or cost of
such changes so that the cost or development can be shared
with the vendor and/or other customers, if other customers who
have purchased the vendor’s product will receive the new
functionality. It is standard practice for vendors to use customer
requests for customization as an economic basis for their
development of new system features, such that the cost of
developing these features (that are included in system upgrades
that everyone gets) is effectively funded by these customers.

5.16 Assessment authoring and delivery

 Provides an internal function to create and deliver a wide variety
of assessment types (with template options). See 4.16 Internal
assessment authoring for more details.

 Can export assessments created within the LMS for use in other
content or LMSs. Assessments created in the system must be
interoperable (using a standard like the SCORM cmi.interactions
data element) in order for this to happen.

 Allows authoring and delivery of a variety of assessment item

types, including:
o Multiple choice (both single and multiple answer)
o Fill in the blank
o Matching
o Drag and drop

Choosing an LMS.docx page 107 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Ranking/ordering
o Image selection
o Word scramble
o Labeling an image
o Essay or short answer (usually requires instructor
intervention to score answers). These should be implemented
as online forms/ fields for easy review, not downloadable
Word documents.

 Includes a grade book function for instructor-led or blended

courses/assignments, with possibility of revising grades

 Allows grades to be assignable to a variety of objects, including

discussion and blog postings

 Provides the ability to export and import from Excel or CSV into
the grade book

 Provides ability to selectively publish assessments based on

parameters such as:
o Date
o Time
o Learner ID
o Grade book criteria
o Multiple criteria

 Tracks time it takes for learners to answer questions via time

stamps, including time stamps on essay submissions

 Includes display options such as all test questions on a single

page vs all on one page

 Can be set to allow navigation back in order to change previous

question vs forward only

 Can set up sections in tests which can be hidden from users for
easy test management

Choosing an LMS.docx page 108 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Includes management features for group assignments, and

apply assignments/topics to multiple groups at once

 Assessments can be set as either summative (for grading

purposes) or formative (self-assessments), with different
features applied to each (related to when and how the learner
can view them)

 Provides a rating or assessment function for mentor/coach/OJT

assessments

 Has flexible options for establishing how and when the

assessment is delivered (e.g., as a self-check quiz, or end of
course test)

 Allows input/upload and management of essay questions,

including use of rubrics to grade sets of questions and flexibly
apply the rubrics across multiple learning objects, semesters,
etc.

 Allows instructors to view and grade essay assignments online

without having to download them as files

 Allows embedding of videos, animations, audio, and graphics

into assessments

 Can be configured to remediate learners to particular content or

locations in content based on assessment results

 Allows importing sets of questions formatted in a standardized

format (e.g., QTI)

 Randomizes the order of questions within an assessment and

the answers within a question

 Allows establishing a standardized student remediation

process/path that is invoked when a student fails an assessment

 Can be set to provide feedback optionally, and after each

question is answered vs at the end of the assessment

Choosing an LMS.docx page 109 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Has an interface for setting up rules for determining what

happens as a result of learner result on assessment (i.e.,
failed=forced remediation, another try, etc.)

 Allows creating pools of questions per objective, such that

different items can be presented in subsequent learner tries (i.e.,
learners won’t see the same question twice)

 Allows assigning weights to individual answer choices for

assessment elements, so that it:
o Gives learners partial credit for correct answer chosen in an
assessment element with multiple correct answers
o Penalizes learners more for answers that are obviously
incorrect

 Allows customizing hints and feedback

 Allows browser lockdown techniques, such as that provided by

Respondus Lockdown Browser®

 Incorporates plagiarism detection software

 Allows score display options other than percentage correct (e.g.,

“9 out of 13 correct”)

 Allows weights to be assigned for individual questions so that

more important questions contribute more to the learner’s score

 Allows use of proctor passwords

 Allows use of enhanced assessment auditing so instructors can

see concurrent logins from different IP addresses

 Allows IP restrictions to ensure assessment can only be taken in

certain locations (for instance, a computer training facility)

 Allows setting of assessments to either practice or test mode

 Allows export of assessments to Word (for printing)

Choosing an LMS.docx page 110 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Supports accepting informal learning assessments such as

learner Web pages and collaborative projects with multimedia,
with ability to set and change file size limits for these

 Allows use of math symbols, i.e., equation editor, while

authoring assessments

 Supports the ability to tie assessments to learning objectives

and outcomes that are input into the system

 Supports the use of learner portfolios for informal learning

assessment, either internally or through system integration with
a third party product. These portfolios should be viewable after
learners have graduated or finished taking an associated
course. Portfolios may need standards support, such as Open
Source Portfolio Initiative (OSPI) or TaskStream.

 Supports the ability to set conditions (i.e., hide/show, dates) for

multiple assessments at the same time

 Assessments created by the LMS assessment authoring tool are

Section 508 compliant. This includes the ability to set different
time constraints for individuals with disabilities who need more
time than other learners.

5.17 Mentoring, coaching, and other developmental scenarios

 Supports infrastructure for managing (assigned on an individual
or class basis) of:
o Mentoring
o Coaching
o Groups
o Projects
o OJT
o Shadowing and apprenticing
o Rotational assignments
o Career programs
o Conferences/forums/seminars/workshops

Choosing an LMS.docx page 111 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.18 Collaboration and communication options

 Allows learners to take notes as they interact with learning
materials. These notes should be persistent between sessions
and automatically associated with locations in the content
(possibly with a “sticky note” look and feel). If the learner
wishes, their notes can be posted, either internally in the LMS,
or publicly outside of the LMS, through APIs to applications like
Twitter and Facebook. They should also be able to be exported
(as a text file).

 Includes collaboration functions to enable users to

communicate with each other, instructors, course
administrators, system administrators, etc. These functions
typically include the following, and are especially common and
relevant to VLE software:
o Email (including group lists)
o Email attachments
o White boarding
o Chat/Instant messenger (IM) (especially for synchronous Q&A
sessions in VLEs)
o Blogs
o Microblogs
o File sharing
o Recording of VLE sessions
o Virtual hand-raising (during synchronous VLE learning
events)
o Passing presentation rights and application sharing (during
synchronous VLE learning events)
o Breakout Rooms (during synchronous VLE learning events)
o Content Annotation
o Ask the Expert exchanges
o Q&A (essentially an open peer help forum)
o Journals
o Threaded discussion (aka forums or discussion boards)

Choosing an LMS.docx page 112 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Desktop sharing
o Community calendar
o Community of practice (CoP)
o Social networking (including as backchannels during VLE
learning events)
o Instant messaging
o Learner-created personal web pages
o Dedicated team spaces. Members/teams can be comprised
either of learner cohorts taking the same course, or functional
teams within the organization.
o Surveys (within and outside of learning events). This includes
spot polls of learners during VLE presentations.
o Peer rating of content
o Webcasting, with the ability for learners to initiate sessions
among themselves (i.e., not just one-way, instructor to learner
webcasting)
o Learner to learner whiteboard (ie, not just instructor to
learner)
o Learner posting of web pages
o Learner creation of videos
o For discussion forums and blogs, provides the ability to:
 Attach documents associated with a posting (learners)
 Embed links (learners)
 Allow creation of groups (instructors)
 Make postings anonymous and/or private that cannot be
viewed by anyone higher in the organizational hierarchy
(i.e., vertical social network)
 Search (learners and instructors)
 Set release conditions (instructors)
 Moderate (instructors)
 Perform “noise management”, e.g., limiting views of
conversations, hide discussion threads (instructors)
 Use rubrics to grade postings (instructors)

Choosing an LMS.docx page 113 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Flexibly configure options to allow and disallow learner

actions (instructors). For instance, controlling whether
learners will be allowed to create, view, or add to new
discussions or threads, or if they are allowed to delete their
own comments.

 Provides the ability to send SMS text messages to learners (as

either system alerts or actual learning content—note that a
considerable amount of higher ed online learning in developing
countries is done through SMS)

 Provides robust WYSIWYG formatting options (including direct

access to HTML code) for collaboration functions such as blogs

 (for LMSs with VLE capability) Allows instructors and learners to

launch virtual classroom/virtual collaboration sessions with
screen sharing, video, etc. on the spot when needed.

 If file sharing is provided:

o Allows learners to include comment tags
o Allow check-in and check-out version controls
o Incorporates a user rating system (for relevancy, quality, etc.)
o Offers a file storage area for each learner
o Offers direct P2P (peer to peer) sharing options
o Accepts a variety of file types, such as;
 PDF
 Video
 Web pages
 MS Office
 Zip archives

 Provides specific functions that enable learners to provide

feedback on the content

 Provides a variety of asynchronous distribution mechanisms for

content, including email attachments, RSS feeds, and podcasts

Choosing an LMS.docx page 114 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Integrates with collaborative tools such as Google Apps, Google

Drive, and Google Plus so that learners can work collaboratively
on projects and attend virtual project meetings.

 Incorporates a recommender system either based on:

o User ratings
o Predefined learning paths
o System inferences based on info in user profiles,
performance data, etc.

 Provides summaries of user ratings and other paradata tags,

with associated comments

 Provides the ability to display a welcome message upon user

login, with info on where to get plug-ins, announcements of new
content, etc.

 Provides a means for learners and instructors to easily and

quickly annotate learning objects with short videos (especially
including videos captured on their mobile device), rather than
being required to embed them into the learning objects
themselves. This includes short, talking head elaborations of
course elements as it is playing as well as course structure (i.e.,
within the navigation functions of the LMS).

 Includes online conferencing capability (this is standard for

VLEs, but not for LMSs)

 Includes logistical communication functions such as a course

calendar and learning assignment pages. Calendars should filter
items for relevancy to the role of the person logged in.

 Effectively manages authorization/authentication; manages

access to materials and conferences

 Provides social media learning functions that can be integrated

into the curriculum to provide social media-based learning
assignments.

Choosing an LMS.docx page 115 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.19 Competency management and development/learning

plans
(NOTE: these are beyond most LMS capabilities as of this writing.
They are included here to guide you as to what advanced features
you might want to look for. For information on these features as an
emerging trend, see 7.12 Adding competency analysis tools).

 Supports competency management and Individual Development

Plan (IDP) HR enterprise infrastructures

 Supports competency-based education, showing progress

towards competencies

 Automatically links training interventions and competency

objects based on user approval

 Features advanced natural language matching algorithms and

associated linking functionality

 Allows learners to create and manage e-portfolios

 Maps individuals/groups to a course/curriculum dynamically

based on rule sets determined by enterprise requirements

 Operates as a standalone product, so that linking training

interventions to competency objects can be performed off-line
and then ported to the LMS

 Uses a variety of competency frameworks, providing a range of

choices for methods of measuring competencies (for example
360-degree Feedback)

 Uses a variety of competency rating scales

 Includes built-in Update functions to reconcile linkages due to

changes in training interventions or competency objects
(additions, deletions, or just word changes)

 Imports/exports competency-related data in common database

formats such as XML or MS Access

Choosing an LMS.docx page 116 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Can provide IDP progress, training completion, and other related

input to competency management, performance appraisal and
other HR components of other systems

 Can provide automated analysis/assessment survey of

employee’s current and anticipated skills and competencies.
Gaps are identified with appropriate courses indicated to
address closure of gap(s).

 Provides career tracking tools for learners to set and monitor

progress towards career objectives

 Prioritizes competencies and courses based on changes in

career, regulations, funding, or organizational vision/mission

 Can import competency inventories and rubrics as well as

learner data from external systems

 Supports individual development plans (IDPs) with the following

options:
o Dynamic IDP that is updated as employee registers, attends,
completes, or does not complete approved training
o Certification/recertification schedules and notification
o Competency decay refresher
o Mandatory/optional training requirements
o Ad-hoc/emergent training requirements
o Full reporting capability
o Compatibility with any competency framework
o Compatibility with any competency rating scale
o Performance thresholds (times to complete)

5.20 Learner tracking

 Capable of tracking, reporting and storing a wide range of
learner performance data by individual, by group and by cohort
groups

Choosing an LMS.docx page 117 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows tracking via a flexible, interoperable standard such as the

xAPI, and uses a published, industry-accepted controlled
vocabulary for xAPI verbs. Note: This requires an internal LRS.

 Contains an electronic training jacket providing a view of career

information, including:
o Training and education
o Qualifications and certifications
o Career history
o Advancement status
o Awards

 Tracks learner data while learners are taking eLearning in a wide

variety of file formats and authoring tools

 (for VLEs) Track “engagement” of users and groups of users

(and reports to the instructor via a dynamically updated
dashboard) using parameters such as:
o User participation in chats
o Number of answers to questions posed by the instructor
o Interactions with other users within breakout meeting rooms
o Status updates

 For LMSs that incorporate social media and other features to

support informal learning, tracking includes “social monitoring”
tools that track:
o Conversation rate (volume of comments about learning
objects)
o Amplification rate (volume of sharing of learning objects)
o Applause rate (volume of “likes” posted for learning objects)

 Includes the ability to add custom fields to track additional

learner information, so that they can be included in analyses and
reports

 Includes RESTful APIs that allow custom views of LMS analytics

Choosing an LMS.docx page 118 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Tracks accredited learning units, for instance, continuing

learning units (CLUs), continuing education units (CEUs), and
continuing professional education (CPEs)

 (for government installations) Includes the ability to “federalize”

data to store SSNs (encrypted), name, CPOID, Activity,
Organization, Pay, Occupational Series, Grade and other
identifying government information. Note that this information is
subject to Personally Identifiable Information (PII) restrictions;
the LMS should have security measures in place to protect it.
See http://csrc.nist.gov/publications/nistpubs/800-122/sp800-
122.pdf

 Provides the ability to print a variety of tracking-related items,

including test scores

 Can track a wide variety of relevant items, including:

o Enrollments
o Withdrawals
o Launches
o Completions
o Attendance (for ILT courses)
o Competencies acquired
o Use of materials
o Evaluations
o Grades
o Assessment scores

 Allows a learner to view their own online course results on a

lesson-by-lesson basis as well as:
 Time spent
 Date and time last accessed
 Number of test tries
 Course grade

Choosing an LMS.docx page 119 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

5.21 Certificates, forms, polls, and surveys

 Allows administrator design/upload and learner delivery of
course completion certificates

 Includes electronic signature capability on external form(s), for

example, the government SF-182. Signature features for
government installations should include SSL, PKI, and
encryption for all authorizing levels.

 Allows easy printing of certificates, surveys, and evaluations

 Provides survey functions as follows:

o Create and edit
o Import
o Copy
o Define properties
o Preview
o Define survey link location (for embedding survey in
eLearning, website, sending by email, etc.)

 Has different options for design of surveys, such as multiple

choice, Likert scale

5.22 Interfaces with external systems and applications

 Includes data migration tools for moving data permanently from
a legacy system to the new one

 Interfaces with systems that you might have in your enterprise

such as:
o HR systems
 HR database
 Performance management systems
 Talent management systems
o Enterprise resource planning (ERP) systems such as such as
SAP®
o Intranets

Choosing an LMS.docx page 120 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Marketing,sales, CRM, and financial platforms (especially

important for “extended enterprise” LMSs)
o Content repositories
o Document management systems
o Learning systems
 Learner registration system
 Collaboration tools
 VLE
 CrMS
 LCMS
 Electronic libraries
 Third party course content
 Webinar systems
 Another LMS or system that you will need to import legacy
learner tracking data from
o IT administrative systems
 Authentication systems
 Authorization systems
 Data validation systems
 Email directories

 Imports and exports to external systems in real-time and batch

mode. This data typically includes not only learner
demographics and identification but such things as
competencies, certifications, and IDPs (individual development
plans).

 Enables add-ons and integration using an open architecture (see

7.5 Open architectures for more details)

 Supports use of a third-party database (in compliance with Open

Database Compliancy (ODBC) requirements

 Import and export of learner and course tracking data using

standardized data interchange formats (e.g., XML, JSON, CSV)
without writing high-LOE integration applications

Choosing an LMS.docx page 121 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Interworks with other systems that manage and deliver training,

such that content can be accessed on another system (for
example, an LCMS’s content repository). This includes
repositories of commercial eLearning courses such as
PeopleSoft® and digital reference libraries such as Safari Books
Online®

 Has the ability to call external applications and code objects

(such as calculators and random number generators), and set
up interfaces to read and write from databases

 Is interoperable with a variety of authoring tool(s), including

direct import from the authoring tool into the LMS. It is important
that you determine which tools your content authors prefer to
use and ensure compatibility with those tools.

 Includes “widgets” (add-ons) that allow the learner to access

search engines, maps, social media sites, etc.

 Allows student accounts to be securely created, managed and

disabled by a connection to an external system of record

 Integrates with Dropbox or other cloud-based file transfer and

storage system (to enable easy transfer of learning object and
media files from developers to LMS administrator)

 Includes automatic learner registration of new hires based on

data that is input to HR system

 Links to employee records in an external system

 Deletes learner ID and training records when employees

terminate based on action in the external system

5.23 Metadata support

 Supports the kind of metadata your organization needs or uses
(LOM, Dublin Core, etc.). See 4.15.4 Standards for metadata for
more details.

Choosing an LMS.docx page 122 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Includes a convenient mechanism for adding metadata or

descriptive labeling to not only courses, but also to other
objects (SCOs, files, activities, etc.)

 Uses metadata to search the course catalog(s)

 Presents options for display of metadata to learners and

administrators at relevant nodes in their workflow

 Allows configuring how metadata tags will be input by content

developers (checkbox, date, popdown, text box, etc.)

5.24 User profiles

 Has the ability to manage profiles for organizations, not just
users

 Has the ability to matrix learner characteristics demographically,

organizationally, etc. (for example, assign learners to more than
one job role, in more than one organization)

 Can be searched on any field

 Incorporates a recommendation engine that pushes

recommendations for learning objects to learners based on
attributes of learning the learner has consumed, according to
their profile. This could function in much the same way as
Amazon.com recommendations.

5.25 Resource management

NOTE: these requirements apply mostly to CrMS products

 Includes features for:

o Scheduling of operational and training assets
o Weapons practice ranges (DoD)
o Expendables
o Computer learning facilities
o Staff assignments
o Planning training events

Choosing an LMS.docx page 123 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Displays filtered view to learners of relevant items above

5.26 Analytics and reports

 Offers a wide variety and number of predefined reports

 Offers permission levels with different kinds of access to

reports.

 Offers flexible, robust abilities to create custom reports, both

internally and by using external tools (including those supplied
by other vendors such as Crystal Reports®)

 Prints reports easily, with appropriate options

 Provides capabilities to:

o Administer and maintain performance and evaluation metrics
o Track individual and group usage statistics
o Integrate evaluation forms internal and external to the
courses
o Perform statistical analysis on the database information
o Report on learner performance data by individual and group
o Easily perform summative evaluations of courses

 Provides direct access to tables used within the LMS for

developing queries and reports. This should be documented in
table and data structure specifications provided with the
product. This is usually a requirement for government
installations.

 Provides reporting on certification status of groups and

individuals, including upcoming renewals, missed renewal
deadlines, etc.

 Provides ways to incorporate data from external systems to

produce reports and analytics that show “big picture” measures
of employee learning progress activity across all knowledge
transfer mechanisms

Choosing an LMS.docx page 124 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 (for “extended enterprise” LMSs) Includes ROI analytics-

oriented reports

 Provides content-centric reports and/or visualizations that

include at a minimum:
o Average time spent by users in the content
o First and last access of content
o Specific sections of content accessed/not accessed, and
duration for each visit
o Number of downloads or user access
o Number of attempts to pass (on an individual assessment
basis)
o Successful attempts to pass (on an individual assessment
basis)
o Type of content (media file, elearning module, etc.)
o Expiration or deadline for users to take content

 Provides user-centric reports and/or visualizations that include

at a minimum:
o Date items (includes content objects, tracks, etc.) assigned
(by admin or instructor)
o Date of start by user
o Date of completion by user
o User score
o Result

 Includes business impact reports for managers that includes at

a minimum:
o Competencies and certifications met by employees
o Progress of employees towards meeting their IDPs (Individual
Learning Plans)
o Contribution of learning towards meeting business objectives
(this may require system integration with ERP systems, or
talent management system)
o Course completions and “not started” courses (especially for
mandatory compliance courses)

Choosing an LMS.docx page 125 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Expired certifications

 Integrated with an xAPI Learning Record Store (LRS) – either

embedded as part of the LMS or an ability to communicate with
one as an external service.

 Includes analytics engines that include graph charting and

visualization options

 Offers features addressing the range of analytics maturity

models (NetDimensions, 2015)
o Passive reporting – canned reporting and dashboards
o Proactive reporting – KPI reporting and dashboards
o Siloed analytics – domain-specific relationship analysis
o Integrated analytics – across HR/talent domain and business
aligned/ connected
o Predictive analytics – dynamic future scenario modeling
o Machine intelligence automation – dynamic automated data-
drive decision making and machine action

5.27 Ease of use for administrators

 Is easy to learn and use, with the ability for users to choose from
tiers of features according to the knowledge and expertise of the
user. This allows users to start using the program quickly and
gradually progress to more complex authoring tiers/feature sets
as their skills mature. In other words, users only see features
that are relevant to their level of skill and the kind of operations
they are capable of performing. Ease of use for administrators is
important since it can reduce the skill set requirements and thus
the cost of administrators.

 Provides user interface customization (not on the level of tiers of

features, as above, but on an individual feature basis), so that
both learners and administrators can optimize for their particular
needs

 Can easily create and restore archives of system (e.g.,

transactions), user (e.g., profiles, enrollments), and content data

Choosing an LMS.docx page 126 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

(e.g., courses and curricula) in a proprietary (such as

Blackboard Vista .bak) or open format

 Is easy to install and reconfigure

 Manages the administration process efficiently with built-in

workflows (for approvals, for instance)

 (for VLEs) Allows assignment of learners to breakout rooms

either randomly by the system (evenly distributed) or on an
explicit individual basis.

 Administrative interfaces are clear, simple, and optimized for

usability. Administrator interfaces are no less important than
learner interfaces. Just because learner interfaces are well-
designed does not mean the administrative interfaces will be
also(!). This is particularly important where there is a need for
non-technical staff to perform administrative functions (such as
for instructors to pull reports and configure courses).

 Includes options for remote administration from outside the

enterprise intranet (through the Internet) and possibly via a
handheld device

 Provides features that allow administrators to view role

structures in a graphical representation (diagrams, outlines,
etc.)

 Provides inline grading tools that allow instructors to grade

learners on the same screen as where the assignment appears

 Provides instructors with the ability to access grading tools

from their mobile device

 Provides clear, specific error messages that aid in

troubleshooting. A generic message that is the same for all
errors is not acceptable. You also want to avoid cryptic,
technical messages that can only be interpreted by the LMS's
software developers. Messages should be understandable not
just to technically inclined LMS administrators, but also to
content developer s. Also, it is ideal for error messages to vary

Choosing an LMS.docx page 127 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

depending on whether you are in the test vs. the production

system.

 Has a feature to store favorite locations within the system

 Allows saving of a workspace

5.28 Ease of use for learners

 Displays interfaces that are consistent and standardized
throughout all screens

 Provides a dashboard that indicates to the learner their progress

towards meeting their learning goals, as well as common system
functions.

 Uses straightforward, simple, and intuitive paths for performing

administrator and learner job task functions. You should test
your most common and important use cases on the system to
verify this. See 8. Process for choosing an LMS, step 12.

 Has a feature to store favorite locations within the system

 Where external content (such as Skillsoft courses) is provided

and resold, a shopping cart feature allows users to organize and
store their purchases.

 (for VLEs only) Allows users to use their telephone rather than
VoIP (computer microphone and speakers) to make an audio
connection to a session. The telephone numbers provided by
the system should be either local or toll free.

 (for VLEs only) Incorporates an option for the system (upon

entering a session) to call the user back to initiate the audio
connection, rather than requiring the user to call in from their
phone and possibly incur long distance charges.

5.29 Transcripts and other documentation

 Allows learners and administrators to print transcripts, course
completion certificates, and learner records with appropriate
options

Choosing an LMS.docx page 128 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Allows a learner to be able to view a transcript of all training that

has been recorded in the LMS for their account along with status
and status date

 Incorporates the ability to import and store information about

learner badges and for instructors to create them, using a
framework such as Mozilla Open Badges. See 7.21 Digital
badges for more information.

5.30 Scalability
 Has a scalable architecture that allows the system to expand as
the number of users increases. The following factors should be
taken into account in your planning:
o Number of concurrent users (current and in the foreseeable
future)
o Database licensing (by seat or site)
o Database volume restrictions

 Has a scalable architecture, enabling evolution of the client

installation without forcing them to go through frequent major
version upgrades

 Allows configuration of a data distribution network (interconnect

distribution peers through a common distribution server)

5.31 Vendor characteristics

 Has a good reputation among acquisition and system owner
communities. Ask the vendor who their other clients are, what
they use the system for, and see if you can talk to these clients
about their experience using the system. Look for negative
comments posted on the Internet by members of these
communities.

 Is willing to openly discuss with you changes in the learning

landscape, and their roadmap for adapting to these changes.

 Has been in the LMS market for at least 5 years. Avoid the first
release of a new system.

Choosing an LMS.docx page 129 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Has not created the product merely as an add-on to an ERP or

some other system, in order to be able to sell it to customers
desperate to add an LMS to their existing system. Although the
cost will probably be lower than purchasing a separate LMS, and
the system will obviously be well-integrated with the ERP, it can
mean that the LMS receives short shrift in design and usability.

 Has a clear technology roadmap with a reasonable time frame

for new versions and additions of new features

 Has consulting experience and arrangements, especially with

complex issues such as data migration and change
management

 Listens to your concerns during interactions with them,

especially during demo sessions of their product. How they are
in these situations probably reflects how responsive and
attentive they will be to your concerns as a customer.

 Is financially sound and not in danger of going bankrupt. You

may want to consider acquiring Dun and Bradstreet reports for
your final vendor candidates, to establish the financial health,
stability, and long term business strategy of them.

 Is of a stable size, as measured by number of employees, annual

revenue, capitalization, etc.

 Has a robust ongoing budget for R&D

 Has a large number of successful clients. Who the clients are

and their industry stature can be important, especially in terms
of their similarity to your mission or infrastructure. If you can,
find out the number of total users served by the LMS product
within this client base.

 Is not about to be acquired or merged with another vendor.

Obsolescence and durability is an important consideration in the
fast-changing landscape of LMSs and enterprise systems in
general. You don’t want a vendor that gets bought out by
another company, and your LMS, with all of your expensive
customizations, no longer functions because it has been

Choosing an LMS.docx page 130 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

reengineered to conform to the acquiring vendor’s architecture,

or worse, has been withdrawn from the marketplace because it
is redundant with a product that the acquiring vendor already
has in place.

 Has worked with many content developers using a variety of

different kinds of content. Ask for references at organizations
that have deployed content similar to yours.

 Is familiar with your business model, market, and content types.

 Is International Standards Organization (ISO) and/or Capability

Maturity Model Integration (CMMI) certified to ensure high-
quality software development output

5.32 User training, technical support, and documentation

 Has robust support for training of all categories of users:
learners, instructors, system administrators, content managers,
etc.

 Has robust support documentation in a wide variety of forms

including tutorials, help, examples, references, and user
manuals

 Has a variety of Help Desk support options for administrators

and learners (telephone, chat, email, etc.). These need to be in
synch with the way your organization normally requests help.

 Offers tiers of support and training (available bundled with the

product or purchased separately from it) so that you are not
forced to spend more for support than you need

 Has a Help Desk system that is structured and process-driven

via trouble call tracking, ticketing, and reporting

 Has Help Desk support that coordinates problem resolution with

the appropriate parties: vendors, SME’s, etc. for problem
resolution

 Has knowledgeable, experienced support personnel

Choosing an LMS.docx page 131 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Is available as close to 24/7 and world-wide as possible

 Offers extensive training options: eLearning, video tutorials, ILT

sessions, webinars, etc.

 Has onsite training options. If training is at vendor site, the

location(s) are a reasonable distance.

 Includes an orientation tutorial for new users

 Has a low average turn-around time for Help Desk support

 Has a feedback function for suggestions on improving the LMS

 Provides technical consulting services options for

customizations, implementation, configuration, architecture
design, needs analysis, change management services, etc.

5.33 Media and content support

 Provides support for industry-standard streaming protocols for
audio and video

 Minimizes latency in delivering high bandwidth media through

robust network throughput and streaming server capacity

 Provides a library function for upload and tracking of user-

generated, internal media (especially videos), or provides direct
access to web-based media (such as videos on YouTube)

 Provides an enterprise Glossary function

 Has robust support for mobile devices and mobile content,

including:
o Adaptive device detection and delivery according to the size
of the device’s screen
o Content provisioning (i.e., while the user is on a free wireless
network so they do not incur cellular data download costs)
o Flexibility in dealing with sudden loss of connectivity

Choosing an LMS.docx page 132 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Seamless mobility from one device to another (i.e., leaving

smartphone and picking up where left off in content on
desktop computer)
o Location-based and context-aware delivery of content
o Spaced learning
o Mobile-optimized just-in-time performance support and
references (e.g. infobases)
o Social media tools optimized for mobile use
o Drill exercises (e.g., electronic flash cards)
o Mobile collaboration platforms that provide access to experts,
mentors, and communities of practice
o Content management for user-generated content (e.g., photos
taken on the mobile device), perhaps via integration with a
3rd party service such as DropBox.
o (for instructors) Grading

 Has gamification options that can be applied to imported

content or content built within the system such as:
o Points
o Levels and achievements
o Badges and trophies
o Rewards and incentives
o Game profiles
o Varied leaderboards
 Master
 Group
 Challenge-based

 Provides support for podcasts and vodcasts

 Supports employee-generated video sharing across the

enterprise with the following features:
o Setting permissions for user groups
o Setting level of user authentication
o Providing security of delivery of the video stream.

Choosing an LMS.docx page 133 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

o Video streaming to prevent videos from being downloaded

into the viewer’s computer.

 Supports immersive learning content (simulations, serious

games, virtual worlds). See 139 Support and optimization for
virtual immersive environments (VIEs) for more details.

 (for VLEs) Provides the ability to push screen shots of the

facilitator’s screen to participants

 Supports interactive video formats with following features:

o Overlays
 Assessments
 Links
 Chapters
 Invisible hot spots (popup additional content or
annotations)
o Multiple timelines for different decisions
o User annotations
o Social video
 Comments scroll as video plays
 Comments can be bookmarked
 Learner participation in commenting on video fills
important content gaps

 Supports a wide variety of media (see below) and media file

formats. Examples include:
o Audio
 MP3
 RealAudio
 WAV
 AAC/MP4
o Video
 MPEG-4
 RealVideo
 Quicktime
 AVI
 WMV
 FLV

Choosing an LMS.docx page 134 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 H.264 (high definition)

o Documents
 Microsoft Office
 Adobe PDF
 HTML5
o Graphics
 JPEG
 PNG
 GIF
 SVG
o 2D animation
 SWF
 HLA Simulations
 HTML5
o 3D animation
 SWF
 WebGL

6. Popularity of features and capabilities

Roche & Upton (2013) conducted a survey across government and
corporate enterprise learning audiences, which resulted in a list of the
most popular “must-have features” among those planning to add,
change, or replace their current LMS/TMS, as follows:
Course tracking and completion..........................91%
Web conferencing/virtual classroom...................78%
Testing & assessment..........................................76%
Integrated reporting.............................................72%
Mobile authoring and deployment.......................70%
Course marketing and e-mail...............................61%
Virtual learning environments.............................46%
Survey management............................................46%
Social network tools............................................46%
Performance management....................................46%
SaaS or cloud computing.....................................46%
Performance review.............................................41%
Embedded training in applications.......................37%
Authoring............................................................33%
Collaborative authoring.......................................30%
On-demand option...............................................30%
Succession planning............................................30%
Mentor/coaching tool...........................................28%

Choosing an LMS.docx page 135 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Recruitment & hiring management......................15%

Compensation management.................................20%
Rapid development..............................................20%

The Brandon Hall Group (2015) reports on the popularity of features as follows:
Ease of use ..........................................................35%
Ability to adapt....................................................26%
Integration with other systems.............................21%
Personalized content............................................21%
Cost.....................................................................18%
Robust reporting features.....................................18%
Mobile learning features......................................18%
Social learning features.......................................12%
Cloud-based deployment.....................................12%

Software Advice (2015) reports on the most used features as follows:

Trainee testing ....................................................73%
Training administration.......................................68%
Record keeping....................................................53%
Virtual classroom................................................45%
Document management.......................................45%
Content creation..................................................39%
Advanced reporting.............................................30%
Mobile learning...................................................30%
Social learning.....................................................27%
eCommerce..........................................................10%
Gamification........................................................10%

Vipond and Clarey (2016) report that their 2013 LMS survey
respondents ranked eLearning delivery and tracking (73%) as the
“most important” requirement, followed by reports (69%), search
(48%), and version control/historical tracking (48%). Other highly
rated 2015 requirements for a future LMS included eLearning delivery
and tracking; version control and content tracking; email
notifications; search capabilities; and the ability to assign due dates
to required training.” (p.5).

7. Emerging trends in LMSs

7.1 Experience API (xAPI) adoption
Currently, most adopters of the xAPI (see 4.15.8 ADL Total Learning
Architecture (TLA), including xAPI for details) are focused on offering
the option of performing SCORM-like functionality using the xAPI
instead of SCORM, rather than leveraging the unique features of the

Choosing an LMS.docx page 136 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

xAPI. In order to achieve the latter, vendors will need to profoundly

rethink their LMS product model. There are three main dimensions for
the kinds of changes for LMS vendors to consider.

One relates to the LMS’s ability to track learning experiences within

content other than standard eLearning, such as mobile “learnlets”,
simulations, and games. And it needs to track it whether it is
launched from the LMS or not.

Second, it needs to track different kinds of data than is possible using

SCORM or proprietary LMS tracking capabilities. This includes such
things as:

 Attempts, levels achieved, and other milestones rather than

simply complete/incomplete or test scores

 Complex learner behaviors that are not part of formal

assessments

 Data from learning activities conducted by groups of learners

 Social media and gamification activities, if those are part of the

LMS feature set

Third, it needs to track, analyze, and report on a wide range of

administrative data other than learner performance, regarding such
things as how content is being used (including content outside of the
LMS), apparent gaps in topics and areas of knowledge, trends in
learner performance, etc.

These three dimensions are particularly apparent in regards to

informal learning that does not necessarily originate from the LMS.
Up to now, LMSs have controlled the learning space by forcing
administrators to predefine and pre-register learning experiences in
the LMS. Now, the LMS can routinely receive data (through the xAPI)
that it has not been made aware of. This gets inherently tricky in
terms of differentiating and giving credit for worthwhile learning
experiences as opposed to meaningless ones. It also may need to
account for learners (and even systems) it has never encountered
before, i.e., that are not pre-registered in the LMS.

Choosing an LMS.docx page 137 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

One issue in LMS xAPI implementations is the use of controlled

vocabularies (for verbs, mainly). This is a complex issue. In general, it
is better for an LMS to use a widely published and accepted
controlled vocabulary, such as the ADL’s controlled vocabulary
(http://www.adlnet.gov/expapi/index.html). See
https://docs.google.com/document/d/1Cu6XCoUzd9ExJILTNMCI2Gzx6
oeGaC8oSZxzbFHsMjA/edit#heading=h.gjdgxs for general information
on controlled vocabularies.

One of the main hurdles to tracking informal learning is incentivizing

learners to manually report their learning experiences outside of the
LMS or outside of any content that is instrumented to communicate
xAPI statements. A user-friendly dashboard will only go so far in
facilitating this; there needs to be a reason and real incentive
structured in the learning environment for learners to take the trouble
to report informal learning experiences.

As a business model, this probably means that LMSs need to choose

between integrating an LRS and therefore accepting xAPI statements
from all quarters of the enterprise, to serve as the authoritative source
of all learning records, or remaining as a system serving separate
learning management and delivery purposes and publishing learning
records to the LRS.

Some industry analysts such as Rustici Software (Rustici, 2014)

predict that third party reporting and analysis tools (based on xAPI)
will become a large value space for organizations, especially in terms
of specialized and niche training systems. LMSs will either need to
have their own tools or integrate with specialized independent
reporting tools.

For more information on LRSs and how they are being integrated into
LMSs, see ADL’s white paper titled Choosing an LRS (available on the
ADL web site at
https://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAnLRS.docx).

Choosing an LMS.docx page 138 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.2 Support and optimization for virtual immersive

environments (VIEs)
There is growing interest in serious games, first person simulations,
and virtual worlds. These are often called “virtual immersive
environments”, or VIEs (pronounced “vees”). LMSs are now starting
to catch up to support these technologies. Most commonly, users
want to access the functionality of LMSs and VIEs in one tool, so VIE
vendors are also trying to add LMS capabilities into their systems.
VIEs integrate LMS functionality in a variety of ways.

The simplest way is for the VIE to offer web browser capability, either
inside of the VIE itself or through a daughter window of the
application. The learner can then log in to their LMS and take
eLearning courses while in the VIE platform.

Another way is for instructors to create assessments or performance-

based assessment nodes in the VIE. Learners complete these and,
either manually or through an automated script, connect to the LMS
(or, at least, the tracking database portion of it) and communicate
tracking data. The LMS in this case does not deliver any of the
learning; it only provides the performance tracking capability. Usually
this requires extensive middleware, though some VIE vendors are
working to include connectivity to selected LMSs.

The above cases start with a user who is operating within the VIE
platform, who then makes the connection to the LMS. The other way
around is also possible, but much less common because it is
technically more difficult: launching a VIE as a learning object from
within an LMS. The difficulty lies in the fact that most VIEs require a
special player, and a VIE server to manage the experience, especially
if multiple concurrent users are involved, as in a virtual world.

A proof of concept called Sloodle is an example of this scenario

applied to a virtual world. Sloodle integrates the Moodle LMS and the
Second Life virtual world by packaging a learning exercise in Second
Life into SCORM eLearning using Second Life scripts and Sloodle
middleware.

One key stumbling block to the “VIE learning object inside a LMS”
scenario is the lack of standards for the middleware and file formats

Choosing an LMS.docx page 139 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

that are needed to be able to import a course containing VIE-based

learning objects into an LMS, and have them launched through the
LMS. There are attempts currently to be able to author VIE learning
objects outside of the VIE platform software. This approach has the
potential to be platform-independent.

The other stumbling block is simply the different paradigms of

learning that each platform (VIE vs. LMS) is optimized for. For
instance, LMSs are designed to afford individual learning
experiences, whereas many VIEs (especially virtual worlds) are
designed to afford shared learning experiences (potentially with high
numbers of participants); LMSs are designed for linear learning paths,
whereas VIEs are designed for non-linear learning paths (often
determined by many performance parameters based on dynamic
events in the VIE). Finally, VIEs vary greatly in their implementations,
from single-user structured to massively multi-user open
environments. This presents a challenge in defining a universal
method for LMSs to integrate with these products.

One logistical requirement for LMSs in supporting VIEs is that

concurrent multi-user VIEs require setting up cohorts in advance, for
example, work teams that will progress through a game-based
exercise together. Automating the assignment of these cohorts
(based on business rules, such as homogeneity of job roles) may not
be that different than assigning learners to an instructor-led class, but
it may be quite different, since it may require some level of
configuration of the VIE software (through an API or other
communications channel).

LMSs may also need to support a live “learner engagement monitor”

who monitors learner scores and progress status in real time and can
tweak system parameters for individuals.

For ideas on what may be in store for LMSs and VIEs, visit:

 http://www.brandonhall.com

 http://wiki.secondlife.com/wiki/Second_Life_Education/
Resources#Blogs

 http://er.educause.edu/

Choosing an LMS.docx page 140 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.3 Support and optimization for informal/social/collaborative

learning
Collaborative, informal learning is well supported by theory and
research (Mayer, 2005), and is often cited as the means by which the
vast majority of learning actually takes place in the enterprise. And in
the education arena, joint projects with learners at other schools
(possibly in other countries) are more and more commonplace,
enabled by social media tools. Through this new learning paradigm,
learners or trainees can be exposed to a variety of different
perspectives, share information, interests, and ideas, and monitor
how these have progressed and evolved over the course of a learning
project. Informal, social media-based learning is especially well-suited
to constructivist learning environments that address higher-order
thinking skills.

Users are now demanding social media features as part of the

learning toolkit that the LMS provides. These functions can be
provided either as applications within the LMS (in other words,
created or provided by the LMS vendor), or linked to external public
sites. As in the case of VIEs, LMSs incorporate social media functions
in a variety of ways. Brandon-Hall (2016) reports that
social/collaborative tools are cited as the #1 focus of technology
exploration (43% of respondents).

Social media tools support many informal learning approaches (such

as coaching, mentoring, and online knowledge sharing), where
learners develop and execute their own learning activities or products
in collaboration with instructors and fellow learners. Sources of
information may not be traditional “authoritative” sources, but
sources determined valuable by learners and their peers, and
methods of finding them can be based on learner research efforts.

In addition to the two parameters just described (learning based on

authoritative source vs user-generated content), informal learning can
be categorized as informal autonomous learning and informal
directed learning.

Informal autonomous learning is learning where the need for learning

and the learning goals are determined by the learners themselves.
The means for learning, i.e., learning experiences, paths, and

Choosing an LMS.docx page 141 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

strategies, are also up to the learner to choose and/or create

themselves. Authoritative source or user-generated learning objects
may be chosen by the learner to fulfill informal autonomous learning
objectives. This type of learning can and often does take place
outside of the workplace, but here we are really referring to learning
within a formal performance context (i.e., the workplace), where there
is some planning and deliberation involved on the part of the learner
to learn something. An example of informal autonomous learning is
where a learner unilaterally decides to take an online tutorial to learn
how to complete a task he or she has never done before. There is no
need for an assessment (and there may not be one available); the
learner simply stops when he or she is satisfied that they have
learned enough.

Informal directed learning is learning where the need for learning and
the goals of it are assigned by an external authority (e.g., supervisor
or corporate training department), but the means of meeting the
learning goals and objectives is left up to the learner (possibly within
a range of predetermined options or suggestions). As with informal
autonomous learning, authoritative source or user-generated learning
objects may be used by the learner. An example would be where a
learner is directed by their supervisor to learn a new procedure
through the enterprise learning tools that are at the learner’s disposal
(subject matter experts, corporate intranet, references, etc.), with an
assessment to be given after the learner is satisfied that they have
met the objective, for the benefit of objectively verifying that to the
authority who assigned the learning.

Finally, there is a category that is in-between formal learning and

informal learning: non-formal learning. Non-formal learning takes
place where the need for learning and the learning goals are
determined by the learner themselves, but the means of learning are
controlled by an external authority. Along with this, authoritative
source content is required, and assessments are given to ensure
conformance to learning and/or performance standards. An example
would be where a learner has unilaterally decided to seek a higher-
skilled, better paying job position in their organization, takes the HR
defined/prescribed learning path of courses (with assessment) to
qualify themselves. The following tables show these four paradigms:

Choosing an LMS.docx page 142 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

It is important to note that these categories are not conceptually rigid.

For instance, in formal and non-formal learning, where the learning
means is authority-controlled, the course design might dictate that
learners MUST collaborate with other learners (possibly in assigned
learning teams), with loose guidelines allowing learner control over
many aspects of the learning tasks. This collaboration itself can be an
important learning process goal, with technology-enabled
assessment of it.

Further, there are design choices for formal and non-formal learning
(Masie, 2013), as follows:
 Time Allocation? How much of allocated time is social?
 Learning Processes? Is the social and collaborative learning
aimed at:
o Evidence Models? How are we evaluating the impact of
various social styles?
o Information Transfer - Peers share first level of knowledge
directly
o Contextual - Peers share "back story" of applying
knowledge
o Collaborative - Peers teach and assess each other
o Remedial - Peers help learners get "unstuck"
o Assessment - Peers provide testing and assessment
o Transfer - Peers support applying knowledge at work

Choosing an LMS.docx page 143 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

LMSs can be designed to support all of these learning and design

paradigms, but most LMSs nowadays are still predicated on formal or
non-formal learning with individual learners taking prescribed
learning paths. LMSs are slowly emerging that allow upload and
display of user-generated content, with the ability for users to tag
content with ratings, comments, tags, etc. (termed “paradata”). They
are also more and more being designed to detect and track user
levels of learner collaboration to achieve learning goals, often a
design feature of informal directed learning programs. For informal
autonomous learning, a well-organized enterprise portal or resource
library and a robust search engine can be sufficient for learners to
independently pursue their learning goals; there is no particular need
for an LMS, especially since learner performance does not need to be
tracked.

Currently, most LMSs provide open-ended support for informal

learning through simply providing access to social media
applications, leaving it up to instructional designers to determine
how, why, where, and when those applications are to be used to meet
learning goals. These social media applications can either be
provided internally and natively within the LMS, but are often
provided through established third party applications. These third
party applications can be installed behind the enterprise’s firewall and
open to use only by employees for business purposes, or available
for any use on the Internet.

Social media tools are often only used as a means to deliver content,
not to provide activities that support true collaborative social
learning. For instance, links to instructional videos are published on
community of practice sites, or Twitter is used to remind learners
about class assignments—without using the full potential of these
tools to facilitate collaboration between learners.

Social media applications to support informal learning include the

following:

 Aggregated platforms that offer a combination of most of the

items below (for example, SharePoint®, Facebook®)

 Application sharing (for example, Google Docs®)

Choosing an LMS.docx page 144 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Blogs (for example, Blogger®)

 Chat (possibly including webcam) (for example, AOL Instant

Messenger®)

 Communities of practice (CoPs) (for example, Ning ®)

 Document sharing (for example, DropBox®)

 Expert exchanges (for example, Experts-Exchange.com ®)

 Forums (for example, Bloomfire®)

 Idea storming (for example, Stormboard®)

 Micro-blogs (for example, Twitter®)

 Picture sharing (for example, Flickr®)

 Podcasts (for example, iTunes®)

 Social bookmarking (for example, Delicious®)

 Social networking (for example, Facebook®, LinkedIn®)

 Social news (for example, Digg®)

 Video conferencing (for example, Skype®)

 Video sharing (sometimes known as Vodcasts) (for example,

YouTube®)

 Wikis (for example, Wikipedia®), often with peer rating of content

These tools are being used to support such informal learning

activities as coaching, knowledge sharing, professional networking,
on-the-job learning assignments, and other work experiences. But
traditionally, LMS products have not been very useful in managing
learning that happens in the workplace through these activities.

Shank (2013) reports the following social media tools as technologies

used by respondents in their survey. The category names are

Choosing an LMS.docx page 145 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Shank’s; they are slightly different in some cases to those in the

above list.

 Videos created by others (YouTube, Vimeo) 46%

 Discussion boards 33%

 Social network (LinkedIn, Facebook) 31%

 Video meetings (Skype, Google+Hangouts) 29%

 Audio created by others (downloadable MP3s) 29%

 Blogs (WordPress, Blogger) 27%

 Microblogging (Twitter) 17%

 Podcasts 24%

 Social network platform (Edmodo, Yammer) 11%

 Community platforms (Ning) 8%

 Social picture sharing (Flickr) 6%

 Social bookmarking (delicious, Diigo) 6.4%

Some LMS vendors are building simple interfaces into their product
that provide access to commercial social media functions and sites,
with no explicit connection to communities of other learners (i.e.,
class cohorts), other learning content in the LMS, or performance
tracking. However, some vendors are creating explicit connections,
whereby the LMS determines, based on performance on an
assessment in the LMS, that a learner would benefit from interacting
with a community of practice (CoP), members of which might be
available to collaborate with and/or mentor them, and automatically
enrolls them. There can also be automated features where the LMS
would assign subscriptions to social media functions to the learner.

LMSs are starting to emerge (for instance UdutuTeach ®) that actually

run on social media sites like Facebook, allowing a high degree of
integration of LMS and social media functions, representing a

Choosing an LMS.docx page 146 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

movement towards turning social media sites into self-contained

learning environments.

Possibly the most important social media feature from the point of
view of training stakeholders is the ability of an LMS to create and
maintain CoPs. These CoPs can be a cornerstone learning activity
within an informal, collaborative learning design, whereby learners
are required to contribute and interact with other learners through
discussion forums, blogs, etc. on a CoP. However, automated
tracking and assessment of these learner contributions can be
difficult. Another popular use of CoPs is to provide a vehicle for
learners interested in (or required to engage in) follow up activities to
a course, or who are interested in further exploration of the subject
matter.

Databases of contactable subject matter experts (sometimes called

expert exchanges or expert locators) are important also, either as a
separate LMS feature or as a core feature of CoPs associated with an
LMS.

Video sharing is emerging quickly as a way for employees to share

best practices and knowledge. Research indicates that a typical
employee generates almost 3 hours of video per month for sharing
purposes, while some organizations estimate that their employees
generate more than 20 hours per month (KZO, 2015). See7.30
Microlearning for more information on LMSs designed for video
sharing.

The advantage of adding social media to an LMS is simply that it

allows the LMS to provide a single access point for all learning
experiences, whether centrally managed and formal, or self-managed
and informal. This mix of structured vs unstructured learning is
quickly gaining acceptance in enterprise learning, and LMSs are
stepping up to the plate to accommodate it. No longer are learning
experiences defined by a curriculum of structured courses predefined
in the LMS course catalog; learners are expected to collaborate and
share knowledge through tools and access points provided by the
LMS. This sharing of knowledge can be through informal messages
posted in blogs, forums, etc., but it can also be through upload of
user-generated content such as slides and videos.

Choosing an LMS.docx page 147 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

This kind of social learning architecture for LMSs often involves

adding the following user affordances to be associated with content
objects:

 Rating

 Categorization

 Comments (with or without moderation)

 Questions (with or without moderation)

 Contact details for subject matter experts (SMEs)

 Recommendations

Social media as a general learning trend can be seen as a threat to the

paradigm of centralization of learning and performance management
that LMSs are currently predicated on, with the value placed on
authoritative content source and control. However, many LMS
vendors are embracing this technology and finding ways to maintain
authoritative content source and control over learning despite its
seeming pull in the other direction.

Despite the fact that LMS vendors are quickly getting better at
integrating social media applications into the delivery of learning
(whether maintaining authoritative control over the content source or
not), they face a serious challenge in terms of tracking the learning
progress of learners within the social media application context. This
has led to dire predictions of the demise of LMSs, due to their no
longer being able to provide centralized monitoring and reporting of
learner progress, one of the core business cases of owning an LMS
(see 7.26 Is the traditional LMS dead?for more information).

A fundamental problem here is the fact that many social media tools
do not in themselves contain any mechanisms for tracking learning;
there is no function for an LMS to connect with (in terms of an API) to
communicate anything resembling learning progress. Indeed, it would
be difficult to define and quantify learning experiences that happen
through use of many of these tools. But users are using them for
learning (in many cases, in ad hoc, home-grown ways) nevertheless.
The ADL xAPI described in 4.15.8 ADL Total Learning Architecture

Choosing an LMS.docx page 148 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

(TLA)) will have a significant effect on the ability of an LMS to track

informal learning experiences. The xAPI tracks both formal and
informal learning via ‘streams’ of learning experiences, similar to
social media streams such as Twitter and Facebook.

As self-directed, crowdsourcing-based learning through popular

social media tools proliferates among users, organizations will have
to embrace this highly decentralized array of tools as legitimate
venues for learning. But, as mentioned above, these tools are not
designed to interoperate with LMSs, and have no inherent drivers to
achieve such interoperability.

A category of system has emerged called “computer-supported

collaborative learning (CSCL) environments”. Many are not
necessarily LMSs in the true sense, in that they do not afford a way to
track learner collaboration and relate it to learning goals. However,
with the attention on finding a way to leverage the effectiveness of
informal learning for enterprise training, CSCL environments are
becoming increasingly instrumented with ways to assess learning.
This is not as straightforward as in traditional learning (where
multiple choice tests still reign), since it is not just the quantity but
quality of communication (both asynchronous and synchronous) that
needs to be tracked, where collaboration is either a learning goal
itself, or a required means of learning.

The following are examples of ways that learning can be assessed in

a CSCL environment (whether designed into the system or through
external means). These include both process assessment (i.e.,
assessing extent of growth in the collaborative learning process) and
product assessment (i.e., assessing quality of learner artifacts
created using the collaborative learning process).

 Instructor evaluation of growth in subject matter knowledge or

skill through transcripts of individual learner postings

 Peer evaluation of the contributions of learners in their learning

teams

 E-portfolio evaluation

 Retrospective self-assessment

Choosing an LMS.docx page 149 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Machine grading of essays, concept maps, wiki entries, etc.

 Discourse analysis by instructor to assess learners’ competence

in the collaborative process itself

The above address the quality of collaborative communication, which

is much more difficult to assess than the quantity of communications
(e.g., frequency and length of postings to blogs, etc.). Learning
stakeholders are quickly learning that the quantity of collaboration,
although a necessary data point, is insufficient alone to truly measure
learning.

One of the most important advantages of use of social media in

training is that the learner group itself can usually provide a bigger
pool of ideas for learning support and scaffolding than the
instructional designer can come up with on their own. Scaffolding
explanations, visual aids, etc. designed into the course by the
instructional designer may work well for the majority of learners.
However, allowing learners to see how some of their peers
understand and relate to the material (through public postings of
some kind) may provide better scaffolding for the statistical outliers
who need scaffolding that only other outlier learners who think or
learn the same way can think of. These learner postings can also be
important where there is insider knowledge or attitudes in the
organization or learner demographic group that the instructional
designer is not privy to or does not understand completely, and
learners can publicly process the material from that insider
perspective.

One way that this learner-generated scaffolding principle can be

implemented is an internal feature in the LMS whereby learners can
take notes and make comments as they are going through the
material. These notes and comments can be persistently stored
between sessions and automatically associated with locations in the
content, and shared with the instructor or other learners. With the
multitude of APIs to external social media applications that are now
available, this can be implemented such that the notes and comments
are posted publicly outside of the LMS, to applications like Twitter
and Facebook.

Choosing an LMS.docx page 150 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

One of the drivers for use of social media in training is project-based,

or experiential training. This is typically used in soft skill learning
domains such as leadership, where a team of learners is given a
project to accomplish and is evaluated on a rubric of parameters
related to both process and product. The project may not be purely a
learning exercise, but may actually be an attempt to solve an
organizational problem. Social media is often leveraged in these
cases to facilitate learners accessing the resources they need to
complete the project.

In this type of learning, the LMS (mostly through social media

applications) needs to able to push required resources out to learners
at planned junctures in the learning experience, as well as enable
learners to find and pull ad hoc resources as needed. In this way, the
LMS becomes the “command and control center” for the learning
experience.

For ideas on what may be in store for LMSs and social media, visit:

 http://www.brandonhall.com

 http://www.elearningguild.com

 https://www.td.org/

 http://www.gartner.com

 http://www.socialmediatoday.com/

 http://www.socialmedia.com

LMSs are now being built around the concept of sharing and
collaboration (e.g., ALTO learning portal®, Spoke®, Origin Konnect®,
and TREK Learning Experience Manager®).

Rather than acquire social media as functionality that is built into the
LMS, there are social media software modules that can be integrated
into LMSs such as MediaWiki (open source –
http://www.mediawiki.org).

As mentioned earlier, one of the biggest problems that training

stakeholders have with informal learning approaches based on social

Choosing an LMS.docx page 151 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

media tools is that there is no easy, straightforward way to assess

and measure the impact of informal learning activities. The ADL xAPI
(xAPI) provides the technical capability of tracking of a wide variety of
informal learning activities (see 4.15.8 ADL Total Learning
Architecture (TLA)), but it does not include the interpretive, contextual
layer that a system needs to assess the true impact of these activities.
LMSs that include competency models and 360-degree assessments
are a good start, but a framework is needed to create actionable goals
and track progress towards them. Systems such as Momentor ®
(https://www.envisialearning.com/) includes goal evaluation tools to
measure and evaluate learning gained from informal collaborative
learning activities.

If you use an informal social media-based learning paradigm that

“mashes up” disparate sites and functions (using your LMS only as
the initial launch pad), you may want to consider a single sign-on
(SSO) mechanism such as OpenID. This may be especially important
if any cloud services outside of the LMS retrieve or post data. OpenID
or some other open authentication mechanism can make
interconnectivity in the cloud trusted.

7.4 Massive Open Online Course (MOOC) support

A recent emerging trend in social media-based courses are “massive
open online courses” (MOOCs). These are courses where both
participants and course materials are distributed across the Internet.
They are usually based on informal learning principles, relying heavily
on social media. Learners participate at the level of their time and
interest, and there is no cost. Universities are usually the sponsors of
MOOCs. Motivations for universities to sponsor MOOCs include:

 Making courses more openly accessible to a wider audience

 Piloting new courses and trying out new ideas for courses in a
low risk environment, where learners are not paying customers
(i.e. matriculating learners)

 Showcasing a course in order to identify prospective learners

and recruit them to the university

Choosing an LMS.docx page 152 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Rather than author and deliver original content, you may be able to
leverage content or curriculum components that are already offered in
a MOOC. Currently some LMSs that are specifically optimized for
MOOCs are emerging. They generally resemble CrMSs (often
including VLE components) rather than LMS. For more information on
MOOCs, see http://en.wikipedia.org/wiki/Mooc.

7.5 Open architectures

“Open architecture” infers that the LMS has APIs that allow
integration of external applications and systems into the LMS,
including, in some cases, swapping an LMS vendor-provided function
with an externally produced one. In some cases, the vendor offers
hundreds of APIs that the customer can pick and choose from. Open
architectures imply a relaxation of proprietary control and constraints
on the part of the LMS vendor, allowing potential users to “look under
the hood” at their implementation.

To enable open architecture, the vendor usually must share all or

parts of its architecture with add-on/system integration developers.
This may require some license agreements between entities sharing
the architecture information.

Open architecture products tend to have a service-oriented

architecture (SOA), and tend to be designed less as closed systems
and more as extensible platforms. Because of this, they tend to
encourage innovation and experimentation more.

In spite of the potential for competitive disadvantages resulting from

publicly exposing the inner workings of their system, some vendors
favor them because their customers want to be able to easily
customize the system by purchasing additions that the LMS vendor
may not feel are important enough to develop themselves.

Open architectures have driven the creation of a substantial

marketplace for third-party applications that can be integrated into the
core LMS system as modules. These modules can provide all sorts of
functions ranging from anything like adding a calendar function to the
learner interface (similar to widgets that you can add to a web portal
or cell phone) to providing the capability to share data with an ERP
system.

Choosing an LMS.docx page 153 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

In the future, some third-party add-on applications may be associated

only with a particular piece of content, rather than adding a global
capability to the LMS. They may customize and maximize the
experience of users taking that particular course.

Open architectures could significantly decrease risk in cases where

changes to your enterprise learning needs and learning technology in
general are expected. In these cases, an open architecture can allow
you to prolong the useful life of your LMS by incrementally adding
needed functionality rather than having to replace it.

As stated in 4.9 Security considerations for LMSs, it is important to

find out what programming language and third party OEM
components were used to build the product you are considering
acquiring. There are innate security considerations for some
programming languages, like PHP. Also, if you will need to customize
the system, your programming staff need to have the skill sets for
that programming language and have licensing access to modify any
third party components.

7.6 Adding authoring capabilities

Many LMSs, in their search for new frontiers of functionality to add to
their system to add value to customers, have turned to authoring and
knowledge management additions. Authoring is a natural addition to
many LMSs, since it moves an LMS closer to being an LCMS,
accruing many of the advantages that an LCMS affords (except for a
content repository) without losing the essential ingredients of an
LMS. For more information on LCMSs, see 3.3 Learning content
management systems (LCMSs).

7.7 Extended enterprise learning

Enterprises are expanding the learner audience served by their LMS
to include partners, distributors, resellers, suppliers, franchisees, and
even customers (B2B and B2C), who may be part of the corporate
“extended enterprise” or “extranet”. No longer is the LMS relegated
only to internal employee training. The learner base and the scope of
learning functions are becoming much broader. Learning in this
paradigm can include such things as product demos, sales training
for retailers, customer surveys, customer support documentation,

Choosing an LMS.docx page 154 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

and more. In some cases, a charge is levied on extranet users. As an

extended enterprise learning system, the LMS is becoming more like
the enterprise portal, in some cases, becoming completely integrated
with or taking over most of the role of the corporate intranet/extranet
portal.

In addition to adding extended enterprise features to their standard

feature set, some LMSs are actually branding themselves as an
“extended enterprise LMS”, and are specifically optimized for
customer and partner training rather than employee learning.
According to Skilljar (2014), there are 4 differences between a
standard LMS and an extended enterprise one:

 Training is more often optional, subject to the needs and whim

of the extended enterprise learners, rather than required, as in
the case of employees.

 Because of the above, these LMSs rely much more on user

experience to attract and retain users. If users do not have a
positive experience, they can opt out, whereas internal
employees do not usually have a choice.

 Extended enterprise LMS users often pay for training, whether

on a per-course or bundled basis, or included in a premium
support plan. This relates to the previous point: where external
learners are paying for the training, learner experience matters
much more.

 Extended enterprise learners are accessing the LMS from a wide

variety of locations and IT environments from outside the
enterprise firewall. Their configurations and identities are not
managed by the enterprise’s HR or IT departments. This
complicates user tracking and makes security a much bigger
concern.

Here is a sampling some of the features that are important to an

extended enterprise LMS (these are included under the appropriate
headings in 5.. List of possible requirements for an LMS):

 Quick system responsiveness

Choosing an LMS.docx page 155 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Ease of self-registration

 Ability to reskin the interface according to different external user

groups (especially differentiated from internal user groups)

 E-commerce features for users paying for training

 Data integration with sales, CRM, and financial systems

 Pricing plan that is flexible due to the unpredictability of the

number of users and patterns of use

LMSs are now appearing that are targeted specifically for extended
enterprise learning, for example, EthosCE®.

7.8 Adding knowledge management architecture and

capabilities
Knowledge management (KM) system features seem like they would
be a useful addition to an LMS, but functionally it is not that simple,
since LMSs deal with content and KM systems deal with information.
However, some vendors are trying to bridge this gap.

Expert locator networks are one common knowledge management

feature that is either added to LMSs or integrated with it.

An LCMS is a better starting point for integration into a KM system,

since that allows you to create small knowledge objects that can be
converted and/or combined into training content, and training content
that can be repurposed as knowledge objects.

Knowledge management implies robust search capabilities; in this

sense, any LMS that provides deep text search of training content is
half of the way there to a KM platform.

7.9 Support for team-based learning

“Team-based learning” can mean nothing more than a group of
learners in a meeting room taking a course together under one login,
presenting themselves to the LMS as if they are one learner and
making group decisions about how to complete course activities. It
can also mean a group (self-organized or assigned by an instructor)

Choosing an LMS.docx page 156 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

synchronously progressing through a course from different locations

and being scored by the average of their individual scores. However,
true team-based learning revolves around the idea of learning
activities that both affect other team members’ activities and are
affected in turn by the actions of others in their team, who may be
using a different version or part of the course based on their
individual role in the team.

Thus, LMS support for team-based learning involves more than just
providing communication functions in the LMS in order to provide
collaboration and peer review by multiple learners. Complicated
assessment and sequencing paradigms must be enabled, with
intelligent agents or middleware automatically tracking and mediating
the activities and performance of each team member, and reporting
rollup progress to the LMS as well as an audit trail for how these
scores were generated (based on individuals’ performance).

As with social media-based learning, team-based learning is

especially well-suited to constructivist learning environments. The
technological challenges in this type of learning are now being
worked out, but there is no universally accepted solution, so no
prominent LMS solutions to supporting it have appeared yet. But as
soon as the team-based learning paradigm becomes an established
part of the training and education space, LMSs will surely move to
support it.

You may see more support for team-based learning in LMSs that
support the xAPI specification, since the xAPI enables it. The “actor”
part of xAPI statements can be an individual or group of individuals.

7.10 “Gadget”- based interface

Gadgets (aka “widgets”, “portlets” or “applets”) are functionalities
that are presented as separate items on a page. They are used in
many commercial e-mail “MyPage” interfaces, and in many enterprise
portal interfaces. They make it possible to completely customize the
user interface; gadgets can be turned off so they do not appear on the
interface, and can be moved to any location on the page. They can be
associated with a specific role so that users only see the ones that
are relevant or permitted for their role.

Choosing an LMS.docx page 157 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Note: Some make a distinction between “widgets”, which are generic

code objects that can be inserted into any web page, like a hit
counter, and “gadgets”, which are proprietary and will only work
within a particular vendor’s system. Here, we are mostly talking about
“gadgets”.

This type of portal-like interface has gained traction with some LMS
vendors, simply because users are more comfortable with this type of
modern interface, and it allows a high degree of interface tailoring to
suit their needs. In some cases, LMSs are being rebranded under the
name “learning portal”.

Integrate of the LMS with a portal through gadgets can be achieved in

either of the following ways:

 Insert “deep links” in the portal interface (essentially gadgets) to

a static location (e.g., a course) in the LMS or use an API to pull
data from your LMS dynamically and post it in the portal.

 The reverse of the above: insert links (static or dynamic) to

portal locations or objects into the LMS interface

7.11 Adding talent management architecture and capabilities

Talent management systems (TMSs) are sometimes called Integrated
Talent Management (ITM) systems. Talent management includes
recruitment, performance management, compensation and benefits,
succession, retention, career planning, skills gap analysis, career
development, and mentoring/coaching administration. These systems
mostly deal strictly with these functions and do not provide the day-
to-day HR processing functions such as payroll. An Elearning!
Magazine Group survey (Roche & Upton, 2013) reports that TMSs
increased in utilization at 28%, up from 22% in 2012, an increase of
27% year to year.

Talent is the most expensive resource to acquire and maintain in

most organizations, so the ROI for TMSs is often attractive, and many
enterprises are eager to adopt them. Three popular features sought
by those wanting to acquire a TMS are: skills gap analysis,
mentoring/coaching administration, and career development.

Choosing an LMS.docx page 158 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

The terms “human capital management” and “workforce productivity”

are also used synonymously with talent management. They overlap
with LMSs in terms of the broad scope of their human resource
development mission. However, whereas LMSs focus on training of
current employees as a solution to a strategic enterprise talent or
competency need, talent management systems focus more on
recruitment as a solution. Talent management systems are often
integrated with applicant tracking systems (ATSs) to manage the
recruitment process, and can include performance management,
compensation and benefits, succession, retention, and career
planning.

Talent management integration or functional merging with LMSs is

seen by many HR stakeholders as strategically important to HR
functional integration, and for this reason, some LMS vendors are
reengineering their LMSs so that they encompass both talent
management and learning. This can result in automation efficiencies
whereby competencies are assessed and result in recruitment and
succession management actions. Career development is of course
only one piece of the Human Resource Development (HRD) picture;
managers must know who needs to be trained and certified based on
what organizational deficiencies exist, and input these deficiencies
directly into the process of acquiring new talent, if that is necessary
based on the existing pool of talent.

For more information on LMS/TMS integrated systems, see

http://elmezine.epubxp.com/i/74275 (Roche & Upton, 2012, p. 20-23).

McIntosh (2014) reports that talent management system vendors are

buying LMSs (for example, SAP purchased Plateau in May, 2011, and
rebranded it as SuccessFactors), accelerating the trend of LMS-talent
management system consolidation.

7.12 Adding competency analysis tools

Some LMS vendors have demonstrated embedded or standalone
competency management or “precision skilling tools” which allow a
user to self-assess their competencies in a specific skill area. The
most common skill areas where this is being applied are information
technology (IT), Microsoft Office applications, and soft skill areas
(“leadership skills” or “financial skills”). In lieu of laying out every

Choosing an LMS.docx page 159 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

competency inherent for a particular job or job category, these tools

allow a user to analyze them independently and choose courses
appropriate to their position, rank, rate, grade, specialty, etc.

Other LMS tools (sometimes called “skill management systems”

(SMSs)) are becoming available to training and HR administrators to
automate the labor-intensive tasks of manually matching training
interventions (courses, units, lessons, topics, OJT, tests, career
experiences, etc.) to the organization’s job competency requirements
(skills, tasks, knowledge, behaviors, etc.). The competency
management process usually includes the following, which is
becoming more and more integral to LMSs:

1. Determine competencies required for jobs

2. Profile competencies and their current levels throughout the

organization

3. Determine the gap between existing and desired

competencies

4. Define objectives and other descriptors of courses that are

needed to close the gap

5. Match learner competency deficiencies to learning tracks,

training programs, and courses

o Define user groups based on competency requirements

o Define courses and curricula based on competency

requirements

o Map competencies to courses or any other training

intervention

o Map learners to courses or any other training intervention

o Map learners to continuum/advancement tracks

6. Plan learning track and training programs that incorporate

these courses, in order to close the gap

Choosing an LMS.docx page 160 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7. Provide training recommendations (to include prerequisites)

to fill competency gaps

8. Evaluate competencies after learning

7.13 Component-based architecture

In a component-based architecture, a vendor licenses a product for
use as an on-demand package of services—customers pay for only
the components they use. It presumes a modular architecture
whereby the vendor compartmentalizes the system so that users only
access (and pay for) the parts that they need at any given time. This
method is attractive to many organizations because it can lower costs
(since you only pay for the features you use), in contrast to licensing
all applications/modules/functionality of the LMS 24/7 throughout the
life of the installation, whether you need them or not.

Products with a “service-oriented architecture” (SOA) imply having a

component-based architecture, in that services can represent
components that can be accessed when needed from the cloud.

Certain aspects of the architecture of such a system must be

designed specifically for component-based architecture by the
vendor, so that features can be turned on or off, depending on the
needs of individual customers. Many current systems offer some
degree of component options; qualifying as “component-based” is
only a matter of the degree to which the system and the pricing model
is optimized for it.

Component-based architectures are usually associated with hosted

solutions (see 4.10 Hosting options). However, a hosted solution may
be sold with or without any compartmentalization. For instance, a
hosted solution may simply be a one-size-fits-all system based on a
flat fee covering a specific number of licenses that cover using all
parts of the system; a component-based architecture solution is
usually hosted but in addition also involves a modular,
compartmentalized approach, as described above.

Be aware that a vendor may offer a component-based architecture,

but only as a possible convenience to allow the customer to reduce
the complexity of the system, not reduce cost. In other words,

Choosing an LMS.docx page 161 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

“turning off” components may be possible, but it may not change the
cost.

7.14 Multimedia LMS (MLMS)

LMSs are starting to appear that call themselves “multimedia LMSs.”
They base their value proposition on the ability to synchronize
moving images with still images on two modular screens. One such
example is Knoodle (www.knoodle.com). This system is essentially an
integrated authoring tool and LMS, since the synchronization is
authored in the MLMS (based on imported PowerPoint and video
assets). A typical use case for this arrangement might be a talking
head video of a senior manager introducing slides introducing a new
corporate policy or structure that is shown on the adjacent modular
screen.

The principle of a dual panel eLearning module, with video or

animation in one panel, and static images in the other, is not new;
VLEs can display content in this manner, and many authoring tools
allow authoring of this format within the content itself. However, these
MLMS products are optimized for this kind of delivery, with the ability
of non-technical authors to rapidly and easily synch static images
(often in the form of PowerPoint slides) to the video or animation. The
content is tightly integrated with the standard LMS functions of
learner tracking, tests, surveys, etc. This approach can work well if
your organization decides that video synched with slides is the type
of content you want to focus on, and you are willing to sacrifice
interactivity, since the screens in this type of LMS are usually static.
You will also need to have the internal resources to create and edit
video.

7.15 Learning Experience Manager

A new kind of LMS has emerged which takes advantage of the xAPI specification’s ability to track
learning of all types, including informal and experiential (see 4.15.8 ADL Total Learning Architecture
(TLA)). It remains to be seen whether the name is accepted as an industry standard for this category of
product; it is only offered by one company currently (TREK product -
http://www.cognitiveadvisors.com/). This product provides an xAPI-required Learning Record Store
(LRS) endpoint for xAPI communications, allowing tracking of such learning experiences as:
 Coaching conversations
 Searches
 Video watching
 On-the-job experience

Choosing an LMS.docx page 162 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

It also allows the awarding of badges (see 7.21 Digital badges),

learning analytics, ePortfolios, and the creation of individual learning
paths.

It remains to be seen whether this concept will take hold in the

marketplace. However, it seems likely that it will emerge as a viable
replacement or “sideware” option for LMSs, given the growing level
of awareness acknowledgement among learning professionals that
only 10% of learning in the workplace is actually accomplished
through formal learning.

7.16 Video conferencing integration

Recent developments such as greater levels of bandwidth, less
expensive dedicated room-based videoconference equipment, and
free personal videoconference capabilities on desktop computers
have contributed to rapid growth in the use of synchronous
videoconferencing in training and education.

Screen sharing (of PowerPoint slides, for instance) and voice over IP
(VoIP) are usually is associated with video conferencing capability;
often this set of capabilities is referred to as “web conferencing”. It is
still common for vendors to partner with a web conferencing vendor
to offer this capability, although some LMS vendors are adopting their
own proprietary technology. Shank (2013) reports that 29% of the
respondents in their survey reported using this technology.

There are three types of videoconferencing capabilities:

 Personal – designed to allow one user to see and talk to another

user. These systems run on personal computers or
smartphones, usually as a free peer-to-peer service. They
usually are limited to one-to-one (between only two persons)
communication, but are beginning to offer one-to-many
capabilities (although quality usually degrades in such cases).
Skype®, iChat®, and LiveMessenger®, and FaceTime® are
examples of these systems.

 Web-based – these resemble the Personal category described

above, except that the video appears in a browser and the
service is managed through a central server. Because of the

Choosing an LMS.docx page 163 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

server-managed aspect, there is no loss of quality as more users

are added to the conference. Examples are WebEx ® and
Elluminate Live®. Web-based videoconferencing is often a core
feature of VLEs.

 Room-based dedicated systems – use dedicated hardware, often

semi-permanently installed in meeting rooms. These proprietary
systems usually include a codec (usually based on the H.323
standard), camera, microphone, large video monitor, and
speaker system, and can be quite sophisticated (and expensive
to use). Examples are Polycom®, Tandberg®, and LifeSize®.

Interoperability between these types of systems is increasing. For

instance, some of the room-based dedicated systems offer software
that allows personal computers (i.e., the “personal” category above)
to connect to room-based conferences.

VIEs often resemble web conferencing systems in the “Web-based”

category in that they allow “talking head” videos of the instructor, but
are different in the respect that VIEs usually do not support student
videos (at least, live ones). In this respect, VIEs resemble webinar
software more closely.

LMSs cannot generally technically integrate or host sessions in the

“Personal” and “Room-based” categories, since these rely on
external proprietary services, with particular software and hardware
requirements. They can facilitate coordinating videoconferencing
sessions as learning activities within a course or curriculum,
however. This allows instructional designers to incorporate
synchronous videoconferencing (of any of the three categories
described above) in their instructional design and mix them with
asynchronous learning objects. Examples of use of
videoconferencing in these cases might include:

 Lecture, discussion, and Q&A with SMEs and instructors

 Virtual field trips

 Real-time collaboration on assignments between classrooms,

assignment working groups, and individual learners, especially
in different parts of the world

Choosing an LMS.docx page 164 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.17 Search-based learning

Google has become the default, preferred source of learning for many
workers.

What does this mean for LMSs? The LMS must be able to handle
federated searches across applications and domains. Searches
should include not only the LMS, but the Internet, content
repositories, and databases of experts. Being able to store content in
a way that is optimal for quick and easy “just-in-time” future retrieval
is an important element of this function. This involves not only
bookmarking sites, but extracting small chunks of content to be
stored in a knowledge management application or database.

A quickly emerging trend in not only search-based learning, but in

knowledge management and the Web in general, is latent semantic
indexing. This is particularly important for unstructured content,
which may not be metadata-tagged, organized within some
meaningful context or hierarchy of objects, etc. Systems are getting
smarter at extracting meaning and relevance from such content, and
not only making it easy for users to find content, but taking it a step
further, as Content Analyst Company (2015) coins the term “the
content finds you”. Content Analyst Company (2015) reports that
knowledge workers spend 16% of their time searching for
information, and they find the information they need only 56% of the
time. Latent semantic indexing systems offer vast improvements over
this, where systems:

 Infer user’s interests

 Adapt to changes in terminology

 Overcome keyword limits

The success of these systems may obviate the need for LMSs in
informal user-directed learning scenarios, if relevant content can
quickly and reliably be found through searches of content
repositories outside of the LMS. If integrated through an API into an
LMS, however, these systems could be a very powerful content
broker extension to the LMS.

Choosing an LMS.docx page 165 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.18 Content delivery networks (CDNs)

If you deliver high definition video content to a highly geographically
dispersed audience, and find that the performance is weak and/or this
service is consuming too much of your LMS server resources to
maintain sufficient quality, you may want to consider a content
delivery network (CDN). Akamai® and Limelight® are examples. These
services have strategically placed local servers operating all over the
globe that make delivery of high-bandwidth video content smoother
to dispersed populations of end users. If you do not have enough
video content to justify having a direct, expensive, long-term contract
with a CDN provider, there are resellers such as Rackspace (a reseller
of Akamai’s CDN) that offer pay-as-you-go agreements. These
resellers enable you to deliver just a few videos very cost efficiently.

CDNs are especially important for “extended enterprise” LMSs (see

section 154. Extended enterprise learning), where the users may be
much more geographically dispersed than a self-contained LMS that
mainly serves internal employees.

7.19 Integration with digital libraries

Digital libraries (usually mostly reference books, but also containing
topical training videos and audios) are an important support for
informal learning. Informal learning (somewhat synonymous with
“constructivist learning”) puts the enterprise learning function into
the role of facilitators and enablers of learning, rather than engineers
of learning. Informal learning can be collaborative, based on use of
social media to contact peers and subject matter experts, or it can be
individually-based, relying mostly on reading and research.

For example, in an individual informal learning scenario, learners

might perform any of the following activities, with varying degrees of
direction and monitoring from instructors or the training organization:

 Self-tailored reading and research prior to a formal training

event

 Enrichment, expansion, extension, and reinforcement reading

assignments between or after formal training events that require
learners to learn more about what they were taught

Choosing an LMS.docx page 166 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Performance support tools and intranet web search tools that

provide on-demand access to references that help them perform
unfamiliar tasks in their job

 Learning objects designed into courses that are compiled from

online references (possibly dynamically)

Digital libraries are the main enabler of these types of informal

learning. There are commercial services such as Safari Books Online ®
that specialize in offering access to references for these types of
learning scenarios. One of the value propositions that these libraries
offer is that of maintaining the currency of their works, to support the
pace of technological change (these libraries even include pre-
publication versions). This value proposition and the authoritative,
expert-level source aspect of digital libraries supports the needs of
employees and managers for certification, reference, and training.

An offshoot of digital libraries is the organization of enterprise book

clubs, which is a good support for both formal and informal learning.
Book clubs involve employees in reading new, relevant books and
meeting regularly (possibly virtually) to discuss them, often with an
assigned facilitator who is either the team lead or a trainer.

Paradata (the ability to rate sources, similar to the rating system in

Amazon.com) and built-in search engines are key component of
digital libraries, as they can help employees target the most useful
information to help them with the task at hand.

Digital library services can be thought of as LMSs of sorts because

they often offer LMS-like features such as:

 Reporting and tracking content accessed by learners

 Assigning items to learning plans and competencies

 Maintaining online reading lists for specific courses, employee

groups, or individuals

If you already have an LMS, you will want to integrate it with the
digital library service and have the LMS present one unified interface
for accessing courses, digital library references, and other learning

Choosing an LMS.docx page 167 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

objects. Digital libraries are thus starting to offer out of the box
integration with major LMSs.

7.20 Gamification of learning

Recently there has been interest in “gamifying” new or existing
learning content, to increase motivation and engagement (through
leveraging the natural drive to competition, achievements, etc.).
Vipond and Clarey (2016) report in their survey that, when asked if
gamification definitely improved learning experience and enhanced
knowledge retention, slightly more than one-half (56%) of
organizations said they “somewhat agree,” while only 31% “totally
agree” with that statement. However, they go on to say that “many
organizations did not seem totally convinced about the value of
gamification, and nearly one-half believed that gamification is ‘nice to
have, but not required’ in a future LMS.” (p. 5)

Kapp (2012, p. 10) defines gamification as: “…using game-based

mechanics, aesthetics and game thinking to engage people, motivate
action, promote learning, and solve problems.” These elements could
be designed into learning content from the start to create a bona fide
serious game, or added to a learning experience that is already built.
In either case, in order to achieve robust gamification, certain items
need to be implemented in the LMS; the gamification cannot rely on
features in the content alone. LMSs are now appearing that do not
require content to have any internally-designed game elements; the
gamification layer could reside entirely in the LMS and be added
solely through LMS configuration.

The LMS needs to support gamification through basic functions such

as special delivery mechanisms, security features, enterprise
integration tools, and reporting and analytics features, as well as
specific features such as interactive leader boards,
points/levels/badges, and intangible trophies and tangible real life
rewards into the learning experience.

If the content itself is gamified, the explicitly designed levels and

points achieved within the content will communicate with the LMS
levels and points tracking and reporting capability. For content that is
not internally gamified, then the LMS can be configured to have
certain achievements or thresholds related to content objects

Choosing an LMS.docx page 168 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

(passing a test in course, for instance) count towards levels and

points as tracked and reported by the LMS.

Some LMSs such as OnPoint CellCast® include features such as the

following to allow formal content or informal learning experiences to
be “gamified”:

 Standard points awarded upon achievement of some threshold

or triggering event

 Bonus points added to standard points for any achievement

completed during a time window or duration frame

 Recognition points that are discretionary and can be awarded by

a manager

 Earned badges associated with any defined achievements and

attained levels

 Trophies & rewards representing different classifications of

winners, each with their own unique trophies and rewards

 Interactive leader boards that show the results of ongoing

competitions between individuals, groups and peers.

 Assigning points to any defined learning assignment or

interaction

 Game Profile templates that can be assigned to any defined

group, location, job code or other collection of workers

 Updating & game analysis features to support changing an

existing Game Profile

 Test-derived points

 Authoring flash card-based games

 SCORM-based or xAPI-based scoring

 Game results & dashboards providing real-time results for every

defined Game Profile

Choosing an LMS.docx page 169 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

7.21 Digital badges

The concept of digital badges are becoming popular in higher
education and some training settings. It derives its popularity from a
new focus on training and educational results (credentials,
competencies, etc.), rather than process (content, instructors, etc.).
Digital credentials are seen as a robust and efficient way to express
those results, and badges are the primary form these credentials take,
allowing them to go far beyond traditional transcripts.

A badge is a file (usually a PNG image file) that contains a record of

some learning accomplishment on the part of the owner. A collection
of badges is called a “backpack”. Before initiatives like the Mozilla
Open Badges Infrastructure, badges did not mean much, since there
is no inherent authentication that a given issuer truly issued it, or that
the owner did not create or modify it themself. But initiatives such as
the Mozilla Open Badges infrastructure provide a means to maintain a
link to the issuer, and can convey the criteria it was issued under, all
with built-in verification.

This type of alternative credentialing can be applied at different levels,

for instance, showing that a learner has successfully completed an
entire course or acquired a high-level skill, as well as micro-level
learning elements involved in those. Because they are visible to
peers, they enable learners to seek and give help to others whose
badge profile indicates that they need it. Badges can applied very
easily in an environment that uses a competency-based model for
curriculum and training. Other environments may have to rework their
learning ecosystem to accommodate the concept. Badges are
currently popular within MOOCs (see 8.3 Massive Open Online Course
(MOOC) support).

LMSs are starting to incorporate badge infrastructure, i.e., the ability

to import and issue learner badges and store badge information in the
learner achievement profile, for example, Trek Learning Experience
Manager.

NOTE: As of January 2017, the IMS Global Learning Consortium will

be taking over the Open Badges specification. The Open Badges spec
developed by the Mozilla Foundation is the most popular
implementation of digital badges. For more information, see

Choosing an LMS.docx page 170 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

https://www.imsglobal.org/initiative/enabling-better-digital-
credentialing

7.22 Adaptive learning systems

The word “adaptive” in this context refers to the manner in which a
system delivering a learning experience may adapt to an individual
learner’s needs, goals, and performance in such a way that the
learning experience is better tailored to meet the specific performance
problem. In general usage, the term “adaptive system” often
describes architectural adaptability, which is the ability to customize
an entire educational/training institution’s delivery platform (or
combination of platforms) for the particular needs of an organization,
e.g., extensibility to support greater numbers of users, ability to
change target language, ability to re-skin the interface and add
“storefronts”, easily add new components/functionality, etc..

By contrast, what is meant by adaptive system in this context stems

from a learning theory/instructional design approach where learning
experiences/performance interventions themselves may be “micro-
structured” in order to permit the broadest possible opportunity for
variation. This capability for variation coupled with a carefully
designed analysis engine driving the choice of which variation(s) to
deliver to the learner enables a robust adaptive system.

“Adaptive systems” is an umbrella term that includes not only

systems that deliver pre-scripted variations in learning paths,
responses, screens, etc., but systems such as intelligent tutoring
systems that generate new content (some might call these
“generative systems”). Whereas, under the pre-scripted variation
paradigm, two learners might actually see the same content, there
may be little or no chance of that in generative systems, which can
create a unique experience for the learner based on what he or she
knows, what an expert knows, and the intended outcome.

“Adaptive LMS” is also sometimes used to refer to a content

brokering system (see 7.23 Content brokering systems) which can
overlap with adaptive learning systems. The differentiation between
an “adaptive system” and a “content brokering system” is somewhat
semantic and arbitrary, but ADL defines it as described in 7.23
Content brokering systems: an adaptive system dynamically orders

Choosing an LMS.docx page 171 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

the elements of a learning experience (i.e., content in most cases)

while the learner is already engaged in it, whereas a content brokering
system suggests or prescribes the next prepackaged learning
experience (often a course) after completion of each content object,
course, etc. based on the learner’s history, profile, and either stated
or detected needs. A simple way to think of it is that an adaptive
system delivers intra-content variations to the learning path, and a
content brokering system delivers inter-content variations. Also,
content brokering systems usually present variations as
recommendations that are not simply delivered to the learner as their
prescribed learning path, as in the case of adaptive systems.

The simplest form of an adaptive system is an LMS that handles

SCORM 2004 sequencing, where, for instance, a pretest determines
your learning path through a course, so that you automatically skip
parts that the pretest shows you already know. Most adaptive learning
systems rely heavily on learner assessment data, as in this example,
as the source of information on which to base adaptations. The
question is how frequent and detailed are the assessments, how fine-
grained are the learning objects, and how dynamic and complex are
the learning paths that are delivered as a result of the assessments.

The following is a summary of parameters on which an LMS can base

its adaptations:

 Embedded assessments (knowledge checks, quizzes, etc.)

measuring their current state of knowledge. In the case of
simulation content, assessment could take place during the
course of the student’s practice performance, without the
student being aware of being assessed (this is sometimes called
“stealth assessment”).

 Past history of the student’s learning performance in other

content

 Student’s own assessment of what they need (such as further

reinforcement exercises, etc.)

 The instructor’s assessment of what the student needs

Choosing an LMS.docx page 172 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Paradata, i.e., what path is most likely to lead to success for this
student based on:

o Paths that other students took that led to success

o Student ratings of content or parts of content

 Location, work context, or environment

Another key question in evaluating an adaptive system is: how

“intelligent” is the system in determining the current
learning/knowledge state of the student and predicting what is the
best learning path through the content for them? Is it based on
explicit, preprogrammed rules or inferential reasoning (based on AI,
cognitive learning science principles, etc.)?

Traditionally, LMSs are more concerned with managing delivery of

predefined learning experiences and content rather than dynamically
micromanaging the delivery and structure of the learning. That has
usually been left up to the content designer. However, with the advent
of a more fine-grained and flexible approach to tracking and tailoring
learning (through standards like the xAPI), and the ability of systems
to assess a much wider palette of the learner’s state and context (e.g.,
see 7.24 Affective computing) the adaptations, the LMS is being
driven towards incorporating these features. There are a number of
LMSs that now purport to be “adaptive”.

ADL has developed a rubric for categorizing adaptive behavior in

learning systems. It focuses on two questions: 1) At what point in the
learning process is the information that determines an adaptation
strategy and resulting behavior generated? 2) What is the source of
the information that appropriately addresses the resulting adaptation
strategy? For #1, there are three possibilities:

 Before the learning experience

 During the learning experience

 After the learning experience (in preparation for further learning

experiences)

For #2, there are also three possibilities:

Choosing an LMS.docx page 173 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Learner actions

 Environmental conditions

 Third party actions

For a more detailed version of this rubric, please contact the authors.

7.23 Content brokering systems

Related to adaptive learning systems (see 7.22 Adaptive learning
systems) are content brokering systems. Both systems result in
personalized learning, and the terms are often used interchangeably.
But, as described in 7.22, for the sake of clarity in this space, ADL
defines a content brokering system as a system that detects a
student’s imminent content needs based on their performance in an
ongoing learning experience, then discovers, recommends, and
(usually) delivers the appropriate content to the learner from a
potentially wide variety of sources, both public and within an
enterprise.

This definition differs from the concept of an adaptive system in the

sense that a content brokering system refers to a system that
suggests or prescribes the next prepackaged learning experience
(often a course) based on the learner’s history, profile, and either
stated or detected needs. As described in 7.22 Adaptive learning
systems, adaptive systems dynamically order the elements of a
learning experience while the learner is already engaged in, or
“inside”, a prepackaged learning experience, not finished with one
and ready to start another (that could be from a different provider or
source). In this sense, one could think of content brokering as “inter-
content adaptation” and an adaptive system as “intra-content
adaptation”. 

LCMSs by design achieve a level of content brokering (see 3.3

Learning content management systems (LCMSs)). However, they are
limited by the fact that the content that they broker must reside in the
LCMS’s native content repository, thus the content is already known
to the system and duly catalogued. That is different from the most
robust form of a content brokering system, one that can theoretically

Choosing an LMS.docx page 174 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

search for, discover, and point to and/or deliver any appropriate

content, no matter where it is located.

In practice, content brokering systems are often configured to access

only certain prescribed content repositories with content objects that
the stakeholders prescribe as authoritative source; the difference
between this scenario and an LCMS is not only the fact that a content
brokering system can access more content than what is in its internal
repository, but the flexibility and intelligence it has in discovering and
evaluating content objects (for appropriate use by learners) that are
not engineered explicitly to be used by it. “Intelligent content” is an
enabling characteristic of content, allowing the system (which could
be an LMS) to understand more easily who, what, when, where, and
how it could best be brokered to learners. For more information on
intelligent content, see Berking (2015a).

Content brokering systems, like adaptive learning systems, must

have robust assessment capabilities in order to identify learning gaps
upon which to base brokering decisions. Systems can use “stealth
assessment” techniques to do this, or can base it on formal, explicit
formative assessments. A true content brokering system also usually
involves more sophisticated methods of determining what content
would be appropriate to deliver or recommend next for the learner, up
to and including AI algorithms that infer and predict the learner’s
knowledge needs and preferences.

Content brokering, like micro-level adaptive capabilities, are not fully

mature technologies currently, whether standalone systems or part of
an LMS. However, with the strong driver of the need for more and
more personalized learning, we will probably see robust content
brokering systems emerge quickly.

Content brokering is a key functional component of ADL’s TLA

architecture concept (see 4.15.8 ADL Total Learning Architecture
(TLA), including xAPI).

A possible starting point for a content brokering system is described

by Quinn (2015). Quinn advocates creating and tagging content
objects according to the following core elements of a learning
experience:
 Introduction

Choosing an LMS.docx page 175 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Concept(s)
 Example(s)
 Practice
 Reflection

These could be manipulated within the content brokering system as

follows:

From Quinn (2015), p.17

7.24 Affective computing

Affective computing can be seen as a logical next step in the
movement towards adaptive learning. Learner tracking using Big
Data, AI-based adaptive algorithms in content, robust learner profiles,
paradata, and other features have paved the way towards highly
adaptive learning systems. In adaptive systems, learners are
presented with or led down optimal learning pathways based on their
past learning performance, preferences, cognitive abilities, and
demographic characteristics.

The next frontier in adaptive learning is affective computing, allowing

a learning application to respond dynamically to the user’s emotions
of confusion, anxiety, boredom, etc.. These obviously have a major
impact on learning, as any classroom instructor knows and leverages
by constantly reading learners’s posture, tone of voice, and facial
expressions.

Choosing an LMS.docx page 176 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Web cams (that can detect where the eye is focused as well as facial
expressions) and voice analysis capability can now detect these
items and apply sophisticated analytics in order to optimally adjust
the pace and content of the learning. There are many concerns about
privacy and accuracy of the detection that need to be resolved before
this technology truly goes mainstream, but it is likely that affective
computing will be integrated into mainstream elearning sooner than
many think. One of the first areas it will be applied is in the high-risk-
of-failure professions or job tasks. In these cases, it is critical that
learning objectives are fully realized, and affective computing
provides a powerful means to ensure that.

Because LMSs are already positioned as the default delivery and

tracking function for elearning, they will likely be the focal point for
adoption of affective computing for learning. That is, the application
that does the detection, tracking, adaptive control of the content will
most likely eventually be integrated into the LMS (either natively or as
a plug-in application). Stay tuned for further developments in this
area.

7.25 Support for performance support tools

The idea of electronic performance support has been around since
the early 1990s. It is captured succinctly by the term “just-in-time”
learning. That is, performance support is delivered to the learner at
the time of need, with minimal interruption in the flow of work. There
is usually no explicit intention that users actually retain the
information presented in the tool; users just refer to the information in
the tool at the point of need and then they can forget about it until the
next time they need that information. This paradigm works
particularly efficiently for knowledge and skills that users only
perform once in a while, and tend to forget between instances
requiring them, no matter how much “just-in-case” training they
receive.

Performance support seemed to hit a plateau in the 2000s, possibly

due to hitting the inherent limits of being only available to workers
while they have access to a desktop or laptop computer. This has
now changed with the advent of mobile learning; the devices are
always with you, and can be used to access performance support not
just in just-in-time mode, but just-in-place as well, using location

Choosing an LMS.docx page 177 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

services. Just-for-me (personalized learning) and just-enough

(content that is narrowly focused on only what the user needs)
learning is following on the heels of just-in-time and just-in-place,
often as part of the mobile learning package.

Often, LMSs offer a “browse” mode that allows users to go to any

location in eLearning content modules, without being forced to take
the assessments by any LMS content sequencing rules. In this way,
traditional course material can be used as performance support,
though the usability of this depends a lot on the content and system
design, i.e., how well the content is topically organized, how focused
and self-contained the topics are, search capability, and the detail and
clarity of topical menus both on the LMS side and the content side.

But what about performance support that is specifically designed as

such? Most LMSs can present content assets (especially media files
like videos) as simple objects that are not part of curriculums and
course catalogs, in some kind of list or folder structure of objects.
However, optimally, these performance support objects ideally need
to have a menu system and organization scheme that is organized
around the workflow, not the logical structure of the information. This
could take the form of a timeline or checklist (with performance
support tools embedded into them), rather than a hierarchy of topics.

A key requirement here is for the LMS to provide templates for

different types of workflows and generic work structures (such as the
aforementioned timeline and checklist) that content authors can
populate, so that users can quickly and efficiently navigate to the
performance support objects they need. In the future, system
integration with detection sensors may evolve, such that the LMS can
detect the work context of the user and automatically present the
appropriate performance support object. Integration with mobile
phone location detection capability is already being achieved as the
first step in this direction.

One could argue that a performance support delivery platform is not

really a learning management system, since, technically speaking,
performance support is not really “learning” (or at least not in the
traditional “just in case learning” sense). That is why performance
support delivery platforms do not generally call themselves LMSs;

Choosing an LMS.docx page 178 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

instead, they are performance support platforms or just-in-time

learning platforms. Examples include WalkMe and Trek.

7.26 Is the traditional LMS dead?

There have been pronouncements from some quarters of the training
industry that “the LMS is dead.” On the face of it, this seems like a
gross exaggeration; Brandon Hall (2011) reported that 92% of their
survey participants do not agree that the LMS is dying. Some industry
analysts also have voiced opinions saying the same (Bates, 2012),
citing the need for structure, a private place to work online, a one-
stop shop for tools, and tracking data. And as mentioned earlier,
Bersin (2014) has reported that the LMS market is well over $2.5
billion and grew by over 21% in 2014.

However, as a measure of discontent with the current functional

capabilities of LMSs, Brown et al (2015) report that, in higher
education, “…15% of institutions intend to replace their LMS in the
next three years, which is far higher than typical for enterprise-class
applications.” (p.2). In the training world, Brandon-Hall (2015) report
that “…among the 85% of organizations that use a Learning
Management System, more than 38% of them are looking to upgrade
and replace their current technology.” In the same article, Brandon
Hall speculates whether this is due to a shift towards organizations
making their learning more learner-centric, just-in-time, and focused
on performance. How well LMSs can adapt to this shift, and can thus
maintain their viability as a central repository of learning functions is
not yet clear. In any case, it seems that LMSs are slipping from this
value proposition, at least from the point of view of end-users.

The most obvious contributor to this is the growing trend of informal

learning, where content can be chosen from publicly available
sources by the learner (e.g., found through Google search), learners
can have tracked (via xAPI) learning experiences while disconnected
to the Internet, and content can be generated and shared by learners
through peer networks.

But even with the idea of connected use of authoritative content in

formal learning experiences still on the table, the idea of having to log
in to a monolithic system (LMS) as a one-stop shop for all learning-
related functions and content is disappearing. Learners expect to be

Choosing an LMS.docx page 179 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

able access content whenever and wherever they want. The learning
delivery function and learning content objects are becoming more
distributed and available across systems, contexts, and devices. On-
demand, granular performance support and learning objects are now
embedded in a wide variety of application contexts. More and more,
these learning objects can be launched anytime, anywhere. For
instance:

 A link to a scenario-based learning object within a corporate

intranet page that announces a new policy (to train employees
on how to handle situations that may come up regarding the
policy)

 A link to a compliance training module in an email sent to a

mobile phone reminding the user of an approaching deadline for
taking this training (which can be taken on the mobile device)

 Screens in a new enterprise system that contain embedded

tutorials and performance support

The LMS function needs to operate in the background to

communicate with these learning objects and delivery functions and
provide consolidated, meaningful measures of learning progress to
stakeholders, while being invisible to the learner. They need to fulfill a
key role of being coordinators and enforcers of community policy,
distributing access privileges, user preferences, content brokering,
etc.

In education, the general trend is moving away from course-centric

and instructor-centric and towards learner-centric and learning
experience-centric, as described in 7.3 Support and optimization for
informal/social/collaborative learning. To keep up with this trend,
LMSs need to enable learning more directly through supporting
different forms of learning. They will always maintain their utility as a
learning administration tool, but to the degree that learners expect the
LMS to support the learning itself in its myriad forms, they must adapt
to survive. The problem is that administration of learning in many
LMSs presumes a standard model of courses; those functions
directly influence each other. Building on to the value of the LMS as
an administrative tool without being bound to a traditional model of
teaching and learning may be too difficult.

Choosing an LMS.docx page 180 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

The other related trend that is contributing to doomsday scenarios for

the traditional LMS is the advent of collections of tools that support
self-directed and group-based learning, with great capacity for
flexibility and customization. These tools are assembled by individual
learners to meet their specific needs, especially for informal learning.
The term “personal learning environments” (PLEs) has emerged to
describe this trend, which is particularly centered on smartphones
and tablets as delivery devices, and apps on these devices to support
ongoing learning needs. The PLE trend has thus driven moving away
from centralized, server-based LMS solutions to distributed and
portable ones.

Contrary to these doomsday trends, Bates (2012) concluded based on

a research study that LMSs are not going away because of the
following:

 Most instructors and learners need a centralized online storage

area to organize their teaching plans and materials

 Instructors and learners need a private place to work online,

outside of publicly accessible social media

 Collaboration and social media tools are not exclusive of an

LMS, they can and are being integrated.

 Institutions are becoming increasingly reliant on LMSs for

reporting and accountability purposes driven by an increasing
need for accountability.

It may be that stakeholders have bought into the enterprise LMS

model so firmly that they (perhaps blindly) trust in the ability of LMSs
to modernize and add the functionality they need; in other words, they
assume that LMS will adapt, as they always have, and effectively
address the needs of L&D and availability of modern learning
technologies instead of going away. This is implied in a Bersin (2014)
research finding that “…61% of companies plan on replacing their
learning platforms in the next 18 months, the most frequently cited
product to be replaced.” Nowadays, stakeholders can get
functionality above and beyond the original course delivery platform
model of an LMS. As described elsewhere in this document, LMS
purchasers can get such new collections of functions as an expert

Choosing an LMS.docx page 181 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

exchange, user generated content repository, and talent management

system. There may come a point, however, when bolting features on
to a system based on old paradigms will not meet new requirements
and opportunities; one of the authors recently heard an LMS vendor
at a conference start their pitch with an emphatic statement that their
new system was created from scratch to meet these modern
requirements and opportunities.

The first step away from a traditional LMS that many organizations are
taking is to provide the content delivery function as a service,
separately from the LMS, using a browser plug-in or cloud-based
application. This enables the “launch anywhere, anytime” paradigm
for content. Many LMSs provide an offline player capability (see
section 5.6 Offline player capability) for disconnected use, but it
seems inevitable that they will need to provide this for everyday
connected use as well, given the growing “anytime, anywhere”
paradigm.

SCORM Cloud® made by Rustici Software is an example of an “LMS in

the background architecture”. It allows you to generate “Dispatch”
SCORM packages that you then import into your LMS. When learners
run the course from your LMS, it actually bounces them over to
SCORM Cloud and plays the copy residing there. Your LMS does all
of the tracking as it normally would.

A major challenge in this regard is tracking learner progress. If the

content is no longer being launched from within the LMS, how can it
find and communicate with the LMS? This problem is being
addressed with APIs, standard data elements, and communication
protocols, and will require industry agreements on standards.

The role of an LMS in a use case involving an intelligent tutoring

system is also currently unclear, although this seems that, given the
appropriate back-end channels of communication, an LMS could at
least provide value in terms of the tracking and reporting function. A
possible start towards this integration is the Generalized Intelligent
Framework for Tutoring (GIFT) being developed by the U.S. Army
Research Laboratory. See
https://www.gifttutoring.org/projects/gift/wiki/Overview.

Choosing an LMS.docx page 182 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Robson (2009) presents one possible way that LMS functionality may
be disaggregated and presented as separate services. The
disaggregated services could include the following components:
 Content Orchestration
 Assessment & Evaluation
 Directory Services
 HR Services
 Rights Management
 Search & Discovery
 Competency Management
 Results & Compliance Tracking
 Social Networking
 Content Management

Brown et al (2015) support a vision of disaggregation of the LMS,

saying “..although the [Next Generation Digital Learning Environment]
might include a traditional LMS as a component, it will not itself be a
single application like the current LMS or other enterprise
applications.” (p.3). One interesting possibility they mention is that
the disaggregated collection of LMS components will become “…a
‘cloud-like space’ to aggregate and connect content and functionality,
similar to a smartphone, where users fashion their environments
directly with self-selected apps.” (p.3). They also say that the model
for the architecture of this confederation of components will be the
mash up, enabled by APIs, standards, and reference models. These
mashups could be sold by vendors offering a buffet of component
possibilities that can be combined into unique blends. These
mashups could mix open source and commercial components. One
large vendor (Adobe) is in fact predicating their system (Adobe
Experience Manager) on an open source core, and basing their
business case on selling add-on components.

The term “litigation mitigation system” has been coined to cynically

describe LMSs. This refers to the fact that LMSs are used to track
employee completion and passing of mandatory courses to ensure
corporate compliance with government or corporate policies that
require such training (Information Assurance and Sexual Harassment
courses are examples). This training removes some of the liability
from the company in cases where an employee has behaved

Choosing an LMS.docx page 183 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

negligently, offering the company a defense of “the employee knew

better.” This cynical terminology points out probably one of the most
enduring use cases for LMSs: delivering assessments and tracking
performance on them. The future of LMSs may ultimately hinge on
this use case primarily, which may ensure their survival; all signs are
that this need is unlikely to go away in the near future (unlike the need
to deliver content from a monolithic source).

7.27 Personal Data Lockers (PDLs)

There is rising support for the idea of personal data lockers (also
called personal learning lockers, or backpacks) that can follow a
learner or employee across different schools, jobs, learning modes,
etc. The technology is readily available. All that is needed is
widespread acceptance of a universal secure web service, and
standards for such, that allow a person to keep their own learning
records in the cloud, from whatever sources, and then be able to
share them with others.

If PDLs take off in the market, LMSs will be faced with a number of
issues to accommodate them, such as whether they will be able to
write and receive records from them, managing privacy and data
ownership of these records (i.e., matching the level set by the user for
their PDL itself), and validating prior learning experiences recorded
by the PDL.

The xAPI may accelerate the creation of PDLs, since it provides a way
to communicate disparate forms of learning data to a cloud-based
web service.

7.28 Web-based client systems

The general trend in many types of software, especially in high
security settings, is to minimize the amount of software that needs to
be loaded on client machines in order to run applications. “Thin
client” or “no client” has been achieved in many types of software,
including such computing-intensive applications as VIEs. IT
departments also like this because it minimizes their burden of client
computer configuration management, version control, etc. It includes
not just applications but plugins, drivers, and content. Of course, this
puts more reliance on network connectivity, which is not a safe bet

Choosing an LMS.docx page 184 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

for mobile devices. But for desktop computers, it is enabled by the

steady improvement in bandwidth and server power and capacity,
enabling more of what would normally be handled as client software
functions to be handled on the server side, with the client computer
effectively acting as a “dumb terminal”.

This trend has already taken hold in LMSs; most of them do not
require players or plug-ins. However, some have required a Java
Virtual Machine loaded on the client computer and are now
advertising “Java-free, web-based clients”. In the case of LMSs
written in Java, this is a significant improvement in the sense that the
LMS would no longer rely on a Java Virtual Machine being loaded and
continually updated on the user’s machine, and no Java applets
downloaded (with attendant possible security risks).

Vendors will need to carefully balance performance with this feature,

since processing on the client machine is almost always much faster
than on the server, depending highly on network bandwidth and
server load.

7.29 The Flipped Classroom

A so-called “flipped classroom” is a type of blended learning where
learners are presented new content material online (often via video,
as in the case of Khan Academy), then do what would traditionally be
thought of as “homework” in the classroom. In this classroom phase,
learners usually work through practice exercises and examples and
can ask questions and get explanations of material presented in the
online phase. The classroom is also the place where learners share
learning with each other. Class time can leverage the presence of the
instructor and other learners to inquire about content learned online
and test and reinforce their skills interactively in applying that
knowledge. Classroom time does not need to be wasted in lecture and
other activities where learners need to simply spend time absorbing
the initial new information.

LMSs do not need to have specific features to support flipped

classroom other than the ability to deliver normal asynchronous
content such as eLearning and video. Additionally, they should
support blended learning in terms of associating online content and
classroom sessions with the same learning experience or course. It

Choosing an LMS.docx page 185 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

remains yet to be seen if LMSs, or, at least, enterprise ones, emerge

that advertise specific support for flipped classroom paradigms, and,
if so, what features that would involve other than those mentioned
here.

7.30 Microlearning
The concept of microlearning largely emerged with the advent of
mLearning. It stems from the fact that short, self-contained pieces of
content are better suited for the mobile platform, as opposed to entire
courses. Quinn (2011) calls them “learnlets”, and defines them as
either microcourses (of just a few minutes in length), featuring a
single feature of a product, aspect of a service, or step of a larger
procedure; or a five-screen module consisting of: Intro, Concept,
Example, Practice with feedback, and Summary. A subset of
microlearning is “microlectures”, consisting of short recorded audio
or video presentations on a single, tightly defined topic.

The term “microlearning” has particular connotations (for example,

Quinn’s as stated above) within the eLearning industry, but
“microlearning” can be broadly applied to any learning asset of about
5 minutes or less duration. It is often associated with blended
learning, where mixed-mode microlearning assets can combined
flexibly within a learning experience. These “blended” assets could
be a mix of performance support and training modules and well as
eLearning and instructor-led training. Short videos (e.g.,
microlectures) often form the backbone of solutions involving
microlearning. Sites such as Khan Academy and TED-Ed specialize in
microlectures, which provide particular support for the flipped
classroom model (see 7.29 The Flipped Classroom).

One of the central tenets of microlearning is that each asset is

relatively self-contained, so that the learner can actually complete it in
the prescribed short session, rather than accepting an “incomplete”
status by leaving a module (albeit perhaps with bookmarking) while it
is in progress. It usually does not have strong dependencies on
learning from other content.

Microlearning often involves user-generated content, for instance,

videos recorded by users on particular topics relating to their work
tasks. Video sharing is emerging quickly as a way for employees to

Choosing an LMS.docx page 186 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

share best practices and knowledge. Cavalier (2015) recommends that

“micro-videos” be limited to 60 seconds or less, with the following
format:

1. Title Bumper (5 seconds)

2. SME/Concept Introduction (10 seconds)

3. Concept/Step 1 (10 seconds)

4. Concept/Step 2 (10 seconds)

5. Concept/Step 3 (10 seconds)

6. Summary/Next Steps (10 Seconds)

7. Closing Bumper (5 Seconds)

The advantages of microlearning are the following:

 Learners, instructors, and administrators have more flexibility to

customize learning programs by sequencing small
microlearning assets that are each finely tuned to a specific
topic or learning objective. Microlearning, especially in the form
of microlectures, can be inserted at any point in a sequence of
learning activities that constitute a learning program, including
after the program begins or after it ends.

 Content brokering and adaptive learning scenarios are easier to

implement.

 It is easier for content authors to update the content, since the

object generally does not involve large complex files and the
impact of changes is self-contained within each file.

 Microlearning doesn’t tend to interfere with work as much (it can

be done in small sessions—while waiting for a conference call to
begin, for example).

 Microlearning is much more suited to “on the go” mobile

learning. Learners can take the learning during otherwise idle
moments. It is also quicker to download, which is helpful for
mobile delivery.

Choosing an LMS.docx page 187 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 It is well adapted to spaced learning scenarios (for more

information on spaced learning, see 8.19 Support for mobile
learning).

 It is consistent with learning science research that posits

average attention spans of about 8 seconds in the general
population (reported in Makhlouf, 2015). Given this fact,
microlearning can be seen as an effective way “…to allow the
learner to experience, absorb, and move on to the next task
before losing interest.” (Makhlouf, 2015).

The implications for LMSs of microlearning are the following:

 There needs to be more focus on flexibly-organized libraries of

learning or performance support assets rather than sequences
of courses, or curricula.

 Microlearning tends to gravitate the learning ecosystem

naturally towards a true performance support environment,
where all learning is consumed in “just-in-time” mode. They also
move the environment towards a knowledge management
ecosystem as well.

 Integration with YouTube, SMS, and Twitter is key, since these

are ideal platforms for microlearning. There are also video
content management systems like KZO that offer more control
and management options over video delivery, with added
functionality for end users as well.

 Metadata tagging (and probably paradata as well) is more

important, since it allows users to search for relevant content
more efficiently and assemble these content objects into
complete learning experiences.

 Microlearning is well suited to an LCMS environment, where

sequences of assets can be dynamically assembled at run time.
In other words, microlearning objects can easily be linked
together like Lego blocks.

Video-based LMSs (most of which are predicated on the

microlearning concept) are now starting to appear. These are now

Choosing an LMS.docx page 188 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

being called Video Content Management Systems (VCMS). Products

include the following:

 Brightcove®

 Kaltura®

 KZO®

These systems feature such things as adaptive bitrate (adjusting the

quality of the video to the available bandwidth), search, enhanced
security and role-based access, concurrent images (e.g., in
PowerPoint) timed to display as the video plays, and synchronous
live video lecture/discussion. Some organizations use these systems
mainly for storing and delivering recorded videos repurposed from
recordings of live virtual classroom sessions. Others use them for
videos generated by end users, SMEs, or training staff, specifically to
be used as asynchronous content.

One unique feature appearing in some of these systems (KZO in

particular) is the ability for end users to overlay comment annotations
on the video as they are watching it. These can then be read by other
users of the video later, and a discussion forum created around the
comments, with users able to click comments and link directly to the
pertinent location in the video.

As with any environment involving user-generated content, there

needs to be a governance and workflow structure for microlearning
videos, to include (Cavalier, 2015):

1. Proper format of the video

2. Metadata and proper tags for video searching

3. Review of video prior to ingestion into the system

4. Voting or likes to make the “cream rise to the top”

5. Empowerment of all subject matter experts

6. Training on how to properly shoot, edit, and post video for

employees

Choosing an LMS.docx page 189 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

LMSs that handle microlearning and microlearning videos need to

account for these in their features and design. This is especially
important in an environment where you allow “self-service videos”,
i.e., users can generate their own videos and post them into the
VCMS. Note that metadata tagging in this case usually requires
changes to business policies and procedures (and enforcement
mechanisms, such as metadata checks that do not allow publishing
of content objects unless the system detects that they have been
properly tagged), since it requires extra work that end users will try to
avoid if they can.

Microlearning videos can be of various lengths, depending on the

limitations of the platform. Here are some examples:

 YouTube® = 10 minutes

 Twitter® = 30 seconds

 Instragram® = 15 seconds

Barry (2016) reports that “According to research conducted by

Bersin, most learners won’t watch videos longer than four minutes.”
No matter what the platform, this sets a realistic upper limit for
microlearning video.

As for much shorter “nanovideos”, it generally does not make sense

for videos of 15 seconds or less to have an audio track. To convey the
learning message effectively, this makes it even more important for
them to be carefully scripted.

Ultra short length microlearning videos often show a process in fast

motion, with the ability to click to step through it in normal or slow
motion. It is also very important to tag microlearning videos, or
microlearning of any kind, with metadata so that individual
microlearning content pieces can be assembled into a meaningful
whole learning experience.

If you are considering incorporating microlearning videos into your

ecosystem, whether you use a VCMS or not, it is very important that
you consider the content management aspect, in order to deal with
issues such as:

Choosing an LMS.docx page 190 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 The difficulty of capturing usage tracking for downloaded

videos. A VCMS, by the fact that videos are streamed from it, is
positioned to handle detailed usage tracking.

 Rules and permissions for creation and use of videos, especially

user-generated videos. Some regulatory environments (like
medical, with HIPAA requirements) may have strict rules for
details that can be shown in the video, or need-to-know based
permissions for viewing them.

 Pushing videos to users (perhaps in a “daily drip”) rather than

simply offering them in a “pull”-based library

One value proposition for a VCMS is the ability to easily capture data
analytics about learner usage patterns (e.g., at what point did users
tend to stop watching? What parts did they rewind and watch over
again?). This relies on streaming of videos from a VCMS; download
and then local storage and play of videos presents a much more
complex challenge to capture data from (xAPI profiles may come in
handy for this).

Commercial cloud-based video sites such as YouTube ® and Vimeo®

can also be used for content management of microlearning videos,
usually at no or very low cost. They often offer private channels.
Despite this, some enterprises do not want their proprietary material
uploaded to a third party site.

One interesting consideration for microlearning videos that can be

used to garner support for them from enterprise fiscal managers is
the fact that they can be put on the books as corporate assets, in the
same way that an entertainment company like Disney considers them
corporate assets.

7.31 Data analytics

Wikipedia cites considerable disagreement among experts as to a
definition of learning analytics, but uses as a starting point “...the use
of intelligent data, learner-produced data, and analysis models to
discover information and social connections for predicting and
advising people's learning." (Wikipedia, 2015). It also differentiates

Choosing an LMS.docx page 191 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

educational data mining from learning analytics, saying that the

former is not hypothesis-driven, in contrast to the latter.

Whatever academic definition one chooses, the broad, practical

aspects of measurement in a learning ecosystem is something that
LMSs are starting to pay more attention to. It is not that measurement
was not possible or important before, but now the xAPI spec enables
you to capture much finer detail of more parameters in an
interoperable way. For more information on the xAPI, see 4.15.8 ADL
Total Learning Architecture (TLA), including xAPI. Through its
semantically-based, flexible data model, it facilitates a new level of
analysis, by fancy data visualization engines or simple rubberneck
checks of data tables, to elegantly answer the perennial questions:
Who? What? Why? Where? When? How?

Taking it down a level, here are some examples of the questions that
can be answered using a combination of xAPI for data capture and
some kind of analytics engine for information output:

1. How well am I doing in this learning experience? (individual

learners)

2. Which learners require or are going to require extra support and

attention, and in what specific areas? (instructors)

3. What design features of learning experiences are most effective

in producing learning in a particular context? (designers)

4. What are the most cost-efficient learning interventions?

(stakeholders)

5. How are particular learning resources actually being used?

(content authors and managers)

6. What are the best logistical arrangements for marketing and

delivering the course? (administrators)

No longer do learning professionals need to be limited to the canned

reports produced by LMSs. Traditional LMS reports have served many
well for a long time, but we are now in the era of data-driven decision
making in the learning space. Data-driven decision-making requires
breaking open the black box of data capture and reporting functions

Choosing an LMS.docx page 192 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

within LMSs to provide a much wider range and depth of information

than can be provided with predefined reports. The xAPI is also the on
ramp towards separating analytics from course delivery and
management functions, which is important given that more and more
content is launched and/or experienced outside of the LMS.

There is some question of how the LMS fits into this xAPI-driven data
analytics picture, since the xAPI does not require an LMS (it requires
a web service called a Learning Record Store (LRS)). However, some
solutions are starting to emerge whereby the LMS retains its
centrality as the repository of learning and learner-related data and
records, by embedding an LRS function or by communicating with an
LRS service. It remains to be seen whether robust data analytics
capability (which would include not only xAPI capability but a data
analytics engine) will become part of the design of LMSs, or will be a
separate system.

With the resurgence of performance support (in many cases,

replacing training), there is a greater need for custom data capture
and analytics solutions, solutions that are difficult for LMS to manage.
One simple reason for this difficulty is that performance support is by
definition devoid of assessments, which are the primary vehicle for
measurement for content in an LMS. Performance support tools
require data on whether and how they are being used (called
“paradata”), not how much learners have learned from them. Paradata
for both performance support and instructional content may include a
range of individual or aggregate user interactions such as viewing,
downloading, time/place/situational context of use, sharing with
others, rating, and using content for derivative products. Silvers &
Torrance (2015), propose the following categories generally related to
paradata:

 Sentiment analysis - What do the words people use tell us about

their disposition to learn?

 Engagement analysis - What’s the activity level with learning

content?

 Cohort analysis - Who forms what groups for what reasons?

Choosing an LMS.docx page 193 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Keyword analysis - How do people seek info & what do they

find?

 Conversion Rate - How many people respond (i.e., comment)?

 Amplification Rate - How many times is something shared?

 Applause Rate - How many likes/favorites/bookmarks?

 Economic Value - Short/Long Term Revenue/Cost Savings?

The xAPI is especially useful for performance support given that work
behavior, output, and productivity can be conveniently measured with
it as well. In other words, you can use the xAPI to instrument the work
environment in addition to the learning environment, and the
integration of these analytics can be very powerful in creating
feedback loops to fine tune your learning interventions (as well as
business processes). Kirkpatrick Level 3 and above evaluations can
be more easily institutionalized within your learning ecosystem in this
way.

The xAPI can not only bridge work and learning data, but it can bridge
a learner’s physical state over time with learning activities or work
performance so that, for instance, heart rate can be correlated with
work or learning tasks to determine points of high stress.

Data visualization that allows recognizing complex patterns and

trends is an important capability enabled by the xAPI. Because the
xAPI allows precise, microscopic statements describing a learner or
system’s state at a specific point in time, trends can be seen easily
with graphs, diagrams, etc. You do not need to create expensive
custom visualization engines to do this. Open source solutions are
available such as ADL’s xAPI Dashboard
(https://github.com/adlnet/xAPI-Dashboard) and Apereo Open
Dashboard
(https://github.com/Apereo-Learning-Analytics-Initiative/OpenDashbo
ard)

The obvious, traditional approach to analytics is to plan your

analytics solution (using xAPI in this case) to answer specific
questions first, then capture data. This works where you have specific

Choosing an LMS.docx page 194 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

measurement needs that are already clearly defined, usually

resembling the typical reports provided by LMSs.
For those inclined towards thinking of data analytics more as research and data mining (ie, without an
initial hypothesis, as mentioned above), the xAPI provides a durable, interoperable basis for analytics
engines to create visualizations that can reveal unexpected patterns. One can think of this approach as
“measure first, ask questions later” - in other words, capture lots of different kinds of data just because
you can, and then explore to see what emerges from it from analysis. The steps for this kind of research-
oriented approach could be:

1. (optional) Formulate baseline research questions. You need some idea of these, even if you are
using this exploratory approach, as a basis for Step 2 below and data analysis/visualization
methods later.
2. Decide what interaction nodes and learner behaviors in the learning experience make sense to
instrument with xAPI.
3. Decide what granularity you need and the right syntax and verbs for your xAPI statements. This
is essentially becomes your hypothesis, if you are using one.
4. Deploy xAPI-instrumented learning experience and collect data.
5. Validate data received against research questions
AND/OR
Look for patterns
6. Refine xAPI granularity, verbs, LRS queries, etc.

For those inclined towards data modeling and “what if” scenarios, not
only can historical data be collected and subjected to various
analyses after the fact, but specific hypothetical data (ie, xAPI
statements) can be substituted for real historical xAPI statements.
The xAPI allows you to insert these hypothetical statements in a
surgically precise way and then play out the scenario in your data
analytics engine, to see what results could emerge that are different
from the real results.

8. Process for choosing an LMS

ADL recommends the following high-level process for choosing an
LMS.

1. Hold stakeholder meetings to determine the basic feasibility of

an LMS acquisition, and how your organizational goals can be
met with it. You need to answer such questions as: What
business problems do you hope to solve with it? What are the
risks? What resources will it require? What new processes and
business rules will it require? What data will it collect? All of this
needs to be looked at under the lens of feasibility. For instance,
if new processes and business rules are required, who will

Choosing an LMS.docx page 195 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

create and enforce them? If you are going towards a

competency-based HR environment, who will create and
maintain the competency data, and who will verify mastery of
competencies? In these meetings, be sure to include all cross-
functional stakeholders for whom implementing the LMS will
have direct or indirect (especially financial) consequences. This
includes HR, T&D (Training and Development), CEO and senior
leadership, and IT staff.

2. With stakeholders, decide on a process and timeline (preferably

with a formal project plan) for how the LMS acquisition project
will proceed, using the high-level steps outlined here, or some
other process.

3. Determine the high-level requirements for your LMS, in each

LMS functional area described in 2.1 What is an LMS? Ensure
that you get input from all groups of potential users, not just
stakeholders, and solicit input from your HR and IT departments.
It is important to stick to only the critical, high-level, and highly
differentiating requirements at this point. That will serve to
quickly filter many unsuitable candidates when you get to step 7
below. This may require a formal requirements definition effort,
especially if you are a large enterprise with many different
groups of potential users who may have different (and hard to
predict) needs.

Be aware that there are many types of requirements (functional,

usability, etc.), representing different points of view (users,
administrators, stakeholders, etc.). See Wiegers’s (2000) article
at http://processimpact.com/articles/reqtraps.html for
information on how to avoid “requirements traps” such as
ambiguous or vague definitions.

If you have never used an LMS before, you may want to consider
gaining a year of experience with a simple, inexpensive or
homegrown system before you buy a major enterprise system.
This could help clarify your goals and requirements
substantially.

Some important general considerations that may impact your

list of high-level requirements at this point include:

Choosing an LMS.docx page 196 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 Whether you will need support for compliance training. This

will require robust tracking features and probably certain
kinds of reports.

 Whether you need to deliver commercial off-the-shelf (COTS)

content, as opposed to content you develop yourself. In the
former case, you will need to ensure that the COTS content
will run successfully in the LMS.

 Whether you will need your LMS to focus broadly on HR and

Talent Development issues rather than strictly on traditional
training.

 Whether you want an “all in one” system that contains

everything you need, or whether you already have some LMS
software functions or components in place that you do not
need included in the LMS. Even if you do not have these
functions already, you may be planning to accumulate them
gradually outside of the LMS you purchase.

4. Determine your budget for purchasing the system and

associated support/training contracts, as well as any
customization you need that you predict that the system will not
provide out of the box. Your budget should ideally be not simply
based on available funds, but a cost-benefit analysis of
implementing the system; at the very least, the cost of the
system should not exceed the true cost of not solving the
training problems that you would be counting on the LMS to
solve. Assigning dollar values to employee training problems is
notoriously difficult, but when acquiring a large expensive
system, a formal cost-benefit analysis may be worth it. (See 4.4
Pricing models for more information about pricing.). You may
want to explore cost sharing opportunities between your
organization and others that may benefit from the system.

5. Determine the category of system you will need (see 3.

Categories of systems to deliver and manage learning) and
types of learning you need to deliver (see 2.4 Types of general
learning goals managed by LMSs). If there are only certain major
capabilities that you really need, you may be able to save money
by buying only the components or services you need. If you

Choosing an LMS.docx page 197 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

already have a CrMS, for instance, you want to consider

acquiring or developing just the course delivery module, or vice
versa, instead of an entire LMS.

6. Identify specific systems that match the category and support

the types of learning you identified in step 5. Because these
categories overlap, you may identify more than one category for
consideration. You may decide at this point to develop your own
product rather than purchase a COTS LMS. Note that if you are a
U.S. government entity, the government acquisition process
requires justifications for acquisition choices. You will need to
validate or justify your decision to develop your own system (vs
buy a COTS product).

7. Develop and populate a system requirements matrix that allows

assessing the systems identified in step 6 against your
requirements developed in step 3. See the Appendix A Sample
System Requirements Matrix for a sample. If you are considering
more than one category of system, you may want to complete a
separate matrix for each different category of system you have
identified as a requirement for your organization, since each
category of system has its own distinct parameters and typical
feature sets. After completing the separate matrices, you will
then need to decide which category you will pursue, if you are
intent on or limited to purchasing only one system.

8. Filter the list of potential candidates, eliminating those that do

not meet your minimum requirements and/or are over your
budget. It is important to focus on your core needs - use
weighting in the provided selection matrix (see Appendix A:
Sample System Requirements Matrix) to establish the absolute
vs “nice to have” requirements.

9. Create and send your list of requirements to remaining

candidates. This includes requests for information (RFIs) or
requests for proposals (RFPs) if necessary—whatever formal
documentation is required for your acquisition process.
Templates for these documents are usually prescribed within
corporate or government organizations. If not, you can find
templates on learning technology consulting firm web sites,
LMS vendor web sites, or by searching on the Web. Note that

Choosing an LMS.docx page 198 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

some small LMS vendors may consider lengthy, detailed RFPs

onerous to respond to, thus may decline to respond.

Altieri (2016) recommends the following when writing

requirements lists for vendors:

 Add one bogus, nonsensical requirement to any list of

requirements sent to vendors, as a check against whether
the vendors actually read the list. If they say that they can
meet all of the requirements, or do not ask about that
requirement, they can be ruled out.

 Don’t mention your current system (i.e., what it cannot do

that you need in a new system)

 State requirements in the positive, not negative.

 Avoid jargon and acronyms or provide a glossary.

 Consider what your needs will be at the time the system

will actually be deployed, not what they are now.
Remember that implementations often take a full year from
actual acquisition. Requirements may change during this
time on a range of issues, from capacity of concurrent
users to types of learning supported.

 Plan to have your LMS for 5 years—that is a typical “useful

life” duration for an LMS, due to changes in learning
technology space that may make it obsolete.

10. Compile a detailed, comprehensive features list for all of

the remaining candidate systems. You may want to start this list
by sampling the features of one system that seems to be the
most feature-rich, and add any features uncovered by your
analysis of other systems as you complete the comparison
process. Or, you can use some or all of the requirements
mentioned in 5. List of possible requirements for an LMS as your
features list. You may want to edit this list of features to only
those that you care about now; however, this may be limiting
since you may be unfamiliar with the usefulness of some
features, or they may become useful sometime in the future.

Choosing an LMS.docx page 199 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

11. Develop a system features rating matrix (see the Appendix

B: Sample System Features Rating Matrix for a sample) that
compares the systems filtered in step 8 using the features list
developed in step 9. Complete as much of this matrix as
possible from the systems’ documentation; if you need more
information, ask their sales representatives for it (though beware
of overblown claims—verify lofty ones independently if
possible). Follow the instructions in the supplied matrix in the
Appendix to assign a numerical rating for each cell in the matrix,
indicating degree of implementation of that feature, and assign a
weighting score to weight each feature according to its
importance to you, enabling a rollup score for each system. This
scoring regime should ensure that an abundance of only nice to
have features does not overshadow the lesser number of core
features you must have.

12. Contact the top scoring vendors (three to five is a

reasonable number) from the previous step and ask for a
presentation/demo. Ask the vendor for a demonstration in your
facility, running your content on their system. The vendor may
want to present a canned demo of their product using
PowerPoint or Flash, and that is fine as a general overview of
the system’s capabilities, but you should see how well the
system expresses these capabilities within your IT environment
using real content. You might also want to ask vendors to
provide a list of three customers who would be willing to host
site visits or talk to you without the vendor present. Some
experiences you might want to ask these customers about are:

 Contract negotiations

 Customizations and turning on/off baseline features

 Implementation process

 Responsiveness and quality of support

You can also investigate blogs, reviews (often offered on

professional organization sites) and other online resources to
assess the quality of the vendor.

Choosing an LMS.docx page 200 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

It is recommended that you consider creating use case scripts

(scenarios that will demonstrate the system's ability to meet
your specific needs), representing common, mission-critical
tasks that an LMS user would perform. During their
demonstration, the vendor performs the steps required to fulfill
each use case. This is a good way to evaluate how effectively
and smoothly the system maps into your use cases. You can
also request that the vendor set up a sandbox for hands-on
testing with the system by your administrators, instructors, and
learners. LMS acquisitions are usually expensive, so it is not
unreasonable to ask for this. See 4.20 The path of least
resistance for an important caveat in evaluating features.

It is important to establish a firm, contractually-binding baseline

of what you would be buying “out of the box” vs what would
require customization above and beyond that baseline. Some
vendors may blithely tell you that their system can meet certain
requirements of yours, but what it really means is that the
system has an architecture that allows integration of those
features with some amount of customization, which is an
additional charge. You should clarify with the vendor what
constitutes “customization” (ie, requires actual programming)
vs “configuration” (ie, changes that can be made by the system
administrator without any programming and system integration).

You may be able to negotiate using the product free for a limited
trial period. This can be very valuable for gathering user
feedback and getting an idea of what the vendor relationship will
be like.

13. Augment the matrix with the additional information gained

from step 11, adding any impressions and notes from the vendor
demos.

14. Make your decision based on feature comparison

(including the weighting you have assigned for each feature)
and experiences from the demo sessions, taking into account
TCO (total cost of ownership), including the application, training,
“software assurance” (yearly cost that includes upgrades,
version releases, etc.), maintenance, hardware that you will need
to run it on, etc.), customer support, and any intangibles. Total

Choosing an LMS.docx page 201 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Cost of Ownership (TCO) is usually a 5-7 year window for LMSs.

As enterprise systems usually require a minimum server
architecture and LAN support, another consideration is whether
a hosted solution (see 4.10 Hosting options) or component-
based architecture solution (see 7.13 Component-based
architecture) may be right for you, if one is available from the
vendor. Get someone (who may not be in your learning
organization) who has negotiation skills and experience
involved to negotiate such important terms as pricing and
licensing.

Holloway and Armstrong (2015) describe a similar process to the

above.

Other processes for selecting LMS or other large systems that rely
more heavily on demonstrations of use cases are possible, for
example, Brandon Hall’s method for selecting an LMS (Brandon Hall
Research, 2011, and Brandon Hall Group, 2012).

A solution that provides 80% of your needs out of the box is generally
a reasonable target; you will probably not get everything you want
without some customization, or resorting to other systems.

After making your decision, be clear in internal communications what

the system can and cannot do. In other words, “promise low, deliver
high”. Make it clear to all of those who will use the system in your
organization what new roles and responsibilities they will have to take
on due to implementing the system, and get their buy-in early on. It is
unrealistic and unfair for them to expect that system administrators
will do everything for them. As users of the system, they should
experience tangible benefits (if they don’t, you need to reevaluate
your requirements). They should understand that “to get, they have to
give”.

After you acquire your new system, before you actually go live, there
are a number of important steps you need to plan for, including
migrating data, marketing, acceptance testing, help desk preparation,
governance, and service interruptions. See Foreman (2013) for
information on issues that arise when changing from one LMS to
another. Lindenberg (2012) describes implementation issues for any
LMS purchase situation, including how to market your new LMS to

Choosing an LMS.docx page 202 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

stakeholders and end users. Porto (2014) describes cultural and

perception issues in migrating to a new LMS. Finally, Ryan et al (2012)
describe a case study of an LMS migration.

9. For more information about LMSs

 Bersin by Deloitte
http://home.bersin.com
This company sells a variety of resources and services related
to eLearning, including buyers guides, comparative ratings, etc.
to aid in the process of choosing an LMS.

 Brandon Hall
http://www.brandon-hall.com
This company sells a variety of resources and services related
to eLearning, including buyers guides, comparative ratings, etc.
to aid in the process of choosing an LMS.

 DOD Instruction 1322.26, Development, Management, and

Delivery of Distributed Learning, June 2006.
http://www.dtic.mil/whs/directives/corres/pdf/132226p.pdf
This document describes DoD requirements for content and
LMSs regarding SCORM conformance.

 Edutools
http://www.edu-tools.info/
This community-driven site offers a variety of resources and
services related to eLearning, including buyers guides,
comparative ratings, etc. to aid in the process of choosing an
LMS (though it mainly focuses on CrMSs).

 eLearning Guild
http://www.elearningguild.com
This professional membership-driven site offers a variety of
resources and services related to eLearning, including buyers
guides, comparative ratings, etc. to aid in the process of
choosing an LMS.

 E-learning! magazine
http://www.2elearning.com/

Choosing an LMS.docx page 203 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

This free magazine contains buyer’s guides and articles that

may be helpful for those involved in choosing an LMS.

 E-learning Centre (UK)

http://www.e-learningcentre.co.uk/
This site is sponsored by a non-profit eLearning consulting
organization. It contains free information resources related to
eLearning systems and tools, including reviews to aid in the
process of choosing an LMS.

 eLearning Industry
http://elearningindustry.com/learning-management-systems-
comparison-checklist-of-features
This site provides a list of features that can be used to compare
LMS vendors.

 Directory of Learning Tools (Centre for Learning and

Performance Technologies) http://c4lpt.co.uk/
This site is sponsored by a non-profit eLearning consulting
organization. It contains free information resources related to
eLearning systems and tools, including reviews to aid in the
process of choosing an LMS.

 Rustici Software
http://www.scorm.com/scorm-explained/scorm-resources/
This site provides a publicly available SCORM-conformant LMS
that can be used for testing and demonstration. It also has a
variety of information pages including such topics as what to
ask for in your LMS RFP to ensure SCORM is what you want,
need, and expect (see
http://www.scorm.com/scorm-explained/scorm-resources/what-
to-ask-about-scorm-in-an-rfp/)

 Tagoras, Inc.
http://www.tagoras.com/catalog/association-lms/
This consulting company has a large collection of resources for
LMS purchasers, particularly oriented towards LMS use in trade
and professional associations

Choosing an LMS.docx page 204 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

 TrainingIndustry.com
http://www.trainingindustry.com/learning-communities/lms-and-
lcms.aspx

This site has a Supplier Directory for LMSs and LCMSs, with
company profiles.

 Training Media Review

http://www.workplacepublishing.com/TrainingMediaReview.htm
This membership-driven site offers a variety of resources and
services related to eLearning, including buyers guides,
comparative ratings, etc. to aid in the process of choosing an
LMS.

 Vendors of Learning Management and ELearning Products

http://www.trimeritus.com/vendors.pdf
This free report (updated 11/11/14) provided by Trimeritus
Elearning Solutions, Inc. includes a lists of LMSs and other
eLearning products.

10. References cited in this paper

Altieri, A. (2016). Choosing an LMS-Tips, Tricks, and Traps. Presentation at Learning Solutions
conference, March 16-18, 2016, Orlando, FL.

ASTD (2008). ASTD Handbook for Workplace Learning Professionals. Biech, E. Editor. Alexandria,
VA: ASTD Press.

Berking, P. (2015a). Choosing Authoring Tools. ADL white paper available at: http://adlnet.gov/adl-
assets/uploads/2016/01/ChoosingAuthoringTools.docx

Berking, P. (2015b). Choosing an LRS. ADL white paper available at:

https://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAnLRS.docx

Barry, S. (2015). 4 Ways to Engage Your Workforce with Mobile Learning. Lynda.com tech report
retrieved 5/3/2016 at http://pages.lynda.com/rs/063-DFS-642/images/Mobile-Learning-v02.14.pdf

Bates, T. (2012). “Why learning management systems are not going away”. Blog post retrieved 4/4/12
from:
http://www.tonybates.ca/2012/04/04/why-learning-management-systems-are-not-going-away/

Beam, P., & Cameron, B. (1998). But what did we learn …?; evaluating online learning as
process. Paper presented at the Sixteen Annual International Conference on Computer
Documentation, Quebec, Canada.

Choosing an LMS.docx page 205 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Bersin, J. (2014). “The Red Hot Market for Learning Management Systems”. Web article published
8/24/15. Retrieved 10/6/15 from http://www.forbes.com/sites/joshbersin/2014/08/28/the-red-hot-
market-for-learning-technology-platforms/#f460d0837fc7

Brandon-Hall Group (2016). “2016 Learning Technology Study: Summary of Top Findings”. Research
report.

Brandon-Hall Group (2015). “Thinking of Switching Your LMS?” White paper September 2015

Brandon-Hall Group (2012a). “Replacing your LMS?” Research brief 2/12.

Brandon-Hall Group (2012b). “De-Mystifying the LMS Selection Process”. Webinar 08/30/12

Brown, M., Dehoney, J., and Millichap, N. (2015). The Next Generation Digital Learning Environment:
A Report on Research. Educause Learning Initiative paper.

Cavalier, J. (2015). “Micro-learning Formats for User-Generated Content”. Slide presentation from
mLearnCon conference 2015. Retrieved 8/3/2015 from:
http://www.elearningguild.com/mLearnCon/sessions/session-details.cfm?session=6570

Cohn, J. and Fletcher, J.D, (2010). What is a Pound of Training Worth? Proceedings of the 31st
Interservice/Industry Training, Simulation and Education Conference. Orlando, FL. Retrieved
11/3/2016 from
https://www.researchgate.net/publication/264897150_What_is_a_pound_of_training_worth

Content Analyst Company (2015). “Game Changing Breakthroughs in Knowledge Management for
Unstructured Content”. Webinar slides retrieved 8/24/15 from:
http://www.kmworld.com/webinars/enter.aspx?eventid=800

Department of Defense Office of the CIO (2009). Clarifying Guidance Regarding Open Source Software
(OSS). Memorandum from the Chief Information Officer. October 16, 2009. Retrieved 4/10/2010
from http://dodcio.defense.gov/Portals/0/Documents/FOSS/2009OSS.pdf

Elearning! (2012). “THE PULSE: Social Learning Still New” (Sept/Oct 2012) Vol. 8, Issue 5, 11.

Fletcher, J.D., and Foster, R. (2002). Computer-Based Aids for Learning, Job Performance, and Decision
Making in Military Applications: Emergent Technology and Challenges. Institute for Defense
Analyses document D-2786. Retrieved 11/2/2016 from
http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA413872

Foreman, S. (2013). “LMS Operation and Governance: Taming the Beast (Part 3 of 4)”. Learning
Solutions. (September 2013). Retrieved 4/14/2014 from:
http://www.learningsolutionsmag.com/articles/1244/lms-operation-and-governance-taming-the-beast-
part-3-of-4

Foreman, S. (2013). “The Six Proven Steps for Successful LMS Implementation (Parts 1 and 2)”
Learning Solutions. Elearning Guild Learning Solutions magazine (July 29, 2013). Retrieved 10/6/15
from:
http://www.learningsolutionsmag.com/articles/1214/the-six-proven-steps-for-successful-lms-
implementation-part-one

Choosing an LMS.docx page 206 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Gallagher, P. S. (2007). Assessing SCORM 2004 for its Affordances in Facilitating a Simulation as a
Pedagogical Model. Doctoral Dissertation, George Mason University, Fairfax, VA.

Gilhooly, K. (2001). Making e-learning effective [Electronic Version]. Computerworld, 35, 52-53.
Retrieved August 5, 2007 from Research Library Core database. (Document ID: 75514405).

Government Elearning! magazine (2012). “LMS Spending Grows”. News article posted 11/07/12 03:16
PM.

Hall, B. (2002). Six steps to developing a successful e-learning initiative: excerpts from the e-learning
guidebook. In A. Rossett (Ed.), The ASTD E-learning Handbook (pp. 234-250). New York: McGraw-
Hill.

Holloway, C. & Armstrong, S. (2015). We Are Ready for a New LMS... Now What? Slide presentation
presented 10/2/15 at DevLearn 2015 conference.

Hougan, S. (2015). Five Tips and an RFP Template: Choosing the Right LMS. ELearning Guild Learning
Solutions magazine (April 8, 2015). Retrieved 10/7/15 from:
http://www.learningsolutionsmag.com/articles/1668/five-tips-and-an-rfp-template-choosing-the-right-
lms/

Kapp, K. (2012). The Gamification of Learning and Instruction. San Francisco: Pfeiffer Publishing.

KZO (2015). Guide to Safely Sharing Video Across Your Enterprise. KZO, Inc. web article retrieved
7/24/15 from:
http://cdn2.hubspot.net/hubfs/157032/Guide_to_Safely_Sharing_Video_Across_Your_Enterprise_06
232015.pdf?submissionGuid=b1a55005-3238-4ad3-9fb9-abb50bdcf82d

Learning Systems Architecture Lab [LSAL]. (2004). SCORM Best practices guide for content developers
(2004 ed.). Pittsburgh: Carnegie Mellon University.

Lindenberg, S. (2012). Implementing and Migrating the New LMS. ATD web article (November 15,
2012). Retrieved October 7, 2015, from
https://www.td.org/Publications/Newsletters/Links/2012/11/Implementing-and-Migrating-the-New-
LMS

Makhlouf, J. (2015) Microlearning: Strategy, Examples, Applications, and More. eLearning Mind web
article. Retrieved 4/20/15 from:
http:/elearningmind.com/microlearning/

Masie, E. (2013). “Social Learning: Design Models?” Blog at

http://www.learning2013.com/blog/item/models-for-social-learning.html

Mayer, R. (Ed.) (2005). The Cambridge Handbook of Multimedia Learning. New York: Cambridge
University Press.

McIntosh, D. (2014). “Vendors of Learning Management and E-learning Products”. White paper.
Retrieved November 2014 from:
http://www.trimeritus.com/vendors.pdf

Choosing an LMS.docx page 207 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

NetDimensions (2015). LMS: Evolution or Extinction – 8 Trends that Change Everything. White
paper retrieved 6/18/15 from:
http://www.netdimensions.com/downloads-2015/infographics/lms-evolution-or-extinction.pdf

Oakes, K. (2002). E-learning [Electronic Version]. T + D, 56, 73-75. Retrieved March 2002 from
Banking Information Source database. (Document ID: 111468179).

Poiry, B. & Wilson, J. (2013). Opportunities for Hands-on Activities in Virtual Instructor‐Led Training.
Slide presentation for Washington Interactive Learning Technologies Conference sponsored by
Society for Applied Learning Technology (SALT), Reston, VA, August 14-16, 2013.

Porto, S. (2014). Navigating the Waves: Curtailing Fear while Managing the LMS Migration. The
Evolllution web article (1/30/14). Retrieved 10/7/15 from:
http:/evolllution.com/opinions/navigating-waves-curtailing-fear-managing-lms-migration/

Quinn, C. (2011). Designing mLearning. San Francisco, CA: Pfeiffer Books.

Quinn, C. (2015). It’s Time to Do Learning Like Grown-ups: Content Systems. DevLearn 2015
conference presentation. Retrieved 11/20/15 from http://www.elearningguild.com/conference-
archive/index.cfm?
id=6941&from=content&mode=filter&source=sessions&showpage=6&sort=titleasc&type=DevLearn
+2015+Conference+%26+Expo

Robson, R. (2009). Slide presentation titled Global Learning Summit: Present & Future of
eLearning Infrastructure (27 Feb 2009).

Roche, J. & Upton, C. (2012). Learning & Talent Systems – Buying on the Rise, Again!. Elearning!
(July/August 2012), Vol. 8, Issue 4, 21-22.

Roche, J. & Upton, C. (2013). Learning & Talent Systems Core to Driving Performance. Elearning!
(April/May 2013), Vol. 9, Issue 3, 30-34.

Roche, J. & Upton, C. (2013). How Learning is Evolving. Elearning! (April/May 2013), Vol. 9, Issue 3,
22-27.

Rustici (2014). Tin Can Impacts. White paper retrieved 8/18/14 from:
https://tincanapi.com/impacts/

Ryan, T., Toye, M., Charron, K., & Park, G. (2012). Learning Management System Migration: An
Analysis of Stakeholder Perspectives. Retrieved August 17, 2015, from:
http://www.irrodl.org/index.php/irrodl/article/view/1126/2078

Scott, T. and Rung, A. (2016). “Federal Source Code Policy: Achieving Efficiency, Transparency, and
Innovation through Reusable and Open Source Software”. Memorandum for the Heads of Departments
and Agencies. Executive Office of the President, Office of Management and Budget (OMB). August
8, 2016. Retrieved 9/26/16 from https://sourcecode.cio.gov/

Shank, P. (2013). Learning Technologies 2013: Where We Are Now. Elearning Guild Research paper.

Silvers, A. & Torrance, M. (2015). Managing Data-Driven Learning Projects. Presentation at Learning
Solutions 2015 conference.

Choosing an LMS.docx page 208 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Skilljar. (2014). Choosing an LMS for Customer Training. Skilljar publication. Retrieved 10/15/14 at
http://info.skilljar.com/choosing-an-lms-for-customer-training

Smith, A. (2015). U.S. Smartphone Use in 2015. Pew Research Center report retrieved 5/3/16 from
http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

Software Advice (2015). 2015 Learning Management Systems UserView. Reported in Upside Learning
(2016).
Szabo, M. & Flesher, K. (2002). CMI Theory and practice: historical roots of learning management
systems. Paper presented at the World Conference on E-learning in Corporate, Government,
Healthcare, and Higher Education 2002, Montreal, Canada.

Thalheimer, W. (2007). Spaced Learning Over Time: A Research-Based Secret. Slides presented at
ASTD Annual Conference, 2007. Retrieved 4/9/13 from: http://astd2007.astd.org/PDFs/Handouts
%20for%20Web/M105.pdf

Towards Maturity (2014). Mobile Learning in the Workplace. Web article retrieved 8/12/14 from:
http://www.towardsmaturity.org/mobile2014

Udell, C. & Woodill, G. (2015). Mastering Mobile Learning. Hoboken, NJ: Wiley Publishing.

Upside Learning (2016). LMS Selection Guide for SMBs. Web article retrieved 8/4/16 from
https://www.upsidelearning.com/us/ebooks/lms-selection-guide-for-smbs

Upside Learning (2014). LMS Switching Guide. Retrieved 8/12/14 from:

http://www.upsidelearning.com/ebooks/lms-switching-guide

U.S. Army (2015). POM 17-21 Reclama MDEP TSPU Army DL Courseware 15 May 2015.Internal
Army document.

Vipond, S. and Clary, J. (2016). Corporate Learning Management Systems 2016 – 2018. eLearning Guild
and Adobe Systems Research Report retrieved 3/3/16 from http://www.elearningguild.com.

Vipond, S. (2016). Understanding BYOD: A Guide to Concepts and Issues for Learning Practitioners.
ELearning Guild Research white paper. Retrieved 11/30/16 from
https://www.elearningguild.com/content.cfm?selection=doc.4704&utm_campaign=research-
byod16&utm_medium=email&utm_source=elg-membern&utm_content=link

Watson, W. R., & Watson, S. L. (2007). An argument for clarity: what are learning management systems,
what are they not, and what should they become? TechTrends, 51(2), 28-34.

Wiegers, K. (2000). Karl Wiegers Describes 10 Requirements Traps to Avoid. Web article at
http://processimpact.com/articles/reqtraps.html. Originally published in Software Testing & Quality
Engineering, January/February 2000.

Wikipedia (2015). Learning analytics article. Retrieved 6/18/15 from:

http://en.wikipedia.org/wiki/Learning_analytics

Choosing an LMS.docx page 209 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

The authors thank Jason Haag, Dean Marvin, Robby Robson, Don
McIntosh, and Mike Rustici for contributing ideas and materials to this
paper.

Choosing an LMS.docx page 210 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

Appendix

Choosing an LMS.docx page 211 of 237

2015 CC: Attribution-Noncommercial-Share Alike 4.0 International
Choosing a Learning Management System ADL Instructional Design Team

A. Sample System Requirements Matrix

The following is a sample of a matrix that can be used in step 7 presented in 8. Process for
choosing an LMS. The step is described as:

Develop and populate a system requirements matrix that allows assessing the systems identified
in step 6 against your requirements developed in step 3. To use the matrix:

1. Enter items you have determined to be your high-level requirements for the system as row
labels in the “High-level requirements” column.

2. Enter the product names at the top of each column, replacing “LMS product 1”, “LMS
product 2”, etc..

3. Research and complete the cells with information indicating whether each product meets
that requirement (may be “yes” or “no”, a more lengthy description of how it meets or
doesn’t meet the requirement, or a number that roughly quantifies the degree to which that
requirement is supported in the product).

LMS Requirements Matrix

LMS product LMS product LMS product LMS product LMS product LMS product LMS product LMS product
1 2 3 4 5 6 7 8
High-level
Requirements

Choosing an LMS.docx page 212 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

Choosing an LMS.docx page 213 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

B. Sample System Features Rating Matrix

The following is a sample of a matrix that can be used in step 10 presented in 8. Process for
choosing an LMS. The step is described as:

Develop a system features rating matrix…that compares the systems identified in step 8 using
the features list developed in step 9. Complete as much of this matrix as possible from the
systems’ documentation; if you need more information, ask their sales representatives for it
(though beware of overblown claims—verify lofty ones independently if possible). Assign a
numerical rating for each cell in the matrix, indicating degree of implementation of that feature;
“0” would indicate that a particular LMS does not have that feature, and “10” indicates that it has
a very robust implementation of the feature. The matrix should weight each feature according to
its importance to you, enabling a rollup score for each system.

To use the matrix:

1. Replace the top row (LMS product 1, LMS product 2, etc.) with the names of the systems
you have identified for consideration.

2. Replace the row names (Feature 1, Feature 2, etc.) with the names of features you have
identified as requirements.

3. For each Weighting factor cell in the column to the right of the Feature name, enter a
number between 1-3 to weight the relative importance of that feature to your organization
(the higher the number, the more important). 1 weighting is a “must have”, 2 rating is a
“should have”, and 3 weighting is a “nice to have”.

4. Research the feature information for each system and complete the cells with the number
indicating the degree to which each system has that feature. We suggest 0-2, 0 being “does

Choosing an LMS.docx page 214 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

not have that feature” and 2 being “has implemented this feature to the fullest extent
possible”. You may want to use a rubric developed by Brandon-Hall (Brandon-Hall Group,
2010) that rates the feature in terms of how “out of the box” it is. Assigning numbers to
their rubric would yield the following rating scale:

 5=Automatic (built-in, out of the box feature)

 4=Semi-automatic (mostly built-in, but requires some programming or customization

to activate)

 3=Semi-custom (partially available. The system can be adapted to implement this

feature through moderate customization)

 2=Custom (not available but can be added, possibly at high cost, with programming)

 1=Not available (would be impossible or cost-prohibitive to customize the system to

add the feature due to incompatibilities with system architecture, etc.)

If a feature is not available, you may also want to note in this matrix whether a feature is
available from another vendor as an add-on, so as not to totally rule out/penalize the vendor
for lack of that feature. This can be incorporated into the rating scale such that a rating of
“3” means that a feature is available as a third party add-on.

5. The rollup score row at the bottom will provide the total weighted score for each system
(right-click on it and select Update Field after you make any changes to the weighting
values or ratings). Formulas in the cells multiply the weighting factor for each feature by
the degree of implementation feature described above; those scores are then added to
make the totals at the bottom of each row.

Choosing an LMS.docx page 215 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

6. If you add columns or rows, copy and paste the Rollup score formula and adjust the row
and column references in the formula accordingly. Right-click the pasted Rollup score and
select Toggle Field Codes to see and edit the formula.

LMS Features Rating Matrix

Weighting LMS product 1 LMS product 2 LMS product 3 LMS product 4 LMS product 5
Feature factor
name
Feature 1
Feature 2
Feature 3
Feature 4
Feature 5
Feature 6
Feature 7
Feature 8
Feature 9
Feature 10
Rollup 0 0 0 0 0
score

Choosing an LMS.docx page 216 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

C. Security Considerations for DoD LMSs

The following are security considerations and requirements for any LMS that will be used within U.S.
DoD. Many of these considerations apply in a more general sense to any military environment that is
acquiring or installing an LMS.

o Unclassified system (NIPRNET)

o Classified system (SIPRNET)

o Certification requirements

o Customer databases

o Defense Enrollment Eligibility

http://usmilitary.about.com/library/milinfo/dodreg/bldodreg
1341-2i.htm

o Reporting System
http://usmilitary.about.com/library/milinfo/dodreg/bldodreg
1341-2i.htm

o Army Knowledge Online

http://www.army.mil/ako/

o Navy Knowledge Online

https://wwwa.nko.navy.mil/portal/home/

o Navy Training Management Planning System

http://www.public.navy.mil/spawar/PEOEIS/SWP/Document
s/FactSheets/FS_NTMPS.pdf

o Security Certification & Accreditation

http://www.dtic.mil/whs/directives/corres/pdf/330513m.pdf

o Transport Layer Security (TLS)

http://en.wikipedia.org/wiki/Transport_Layer_Security

o FIPS PUB 112 standard, Password usage, National Institute

of Standards and Technology (NIST) (for user identification
and authentication and DoD password management

Choosing an LMS.docx page 217 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

guideline)
http://www.dtic.mil/dtic/tr/fulltext/u2/a406544.pdf

o Public-Key Infrastructure (PKI)

http://www.dartmouth.edu/~deploypki/overview.html

o Support for multiple levels of customizable security access

o Security considerations for private and public cloud

solutions

o Cybersecurity
http://www.dtic.mil/whs/directives/corres/pdf/850001_2014.
pdf

o Security System Authorization Agreement – Required by

DoDI 5200.40 - DoD Information Technology Security
Certification and Accreditation Process (DITSCAP)
http://en.wikipedia.org/wiki/System_Security_Authorization
_Agreement

o BYOD considerations for mobile LMSs

http://www.infolawgroup.com/2012/03/articles/byod/the-
security-privacy-and-legal-implications-of-byod-bring-your-
own-device/

Choosing an LMS.docx page 218 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

D. Sources of Possible Requirements for U.S. DoD LMS

Acquisitions and Installations
o Common Human Resource Information Standards (CHRIS)
http://www.prim.osd.mil/init/listing_chris.html

o DoD 5220-M-SUP - National Industrial Security Program

Operating Manual Feb 2006
http://www.dss.mil/documents/odaa/nispom2006-5220.pdf

o DoD Information Assurance Certification and Accreditation

Process (DIACAP) Nov 2007
http://www.prim.osd.mil/Documents/DIACAP_Slick_Sheet.p
df

o NSTISSI No. 4009 - National Information Systems Security

(INFOSEC) Glossary May 2003
http://handle.dtic.mil/100.2/ADA433929

o OMB A130 Transmittal Number 4 - Management of Federal

Information Resources Various
http://www.whitehouse.gov/omb/circulars/a130/a130trans4.
html

o Public Law 107-347– Federal Information Security Act

https://en.wikipedia.org/wiki/Federal_Information_Security_
Management_Act_of_2002

o Subsection 552a of title 5, United States Code Jan 06, 2003

http://uscode.house.gov/view.xhtml?
req=(title:5%20section:552a%20edition:prelim)

o DODD 8500.1 Cybersecurity

http://www.dtic.mil/whs/directives/corres/pdf/850001_2014.
pdf

o DoD 5200.1-R DoD Information Security Program

Regulation
http://www.dtic.mil/whs/directives/corres/pdf/520001_vol1.p
df

Choosing an LMS.docx page 219 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

o 44 U.S.C. § 3541, United States Code, "Federal Information

Security Management Act of 2002" (FISMA)

o Defense Acquisition Workforce Improvement Act (DAWIA)

(dictates keeping training records and test scores for 7
years, with caveats)
http://www.dau.mil/doddacm/Pages/Certification.aspx

o Federal Risk and Authorization Management Program

(FedRAMP+)
https://www.fedramp.gov/#

o NIST 800-53
http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.
SP.800-53r4.pdf

o USA Learning
https://usalearning.gov/

Choosing an LMS.docx page 220 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

E. Additional requirements for LCMSs

The following requirements are supplemental to the list of
requirements presented in 5. List of possible requirements for an
LMS. If you are acquiring an LCMS, you should consider adding these
requirements to that list. The criteria described here are relevant only
to LCMSs, since LMSs do not normally include the content authoring
and content repository features that are the hallmark of LCMSs.

If you are looking at an LCMS solution, it is important that you also

focus on the requirements for content authoring features, in addition
to the requirements found in this list and the list in 5. List of possible
requirements for an LMS. For a list of requirements related to
authoring capabilities, see ADL’s Choosing Authoring Tools paper at
http://adlnet.gov/adl-assets/uploads/2016/01/ChoosingAuthoringTools
.docx. The list below does not include features that are related to
content authoring, to avoid repetitive overlap with the Choosing
Authoring Tools paper.

A high-quality LCMS, in addition to the criteria presented in 5. List of

possible requirements for an LMS, will include the following features:

o Navigation and administrative views

o Allows filtering of views so that you can view only a

particular level of content in the hierarchy, or branch of the
content tree.

o Clearly shows where an object is being reused.

o Allows filtering of views of content being developed, using

metadata.

o Content import

o Has mapping feature that allows you to indicate how the

styles and items in Microsoft Office documents to be
imported relate to the level of object in the LCMS. For
example, an “H1” heading in a Microsoft Word document
becomes a separate screen with that title.

Choosing an LMS.docx page 221 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

o Allows bulk media import (e.g., collection of media files

within zip file) into content repository.

o Imports containers that store external files, for example, a

web site, with internal links between files maintained after
import.

o Creating ancillary content objects

o Easy to create ancillary course objects like bibliographies,

glossaries, assessments.

o For assessments, has an API that allows setting the values

required to communicate scoring info between a
simulation format like Flash and the LCMS.

o Manipulating content objects

o Allows establishing objects at at least four levels of

content object hierarchy (for example, course, module,
learning object, topic).

o Uses drag and drop as much as possible for moving

objects within output structures.

o Allows and has flexible options for orphan objects that are
not assigned to parent objects.

o Allows you to assign properties to multiple objects at once,

without interfering with already existing settings.

o Developers can lock container objects in a course even if

there are child objects locked by other users.

o Has flexible options for deleting content that is linked/

reused in other containers.

o Has templates that can be applied at all levels of course

structure.

Choosing an LMS.docx page 222 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

o Has robust prohibitions to maintain the integrity of

relationships of objects. For example, it won’t allow you to
delete an object that is referenced by another object.

o Allows overriding personalization settings for individual

objects (that are subject to inheritance rules from parent
objects)

o Has features optimized for system training, such as built-in

screen capture.

o Content preview

o Has viewers that emulate the way the content will look
delivered for web, print, mobile device, 508 accessible
eLearning, etc.

o Allows viewing of content dynamically as it is created, with

different viewers, themes, contexts, etc.

o Repository storage, documentation, and management

o Has robust features for storage, documentation, and

management of content versions.

o Can restore easily to a previous version of an object.

o Can export assets/media from the repository easily.

o Contains a log for all system actions taken in the

repository.

o Allows audit of data movement in and out of the repository

(e.g., “process viewer”).

o Allows partition administration. This is useful where

multiple organizations share the same content repository.

o Allows synchronization of data on different servers, if

multiple servers are to be networked for different data
sharing needs. For example, data on a development server
could be automatically updated on production servers and

Choosing an LMS.docx page 223 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

review servers. This requires functions that identify the

servers used and the content that will be delivered from
one to the other.

o Has a task broker that manages and distributes the load

put on the server by tasks initiated in the repository.

o Exports data from the repository into a packaged XML or

JSON format. This means you can export data for the entire
repository, not just for a particular object.

o Allows caching of media contained in the database,

eliminating the time needed to retrieve it from the
repository when content is launched.

o Supports PENS (Package Exchange Notification Services).

With PENS, you can automatically export content to a
PENS server as SCORM or AICC with notification.

o Content delivery and output formats

o Can set up rules for whether an object displays to the

learner or not (possibly by leveraging SCORM 2004
sequencing capability).

o Has a slideshow feature for delivery of a series of raw

images.

o Is interoperable with PDFs such that a particular page can

be opened vs always opening to the first page.

o Exports content to a variety of content formats, such as

Word, PowerPoint, Framemaker.

o Incorporates viewer objects that allow dynamic

configuration of the way a learner will see delivered
content.

o Has an encrypted export option.

o Allows export as compiled Help (.chm format) that has TOC

and index.

Choosing an LMS.docx page 224 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

F. Examples of products
NOTE: these lists are for illustrative purposes and do not constitute an endorsement by ADL.
This list is not represented as a comprehensive list of all available systems.

General purpose LMSs

 ABC Academy®
www.danishprobe.com

 Absorb LMS®
http://www.absorblms.com/

 Adobe Captivate Prime®

http://www.adobe.com/products/captivateprime.html

 BizLibrary LMS®
www.bizlibrary.com

 Bridge®
http://www.getbridge.com/

 Canvas® [oriented towards use by higher ed community]

http://www.instructure.com/

 Chamilo [open source]

http://www.chamilo.org/en

 CLIX®
www.im-c.com

 ComplianceWire®
www.kaplaneduneering.com

 Cornerstone OnDemand Talent Management Suite®

http://www.cornerstoneondemand.com

 CourseAvenue Deliver®
http://www.courseavenue.com/

 CrossKnowledge Learning Suite®

http://www.crossknowledge.com/en_GB/elearning/technologies/l
ms-platform.html

Choosing an LMS.docx page 225 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 Digital Chalk®
http://www.digitalchalk.com

 Docebo [open source]

http://www.docebo.org/doceboCms

 Dokeos [open source]

http://www.dokeos.com

 DuPont Sustainable Solutions LMS®

http://www.training.dupont.com/dupont-elearning-suite/learning-
management-system

 Edubrite®
http://www.edubrite.com/oltpublish/site/cms.do

 EthosCE®
http://www.ethosce.com

 Exceed LMS®
http://www.intellum.com/exceed-lms/

 Brightspace Suite® [heavy emphasis on personalization and

analytics]
http://www.d2l.com/products/

 learningCentral®
http://www.netexlearning.com/en/learningcentral/

 LearnUpon®
www.learnupon.com

 eFront [open source]

http://www.efrontlearning.net

 Flex®
http://mobileagility.com/products/flex/

 Google CloudCourse®
http://google-opensource.blogspot.com/2010/05/cloudcourse-
enterprise-application-in.html

Choosing an LMS.docx page 226 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 GreenLight Learning Management System®

http://www.silkroad.com

 Grovo®
http://www.grovo.com/platform

 iCohere Unified Learning®

http://icohere.com/

 iLearningPLUS® [built on Oracle iLearning platform]

http://www.seertechsolutions.com

 ILIAS [open source]

http://www.ilias.de

 Infor Learning Management [formerly CERTPOINT VLS]

http://www.infor.com/product-summary/hcm/learning-
management/

 Instancy Learning Gateway®

http://www.instancy.com/

 JoomlaLMS® [open source]

http://www.JoomlaLMS.com

 KnowledgeHub®
http://www.elementk.com

 KeneXa Learning Suite (formerly OutStart Training Edge) ®

http://www.outstart.com/trainingedge-lms.htm

 KMx Enterprise®
http://www.kmsi.us/kmx_product_information.htm

 Krawler LMS®
http://www.krawlerlms.com

 Learning Studio®
http://www.pearsonlearningsolutions.com/pearson-learning-
studio/

Choosing an LMS.docx page 227 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 Litmos®
http://www.litmos.com/

 Meridian Global LMS®

http://www.meridianksi.com

 Moodle [open source] [oriented towards use by academic

community]
http://moodle.com/

 Mzinga Social Learning Suite®

http://www.mzinga.com

 Net Dimensions Learning®

http://www.netdimensions.com/talent-suite/learning.php

 NTER (National Training and Education Resource) [open source]

http://www.nterlearning.org

 OpenClass® [free, hosted solution]

http://www.pearsonlearningsolutions.com/pearson-learning-
studio/

 Openelms®
http://www.openelms.org/

 Oracle Learning Management®

http://www.oracle.com

 PeopleSoft Enterprise Learning Management (ELM) ®

http://www.oracle.com/applications/

 Saba Enterprise Suite®

http://www.saba.com

 SAP Enterprise Learning®

http://www.sap.com

 SCORM Cloud®
http://www.scorm.com/scorm-solved/scorm-cloud/scorm-test-
track-scorm-cloud/

Choosing an LMS.docx page 228 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 ShareKnowledge LMS® (built on Microsoft SharePoint)

http://shareknowledge.com/sk?
gclid=CKXxgaLN1MECFUNp7Aod4A4Arg

 Skilljar LMS®
http://www.skilljar.com/

 Skillport LMS®
http://www.skillsoft.com/business-solutions/skillport.asp

 SuccessFactors Learning (LMS)

http://www.successfactors.com/en_us.html

 SumTotal® TotalLMS®
http://www.sumtotalsystems.com

 TEDS Learning on Demand®

http://www.teds.com/index.php/solutions/learning-management

 TalentLMS®
http://learn.unbundled.org/

 Taleo Learn®
http://www.taleo.com/solutions/learning

 TM SIGAL®
http://www.technomedia.com

 Topyx®
http://interactyx.com/

 Totara LMS® [customized version of Moodle]

http://www.totaralms.com/

 Training Jungle®
http://www.redtray.co.uk

 Training Partner®
http://www.trainingpartner.com/LMS.aspx

 Travitor®
http://travitor.com

Choosing an LMS.docx page 229 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 Trellis®
http://home.learning.net/

 UdutuTeach/Learn®
http://udutu.com/products-udututeach-and-udutulearn.html

 Veloce®
http://www.syslps.com

 ViewCentral®
http://www.viewcentral.com

 UpsideLMS®
http://www.upsidelearning.com

 Virtual Training Assistant®

http://risc-inc.com/

 WiseTail Learning Ecosystem®

http://www.wisetail.com/solutions/

 WordPress LMS®
http://www.learndash.com/

 Xerox Learning Services®

http://www.acs-inc.com/learning-services/learning-outsourcing/l
earning-administration.aspx

 XStream RapidShare LMS®

http://www.xstreamsoftware.com

Mobile Learning LMSs

 Bridge®
http://www.getbridge.com

 Blackboard Mobile®
http://www.blackboard.com/Platforms/Mobile/Overview.aspx

 Certpoint VLS Mobile®

http://www.certpointsystems.com/products-and-services/enterpr
ise-learning-platform/mobile-learning.html

Choosing an LMS.docx page 230 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 CourseAvenue Enterprise Mobile Solution

http://www.courseavenue.com/

 Flex®
http://mobileagility.com/products/flex/

 eXact Learning Mobile®

http://www.exact-learning.com/

 Inkling
https://www.inkling.com/platform/

 Intuition Rubicon®
http://www.intuition.com/Mobile/home.aspx

 Instancy Mobile LMS®

http://www.instancy.com/mobilelearning.aspx

 KMxMobile®
http://www.kmsi.us/white_paper13.htm

 KO-SU®
https://ko-su.com/

 Litmos LMS®
http://www.litmos.com/mobile-learning

 Mobile Coach®
http://mobilecoach.com [delivers learning content via text
messages]

 Moodle Mobile®
http://docs.moodle.org/en/Mobile_Moodle_FAQ

 NetDimensions Talent Suite Mobile®

http://www.netdimensions.com/solutions/mobile-learning.php

 OnPoint Digital CellCast®

http://www.mlearning.com/

 Train by Cell®
http://trainbycell.com/

Choosing an LMS.docx page 231 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 Trivantis Coursemill®
http://www.trivantis.com/coursemill-learning-mangement-
system-features

 Upside2Go®
http://www.upsidelearning.com/us/mobile-learning-solution-
upside2go.asp

 Xyleme Mobile Learning Solution®

http://www.xyleme.com/solution/mobile-learning

Specialized LMSs

 Adobe Experience Manager [social learning optimized]

http://www.adobe.com/marketing-cloud/enterprise-content-
management.html?promoid=KFBZO

 ALTO LMS® [social learning optimized]

http://www.commelius.com/lms/

 Booster Learn®[spaced learning delivery platform]

https://boosterlearn.com/science-of-booster

 Brightcove® [video content management system – see 7.30

Microlearning]
https://www.brightcove.com/en/

 Desire2Learn® [oriented towards use by academic community]

http://www.desire2learn.com/

 Its Learning® [oriented towards K-12)

http://www.itslearning.net

 Kaltura® [video content management system – see 7.30

Microlearning]
http://corp.kaltura.com/

 Konnect® [social learning platform]

http://originlearning.com/technology/konnect/

Choosing an LMS.docx page 232 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 KZO Innovations® [video content management system – see 7.30

Microlearning]
http://www.kzoinnovations.com

 Mlevel® [game-based, “casual” learning platform]

http://www.mlevel.com/

 Mindmarker® [spaced learning delivery platform]

http://www.mindmarker.com

 NovoEd® [optimized for social learning]

https://novoed.com

 Poll Everywhere® [platform for doing live classroom polls]

https://www.polleverywhere.com/

 Skillaware® [performance support and learning analytics

platform – integrates BPMN, xAPI, and DITA standards]
http://www.skillaware.com

 Skytap® [virtual training lab platform]

http://www.skytap.com

 Spoke® [optimized for social learning]

http://www.unboxedtechnology.com/products/spoke/

 TREK Learning Experience Manager® [optimized for managing

coaching and “on the job” learning approaches – integrates
xAPI standard for tracking]
http://www.cognitiveadvisors.com/trek

 Valamis® [optimized for phenomenon-based learning and

learning by swarming]
http://valamis.arcusys.com/

 Viddler® [platform for delivering interactive video]

http://www.viddler.com

 Walkme® [performance support delivery platform]

http://www.walkme.com

Choosing an LMS.docx page 233 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 Nimble® [content distribution platform]

http://www.xanedu.com/business

LCMSs

 ATutor [open source]

http://www.atutor.ca/atutor/index.php

 Claro®
http://www.dominknow.com/products/lcms.cfm

 Cornerstone OnDemand®
http://www.cornerstoneondemand.com/

 Docebo [open source]

https://www.docebo.com/

 eXact LCMS®
http://www.exact-learning.com/en/products/learn-exact-suite/exa
ct-lcms-learning-content-management-system

 GreenLight Learning Content Management System ®

http://www.silkroad.com

 IBM Learning Content Management System®

http://www.ibm.com/marketplace/cloud/learning-content-
management-system/us/en-us

 Kenexa LCMS®
http://www.outstart.com/outstart_lcms.htm

 Knowledge Guru [heavy emphasis on gamification]

http://www.theknowledgeguru.com/

 Mediasite Enterprise Video Platform®

http://www.sonicfoundry.com

 Saba Content Management®

http://www.saba.com

 SAP Enterprise Learning®

http://www.sap.com

Choosing an LMS.docx page 234 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 SumTotal LCMS®
http://www.sumtotalsystems.com/products/learning-content-
management-system.html

 TrainingRelief®
http://trainingrelief.com

 Xyleme LCMS®
http://www.xyleme.com

CrMSs

 .LRN [open source]

http://dotlrn.org/

 Adrenna Academic [open source]

http://www.adrenna.com/open-source-lms#adrenna_academic

 Blackboard® [oriented towards use by academic community]

http://www.blackboard.com

 Canvas Network [oriented towards use by higher ed community]

http://www.instructure.com/

 Claroline [open source]

https://github.com/claroline/Claroline

 CourseCompass®
http://www.pearsonlearningsolutions.com/career-schools/lms-
cms.php

 Coursera® [used exclusively for MOOCs]

https://www.coursera.org/

 Edvance360®
https://www.edvance360.com/

 EdX [open source, used exclusively for MOOCs; Google is

contributing code. See
http://googleresearch.blogspot.com/2013/09/we-are-joining-
open-edx-platform.html]
https://www.edx.org/

Choosing an LMS.docx page 235 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 GoToTraining®
http://www.citrix.com/English/ps2/products/product.asp?
contentID=1862273&ntref=prod_top

 Haiku®
http://www.haikulearning.com/

 Learning Studio®
http://www.pearsonlearningsolutions.com/pearson-learning-
studio/

 LON-CAPA [open source]

http://www.lon-capa.org/

 Moodle [open source]

http://moodle.com/

 OLAT [open source]

http://www.olat.org/website/en/html/index.html

 Sakai [open source]

http://sakaiproject.org/portal

 WebStudy®
http://www.webstudy.com

VLEs

 Adobe Connect®
http://www.adobe.com/products/adobeconnect.html

 Collaborate®
http://www.blackboard.com/Platforms/Collaborate/Products/Blac
kboard-Collaborate/Web-Conferencing.aspx

 Centra®
http://www.saba.com

 Connect®
http://www.adobe.com/products/acrobatconnectpro/

Choosing an LMS.docx page 236 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0
Choosing a Learning Management System ADL Instructional Design Team

 iCohere Unified Learning®

http://icohere.com

 LiveRoom®
http://www.desire2learn.com

 Social Learning Suite®

http://mzinga.com/

 WebEx Training Center®

http://www.webex.com

 WizIQ® [hosted service that allows the public to run their own
classes]
https://www.wiziq.com/

Choosing an LMS.docx page 237 of 237

2010 CC: Attribution-Noncommercial-Share Alike 3.0