Benchmarking of

Mobile Data
Collection Solutions
W HAT ASPECTS TO CONSIDER WHEN CHOOSING A TOOL /P LATFORM

Solutions tested: December/January 2017

Report compiled: May-June 2017

This publication has been produced with the assistance of the Office of the United
Nations High Commissioner for Refugees (UNHCR) and Terre des hommes (Tdh). The
contents of this publication are the sole responsibility of CartONG and can in no way be
taken to reflect the views of UNHCR or Tdh.
 Benchmarking of Mobile Data Collection Solutions

Table of Contents
Table of Contents ..................................................................................................... 1
Acronyms ................................................................................................................. 3
Executive Summary .................................................................................................. 4
Part I. Where the sector stands on Mobile Data Collection ................................... 5
A. What are the main advantages and challenges of Mobile Data Collection? ...... 5
Advantages .......................................................................................................... 5
Challenges............................................................................................................ 7
B. What are the most differentiating factors of the day concerning MDC? ........... 9
Analysis possibilities of MDC platforms ..................................................................... 9
Monitoring of a situation over time .......................................................................... 9
ODK-Based Technologies ........................................................................................ 9
Summary graph .................................................................................................. 10
C. Is having “one MDC tool for your whole organization” a viable option? ........ 11
Advantages of Using a Single Platform ................................................................... 11
How to Choose a Preferred Solution? ..................................................................... 12
D. What are the aspects to keep in mind when you budget your MDC? .............. 13
Budgeting for Mobile Data Collection in general ....................................................... 13
Part II. Product Evaluation .................................................................................. 16
1. Common Aspects and Features of All MDC Platforms .................................... 16
2. Platform Reviews .......................................................................................... 18
A. ODK- based solutions .................................................................................... 19
KoboToolBox ....................................................................................................... 19
ODK Aggregate ................................................................................................... 21
ONA................................................................................................................... 23
SurveyCTO ......................................................................................................... 25
B. Other Xform-based solutions......................................................................... 27
CommCare ......................................................................................................... 27
DeviceMagic ....................................................................................................... 29
Magpi ................................................................................................................ 31
Survey123 for ArcGIS .......................................................................................... 33
C. Other solutions .............................................................................................. 35
Akvo Flow........................................................................................................... 35
Dharma Platform ................................................................................................. 37
Fulcrum ............................................................................................................. 39
IformBuilder ....................................................................................................... 41
Mobenzi Researcher ............................................................................................. 43
Poimapper .......................................................................................................... 45
Traxilo ............................................................................................................... 47
info@cartong.org | http://www.cartong.org Page 1 | 66
 Benchmarking of Mobile Data Collection Solutions

D. Table 1 – ODK and Xform based solutions ..................................................... 49

E. Table 2 – Other solutions .............................................................................. 54
Annexes ................................................................................................................. 57
1. List of All Solutions ....................................................................................... 57
2. MDC Technical Requirements Checklist ......................................................... 58
3. What are The Advantages to Using ODK-Based Technologies for NGO/IOs? . 63
4. Resources...................................................................................................... 65

info@cartong.org | http://www.cartong.org Page 2 | 66

 Benchmarking of Mobile Data Collection Solutions

Acronyms
API Application Programming Interface (allows communication between different programs
and interaction with data stored in the cloud)

CAPI Computer Assisted Personal Interview

CSV Comma Separated Values, spreadsheet format

DDC Digital Data Collection

GIS Geographic Information Systems

HIPAA Health Insurance Portability and Accountability Act; prescribing standards for data
privacy and security.

JSON Javascript Object Notation, open standard format for sharing data over the internet
KML Keyhole Markup Language, file format for sharing geographic information

MDC Mobile Data Collection

NGO Non-Governmental Organization

PAPI Paper Assisted Personal Interview (in contrast to CAPI)

XLS Microsoft Excel spreadsheet format (newer version XLSX)

XML Extensible Markup Language, open standard format for encoding documents in a
human and computer readable format

Icons used in this report were purchased from Noun Project.

info@cartong.org | http://www.cartong.org Page 3 | 66

 Benchmarking of Mobile Data Collection Solutions

Most mobile data collection solutions evolve fast, new versions are often released
several times a year. This report is based on tests conducted in January 2017.
Readers are strongly advised to verify features and functions of newer releases
when interested in a particular tool/platform. This report only serves to give a
general overview and a comparison based on a snapshot in time.

Executive Summary
Humanitarian response operations as well as development projects require accurate
information to use their resources in the best possible manner, be it to determine the urgent
needs of communities affected by natural disasters or conflict, or to ensure that people and
households receive ongoing support to improve their situation.

In the best case, making decisions based on no, limited, outdated or incorrect information
means that time and money is lost. In the worst case it can mean that people’s lives are
irrevocably harmed.

Mobile Data Collection (MDC) can help improve the quality of data, information, analysis and
decision making. By using one of the MDC platforms described in this report, organizations can
collect data faster and with fewer errors than on paper. The sharp decline in hardware costs
for mobile phones also means that MDC is often cheaper than doing a survey on paper.

As the report shows, the question is no longer if organizations should use MDC, but how. The
short profiles for each solution, as well as the detailed table at the end of the report, show the
main strengths and weaknesses of the different platforms.

The most significant differences between the tools tend to fall into three different categories:

1) How easy is it to import or export data and forms to and from other applications?
2) Does the platform support individual case management and/or monitoring situations
over time?
3) What level of data analysis is supported out of the box?

There is no single solution that can fit all possible needs and the report advises against trying
to force a single solution on staff. Instead, the report recommends that organizations either
agree on a small pool of approved platforms from which staff can choose, or on a preferred
solution, from which teams can diverge if necessary. This approach ensures a blend between
uniformity - which is important to maximize familiarity, compatibility and support for the
platform - and flexibility, which enables teams to respond quickly to operational needs and
changes in the MDC marketplace.

info@cartong.org | http://www.cartong.org Page 4 | 66

 Benchmarking of Mobile Data Collection Solutions

Part I. Where the sector stands on Mobile Data Collection

According to a survey conducted during the NOMAD 1 MDC event in Amman 2016, only 25% of
the participants had never used MDC before, whilst during a similar NOMAD event in Paris in
2013, 58% of the participants claimed they had not used MDC before.

The rapid proliferation of smartphones, as well as the massive decline of their price, has turned
smartphones from a luxury item to a common, multi-use tool that hardly raises an eyebrow
anywhere in the world. Where previously, the high value or the high status associated with
owning a smartphone or tablet exposed NGOs and their staff to potential security risks, these
concerns have abated to a certain extent 2 and many NGOs are providing their staff with mobile
devices.

Programmatic areas that have benefited greatly from the proliferation of smartphones are
those that are related to monitoring or data capture - be it through spontaneous photos of
project assets (for example when something has broken) or through highly structured
assessment tools that run on digital devices. This report looks at the latter and provides the
reader with an overview of the tools that are state of the art in late 2016/early 2017.

Given that the main audience of this report are non-profit organizations working in developing
countries, the report limits itself to mobile data collection (MDC) applications and services that
run on low-cost Android devices, even when no data or cellular network is available.
Applications and services that require iOS, a stand-alone notebook computer or a stable
internet connection are not part of this overview, since these are less common in the field.

A. What are the main advantages and challenges of Mobile Data

Collection?
Advantages

Mobile data collection has many advantages over paper-based alternatives:

Fewer errors: “Garbage In - Garbage Out” is the mantra of many evaluators. MDC is
able to reduce the amount of “garbage” significantly by eliminating or reducing two
potential sources of errors:

• Inconsistent/impossible/missing data: All

products and services tested for this report include
internal checks that highlight impossible or inconsistent
data during the data capture phase in the field, so that
it is not possible to enter 114-year-old children or
pregnant men. In many cases this type of automatic
plausibility check also includes errors of summation, for
example during household survey when expenditure for
individual household items don’t add up to the total
amount. Since the software can alert the enumerator to issues on the spot, the data

1
NOMAD (HumanitariaN Operations Mobile Acquisition of Data) is a project that is co-lead by
iMMAP and CartONG with the goal to promote the use of mobile data collection by humanitarian professionals in order
to improve their efficiency and impact. It both helps organizations by offering an online selection tool to help them
select which tools are relevant to their needs (https://humanitarian-nomad.org/online-selection-tool). NOMAD also
organizes yearly workshops where interested parties can meet other MDC users and service providers.
2 Exceptions to this are some very specific contexts where state or non-state security forces do not accept the

presence of smartphones, for example Al-Shabaab controlled areas of Somalia or in conflict situations like Syria
where it might put a person at risk if he/she can be localized through a smartphone.

info@cartong.org | http://www.cartong.org Page 5 | 66

 Benchmarking of Mobile Data Collection Solutions

can be corrected immediately. The same goes for fields that on a paper survey might
be accidentally skipped or omitted. This is particularly relevant since research 3 has
shown that errors in the data capture phase are not random, but biased towards
households with distinct characteristics, which might skew the whole data set.

• Re-keying errors: At some point, data from all paper based surveys has to be entered
into a computer. During this process errors invariably happen. Because data collected
digitally does not need to be re-entered, this source of errors is effectively eliminated.

Faster data collection: Data collection via mobile devices tends to be faster than on
paper, partially because of built-in functions that can automatically skip questions
based on previous answers. For example: if a household does not have any children,
questions related to the children can be skipped automatically. The time savings increase with
complexity and length of the survey. For example, Fitzgerald et al. found that by using MDC
with skip-logic, they were able to save close to one hour per household during an in-depth
household survey in Ethiopia and Malawi that, on paper, ran to 50 pages. 4 Given that most
households were subsistence farmers, the researchers also found that respondents were more
likely to answer all questions when the survey took less time.

Faster analysis: Because the data doesn’t have to be manually entered, it is also
much faster to run simple analyses on the data, even while the survey is still underway.
All applications and services tested for this report include at least a basic tool to
visualise data out of the box. In addition to providing NGO staff with answers more quickly,
this can also be an important feedback tool for communities that have been surveyed.

Better quality control: Many MDC applications are able to capture the GPS
coordinates where an interview takes place, as well as the time the interviewer took
to complete the interview. The GPS coordinates allow supervisors to ensure that staff
have visited the right location and facilitates repeated visits which might be necessary for
monitoring. The duration can help to identify enumerators who are either extremely fast or
extremely slow, either of which might be an indicator of quality issues. 5

Costs: Costs are frequently listed as one of the areas where paper-based data
collection has an advantage over digital data collection. However, this depends heavily
on the individual case especially on the number of surveys conducted and the number
of submissions and length of each survey. While MDC have higher initial costs for software,
development and hardware and capacity building, they do have cost benefits in other areas.
Changes to digital surveys, for example, can be rolled out easily to all enumerators, while paper
survey forms might have to destroyed and reprinted. Also, the costs for subsequent data entry
are completely removed with MDC. In a 2015 study in Thailand and the Philippines, Oxfam
found that MDC was cheaper than paper unless new mobile devices had to be bought for the

3
Improving Consumption Measurement and other Survey Data through CAPI: Evidence from a
Randomized Experiment, Caeyers et al., Journal of Development Economics DOI: 10.1016/j.jdeveco.2011.12.001,
December 2011
4
A Comparative Analysis of Traditional and Digital Data Collection Methods in Social Research in LDCs -
Case Studies Exploring Implications for Participation, Empowerment, and (mis)Understandings,
http://www.validnutrition.org/wp-content/uploads/2015/03/A-Comparative-Analysis-of-Traditional-and-Digital-Data-
Collection-Methods.pdf Gretta Fitzgerald and Mike Fitz Gibbon, Preprints of the 19th World Congress The
International Federation of Automatic Control Cape Town, South Africa, 24 - 29 August 2014, Retrieved: 6 December
2016
5
See also. Electronic Versus Paper-Based Data collection: Reviewing the Debate
http://blogs.worldbank.org/impactevaluations/electronic-versus-paper-based-data-collection-reviewing-debate,
Sacha Dray, Felipe Dunsch, and Marcus Holmlund, Wold Bank, 25 May 2016, Retrieved: 6 December 2016

info@cartong.org | http://www.cartong.org Page 6 | 66

 Benchmarking of Mobile Data Collection Solutions

survey and were not used for any other survey afterwards. 6 The cost benefits of MDC grow
even further when the same survey is run multiple times, for example for projects where
changes over time are being tracked with the same indicators.

Multimedia: Smartphones are much more than a touchscreen to enter data. Organizations
can use a variety of different tools and features to enrich the data by collecting GPS points
without a stand-along GPS receiver, taking photos without bringing along a separate camera,
scan barcodes, record audio and video etc. Most of the time this additional data is automatically
integrated into the survey without requiring any extra efforts or manual work.

In short: through mobile data collection, NGOs are able to get more accurate
information faster and at a lower cost than with paper.

Free Online Course: Introduction to Mobile Data Solutions

https://course.tc/catalog/course/c06a1489-51e4-43bb-9b50-27fa4446327f
TechChange offers a free, self-paced online course that provides a basic introduction to
mobile data solutions. The course was developed with assistance from USAID and FHI360.

Challenges

Survey design: As described above, mobile surveys can prevent enumerators from
entering impossible data or omitting questions. 7 However, other risks are only
encountered in MDC. Sometimes the excitement about a new technology leads to an
increased focus on the technical aspects of a survey, at the expense of designing the survey
itself conceptually. Often, such a shift in focus means that creating a complex form logic is
perceived as the key to a good survey while other important elements, such as defining the
goals of the survey or questioning the ethics of questions are neglected.

Survey coding: Unlike paper surveys, MDC surveys require that someone implements the
desired skip logics and other restrictions on data entry, usually through some visual form
builder or template that will provide the smartphone application with the instructions as to how
the survey should behave. There is a learning curve associated with the acquisition of these
skills and some level of competence is required to be able to reap the full MDC potential with
regards to data quality.

Hardware failure: Applications can bug, mobile devices can break, run out of
electricity and their batteries are particularly sensitive to high or low temperatures.
Replacement devices, paper forms as backup, car chargers and battery packs can
mitigate these issues, but in many cases, a severe hardware failure will mean that an
enumerator cannot continue her/his work until s/he has returned to the office. In some cases,
the data stored on the device might be lost as well.

6
Going Digital - Using digital technology to conduct Oxfam’s Effectiveness Reviews,
http://oxfamilibrary.openrepository.com/oxfam/bitstream/10546/578816/4/cs-going-digital-effectiveness-reviews-
290915-en.pdf, Emily Tomkys and Simone Lombardini, Oxfam GB, September 2015, Retrieved: 6 December 2016
7
For more details see: http://blog.cartong.org/2015/10/15/conceiving-survey-1/ and
http://blog.cartong.org/2015/11/10/conceiving-survey-2/

info@cartong.org | http://www.cartong.org Page 7 | 66

 Benchmarking of Mobile Data Collection Solutions

Lack of connectivity: Most MDC solutions require an active online connection to

synchronise data, which can add logistical issues to the deployment of MDC. Given that
this report is primarily intended for NGOs working in developing countries, only
products have been included that allow data collection in offline mode.

Lack of compatibility: All MDC solutions in this report can export data at least as
comma-separated-values (csv) or Excel files (xls), but many provide additional export
options that are better suited for further analysis and visualisation in tools such as
SPSS, Tableau or other Business Intelligence tools. Most solutions even offer access to an API
that once set up allow creating visualisations (such as online dashboards or web maps) showing
the collected data in real-time. Nevertheless, when exporting results or questionnaires or
accessing the data through an API, the structure and format of the outputs will often vary
between competing providers. This can make it difficult and time-consuming to collate data
collected with different MDC solutions and it can make it impossible to switch platforms during
an ongoing survey. The reviews in this report make note of compatibility options and issues
where relevant.

Familiarity with the technology: While smartphone literacy is increasing steadily,

this is not the case across all demographic groups and geographic zones. Smartphone
literacy can be an issue especially in surveys where, for example, enumerators should
be older because the survey is aimed at the elderly.

Languages: Many MDC solutions provide the user interface for their server and
analysis module only in very few languages - sometimes just in English. But, apart
from two exceptions, all tools included in this report can create surveys in any major
alphabet including Arabic and Hindi. However, not all solutions allow enumerators to switch
between languages within the same survey which, is necessary for examples when you work
with different ethnic groups who speak different languages.

Security and privacy: Surveys often collect personal information. Based on the right
to privacy, recognised in most international human rights treaties, such data is
protected. 8 It is the responsibility of the organization collecting the data to ensure that
the collection, storage, analysis and publication of data conforms to security and privacy
standards and do not pose threat to the individual or his rights. Depending on the type of
survey, different levels of security can be acceptable. For example, data on the health of a
patient requires a very high level of security and care should be taken that any public
visualisation of health data can never be traced back to an individual. A key informant interview
on general needs in a camp, on the other hand, might require much less protection. While only
a few solutions encrypt data when stored on server or phone, all solutions benchmarked here
can use HTTPS for data transfer between phone and server. The reviews in Part II will mention
if solutions adhere to recognised security standards. User authentication processes and security
were not benchmarked for this document. This could, however, be an additional requirement
if a very secure environment is needed.

8
“Professional Standards for Protection Work” (2nd Edition, 2013, ICRC):
https://www.icrc.org/eng/assets/files/other/icrc-002-0999.pdf

info@cartong.org | http://www.cartong.org Page 8 | 66

 Benchmarking of Mobile Data Collection Solutions

B. What are the most differentiating factors of the day concerning MDC?
MDC solutions evolve over time, and the differentiating factors between solutions also evolve
based on the requirements of the user community and the technical constraints of the moment.
Here are three of the most important of these factors today:

Analysis possibilities of MDC platforms

An important aspect where tools differ is their ability to make the analysis of the data on the
server easy for the user. Here we can distinguish between solutions providing support
throughout the entire surveying process including the data analysis and those that focus more
on the collection and simply provide a range of export formats to allow an easy integration with
external analysis tools. More sophisticated platforms will allow you to filter your data directly
on the server and represent the filtered information in graphs or maps, have data quality
checks embedded and even export reports directly from the platform.

Monitoring of a situation over time

A second aspect where we can cluster the solutions into two separate groups is their ability to
monitor a situation over time. This option requires that the user can either edit existing records
and a history of the different submissions is kept on the server or the user can submit several
submissions all relating to a “parent” entry. For example the enumerators could first collect
data on water points or patients with some unchangeable attributes (such as a unique identifier,
the water point’s location or a patient’s date of birth) and then submit variable data associated
with each entry (such as a flow rate at a water point measured regularly or the blood pressure
of a patient). The platforms making this possible are few and are usually those that are at the
higher end in terms of costs seeing the added value that this component brings.

ODK-Based Technologies

A third aspect that is very frequently used to differentiate MDC solutions is whether it belongs
to the ODK-based technologies.

The OpenDataKit is one of the projects that uses a common language-XML 9- and a common
standard- Xforms 10- for data collection. They developed an open-source suite of tools and apps
to collect data. Being one of the first to use Xforms, their contribution to defining and using
the Xform standards was key to the subsequent developments. Many other tools have emerged
that either built on top of ODK directly or that used the Xforms standards as laid out by the
ODK team (with only minor adaptations or changes) for developing their own solutions. The
advantage of these developments for the user is the interoperability between the tools
especially for those being built directly on top of ODK. It ensures that data and surveys can be
shared between tools and platforms - the format of the data is, in fact, independent of the
platform, a key idea in the age of file sharing and multi-machine networks. Being based on
standards specifically developed for data collection both ODK and Xform based solutions
support natively difficult logical operations (skip patterns, cascading selects,….) or question
types (calculations, dates,...). In the remainder of this document the authors will refer to:

- ODK-based solutions - for solutions that are built on ODK and use the Xform standards

9
XML is an eXtensible Markup Language - it defines how to structure a file in order to make it easily readable to
humans and machines alike using tags. XML is used as a framework/a set of rules based on which more explicit
languages where created further defining the tags such include HTML or KML.
10
The Xform standards further define and detail how XML can be used for data collections: this includes key tags
representing questions types and logical operations such as skip patterns.

info@cartong.org | http://www.cartong.org Page 9 | 66

 Benchmarking of Mobile Data Collection Solutions

- Xform-based solutions - for solutions which use Xforms but are not necessarily built on
top of ODK
- other toos - tools that do not comply with the open Xform standards. Those may use
un-standardized XML or a proprietary format for their data
As the ODK based solutions are in many ways similar to each other and provide often a certain
number of similar functionalities, we will regroup them in our comparison of solutions to make
it easier to compare them.
If you want to know more about what advantages there can be for NGOS to use ODK-based
technologies, you can refer to Annex 3: What Are the Advantages in Using ODK-based
technologies for NGOs/IOs?

Summary graph

Here is an infographic regrouping all solutions looked into through the prism of these
differentiating factors:

info@cartong.org | http://www.cartong.org Page 10 | 66

 Benchmarking of Mobile Data Collection Solutions

C. Is having “one MDC tool for your whole organization” a viable option?
Selecting a single MDC platform to be the solution for the entire organization is an aspiration
for many International Organizations and NGOs. The idea is to select and promote, whenever
possible, one specific solution to be used across operations. Advocates of this approach
emphasize the better integration and better support that it can entail. The opponents criticise
that it is often a bad compromise that never fully fits the needs of all cases.

This chapter looks more closely at the advantages and inconveniences of a one-platform
approach for Mobile Data Collection and highlights key questions that need to be answered
before making such a decision.

Advantages of Using a Single Platform

Four arguments are key to understanding why a one-platform approach can be beneficial for
an organization:

Better integration: If data needs to be integrated into existing workflows, synchronised with
another organizational database, and especially if IT support is required for these tasks, a
single solution for the whole organization can help to ensure that data and workflows are stable
and that IT can adjust processes as needed.

Better knowledge: This is an advantage for both IT and users: enumerators, form builders,
project managers and IT staff know what to expect from a solution, where to find support and
how to best use the tool.

Compliance with organizational standards: Key departments including IT can verify that
a tool meets requirements which have been established for the entire organization. This is
specifically relevant for security standards. While an operational department might not have
the capacity to evaluate the security features of various products, for an organizational tool
those key features (should) have already been evaluated and departments can trust that the
solution meets the requirements.

Bargaining power: Having an entire organization (especially one with several projects in the
pipeline) can improve the bargaining position when discussing the licensing and pricing with a
provider. It can also impact the priority of feature requests as the provider knows that there is
a concrete need for improvement in the requested area.

Disadvantages of the One-Platform Approach

Despite these advantages, imposing a single platform as the only option to all departments is
not always the best approach:

Varying needs: Different MDC tools have different features and no tool can meet all needs.
Often that means making compromises and whether those compromises are acceptable needs
to be decided during the project’s inception phase.

Staff frustration: The two main reasons for staff frustration are: 1) being forced to use a tool
that does not meet the exact needs and 2) having to switch tools, which means investing time
in training and data migration (not always compatible with skills, budget or project deadlines).

Keeping pace with new developments: The MDC sector is a fast-paced environment where
different solutions appear and disappear from one year to the next and where providers release
new versions with additional features several times a year. Different solutions improve or
decline each year. This means that there is no guarantee that a solution which is best suited

info@cartong.org | http://www.cartong.org Page 11 | 66

 Benchmarking of Mobile Data Collection Solutions

for a task this year will still be the best, or still be supported, next year. This makes it hard to
make a single tool part of the organisation’s MDC strategy over time.

Organizational inertia: An organization that once decided on a one-platform approach will

find it more difficult to switch to another tool. It becomes more difficult to justify changing
workflows and requesting staff to learn another tool once a solution is put in place, even when
a new tool might have become better suited for the majority of tasks or when organizational
needs might justify a transition.

From a Single Platform to a Preferred Solution

Whether or not a one-platform approach is beneficial will vary from one organization to the
next. The main question is whether the benefits can outweigh the inconveniences that come
with the decision.

This is, in fact, not a binary question with only “yes” or “no” as an answer. There is no reason
why all MDC projects need to be treated alike. It can be possible to recommend a preferred
solution but embrace other solutions for certain projects. Another possibility is to let teams
choose from a pool of pre-approved options.

Generally speaking, projects which require integration with an organization’s technical

infrastructure benefit from selecting and staying with one solution. Examples include when a
server for data collection is set up behind an organization’s firewall or where data from the
project is used in different tools and databases across the organization.

On the other hand, projects that are independent of the existing enterprise data and
infrastructure and do not require any special IT support can use different platforms more easily
as they don’t require the entire organization to restructure its workflows. In these cases, there
is no harm in using several solutions within the same organization, provided that the
departments switch platforms based on needs, and not out of curiosity for the latest innovation,
and don’t unnecessarily burden staff and enumerators with ever changing solutions.

For these independent projects, there is often an added value in giving staff the flexibility to
use the tool they deem best for a given project. Not narrowing down their options unnecessarily
allows them to take advantage of new product developments, be it new features and
functionality in an existing solution or an entirely new product. After all, the MDC sector is a
fast-paced environment where considerable changes take place from one year to the next.

How to Choose a Preferred Solution?

Organizations should consider the following five key factors when attempting to find a platform
that can serve as the preferred solution for the whole organization:

Factor Component

User and role - Which levels of management are required? Organization, Region,
management Country, Project, …
- How strictly do tasks need to be attributed to certain roles? Viewer,
Enumerator, Administrator, Project Manager can do X, Y or Z in the
system.
- How strictly do projects need to be separated from each another?
Access to folders/projects can be set independently of roles.

info@cartong.org | http://www.cartong.org Page 12 | 66

 Benchmarking of Mobile Data Collection Solutions

Security - How sensitive is the data collected?

- What are the legal and moral obligations to protect (sensitive) data
that the organisation has?
- Are there organizational standards where and how such data is
stored?
Cloud, encryption, behind organizations firewall, …

Data - Are specific data formats or a certain data structure needed for
integration analysis with existing software?
and analysis - How is data linked or integrated into existing enterprise databases?
Only manual export, API, custom developed workflows

Additional - Is monitoring or case management needed?

requirements - What are the language requirements? Application and online interface
languages (especially if non-Latin characters)

Migration of - Can the new tool continue to perform the same tasks? Specific needs
existing of existing projects
projects (if - Is it easy enough for staff to adapt and adopt the new tool? Capacity
required)
building needs
- Does data need to be migrated from existing solution(s) to the one
chosen and if so, what are the available options?

Only if at least one solution can be found that fits the identified requirements or is an acceptable
compromise, does it make sense to choose a preferred solution for an organization.
Nevertheless, a compromise is always possible and it needs to be established how strictly a
preferred solution should be imposed. Different options are available, for example to

a) use only one solution in the future but allow existing projects to stay with their solution
b) ensure capacity building, IT support and allow stronger integration for only one solution,
while accepting other solutions for more independent projects (potentially requesting
justification for a deviating choice)
c) suggest a pool of 3 to 4 solutions to account for different project requirements

A less rigid setup such as this allows staff to keep some flexibility while, at the same time,
providing guidance on which solution(s) should be considered.

D. What are the aspects to keep in mind when you budget your MDC?
Budgeting for Mobile Data Collection in general

Budgeting for data collection, independent of the approach, comprises two types of expenses:
initial and recurring costs. The initial costs for a mobile approach to data collection are often
significantly higher than for a paper-based approach given that phones have to be purchased
and staff or enumerators trained to use the software and hardware. On the other hand, a paper
based approach has considerably higher recurring costs, particularly when several surveys are
conducted or a situation is monitored over time. The biggest cost differentiator between paper
and mobile based surveys are the costs associated with data entry clerks. For example, in 2015
Oxfam estimated that it would cost close to 1,500 GBP (approx. 1,900 USD) to employ clerks
for 15 days to enter data from a survey they were planning to conduct in Thailand and the

info@cartong.org | http://www.cartong.org Page 13 | 66

 Benchmarking of Mobile Data Collection Solutions

Philippines. This was 10 times as much as was budgeted for the same survey using mobile
devices. 11

Another recurring expense of paper-based surveys are printing costs, which Oxfam, in the
same study, estimated to be 200 GBP (approx. 250 USD) – or approximately the price of one
smartphone.
In addition, even with a paper-based approach, IT and software costs can be considerable
depending on where and how data is stored and analysed.

The overall costs of data collection depend on the factors listed in the table below. For
comparison we list the most important expenses for both, paper-based and mobile data
collection:

MDC approach Paper-based

approach
Hardware, including replacement handsets, batteries and yes no
chargers, Sim cards if required
Software development possible no
MDC software licenses (possibly recurring) possible no
Analysis software license (possibly recurring) possible possible
Initial costs

In-house IT costs yes possible

Training enumerators to use the hard- and software yes no

Training data entry clerks to use the software minimal, only yes
for backups
Training enumerators in survey techniques yes yes
Printing costs minimal, only yes
for backups
External MDC expert/consultant possible possible
Piloting the survey questions yes yes
Recurring costs

(incl. technical implementation)

Enumerators’ salaries yes yes
Staff time for data entry no yes
Staff time for data validation/verification minimal yes
Staff time for administration/user management of platform yes no
Staff time for in-house data analysis yes yes
Backup costs yes yes

One of the biggest budget items for any MDC project are the costs for using the chosen MDC
software platform. This also applies to Open Source solutions, as the use of Open Source tools
also incurs costs. Unfortunately, these costs can also be surprisingly hard to calculate.

The three most common ways to charge for MDC platforms are:

• Pay per user, submission, form or question

11
Going Digital - Using digital technology to conduct Oxfam’s Effectiveness Reviews,
http://oxfamilibrary.openrepository.com/oxfam/bitstream/10546/578816/4/cs-going-digital-effectiveness-reviews-
290915-en.pdf, Emily Tomkys and Simone Lombardini, Oxfam GB, September 2015, Retrieved: 6 December 2016

info@cartong.org | http://www.cartong.org Page 14 | 66

 Benchmarking of Mobile Data Collection Solutions

• Pay for features and functionality

• Pay for storage space
Many plans combine these three elements, for example by charging a fixed amount for certain
features multiplied by the number of users. Since each tool in this report weighs and combines
these elements differently, it is impossible to fully compare the plans and identify an overall
best-priced option since this will depend on the design of the project.

In addition, many of the tools included in this report offer enterprise plans that provide clients
with more flexibility than their basic plans. This can be discussed with the sales teams. Some
providers also offer a special discount for non-profit organizations.

info@cartong.org | http://www.cartong.org Page 15 | 66

 Benchmarking of Mobile Data Collection Solutions

Part II. Product Evaluation

New MDC solutions are appearing and disappearing every year, making the pool of potential
products overwhelming and hard to assess. For this report, the authors have not attempted to
assess all possible solutions but rather to provide an overview of good solutions for different
scenarios in the field. Only platforms that met the following criteria were considered:

• Must facilitate data collection on the ground and not remotely (e.g. by SMS)
• Must allow data collection without any network (internet and/or 3G)
• Must have significant user buy-in or track record in humanitarian organizations
• Must have been used in multiple contexts and countries
• Must be designed for MDC (MDC is not just as an add-on)

Initially, 26 products were identified for this report, a list that was eventually narrowed to 16,
all of which can add value to programs in humanitarian and development contexts. The list of
all 26 tools and the reasons for the exclusion of certain solutions of this benchmarking is
available in the annex (see List of All).

1. Common Aspects and Features of All MDC Platforms

As mentioned, this report only includes viable solutions for organizations that are working in a
humanitarian or development context. As such, many of the solutions seem very similar. In
fact, the main differences are frequently related to price, specific sectors where the product
originated from, or usability. In other cases, it is a question of specific features that are better
in some products than in others.

All solutions discussed on the following pages can be used to create surveys and to collect and
manage the data. However there are big differences where the analysis of the data is
concerned. Here, some solutions provide no or only very limited support.

The following paragraphs lists the basic features and functionality that you can expect from the
tools during the different steps of MDC:

All tested solutions offer a visual form builder where surveys can be created, for example by
dragging different types of question (such as multiple choice or text field) onto a blank form
where details for each question can be added. For some solutions creating complex forms might
require the use of Excel (following Xform standards). While the products differ in the question
types they support, all solutions include at least: free-text fields, single and multiple-choice
questions, GPS points and permit skip-logic. Some, more advanced products include features
to monitor aspects over time, collect dates, photos, repeat certain questions several times,
scan QR- or barcodes or calculate values based on previous questions.

Once a survey has been created, data can be collected in the field and saved offline on the
phone. None of the reviewed products require a constant internet connection or a SIM card to
collect data. All solutions, with the exception of DeviceMagic and Traxilo, use UTF-8 encoding
for their survey questions so that questionnaires can be created in almost any language and
alphabet. However, not all solutions can switch between languages during a survey which is
useful in a culturally diverse setting.
After the data has been collected, it has to be sent to the MDC platform. For this step, most
solutions require a WiFi or 3G internet connection. Only ODK tools that are used with an offline
server setup, as well as Briefcase, can retrieve data manually from the phone and push it to

info@cartong.org | http://www.cartong.org Page 16 | 66

 Benchmarking of Mobile Data Collection Solutions

the server to work entirely offline. On the server the data is displayed in a table. In some
solutions it is possible to edit, add or delete data directly on the server, others only allow the
data to be viewed but not modified. It is also through the server interface that forms, projects
and users can be managed.

Analysis is where the platforms’ abilities vary the most. The two main differences are:
• How many analysis features (and which ones) are available
• The available level of refinement (basic/advanced).

The following table summarises typical analysis features.

Analysis features Basic features Advanced features

Mapping (assuming GPS • Shows data points on a • Ability to show data on a

coordinates collected) map. map, based on attributes
• Ability to filter which
answers are shown
• Some level of formatting is
possible (custom base
maps, colour theme,
legends, etc.)
Graphs • Bar graph and/or pie chart • Ability to edit styles to fit
• May or may not be existing report templates
possible to select the type • Filters: showing only data
of graph for specific admin levels
• Little to no ability to adapt and/or exclude some
the graph presentation answers (such as NA, DNK)
(colour themes etc.) • Cross-tabulation
Indicators • 1 question = 1 indicator • Scoring or other
calculations based on the
data are possible
• To some extent it is
possible to build indicators
by combining questions or
making calculations
Dashboards • Not available • Ability to construct a set of
graphs for questions
and/or indicators and save
them
• May be possible to share
the dashboard
• Dashboard can be saved &
edited

No matter which solution you choose, the option to export the collected data at least in csv/xls
format is available in all platforms. This enables you to analyse or visualise the data further
using external software such as Excel, Tableau or SPSS.

info@cartong.org | http://www.cartong.org Page 17 | 66

 Benchmarking of Mobile Data Collection Solutions

2. Platform Reviews
CartONG staff tested the selected platforms using two existing surveys (see annex). They
created similar forms on the platform (or imported the existing form where possible), collected
sample answers offline, synced the phone data with the platform, visualised and filtered the
data in the online interface where possible and tested the export options and format of the
output. In addition, staff used internet research and direct discussions with the solution
providers to get additional information on API access, security features, user management,
offline setup and different pricing schemes.
The snapshots on the following pages summarise the main strengths and weaknesses of each
product. They consist of a short introductory text for each solution, a table highlighting the
pros and cons and one or several screenshots of the user interface.

The test criteria, including examples, are available in the annex (see MDC Technical
Requirements)

Features that are common to all platforms (see: Common Aspects and Features of All MDC
Platforms) will not be mentioned again. To save space, other features are only mentioned in
the short profiles if they stand out or differ significantly from the average solution (positively
or negatively). This was done to save space. A detailed table with all features for all solutions
is available at the end of the chapter.

The tools are for simplicity’s sake presented in three sections first looking at ODK-based
solutions then at other Xform based solutions and finally looking at other solutions. Within each
section, the tools are listed alphabetically and the order does not indicate any preference or
ranking.

info@cartong.org | http://www.cartong.org Page 18 | 66

 Benchmarking of Mobile Data Collection Solutions

A. ODK- based solutions

KoboToolBox

http://www.kobotoolbox.org/

Tested version: with ODK Collect v1.4.14 (31.01.2017)

KoBoToolbox is a free and open source MDC tool that is supported by OCHA (United Nations
Office for the Coordination of Humanitarian Affairs) and that was developed by the Harvard
Humanitarian Initiative. It is used widely in the humanitarian and development community.
KoBoToolbox is based on the Xform standard which makes it very easy to share forms between
many MDC platforms.

Strengths and weaknesses

KoBoToolbox is free, open source and easy to set up and deploy for a majority of mobile data
collection needs. It is possible to map results or access data through the API for use in more
advanced analysis tools.

Its main weaknesses are issues of user-friendliness, a lack of stability as well as a lack of
related user support and communication.

When is it a good solution?

KoBoToolbox is a good and to date free solution for most mobile data collections that do not
require certain features, such as monitoring, sensitive data protection, or advanced user/role
management.

Figure 1: Kobo Map interface showing a submission, including photo

info@cartong.org | http://www.cartong.org Page 19 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- All standard questions - External lists don’t work on

Form features - Standard metadata web form
- Possible to switch language in - No monitoring or editing on
form phone
- Edit & delete of individual
submissions possible (uses
Server Enketo)
features - Interface in four languages
including Arabic
- Versioning of projects after
deployment possible
- Briefcase compatible
- Export format includes sav
- Form and data upload possible
Import/export through Briefcase
- Flexible options to export
groups
- API with read/write access
Analysis & - Map offers data viewing - No filters
visualisation options based on responses - Only basic charts
and legend
- Prepackaged Excel tool for
offline analysis available
- Forms are shared with users - No custom roles
User rights who have their own account - Some rights are at account
and not at project level
- Encryption on phone and server - No specific management for
Security possible Personal Identifiable
Information Not HIPAA
compliant
Additional - Free plan for humanitarian orgs
details - Offline setup possible although
this requires IT skills

info@cartong.org | http://www.cartong.org Page 20 | 66

 Benchmarking of Mobile Data Collection Solutions

ODK Aggregate

https://opendatakit.org/

Two setups were tested for this report:

Aggregate v 1.4.13

a) servlet Tomcat 8 on AWS with Mysql DB 5.7.18.

Install on Ubuntu Server 16.04 LTS (16.01.2017)
b) on Google App Engine (17.01.2017)
ODK Aggregate is the core engine at the heart of many of the closely related platforms. It has
been developed with flexibility in mind so that users can download, install and configure their
own server.

Strengths and weaknesses

An installation on Google App Engine retains much of the flexibility and is easier to setup than
the alternative, a custom installation in the cloud (Amazon Web Service was tested, but many
other providers exist). This approach allows any organization to run their entire MDC operation
on their own servers and keep full control over all updates. It is, however, more technically
demanding and the challenges of maintaining software should not be underestimated.

When is it a good solution?

ODK Aggregate is a good solution for organizations with strong IT resources that have specific
needs for their MDC infrastructure. For example, because they require certain security features
such specific standards that are not easy to find with providers or that need to operate behind
an organization’s firewall. Another reason could be the need to retain full control of the
database, for example, to integrate MDC data with other processes.

Figure 2: Aggregate User interface, including pie chart

info@cartong.org | http://www.cartong.org Page 21 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- all question types supported - no monitoring or editing on

(but not all available in visual phone
Form features form builder)
- all standard metadata
- possible to switch language in-
form
Server - filters available (but not very - no edits on server (except if
features user-friendly) set up with Enketo)
- delete possible - English only interface
- Additional formats: kml, json
Import/export - Form & data upload
- API (read & write) available
- Briefcase compatible
- Charts (pie charts & bar graphs) - Analysis is are much more
Analysis &
possible advanced on Google
visualisation Earth/Fusion tables (but this
- Some mapping possible
is easy to set up)
- Anonymous submission can be - No custom roles and only
User rights set up and also prevented basic predefined roles
- Access only on full account
- Encryption on phone and server - If https is required, must be
possible configured separately for
Security
- https can be used by default on AWS installation
Google App Engine
- Free but server & installation - On AWS: hosting and
required bandwidth costs
Additional
- On Google: free quota for - Offline setup possible with pre-
details
submission (then manipulation built VM
costs)

info@cartong.org | http://www.cartong.org Page 22 | 66

 Benchmarking of Mobile Data Collection Solutions

ONA

https://ona.io/home

Tested version: v1.3.25 (19.01.17)

ONA is an actively maintained solution whose developers are responsive to the needs of their
users, which include a large number of humanitarian organizations.

Strengths and weaknesses

ONA is based on the Xform standard, therefore the coding of the survey will meet most
requirements. The analysis features on the website are getting better and while ONA is still not
a complete analysis solution in itself, it can probably answer a significant share of an operation’s
needs, especially closer to field level. User management options are particularly rich, especially
for an organization looking at managing different units (country, regions, types of programs)
on a single account.

It doesn’t offer advanced features such as case management and isn’t well-suited for
monitoring. There’s also no option to filter data on the website.

When is it a good solution?

ONA is particularly well-suite for simple MDC (with no monitoring or case management),
especially if many different users require limited access to a large number of surveys. It really
shines when managing larger work teams is a requirement.

Figure 3: The online data analysis platform

info@cartong.org | http://www.cartong.org Page 23 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- All question types supported - No monitoring, only csv

Form features - Standard metadata preloading
- Possible to switch language
Server - Editing & deleting possible - Interface only in English
features (except for data pushed with
Briefcase from another platform)
- Briefcase compatible - Data upload only at beginning
- Form upload possible of project
- Data upload possible
- Export formats include json, sav,
Import/export kml
- Flexible export options for
groups & choices
- API to view, query, for form
definition, …
Analysis & - Map with advanced options - No filters
visualisation including filters and legend - Very basic charts

- Four predefined roles - No custom roles

- Access can be limited, inviting
User rights “collaborators” to a project
- Access can be set on a project
basis (many projects per
account possible)
Security - No encryption

Additional - Free plan available but with

details limited forms and features

info@cartong.org | http://www.cartong.org Page 24 | 66

 Benchmarking of Mobile Data Collection Solutions

SurveyCTO

http://www.surveycto.com/

Tested version: v2.212 (19.01.2017)

SurveyCTO is a platform that has fed back a number of features to the

broader ODK community and seems to have gotten many important elements of MDC right.
While it isn’t a fully-fledged case management tool, it is possible to configure a survey drawing
in responses that have been submitted before to be displayed on the phone.

Strengths and weaknesses

All of the basic features for MDC are in place and work reliably, with extensive documentation
and user support to help in their use. Some advanced features are also in place: analysis on
the account is improving and includes filtering and grouping options as well as the capacity for
combining data from different surveys for analysis.

User management, on the other hand, isn’t very advanced compared to other platforms: rights
are very broadly disaggregated. Also, rights are always granted to an account, meaning it isn’t
possible to limit access to a specific survey, or only to the analysis tabs.

While the documentation is extensive, it is not well illustrated and perhaps a little dense for
new users.

When is it a good solution?

Organizations that are looking for a complete, flexible and versatile platform and that have
only very limited needs for user and access management should consider SurveyCTO. Short of
full-on case management and monitoring, this platform covers most use cases.

Figure 4: Data workflows one can easily set up

info@cartong.org | http://www.cartong.org Page 25 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- All question types supported - Limited monitoring and case

- Standard metadata management are paid
- Possible to switch language in features
form
Form features - External data works on webform
as they do on the phone
- Monitoring and case
management capacities (albeit
limited and as paid features
only)
Server - Interface in English only
features

- Briefcase compatible - Info on API not part of

- Data upload available documentation
Import/export - Form upload during test
- Options for export of groups and
choices
Data - Advanced charts
visualisation - Images available on server
options - Basic map (display data point
location only)
- Basic filtering
- No custom roles
User rights - Access limitations cannot be
set
- Encryption on phone and server - Can’t prevent anonymous
Security submissions
possible

Additional - Free plan available but with

details limited forms, data and features

info@cartong.org | http://www.cartong.org Page 26 | 66

 Benchmarking of Mobile Data Collection Solutions

B. Other Xform-based solutions

CommCare

http://www.dimagi.com/products/

Tested version: 2.32.1 (17.01.2017)

CommCare, a solution developed by Dimagi, has originally been designed for field based,
mobile health care workers. Because of this legacy, the solution comes with a suite of case
management features, such as setting up a list of cases (patients) and having forms and
surveys associated with each case. CommCare can also be used for ordinary surveys and comes
with an online form builder assisting users to set up their forms.

Strengths and weaknesses

Whilst CommCare is using XForms and offers a formbuilder as well as direct xml editing, only
forms directly created in CommCare import easily, other xml forms can only partially import.

To access the forms, users need a dedicated user account. Using the mobile app is
straightforward, however, the platform itself is less intuitive and has a steep learning curve
before someone can navigate and use it with ease. Analysis and visualisation options are limited
and most users requiring instant visualisations, maps or dashboard-like features will have to
set up a third-party tools and use the API. CommCare comes with advanced activity monitoring
features, which can be useful if the project requires that staff and case activities are closely
watched. Reports and data exports can be customized so that only selected fields get exported,
which can come in handy when the core data needs to be shared but not all of it should be
made available.

When is it a good solution?

If your project would benefit from case management features, such as reminders for your staff,
and surveys linked to case management, then it can be well worth investing the extra time
necessary to become comfortable with the platform. This is particularly true if you would like
to submit reports from the field by SMS in areas where mobile internet is not readily available.

Figure 4: CommCare’s form builder interface

info@cartong.org | http://www.cartong.org Page 27 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- All main question types - Lacking support for external

- All standard metadata lists
Form features - Monitoring possible (with Pro - Setting up calculations is
account) not straight forward even
- Possible to switch language in with XLS and XML coding
form knowledge
- Editing and deleting on the - No filters for questions
server is possible (with Pro+ (only metadata)
Server account) - The customizable report
features - Interface available in EN, FR and builder is only available as
ES an add-on to a Pro account
for an extra fee
- No maps
- Data upload possible incl. bulk - Xml forms created by other
upload (only for cases, not for applications using Xform
all data) standards can’t by easily
Import/export - Form upload possible (but imported: labels do not
different structure from other import fully, languages
solutions) aren’t imported
- Exports can be created to only - Only csv and Excel as
include some of the fields exporting formats
- Limited to monitoring activities, - No built-in server options to
Analysis & like number of submissions, visualise the data
visualisation mobile workers, etc.
- API (on Pro plan)
- Access can be limited on form
User rights basis, anonymous submissions
prevented
- Variety of predefined roles
- Server encryption possible
Security - Compliance with HIPAA and de-
identifying data available as part
of advanced and enterprise plans
- Free plan available but limited - Not available with free plan:
features and number of users Case management, web-
Additional based app, case importer,
details Excel dashboard analysis,
HIPAA compliance,
advanced user
management, API access

info@cartong.org | http://www.cartong.org Page 28 | 66

 Benchmarking of Mobile Data Collection Solutions

DeviceMagic

https://www.devicemagic.com/

Tested version: Demo account (19.01.2017)

DeviceMagic supports most of the features you would expect from an MDC solution. Plans are
priced by device and by month.

Strengths and weaknesses

Overall, DeviceMagic is easy to setup and use, including the graphical form builder. It is very
advanced when it comes to data interoperability and sharing or sending data to shared folders
like Dropbox and Google Drive, or common communications platforms like Slack, Evernote,
Podio, Box, and Zapier.

Its main weaknesses are: lack of support for some non-Latin languages, inability of switching
languages in a form, as well as a lack of data editing features. Another feature might be an
advantage for some and a disadvantage for other organizations: all parent and child files are
automatically combined in one Excel file which can be useful if this is required for your analysis
as it allows easy filtering based on your meta or household data- however for surveys that
collect quantitative questions about households and household members, this feature can make
it challenging to extract indicators and statistics from both.

When is it a good solution?

DeviceMagic is easy to setup and use. As the pricing is device based, DeviceMagic can be an
economical solution for organizations working with a small number of key informants that
report regularly (for example once per week for a whole year).

Figure 5: DeviceMagic's form builder interface

info@cartong.org | http://www.cartong.org Page 29 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Most question types supported - Missing question types or

- All standard metadata logic: no notes
- GPS comparatively slow
Form features - No right-to-left language
support
- No monitoring
- No option to switch
language in form
Server - Deleting on the server is - No edits on server
features possible - English-only interface

- Additional formats: json, xml - No data upload

- Form upload possible but only in - Form conversion service
json format was not working, labels
- Additional link with repositories converted also to column
such as drop box, OneDrive names
Import/export - API access for enterprise - When exporting forms with
account looped groups in xls or csv,
the parent data will be
duplicated and added to
child data. This can make
analysis on the parent file
difficult.
- Map - No filters except on map by
- Pushing to Google Spreadsheet is submission date
Analysis &
easy - No charts except for staff
visualisation tracking (not based on
- API available questions)
- Map is only basic
- Customizable roles with detailed
User rights options
- Access limitations by group
Security - No encryption on phone or
server
Additional - No free plan
details

info@cartong.org | http://www.cartong.org Page 30 | 66

 Benchmarking of Mobile Data Collection Solutions

Magpi

https://magpi.com/

Tested version: DataDyne 5.4.8 (25.01.2017)

Magpi is part of the “wider” ODK family of platforms. It has a different feel than other ODK
solutions and contains a number of less common features, such as support for IVR (Instant
Voice Response) and SMS data collection (both for single-question surveys and more complex
ones).

Strengths and weaknesses

Although the support for different question types is good, some more advanced form building
features are not available, such as calculations based on dates and time. Some features are
available but in a different structure compared to ODK. Magpi Enterprise users can easily
integrate their data into other solutions, more information to be found here.

When is it a good solution?

Magpi is a good candidate for organizations that need to collect data via SMS or voice
messages, for example crowd-sourced data or feedback from beneficiaries. It is good for MDC
projects of average complexity. If your organisation is using Zapier, Salesforce, Zoho,
MailChimp or any other similar solutions to manage contacts or other operational data and is
planning to augment with MDC Data, Magpi might be a good match to complement your other
systems.

Figure 6: Magpi online interface to download the app and follow the fleet of phones

info@cartong.org | http://www.cartong.org Page 31 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Standard metadata - Missing questions: calculation

options are limited, no
Form features signature, images only with
paid account
- No monitoring possible
- Only one language per form
Server - Editing and deleting is possible
features - Several language options incl.
CN
- Additional mdb format - API access only with
- Form and data upload possible Enterprise account
Import/export - Advanced data interoperability
options for many other
systems
Analysis & - Advanced options to filter,
Visualisation cross-tabulate data
- Advanced chart/ reports with
Pro+ account
- Advanced map options
User rights - Form can be shared with other - Custom roles only with
users Advanced+ account
Security - No encryption on phone or
server
Additional Free plan available with very
details limited features and forms

info@cartong.org | http://www.cartong.org Page 32 | 66

 Benchmarking of Mobile Data Collection Solutions

Survey123 for ArcGIS

http://survey123.esri.com/

Tested version: 1.10.25 (16.01.2017)

Survey123 for ArcGIS is an MDC solution designed by ESRI, a company specialised in

geographic information. Accordingly Survey123 has a strong geographical component and is
best used in organizations that have dedicated GIS staff who are already using ESRI products.
Survey123 requires an ArcGIS Online (AGOL) account or alternatively a Portal setup 12. While
there is a Survey123 online interface which gives the user access to the uploaded forms and
details some standard statistics, in order to interact with the data, to filter by fields or to see
the data points on a map it is best to go through the standard ArcGIS Online interface.

Strengths and weaknesses

Survey123 is a solution where surveys have a focus on geographic data and mapping is a
requirement. In addition, Survey123 supports all main question types except for monitoring
and even allows anonymous submissions, with a server interface in over 30 languages.

Data analysis apart from mapping is the clear weakness of Survey123. While it is possible to
filter data for mapping and visualise it creating heat maps, etc., there is no option to filter data
for other visualisations such as graphs. In addition, the tool is weak if data is not needed in a
GIS compatible format. Images and attachments can only be downloaded to the gdb format
which is only useful if the users also work with ArcGIS Desktop..

When is it a good solution

Survey123 for ArcGIS is a solution which is ideal for an organization with strong GIS usage.
If an organization has the ArcGIS Suite (Desktop, Server, Online/Portal) and also intends to
collect data through mobile devices, Survey123 is an obvious choice especially given that
enterprise support is available and that the solution has frequent updates.

While it is possible to collect non-geographic data with Survey123, there is no point in doing
so as data is stored in geographic formats. Given that anonymous submissions are possible
since early 2017, Survey123 has a strong advantage over most other solutions for citizen
science projects which collect spatial data.

In April 2017 a new version was released including features like previously collected data can
be edited on the phone.

12 Portal for ArcGIS is a platform for geographic data similar to ArcGIS Online but instead of being stored

in the cloud and managed by ESRI it is stored behind an organization’s firewall and managed by the
organization

info@cartong.org | http://www.cartong.org Page 33 | 66

 Benchmarking of Mobile Data Collection Solutions

Figure 5: Survey123's interface

for analyzing data

PROS CONS

- Standard question types

supported
Form features - Standard metadata
- Possible to switch language in
form
Server - Large variety of server interface - Features can only be edited
features languages (>30) in AGOL, but no option to
add related table data
- Additional export formats: - Only export as gdb allows
Import/export json,shp, gdb download of images
- ESRI REST API to access data - Not possible to upload data
- Form upload possible
- Advanced mapping options in - No filtering in Survey123
AGOL incl. heat maps etc. interface (layers can be
Analysis &
filtered in AGOL)
visualisation - Basic charts that do not
allow any filtering except by
submission date
- Anonymous submissions are
User rights possible
- Customizable user roles
Security - No encryption on server or
phone
- Possible to set up on private
Additional server with Portal for ArcGIS
details (only with built-in identity store)
- Active user community under
GeoNet

info@cartong.org | http://www.cartong.org Page 34 | 66

 Benchmarking of Mobile Data Collection Solutions

C. Other solutions

Akvo Flow

http://akvo.org

Tested version: Flow v.2.2.9 Dashboard v.1.9.11.1 (16.01.2017)

Akvo Flow is a relatively simple tool for basic monitoring operations that is well suited for
situations where data with a geographic component (infrastructure, water points, etc.) has to
be collected repeatedly. The platform is easy to setup and use, with acceptable user
documentation and support. Data collection points can be displayed on an offline-map on the
phone and historical data can be accessed easily, which is particularly useful for programs that
include monitoring component. These features are not common in other products.

Strengths and weaknesses

The solution is expensive if only comparatively simple data collection is needed for a project
and if the monitoring and mapping components are not required or not widely used. The
analysis features provided by Akvo Flow are unlikely to be sufficient on their own and the
platform has limitations that lessen the usefulness of some features. For example. GPS
coordinates cannot be edited and while it is possible to bulk upload monitoring data related to
a data point, the initial creation of these data points cannot be done in bulk.

When is it a good solution?

Akvo Flow is fairly easy to setup and use. It is a good solution in situations where data needs
to be monitored over time and where being able to see the data collection points (such as
water points) on an offline map on the phone adds real value. It can be a good starting point
for MDC projects that require cloud-based infrastructure monitoring.

Figure 6: Interface to export data

info@cartong.org | http://www.cartong.org Page 35 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Monitoring possible - Missing question types &

logic: calculations, notes, no
Form features external lists
- Missing metadata: recording
time
- Editing & deleting possible
Server
(except for GPS coordinates)
features - Several language options:
EN,FR,ES,PT,VN,ID
- Data upload possible in bulk - No form upload (must use
Import/export (but not to bulk-upload new the visual form builder)
data collection points) - API uses HTTP only
- Simple API (read-only)
- No filters
- Only very basic charts (one
Analysis &
chart at a time)
visualisation - Very basic map
- Images not accessible on
server
- Custom role creation
- Access on folder basis
User rights - Detailed breakdown of tasks
that can be assigned to each
user
Security - No encryption options

Additional
details

info@cartong.org | http://www.cartong.org Page 36 | 66

 Benchmarking of Mobile Data Collection Solutions

Dharma Platform

http://dharmaplatform.com/

Tested version: Version 0.47 (02.03.2017)

Dharma Platform is a stand-alone tool that can manage different data collection projects within
one platform. Although very expensive, its way of grouping projects into teams and regions
make it a good tool for globally operating NGOs and IOs.

Strengths and weaknesses

Dharma excels in monitoring and case management. Users can create a main entry which is
then linked to attributes that are monitored and updated regularly. The server interface is at
the same time easy to use and powerful, especially when it comes to filtering and visualisation,
which makes Dharma a good choice for users that don’t have a lot of experience with these
type of tools.

However, there is also weaknesses: some very commonly needed field types or options are not
(yet) supported: most importantly, calculations or the option to require an answer before
submitting a form is not or not sufficiently implemented. For example, at the moment it is only
possible to mark text input fields as mandatory but not multiple choice question.

When is it a good solution?

It is a good solution for projects that aim to monitor information regularly, such as health-
related indicators for different patients. The focus on health is also reflected in the platform’s
security features, especially its HIPAA compliance.

Figure 7: Visualisation of the analysis platform

info@cartong.org | http://www.cartong.org Page 37 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Full case management - Only limited question types,

options (“longitudinal study” missing are: calculations,
(incl. edit and delete on pictures, barcode support,
phone) set up input constraints.
Form features - Simple, yet powerful form Only text fields can be set
creation interface as mandatory
- Can split projects by different - Not possible to switch
regions, teams and compare language within a form
them with each other
- Standard metadata
- Map only available for staff
tracking not as part of
Server
analysis/reporting
features - No option to manipulate the
data on the server
- Interface only in English
- Dharma team can provide - No form upload
Import/export support for potential data - No data upload
migration
Analysis & - Advanced filtering and cross-
visualisation tabulation options
- Charts based on filter
- Access on project basis with - No custom roles
User rights subset by team or region - No anonymous submissions
possible
- Persistent encryption at
database level on server and
Security phone
- Compliant with ISO 27001-
27008, HIPAA, USFDA
Additional - No free plan
details

info@cartong.org | http://www.cartong.org Page 38 | 66

 Benchmarking of Mobile Data Collection Solutions

Fulcrum

http://www.fulcrumapp.com/

Tested version: 2.22.1 (19.01.2017)

Fulcrum is used by many private companies but also counts UNICEF as one of its clients. It has
a strong geographical component and provides most of the features that we find in advanced
MDC tools, including editing. A free trial is available, which can help get to know the app.

Strengths and weaknesses

The app excels through its ability to work with geographic data. It provides an option to include
basemaps from OpenStreetMap, Mapbox or custom mbtiles to identify survey locations more
easily in an offline environment. In addition, a tutorial on how to integrate data into the online
mapping platform CARTO.com is available and users can export their data in various geographic
formats including shp, kml or gdb. This makes the integration with desktop GIS application
such as Google Earth, ArcGIS or QGIS very easy. At the same time, almost all question types
can be created with the exception of custom constraints or references to previous answers.
Fulcrum will even keep a history of the data allowing users to see changes made to the entries.
This makes it a very useful tool for many data collection projects.

However, Fulcrum does not perform as well when it comes to analysis and security. The app
doesn’t provide advanced security features like encryption and the only option to view the data
on the server is using a map or table. Charts or filters cannot be created directly through the
server interface so that additional tools are needed for most forms of analysis.

When is it a good solution?

While Fulcrum is not a full case management solution, it allows users to update information on
the phone and keeps a history of that information. Given that it also provides strong
geographical features, it can be very useful for monitoring and mapping facilities and
infrastructures.

Figure 8: Geographical interface to view the data and form conception interface

info@cartong.org | http://www.cartong.org Page 39 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Edits on phone possible - No switching between

Form features - All standard metadata languages
- No custom constraints or
reference to previous answer
Server - Edit & delete on server - Only in English
features

- Many GIS formats incl. shp, - No form upload

gdb sql (PostGIS)
Import/export - Advanced options for uploading
data matching fields (incl. in
bulk)
- API allows querying
Analysis & - No filters or charts
Visualisation - Only basic map

- Detailed options for custom

User rights roles
- Access to form can be limited
to certain users
Security - No encryption available

Additional - No free plan

details

info@cartong.org | http://www.cartong.org Page 40 | 66

 Benchmarking of Mobile Data Collection Solutions

IformBuilder

https://www.zerionsoftware.com/iformbuilder/

Tested version: App: 6.9.11.266.

Dashboard: 10.0.2.3938 (17.01.2017)

An intuitive tool with some advanced features and comprehensive documentation. The tool is
easy to use and based on a graphic interface. It supports basic case management, such as
linking different forms to a specific case, or assigning a case to a specific staff member. It is
developed actively and offers good documentation as well as an active community forum.

Strengths and weaknesses

IformBuilder’s plans are comparatively expensive and although the platform offers advanced
features that are lacking elsewhere (such as email alerts), it doesn’t offer some features that
are very basic for MDC: for example, calculations are possible but use of logical statements
(true/false) is limited.

When is it a good solution?

IformBuilder is a good choice when a simple tool to perform basic monitoring is needed, as the
platform is easy to learn. An iOS version of the application is also available, which may be
useful if the existing hardware in the organization is also based on Apple products.

Figure 9: Form conception interface

info@cartong.org | http://www.cartong.org Page 41 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Monitoring possible - Missing option: calculations

through “smart tables” cannot be set to invisible to
Form features but export options limited the user
- Metadata details vary by
output format (most in
json)
Server - Editing and deleting on - Interface is only in English
features the server is possible

- Additional export - Creates empty rows in csv if

Import/export formats: json, xml, atom no child (for groups)
- Bulk data upload possible

Analysis & - No filters

visualisation - Only one point can be
shown on a map at a time
User rights - - Only full account access
- No roles
Security - No encryption for current
version of the app
Additional - Onsite forum for help & - No free plan
details questions

info@cartong.org | http://www.cartong.org Page 42 | 66

 Benchmarking of Mobile Data Collection Solutions

Mobenzi Researcher

http://www.mobenzi.com/researcher/Home

Tested version: “Researcher” v5.5.1-i (07.02.2017)

Mobenzi count Oxfam, amongst other, as one of their most

prominent NGO clients. While the company provides different solutions, only Mobenzi
Researcher was included in this report.

Strengths and weaknesses

Mobenzi Researcher comes with some very strong features, including detailed user and role
management, analysis options that allow for very detailed filtering as well as the ability to work
in different languages within the same project. However, the platform also falls short in areas
that are standard in other solutions. For example during an export images are simply copied
into the cells of an excel file with no clear identifier or link to the original input. In addition the
GPS did not always work properly during testing. Finally, the interface, while powerful, is not
intuitive and user will have to spend some time to learn how to use some functions such as
setting custom filters for analysis.

When is it a good solution?

The platform’s strengths in user management and the ability to export data in the stat/transfer
format make it a good solution for larger data collection projects that need to be analysed
further in a statistical tool.

Figure 10: Interface to visualize data

info@cartong.org | http://www.cartong.org Page 43 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- All standard metadata - Missing question types:

custom constraints, hints;
Form features - Problems with GPS during
testing
- No monitoring or editing on
phone possible
Server - Editing and filtering is possible - Interface only in English
features

- Stat/transfer as additional - No data upload (feature still

Import/export export format in beta)
- API allows query, update, … - Picture embedded in xls
without strong link to cell
- Advanced filter options (but - Map view allows no
Analysis & text input not very user- customization
visualisation friendly)
- Charts take into account filter
options
- Very detailed support for role
User rights definition
- Access limited on a project
(“study”) basis
Security - No encryption

Additional - No free plan

details

info@cartong.org | http://www.cartong.org Page 44 | 66

 Benchmarking of Mobile Data Collection Solutions

Poimapper

http://www.poimapper.com/en/

Tested version: POI Mapper Plus 1.6.5m (24.01.2017)

Poimapper, as the name suggests, focuses on mapping points of interests in the field and
putting them on a map. However, it is not just a geographic application but has become a full
MDC tool. Complex form logic is supported as well as visualisations including charts and filters.

Strengths and weaknesses

Poimapper supports the collection of GPS points and can display data on a map, both on the
phone and on the server. Unfortunately, the map’s functionalities are limited and cannot be
customised, aside from clustering. However, Poimapper provides other options that make it a
compelling tool. Most data types are accepted, data can be edited on the phone and a history
is kept on the server. Filtering is strong which helps when analysing the data.

Less impressive is the user and role management, which is relatively limited, and the visual
form builder. While being powerful and working without problems, the form builder’s design
makes it often difficult for the user to verify the logical setup of the form, especially when
several skip patterns were implemented.

When is it a good solution?

NGOs can use Poimapper to collect and even monitor data on facilities that have a clear
geographical component. As data can be exported in SPSS, statistical analysis of the results is
possible.

Figure 11: Poimapper's form builder and reporting and analysis interface

info@cartong.org | http://www.cartong.org Page 45 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Most questions types are - No hints or signature

possible questions
- Standard metadata - Form creation interface
Form features - Different languages are kept in needs some time to get
different “views” of the same used to
form
- Keeps history of edits
Server - Editing and deleting on the
features server is possible

- Export format includes SPSS, - API for enterprise accounts

Import/export kml mentioned but no
- Allows upload of forms and data documentation openly
accessible
Analysis & - Advanced filter options - Map view is only very basic
visualisation - Good charts support that takes
into account the filter settings
- Access can be limited on a per- - No anonymous submissions
User rights form basis - Only three basic roles and
no customizable roles
- For highly sensitive data a security
Security chain-of-custody solution can be
provided
Additional - 30% discount for NGOs - Free plan available but
details limited to one user

info@cartong.org | http://www.cartong.org Page 46 | 66

 Benchmarking of Mobile Data Collection Solutions

Traxilo

https://traxilo.com/

Tested version: 1.5.5 (03.02.2017)

Traxilo is a tool that was developed to follow clients/beneficiaries as well as the activities linked
to them. Traxilo is used by a number of NGOs including Doctors of the World. It is extremely
easy to set up and to use.

Strengths and weaknesses

Traxilo’s strength lies in tracking services that are linked to specific beneficiaries. The interface
is very user-friendly. The system is very secure and built to favour anonymised beneficiaries
for better protection of sensitive data. For this purpose Traxilo can easily replace a person’s
name with a code. The idea is that data collection should mainly be used to focus on trends,
not on individuals. Whenever questionnaires are modified, they are automatically updated on
the phone.

When is it a good solution?

Traxilo is an ideal tool for one very specific use case: if an organization wants to build a list of
beneficiaries and then gather data linked to those beneficiaries (for example services provided
by the organization) or to monitor general services that are not related to a specific beneficiary,
but are part of a program.

Figure 8: The type of dashboard available to view

information on a cohort

Figure 12: Traxilo’s advanced filtering and chart options for beneficiary tracking, and below its mobile
interface

info@cartong.org | http://www.cartong.org Page 47 | 66

 Benchmarking of Mobile Data Collection Solutions

PROS CONS

- Monitoring or beneficiary - Key question types are

tracking missing; Only number,
single/multiple choice, text
Form features and required fields are
available.
- Only Latin characters
- No option to switch
language inside a form
Server - Editing and deleting on the - Only English interface
features server is possible

Import/export - - No data or form upload

- No API
Analysis & - Advanced filter options - No map view
visualisation - Advanced charts based on filters

- Easy access management via - Only two roles with no

email customization
User rights - Access can be limited on form
basis and shared with other
accounts
- Phone and server encryption -
- Demographic data for a cohort
Security can be captured without
identifying them (a code can be
used)
Additional - Not adapted for non-cohort
details follow-up use cases

Figure 9: Print screens of the mobile application

info@cartong.org | http://www.cartong.org Page 48 | 66

 Benchmarking of Mobile Data Collection Solutions

D. Table 1 – ODK and Xform based solutions

KoBoToolbox ODK Aggregate ONA SurveyCTO CommCare Device Magic Magpi Survey123

a) Aggregate v
1.4.13 on Tomcat
Tested version Ubuntu
ODK Collect v1.4.14 v1.3.25 V2.20 v 2.32.1 v 1.72.0 Magpi DataDyne 5.4.8 v 1.10.25
b) Aggregate
v1.4.13 on Google
App Engine

most: no notes,
difficult to set
constraints, only most: no calculations/
all but importing all but importing most: limited
calculation fields can reference in label text
external lists does external lists does most: no external calculations, no
Question types all all be hidden, GPS (i.e. data input cannot
not work on Enketo not work on Enketo list signature, no external
doesn't use WiFi be included in label
web form web form lists
network to improve (only answer)
location accuracy and
speed
Form features

all all, except IMEI, some

all
(start/end time, all are loaded by default.
all (can be set or (start/end time, all (included by all (identifies by all (identifies by
Metadata device id, phone nb, (start/end time can Optional: geostamp,
not) device id can be set default) username, not device) username, not device)
sim serial can be set be set or not) time stamp identifies
or not)
or not) user not device

Language/ limited, no right-to-

all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8)
charset support left languages

Switch language
possible possible possible possible no no possible
in form
limited (only csv limited (only csv limited (no csv with no (released in April
Monitoring no yes with Pro account no no
preloading) preloading) paid account) 2017)
Details

advanced, but not

advanced: filters, directly in survey123
advanced but no (data can only be no (only metadata no (only option is to
Filters no basic disaggregation, cross- interface. Can be set
difficult set up sorted) filters) filter map by date)
tabulation up in AGOL viewer
using sql
Server features

basic: the more

customizable report-
yes, on selected
basic: can't be builder is only advanced options with
Charts & fields, but not no: only to track staff, basic: can't be filtered
filtered or basic advanced available as an add- filters (for paid
reporting possible to change not based on input or customised
customised on to a standard+ account only)
layout
account for an extra
fee.

info@cartong.org | http://www.cartong.org Page 49 | 66

 Benchmarking of Mobile Data Collection Solutions

KoBoToolbox ODK Aggregate ONA SurveyCTO CommCare Device Magic Magpi Survey123

a) Aggregate v
1.4.13 on Tomcat
Tested version Ubuntu
ODK Collect v1.4.14 v1.3.25 V2.20 v 2.32.1 v 1.72.0 Magpi DataDyne 5.4.8 v 1.10.25
b) Aggregate
v1.4.13 on Google
App Engine

advanced when using

yes, if Google Map
advanced: with the AGOL map
API has been set up; advanced: with filter basic: displays data basic: GPS accuracy advanced: with filters
Map category options and no interface: heat maps,
applying a filter to a options and legend points on a map reduced (paid account only)
legend filters, custom colours,
map is possible
legend, ...

pictures are an
View images yes yes yes yes yes yes advanced feature (not yes
tested)

possible but not for possible through

possible (opens in no, unless Enketo is no, only possible to possible with PRO
Edit data pushed with no possible ArcGIS Online viewer
Enketo) set up edit datasets account
briefcase (not survey123 page)

possible but not to

Possible (data, but possible with PRO
Delete possible possible possible possible possible delete related entry
not form) account
(group)

~20 languages
EN, FR, ES, AR,
(defined by AGOL
Languages though translation EN EN EN EN, FR and some ES EN EN, FR, ES, PT, CN
profile), including
incomplete for some
Arabic, Hindi,...

Enketo can be
installed to edit.
Alternative
workaround: export,
Details:
change and delete
the original, then
upload changed
dataset
csv, kml, json (some
Data export csv/xls, kml, json, csv/xls, json, xml,
csv/xls, kml, sav Briefcase, some csv/xls csv/xls csv/xls, mdb csv/xls, json, shp, gdb
format sav docx, pdf
server export)
Export & Import

possible: xls (through

possible: xml but
possible: json but only Survey123 Connect
adaptations required
Form upload possible: xls possible: xml possible: xls possible: xls or xml coming from Device possible: xml but adaptations
if form comes from
magic required if data comes
other tools
from other tools)

possible (Briefcase
possible for new data possible: Excel case
possible with or CTO sync).
possible with (Briefcase or csv uploader (Pro possible: copy-paste
Data upload Briefcase (although Another option is to no no
Briefcase upload) but not for feature, could not be or upload
not always reliable) upload a csv to a
ongoing project tested)
dataset

info@cartong.org | http://www.cartong.org Page 50 | 66

 Benchmarking of Mobile Data Collection Solutions

KoBoToolbox ODK Aggregate ONA SurveyCTO CommCare Device Magic Magpi Survey123

a) Aggregate v
1.4.13 on Tomcat
Tested version Ubuntu
ODK Collect v1.4.14 v1.3.25 V2.20 v 2.32.1 v 1.72.0 Magpi DataDyne 5.4.8 v 1.10.25
b) Aggregate
v1.4.13 on Google
App Engine
flexible: direct
flexible: excel - flexible: csv: flat export - flat; defined: subforms are
Group/sub-form defined: csv flat defined: csv flat
related table, csv - table, briefcase or CTOsync defined: xls flat table exported defined: related tables
exporting table table
flat table xls: related tables export: related independently
tables

Multiple choice defined: both all in flexible: all in one flexible: all in one flexible: each choice
defined: all in one defined: each choice defined: all in one cell,
questions one column and each column or each column or each one column or all in defined: all in one cell
cell one column separated by comma
exporting choice in one column choice one column choice one column one cell

direct download: url link: full url in table, pictures are an only downloaded if
export media as .zip link: full url in table,
Picture export in table briefcase: briefcase-export link: full url in table link: full url in table advanced feature (not gdb chosen, no cell in
and with id in table export media as .zip
media folder as zip media as .zip tested) csv table

yes (but pushing

Briefcase
data has often been yes yes Yes no no no no
compatible
unreliable)

yes, (at least yes, with Enterprise yes, ESRI REST API;
yes: HTTPS read
yes: (contact standard plan): read account view, query, update,
yes, read & write yes: view, query, (Enterprise only)
yes, read & write customer support for & write using HTTP https://devicemagic.z delete
https://opendatakit. form definition,… http://support.magpi.
API https://kc.kobotoolb more details - not https://confluence.di endesk.com/hc/en- http://resources.arcgi
org/use/aggregate/d https://ona.io/static/ com/support/solutions
ox.org/api/v1/ part of general magi.com/display/co us/articles/218720498 s.com/en/help/rest/ap
ata-transfer/#APIs docs/index.html /articles/4865-magpi-
documentation) mmcarepublic/Comm -Device-Magic- iref/index.html?mapse
outbound-api
Care+HQ+APIs Database-API rver.html

Excel, Google Fusion, Excel, Drives (Google, Excel, ArcGIS

Easy integration Excel, SPSS, Google Excel, Google Earth, Excel, SPSS, Google Excel, Google Fusion,
Google Docs and OneDrive, Dropbox, (especially Online),
with Earth Google Fusion Fusion, Google Earth Access
Google Earth Box, ...) QGIS

to view data when exporting

attributes in Google without briefcase
Details Earth, better to sub-forms not
import the csv than exported (only url
the kml reference given)

Anonymous possible; depends on no, always requires no, always requires no, registering the no, always requires possible (since version
User Rights

possible possible in web form

submission server setup authentication authentication phone is required authentication 1.9)

5 (admin, app editor,

4 (collect, view, 5 (view, submit, 4 (viewer, user,
Predefined roles/ 3 (view, edit, 4 (view, view & billing admin, field 3 (form admin, data
manage form, site create form, admin no, only "owner" publisher, viewer in
permissions submit) submit, submit, edit) implementer, read- manager, collector)
admin) user, full admin) ArcGIS Online)
only)

info@cartong.org | http://www.cartong.org Page 51 | 66

 Benchmarking of Mobile Data Collection Solutions

KoBoToolbox ODK Aggregate ONA SurveyCTO CommCare Device Magic Magpi Survey123

a) Aggregate v
1.4.13 on Tomcat
Tested version Ubuntu
ODK Collect v1.4.14 v1.3.25 V2.20 v 2.32.1 v 1.72.0 Magpi DataDyne 5.4.8 v 1.10.25
b) Aggregate
v1.4.13 on Google
App Engine

yes, very detailed

options that can be
Custom yes defined in ArcGIS
chosen separately
permissions or no no no no no no Online e.g. to permit
including sending
role creation editing on server
messages and
checking errors

yes, on group basis. yes, on group basis.

yes, on form basis
(forms are shared (forms are shared
Access per role (except anonymous yes, on project basis
no, only full account no, only full access yes, on form basis with a group and its yes, on form basis with a group and its
limitations submission, on or account basis
members have members have
account basis)
access) access)

users are not added ONA allows adding

users are not added to
to an account. collaborators who
an account. Instead,
Instead, forms are have their own ONA
Details forms are shared with
shared with users account. Therefore
users who have their
who have their own the options are quite
own accounts
accounts flexible.

Phone
possible possible possible possible no no no
encryption
Server yes, encrypted on
possible possible possible possible no no no
encryption the server

Compliance with HIPAA for Advanced

Security

security and Enterprise plans FISMA low

standards (not for Pro or lower)

depending on install
HTTPS for transfer https://wiki.commca
possible to install on
has to be set up rehq.org/display/co
Details your own server for
manually (Tomcat), mmcarepublic/De-
increased security
install on own server Identify+Data
possible
possible: allows
Additional details

sending data to CTO

Sync (similar to
possible but would
possible but still in Briefcase). Data can
possible with prebuilt need development,
Offline setup beta, requires high no be exported into csv. no no no
VM i.e. use code from
IT skills Requires paid
Github
account. Web
interface necessary
to manage surveys.

info@cartong.org | http://www.cartong.org Page 52 | 66

 Benchmarking of Mobile Data Collection Solutions

KoBoToolbox ODK Aggregate ONA SurveyCTO CommCare Device Magic Magpi Survey123

a) Aggregate v
1.4.13 on Tomcat
Tested version Ubuntu
ODK Collect v1.4.14 v1.3.25 V2.20 v 2.32.1 v 1.72.0 Magpi DataDyne 5.4.8 v 1.10.25
b) Aggregate
v1.4.13 on Google
App Engine
yes: hosting of
yes: for aggregate costs
yes but very limited
humanitarian depending on yes: limited forms yes: limited forms, yes: limited features
Free plan features and limited no, only 15 days trial no
organisations - no bandwidth and size. and features data and features and forms
users
limits For App Engine, free
quota available

pay for number of

free solution but pay for number of pay for number of pay per user (ArcGIS
free for humanitarian forms/submissions
Pricing scheme potential charges for projects/forms & pay for features pay per user forms and many form Organization), not-for-
organisations and advanced
hosting server advanced features feature profit discount
features

only limited
GeoNet forum & blog,
documentation,
active Google Group, Google user group, other support depends
email support, email support, ODK Skype, phone, on-
Help & support website and email active Google Group dedicated support for forum, email support on AGOL account,
Google Groups Forum/Google Group ground-support
support Pro users dedicated support for
available (not clear
Enterprise users
about extra costs)

info@cartong.org | http://www.cartong.org Page 53 | 66

 Benchmarking of Mobile Data Collection Solutions

E. Table 2 – Other solutions

Akvo Flow Dharma Fulcrum Iformbuilder Mobenzi Researcher Poimapper Traxilo

Flow 2.2.9 on
Tested version App: 6.9.11.266.
Dashboard version v0.47 v.2.22.1 v5.5.1-i Poimapper Plus 1.6.5m Mobile version 1.5.5
Dashboard:10.0.2.3938
v1.9.11.1
few: no custom
constraints, hiding
few: no photo,
calculation questions,
signature, external list,
most but no external lists. Only text most but no reference to most, but GPS buggy. most but input cannot
most, but no hidden no calculations, skip
Question types calculations, notes, input can be defined as previous questions, no No hints, limited be referenced in a label,
calculation questions patterns, group
external lists required questions, no custom rules/constraints constraints no hints, no signature
questions, barcodes,
calculations, no
structured date type
signature / barcode /
picture
all. Uses create and most:
all (identifies by user all (identified by user all (identified by user most:
Metadata update time, not depends on output (json all
not device) not device) not device) no device ID
submission time. best)
Form features

language/ limited, Latin characters

all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8) all (UTF-8)
charset support only
only through a
workaround: languages
Switch are managed through create different views create different views
no, requires creating a
language in yes no the phone’s settings for different languages for different languages no
new survey
form (must change phone but link is kept but link is kept
language to change
form’s language)
yes, full case
yes, full longitudinal limited for some data
management yes, keeps history of yes, keeps history of yes, full beneficiary
Monitoring study, case types but possible no
(longitudinal study) edits edits tracking
management options through "smart tables"
options

The app can be set to

metadata exported to a
Details English or French (but
different tab/sheet
not the forms itself)

no (only filter by advanced: but

no (data can only be advanced: filter or no (data can only be advanced: filter any
Filters metadata: status, statements need to be advanced
sorted) cross-tabulation sorted) field type and input
project, enumerator,..) written as text input
Server features

Charts & basic: can't filter or advanced: based on yes, based on filter and yes based on fields and
basic yes, based on filters no
reporting customise fields and filters fields (pro account) filters

basic: only one point at

basic: only as "staff a time (as part of the basic: markers on basic: based on filter,
Map basic: no layout options basic basic
tracking" points on map details when clicking on survey location choose to cluster or not
a record)

View images no (no picture support) yes yes Yes yes (no picture support)

possible, but not the

Edit no, but on phone possible possible possible possible possible
GPS coordinates

info@cartong.org | http://www.cartong.org Page 54 | 66

 Benchmarking of Mobile Data Collection Solutions

Akvo Flow Dharma Fulcrum Iformbuilder Mobenzi Researcher Poimapper Traxilo

Flow 2.2.9 on
Tested version App: 6.9.11.266.
Dashboard version v0.47 v.2.22.1 v5.5.1-i Poimapper Plus 1.6.5m Mobile version 1.5.5
Dashboard:10.0.2.3938
v1.9.11.1

Delete possible no, but on phone possible possible possible possible possible

Languages EN,FR, ES, PT, VN, ID EN EN EN EN EN, FR, ES, DE, FI EN

Details

csv/xls, json, kml,

Data export
csv/xls csv/xls sqlite, shp, gdb, sql csv/xls, json, xml, atom csv/xls, stat/transfer csv/xls, kml, SPSS, doc csv/xls
format
(postgis)

possible: xml (but not

Form upload not possible no no possible possible no
xform standards)

possible: advanced
possible but not for GPS
options for matching possible but in beta (not possible (but skip-logic
Data upload points and not to add no possible: xls no
columns, indicating tested) not supported)
new points to monitor
conflicts, ...

flexible with four

defined: related table
options: flat (repeat as
(either export only sub-
Group/sub-form additional row or
Export & Import

defined: flat table defined: flat table defined: related table form or as part of main n/a
exporting column), related (each
form but never flat main
repeat group extra or all
table)
repeats in one)

Choice defined: each choice one defined: each choice one defined: each choice one
defined: all in one cell defined: all in one cell defined: all in one cell defined: all in one cell
exporting column column column

embedded in xls table, csv: image info is lost

Picture export Link: full url in csv table (no picture support) Link: full url in table Link: full url in table but lost in csv, download xls: choice to download (no picture support)
one by one possible media as .zip

Briefcase
no no no no no no no
compatible

yes: query, read,

yes: view only, uses yes: view only
yes: DJANGO REST API, yes: view, query update, add
HTTP https://iformbuilder.zen yes, but not part of
contact customer http://developer.fulcrum http://help.mobenzi.co
API https://github.com/akvo desk.com/hc/en- documentation, contact no
support not part of app.com/query- m/article/AA-
/akvo-flow/wiki/Akvo- us/articles/205353910- support
existing documentation api/intro/ 00431/40/Guides/API/A
FLOW-API API-6-0-Documentation
PI-Overview.html

Easy integration Excel, QGIS, ArcGIS, Excel, SPSS, Google

Excel Excel Excel Excel, SAS, Stata Excel
with Google Earth Earth

info@cartong.org | http://www.cartong.org Page 55 | 66

 Benchmarking of Mobile Data Collection Solutions

Akvo Flow Dharma Fulcrum Iformbuilder Mobenzi Researcher Poimapper Traxilo

Flow 2.2.9 on
Tested version App: 6.9.11.266.
Dashboard version v0.47 v.2.22.1 v5.5.1-i Poimapper Plus 1.6.5m Mobile version 1.5.5
Dashboard:10.0.2.3938
v1.9.11.1
sub-form export has
Details empty rows if there is
no "child" to parent data

info@cartong.org | http://www.cartong.org Page 56 | 66

 Benchmarking of Mobile Data Collection Solutions

Annexes

1. List of All Solutions

The following table lists all solutions which were researched in a pre-benchmarking effort. Not
all of them were retained for this benchmarking, reasons for exclusion listed in the last column.

Tool Included URL Reason for exclusion

Akvo Flow yes http://akvo.org

http://www.dimagi.com/
yes
Commcare products/

https://www.devicemagi
yes
DeviceMagic c.com/

http://dharmaplatform.c
yes
Dharma om/

http://plus.epicollect.net
no
EpiCollect / Still in beta

Formhub no https://formhub.org/ No longer maintained

No considerable track record

no
Formitize http://formitize.com/en/ with humanitarian orgs

http://www.fulcrumapp.
yes
Fulcrum com/

https://www.goformz.co No considerable track record

no
GoFormz m/ with humanitarian orgs

https://www.zerionsoft
yes
iFormbuilder ware.com/iformbuilder/

http://www.kobotoolbox
yes
Kobo online .org/

Magpi yes https://magpi.com/

http://www.mobenzi.co
yes
Mobenzi Researcher m/researcher/Home

WASH specific initially, although

no
mWater http://www.mwater.co/ this is changing

ODK with Aggregate on Tomcat

online (Amazon web services yes
install) https://opendatakit.org/

ODK with Aggregate online on yes https://opendatakit.org/

App Engine updated instance

info@cartong.org | http://www.cartong.org Page 57 | 66

 Benchmarking of Mobile Data Collection Solutions

with Aggregate: 1.4.7; released

7 May 2015

Ona yes https://ona.io/home

http://www.poimapper.c
yes
Poimapper om/en/

http://www.getpushfor No considerable track record

no
Pushforms ms.com/ with humanitarian orgs

https://socialcops.com/c
no
Socialcops Collect ollect/ Used in limited context so far.

http://survey123.esri.co
yes
Survey 123 (ESRI) m/

No considerable track record

with humanitarian orgs but
no
rather with universities and
SurveyBe http://surveybe.com/ donors

http://www.surveycto.co
yes
SurveyCTO m/

TaroWorks no https://taroworks.org/ No demo in time to test

http://www.trackvia.co No considerable track record

no
TrackVia m/ with humanitarian orgs

Traxilo yes https://traxilo.com/

2. MDC Technical Requirements Checklist

All platforms were tested against the same criteria and assessed for the same features and
functions. To make it comparable, a ranking system was set up for easier comparison.

Question Check for & example Ranking

Simple fields: integer, double, free text

Date fields (not a string)

Form features

Picture/ photo and signature

all: 14
Question GPS coordinates (points)
most: 8-13
type
Select: multiple & single few: < 8

Hints & Notes

Calculations including based on previous answers: age based

on today’s date and date of birth

info@cartong.org | http://www.cartong.org Page 58 | 66

 Benchmarking of Mobile Data Collection Solutions

Simple constraints: answer must lie between 1-10

Custom constraints: e.g. for multiple choice answer a and b

accepted together but not a and c

Skip logic: skip question(s) if the previous answer was no

Hide questions: A calculation field is used but not shown to

the enumerator

Require questions: form can't be submitted without an

answer in this field

External lists: use a csv list to fill in multiple-choice options

Recurring group, "child" questionnaire, sub-form: for each

child in household answer certain questions (number varies
per hh)

Unique ID for submission

User or device identifier

Submission date and time all: 7

Metadata Start/end time of a survey most: 4-6

User or device identifier: user name or device ID few: <4

Submission date and time

Start/end time of a survey

Supports UTF-8 encoding, incl. right-to-left and non-Latin all

Language &
charset
Allows only left-to-right languages or Latin characters limited

Switch language in form: Enumerator switches from EN to

good
AR if question not understood by informant

Switch language on phone: Enumerator can choose to open

Switch
the survey in EN or AR before each interview. All languages ok
language
stored and analysed together on server

One language - one form: Each language in a separate form

no
not analysed together on server

Monitoring OR editing on the phone keeping track of edits or

yes
link different submissions to the same parent
Monitoring
Neither monitoring nor editing on phone no

Filter by any field, allow several filters at a time advanced

features
Server

Filters Filter by some fields, only one at a time basic

No filters on question fields no

info@cartong.org | http://www.cartong.org Page 59 | 66

 Benchmarking of Mobile Data Collection Solutions

Takes filters into account when creating charts advanced

charts &
Only very simple pie/bar charts no filtering on chart basic
graphs
No charts on question fields no

Customise map using filters, categories,… advanced

map Only display of GPS points basic

No map available no

Images accessible on server yes

view images
Images NOT accessible on the server no

Edit any answer on server yes

Edit Editing possible with the exception of certain fields partly

No editing possible on the server no

Deleting is possible yes

Delete
No deleting no

Languages List language options for server interface

Export
Lists formats to export data
format

Forms upload if formatted correctly yes

Form upload
Forms can NOT be uploaded no

Any records in correct format can be uploaded (except

possible
images)
Data upload
Upload possible but with limitations partly

No option to upload data no

Various options: as flat table (all entries in one table each

flexible
Group child one line) or related (child in separate table)
exporting
One option only pre-defined

Various options: all in one cell separated by comma or each

flexible
Choice choice one column
exporting
One option only pre-defined

info@cartong.org | http://www.cartong.org Page 60 | 66

 Benchmarking of Mobile Data Collection Solutions

Picture Details how pictures are downloaded and link between table entry and
export image file kept

Possible to push and pull data with Briefcase yes

Briefcase
compatible
NOT possible to push and pull data with Briefcase no

An API is provided for accessing the data and documentation

yes
available
API
NO API is provided no

Easy Lists tools with which the data can easily integrate based on export formats
integration or available tutorials

Possible and avoidable yes

Anonymous
Possible but has limitations partly
access
Never possible no
User management

Predefined
Lists predefined roles or permissions
roles

Allows creating custom roles yes

Custom
roles
Not possible to create custom roles no

Access per Possible to limit access to certain forms/projects/… yes

role
limitations NOT possible to limit access no

Data is (or can be) stored encrypted on the phone yes

Phone
encryption Data is NEVER encrypted on phone (other than phone
no
encryption)
Security

Data is (or can be) stored encrypted on the server yes

Server
encryption
Data is NEVER encrypted on server no

Compliance
with Lists compliance such as FISMA low/high, HIPAA,…
standards

Possible to have a workflow without any connectivity or

yes
behind the organizations firewall
Additional

Offline
details

setup
NOT possible to have a workflow without any connectivity or
no
behind the organizations firewall

Free plan A free plan (other than trial) is available yes

info@cartong.org | http://www.cartong.org Page 61 | 66

 Benchmarking of Mobile Data Collection Solutions

NO free plan is available no

Business
Details on pricing scheme: per user/submission/…
model

Help &
Detail ways to get help and support
Support

info@cartong.org | http://www.cartong.org Page 62 | 66

 Benchmarking of Mobile Data Collection Solutions

3. What are The Advantages to Using ODK-Based Technologies for

NGO/IOs?
As many humanitarian organizations have opted to use ODK, there is sufficient technical
knowledge within the sector to exchange sector-specific experience, code and standard
operating procedures within the humanitarian community of practice. This can save time and
avoid duplicating efforts.

Since ODK-based solutions share the same core architecture, some tools inside the ODK suite
can be used across different products, which makes it easier to switch from one platform to
another if necessary. Examples include the Briefcase application to download all data, including
media files or Enketo, a tool to enter and edit data through a web interface.

While Open Source tools as such do not incur costs on software (though potentially on hosting),
it strongly depends on internal resources and the availability to build and retain the
competencies internally on whether or not they are really cost effective. Certainly, key
considerations are the aspect highlighted above, how active are the developers, how active the
user community and how long has a solution been around. Because ODK has been around for
a while, the number of efficient users increases steadily and since easier ways of coding have
been conceived as well as carefree hosting services which do not require any IT skills, chances
are good that a respective organization will find or already has internal resources to support
surveys built on ODK/related tools. This then leads to cost effective deployments and
guarantees a certain sustainability.

The graph below highlights the interoperability between the closely related ODK tools. It is
usually possible to build a form which can be used with any of the mobile apps or through a
web form accessed through computer. The data can then be sent to the web platform of choice
and analysed there if possible or exported and used in any external statistical packages.

info@cartong.org | http://www.cartong.org Page 63 | 66

 Benchmarking of Mobile Data Collection Solutions

info@cartong.org | http://www.cartong.org Page 64 | 66

 Benchmarking of Mobile Data Collection Solutions

4. Resources

Assessment of Digital Data Collection Apps to Support ACIAR’s M&E,

http://aciar.gov.au/publication/fr2016-03, Stuart Higgins, Australian Centre for
International Agricultural Research, November 2015, Retrieved: 6 December 2016

Choosing the Right Tool for Data Collection: Paper vs. Digital Tools vs. IVR,
https://blog.socialcops.com/academy/resources/choosing-right-tool-data-collection-
digital-tools-paper-digital-ivr/, Gaurav Jha, SocialCops, Retrieved: 6 December 2016

Collecte Mobile de Données,

http://odk.reliefapps.org/documents/rapport_etude_odk_crf.pdf, French Red Cross,
January 2015, Retrieved: 13 December 2016

A Comparative Analysis of Traditional and Digital Data Collection Methods in

Social Research in LDCs - Case Studies Exploring Implications for Participation,
Empowerment, and (mis)Understandings, http://www.validnutrition.org/wp-
content/uploads/2015/03/A-Comparative-Analysis-of-Traditional-and-Digital-Data-
Collection-Methods.pdf Gretta Fitzgerald and Mike Fitz Gibbon, Preprints of the 19th
World Congress The International Federation of Automatic Control Cape Town, South
Africa, 24 - 29 August 2014, Retrieved: 6 December 2016

Digital Data Collection in Plan: A Review of Current Practice and Lessons

Learned.
http://www.ictworks.org/wp-content/uploads/2016/03/Digital-Data-Collection-Plan.pdf,
October 2015, Erica Packington and Hannah Beardon, Plan International Finland,
Retrieved: 6 December 2016

Digital Data Collection Vs. Data Collection on Paper,

http://www.developmentoutlook.org/2012/07/digital-data-collection-vs-data.html,
Development Outlook, Retrieved: 6 December 2016

Electronic Versus Paper-Based Data collection: Reviewing the Debate

http://blogs.worldbank.org/impactevaluations/electronic-versus-paper-based-data-
collection-reviewing-debate, Sacha Dray, Felipe Dunsch, and Marcus Holmlund, Wold
Bank, 25 May 2016, Retrieved: 6 December 2016

Going Digital - Using Digital Technology to Conduct Oxfam’s Effectiveness

Reviews,
http://oxfamilibrary.openrepository.com/oxfam/bitstream/10546/578816/4/cs-going-
digital-effectiveness-reviews-290915-en.pdf, Emily Tomkys and Simone Lombardini,
Oxfam GB, September 2015, Retrieved: 6 December 2016

info@cartong.org | http://www.cartong.org Page 65 | 66

 Benchmarking of Mobile Data Collection Solutions

Improving Consumption Measurement and other Survey Data through CAPI:

Evidence from a Randomized Experiment, Caeyers et al., Journal of Development
Economics DOI: 10.1016/j.jdeveco.2011.12.001, December 2011

Mobile Survey Toolkit, http://policy-practice.oxfam.org.uk/publications/mobile-survey-

toolkit-617456, Emily Tomkys and Laura Eldon, Oxfam GB, 27 July 2016, Retrieved: 6
December 2016

Paper-to-Mobile Data Collection: A Manual, U.S. Global Development Lab,

https://www.fhi360.org/sites/default/files/media/documents/Paper_to_Mobile_Data_Coll
ection_Manual_1.0.pdf, Erin Satterlee, Leela McCullough, Michael Dawson, and Kelly
Cheung, 2015, Retrieved: 13 December 2016

Technologies for Monitoring in Insecure Environments,

http://www.gppi.net/fileadmin/user_upload/media/pub/2016/savE__2016__Toolkit_on_
Technologies_for_Monitoring_in_Insecure_Environments.pdf, GPPi / Humanitarian
Outcomes, September 2016, Retrieved: 6 December 2016

info@cartong.org | http://www.cartong.org Page 66 | 66