MicroVote General Corporation Quality Assurance Audit

September 8-9, 2014

Indianapolis, Indiana

Conducted by: United States Election Assistance Commission

Testing and Certification Division
1335 East-West Highway
Ste. 4300
Silver Spring, MD

December 15, 2014

 United States Election Assistance Commission – Testing and Certification Division 2014

Executive Summary
MicroVote General Corporation (MicroVote) is an EAC registered voting system manufacturer
based in the Indianapolis, Indiana metropolitan area and has been registered with the U.S.
Election Assistance Commission (EAC) since 2007. On September 8-9, 2014, the Election
Assistance Commission conducted a quality assurance audit of MicroVote in order to collect
sufficient data to assess the manufacturer’s quality systems and their compliance with the
quality assurance requirements of the EAC certification program and the 2005 Voluntary Voting
System Guidelines.

The quality assurance audit found that while MicroVote had very recently developed and
incorporated a quality assurance manual into their company processes, the manual needs to be
strengthened, fully implemented with the backing and support of senior management and
backed up with internal procedures that would allow independent auditors to determine if
MicroVote is actually meeting their stated quality goals.

This report provides six specific recommendations for MicroVote in order for the company to
improve overall quality management and quality assurance and to bring their current process
more in line with the intention of the requirements of VVSG Section 8. These
recommendations are detailed in the Audit Recommendations Section of this report.

1 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

Introduction
On September 8-9, 2014, the Election Assistance Commission conducted a quality assurance
audit of MicroVote at the company headquarters in Indianapolis, Indiana. Participating in the
audit for EAC were Brian Hancock, Director, Testing and Certification, Jessica Myers, Program
Specialist, Mark Skall, Technical Reviewer and Tom Caddy, Technical Reviewer.

The quality assurance audit was performed pursuant to Section 2.3.1.4 and Section 8 of the EAC
Testing and Certification Program Manual as well as Section 8 of the 2005 Voluntary Voting
System Guidelines (VVSG).

This report, along with the attached appendices, documents the audit findings, conclusions and
recommendations and will be forwarded to MicroVote and included as an attachment to the
MicroVote EMS 4.1 Test Report in order to assist the manufacturer with meeting the
requirements of VVSG Section 8 and improving their overall operations and quality control.

Purpose
The EAC conducted the audit because of ongoing questions about the quality assurance
practices of the manufacturer based on concerns noted by NTS Laboratories and previous EAC
experience on MicroVote test campaigns.

Scope
This audit was conducted in order to collect sufficient data to assess the manufacturer’s quality
systems, their compliance with the quality assurance requirements of Section 8 and the
configuration management requirements of Section 9, Volume 1 of the 2005 Voluntary Voting
System Guidelines (VVSG), and to compare MicroVote quality practices to IT industry standard
QA practices.

To accomplish the audit objectives noted above, the EAC:

 Met with MicroVote management and senior staff as well as the management and staff
of Carson Manufacturing (Carson).
 Listened to briefings and participated in discussions regarding MicroVote management
and quality systems.
 Reviewed documentation related to the MicroVote QA system.
 Use EAC Quality Audit Checklist (Appendix A) to determine compliance.

2 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

Why Conduct a Quality Audit?

Quality assurance is often defined as a process-centered approach to ensuring that a company
or organization is providing the best possible products or services to its customers. Quality
assurance focuses on enhancing and improving the process that is used to create the product,
rather than focusing on the product itself. Among the parts of the process that are considered
in QA are planning, design, development, production and service.

Quality assurance demands a degree of detail in order to be fully implemented at every step.
Planning, for example, could include determining specific levels of quality or measurable results
that the organization wants to achieve. Checking could involve testing and other objective
measurements to determine whether the goals were met, rather than mere subjective
evaluation of quality. Acting could mean a total revision in the manufacturing process to correct
a technical or cosmetic flaw or very small changes to improve efficiency or accuracy.

Quality assurance verifies that any product, regardless whether it is new or modified, is
produced and offered with the best possible materials, in the most comprehensive way and
with the highest standards. Quality assurance provides the mechanism to exceed customer
expectations in a measurable and accountable process.

ISO 9000 is a family of standards published by ISO, the International Organization for
Standardization, related to quality management systems and designed to help organizations
ensure that they meet the needs of customers and other stakeholders while meeting statutory
and regulatory requirements related to the product. ISO 9001 is a global quality management
standard dealing with the requirements that organizations wishing to meet the standard must
fulfill. As of 2011, more than a million organizations worldwide were certified to the ISO 9001
standard. While not a panacea for every quality related problem an organization may face, the
principles of ISO 9001 have been the guiding force in organizational quality since the early
1990’s.

ISO 9001 defines quality as something that can be determined by comparing a set of inherent
characteristics with a set of requirements. If those inherent characteristics meet all
requirements, high or excellent quality is achieved. If those characteristics do not meet all
requirements, a low or poor level of quality is achieved. Quality assurance (QA) is defined as a
set of activities intended to establish confidence that quality requirements will be met. QA is
one part of quality management. A quality management system (QMS) is a set of interrelated
or interacting elements that organizations use to direct and control how quality policies are
implemented and quality objectives are achieved.

Finally, a quality manual documents an organization's quality management system (QMS). It

can be a paper manual or an electronic manual. According to ISO 9001 section 4.2.2, a quality
manual should:

 Define the scope of your QMS.

o Explain reductions in the scope of your QMS.

3 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

o Justify all exclusions (reductions in scope).

 Describe how your QMS processes interact.
 Document your quality procedures or refer to them.

While not requiring ISO 9001 certification from voting system manufacturers, the 2005 VVSG
recognizes the importance of quality assurance in voting systems with the specific requirements
related to quality contained in Section 8.

VVSG Section 8.1 defines the scope of QA.

“Quality assurance provides continuous confirmation that a voting system conforms with
the Guidelines and to the requirements of state and local jurisdictions. Quality assurance is
a vendor function that is initiated prior to system development and continues throughout
the maintenance life cycle of the voting system. (Emphasis added) Quality assurance
focuses on building quality into a voting system and reducing dependence on system tests at
the end of the life cycle to detect deficiencies, thus helping ensure the system:
• Meets stated requirements and objectives
• Adheres to established standards and conventions
• Functions consistently with related components and meets dependencies for use
within the jurisdiction
• Reflects all changes approved during its initial development, internal testing, national
certification, and, if applicable, state certification processes.”

VVSG Section 8.2 defines the general requirements for quality assurance:

“The voting system vendor is responsible for designing and implementing a quality
assurance program to ensure that the design, workmanship, and performance requirements
are achieved in all delivered systems and components. At a minimum, this program shall:

a. Include procedures for specifying, procuring, inspecting, accepting, and

controlling parts and raw materials of the requisite quality
b. Require the documentation of the hardware and software development
process
c. Identify and enforce all requirements for:
i. In-process inspection and testing that the manufacturer deems
necessary to ensure proper fabrication and assembly of hardware
ii. Installation and operation of software and firmware
d. Include plans and procedures for post-production environmental screening and
acceptance testing
e. Include a procedure for maintaining all data and records required to document
and verify the quality inspections and tests.”

Because determinations of quality can often be subjective, the EAC uses the Quality Audit
Checklist to focus auditors and to provide a general basis for determining if the

4 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

manufacturer meets the quality requirements of the VVSG and the general principles of
quality outlined in ISO 9001.

Current MicroVote Quality Assurance Processes

The formal QA process at MicroVote is essentially a new development triggered by the EAC
Quality Audit notification and some issues related to testing the MicroVote QA/CM process as
required by in the VVSG for the current MicroVote test campaign. This statement is reflective of
the fact that the MicroVote Quality Assurance Manual (Version 1.0 & 1.1) are dated September
1st and 2nd, 2014. In addition, the MicroVote Training Manual (Version 1.0) is dated August 25,
2014, and the one and only staff training session related to the new quality manual and process
was held on September 2, 2014 from 10:00am until approximately 12:40pm.

MicroVote is a small company with approximately 15 full time equivalent employees, including
the Chief Executive Officer and President. Additional temporary staff is hired prior to each
election and may or may not receive updated voting system/election training from MicroVote.
Source code change control is managed via a Microsoft Team Foundation Server. Commercial
Off-the-Shelf (COTS) component and end-of-life (EOL) components as well as supplies and
vendors are managed by the MicroVote Chief Operating Officer. Internal testing is conducted
under the QA department who develop their own test cases. It was unclear how validation of
these test cases was performed by MicroVote. Hardware EMC testing has just recently (3-4
months) been contracted out to Technicolor Laboratory Service, located in the Indianapolis
area.

A unique and potentially problematic scenario exists because MicroVote owns the Intellectual
Property (IP) rights to their software, while the IP rights to the hardware that runs the voting
system is owned by Bill Carson, Technical Support Specialist of Carson Manufacturing Company,
Inc. (Carson). Carson is the sole manufacturing facility for the MicroVote system and, like
MicroVote, is located in the Indianapolis metropolitan area. . This relationship led the EAC audit
team to also review the Carson quality process at their facilities on September 9, 2014. Because
no voting system products were being manufactured by Carson at the time of the EAC visit, EAC
audit team concentrated on a review of the Carson quality program.

The EAC audit team met with Carson staff, reviewed documentation related to the quality
system and toured the manufacturing area of the facility. The Carson Quality Manual reviewed
by EAC was Revision C, dated November 1st, 2011. No updates or revisions have been
incorporated since that time. No separate quality unit or department exists at Carson. Currently
no quality manager has been designated, and currently, all quality functions default to the
company Chief Executive Officer (Barbara Ferguson, President and CEO).

5 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

Carson in general appears to have a very minimal quality program.

Audit Results
This section details the results of the quality audit by highlighting findings noted by the audit
team in their quality audit checklist. The EAC quality audit checklist contains five major sections
covering:

 Organizational Quality Management System

 Product Design and Development
 Pre-Production Design and Development Testing
 Identify and Control Nonconforming Products
 Labeling

1. Organizational Quality Management System

EAC auditors conclude that MicroVote has not operated under any corporate quality policy
prior to the EAC audit, since the initial version of their quality manual was dated just days
before the EAC audit team arrived in Indianapolis. MicroVote has no separate division of
quality responsibilities but instead splits QA duties between the Director of Software
Development, who is also Director of QA, and the Director of Customer Service, who is also
the QA Manager. Since no separate QA unit exists at MicroVote, the EAC was told that QA is
reviewed by the division directors who ultimately report to the Chief Executive Officer or
Chief Operating Officer. This bifurcation of responsibility can lead to questions related to
who has the ultimate authority over QA decisions as well as providing the potential for
issues to slip through gaps between the two individuals if responsibilities are not clearly and
expressly defined. The MicroVote Quality Manual does not include a process for the review
of production records and no records or documentation was provided to the EAC.

The MicroVote individuals currently responsible for QA duties have no claimed or

documented training in QA processes (Either ISO 9001 or otherwise), and because the QA
Manual and associated processes are new, no formal internal audits have been conducted
and only one short and minimal training session was claimed and documented for
MicroVote staff to be informed of the existence of the Quality Manual.

2. Product Design and Development

The MicroVote Quality Manual does not contain a design review policy statement. This
would obviously preclude any systematic internal review or audit of design concepts as well
as any systematic method to determine if design objectives are being achieved. Because

6 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

formal design reviews are not a part of the MicroVote process, record keeping for this
function is not applicable at this time. General review of product functionality is a joint
effort between the Director of Software Development and the Director of Customer Service.
Lack of a design review policy may be partially related to the fact that the MicroVote
product has remained essentially unchanged at least since its initial certification efforts as
part of the NASED program in 2006. Any design review done is therefore ad hoc in nature.
MicroVote noted that they may develop a formal product design and review policy if the
EAC recommends such an activity.

3. Pre-Production Design and Development Testing

Some internal design and developmental as well as general functional testing is done on the
MicroVote software by the Director of Software Development. This individual develops his
own test cases and it was unclear how these test cases were validated, or even if the test
cases were validated. Because of this limited process, no ongoing formal test cases appear
to exist which brings into question the repeatability of such test cases.

4. Identifying and Controlling Nonconforming Products

Section 12.0 and 13.0 of the MicroVote Quality Assurance Manual outline general
requirements for the Control of Nonconforming Articles and Corrective Actions.
Unfortunately it appears that MicroVote has no accompanying procedures to enable the
company to consistently incorporate these requirements into product development,
product testing or product delivery.

Section 13.0 notes that “In the case of significant [Emphasis added] conditions adverse to
quality, the root cause of the condition shall be determined and action planned to correct
and preclude repetition.” As with the rest of this document, no detailed process for
implementing a root cause analysis was defined leaving open questions of the consistency
and quality of root cause analysis. In addition, MicroVote should clearly define terms used
in this section such as “significant” in order to be clear on what level of system anomaly
would initiate root cause analysis.

Customers are notified of non-conforming products either via email, telephone or by mail
through the United States Postal Service.

5. Labeling
The EAC auditors were unable to find detailed policies and procedure related to product
labeling and the EAC Mark of Certification although Section 9.0 of the Quality Manual states
that: “Labeling shall be verified during customer acceptance testing.” This is surprising given
that the EAC has provided the basis for such procedures in EAC Notice of Clarification (NOC)
2008-002 EAC Mark of Certification available on the EAC web site.

7 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

Audit Recommendations
In consideration of the findings outlined in this audit report, the EAC recommends MicroVote
take the following six (6) steps to improve overall quality management and quality assurance
and to bring their current process more in line with the intention of the requirements of VVSG
Section 8:

1. While formal ISO 9001 certification is not recommended at this time, the EAC does
recommend that MicroVote develop a formalized organizational quality management
system based on the principles of ISO 9001. Quality management is defined as all
activities carried out by the organization to direct, control and coordinate quality. The
activities should at a minimum include formulating a quality policy, and setting quality
objectives. This recommendation can best be met by one or more MicroVote staff
members receiving formal training in ISO 9001 concepts via one of the numerous
commercial ISO training organizations.

2. Augment and fully implement the new organizational quality manual. The quality
manual documents an organization's quality management system (QMS) and should:

 Define the scope of the QMS.

o Justify all exclusions (reductions in scope).
 Describe how your QMS processes interact.
 Document your quality procedures.

3. Conduct regularly scheduled internal quality audits in order to monitor and measure
your QMS, document any nonconforming procedures or products and perform
corrective action to improve the nonconforming process or product.

4. Develop a systematic process for the review of new product design and design changes
to already developed products. This process should include specific measurements to
determine if design objectives are being met as well as a system of maintaining records
of all design reviews.

5. Develop and implement detailed root cause analysis procedures to satisfy the
requirements of Section 13.0 of the MicroVote Quality Manual.

6. Undergo another EAC quality assurance audit within one year of the date of this report
to allow the EAC to assess MicroVote progress in meeting the recommendations of this
audit.

Although the above recommendations are purely voluntary, the EAC strongly suggests that
MicroVote implement the recommendations for the following reasons:

8 MicroVote QA Audit Report

 United States Election Assistance Commission – Testing and Certification Division 2014

 ISO 9001 is the standard best practice specification for QMS in use worldwide. ISO 9001
has a track record of saving money, streamlining operations and reducing waste, and
increasing customer satisfaction.
 As the EAC moves towards improving and truncating the certification process through
the use of Manufacturer Declaration of Conformity (DoC), it is likely that ISO 9001
certification will eventually become a requirement for all EAC registered manufacturers
in order to provide some additional assurance to the DoC and ultimately, to MicroVote
customers.

The EAC requests that an initial written response to this report be submitted within 45 days of
the receipt date of this document.

9 MicroVote QA Audit Report

 Manufacturer Quality Audit Date: 9/8/14 – 9/9/14 Appenix A
Microvote Reviewer:

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

1 Organizational Quality
Management System
1.1 Does this organization operate under a
corporate quality policy?

1.2 Does a Quality Assurance unit

(department) exist as a separate
organizational entity?

13 Does the Quality Assurance unit alone

have both the authority and responsibility
to approve or reject all designs, parts,
components, and finished products?

1.4 Does the QA department routinely review

production records to ensure that
procedures were followed and properly
documented?

1.5 Does the organization have a documented

Quality Assurance program? (Quality
Manual)
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

1.6 Does the Quality Manual contain (at

minimum)

 A quality policy statement,

including objectives and
commitments by executive
management?
 The reporting relationship
between management, technical
operations, production, support
and the quality system?
 The organizations general scope
of product inspection and
testing?
 Appropriate and clear reference
to inspection verification and
test procedures to be used?
 Reference to any procedures for
inspection, calibration and
maintenance of test equipment?
 Procedures for handling non-
conforming materials and
products?


1.7 Does the Quality Manual provide means

for finished products to be traced back to
the production and quality control
records at the manufacturing facilities?

2
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

1.8 Is the Quality Manual reviewed and/or

revised at planned intervals?

1.9 Has the organization implemented and

maintained document and data control
procedures for the Quality
Manual/Quality System?

1.10 Are the procedures followed? (Examine

records to ensure consistent record-
keeping and documentation.)

1.11 Are QA supervisory personnel qualified by

way of training and experience?

1.12 Is a copy of the Quality Manual readily

available to all employees?

1.13 Is training provided in Quality Assurance

and quality improvement?

1.14 If "yes" to above, when provided?

__________________

1.15 Does a formal auditing function exist in

the Quality Assurance department?

2 Product Design and

Development

3
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

2.1 Do reviews occur at planned stages

throughout the design process?

2.2 Are reviews carried out in a systematic

way involving representatives of all
organizational functions concerned with
the product development?

2.3 When designs change from the original

concept, are revised inputs and outputs
reviewed and approved by the
appropriate authorized individuals?

2.4 Does the output demonstrate the

suitability and conformance to
specifications of the designed product?

2.5 How is it determined if design objectives

are being achieved?

2.6 Are adequate records of design reviews

maintained?

3 Pre-Production Design and

Development Testing

4
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

3.1 Is design and development testing done

to validate that the product complies with
the organizations own performance
requirements and the requirements of
customers and regulators (Federal and
State certification authorities)?

3.2 Are test methods traceable to pertinent

system functional requirements and
applicable Federal or State certification
requirements?

3.3 Are test records/reports maintained and

do they confirm that appropriate testing
has been carried out?

5
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

3.4 Do test records/reports include all

information necessary for the reliable
interpretation of test results?

This information should include at

minimum:

 Descriptive title
 Description and clear
identification of the product
being tested.
 Date of test.
 Identification of the test method
used.
 Clear and unambiguous
description of the results of the
test (Pass/Fail).
 Signature and title of individual
accepting responsibility for the
content of the record/report.

3.5 Do the records for each test contain

sufficient information to permit
repetition?

6
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

3.6 If test records/reports are performed and

compiled by a contractor, that contractor
must be audited by the manufacturer to
ensure that the contractor is qualified to
perform the testing contracted for
(Including having appropriate
accreditation) and meets the
manufacturer’s quality requirements.

3.7 Manufacturer shall maintain and retain all

records of contracted tests.

4 Identify and Control of

Nonconforming Products
4.1 Does the organization define procedures
used to identify, evaluate, and address
any nonconforming product detected by
inspection, customer report/complaint or
Federal or State certification authority
determination?

4.2 Are corrective actions in place to explore

the root cause of the nonconformance
and provide a plan for eliminating the
root cause?

4.3 Is Information related to occurrences of

nonconforming work recorded and
maintained?

7
Manufacturer: Microvote Date: 9/8/14 – 9/9/14

Question QA Reference Evidence (Note any additional evidence or comments may be Yes, No, or NA
included in a supplementary notebook).

4.4 How are customers notified of

nonconforming products and product
upgrades resulting from root cause
analysis and product redesign?

5 Labeling
5.1 What are the organizational policies on
labeling (both marks of certification and
other required labels such as URL)?

5.2 How and when is correct labeling

verified?