Disclaimer:

As a condition to the use of this document and the information contained herein, the Facial Identification
Scientific Working Group (FISWG) requests notification by e-mail before or contemporaneously to the
introduction of this document, or any portion thereof, as a marked exhibit offered for or moved into evidence in
any judicial, administrative, legislative, or adjudicatory hearing or other proceeding (including discovery
proceedings) in the United States or any foreign country. Such notification shall include: 1) the formal name of
the proceeding, including docket number or similar identifier; 2) the name and location of the body conducting
the hearing or proceeding; and 3) the name, mailing address (if available) and contact information of the party
offering or moving the document into evidence. Subsequent to the use of this document in a formal proceeding,
it is requested that FISWG be notified as to its use and the outcome of the proceeding. Notifications should be
sent to: chair@fiswg.org

Redistribution Policy:

FISWG grants permission for redistribution and use of all publicly posted documents created by FISWG,
provided that the following conditions are met:

Redistributions of documents, or parts of documents, must retain the FISWG cover page containing the
disclaimer.

Neither the name of FISWG, nor the names of its contributors, may be used to endorse or promote products
derived from its documents.

Any reference or quote from a FISWG document must include the version number (or creation date) of the
document and mention if the document is in a draft status.

0
 Version 1.0 2013.05.16

Facial Recognition Systems

Methods and Techniques

Facial Recognition System: Methods and Techniques

This document provides a general outline of Methods and Techniques that can be helpful or
considered when planning or operating a Facial Recognition (FR) system. The goal of this
document is to provide guidance on methods and techniques to increase the likelihood of
obtaining a true match in the candidate list for a submitted probe within a 1:N search. Please
refer to the FISWG web site for guidance on human 1:1 comparison processes that may be
required following a 1:N search.

Figure 1 is important to both system administrators and system users as it displays the
relationship and information represented by the system, the flow of search strategies, and the
management of the data in the facial gallery.

Figure 1 – System Flow (post enrollment)

FISWG FR Systems Methods and Techniques 1

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

NOTE: FISWG defines metadata binning as: A technique used by a FR system to organize
enrollment of data to facilitate and optimize searching using filters based on information
associated with an image.1

Target audiences:
System administrators or developers of the FR system.
o System developers are responsible for the overall design of the system features
that allow and enable these methods and techniques.
o System administrators are responsible for verification and proper deployment,
implementation, and usage of these methods and techniques.
System users.
o System users are the examiners, operators, or other personnel who actually
utilize the system for facial searching purposes.

A number of areas are described in overview below. Each area will be covered in depth in future
FISWG standalone specific documents:
1. Metadata system setup and usage
2. Facial gallery management
3. Enrollment of the facial image
4. Search tuning
5. Search strategy and options
6. Image preprocessing
7. Eye locations
8. Metrics reporting auditing

1. Metadata system setup and usage

Metadata is anything associated with but excluding the facial image and can include a
system generated identity number for the person and/or gender, image header information,
age, race, scars, marks, tattoos, ethnicity, etc.

Metadata usage can be broken down into two main areas: system setup of the metadata by
the system administrators and actual usage of the metadata by the system users.

Metadata binning and subsequent filtering is an efficient approach that utilizes investigative
data to refine a search and improve search results. If the metadata associated with the
probe image is available to the practitioner and the FR system being used allows a metadata
search, metadata filtering could be used to refine the initial search or to further refine the
search results. Agency policy should govern at what point metadata is used in the search
process, and user preference or agency policy will determine whether filters are added
separately or jointly. If refining a search using metadata filters is an option, experimenting
to determine the results each filter will produce will assist the practitioner in learning the
limits of this type of search.

It is suggested that the initial search be conducted using only the probe facial image, with
no metadata search included. This will result in the largest-possible candidate list for
comparison purposes. Subsequent searches using metadata as a filtering tool can then be
performed in an effort to produce a more specific result and the best candidate list for
analysis. It is prudent to use metadata searches even if a likely candidate is returned as a

1
FISWG Glossary Version 1.1, dated 2/12/2012

FISWG FR Systems Methods and Techniques 2

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

result of the initial search, as there may be additional photos/candidates available for
comparison within the database that a metadata search would disclose. It is important to
note that a photo-only or metadata-only search may result in candidates that the other
method of searching would not. If the option is available, a subsequent metadata-only
search may produce an additional useful candidate list.

Using metadata filtering to refine a search can also be used to test a FR system algorithm.
Limiting a search to specific parameters while searching for an image that is known to be a
part of the FR database can disclose algorithm problems if the known photo is not a part of
the resulting gallery. Additionally, by observing how a system responds, for example, to an
image-only search vs. a metadata-only search, a practitioner can improve his/her own
search strategy.

Metadata system setup is the phase where the metadata accessible for FR system usage is
defined and categorized. This requires the textual information (e.g., demographic,
biographic, contextual etc.) associated with the facial images to be defined as pick lists,
numeric ranges, dates, or free text.

Metadata may also be created from indirect information not directly associated with a
person. Examples here include:
a) Recidivism
b) Criminal behavior correlations
c) Gang or other affiliations
d) Watchlisting
e) Categorizing metadata sets into larger groupings (e.g., Regional Originating Agency
(ORI) sets)

If metadata is known at the time of the FR enrollment then this information can be used
(e.g. binning) to logically reduce the size of the gallery to be searched/filtered in a controlled
and deterministic manner. Usage of metadata should be appropriately integrated into the
overall search strategy because improper usage can be detrimental to providing successful
search results. If the consistency of the metadata is low (i.e. there are data entry errors
where, for example, gender is incorrectly entered) then filtering based on this demographic
will result in higher error rates and the correct match to the probe could be filtered out.

If binning is utilized then it should be understood that metadata usage is a logical pre-filter
and is separate from the algorithmic portion of the biometric matching process. If the
consistency is known to be high, binning can improve performance, both in time and
likelihood of returning a true mate.

2. Facial gallery management

Facial gallery management can be described as:
a) Monitoring and maintenance processes done on the overall gallery as it grows and
changes.
b) Facial search techniques tuned to the galleries that are not static, can be changed or
adjusted.
c) Applying user access controls and restrictions to subsets of data that have been
deemed operationally sensitive.

Operational performance can be more effective if data is organized per algorithm sensitive
characteristics and appropriate search strategies are used. For example, small images may
require a different search strategy than large images. Further, off pose face images may be
better suited for one algorithm while frontal images may produce better results using

FISWG FR Systems: Methods and Techniques 3

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

another algorithm. Data quality metrics, demographics, and contextual data can all be used
to analyze, profile, locate, present, repair, or exclude images.

Facial gallery management can be broken down into two main areas: data profiling and data
cleansing.

Data Profile
a) Facial galleries can be collections of various types and quality of imagery from
different capture systems that can be characterized based on their core similarities
(e.g., image file size, image quality, expression, etc.). This has also been referred to
as “sameness”. Galleries should be profiled in order to gain an understanding as to
how many collections exist.
b) These collections can be managed and search strategies defined taking into
consideration the aforementioned characteristics of the galleries that may improve
search performance.
c) Proper profiling involves knowing the collections in the facial galleries. Operational
pilots have shown significant increases in accuracy by choosing the appropriate
search strategy for a given image set within a larger gallery of assorted images.

Data Cleansing
a) Many images in a facial gallery are sub-optimal due to reasons that include but are
not limited to:
Non-frontal faces
Images not of a person
Incorrectly detected eye positions
b) These images need to be identified so they may be isolated, corrected, or marked for
exclusion

3. Enrollment of the facial image

The timing of the enrollment of a facial image into an automated FR system will have an
effect on 1) whether subsequent images are providing the most-comprehensive search and
2) the timing of a response to a requestor. The best-case scenario would consist of a near
real-time operational environment – an FR system enrolling an image as the image is
entered into the system to be searched. This would ensure that the image is enrolled to the
system’s photo gallery immediately, that it is immediately available to be searched against
subsequent probe images, and that it is searched against every previously-submitted image
maintained within the database. However, since all FR systems are not the same, this is not
always the case.

With a time-delayed environment, some amount of time will pass before a probe photo is
enrolled into the database following a search. If subsequent probes are searched prior to
previous probes being enrolled (i.e., if an agency waits until a certain time of day to enroll
all of the day’s probe images), a possible candidate(s) may not yet be in the system’s
database and, therefore, cannot/will not be included in the resulting candidate list.
Conversely, if an agency waits to search probes until the system has been updated by the
enrollment of the day’s previously-searched probes, searches will not be performed in a
timely manner and investigations may be impeded. This is also true of batch process
enrollments.

Practitioners should be aware of their agency’s system enrollment policy and adjust their
search strategy accordingly. If the system environment is not real-time the search of a
probe image prior to the day’s system update may necessitate a re-search of the probe once
all of the day’s images have been enrolled.

FISWG FR Systems Methods and Techniques 4

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

4. Search tuning
The purpose of search tuning is to improve overall system performance. Search tuning is
defined as analysis or testing that has been undertaken on operational data that results in a
set of predefined or range(s) of settings or options that can be used when searching. Any
search tuning should incorporate information from (i) system developer and/or integrator,
(ii) objective testing/testers and (iii) operational user analysis with respect to the given FR
system, its data, and its targeted goals.

Information from the system integrator should include but is not limited to:
a) What is the overall approach used for the FR system? Describe the FR system
sensitivity to: geometric shapes of the face, facial features, skin texture, facial
landmarks, or other facial representations.
b) How much roll, pitch, or yaw can be tolerated before pose correction should be
considered?
c) Is there any known bias in the system (e.g., age, ethnic, other)?
d) Is multi-pass searching used? If so what options exist to vary the search pass
settings?
e) Is there a trade-off between accuracy and search speed? If so, how is the intensity
of the searching changed? Who can make these changes?
f) How does facial gallery growth and size impact FR search times?
g) How do you interpret a facial match score?
h) Is any scoring normalization used or available? If so what types and kinds? Is each
gallery dependent or gallery independent?
i) Are there any effects on facial match scores as the gallery size changes e.g. quality of
match performance with more images of more candidates?

Objective tests can then help provide assurance that any information provided is accurate
when it applies to critical statements or assumptions. Objective tests should be performed
on ground truth data. If it is not possible to ground truth operational data then the test data
should aim to be as representative as possible to the intended data type(s) of the system.
For example, if the system is to be used with a combination of mugshot and surveillance
images, then testing should be undertaken on galleries consisting of both of these image
types.

5. Search Strategy
As noted in previous FISWG documents, “it must be recognized that agencies (and
individuals) perform facial comparison for a wide variety of purposes, often under
operational conditions that do not allow for a great deal of time or effort to be expended.
Agencies that choose to utilize such methods must recognize this fact and the associated
risks (i.e., greater chance of error).”2 This applies to other operational constraints including,
but not limited to, enrollment of images, varying system algorithms, requestor’s directives,
and agency policies. A comprehensive search is a trade-off. If agency-specific constraints
such as workload, workforce, and/or deadlines and outside influences such as a requestor’s
directives are predominant concerns, results will suffer. Search strategies employed by
practitioners should take into consideration any known policies, constraints, and customer
expectations.

2
FISWG Guidelines for Facial Comparison Methods Version 1.0, dated 2/2/2012

FISWG FR Systems Methods and Techniques 5

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

Agency policy and outside influence will dictate the extent of searches performed. Any
system designed to hold operationally sensitive data needs to consider levels of user access
and restrictions to subsets of data. Operational policy should be an agency decision, but
workload, workforce, and deadline may dictate and constrain searching strategy/possibilities
and, therefore, results. As previously noted by FISWG, “Facial comparisons are performed
for a number of reasons and the comparison methods employed should be chosen based on
the timeframe required for a decision and the level of confidence required. Comparisons
that need to be immediate require the use of faster processes that will necessarily lead to a
result with a lower confidence. In certain scenarios, this lower confidence is an acceptable
trade-off for the speed of the analysis.”3 This applies to a modified searching strategy
resulting from policy-driven or requestor constraints, as well.

Requesting agencies potentially constrained by policy, may ask that certain procedures be
followed, such as a request to search by specific metadata, to search additional databases
that are external to the initial searching agency, or even to request there be a certain (i.e.,
limited) number of images in the candidate list that is returned. In such cases, search
results will be dependent on information provided by the contributing agency, and results
may differ from those that would be produced had no constraints or directives been issued.

In all contingencies, the practitioner must understand his/her FR system’s capabilities and
limitations before asking it to search by specific information, and in order to develop the best
strategy for his/her operational environment and the constraints put in place by the agency
and/or the requestor.

Search options are defined as the options or feature sets a user has at their disposal
when doing a facial search. This is the culmination of all methods and techniques defined
within this document, that if done properly should increase the likelihood of a successful
search.

Accurate comparison of facial images is highly-dependent on the quality of both the probe
and the gallery image. A practitioner’s ability to note similarities and differences between
the probe and gallery image(s) is reduced when both are not of optimal image quality, and
he/she may be unable to reach a definitive conclusion.

Comparison of:
A high-quality probe against the high-quality portions of the facial gallery

As FISWG has noted previously, “Optimal images for facial comparison are high
resolution and have sufficient focus to resolve features of interest, such as blemishes
and wrinkles, with minimal compression artifacts or distortion…”4 The obvious
advantage to comparing a high-quality probe against a high-quality gallery image is
that, with pristine images, the practitioner will be able to clearly view, on each image,
every feature that is typically compared during the morphological analysis of the face.
The higher the quality of the probe image, the better the chance of producing a
candidate list that will result in a likely candidate and the stronger the conclusion that
can be drawn.

3
FISWG Guidelines for Facial Comparison Methods Version 1.0, dated 2/2/2012
4
FISWG Guidelines for Facial Comparison Methods Version 1.0, dated 2/2/2012

FISWG FR Systems Methods and Techniques 6

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

A low-quality probe against the high-quality portions of the facial gallery

Each agency and practitioner will have his/her own definition of what constitutes a
low-quality probe image. These include, but are not limited to, distorted photos, low
resolution face, and limited dynamic range, each of which impede the practitioner’s
ability to clearly discern the subject’s facial features. A FR system may accept a less-
than-optimal probe image, but the lack of discernible facial features will result in a
less-than-optimal candidate list, regardless of quality of the photos within the FR
system. If an experienced practitioner with the proper training in the analysis of such
photos is able to discern a clear feature on a poor-quality probe image, he/she will be
more likely to match the probe to a gallery image; however, the conclusion drawn
may be weak. Metadata binning may be considered as a way to improve
searching/filtering candidates that closely match general, obvious, or known features
of the probe image. The best-case scenario may be to utilize this situation as an
opportunity to eliminate those photos with obvious differences, and/or offer any
conclusions drawn to the requestor as an investigative lead as opposed to
identification.

A high-quality probe against the low quality portions of the facial gallery

While a high-quality probe will increase the probability of a more thorough image
search against the photos within a FR system and may produce a more
comprehensive candidate list for comparison, the gallery may still include images of
low quality. As with the scenario noted above, if an experienced practitioner with the
proper training in the analysis of such photos is able to discern a clear feature on a
poor-quality probe image, he/she will be more likely to match the probe to a gallery
image; however, the conclusion drawn may be weak. Metadata binning may be
considered as a way to improve searching/filtering candidates that closely match
general, obvious, or known features of the probe image. The best-case scenario
may be to utilize this situation as an opportunity to eliminate those photos with
obvious differences, and/or offer any conclusions drawn to the requestor as an
investigative lead as opposed to identification.

A low-quality probe against the low-quality portions of the facial gallery

Obviously the most-challenging scenario, the submission of a low-quality probe

image for search by an FR system will return a less-than-optimal candidate list, and
the comparison of a low-quality probe against a low-quality gallery image should be
attempted only by an experienced practitioner who has been properly trained in
handling this type of comparison. Metadata filtering may produce a more productive
candidate list than image search alone, but poor quality renders it difficult to discern
blemishes, shapes, and features of the face. Practitioners will be less able to render
definitive conclusions. Eliminations may be easier to make based on gender, race,
and ethnicity.

FISWG FR Systems Methods and Techniques 7

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

Example operational scenarios that should be discussed include:

a) When and how to use metadata filtering when searching
Are there specific instances where metadata filtering can be used or should
not be used?
When searching the gallery, should the search start with no filtering, followed
by adding metadata filters? Or should metadata filtering be applied on the
initial searches and then removed or altered based on the character and
content of the result sets?
b) How is the searching strategy affected by having multiple probes?
Using the same image with different variations from image preprocessing
Using multiple images of the same person of interest in entirely different
images
c) How or when can the number of results be changed to augment the search process?
d) How can the options or features in the biometric algorithm be used to augment the
search process?

Search strategies should also be planned around any known operational constraints. An
example of this is how or when new images are enrolled into the gallery and does this affect
how facial searching is done on new probes that need to be searched? If gallery images are
enrolled twice a day, does this cause a deliberate time delay in searching a new facial
image?

Within the context of this document, metadata filtering is assumed to be done within the
search process and not a post search user based operation. If the client used for reviewing
search results offers post search filtering, then this can greatly enhance the reviewing of
candidate list results.

6. Image preprocessing
Ideally, image preprocessing enhances a probe facial image in order to improve the
matching prospects. The system developer should provide any appropriate guidelines for
optimized facial data to be used by the system. Preprocessing should only be done on poor
quality images as determined by the quality attributes provided by the developer or quality
metrics supplied by the face recognition system. Improper use of image preprocessing
can degrade system performance and therefore only properly trained personnel using
industry accepted image processing applications within approved agency guidelines should
perform image preprocessing.

Image preprocessing can include both image enhancement and facial processing. In all
steps the original image is always preserved for reference and forensic comparison
purposes. It is left up to agency policy to determine if the original image should always be
searched.

a) Image enhancement uses standardized 2D filters including but not limited to image
lighting, histogram equalization, color corrections, de-blur, etc.) Such
enhancements are strictly reliant on information within the image itself. The
geometric aspects of the person in the facial image are NOT changed when doing
this.
b) Facial preprocessing is applied to just the face to clarify and improve the facial image
in order to render a more compliant search probe. Techniques include three
dimensional modeling such as pose correction. These are separate and distinct from
two dimensional modifications because the geometric aspects of the person in the
facial image are changed when doing this.

Some current FR systems provide options, although they may be limited, that will allow a

FISWG FR Systems Methods and Techniques 8

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

practitioner to enhance a probe image, as necessary, once it has been submitted to the
system for search. Much like enhancements made with software such as Photoshop®, these
options will permit a practitioner to make changes to the original probe photo, therefore
allowing a more comprehensive search and possibly resulting in higher ranked or additional
candidate list images. If a FR system produces a poor candidate list, the user can take
advantage of image preprocessing.

Such enhancements could include, but are not limited to:

a) Adjusting brightness
b) Adjusting colors/tinting
c) Adjusting contrast
d) Cropping the image
e) Enlarging the image
f) Adjusting roll, pitch, or yaw
g) Marking the center of eye
May help algorithm with eye placement
Distance between eyes may also assist search parameters
h) Adding metadata to the search (e.g., sex, race, etc.) after the initial image search

A practitioner may find that searching a number of probe photos, the same image with
different variations of enhancements, and/or multiple images of the same subject – provided
the images are clear – improves the chances of an image search resulting in a candidate.
However, regardless of how many probes are submitted by a requestor or to what extent the
practitioner enhances the probe, all available probe images should be searched, and the
same basic search strategies should be used.

Using Photoshop® or comparable software, probe images can be modified from color to
black and white or enhanced, as necessary, to reveal facial features. At the discretion of the
practitioner, the image search can begin with the best probe image or all available probes
can be submitted for search at one time. Regardless of the search order, all available
probes should be searched, whether it is assumed that any will be rejected by the system, or
whether a candidate list has already been produced as a result of any other probe. Using
this approach will ensure a comprehensive search and a more robust candidate list for
comparison purposes. All candidate lists resulting from the search of any of the probe
images should be reviewed.

If metadata is submitted with a probe image, a metadata search should be conducted,

regardless of the size of the candidate list returned as a result of the image searched.

7. Eye Locations
In all steps involving image preprocessing, it is key to ensure that proper eye location and
verification is done. This is either a manual placement of eye locations on an image or the
verification that the FR algorithm can automatically locate eyes in the final search probe.

Eye location verification is a key part of the facial image search process, and essential to an
accurate image search by an automated FR system, as it improves the algorithm search.
Agencies should take this into consideration when purchasing an automated FR system.
Taking into consideration all existing FR systems, however, a practitioner may not have the
option of marking the center of the probe photo’s eyes prior to search. To ensure searching
consistency, each agency should know how their FR system operates. For example will it
mark the eyes (or the chin, or the ears)? Individual agencies should establish an eye
location verification policy that will ensure that the center of the eye is marked prior to
searching or, if this feature is not available, that the probe photo’s roll, pitch, and/or yaw is

FISWG FR Systems Methods and Techniques 9

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

adjusted so the eyes are level. Agencies must be aware of how their FR system operates –
this will drive policy.

8. Metrics Reporting and Auditing

This is defined as the collection, summary, and analysis of any and all information presented
to, acted on, or produced by the FR system. The outcome of this can be used to understand
the system operation, defend the performance of the system, or develop understandings of
how to improve or optimize the system as a whole.

The following FR performance metrics may include:

a) Searches done
Date and time
Workstation
User
Search strategy
Metadata filter(s) applied
Probe characteristics
Search results
b) Average search time
Search strategy
Facial gallery size
Metadata filter(s) applied
c) Failure to acquire / inability to create template
d) Characteristics of the result sets
Number of results
Scores and distributions
Metadata of interest
e) Confirmed matches
Scoring
Ranking
f) Overall quality of the facial images as it pertains to the FR matching

These metrics should be routinely reviewed for continual operational tuning and overall
effectiveness.

FISWG FR Systems Methods and Techniques 10

This document includes a cover page with the FISWG disclaimer

 Version 1.0 2013.05.16

Reference List
FISWG Glossary, version 1.0

FISWG Guidelines for Facial Comparison Methods Version 1.0

FISWG documents can be found at: www.FISWG.org

FISWG FR Systems: Methods and Techniques 11

This document includes a cover page with the FISWG disclaimer