Address-Result Connection

When a query address and a result address are not exactly the same, the kind of connection they have
depends on their relationship:
• Street number is the same in both query and result but the unit number is different or missing:
• If neither address is a street extension, rate result relevance Good when:
▪ The query contains a unit number and the result does not.
▪ The result contains a unit number and the query does not.
▪ The query contains one unit number and the result contains another.
• The query is full address including street number and name and the result is the street name
only:
• Since this result is an unlikely secondary intent, rate relevance as Acceptable.
• Query is for a street [Main Street, Pleasanton, CA] result is just the locality (Pleasanton, CA).
• Rate the result relevance Bad as it does not satisfy the user intent

Lack of Connection
If the result does not satisfy the user intent either because there is no relationship between the query intent
and the result or because the connection between the query intent and result will not be immediately obvious
to the user, rate the result relevance Bad.

Satisfying User Intent

The user’s intent can be determined by considering the query, the user’s location/viewport, and your local
knowledge. Sometimes queries are ambiguous or can have multiple interpretations. In order to determine a
query’s primary and secondary intent, look at the relationship between the query and the result. Ask
yourself:
• Is there a logical relationship between the query and the result?
• How likely is the user to be looking for this result given the query and the location intent?

Intent Type, Definition, Highest Initial Rating

Primary Intent Result: satisfies the most obvious and likely user intent (Navigational or Excellent)
Secondary Intent A result: which is less likely to be the user’s intent. Results matching secondary intent are
often not as prominent as the originally queried entity but still satisfy the intent of the query. Good (User
Intent)
Unlikely Intent A result: which matches the query but is very unlikely to be the user’s intent. Acceptable
(User Intent)
Non-Relevant Intent: There are issues that make the result useless for the user. Bad (User Intent
For any Relevance Rating of Good or below, you must select the appropriate check box(es) to indicate
the reason(s)for demotion: User Intent and/or Distance/Prominence. If both reasons apply, use both
checkboxes.

When determining intent, keep in mind:

• It is rare that a navigational query will have multiple primary intents.
• Rating one result Navigational and one Excellent within the same set of results is also rare, but possible
(see examples in Navigational Results for Category Queries).
• A navigational query will most often produce one result that satisfies the distinct intent and multiple other
results that satisfy a secondary intent or do not satisfy the intent at all. This means there will be one result
rated Navigational and all other results will be rated Good or Acceptable for secondary intent or Bad for not
satisfying user intent.
• Most non-navigational queries will produce results that are rated from Excellent to Bad.
• Since the queries were made by actual users, there will be cases where a query is gibberish or has no maps
intent. Do your best to understand it and see how well the results fit the user’s intent, but don't be shy about
assigning a Bad rating if a result doesn’t fit
Satisfying the user intent is only the first step in the rating process and covers the minimum
requirements for a result to be considered relevant. The rating determined by deciding how well intent
is met is called the Initial Rating. Distance and prominence will be considered next in order to
establish the final rating

Once you’ve made an initial rating by establishing that there is a relationship between the query and the
result and determining the intent type, consider the result’s prominence. The prominence of a feature refers
to its popularity, including the number of people visiting and media sources referencing it. Prominence can
vary based on the test locale and even local knowledge. Consider the following list to get a general idea how
to establish prominence, ordered from the most prominent to the least prominent:
• The feature is known internationally
• The feature is known in the country
• The feature is known in the region
• The feature is known locally
• The feature may not even be known locally
A result that may not directly appear to be user intent can be promoted for being internationally prominent.

Distance vs. Prominence

Once you’ve decided how relevant the result is based on user intent, consider the result in the context of all
possible results in the real world. Use the distance to the user or viewport and the prominence of the result to
decide if a demotion should be applied. A result’s fit to user intent, prominence, and distance should all be
relative to the query and the context around the query. Before deciding whether to demote because of
prominence or distance, it’s important to look at:
• The type of area around the result – rural, urban, or suburban
• The number of businesses that satisfy the query
• The distance to the user or viewport
The following examples of multiple query/result pairs will demonstrate how to interpret the relationship
between distance, prominence, and satisfying the user’s intent. The examples address common rating
situations and explain how to evaluate the relevance of the results.

Many Possible Results

Some queries, like [starbucks], have many results that could satisfy the user. In cases like this, distance is
more important than prominence, so results that are closer to the user’s location/viewport should be rated
higher than results that are farther away. To illustrate the high density of Starbucks, possible results are
shown as purple dots
Rate Excellent for Starbucks locations that are in close proximity to the user.
Relevance Good (Distance/Prominence) Rate Good for locations that are a bit farther away from the closest
relevant locations to the user.
Relevance Acceptable (Distance/ Prominence) Rate Acceptable for relevant locations that are even farther
away from the user but still inside the viewport.
Relevance Bad (Distance/Prominence) When there are many locations close to the user, rate Bad for
locations that are significantly farther away and outside the viewport.
Relevance Excellent Rate Excellent for Starbucks locations that are in close proximity to the user, even if
they are outside the viewport.

User Outside Fresh Viewport

When a user is outside the viewport, results are expected in or near the viewport. In cases where there are a
large number of possible results inside the viewport, even if they are not returned for rating (purple dots),
rate Bad for those outside it.
Relevance Excellent Rate any Starbucks within the fresh viewport Excellent when the user is outside the
viewport.
Relevance Bad (Distance/Prominence): When there are many possible results within the viewport, including
those not returned for rating (purple dots), rate Bad for results outside it.

Few Possible Results

When the query has only a few results in the real world, be more lenient when considering distance. Because
there are only a few Zara clothing stores in Miami, results for the query [zara] are demoted more leniently
for distance than the many Starbucks locations in the example above.
Relevance Excellent: Rate Excellent for the closest Zara location, even when outside the fresh viewport.
Relevance Good (Distance/Prominence): When there aren’t any possible results in the fresh viewport, rate
Good for locations that are a bit farther away from the user/viewport.
Relevance Acceptable (Distance/ Prominence): The Zara store is located significantly further away than the
second closest stores and is therefore rated Acceptable

Few Possible Results and Greater Distance

Leniency on distance is acceptable when there are only a few possible results for the query in the requested
area. For example, a user close to Berlin queries for [Wartestraße], which is a street name. There are only
four streets in Germany called “Wartestraße” that can satisfy the user’s intent and all four streets are returned
as results. Because of the limited number of results, each result’s relevance remains fairly high even over
longer distances
Relevance Excellent: Wartestraße in Gransee is the closest to the user and viewport and is therefore the most
relevant.
Relevance Good (Distance/Prominence): Wartestraße in Siegen is far away from the user, but is the second
closest possible result. Wartestraße in Wiesbaden is at a similar distance from the user as the one in Siegen
and receives the same rating.
Relevance Acceptable (Distance/ Prominence): Wartestraße in Geislingen is the fourth closest result but
significantly farther away. Given all other results and their distances, this result is rated Acceptable

Rural Areas
Results are often farther away in rural areas, but can still be considered relevant if there are no reasonable
results close to or within the fresh viewport. In a rural area, generosity can be applied to ratings with respect
to distance. The criteria for rating are similar to the [zara] example in the Few possible results section, but
with an emphasis on distance. Here is an example for the query [american legion] in North Dakota.

Relevance Excellent: This result is closest to the user based in Bismarck, ND.
Relevance Good: The second-closest result is significantly further north than the closest result.
Relevance Acceptable While quite remote from the user, this result is still a good option for the user given
that there are only two closer results.
Relevance Acceptable: Like result 3, this location quite far away but is still an acceptable result given that
there are only two significantly closer results available to the user.

Location – User Intent Deviation

When the location intent and user intent do not allow for a clear result (because either the user or viewport
are in an atypical location, like the middle of the ocean), rate with a focus on high prominence. For example,
the query [Hong Kong] in a test locale set to en_US with the viewport and user location around Naples,
Italy. There is no real change in rating if the user is inside or outside of the viewport, or if the viewport is
fresh or stale
Navigational: The most prominent result is the territory of Hong Kong in Asia.
Excellent Businesses/locations with the same name that are within the viewport or close proximity to the
user and prominent. (This will be rare.)
Bad (Distance/ Prominence): Locations that are too far away and less prominent to be considered relevant
should be rated Bad. A restaurant in Rome named “Hong Kong” should be rated Bad because it is very far
away from the user location, which is in Naples.

Rating for User Inside and Outside Viewport

The following two examples show the differences in rating when a user is inside and outside a fresh
viewport. The query is for a chain business called Great Wolf Lodge found across the United States. The
examples show all possible results within and around the large viewport. In general, when the user is outside
the fresh viewport, no result within the viewport is demoted for distance. The only exception to this is when
the viewport is large enough to include an unusually large area, like an entire continent, and the results
within it would not realistically be useful or interesting to a user with this query
Relevance Excellent: The result is within the fresh viewport and fulfills the primary user intent.
Relevance Good/Acceptable (Distance/Prominence) The result is outside the fresh viewport. Because there
are a limited number of possible results within the viewport and this one is relatively close, demote to Good
or Acceptable.
When the user is inside the fresh viewport, the user's location is used as location intent and results are
demoted for distance.
Relevance Excellent This result is amongst the closest possible to the user and is inside the fresh viewport.
Relevance Good (Distance/ Prominence) Though still inside the fresh viewport, this result is significantly
farther away from the user location with closer options available.
Relevance Bad (Distance/ Prominence) This result not only is very far away from the user, it is also outside
the fresh viewport.
Relevance Acceptable (Distance/Prominence) While very far away from the user, this Suggestion is still
within the fresh viewport and therefore rated Acceptable.

Multiple Transit POIs with the Same Name

Many large transit POIs include multiple means of transportation at the same location. For example, large
airports can have metro or train service within the same location, or train stations can be a hub for trains,
metros, and buses. Other transit POIs at the queried transit POI should be treated as secondary intent with an
initial rating of Good, but can also be considered Acceptable if the result is less likely to satisfy the user’s
intent. Additional demotions can be applied based on the prominence of the returned feature

POIs and Transit Intent

A non-transit POI can be associated with a transit POI with the same name. You can establish the
relationship between the two entities by using local knowledge, researching prominence, and understanding
how well the result satisfies user intent. This way you will learn whether the query has a distinct
navigational intent or if the prominence of the two POIs that share a similar name is so strong that both can
be the intent

Transit Queries
The criteria described above should be applied to understand the transit queries. Additionally, if it is
determined that a query has a clear navigational intent, all other results will be Bad (see [12th st oakland
bart] example below). If a result is promoted to Navigational, other results that could potentially satisfy the
user intent should be demoted further for distance and prominence (see [BART daly city] and [stockport
station] examples below). In general, the fewer the results that satisfy the user intent, the farther away the
results can be and still be considered relevant. The more available results that can satisfy the user intent, the
closer they need to be

Stops vs. Stations

A transit stop is a location typically designated by a bench or covered waiting area, a posted sign, or street
markings. Transit stations, on the other hand, are typically larger areas associated with major structures
including, but not limited to, platforms, ticketing offices, enclosed waiting/seating areas, or terminal
buildings/structures encompassing the transit area. The term “station” is primarily used to refer to transit
locations that are larger and typically a terminal housing multiple routes. All stations are stops, but not all
stops are stations.
Since all stations are stops, but not all stops are stations, returning a station for a stop query is ok, but
returning a stop when the query is station should result in a rating of Bad with the User Intent checkbox
selected. Try to identify these corresponding terms in the test locale and understand the relationship between
them.
For the category [bus stops], relevance should be based mainly on distance from the user location or fresh
viewport and the physical size of the bus stop. If the result is a small bus stop in a rural area and the bus stop
is among the closest ones to the user’s location intent, it should be rated Excellent. On the other hand, if the
result is a large bus hub that is a bit farther away from the user, it should still receive an Excellent because of
its size, regardless of the fact that there are closer, smaller bus stops. Results close to the user should not be
demoted because there is a larger bus stop farther away, instead, the large bus stop should be promoted
because of its importance

Parking Intent
When evaluating [parking] queries, note that:
• Free and paid parking are equally relevant.
• Parking time limits do not affect relevance rating. Your rating should not be changed by the
fact that a parking lot is long term, short term, or limited-time (like a two-hour limit) parking.
• Parking for small cars, large cars, RVs, or motorcycles is equally relevant.
• If you can find evidence that a result is for private parking that cannot be used by the general
public, give it a rating of Bad

Relevance: Service-Level Mismatch

Sometimes a query specifically requests a certain level of service, which may or may not be reflected in the
result. Common requests include:
• Specific stores (e.g. Walmart Supercenter)
• Outlet stores (e.g. J. Crew Factory)
When the query requests a specific level of service which is not reflected in the result, give an initial rating
of Good and demote as necessary depending on how well the result would satisfy the user intent.
When the query requests a type of business with a generic level of service, providing a result that offers
more service than requested still satisfies user intent and should receive an initial rating of Excellent.
When the query does not specify the type of service, any relevant result can be rated Excellent.

Rating Relevance When the Result Status is PERMANENT_CLOSURE

A business/POI that has its status set to PERMANENT_CLOSURE requires special consideration
during relevance rating. Note: A result without any status indicated should be rated as usual, including when
research indicates the business/POI is closed or does not exist. There are some situations where you can
expect to see results with the status PERMANENT_CLOSURE and others where such results are not
expected. The following sections will describe how to identify and rate them.

Rating Approach
Rate results with an expected status of PERMANENT_CLOSURE as if they were open. This means that in
most cases the rating will be either Navigational or Excellent, because even though they are closed, these are
the best or only results that could ever be returned
Name
The Name and Category Accuracy rating is used to evaluate the accuracy of a Business/POI name. Your
final rating will take into consideration both the accuracy of the Business/POI name and the category
assigned to it. Inaccuracies in either or both of these elements will result in rating demotions.
You’ll see this rating referred to throughout the guidelines as both “Name Accuracy” and "Name and
Category Accuracy.”
Business/POI Name and Category Accuracy can be rated as:
• n/a
• Correct
• Partially Correct
• Incorrect
• Can’t Verify
If you give a final Name and Category Accuracy rating of Partially Correct or Incorrect, you may be
presented with two checkboxes:
• Name Issue
• Category issue
If they appear, use one or both them to select the reason(s) for your rating.
N/A Name
The n/a rating should be used for all address type results, including residential addresses, streets, localities,
and so on. Unlike Business/POI type results, address type results do not actually have a result name. Instead,
the first line of address type results appears as the result title. This is because this field is meant to present
the most relevant part of the address information.

Correct Name:
A business/POI result name is accurate when it is used on the POI’s official website or on other official
resources. Even if the official website does not use the name, if it is used on other official sources, the name
is correct. Note that the name has to be used to refer to the particular POI. The corporate name by itself is
not automatically the correct name

Partially Correct Name

A partially correct name differs from the official versions but can still be recognized by the user. Partially
correct names can include minor and moderate misspellings, service level mismatches, and missing or
unnecessary name parts, including holding names/corporate structures. When the business name on the
storefront does not include the corporate status (Ltd., Inc., etc.) seen in the result, it is partially correct.
Minor and Moderate Misspellings and Missing or Extra Words Consider names partially correct when they
contain issues that don’t prevent the user from identifying the business:
• Name contains repeated, redundant, or missing information or parts
• Incorrect or missing punctuation or special characters
• Unnecessary or missing spaces
• Unexpected use of lower/upper case and ALL CAPS letters
• Acronyms are often expected in ALL CAPS: YMCA, TGIF
• Minor and moderate misspellings that don’t influence the user’s understanding. See your Country-Specific
guidelines for more information on misspellings.
• Mix of expected languages in the result title. Expected languages are the languages of the query, test
locale, or region, or of an official business name

Incorrect Name
An incorrect result name is one that can’t be recognized because of severe misspelling or ambiguous or
unnecessary/missing parts in the name. Consider names incorrect when they contain issues including:
• Severe misspellings that prevent the user from identifying the business because of:
• Change in meaning
• Misspelling that results in an entirely different word or gibberish
• Holding names that are completely different than the recognized name
• Unnecessary or missing parts in the name that prevent the user from recognizing the business
• Slang and inappropriate language
• Using the previous name of the same entity that is not a variation of the current name and therefore cannot
be identified
Note: If a result name is incorrect, the final Name and Category Accuracy rating will always be Incorrect,
even if the category is correct.
PIN ACCURACY
Best Available Evidence Pin rating for most features found under rooftops is based on the best available
evidence for that feature's location. That is, the more evidence that can be found to verify a feature’s
location, the more precise the pin’s location must be in order to be rated Perfect. This is true whether or not
the feature's address includes a unit number, because the goal is to find the location of the feature. If street
imagery or other strong evidence can be found to confirm a feature’s specific location under a shared
rooftop, only that location will be considered Perfect. If a shared rooftop can be identified but there’s no
strong evidence to confirm the feature’s specific location under it, the entire rooftop will be considered
Perfect. If several rooftops share an address and there’s no strong evidence to confirm which rooftop the
Feature is under, all rooftops will be considered Can’t Verify.
Note: This general pin rating rule does not apply to transit POIs, which have their own pin rating rules
Approximate: The result is a business (or an address) under a connected rooftop with multiple street
numbers under it. This result shares a parking lot with other features.
Note: If there are any other buildings within the shared parking lot that are not connected to the Perfect
location, they will also be rated Approximate
 Little Guide Line

When the user is inside the fresh viewport, the location intent becomes the user location unless there is an
explicit location intent for the query. Many candidates fail to use the user location as the location intent in
this scenario, a common mistake would be to rate every result inside the viewport as excellent or demote
results further away from the user but still inside the fresh viewport to bad. Here is an example from the
guidelines on how to rate a scenario like that correctly:
User Inside Fresh Viewport
When the user is inside the viewport, consider all possible results, including real-world results not returned
for rating (purple dots), and demote by distance from the user.
Query:
starbucks
User and fresh viewport in San Francisco, CA
Result, Rating and Explanation
① Starbucks 865 Market Street
Relevance Excellent
Rate Excellent for Starbucks locations that are in close proximity to the user

② Starbucks 170 O’Farrell St

Relevance Good (Distance/Prominence)
Rate Good for locations that are a bit farther away from the closest relevant locations to the user
3 Starbucks 264 Kearny St
Relevance Acceptable (Distance/ Prominence)
Rate Acceptable for relevant locations that are even farther away from the user but still inside the viewport.

4 Starbucks 580 California St

Relevance Bad (Distance/Prominence)
When there are many locations close to the user, rate Bad for locations that are significantly farther away
and outside the viewport.

⑤ Starbucks 140 Mason Street

Relevance Excellent
Rate Excellent for Starbucks locations that are in close proximity to the user, even if they are outside the
viewport.
Business Result for Address Query
When the query is an address query for a street and the returned result is a business on the same street the
user was looking for, that result does not satisfy the user intent. Many candidates fail to demote such results
to bad due to user intent and instead rate relevance as good or higher when it should be demoted to bad. Here
is an example from the guidelines on how to rate a scenario like that correctly:
Query
stevens creek blvd cupertino
User and fresh viewport in
Cupertino, CA
Result
1. Happy Lamb Hot Pot
19062 Stevens Creek Blvd,
Cupertino, CA, 95014
Relevance: Bad (User Intent)
The query is for a street and the result is a specific business on that street. This specific result is unlikely to
satisfy the broad intent of the query.
2. 19062 Stevens Creek Blvd
Cupertino, CA, 95014
Relevance: Bad (User Intent)
The result includes a street number when the query is for a street, making this result too
specific for the more general query.

Remember to continue to ask yourself the question - could this result be what the user is looking for? If I
received this as a result, would I be satisfied?

Chain Business Relevance Rating

A query for a chain business is generally not considered an explicit query and is therefore not eligible a for
Navigational rating unless the query contains a location modifier that points to a single unique location.
Many candidates rate these results with a relevance rating of Navigational which is considered incorrect as
the highest rating a chain business can receive is Excellent.
Here is an example from the guidelines on how to rate a scenario like that correctly:
Query
[whole foods]
User and Viewport: User and fresh viewport in Irvine, California
User Intent: The user is clearly looking for the grocery chain Whole Foods close to their location or at least
within the viewport.
Is there a Navigational Result for this Query? No

Result
① Whole Foods, The District, 2847 Park Ave, Tustin, CA 92782 Category: Grocery
Is there a Navigational Result for this Query? No
Ratings and Explanation
Relevance Excellent
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Approximate
Reason: This is a Whole Foods store within the user’s viewport as well as the closest Whole Foods
location to Irvine. The pin is dropped within the shared parking lot of the result, making it
Approximate.

Result
② Whole Foods Fashion Island, 415 Newport Center Dr, Newport Beach, CA 92660 Category: Grocery
Ratings and Explanation
Relevance Good (Distance/Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: This is a Whole Foods store in the same metro area, but is outside the user’s viewport

Result
③ Whole Foods Aliso Village Shopping Center, 23932 Aliso Creek Rd, Laguna Niguel, CA 92677 Category:
Grocery
Relevance Acceptable (Distance/ Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: This is a Whole Foods store in a neighboring city, outside of the viewport and a considerable
distance away. As such this result is less relevant to the user and is rated Acceptable
Navigational Chain Business
If the query for a chain business includes a location modifier that points to a single unique branch of that
business, that specific branch can be eligible for a Navigational rating.
When there are multiple results available for the location modifier, the highest possible rating for all results
will be Excellent. Remember:
User and viewport location should always be ignored when there is a location modifier
Ratings must always be based on possible real-world results
Many candidates fail to either demote results outside the location modifier or are too generous with the
relevance rating. They either forget to check real-world results that did not appear on the map or fail to do
enough research to understand the boundaries of the location modifier before rating relevance.
Here is an example from the guidelines on how to rate a scenario like that correctly
Query: Starbucks san francisco
User and Viewport: User and fresh viewport in San Francisco, CA
User Intent: The user is seeking a Starbucks in a specific location. The fresh viewport and user location can
be ignored since the user has an area in mind for results. Because there are so many Starbucks within the
named area, results outside it are rated Bad.
Is there a Navigational Result for this Query? No
Results: ① Starbucks, 2165 Polk St, San Francisco, CA 94102 Category: Coffee
Ratings and Explanation
Relevance Excellent
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: This result is one of many Starbucks in the specified location, San Francisco, and is rated
Excellent. All results within San Francisco are eligible for the highest possible initial rating of
Excellent. All the data for the result is correct

Results: ② STARBUCKS 1231 Market St, San Francisco, CA 94103 Category: Coffee
Ratings and Explanation
Relevance Excellent
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: This result is one of many Starbucks in the specified location, San Francisco, and is rated
Excellent. All results within San Francisco are eligible for the highest possible initial rating of
Excellent.
The STARBUCKS name is written in capital letters. Because this is how the business refers to itself on
storefront signs, it is Correct. The address is Correct and the Perfect pin lands on the correct rooftop

Results: ③ Starbucks 1555 40th St. Emeryville, CA 94608 Category: Coffee

Ratings and Explanation
Relevance Bad (Distance/Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Approximate
Reason: This Starbucks is outside of the boundaries of the specified location and should be rated Bad
since there are so many Starbucks to choose from in San Francisco. The name and address are
Correct. Using the best available evidence — in this case, street imagery — the location of the
Starbucks under the rooftop can be found. Since the pin falls on the correct rooftop, but not in the
correct location, it is rated Approximate

④ Starbucks 601 Westlake Center, Daly City, CA 94015 Category: Coffee

Results: Ratings and Explanation
Relevance Bad (Distance/Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: This Starbucks is outside of the boundaries of the specified location and should be rated Bad
since there are so many Starbucks to choose from in San Francisco. All the data for the result is
correct

Results: ⑤ Starbucks 12 Manor Plaza, Pacifica, CA 94044 Category: Coffee

Results: Ratings and Explanation
Relevance Bad (Distance/Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Approximate
Reason: This Starbucks is outside of the boundaries of the specified location and should be rated Bad since
there are so many Starbucks to choose from in San Francisco. The pin drops within the Half n' Half rule
boundaries of the shared parking lot
Unique Transit Stations
When the user is looking for a specific transit station with a unique name in a locality that does not share the
name as the transit station, we expect a Navigational result and rate any other transit station in that locality
as bad for user intent.
For example, let’s take the transit stations in San Franciso - BART stations. There is a unique BART station
name that is not the same as the name of the place where the station is located.
The expectation for results returned for a different BART station that does not satisfy the user intent is to be
demoted to Bad when rating relevance. Many evaluators fail to demote results that do not satisfy the user
intent.
Here is an example from the guidelines on how to rate a scenario like that correctly
Query 12th st Oakland bart] User and viewport in California
Result: 12th St. Oakland Station Oakland, CA
Rating and Explanation:
Relevance Navigational
Reason: The query is for a specific BART station with a unique name that is not the same as the name
of the locality where the station is found. (Compare this to the examples below). This result matches
the query exactly and fully satisfies the user’s intent for this specific station.

Result: 19th St. Oakland Station Oakland, CA

Rating and Explanation:
Relevance Bad (User Intent Issue)
Reason: The query intent is for a specific BART station with a unique name that is not the same as the
name of the place where the station is located. The result returned is for a different BART station and
does not satisfy the user intent. This result should receive a rating of Bad

Location or Category Intent

When the query is for a transit station with a location intent, keep in mind that it may be a transit station
with a location modifier, or potentially the user looking for a transit station in that location (category intent).
Many candidates fail to make that connection due to a lack of research and failing to make the connection
between when the query can be interpreted as both a category (i.e. Locality + Station) or as an explicit
location, (Direct Match Name).
Results inside the intended locality should get a high rating while results outside the locality should be
demoted accordingly based on distance and the number of available results.
Here is an example from the guidelines on how to rate a scenario like that correctly:
Query: stockport station User and viewport in Stockport, England
Result: Stockport Station Stockport, England
Relevance Navigational
Reason: The query intent is for a station in Stockport. Research shows that there are several stations
in Stockport, but only one named Stockport Station, so this result fully satisfies the user’s intent

Query: Davenport Station Stockport, England & Heaton Chapel Station Stockport, England
Relevance Excellent
Reason: The query intent is for a station in Stockport. There is one station with a name that matches
the Navigational exactly, which receives the Navigational rating. All other stations within the queried
locality should receive Excellent
Query: Burnage Station Manchester, England
Relevance Good (Distance/Prominence)
Reason: This station is just outside of the requested locality, in a neighboring locality. Even though the
station is outside of the requested locality, it provides a choice of stations in the area for the user. The
fewer choices there are available, the farther away the results can be and still be relevant. This result
is demoted due to distance.

Location Modifier Intent

When the query is for a chain-type business with a location modifier attached, results are expected
within the given location modifier and user/viewport position are to be disregarded.
In cases where few results exist in the location modifier, results outside of the modifier can still be viable
options based on the distance from the intended location.
Many candidates fail to disregard the user and viewport location and rate results within or near them with a
high relevance rating when they should be disregarded.
Here is an example from the guidelines on how to rate a scenario like that correctly:
Query: Aldi waco tx
User and Viewport: User and fresh viewport in Houston, TX
User Intent: The user asked for a specific chain business in a specific location. The fresh viewport can be
ignored since the user has a specific area in mind for results. Because there are very few stores within the
specified area, results outside it are gradually demoted.
Is there a Navigational Result for this Query? Yes
Results: ① ALDI 1220 N Valley Mills Dr, Waco, TX 76710 Category: Grocery
Ratings and Explanation
Relevance Navigational
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Approximate
Reason: ALDI is a well-known chain business. Research confirms that the location modifier in the
query narrows possible results to a single unique location. This is the only location eligible for a
Navigational rating.
The pin drops within the shared parking lot.

Results: ② ALDI 3310 S 31st St, Temple, TX 76502 Category: Grocery

Ratings and Explanation
Relevance Good (Distance/Prominence)
Name Accuracy Correct
Address Accuracy Correct
Pin Accuracy Perfect
Reason: Although this result is not in Waco, research shows that, unlike the many Starbucks in San
Francisco, there are very few Aldi stores in the Waco area. That is why this nearby result should be
demoted -2 for distance from the highest possible initial rating of Navigational and rated Good. The
pin drops on the edge of the appropriate rooftop and can be rated Perfect

Results: ③ ALDI 3623 W State Highway 31, Corsicana, TX Category: Grocery

Ratings and Explanation
Relevance Acceptable (Distance/ Prominence)
Name Accuracy Correct
Address Accuracy Incorrect – Postal Code
Pin Accuracy Perfect
Reason: Although this result is not in Waco, research shows that, unlike the many Starbucks in the
San Francisco example above, there are very few Aldi stores in the Waco area. This result has been
demoted -3 from the highest possible initial rating of Navigational to Acceptable because it is
significantly farther from the area of expected results. The postal code, which is a mandatory address
component, is missing from the result address.

Find the Connection

If the result does not satisfy the user intent because there is no relationship between the query intent and the
result, or because the connection between the query intent and result will not be immediately obvious to the
user, rate the result relevance Bad. Many candidates fail to do enough research to understand the user intent
and what the result returned is.
Here is an example from the guidelines on how to rate a scenario like that correctly
User Query: airport
Result: Santa Cruz Boardwalk Santa Cruz
Type of Connection: None
Description: Does not satisfy user intent: Research shows that there is no connection between the query
and the result. Rate Bad.

User Query: Raging Waters 2333 South White Rd San Jose

Result: 2333 South White Rd San Jose
Type of Connection: None
Description: Does not satisfy user intent: Result is for the correct address, but does not include the business
named in the query, so users will have no way of knowing if this is truly connected to the business or not.
Rate Bad.

User Query: Raging Waters

Result: 2333 South White Rd San Jose
Type of Connection: None
Description: Does not satisfy user intent: Query is the name of a business and result is for the correct
address, but does not include the business name. Users will have no way of knowing whether or not this
address is connected to the business they asked for. Rate Bad.

User Query: valley fair mall

Result: Macy’s 2801 Stevens Creek Blvd, Santa Clara
Type of Connection: None
Description: Does not satisfy user intent: Result is for a store, not a mall. This store is inside the requested
mall, but does not satisfy the intent of the query, which is the whole mall. Rate Bad

Implicit Location Intent

If the user is outside of the viewport and the viewport is fresh, and the query is for a chain business and a
result matches the query intent perfectly, we use the fresh viewport as the location modifier.
In cases like this, ratings should be based on the fresh viewport and results outside the viewport should be
demoted for distance if there are closer real-world options. Many candidates fail to use the fresh viewport as
the location modifier.

Non-Specific Addresses
For queries where the location intent is not explicitly stated in the query, you must infer the location intent
from the user’s viewport and the user’s location. One difficulty in evaluating address queries has to do with
partial addresses, which could refer to multiple locations. Generally, for partial addresses, users are looking
for the location closest to their position or within / near the fresh viewport.
 Results that are farther away from an ideal result but can potentially satisfy the user’s intent can be
rated as high as Good (Distance/Prominence)
 Results are demoted based on distance
  Results are demoted based on the density of potential results in a given area
 Results that are too far away will be rated Bad (Distance/Prominence)
Many candidates fail to do real-world research hence missing that there might be closer results to the user or
fresh viewport.

Name Accuracy for Address Results

The n/a rating should be used for all address type results, including residential addresses, streets,
localities, and so on.
Many evaluators fail to rate name accuracy as n/a when rating street names, localities and so on

Non-Specific Addresses
Ensure that the classification is correct in order to rate name accuracy as correct. Many candidates fail to
make the connection when rating name accuracy between the name and the correct classification by only
taking the name into consideration and rating name accuracy correct when it should be demoted when the
classification is incorrect

Results to Rate
You are only required to rate tasks with rating options beneath them. Avoid greyed-out results. You may
occasionally see one or more normal-looking or greyed-out results with no rating fields beneath them. No
rating is required for these results. This is expected and is not a technical issue or a reason to release the
survey.
Many candidates fail to remember this statement and get confused when they see 3 results, 2 of which are
greyed out with no rating fields beneath them. These other results are there for context. In your exam, you
are only going to be rating one result per task. Keep an eye out for the result that you need to rate.