International - Localization Quality Defect - SOP - V1
International - Localization Quality Defect - SOP - V1
Evaluation SOP
Purpose
This SOP provides instructions to evaluate whether the Rufus response to a given question
has any localization quality defect. Localization quality defect measures the % of Rufus
responses that have errors in local grammar, spelling, unit of measure, terminology, etc.
This SOP includes sections per International locale (see details in the individual locale
section).
Workflow Diagram
1
Q1: Do you understand the query?
Answers:
• Yes
• No
Guidelines:
• Read the query and make sure you understand what the intent of the customer is.
2
Exit clause: If you don’t understand what the customer wants (e.g., empty
query, unintelligible, unclear, other language), then select "No", leave a comment,
and then skip the entire entry.
Note: If the query is very strange (e.g., product question unrelated to the
context page), but you still understand the customer intent, then annotate it
normally.
Description: First, we want to classify the query type based on the customer intent and
entity that the customer is looking for. When the customer enters a query in the
search/conversation bar, we can map its meaning/intent to one of our pre-defined
customer intents. Select the appropriate Intent-Entity pair for the given query. For detailed
definitions of each intent and entity, please refer to the table “Query Goal” in the
Definitions document.
Intent Selections:
• Recommendation
• Factoid
• Opinion
• Description
• Instruction
• Comparison
• Other
o When selecting “Other”, skip the entity selection
Entity Selections:
• ASIN
• ASIN attribute
• product type
• product type attribute
• Amazon service
3
Answers:
• Yes
• No
Guidelines:
• If the answer is empty, select "No", leave a comment as necessary, and then skip the
remaining questions.
Answers:
• Yes
• No
Guidelines (en-UK)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
4
• Incorrect measurement format (e.g., 1,000 in UK versus 1.000 in EU, millions in UK
versus lakhs in IN)
• Tone: The shopping assistant must use a conversational and supportive/empathetic
tone. A conversational tone means that the answer should be written in informal or
conversational speech from the perspective of a shopping assistant, i.e. not too
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (de-DE)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
5
instead of formal (“sie”) register. However, if the customer starts the conversation in
a formal register (“sie”), the assistant should also respond in a formal register
(“sie”). The assistant should avoid using any slang. A supportive/empathetic tone
means that the answer must demonstrate an appropriate level of empathy and
support to the customer. The assistant should be empathetic to the customer’s
problem, their happy or bad news, or an exciting prospect. It should adapt its tone
and react appropriately with natural and idiomatic expressions like “Kudos to you!”,
“Great to hear!”, “I’m sorry to hear that”, “That’s exciting!”, etc. Outside of these
kinds of reactions, the assistant’s tone should be neutral.
• The answer is cut off / not fully displayed
Guidelines (en-DE)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
6
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (fr-FR)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
7
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (it-IT)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
8
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (es-ES)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
Guidelines (pt-ES)
9
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
Guidelines (en-IN)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
• Answer is not in same language as question. Note that in en-IN, questions could
also be in Hindi / Hinglish (in Latin script) – if Rufus responds to such a
Hindi/Hinglish question with a 100% English answer, mark this as a localization
quality defect.
10
• Incorrect grammar (e.g., these shoes has a great deal on them)
• Incorrect spelling (e.g., color versus colour – check for British English (used in IN)
versus American English)
• Incorrect capitalization of words (i.e., first word in a sentence should start with a
capital letter)
• Non-local terminology. As long as the terminology makes sense broadly within the
country (even if it is a regional reference), it is correct.
• Incorrect unit of measure (e.g., INR versus $, miles versus km, kg and grams over
pound). Note that in rare cases, specific product types have an established unit of
measure and it is correct for this to supersede the country’s official unit of measure.
For example, TV screen sizes are usually presented in inches, even in the IN,
where centimeters is the official measurement system versus inches.
• Incorrect measurement format (e.g., 1,000 in IN versus 1.000 in EU, millions versus
lakhs in IN)
• Tone: The shopping assistant must use a conversational and supportive/empathetic
tone. A conversational tone means that the answer should be written in informal or
conversational speech from the perspective of a shopping assistant, i.e. not too
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (en-CA)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
11
• Non-local terminology (e.g., nappies in UK versus diapers in CA). As long as the
terminology makes sense broadly within the country (even if it is a regional
reference), it is correct.
• Incorrect unit of measure (e.g., CAD versus $, miles versus km). Note that in rare
cases, specific product types have an established unit of measure and it is correct
for this to supersede the country’s official unit of measure. For example, TV screen
sizes are usually presented in inches, even in CA, where centimeters is the official
measurement system versus inches.
• Incorrect measurement format (e.g., 1,000 versus 1.000 in EU, millions versus
lakhs in IN)
• Tone: The shopping assistant must use a conversational and supportive/empathetic
tone. A conversational tone means that the answer should be written in informal or
conversational speech from the perspective of a shopping assistant, i.e. not too
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (fr-CA)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
12
o Use of fr_FR vs. fr_CA : e.g. faire du shopping vs. magasinage or email vs.
courriel
o use of anglicisms such as Thanksgiving vs. Action de grâce or Black Friday
vs. Vendredi fou ).
• Incorrect unit of measure (e.g., € versus $, miles versus km). Note that in rare
cases, specific product types have an established unit of measure and it is correct
for this to supersede the country’s official unit of measure. For example, TV screen
sizes are usually presented in inches, even in the EU, where centimeters is the
official measurement system versus inches.
• Incorrect measurement or size format (e.g., e.g. small, medium, large vs. petit,
moyen, grand or “lit king” vs “très grand lit”)
• Tone: The shopping assistant must use a conversational and supportive/empathetic
tone. A conversational tone means that the answer should be written in informal or
conversational speech from the perspective of a shopping assistant, i.e. not too
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (es-US)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
13
• Incorrect measurement format (e.g., 1,000 in US versus 1.000 in EU, millions in US
versus lakhs in IN)
• Tone: The shopping assistant must use a conversational and supportive/empathetic
tone. A conversational tone means that the answer should be written in informal or
conversational speech from the perspective of a shopping assistant, i.e. not too
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (en-JP)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
14
formal but not too familiar. The assistant should avoid using any slang. A
supportive/empathetic tone means that the answer must demonstrate an
appropriate level of empathy and support to the customer. The assistant should be
empathetic to the customer’s problem, their happy or bad news, or an exciting
prospect. It should adapt its tone and react appropriately with natural and idiomatic
expressions like “Kudos to you!”, “Great to hear!”, “I’m sorry to hear that”, “That’s
exciting!”, etc. Outside of these kinds of reactions, the assistant’s tone should be
neutral.
• The answer is cut off / not fully displayed
Guidelines (ja-JP)
Please mark ‘Yes’ if the answer has at least 1 of the below localization quality defects.
口調について
• 「てにをは」を端折らない
• い抜き言葉は使わない
15
• 体言止めにせず、文章の形でしゃべる
表記について
・英数字は半角で
・()は半角で
・!や?の後ろは半角スペースをあける
・返答の文末には。が必要
敬称について
原則として、必ず人名には敬称をつける。敬称は「氏、さん、君、ちゃん」を用いる。現
役のスポーツ選手は「〇〇選手」
例外:徳川家康 など歴史上の人物
人名のガイドライン
天皇について
皇族の敬称については右のリンクを参照し、正しい敬称を用いる
16
参考:天皇・皇室の敬称
送りがな
○ ×
行う 行なう
終わり/終わる 終り/終る
書き込み/書き込む 書込み/書込む
組み合わせ 組合せ/組み合せ
支払い/支払う 支払
取り扱い/取り扱う 取扱、取扱い/取扱う
取り消し/取り消す 取消、取消し/取消す
取り引き 取引、取引き
払い戻し/払い戻す 払戻、払戻し/払戻す
貼り付け 貼付け
読み込み/読み込む 読込み/読込む
読み取り/読み取る 読取り/読取る
○ ×
受付期間 受け付け期間
申込期間 申し込み期間
取扱店 取り扱い店
経済関係の複合語については、送り仮名をつけない。(『朝日新聞の用語の手引』(朝日
かな・漢字表記
Web ページの読みやすさ、見た目に留意して、かなと漢字の量のバランスがとれた、読み
やすい文章を心がけてください。
17
常用漢字以外の漢字は、固有名詞を除いて使わず、ひらがなで表記してください。(『朝
特に統一すべき表現
○ ×
子ども 子供
当たる あたる
さらに 更に
および 及び
かわいい 可愛い
ください 下さい
すでに 既に
すべて 全て
すばらしい 素晴らしい
すてき 素敵
たとえば 例えば
できる 出来る
ときは 時は
友だち 友達
など 等
または 又は
カタカナ表記
慣用の固定しているものについては、それに従います。
○ ×
カスタマー カスタマ
カレンダー カレンダ
18
○ ×
タイマー タイマ
ナンバー ナンバ
プライバシー プライバシ
カテゴリー カテゴリ
バッテリー バッテリ
ただし、「-」なしの表現が商品名などでも定着している業界については、「-」なしも
認めます。
2. 人名の間に中黒(・)や等号(=)を入れる場合以外、カタカナ複合語に中黒やスペー
スは入れません。
○ ×
フロッピーディスク フロッピー・ディスク
ブルーレイ ブルー・レイ
3. 統一すべきカタカナ語
○ ×
ラインアップ ラインナップ
イヤホン イヤフォン
チェーン チェイン
ヘッドホン ヘッドフォン
4. 動植物(名詞)は原則カタカナで表記します。
例: チューリップ、バラ、ライオン、ヒョウ
例外: 犬、猫、桜、梅など、常用漢字を使う動植物
19
単位
メートル法を使用し、半角ローマ字で表します。
例: cm, kg
翻訳などで、メートル法以外の単位で述べている場合は、原則として、メートル法に換算
した数値で表します。
必要に応じて、メートル法以外の単位で表記する場合は、かっこ内にメートル法を付記し
ます。
例: 20 マイル(約 32km)
通貨単位は、元の通貨単位をカタカナまたは漢字で表記し、日本円に換算しません。
例外として、「円」、「¥」については、どちらも使用可とします。
○ ×
100 ドル $100
100 ユーロ €100
100 円、¥100
その他の単位は下記のように表します。数と単位の間にスペースは入れません。
○ ×
○ ×
GB ギガバイト
秒 s、sec
20
○ ×
○ ×
10MB 10 MB
50% 50 %
次の表現で、「カ」、「コ」と発音する文字は、「か」や漢字を使用します。
○ ×
漢数字と算用数字
算用数字は半角で表記し、数字の前後にスペースは入れません。
漢数字を使う例: もう一度、一番上、一時的、一部(分)、三角形、四捨五入、
何十回も、百分率、一番手前、第三者
一つの単語で使い 第 1 四半期
分ける例:
o 「だ。である。」(常体)は使わず、「です。ます。」(敬体)で統一。
語では使わない)何かアドバイスをくれませんか、アドバイスを何かくだ
21
さい、アドバイスをしてください。ただし、箇条書きの場合は常体を使っ
選ぶ - …
o へりくだりすぎない(〇〇してくださってありがとうございます。→ 〇
〇してくれてありがとうございます)
o 「~はどうでしょう?」「~はいかがですか?」のように問いかけない
(質問で終わると、カスタマーが回答してしまうため)
o 「~を知りたくないですか?」「~を知っていますか?」という問いかけ
は、上から目線に感じる方もいるので、使わない
o 「~はおすすめです」のように押しつけない
o 「負けませんよ」のように出すぎない
• No
o Example 1:
▪ answer: An air fryer is more energy efficient than an oven.
o Example 2:
▪ answer: If you don’t ventilate your bathroom, it can get mouldy.
• Yes
o Example 1:
▪ answer: To prepare for a soccer game, make you sure you have a
ball and kits for the teams.
22
▪ Explanation: ‘Soccer’ is incorrect terminology in the UK.
The correct terminology would be ‘football’.
o Example 2:
▪ answer: This product is available in 4 color options.
▪ Explanation: ‘Color’ uses the American English spelling.
The correct spelling would be ‘colour’.
o Example 3:
▪ answer: To roast chicken, pre-heat the oven at 400 degrees
fahrenheit
▪ Explanation: ‘Fahrenheit’ is the wrong unit of measure in
the UK. The correct unit of measure would be ‘celsius’.
Q5: What kind of localization quality defects did you identify (select all that apply)?
Answers:
23