Introduction to Information Retrieval Introduction to Information Retrieval

Recall the basic indexing pipeline

Documents to Friends, Romans, countrymen.
be indexed
Introduction to
Information Retrieval Tokenizer
Token stream Friends Romans Countrymen
Document ingestion Linguistic modules
Modified tokens friend roman countryman

Indexer friend 2 4
roman 1 2
Inverted index
countryman 13 16

Introduction to Information Retrieval Sec. 2.1 Introduction to Information Retrieval Sec. 2.1

Parsing a document Complications: Format/language

 What format is it in?  Documents being indexed can include docs from
many different languages
 pdf/word/excel/html?
 A single index may contain terms from many languages.
 What language is it in?  Sometimes a document or its components can
 What character set is in use? contain multiple languages/formats
 (CP1252, UTF-8, …)  French email with a German pdf attachment.
 French email quote clauses from an English-language
contract
Each of these is a classification problem, which
we will study later in the course.
 There are commercial and open source libraries that
But these tasks are often done heuristically … can handle a lot of this stuff

Introduction to Information Retrieval Sec. 2.1 Introduction to Information Retrieval

Complications: What is a document?

We return from our query “documents” but there are
often interesting questions of grain size: Introduction to
What is a unit document? Information Retrieval
 A file?
 An email? (Perhaps one of many in a single mbox file)
 What about an email with 5 attachments?
Tokens
 A group of files (e.g., PPT or LaTeX split over HTML pages)

1
Introduction to Information Retrieval Sec. 2.2.1 Introduction to Information Retrieval Sec. 2.2.1

Tokenization Tokenization
 Input: “Friends, Romans and Countrymen”  Issues in tokenization:
 Output: Tokens  Finland’s capital 
 Friends Finland AND s? Finlands? Finland’s?
 Romans
 Hewlett-Packard  Hewlett and Packard as two
 Countrymen tokens?
 A token is an instance of a sequence of characters  state-of-the-art: break up hyphenated sequence.
 Each such token is now a candidate for an index  co-education
 lowercase, lower-case, lower case ?
entry, after further processing
 It can be effective to get the user to put in possible hyphens
 Described below
 San Francisco: one token or two?
 But what are valid tokens to emit?  How do you decide it is one token?

Introduction to Information Retrieval Sec. 2.2.1 Introduction to Information Retrieval Sec. 2.2.1

Numbers Tokenization: language issues

 3/20/91 Mar. 12, 1991 20/3/91  French
 55 B.C.  L'ensemble  one token or two?
 B-52  L ? L’ ? Le ?
 My PGP key is 324a3df234cb23e  Want l’ensemble to match with un ensemble
 Until at least 2003, it didn’t on Google
 (800) 234-2333
 Internationalization!
 Often have embedded spaces
 Older IR systems may not index numbers  German noun compounds are not segmented
 But often very useful: think about things like looking up error
 Lebensversicherungsgesellschaftsangestellter
codes/stacktraces on the web
 ‘life insurance company employee’
 (One answer is using n-grams: IIR ch. 3)
 German retrieval systems benefit greatly from a compound splitter
 Will often index “meta-data” separately module
 Creation date, format, etc.  Can give a 15% performance boost for German

Introduction to Information Retrieval Sec. 2.2.1 Introduction to Information Retrieval Sec. 2.2.1

Tokenization: language issues Tokenization: language issues

 Chinese and Japanese have no spaces between  Arabic (or Hebrew) is basically written right to left,
words: but with certain items like numbers written left to
 莎拉波娃现在居住在美国东南部的佛罗里达。 right
 Not always guaranteed a unique tokenization  Words are separated, but letter forms within a word
 Further complicated in Japanese, with multiple form complex ligatures
alphabets intermingled
 Dates/amounts in multiple formats  ← → ←→ ← start
フォーチュン500社は情報不足のため時間あた$500K(約6,000万円)  ‘Algeria achieved its independence in 1962 after 132
Katakana Hiragana Kanji Romaji years of French occupation.’
 With Unicode, the surface presentation is complex, but the
End-user can express query entirely in hiragana! stored form is straightforward

2
Introduction to Information Retrieval Introduction to Information Retrieval Sec. 2.2.2

Stop words
 With a stop list, you exclude from the dictionary
Introduction to entirely the commonest words. Intuition:
Information Retrieval  They have little semantic content: the, a, and, to, be
 There are a lot of them: ~30% of postings for top 30 words
 But the trend is away from doing this:
Terms  Good compression techniques (IIR 5) means the space for including
stop words in a system is very small
The things indexed in an IR system  Good query optimization techniques (IIR 7) mean you pay little at
query time for including stop words.
 You need them for:
 Phrase queries: “King of Denmark”
 Various song titles, etc.: “Let it be”, “To be or not to be”
 “Relational” queries: “flights to London”

Introduction to Information Retrieval Sec. 2.2.3 Introduction to Information Retrieval Sec. 2.2.3

Normalization to terms Normalization: other languages

 We may need to “normalize” words in indexed text  Accents: e.g., French résumé vs. resume.
as well as query words into the same form  Umlauts: e.g., German: Tuebingen vs. Tübingen
 We want to match U.S.A. and USA  Should be equivalent
 Result is terms: a term is a (normalized) word type,  Most important criterion:
which is an entry in our IR system dictionary  How are your users like to write their queries for these
 We most commonly implicitly define equivalence words?
classes of terms by, e.g.,
 deleting periods to form a term  Even in languages that standardly have accents, users
 U.S.A., USA  USA often may not type them
 deleting hyphens to form a term  Often best to normalize to a de-accented term
 anti-discriminatory, antidiscriminatory  antidiscriminatory  Tuebingen, Tübingen, Tubingen  Tubingen

Introduction to Information Retrieval Sec. 2.2.3 Introduction to Information Retrieval Sec. 2.2.3

Normalization: other languages Case folding

 Normalization of things like date forms  Reduce all letters to lower case
 7月30日 vs. 7/30  exception: upper case in mid-sentence?
 Japanese use of kana vs. Chinese characters  e.g., General Motors
 Fed vs. fed
 SAIL vs. sail
 Tokenization and normalization may depend on the  Often best to lower case everything, since users will use
language and so is intertwined with language lowercase regardless of ‘correct’ capitalization…
detection Is this
Morgen will ich in MIT … German “mit”?  Longstanding Google example: [fixed in 2011…]
 Query C.A.T.
 Crucial: Need to “normalize” indexed text as well as
 #1 result is for “cats” (well, Lolcats) not Caterpillar Inc.
query terms identically

3
Introduction to Information Retrieval Sec. 2.2.3 Introduction to Information Retrieval

Normalization to terms Thesauri and soundex

 Do we handle synonyms and homonyms?
 An alternative to equivalence classing is to do  E.g., by hand-constructed equivalence classes
 car = automobile color = colour
asymmetric expansion  We can rewrite to form equivalence-class terms
 An example of where this may be useful  When the document contains automobile, index it under car-
 Enter: window Search: window, windows automobile (and vice-versa)
 Enter: windows Search: Windows, windows, window  Or we can expand a query
 Enter: Windows Search: Windows  When the query contains automobile, look under car as well

 Potentially more powerful, but less efficient  What about spelling mistakes?
 One approach is Soundex, which forms equivalence classes
of words based on phonetic heuristics
 More in IIR 3 and IIR 9

Introduction to Information Retrieval Introduction to Information Retrieval Sec. 2.2.4

Lemmatization
 Reduce inflectional/variant forms to base form
Introduction to  E.g.,
 am, are, is  be
Information Retrieval  car, cars, car's, cars'  car
 the boy's cars are different colors  the boy car be
Stemming and Lemmatization different color
 Lemmatization implies doing “proper” reduction to
dictionary headword form

Introduction to Information Retrieval Sec. 2.2.4 Introduction to Information Retrieval Sec. 2.2.4

Stemming Porter’s algorithm

 Reduce terms to their “roots” before indexing  Commonest algorithm for stemming English
 “Stemming” suggests crude affix chopping  Results suggest it’s at least as good as other stemming
 language dependent options
 e.g., automate(s), automatic, automation all reduced to  Conventions + 5 phases of reductions
automat.  phases applied sequentially
 each phase consists of a set of commands
 sample convention: Of the rules in a compound command,
for example compressed for exampl compress and select the one that applies to the longest suffix.
and compression are both compress ar both accept
accepted as equivalent to as equival to compress
compress.

4
Introduction to Information Retrieval Sec. 2.2.4 Introduction to Information Retrieval Sec. 2.2.4

Typical rules in Porter Other stemmers

 sses  ss  Other stemmers exist:
 ies  i  Lovins stemmer
 http://www.comp.lancs.ac.uk/computing/research/stemming/general/lovins.htm
 ational  ate  Single-pass, longest suffix removal (about 250 rules)
 tional  tion  Paice/Husk stemmer
 Snowball
 Weight of word sensitive rules  Full morphological analysis (lemmatization)
 (m>1) EMENT →  At most modest benefits for retrieval
 replacement → replac
 cement → cement

Introduction to Information Retrieval Sec. 2.2.4 Introduction to Information Retrieval Sec. 2.2.4

Language-specificity Does stemming help?

 The above methods embody transformations that  English: very mixed results. Helps recall for some
are queries but harms precision on others
 Language-specific, and often  E.g., operative (dentistry) ⇒ oper
 Application-specific  Definitely useful for Spanish, German, Finnish, …
 These are “plug-in” addenda to the indexing process  30% performance gains for Finnish!
 Both open source and commercial plug-ins are
available for handling these

Introduction to Information Retrieval Introduction to Information Retrieval Sec. 2.3

Recall basic merge

 Walk through the two postings simultaneously, in
Introduction to time linear in the total number of postings entries

Information Retrieval
2 4 8 41 48 64 128 Brutus
2 8
Faster postings merges: 1 2 3 8 11 17 21 31 Caesar
Skip pointers/Skip lists
If the list lengths are m and n, the merge takes O(m+n)
operations.

Can we do better?
Yes (if the index isn’t changing too fast).

5
Introduction to Information Retrieval Sec. 2.3 Introduction to Information Retrieval Sec. 2.3

Augment postings with skip pointers

(at indexing time) Query processing with skip pointers
41 128 41 128
2 4 8 41 48 64 128 2 4 8 41 48 64 128

11 31 11 31
1 2 3 8 11 17 21 31 1 2 3 8 11 17 21 31

 Why? Suppose we’ve stepped through the lists until we process 8

on each list. We match it and advance.
 To skip postings that will not figure in the search
results. We then have 41 and 11 on the lower. 11 is smaller.
 How?
But the skip successor of 11 on the lower list is 31, so
 Where do we place skip pointers? we can skip ahead past the intervening postings.

Introduction to Information Retrieval Sec. 2.3 Introduction to Information Retrieval Sec. 2.3

Where do we place skips? Placing skips

 Tradeoff:  Simple heuristic: for postings of length L, use L
 More skips  shorter skip spans  more likely to skip. evenly-spaced skip pointers [Moffat and Zobel 1996]
But lots of comparisons to skip pointers.  This ignores the distribution of query terms.
 Fewer skips  few pointer comparison, but then long skip
spans  few successful skips.
 Easy if the index is relatively static; harder if L keeps
changing because of updates.

 This definitely used to help; with modern hardware it

may not unless you’re memory-based [Bahle et al. 2002]
 The I/O cost of loading a bigger postings list can outweigh
the gains from quicker in memory merging!

Introduction to Information Retrieval

35