Scribd
Upload
51 views2 pages

PR Example Sol

An IR system retrieves documents for a query where the relevant documents are d3, d12, d15, d21, d22, and d30. The precision and recall are calculated at each retrieved document. Interpolated precision is calculated at standard recall levels by taking the maximum precision between levels. R-precision, which is precision at the number of relevant documents retrieved, is 0.5. Coverage ratio and novelty ratio are calculated for two users based on which documents they consider relevant.

Uploaded by

Rihab BEN LAMINE
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
51 views2 pages

PR Example Sol

An IR system retrieves documents for a query where the relevant documents are d3, d12, d15, d21, d22, and d30. The precision and recall are calculated at each retrieved document. Interpolated precision is calculated at standard recall levels by taking the maximum precision between levels. R-precision, which is precision at the number of relevant documents retrieved, is 0.5. Coverage ratio and novelty ratio are calculated for two users based on which documents they consider relevant.

Uploaded by

Rihab BEN LAMINE
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Given a query q, where the relevant documents are d3, d12, d15, d21, d22, and d30, an IR

system retrieves the following ranking: d5, d3, d21, d36, d30, d45, d80, d28, d23, and
d12.

1. What are the precision and recall values at each retrieved document for this ranking?

Recall Precision
d5 0/6 = 0 0/1=0
d3 1/6 = 0.166 1/2 = 0.50
d21 2/6 = 0.33 2/3 = 0.66
d36 2/6 = 0.33 2/4 = 0.50
d30 3/6 = 0.50 3/5 = 0.60
d45 3/6 = 0.50 3/6 = 0.50
d80 3/6 = 0.50 3/7 = 0.42
d28 3/6 = 0.50 3/8 = 0.37
d23 3/6 = 0.50 3/9 = 0.33
d12 4/6 = 0.66 4/10 = 0.40

2. Interpolate the precision scores at 11 recall levels. Note: The interpolated precision at
the j-th standard recall level is the maximum known precision at any recall level between
the j-th and (j + 1)-th level: P( rj ) = max P( r )
r j ≤ r ≤ r j +1

Recall Interpolated Precision


0% 0.66
10% 0.66
20% 0.66
30% 0.66
40% 0.6
50% 0.6
60% 0.4
70% 0
80% 0
90% 0
100% 0

Explanation (uP is the uninterpolated precision):


P(100%)=max 100%<= r <=100% {uP(100%)=0}=0
P(90%)=max 90%<= r <=100% {P(100%)=0}=0
P(80%)=max 80%<= r <=90% { P(90%)=0}=0
P(70%)=max 70%<= r <=80% { P(80%)=0}=0
P(60%)=max 60%<= r <=70% {uP(66%)=0.40, P(70%)=0} =0.40
P(50%)=max 50%<= r <=60% {uP(50%)=0.60, P(60%)=0.40} =0.60
P(40%)=max 40%<= r <=50% {P(50%)=0.60} =0.60
P(30%)=max 30%<= r <=40% {uP(33%)=0.66, P(40%)=0.60}=0.66
P(20%)=max 20%<= r <=30% {P(30%)=0.66}=0.66
P(10%)=max 10%<= r <=20% {uP(16.6%)=0.50, P(20%)=0.66}=0.66
P(0%) =max 0%<= r <=10% {P(10%)=0.66}=0.66

3.Why is interpolation of precision scores necessary when evaluation an IR system?


To be able to evaluate the precision for many queries at the same levels of recall (queries
tend to have different recall points).

4. What is the R-precision? (precision at first R retrieved documents where R is the total
number of relevant documents)

R-Precision 3/6 = 0.5

5. If we have two users, the first user decided that d12, d15, d21 are relevant to the query,
and the second user decided that d3, d12, d15, d21, d22 are relevant to the query, what is
the coverage ratio and the novelty ratio for this user? (Remember that the coverage ratio
is the proportion of relevant items retrieved out of the total relevant documents known to
a user prior to the search. The novelty ratio is the proportion of retrieved items, judged
relevant by the user, of which they were previously unaware.)

Coverage ratio Novelty ratio


User 1 2/3 = 0.66 2/4 = 0.50
User 2 3/5 = 0.60 1/4 = 0.25

Explanation:
User1 knows 3 documents (d12, d15, d21). The system retrieves 4 relevant documents, 2
of them are the ones expected by user1 (d12, d21). Therefore the coverage ratio for user1
is 2 retrieved out of 3 expected. Among the 4 relevant documents retrieved by the system
2 are known to user1 and 2 are new. So the novelty ratio for user1 is 2 out of 4.
For user2, please compute yourself.

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy