24小时全年无休客服
本公司旗下写手皆为公司直聘 网络写手勿扰
案例中心
CMPT 456 Assignment
Assignment
Hello, dear friend, you can consult us at any time if you have any questions, add WeChat: THEend8_
CMPT 456 Assignment
Question 1 (25 points)
This question is about Kappa statistic. Consider the following judgements by two judges A and B.
A
Yes No
B Yes 320 480
No 550 150
Please calculate the Kappa statistic and explain your calculation. Are the two judges in agreement?
Question 2 (25 points)
This question is about NDCG. Suppose a 5-point scale is used in relevance judgements, and a
ranking of answers to a query is: 4, 3, 1, 0, 3, 2, 1, 4, 0, 2. What is the value of NDCG @ 10?
Question 3 (25 points)
Consider an information need for which there are 4 relevant documents in the collection.
Contrast two systems run on this collection. Their top 10 results are judged for relevance as
follows (the leftmost item is the top ranked search result):
System 1 R N R N N N N N R R
System 2 N R N N R R R N N N
1. What is the MAP of each system? Which has a higher MAP?
2. What is the R-precision of each system? (Does it rank the systems the same as MAP?)
Question 4 (25 points)
This question is about break-even point. Recall that the break-even point is the point where the
precision and the recall on the top-n returned answers are identical. What is the relationship
between the value of F1 and the break-even point? Must there always be a break-even point
between precision and recall? Either mathematically prove that there must be or give a
counterexample.