Trjtdtk

I'm currently reading

Hu, Koren, Volinsky: Collaborative Filtering for Implicit Feedback Datasets

One thing that confuses me is the "expected percentile ranking", an function the authors define to evaluate the goodness of their recommendations. They define it in the Evaluation methodology on page 6 as:

$$overlinetextrank = fracsum_u,i r^t_ui textrank_uisum_u,i r^t_ui$$

where $u$ is a user, $i$ is an item (e.g. a TV show), $r_ui in [0, infty)$ is the amount how much user $u$ did watch show $i$. $textrank_ui in [0, 1]$ is the percentage rank of item $i$ for user $u$. For example, it is 0 if for user $u$ the item $i$ has the highest $r$ value and 1 if the item $i$ for user $u$ has the lowest $r$ value.

I'm not super sure if I understood it correctly.

The authors write that lower values of $overlinetextrank$ are more desirable and for random predictions would lead to an expected value of $overlinetextrank$ of 0.5.

Examples

• Assume there is only one item. In this case $textrank = 0$. Makes sense, as there cannot be any predictions.


• Assume there is only one user and two items with $r_1,1 = 1$ and $r_1,2 = 2$. Then:

$$overlinetextrank = frac1 cdot textrank_1, 1 + 2 cdot textrank_1, 21+2$$

This means $overlinetextrank in 2/3, 1/3$.

• If there is only a single user and all $|I|$ values of $r_ui$ are the same, then $overlinetextrank = sum_ui textrank_ui = frac2$
  

Questions

1. Is my understanding of the metric correct? Especially my last example and the statement by the authors that $overlinetextrank geq 50%$ indicated an algorithm is no better than random seem off.


2. What is $t$?

What is $t$?

It means observed $r_ui$ in the one-week test set (page 6-left).

Is my understanding of the metric correct?

First two examples are correct. Assuming user-item relation $r_ui^t$ is constant $a$ for all items in the test set, and predicted ranks are uniform across $[0, 1]$, then, the third one would be:

$$overlinetextrank = fracsum_u,i r^t_ui textrank_uisum_u,i r^t_ui=fracsum_u,i a text rank_uisum_u,i a=frac1sum_u,i text rank_ui=frac1frac2=frac12$$
This makes sense. Items are identical to the user, therefore no model can do better than random guessing, since there is no observed preference to help the model favor one item over the other. Of course, another assumption here is that training (4 weeks) and test (next week) sets are from the same distribution.