Supporting users in finding successful matches in reciprocal recommender systems

Abstract

Online platforms which assist users in finding a suitable match, such as online-dating and job recruiting environments, have become increasingly popular in the last decade. Many of these environments include recommender systems which, for instance in online dating, aim at helping users to discover a suitable partner who will likely be interested in them. Generating successful recommendations in such systems is challenging as the system must balance two objectives: (1) recommending users with whom the recommendation receiver is likely to initiate an interaction and (2) recommending users who are likely to reply positively to the recommendation receiver initiated interaction. Unfortunately, these objectives are partially conflicting since very often the recommendation receiver is likely to contact users who are not likely to respond positively, and vice versa. Furthermore, users in these environments vary in the extent to which they contemplate the other side’s preferences before initiating an interaction. Therefore, an effective recommender system must effectively model each user and balance these objectives. In our work, we tackle this challenge through two novel components: (1) an explanation module, which leverages an estimate of why the recommended user is likely to respond positively to the recommendation receiver; and (2) a novel reciprocal recommendation algorithm, which finds an optimal balance, individually tailored to each user, between the partially conflicting objectives mentioned above. In an extensive empirical evaluation, in both simulated and real-world dating Web platforms with 1204 human participants, we find that both components contribute to attaining these objectives and that the combinations thereof are more effective than each one on its own.

Appendices

Appendix 1: User experience questionnaire for evaluation of reciprocal explanations

Our questionnaire, which evaluated the effect of explanation on the user experience (Sect. 5), included 5 Likert-scale questions, with a scale ranging from 1 (“strongly disagree”) to 5 (“strongly agree”). These questions measured five prominent factors of user experience in recommender systems: user satisfaction from the recommendations, perceived competence of the system, perceived transparency of the system, and trust in the system (Cramer et al. 2008; Pu et al. 2011; Knijnenburg et al. 2012). In addition, the users were asked specifically about the explanation usefulness, namely the extent to which the users considered the explanations to be helpful. The questions are presented in Table 5. The second question, which is ‘negatively worded’, was reverse-scored Hartley (2014). In order to make sure that the different questions actually evaluate different measures, we calculated the cross-scale Pearson correlation coefficients Goodman (2001) which show that the answers to the questions are not strongly correlated. The full correlation table is presented in Table 6.

Table 5 User experience questionnaire

Table 6 Cross-scale Pearson correlation coefficients

1.1 Questionnaire results in the simulated environment

In this section, we present the results of the questionnaire, which evaluated the user experience in the simulated environment, for both negligible cost and explicit cost settings.

1.1.1 Negligible cost

In the negligible cost setting, the one-sided condition outperformed the reciprocal condition also in the user experience, as in the relevance measure (Sect. 5.2.1). Specifically, the satisfaction (${\mathrm{mean}}= 4$, ${\mathrm{s.d.}}= 0.85$ vs. ${\mathrm{mean}}=3.57$, ${\mathrm{s.d.}}=0.86$, $p\le 0.05$) and perceived competence (${\mathrm{mean}}= 4.13$, ${\mathrm{s.d.}}= 0.83$ vs. ${\mathrm{mean}}= 3.27$, ${\mathrm{s.d.}}= 0.9$, $p\le 0.01$) were found to be significantly superior for the one-sided explanation condition. No statistically significant difference was found between the conditions for the remaining measures. The results are presented in Fig. 13.

Fig. 13

Reciprocal versus one-sided explanations in MM with negligible cost. Error bars represent the standard error

1.1.2 Explicit cost

In the explicit cost setting, in addition to the acceptance (Sect. 5.2.2), the participants’ trust in the system was found to be higher under the reciprocal explanation condition (one-sided: ${\mathrm{mean}}=2.93$, ${\mathrm{s.d.}}=1.14$ vs. reciprocal: ${\mathrm{mean}}=3.38$, ${\mathrm{s.d.}}=1.01$, $p\le 0.05$). No statistically significant difference was found between the conditions for the remaining measures. The results are presented in Fig. 14.

Fig. 14

Reciprocal versus one-sided explanations in MM with explicit cost. Error bars represent the standard error

Appendix 2: Choosing the explanation method

Before the evaluation of one-sided and reciprocal explanations in RRSs, we performed a preliminary investigation in order to find the best suited explanation method for online dating, the domain on which we focus throughout this paper.

1.1 Comparison of correlation-based and transparent explanation methods

In addition to the correlation-based explanation method, which is described in Sect. 4, we designed a similar explanation method based on the same guidelines (described above in Sect. 2.1). We called this explanation method the “transparent” explanation method.

The transparent explanation method, which aims to reflect the actual reasoning for the recommendations provided by the RECON algorithm, works as follows: to explain to user x a recommendation of user y, the method returns the top-k attributes of y which are the most prominent among users who received a message from user x.

To illustrate the difference between the transparent and the correlation-based explanation methods, we revisit Example 1. Assume an RRS has decided to recommend Alice, who never smokes and is slim, to Bob. Recall that Bob sent 6 messages to users who never smoke and 4 to slim users. For $k=1$, the transparent explanation method would provide “never smoke” as an explanation because Bob sent more messages to users who never smoke than to users who are slim. Now say Bob viewed a total of 25 users, of whom 18 never smoke and 4 were slim. In other words, Bob sent messages to only a third of the users he viewed who never smoke, and to all users he viewed who are slim. Thus, the correlation-based method would find a stronger correlation between the presence of “slim body” and Bob’s messaging behavior; hence, “slim body” would be provided as an explanation.

In order to compare the transparent and the correlation-based explanation methods, we used the MM simulated system discussed in Sect. 5.1. We asked 59 of the 118 participants who took part in the data collection phase and did not take part in the negligible-cost experiment (Sect. 5) to enter the MM platform, where each participant then received a list of five personal recommendations generated by the RECON algorithm along with either transparent explanations (30 participants) or correlation-based explanations (29 participants). Participants were randomly assigned to one of the two conditions. As in the negligible-cost experiment, participants were asked to rate the relevance of each recommendation separately, on a five-point Likert scale from 1 (extremely irrelevant) to 5 (extremely relevant). Next, participants answered the questionnaire (available in Appendix 1), debriefing them on their user experience.

1.1.1 Results

All collected data were found to be approximately normally distributed according to the Anderson-Darling normality test Razali and Wah (2011). All reported results were compared using an unpaired t test. The results from the questionnaire showed that participants in the correlation-based condition were more satisfied than those in the transparent explanation condition and perceived the system as more useful ($p\le 0.05$). We did not find a significant difference in the way participants rated the relevance of the provided recommendations nor did we find a significant difference in the reported trust of the system.

1.2 Additional explanation methods evaluated in the MM environment

1.2.1 Comparison to baseline

Prior to our main experiment, we first compared the correlation-based explanations with a baseline condition: recommendations without any explanation. We recruited an additional group of 30 participants who were asked to enter the mm environment. We used the same experimental methodology described in Sect. 5. We measured all evaluation measures with the exception of the explanation usefulness (which was not relevant to the baseline condition). We found that the correlation-based condition significantly outperformed the baseline condition in the relevance measure ($p\le 0.05$).

1.2.2 Comparison to collaborative filtering explanation style

We further examined another explanation method, similar to a method which was presented in previous work Herlocker et al. (2000). This explanation method justifies the recommendation by simply stating that “similar users” to the service user have shown interest in the recommended match. We call this explanation style “collaborative filtering,” because the explanation indicates that the recommendation was generated using collaborative filtering methodology, where recommendations are based on similarity measures. Unlike the previous methods, these explanations do not include any information about the attributes of the recommended users. Of course, this explanation does not reflect the actual reasoning for the recommendation, since the underlying algorithm is content-based. Nevertheless, previous work has shown that explanations which are not related to the underlying algorithm can also be highly effective Herlocker et al. (2000).

For the evaluation of this explanation method, we recruited an additional group of 25 subjects. All of the experimental setup was identical to the setup in the previous experiment, with the only difference being the explanation method.

Our results show that the correlation-based recommendation method was significantly superior to the collaborative filtering explanation style. Specifically, the relevance rate in the correlation-based condition was significantly higher than the collaborative filtering (correlation: mean = 3.34 vs. collaborative filtering: mean = 2.36, $p\le 0.01$). In addition, the subjects in the experiment with the correlation-based method were significantly more satisfied than the users in the experiment with the collaborative filtering method (correlation: mean = 4 vs. collaborative filtering: mean = 3.28, $p\le 0.01$).

Appendix 3: Features in the reply prediction model

Public Profile Features of Sender and Receiver:

These features are part of the users’ profile and are public to all the users in the environment.

• 1. Age

• 2. Gender

• 3. Marital status

• 4. Number of children

• 5. Height

• 6. Smoking habits

• 7. Number of pictures in profile

• 8. Are pictures public? (The users on the site had an option of keeping their pictures private)

• 9. Religious observance level

• 10. Dating goal

• 11. Living area

• 12. Self-description length (Number of characters in the user’s self description)

• 13. Preferences description length (Number of characters in the user’s description of his/her preferences)

• 14. Economic status

• 15. Ethnic background

Each feature corresponds to two features in the model—one for the sender and one for the receiver.

Interaction and Activity features of Sender:

• 1. Number of profiles he/she viewed.

• 2. Number of users who viewed him/her.

• 3. Number of users he/she liked.

• 4. Number of users who liked him/her.

• 5. Number of messages he/she sent.

• 6. Number of messages he/she received.

• 7. Number of his/her messages which were positively replied to before current message.

• 8. Percent of positively replied messages before current message.

• 9. Number of received messages which he/she did not view.

• 10. Number of users who viewed him/her which he/she did not view.

• 11. Number of users who liked him/her which he/she did not view.

Interaction and Activity features of Recipient:

• 1. Number of users who viewed him/her.

• 2. Number of users he/she viewed.

• 3. Number of users who liked him/her.

• 4. Number of users he/she liked.

• 5. Percent of messages he/she replied to positively from all received messages.

• 6. Number of messages he/she received.

• 7. Did he/she send a message to the sender before?

• 8. Did he/she like the sender before?

• 9. Has he/she replied positively to any message before?

• 10. Was he logged-in while the message was received?

• 11. Log-ins to the environment in the week before the message.

• 12. Average duration of logins in previous week (before the message).

• 13. Number of sent messages in previous week (before the message).