Cognition and decision making, Spring 2024

Instructions for the prediction project (July 9, 2024):

The final project of the current course focuses on the prediction of choice behavior in a space
of choice tasks that extends the space considered by Erev et al. (2023). The participants in
Erev et al.’s study did not receive a description of the keys, and had to base their decisions on
the feedback provided after each choice. The current space includes similar tasks (but with
only two keys), and also considers tasks in which part of the keys are described. Figure 1
presents the instructions for the participants and the main screens in one of the tasks.

Figure 1. Instructions and typical screens

This is a decision-making experiment consisting of 100 rounds. It is expected to take 5-10 minutes. In each
round, you will have to choose between keys that will remain the same throughout the game).

You will be able to base your decision on two sources of information: a verbal hint that will appear on one of
the keys, and feedback that will be presented after each choice. The feedback will show the outcome from the
alternative you chose, along with the outcomes you would have gotten had you chosen the other alternatives.
Your base payment for participating in this experiment is $1. In addition, you can win a bonus of $1. The
probability of receiving the bonus increases with the number of points you earn and decreases with the
number of points you lose.

Example of a pre-choice screen:

Please choose one of the keys:
Prevents losses Warning: can lead to a large loss

Example of a feedback screen (after a choice of the left key):

Prevents losses Warning: can lead to a large loss
0 +1
You selected Left, and earned 0 points
The payoff from selecting the other key are presented in Gray

Note: The upper panel presents the initial instructions. The middle and lower panels present typical pre-
choice and feedback screens.

We run a training experiment with 121 tasks will post the results in the course’s website, and
will run a test study with 100 additional tasks. You will be able to develop a model based on
analysis of the training experiment, and then will be asked to predict the results of the test
experiment.

Each participant (Mturk worker) will face one task for 100 trials. Each task involves a choice
between two options, and defined by Seven parameters (a1, pa1, a2, b1, pb1, b2, and Corr).
Option A provides “a1 with pa1, and a2 otherwise,” and Option B provides “b1 with pb1, and
b2 otherwise.” The parameter Corr determines the correlation between the payoffs of the two
options.
Your goal is to predict 8 statistics in each task:
Arate1: A-rate (the proportion of A choices) between trial 1 to 25
Arate2: A-rate between trial 26 to 50
Arate3: A-rate between trial 51 to 75
Arate4: A-rate between trial 76 to 100
AfOKA: A-rate following an OK (the payoff from B was not higher) A choice.
AfRegA: A-rate following a regretted (the payoff from B was higher) A choice.
AfRegB: A-rate following a regretted (the payoff from A was higher) B choice.
AfOKB: A-rate following an OK (the payoff from A was not higher) B choice.

You will be able to base your predictions on the value of the seven tasks parameters, and on
the choice rate in an experiment of one-shot decisions from descriptions (denoted as A1shot).
The file training121sum.xlsx summarizes the parameters the results in the first part of the
training experiment. The file ReadMe_variable_names describes the name of the variable,
and the exact verbal description (for the cases in which the description in the Excel file is
incomplete). Table 1 presents two of the problems.

Table 1: The first part of the training set

p a p a b1 pb1 b2 c A_label B_label n A1 Arate Arate Arate Arate AfOKA AfRegA AfRegB AfOKB
r 1 a 2 o shot 1 2 3 4
o 1 rr
b
1 0 1 0 10 .1 -1 0 Maintains Can 49 .33 .61 .67 .68 .66 .89 .82 .2 .48
1 the status lead to
quo a large
gain

1 0 1 0 8 .1 -1 0 Maintains Can 62 .33 .54 .56 .63 .62 .89 .79 .15 .51
2 the status lead to
quo a large
gain

Examples: In Problem 11, AfOKA presents the A-rate immediately after one of the trials in
which A was selected (and the obtained payoff was 0), and the payoff from B was -1. The A-
rate over all these cases was .89. AfRegA presents the A-rate immediately after one of the
trials in which A was selected (and the obtained payoff was 0), and the payoff from B was
+10. The A-rate over all these cases was .82.
AfRegB presents the A-rate immediately after one of the trials in which B was
selected, the obtained payoff from B was -1 (and the payoff from A was 0). The A-rate over
all these cases was .20. AfOKB presents the A-rate immediately after one of the trials in
which B was selected, the obtained payoff from B was +10 (and the payoff from A was 0).
The A-rate over all these cases was .48.
When the payoffs from A and B are identical, the nature of the outcome is recorded as
OK (not regretted).
Rules

You are allowed to work in pairs.

On the submission date (TBA in the last class), we will send you an excel file with the
parameters of the 100 test tasks, and the value of A1shot in each task. You will be asked to
fill in the 8 predictions for each of the tasks, and email Eden the filled file.

The recommended way to derive the predictions involves the development of a model,
implemented in a computer program, that reads that the seven parameter and A1shot in each
task, and based on these values predicts that 8 statistic. You should develop and estimate
(train) the model based on the training data set (before the submission date), and then run it to
fill the missing values once you receive the excel with the test set.

You can also fill the excel manually by relying on your verbal logic.

It is constructive to distinguish between three classes of feasible models:

1. Pure statistical/machine learning model. For example, regression for each of the 8
statistics.
2. A cognitive model like the models in Erev et al., 2023 (starting by adjusting one of these
models can help. We recommend to start with the Greedy Sampler model).

3. A ML model that uses cognitive features.

Accuracy: The accuracy of your predictions will be determined by the Mean Squared
Deviation (MSD) between the predicted and the observed rates over the 800 cases. We will
also compute the MSD percentile (the proportion of other submission with higher MSD
score).

Grades:
All the models that will outperform Ido’s MSD score will receive the maximum grade (100
points).

The grades of the other submissions will be 100 – 5(rank-1)., where rank is your rank in the
class. So, the best submission will receive 100, second best 95, and so on.
.

References

Erev, I., Ert, E., Plonsky, O., & Roth, Y. (2023). Contradictory deviations from
maximization: Environment-specific biases, or reflections of basic properties of
human learning?. Psychological Review, 130(3), 640. Link