Tutorial#
Structure of Arduino code#
Here we will go over the basic structure of the FED3Bandit task. We will use the example in the FED3 arduino library for this explanation.
Sample code#
Here is the sample code:
1 #include <FED3.h> //Include the FED3 library
2 String sketch = "Bandit"; //Unique identifier text for each sketch, change string only.
3 FED3 fed3 (sketch); //Start the FED3 object - don't change
4
5 int pellet_counter = 0; //pellet counter variable
6 int timeoutIncorrect = 10; //timeout duration in seconds, set to 0 to remove the timeout
7 int probs[2] = {80,20}; //Reward probability options
8 int new_prob = 0;
9
10 void setup() {
11 fed3.countAllPokes = false;
12 fed3.LoRaTransmit = false;
13 fed3.pelletsToSwitch = 30; // Number of pellets required to finish the block and change reward probabilities
14 fed3.prob_left = 80; // Initial reward probability of left poke
15 fed3.prob_right = 20; // Initial reward probability of right poke
16 fed3.allowBlockRepeat = false; // Whether the same probabilities can be used for two blocks in a row
17 fed3.begin(); // Setup the FED3 hardware, all pinmode screen etc, initialize SD card
18 }
19
20 void loop() {
21 /////////////////////////////////////////////////////////////////////
22 // This is the main bandit task.
23 // In general it will be composed of three parts:
24 // 1. Set up conditions to trigger a change in reward probabilities
25 // 2. Set up behavior upon a left poke
26 // 3. Set up behavior upon a right poke
27 /////////////////////////////////////////////////////////////////////
28 fed3.run(); //Call fed.run at least once per loop
29
30 // This is part 1.
31 if (pellet_counter == fed3.pelletsToSwitch) {
32 pellet_counter = 0;
33 new_prob = probs[random(0,2)];
34 if (! fed3.allowBlockRepeat) {
35 while (new_prob == fed3.prob_left) {
36 new_prob = probs[random(0,2)];
37 }
38 }
39 fed3.prob_left = new_prob;
40 fed3.prob_right = 100 - fed3.prob_left;
41 }
42
43 // This is part 2.
44 if (fed3.Left) {
45 fed3.BlockPelletCount = pellet_counter;
46 fed3.logLeftPoke(); //Log left poke
47 delay(1000);
48 if (random(100) < fed3.prob_left) { //Select a random number between 0-100 and ask if it is between 0-80 (80% of the time). If so:
49 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
50 fed3.Feed(); //Deliver pellet
51 pellet_counter ++; //Increase pellet counter by one
52 }
53 else { //If random number is between 81-100 (20% of the time)
54 fed3.Tone(300, 600); //Play the error tone
55 fed3.Timeout(timeoutIncorrect, true, true);
56 }
57 }
58
59 // This is part 3.
60 if (fed3.Right) {
61 fed3.BlockPelletCount = pellet_counter;
62 fed3.logRightPoke(); //Log Right poke
63 delay(1000);
64 if (random(100) < fed3.prob_right) { //Select a random number between 0-100 and ask if it is between 80-100 (20% of the time). If so:
65 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
66 fed3.Feed(); //Deliver pellet
67 pellet_counter ++; //Increase pellet counter by one
68 }
69 else { //If random number is between 0-80 (80% of the time)
70 fed3.Tone(300, 600); //Play the error tone
71 fed3.Timeout(timeoutIncorrect, true, true);
72 }
73 }
74 }
This example shows a simple 2-armed bandit task. Here, the reward probabilities of left and right always add 100, and change simultaneously. Thus, this is a special case of the 2-armed bandit task that is equivalent to a probabilistic reversal task. This simple version of the bandit task, however, contains all the fundamental elements for a more sophisticated task. Broadly, FED3Bandit Arduino code is composed of two parts. First, variable set up. Second, bandit task setup. The task setup part can be further considered in 3 parts:
Reward probability switching
Behavior after left poke
Behavior after right poke
In the following subsection we will dissect each of the parts of the FED3Bandit bakbone, and in the next section we will show how to customize each of these parts.
Variable setup#
From the code above, these is where all variables are set up:
1 #include <FED3.h> //Include the FED3 library
2 String sketch = "Bandit"; //Unique identifier text for each sketch, change string only.
3 FED3 fed3 (sketch); //Start the FED3 object - don't change
4
5 int pellet_counter = 0; // pellet counter variable
6 int timeoutIncorrect = 10; // duration in seconds, set to 0 to remove the timeout
7 int probs[2] = {80,20}; // probability options
8 int new_prob = 0; //
9
10 void setup() {
11 fed3.countAllPokes = false; // Whether all pokes are counter
12 fed3.LoRaTransmit = false; // Wireless data transmission (future implementation)
13 fed3.pelletsToSwitch = 30; // Number of pellets required to finish the block and change reward probabilities
14 fed3.prob_left = 80; // Initial reward probability of left poke
15 fed3.prob_right = 20; // Initial reward probability of right poke
16 fed3.allowBlockRepeat = false; // Whether the same probabilities can be used for two blocks in a row
17 fed3.begin(); // Setup the FED3 hardware, all pinmode screen etc, initialize SD card
18 }
If you have experience with Arduino programming, this should look very familiar.
The first block of code includes the FED3 library, and creates a FED3 object with a “Bandit” identifier. If you are using any version
of the bandit task, make sure not to modify the value of sketch, as this initializes sessiontype=="Bandit" which has unique features
that will not work is sketch has a different value.
In the second block of code, all variables that are particular to this sketch are declared/initialized. These variables are necessary for the proper bandit task functioning, but may look different for each version of the task.
Finally, in the third block of code variables that are contained within the FED3 library are initialized. These variables are essential for any version of FED3Bandit and are doing some work under the hood for all FED3Bandit functions to work properly (specially the logdata() function). You may modify the value of these variables. For further reference see ARDUINO DOCUMENTATION
Task setup#
Now that we have discussed the declaration and initialization of all the necessary variables, let’s discuss the task set up. Here’s the code of the task:
1 void loop() {
2 /////////////////////////////////////////////////////////////////////
3 // This is the main bandit task.
4 // In general it will be composed of three parts:
5 // 1. Condition(s) to trigger a change in reward probabilities
6 // 2. Behavior upon a left poke
7 // 3. Behavior upon a right poke
8 /////////////////////////////////////////////////////////////////////
9 fed3.run(); //Call fed.run at least once per loop
10
11 // This is part 1.
12 if (pellet_counter == fed3.pelletsToSwitch) {
13 pellet_counter = 0;
14 new_prob = probs[random(0,2)];
15 if (! fed3.allowBlockRepeat) {
16 while (new_prob == fed3.prob_left) {
17 new_prob = probs[random(0,2)];
18 }
19 fed3.prob_left = new_prob;
20 fed3.prob_right = 100 - fed3.prob_left;
21 }
22 else {
23 fed3.prob_left = new_prob;
24 fed3.prob_right = 100 - fed3.prob_left;
25 }
26 }
27
28 // This is part 2.
29 if (fed3.Left) {
30 fed3.BlockPelletCount = pellet_counter;
31 fed3.logLeftPoke(); //Log left poke
32 delay(1000);
33 if (random(100) < fed3.prob_left) { //Select a random number between 0-100 and ask if it is between 0-80 (80% of the time). If so:
34 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
35 fed3.Feed(); //Deliver pellet
36 pellet_counter ++; //Increase pellet counter by one
37 }
38 else { //If random number is between 81-100 (20% of the time)
39 fed3.Tone(300, 600); //Play the error tone
40 fed3.Timeout(timeoutIncorrect, true, true);
41 }
42 }
43
44 // This is part 3.
45 if (fed3.Right) {
46 fed3.BlockPelletCount = pellet_counter;
47 fed3.logRightPoke(); //Log Right poke
48 delay(1000);
49 if (random(100) < fed3.prob_right) { //Select a random number between 0-100 and ask if it is between 80-100 (20% of the time). If so:
50 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
51 fed3.Feed(); //Deliver pellet
52 pellet_counter ++; //Increase pellet counter by one
53 }
54 else { //If random number is between 0-80 (80% of the time)
55 fed3.Tone(300, 600); //Play the error tone
56 fed3.Timeout(timeoutIncorrect, true, true);
57 }
58 }
59 }
As previously mentioned, the body of the FED3Bandit task consists of three parts:
Conditions to trigger a change in reward probabilities:
1 // This is part 1.
2 if (pellet_counter == fed3.pelletsToSwitch) {
3 pellet_counter = 0;
4 new_prob = probs[random(0,2)];
5 if (! fed3.allowBlockRepeat) {
6 while (new_prob == fed3.prob_left) {
7 new_prob = probs[random(0,2)];
8 }
9 fed3.prob_left = new_prob;
10 fed3.prob_right = 100 - fed3.prob_left;
11 }
12 else {
13 fed3.prob_left = new_prob;
14 fed3.prob_right = 100 - fed3.prob_left;
15 }
16 }
In this example, reward probabilities change when the mouse have obtained 30 pellets (fed3.pelletsToSwitch = 30).
pellet_counter is the variable that tracks the number of pellets that have been received in the current block.
After 30 pellets have been received, pellet_counter goes back to zero, a new probability from the reward
probability options probs is then randomly chosen (in this case there are only two options, 0 or 80).
Since fed3.allowBlockRepeat was set to false, a new probability will keep being chosen until new_prob is
different from fed3.prob_left, and this will be the new value of fed3.prob_left.
In other words, since there are only two possible probabilities, fed3.prob_left will always be
80 -> 20 -> 80 -> .... In this case, the new reward probability of right will always be 100-fed3.prob_left.
Leading to the following behavior
Block |
fed3.prob_left |
fed3.prob_right |
|---|---|---|
1 |
80 |
20 |
2 |
20 |
80 |
3 |
80 |
20 |
And so on. This is behavior is identical to a probabilistic reversal task, showing that this task is a special case of a two-armed bandit task.
Behavior after left poke:
1 // This is part 2.
2 if (fed3.Left) {
3 fed3.BlockPelletCount = pellet_counter;
4 fed3.logLeftPoke(); //Log left poke
5 delay(1000);
6 if (random(100) < fed3.prob_left) { //Select a random number between 0-100 and ask if it is between 0-80 (80% of the time). If so:
7 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
8 fed3.Feed(); //Deliver pellet
9 pellet_counter ++; //Increase pellet counter by one
10 }
11 else { //If random number is between 81-100 (20% of the time)
12 fed3.Tone(300, 600); //Play the error tone
13 fed3.Timeout(timeoutIncorrect, true, true);
14 }
15 }
Here, when the rodent pokes left (fed3.Left == true), fed3.BlockPelletCount is first updated and
the left poke is logged. Then, after a delay of one second (delay(1000)), an int between 0 and 100
is selected.
If the integer is smaller than fed3.prob_left, then a short tone will be played
(fed3.ConditionedStimulus()) and a pellet will be delivered (fed3.Feed()). Due to the inner working
of the feeding funciton, program will stay in this function until the pellet is retrieved.
After the pellet is retrieved, the pellet_counter will be updated.
If the integer is greater than fed3.prob_left, an error tone will be played (fed3.Tone(300, 600)) and
a time out of duration timeoutIncorrect will be triggered. Since the other two argument in the time out
function are true, true, this means that the time out will reset if rodent pokes during time out, and that
pokes that occur during timeout will not be counted on the FED3 screen.
In average, the random integer will be smaller than fed3.prob_left fed3.prob_left percent of the times.
For example if fed3.prob_left=80, a pellet will be delivered 80% of the times, in average.
Behavior after right poke
In this example, behavior after a right poke follows the same logic as the behavior after a left poke.
Customizing Task#
Clearly, a bandit task can be customized in multiple ways. Here we describe a few customization examples. The goal of this section is to develop intuition of the FED3Bandit structure. Recipes for different versions of the bandit task can be found in the HOW-TO GUIDES section. All customizations are modifications of the sample task described above. All modifications of the sample task are highlighted.
Reward probabilities from a normal distribution#
As we described in the overview section, reward probabilities can be a fixed number or it can come from a distribution. Let’s say that we want to use the same task as our previous example, but now we want the reward probabilities to come from a normal distribution. Let’s modify the setup of the variables:
1 #include <FED3.h> //Include the FED3 library
2 #include <random>
3 String sketch = "Bandit"; //Unique identifier text for each sketch, change string only.
4 FED3 fed3 (sketch); //Start the FED3 object - don't change
5
6 int pellet_counter = 0; // pellet counter variable
7 int timeoutIncorrect = 10; // duration in seconds, set to 0 to remove the timeout
8 int probs_mean[2] = {80,20}; // probability options
9 int probs_std = 10:
10 int new_prob = 0; //
11
12 void setup() {
13 fed3.countAllPokes = false; // Whether all pokes are counter
14 fed3.LoRaTransmit = false; // Wireless data transmission (future implementation)
15 fed3.pelletsToSwitch = 30; // Number of pellets required to finish the block and change reward probabilities
16 fed3.prob_left = 80; // Initial reward probability of left poke
17 fed3.prob_right = 20; // Initial reward probability of right poke
18 fed3.allowBlockRepeat = false; // Whether the same probabilities can be used for two blocks in a row
19 fed3.begin(); // Setup the FED3 hardware, all pinmode screen etc, initialize SD card
20 }
21
22 std::default_random_engine generator;
23 std::normal_distribution<float> distribution_left(fed3.prob_left, probs_std);
24 std::normal_distribution<float> distribution_right(fed3.prob_lright, probs_std);
First, notice that we included the <random> library.
We have also changed int probs[2] = {80,20} to int probs_mean[2] = {80,20} to reflect that these will not
be fixed but the mean of the normal distribution. Similarly, we added a new variable int probs_std = 10 that
will be the standard deviation of the normal distribution. Finally, we create a random_engine
instance (generator), which will help us select a random number from the normal distribution, and two normal
distributions (distribution_left and distribution_right).
Since we want to change the mean of the distribution after the block switchin condition has been met (in this case after 30 pellets have been delivered) we modify the condition section as follows:
1 // This is part 1.
2 if (pellet_counter == fed3.pelletsToSwitch) {
3 pellet_counter = 0;
4 new_prob = probs[random(0,2)];
5 if (! fed3.allowBlockRepeat) {
6 while (new_prob == fed3.prob_left) {
7 new_prob = probs[random(0,2)];
8 }
9 fed3.prob_left = new_prob;
10 fed3.prob_right = 100 - fed3.prob_left;
11 std::normal_distribution<float> distribution_left(fed3.prob_left, probs_std);
12 std::normal_distribution<float> distribution_right(fed3.prob_lright, probs_std);
13 }
14 else {
15 fed3.prob_left = new_prob;
16 fed3.prob_right = 100 - fed3.prob_left;
17 std::normal_distribution<float> distribution_left(fed3.prob_left, probs_std);
18 std::normal_distribution<float> distribution_right(fed3.prob_lright, probs_std);
19 }
20 }
Here instead of just changing the value of probs_left and probs_right, we are creating two new normal
distributions that have the new mean (distribution_left, distribution_right).
Now, let’s see how we need to adapt the behavior after a left poke (and right poke identically) to deliver a pellet with a probability drawn from a normal distribution:
1 // This is part 2.
2 if (fed3.Left) {
3 fed3.BlockPelletCount = pellet_counter;
4 fed3.logLeftPoke(); //Log left poke
5 delay(1000);
6 float normal_left = distribution_left(generator);
7 if (random(100) < normal_left) { //Select a random number between 0-100 and ask if it is between 0-80 (80% of the time). If so:
8 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
9 fed3.Feed(); //Deliver pellet
10 pellet_counter ++; //Increase pellet counter by one
11 }
12 else { //If random number is between 81-100 (20% of the time)
13 fed3.Tone(300, 600); //Play the error tone
14 fed3.Timeout(timeoutIncorrect, true, true);
15 }
16 }
Here after a left poke, but before evaluating the outcome, we draw a number from distribution_left
(normal_left) and evaluate the outcome based on that number. A similar modification would be needed
to the FED3 behavior after a right poke.
Independence of arms#
Up until now, the two “arms” of the bandit task have been dependent on each other. The sum of prob_left
and prob_right has always been equal to 100. However, this does not need to be the case. Let’s say that
in this version of the task we want more than two probability options and we want the probabilities to
be chosen independently for each arm. To do this we adapt the variable setup as follows:
1 #include <FED3.h> //Include the FED3 library
2 String sketch = "Bandit"; //Unique identifier text for each sketch, change string only.
3 FED3 fed3 (sketch); //Start the FED3 object - don't change
4
5 int pellet_counter = 0; // pellet counter variable
6 int timeoutIncorrect = 10; // duration in seconds, set to 0 to remove the timeout
7 int probs[5] = {90,70,50,30,10}; // probability options
8 int probs_std = 10:
9 int new_prob_left = 0; //
10 int new_prob_right = 0;
11
12 void setup() {
13 fed3.countAllPokes = false; // Whether all pokes are counter
14 fed3.LoRaTransmit = false; // Wireless data transmission (future implementation)
15 fed3.pelletsToSwitch = 30; // Number of pellets required to finish the block and change reward probabilities
16 fed3.prob_left = 90; // Initial reward probability of left poke
17 fed3.prob_right = 90; // Initial reward probability of right poke
18 fed3.allowBlockRepeat = false; // Whether the same probabilities can be used for two blocks in a row
19 fed3.begin(); // Setup the FED3 hardware, all pinmode screen etc, initialize SD card
20 }
Here, we have modified probs to contain 5 values: 90, 70, 50, 30, 10. This provides more options of reward
probabilities. We have also change the name of new_prob to new_prob_left and initialized a new variable
called new_prob_right Note that we also changed the initial value of fed3.prob_left and fed3.prob_right
to be one of the options from probs. Also note that the sum of the two does not equal to 100.
Now, we need to change what happens when the condition for switching probabilities is met:
1 // This is part 1.
2 if (pellet_counter == fed3.pelletsToSwitch) {
3 pellet_counter = 0;
4 new_prob_left = probs[random(0,5)];
5 new_prob_right = probs[random(0,5)];
6 if (! fed3.allowBlockRepeat) {
7 while (new_prob_left == fed3.prob_left) {
8 new_prob_left = probs[random(0,2)];
9 }
10 while (new_prob_right == fed3.prob_right) {
11 new_prob_right = probs[random(0,2)];
12 }
13 fed3.prob_left = new_prob_left;
14 fed3.prob_right = new_prob_right;
15 }
16 else {
17 fed3.prob_left = new_prob_left;
18 fed3.prob_right = nw_prob_right;
19 }
20 }
Here we are repeating the procedure of choosing a probability from probs for new_prob_left
and new_prob_right. Since random() is called twice, two different values will likely be assigned
to each.
Creating conditions for reward probability changes#
Up until now, the condition for switching reward probabilities has been to reach 30 pellets delivered. Let’s say that we want to change this condition. The new condition will be that the rodent needs to poke seven times on the high probability port, and after that the reward probabilities will change. Let’s modify the variable setup first:
1 #include <FED3.h> //Include the FED3 library
2 String sketch = "Bandit"; //Unique identifier text for each sketch, change string only.
3 FED3 fed3 (sketch); //Start the FED3 object - don't change
4
5 int pellet_counter = 0; // pellet counter variable
6 int timeoutIncorrect = 10; // duration in seconds, set to 0 to remove the timeout
7 int probs[2] = {80,20}; // probability options
8 int new_prob = 0; //
9 int high_p_pokes = 0;
10
11 void setup() {
12 fed3.countAllPokes = false; // Whether all pokes are counter
13 fed3.LoRaTransmit = false; // Wireless data transmission (future implementation)
14 fed3.pelletsToSwitch = 7; // Number of pellets required to finish the block and change reward probabilities
15 fed3.prob_left = 80; // Initial reward probability of left poke
16 fed3.prob_right = 20; // Initial reward probability of right poke
17 fed3.allowBlockRepeat = false; // Whether the same probabilities can be used for two blocks in a row
18 fed3.begin(); // Setup the FED3 hardware, all pinmode screen etc, initialize SD card
19 }
Here, we initialized a new variable called high_p_pokes which will be the counter of high probability pokes.
We also changed the value of fed3.pelletsToSwitch to 7. Note that the new condition will not require any
specific number of pellets, but fed3.pelletsToSwitch is printed in the .csv output file. Alternatively,
we could also set fed3.pelletsToSwitch = 0 and initialize a new variable int high_p_cond = 7 and replace
all fed3.pelletsToSwitch in the rest of the code. However, using this alternative the condition will not be
printed in the csv file. Now let’s take a look at part 1 of the task setup:
1 // This is part 1.
2 if (high_p_pokes == fed3.pelletsToSwitch) {
3 high_p_pokes = 0;
4 new_prob = probs[random(0,2)];
5 if (! fed3.allowBlockRepeat) {
6 while (new_prob == fed3.prob_left) {
7 new_prob = probs[random(0,2)];
8 }
9 }
10 fed3.prob_left = new_prob;
11 fed3.prob_right = 100 - fed3.prob_left;
12 }
The only change we made here was to replace pellet_counter with high_p_pokes. Finally, we need to
when high_p_pokes will increase. Let’s see how this looks after a left poke
1 // This is part 2.
2 if (fed3.Left) {
3 fed3.BlockPelletCount = pellet_counter;
4 fed3.logLeftPoke(); //Log left poke
5 delay(1000);
6 if (prob_left > 50) {
7 high_p_pokes ++;
8 }
9 else {
10 high_p_pokes = 0;
11 }
12
13 if (random(100) < normal_left) { //Select a random number between 0-100 and ask if it is between 0-80 (80% of the time). If so:
14 fed3.ConditionedStimulus(); //Deliver conditioned stimulus (tone and lights)
15 fed3.Feed(); //Deliver pellet
16 pellet_counter ++; //Increase pellet counter by one
17 }
18 else { //If random number is between 81-100 (20% of the time)
19 fed3.Tone(300, 600); //Play the error tone
20 fed3.Timeout(timeoutIncorrect, true, true);
21 }
22 }
Here we have added a new conditional statement. If the left port is the high probability port (prob_left > 50)
then high_p_pokes will increase by one. Otherwise, it will be reset to 0. If we add the same conditional
statement after a right poke (replacing prob_left > 50 with prob_right > 50), high_p_pokes will only
reach 7 (number needed to trigger a reward probability change) after the rodent pokes 7 consecutive times in the
higher probability port.
Other Customizations#
The objective of this tutorial is to develop intuition on how the FED3Bandit task is setup. For additional versions of FED3Bandit, please see the HOW-TO GUIDE section.