In instrumental animal decision making experiments, subjects choose an action and are rewarded according to which action they chose with some probability (often 70, 80, 90 or 100% probability for choosing the correct action). With more experience of the same reward contingencies, the animal learns and improves its performance, receiving rewards more consistently. Counterintuitively, when the reward contingencies switch and a different action produces a reward, animals who have more experience on the original reward contingency update their preferences more quickly than animals that have less experience with the original reward contingency. In other words, simple reinforcement learning algorithms such as Q-Learning would predict that a mouse who receives a reward every time he chooses a left water port would take longer to update to choosing the right side when he starts getting rewarded only for right port choices than a mouse who has less experience with the left side being rewarded. Of course, this is not the case and mice with more experience with the left port update more quickly. This phenomenon is called the overtraining reversal effect (ORE).
This repo is an attempt to explain the phenomenon by comparing computational models fitted to normally trained and overtrained mice. I used Hidden Markov Models where each state corresponded to a logistic regression model representing a decision making strategy. The transition matrix of the HMM was either static or was itself a logistic regression model. Each dicrete strategy from the HMM was then simulated to assess its performance, and classified into model free or model based learning with a kNN classifier. This is the code section of my Masters dissertation for the University of Nottingham.
To download the repository on MacOS or Linux, go to your preferred directory such as ~/workspaces and clone the repo:
git clone https://github.com/mikecondon/ORE-Modelling.gitThen download and install uv package manager. Once this is installed, move in to the cloned repo directory and use uv to install all dependencies into a new virtual environment.
cd ORE_Modelling
uv syncFinally, to run the Jupyter Notebook files, ensure jupyter notebook is installed and run:
uv run --with jupyter jupyter lab --notebook-dir=.The data used in this project is generated by a modified Q-learning algorithm in the q_visualise.ipynb and q_utils.py files. This timeseries data containing 30 simulations of normally trained and overtrained mice are stored in the data/ directory.
We fit models and stored their parameters using either the train_single.ipynb file or hyperparameter_search.py file. This also allowed us to find the best model fit.
We compared models in the logreg_hmm_investigate.ipynb, trans_glm_hmm.ipynb, logreg_visualise.ipynb notebooks for static GLM-HMM, non-stationary GLM-HMM or single state GLM. To determine whether a strategy was model free or model based, a kNN classifier was trained in the detector.ipynb file.
Modifications to the DynaMax state space models package are contained in the custom_hmm.py file. Plotting and data processing functionality is contained in the dynamax_utils.py file.