Chapter 4 From verbal descriptions to formal models

This chapter bridges the gap between verbal theories and computational implementations of cognitive models. Building on our observations of the matching pennies game, we now develop precise mathematical formulations that can generate testable predictions.

4.1 Learning Goals

After completing this chapter, you will be able to:

  • Transform verbal descriptions of decision-making strategies into precise mathematical formulations, which implications can be more easily explored and that can be empirically tested

  • Create computational implementations of these mathematical models as agent-based models in R

  • Generate and analyze simulated data to understand model behavior under different conditions

4.2 The Value of Formalization

Moving from verbal to formal models represents a crucial step in cognitive science. When we describe behavior in words, ambiguities often remain hidden. For instance, a verbal description might state that players “tend to repeat successful choices.” But what exactly constitutes “tend to”? How strongly should past successes influence future choices? Mathematical formalization forces us to be precise about these specifications.

By computationally implementing the our models,

  • we are forced to make them very explicit in their assumptions;

  • we become able to simulate the models in a variety of different situations and therefore better understand their implications

So, what we’ll do throughout the chapter is to:

  1. choose two of the models and formalize them, that is, produce an algorithm that enacts the strategy, so we can simulate them.

  2. implement the algorithms as functions: getting an input and producing an output, so we can more easily implement them across various contexts (e.g. varying amount of trials, input, etc). See R4DataScience, if you need a refresher: https://r4ds.had.co.nz/functions.html

  3. implement a Random Bias agent (choosing “head” 70% of the times) and get your agents to play against it for 120 trials (and save the data)

  4. implement a Win-Stay-Lose-Shift agent (keeping the same choice if it won, changing it if it lost) and do the same.

  5. scale up the simulation: have 100 agents for each of your strategy playing against both Random Bias and Win-Stay-Lose-Shift and save their data.

  6. figure out a good way to visualize the data to assess which strategy performs better, whether that changes over time and generally explore what the agents are doing.

4.3 Defining general conditions

pacman::p_load(tidyverse, patchwork)

# Number of trials per simulation
trials <- 120  

# Number of agents to simulate
agents <- 100  

# Optional: Set random seed for reproducibility
# set.seed(123)

4.4 Implementing a random agent

Remember a random agent is an agent that picks at random between “right” and “left” independently on what the opponent is doing. A random agent might be perfectly random (50% chance of choosing “right”, same for “left”) or biased. The variable “rate” determines the rate of choosing “right”.

rate <- 0.5

RandomAgent <- rbinom(trials, 1, rate) # we simply sample randomly from a binomial

# Now let's plot how it's choosing
d1 <- tibble(trial = seq(trials), choice = RandomAgent)

p1 <- ggplot(d1, aes(trial, choice)) + 
  geom_line() + 
  labs(
    title = "Random Agent Behavior (rate 0.5)",
    x = "Trial Number",
    y = "Choice (0/1)"
  ) +
  theme_classic()
p1

# What if we were to compare it to an agent being biased?
rate <- 0.8

RandomAgent <- rbinom(trials, 1, rate) # we simply sample randomly from a binomial

# Now let's plot how it's choosing
d2 <- tibble(trial = seq(trials), choice = RandomAgent)
p2 <- ggplot(d2, aes(trial, choice)) + 
  geom_line() + 
  labs(
    title = "Biased Random Agent Behavior",
    x = "Trial Number",
    y = "Choice (0/1)"
  ) + 
  theme_classic()

p1 + p2

print("This first visualization shows the behavior of a purely random agent - one that chooses between options with equal probability (rate = 0.5). Looking at the jagged line jumping between 0 and 1, we can see that the agent's choices appear truly random, with no discernible pattern. This represents what we might expect from a player who is deliberately trying to be unpredictable in the matching pennies game.
However, this raw choice plot can be hard to interpret. A more informative way to look at the agent's behavior is to examine how its average rate of choosing option 1 evolves over time:")
## [1] "This first visualization shows the behavior of a purely random agent - one that chooses between options with equal probability (rate = 0.5). Looking at the jagged line jumping between 0 and 1, we can see that the agent's choices appear truly random, with no discernible pattern. This represents what we might expect from a player who is deliberately trying to be unpredictable in the matching pennies game.\nHowever, this raw choice plot can be hard to interpret. A more informative way to look at the agent's behavior is to examine how its average rate of choosing option 1 evolves over time:"
# Tricky to see, let's try writing the cumulative rate:

d1$cumulativerate <- cumsum(d1$choice) / seq_along(d1$choice)
d2$cumulativerate <- cumsum(d2$choice) / seq_along(d2$choice)

p3 <- ggplot(d1, aes(trial, cumulativerate)) + 
  geom_line() + 
  ylim(0,1) + 
  labs(
    title = "Random Agent Behavior",
    x = "Trial Number",
    y = "Cumulative probability of choosing 1 (0-1)"
  ) + 
  theme_classic()

p4 <- ggplot(d2, aes(trial, cumulativerate)) + 
  geom_line() + 
  labs(
    title = "Random Agent Behavior",
    x = "Trial Number",
    y = "Cumulative probability of choosing 1 (0-1)"
  ) + 
  ylim(0,1) + 
  theme_classic()

p3 + p4

print("This cumulative rate plot helps us better understand the agent's overall tendencies. For a truly random agent, we expect this line to converge toward 0.5 as the number of trials increases. Early fluctuations away from 0.5 are possible due to random chance, but with more trials, these fluctuations tend to even out.
When we compare agents with different underlying biases (rate = 0.5 vs rate = 0.8):")
## [1] "This cumulative rate plot helps us better understand the agent's overall tendencies. For a truly random agent, we expect this line to converge toward 0.5 as the number of trials increases. Early fluctuations away from 0.5 are possible due to random chance, but with more trials, these fluctuations tend to even out.\nWhen we compare agents with different underlying biases (rate = 0.5 vs rate = 0.8):"
## Now in the same plot
d1$rate <- 0.5
d2$rate <- 0.8
d <- rbind(d1,d2) %>% 
  mutate(rate = as.factor(rate))

p5 <- ggplot(d, aes(trial, cumulativerate, color = rate, group = rate)) + 
  geom_line() + 
  labs(
    title = "Random Agents Behavior",
    x = "Trial Number",
    y = "Cumulative probability of choosing 1 (0-1)"
  ) + 
  ylim(0,1) + 
  theme_classic()
p5

print("We can clearly see how bias affects choice behavior. The unbiased agent (rate = 0.5) stabilizes around choosing each option equally often, while the biased agent (rate = 0.8) shows a strong preference for option 1, choosing it approximately 80% of the time. This comparison helps us understand how we might detect biases in real players' behavior - consistent deviation from 50-50 choice proportions could indicate an underlying preference or strategy.")
## [1] "We can clearly see how bias affects choice behavior. The unbiased agent (rate = 0.5) stabilizes around choosing each option equally often, while the biased agent (rate = 0.8) shows a strong preference for option 1, choosing it approximately 80% of the time. This comparison helps us understand how we might detect biases in real players' behavior - consistent deviation from 50-50 choice proportions could indicate an underlying preference or strategy."
# Now as a function
#' Create a random decision-making agent
#' @param input Vector of previous choices (not used but included for API consistency)
#' @param rate Probability of choosing option 1 (default: 0.5 for unbiased)
#' @return Vector of binary choices
#' @examples 
#' # Create unbiased random agent for 10 trials
#' choices <- RandomAgent_f(rep(1,10), 0.5)
RandomAgent_f <- function(input, rate = 0.5) {
  # Input validation
  if (!is.numeric(rate) || rate < 0 || rate > 1) {
    stop("Rate must be a probability between 0 and 1")
  }
  
  n <- length(input)
  choice <- rbinom(n, 1, rate)
  return(choice)
}

input <- rep(1,trials) # it doesn't matter, it's not taken into account
choice <- RandomAgent_f(input, rate)
d3 <- tibble(trial = seq(trials), choice)

ggplot(d3, aes(trial, choice)) + 
  geom_line() + 
  labs(
    title = "Random Agent Behavior",
    x = "Trial Number",
    y = "Cumulative probability of choosing 1 (0-1)"
  ) + 
  theme_classic()

## What if there's noise?
RandomAgentNoise_f <- function(input, rate, noise){
  n <- length(input)
  choice <- rbinom(n, 1, rate)
  if (rbinom(1, 1, noise) == 1) {choice = rbinom(1,1,0.5)}
  return(choice)
}

4.5 Implementing a Win-Stay-Lose-Shift agent

#' Create a Win-Stay-Lose-Shift decision-making agent
#' @param prevChoice Previous choice made by the agent (0 or 1)
#' @param feedback Success of previous choice (1 for win, 0 for loss)
#' @param noise Optional probability of random choice (default: 0)
#' @return Next choice (0 or 1)
#' @examples
#' # Basic WSLS decision after a win
#' next_choice <- WSLSAgent_f(prevChoice = 1, feedback = 1)
WSLSAgent_f <- function(prevChoice, feedback, noise = 0) {
  # Input validation
  if (!is.numeric(prevChoice) || !prevChoice %in% c(0,1)) {
    stop("Previous choice must be 0 or 1")
  }
  if (!is.numeric(feedback) || !feedback %in% c(0,1)) {
    stop("Feedback must be 0 or 1")
  }
  if (!is.numeric(noise) || noise < 0 || noise > 1) {
    stop("Noise must be a probability between 0 and 1")
  }
  
  # Core WSLS logic
  choice <- if (feedback == 1) {
    prevChoice  # Stay with previous choice if won
  } else {
    1 - prevChoice  # Switch to opposite choice if lost
  }
  
  # Apply noise if specified
  if (noise > 0 && runif(1) < noise) {
    choice <- sample(c(0,1), 1)
  }
  
  return(choice)
}


WSLSAgentNoise_f <- function(prevChoice, Feedback, noise){
  if (Feedback == 1) {
    choice = prevChoice
  } else if (Feedback == 0) {
      choice = 1 - prevChoice
  }
  if (rbinom(1, 1, noise) == 1) {choice <- rbinom(1, 1, .5)}
  return(choice)
}

WSLSAgent <- WSLSAgent_f(1, 0)


# Against a random agent

Self <- rep(NA, trials)
Other <- rep(NA, trials)

Self[1] <- RandomAgent_f(1, 0.5)
Other <- RandomAgent_f(seq(trials), rate)
  

for (i in 2:trials) {
  if (Self[i - 1] == Other[i - 1]) {
    Feedback = 1
  } else {Feedback = 0}
  Self[i] <- WSLSAgent_f(Self[i - 1], Feedback)
}

sum(Self == Other)
## [1] 73
df <- tibble(Self, Other, trial = seq(trials), Feedback = as.numeric(Self == Other))

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, Self)) +
  geom_line(color = "blue", aes(trial, Other)) +
  labs(
    title = "WSLS Agent (red) vs Biased Random Opponent (blue)",
    x = "Trial Number",
    y = "Choice (0/1)",
    color = "Agent Type"
  )

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, Feedback)) +
  geom_line(color = "blue", aes(trial, 1 - Feedback)) +
  labs(
    title = "WSLS Agent (red) vs Biased Random Opponent (blue)",
    x = "Trial Number",
    y = "Feedback received (0/1)",
    color = "Agent Type"
  )

print("These plots compare how a Win-Stay-Lose-Shift (WSLS) agent performs against different opponents. The red line shows the WSLS agent's choices, while the blue line shows the opponent's choices. When playing against a biased random opponent, we can see clearer patterns in the WSLS agent's behavior as it responds to wins and losses. Against another WSLS agent, the interaction becomes more complex, as each agent is trying to adapt to the other's adaptations. This kind of visualization helps us understand how different strategies might interact in actual gameplay.")
## [1] "These plots compare how a Win-Stay-Lose-Shift (WSLS) agent performs against different opponents. The red line shows the WSLS agent's choices, while the blue line shows the opponent's choices. When playing against a biased random opponent, we can see clearer patterns in the WSLS agent's behavior as it responds to wins and losses. Against another WSLS agent, the interaction becomes more complex, as each agent is trying to adapt to the other's adaptations. This kind of visualization helps us understand how different strategies might interact in actual gameplay."
df$cumulativerateSelf <- cumsum(df$Feedback) / seq_along(df$Feedback)
df$cumulativerateOther <- cumsum(1 - df$Feedback) / seq_along(df$Feedback)

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, cumulativerateSelf)) +
  geom_line(color = "blue", aes(trial, cumulativerateOther))  +
  labs(
    title = "WSLS Agent (red) vs Biased Random Opponent (blue)",
    x = "Trial Number",
    y = "Cumulative probability of choosing 1 (0-1)",
    color = "Agent Type"
  )

# Against a Win-Stay-Lose Shift
Self <- rep(NA, trials)
Other <- rep(NA, trials)

Self[1] <- RandomAgent_f(1, 0.5)
Other[1] <- RandomAgent_f(1, 0.5)

for (i in 2:trials) {
  if (Self[i - 1] == Other[i - 1]) {
    Feedback = 1
  } else {Feedback = 0}
  Self[i] <- WSLSAgent_f(Self[i - 1], Feedback)
  Other[i] <- WSLSAgent_f(Other[i - 1], 1 - Feedback)
}

sum(Self == Other)
## [1] 60
df <- tibble(Self, Other, trial = seq(trials), Feedback = as.numeric(Self == Other))

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, Self)) +
  geom_line(color = "blue", aes(trial, Other))

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, Feedback)) +
  geom_line(color = "blue", aes(trial, 1 - Feedback))

df$cumulativerateSelf <- cumsum(df$Feedback) / seq_along(df$Feedback)
df$cumulativerateOther <- cumsum(1 - df$Feedback) / seq_along(df$Feedback)

ggplot(df) + theme_classic() +
  geom_line(color = "red", aes(trial, cumulativerateSelf)) +
  geom_line(color = "blue", aes(trial, cumulativerateOther))

print("This cumulative performance plot reveals the overall effectiveness of the WSLS strategy. By tracking the running average of successes, we can see whether the strategy leads to above-chance performance in the long run. When playing against a biased random opponent, the WSLS agent can potentially exploit the opponent's predictable tendencies, though success depends on how strong and consistent the opponent's bias is.
When we pit the WSLS agent against another WSLS agent, the dynamics become more complex. Both agents are now trying to adapt to each other's adaptations, creating a more sophisticated strategic interaction. The resulting behavior often shows interesting patterns of mutual adaptation, where each agent's attempts to exploit the other's strategy leads to evolving patterns of play.")
## [1] "This cumulative performance plot reveals the overall effectiveness of the WSLS strategy. By tracking the running average of successes, we can see whether the strategy leads to above-chance performance in the long run. When playing against a biased random opponent, the WSLS agent can potentially exploit the opponent's predictable tendencies, though success depends on how strong and consistent the opponent's bias is.\nWhen we pit the WSLS agent against another WSLS agent, the dynamics become more complex. Both agents are now trying to adapt to each other's adaptations, creating a more sophisticated strategic interaction. The resulting behavior often shows interesting patterns of mutual adaptation, where each agent's attempts to exploit the other's strategy leads to evolving patterns of play."

4.6 Now we scale it up

trials = 120
agents = 100

# WSLS vs agents with varying rates

for (rate in seq(from = 0.5, to = 1, by = 0.05)) {
  
  for (agent in seq(agents)) {
    Self <- rep(NA, trials)
    Other <- rep(NA, trials)
    
    Self[1] <- RandomAgent_f(1, 0.5)
    Other <- RandomAgent_f(seq(trials), rate)
    
    
    for (i in 2:trials) {
      if (Self[i - 1] == Other[i - 1]) {
        Feedback = 1
      } else {Feedback = 0}
      Self[i] <- WSLSAgent_f(Self[i - 1], Feedback)
    }
    
    temp <- tibble(Self, Other, trial = seq(trials), Feedback = as.numeric(Self == Other), agent, rate)
    
    if (agent == 1 & rate == 0.5) {df <- temp} else {df <- bind_rows(df, temp)}
  }
}

## WSLS with another WSLS

for (agent in seq(agents)) {
    Self <- rep(NA, trials)
    Other <- rep(NA, trials)
    
    Self[1] <- RandomAgent_f(1, 0.5)
    Other[1] <- RandomAgent_f(1, 0.5)
    
    
    for (i in 2:trials) {
      if (Self[i - 1] == Other[i - 1]) {
        Feedback = 1
      } else {Feedback = 0}
      Self[i] <- WSLSAgent_f(Self[i - 1], Feedback)
      Other[i] <- WSLSAgent_f(Other[i - 1], 1 - Feedback)
    }
    
    temp <- tibble(Self, Other, trial = seq(trials), Feedback = as.numeric(Self == Other), agent, rate)
    
    if (agent == 1 ) {df1 <- temp} else {df1 <- bind_rows(df1, temp)}
  }

4.6.1 And we visualize it

ggplot(df, aes(trial, Feedback, group = rate, color = rate)) +
  geom_smooth(se = F) + theme_classic()

We can see that the bigger the bias in the random agent, the bigger the performance in the WSLS (the higher the chances the random agent picks the same hand more than once in a row).

Now it’s your turn to follow a similar process for your 2 chosen strategies.

4.7 Conclusion

Moving from verbal descriptions to formal computational models represents a crucial step in cognitive science. Through our work with the matching pennies game, we have seen how this transformation process requires careful consideration of theoretical assumptions, mathematical precision, and practical implementation details.

The development of formal models forces us to be explicit about mechanisms that might remain ambiguous in verbal descriptions. When we state that an agent “learns from experience” or “responds to patterns,” we must specify exactly how these processes work. This precision not only clarifies our theoretical understanding but also enables rigorous empirical testing. Our implementation of different agent types - from simple random choice to more sophisticated strategies - demonstrates how computational modeling can reveal surprising implications of seemingly straightforward theories. Through simulation, we discovered that even basic strategies can produce complex patterns of behavior, especially when agents interact with each other over multiple trials.

Perhaps most importantly, this chapter has established a foundational workflow for cognitive modeling: begin with careful observation, think carefully and develop precise mathematical formulations, implement these as computational models, and validate predictions against data. Don’t be afraid to make mistakes, or rethink your strategy and iterate the modeling process. This systematic approach will serve as our template as we progress to more complex cognitive phenomena in subsequent chapters.

While our matching pennies models may seem simple compared to the rich complexity of human cognition, they exemplify the essential principles of good modeling practice: clarity of assumptions, precision in implementation, and rigorous validation against empirical data. These principles will guide our exploration of more sophisticated cognitive models throughout this course. For more advanced examples of models that can underly behavior in the Matching Pennies game check:

  • Chapter 12 on reinforcement learning.

  • the paper by Waade et al mentioned at the beginning of the chapter.