alluvial filled with pattern

I have the following sankey plot, and I'd like to fill the bars with patterns, instead of different colors.

based on a df reproducible with the following code:

data<- data.frame("ID"=rep(c(1:20),2), "time"=c(rep(1,20),rep(2,20)), "value"=c("B","G","B","G","B",rep("G",3),rep("B",2),"G",rep("B",3),"G",rep("B",6),"G","B","G","B",rep("G",3),rep("B",2),"G",rep("B",3),"G","B","switch",rep("B",2),"switch"))

Here the code for the sankey:

library(dplyr)
library(ggplot2)
library(ggalluvial)
library(ggpattern)

data %>% 
  group_by(time) %>% 
  mutate(w = n()) %>% 
  ggplot(aes(x = time, stratum = value, alluvium = ID,
             fill = value, label = value, y = 1/w)) +
  scale_x_discrete() +
  scale_y_continuous(name = "", label = scales::percent_format(accuracy = 1)) +
  geom_flow(stat = "flow", knot.pos = 1/4, aes.flow = "forward") +
  geom_stratum() +
  theme(legend.position = "bottom") +
  labs(x = "", fill = "")

I'd like to fill the bars with patterns, instead of different colors.

I tried with scale_pattern_type_manual(values=c('stripe', 'wave', 'wave', "crosshatch", "none","stripe", "crosshatch","none")) but it does not work.