Multilevel contrast with sex as fixed effect in the model: Limma within R environment

Usually I se contrasts made using factorial approaches (case vs control, 0 vs 1). Using limma in R, how do I create a multilevel contrast including a sex effect in the model? Here you have one example of the input groups I would like to use:

library("limma")
pheno <- rep (1:4, 5)
sex <- rep(1:2, 10)
test<- as.data.frame(cbind (pheno, sex))
group <- as.factor(test$sex)
group2 <- as.factor(test$pheno)
group3 <- as.factor(paste(test$sex , test$pheno , sep = "_"))

I would like to create a contrast to analyze the effect of the pheno (varying from 0 to 4) using sex as fixed effect in the model. Here I am including some lines related to the construction of contrasts withim limma package

design <- model.matrix(~0+group3)
colnames(design) <- gsub("group3", "", colnames(design))
contr.matrix <- makeContrasts(
  comp1= 0-4,
  levels = colnames(design))