Building a self-maintaining collection that removes items as they expire?

I am building a system that enables users to enter a phone number to add to their account. The system allows a user to enter the number, that number is then added to my PendingNumberManager with a set expiration time. We then dispatch a text message to that number, and wait for the user to enter their verification code in our application. If they enter the correct verification code for their account and number, that number is removed from the PendingNumberManager and added to their user account (and persisted to the database).

Overall, the functionality is relatively straightforward until I start factoring in the automatic expiry. Some of the options that jump out to me immediately are:

• Never remove expired entries unless someone tries to fetch them, and then remove the singular expired entry. (Will potentially cause the list to get more and more full over time).
• Check for expired entries on every lookup. (Causes additional overhead on every lookup).
• Have some sort of loop running in another thread that just periodically checks for expired items. (May eat up additional resources when they're not required)

Overall I'm somewhat at a loss on how to handle this the most efficient way. Is there any design pattern or approach that lends itself well to this problem?