how to conduct smoothing on a stream of images and identify irrelevant images

I am trying to cam record computer user behaviors and will try to predict the action as final goal. However, before moving on predictive model implementation, I need to reduce the noise and eliminate irrelevant images from a training set. For example I have a folder name eating food and I recorded 10 different users for 2 minutes while they are eating the food. Now I will train this folder as "eating" but not all the frames in these recorded set is eating. Lets say I have 500 frames recorded for this action but only 400 of them is really eating. The rest 100 might be just some other actions user engage in while recording (for example he receives a phone call and presents a different action other than eating while I record eating action). So the question is how can I eliminate these irrelevant actions from a stream of action frames?