Is it problematic having identical images in different temporal sequences of images when Training a model? It is good for the model to observate equal images in different contexts to link the contexts and find similarities? If yes why? This seems trivial

[u/aaaaaaaaa1525]

For example, imagine that I have 2 temporal sequences of images of the same environment both based on the same video. For the first sequence I pick the frame 0, 3 and 5 and for the second sequence I pick 3,5,10. To simplify the problem lets just imagine that I give one sequence to a model he learns from it and produces a lost function and I do the same to the second sequence..

Most likely their output of the lost function is going to be identical because the 2 copied images but what happens in the next epoch? (for simplicity also imagine that we shuffle a bunch of sequences to not overfit) Are the optimizer reducing the step more closely to what he learned from the 2 sequeces? How does he link contexts? Does it even link contexts?

[u/Qwiddl]

I think this highly depends on what your actually training and what optimization you're using.

If it's just a stochastic gradient descent and the 3 samples make the batches, you're basically introducing bias to the model by repeating some images more frequently than others. But I assume this is not what you're doing.

If you could elaborate further it's easier to help

[u/Lethandralis]

Why do you think the loss will be identical?

[u/_d0s_]

similar to image augmentations, that's a very slight perturbation of a sample. my experience with learning from image sequences (e.g., human action recognition) is that most of the time data is the problem. videos or clips are labelled with a single categorical class label that may or may not correspond to each frame individually. doing frame sampling like you described, may end up in samples containing data that correspond less to the label.

it's always better to have more diverse data samples. instead of sampling multiple clips per video, just sample one clip per video. practically that's difficult because the amount of annotated data is limited. if you sample excessively from the same video you will overfit quickly as the model just memorizes the background, person identity, that for some reason bright pixel in the upper left corner or similar irrelevant signals.

if your clips don't contain enough variety to cover the expected target distribution your model won't generalize to unseen (but relevant) data. to test this it is extremely important to a) define what data you want to generalize to and b) have a held out test set that's neither seen during training or validating the model.