The Cause
Especially on websites with a large number of videos, the plugin can have problems finding all the videos, due to (a shortage of) memory.
The Solution
You should try going to SEO-Video and then re-indexing a number of times. You may need to scroll down to find this feature. However, be sure to leave the 『Force Re-Index』 box unchecked. Once the number in the sitemap is static, it has found all the videos.