Html5 Video frame accurate

The Problem

Unfortunately there is no frame API for us to use - Firefox has some experimental features, but other than that we can only deal with time and how the browser handles frame decoding as well as frame rate conversion. See below for an update.

For example, the browser may or may always convert/normalize to 30 fps, and if so may or may not assume drop-frame (for 29.97/59.94 fps/NTSC). Over time this will affect the frame count and will eventually show as offsets as there is no true link between time and frame count.

In addition there could be rounding errors due to the integer based conversion and also potentially depending on the start position in the video. There are also different solutions to how video is decoded under the hood depending on the broswer, for example, Chrome uses FFMpeg underneath, Firefox uses whatever is available in the OS* (for MP4 encoded videos) and so forth.

Possible solution

(Update: forgot to mention -) There is the Media Source Extensions API which support the VideoPlaybackQuality object. This has currently [support in Chrome and IE (on Windows 8+) as well as in Firefox.

To use you can call it directly on the HTMLVideoElement:

var q = video.getVideoPlaybackQuality();

Then read out the values using its property:

var frames = q.totalVideoFrames;

Just be aware of that this value may accumulate all frames regardless of the actual play position. You can perhaps get around this working with MSE directly. It's a little more tedious to setup and handle, but this link provides a good starting point in case you're unfamiliar with MSE.

An alternative idea would be to encode a special bar-code in the video, then read it using canvas. It would be the equivalent on an actual SMPTE time-code, but this would require an overlay being visible in the video to some degree (ala the old way of encoding tele-text into the over-scan area of the video). If the video is full-frame other problems will arise such as false detection depending on colors used for the coding (or make a solid background for it).

^*) may have changed in recent versions after Cisco released their mp4 solution for free.

I posted this answer on the link you referenced, but I think it might be relevant here too:

Starting in M83, Chromium has a requestVideoFrameCallback() API, which might solve your issue. You can use the mediaTime to get a consistent timestamp, as outlined in this Github issue. From there, you could try something like this:

var frameCounter = (time, metadata) => {
   let count = metadata.mediaTime * frameRate;
   console.log("Got frame: " + Math.round(count));

   // Display timecode code here.

   video.requestVideoFrameCallback(frameCounter);
}

video.requestVideoFrameCallback(frameCounter)

This will only fire on new frames, but you may occasionally miss one (which you can detect from a discontinuity in the metadata.presentedFrames count). You might also be slightly late in capturing the frame (usually 16ms, or one call to window.requestAnimationFrame() later than when the video frame is available).

If you're interested in a high level overview of the API, here's a blogpost, or you can take a look at the API's offical github.