Videos are a sequence of images played back at a variable speed in order to create the illusion of motion. The first ever video/movie created was called The Horse In Motion, in 1878.
What is video scrubbing?
Since that ground-breaking first movie, videos have come a long way. The technique of video scrubbing is an interaction in which a user can fast-forward/reverse a video by dragging a play-head or cursor over a segment track to a particular location.
We set out an objective to build a video scrubbing experience that can be controlled by the user on scroll and toggle slides in view set at various time-points of the video. Before deciding on the final front-end implementation for the video scrub we experimented with three approaches:
- Video current time
- Video seek unpack frames canvas
- Video server frames sequence
1. Video current time
Using the simplest approach, we loaded the video in an HTM5 video tag with source and manipulated the currentTime property of the video as the user is scrolling. This allows the default playback web experience in the browser to take over.
This approach worked fairly well on high-end devices and with videos no larger than 5mb and a resolution of maximum 720p. However, increasing the length and resolution of the video increased the frame drops while scrubbing, and the experience decreased drastically on mobile devices.
Nevertheless, this approach is great for short and low resolution reels, but not suited for larger/high definition ones.
2. Video seek unpack frames canvas
With this approach, first we had to load the video off-screen and unpack the frames by playing the video, listening to time update events and pausing it as the video plays. The reason for this was to extract the frame and generate a static image from it into a Canvas element. We repeated this method until the video was complete and all frames were extracted.
This method is more reliable than the previous in terms of experience, but not very versatile as the time to extract the frames takes as much time as the video itself.
3. Video server frames sequence
After experimenting with fist two methods we settled on the video server frames sequence approach. This relies on the server to provide the individual video frames, download them and build up the scrub experience from the client side.
The entire experience relies on a set of frames to be downloaded as individual images and the pointing on scroll for each frame to be done on a Canvas element.
Processing video file into frames
Firstly, we had to process the video to be scrubbed through into frames and uploaded them in Sitecore's Media Library as a set of folders, which were then transformed into a JSON array of strings with URLs stored in a data-attribute handled by front-end.
Before setting the final frames and uploading them into Sitecore CMS we optimised the rendered frames. By breaking down the video into frames, the total size was slightly more than the video itself. In order to keep this size increase as low as possible we ran a series of optimisations for the video. Starting with the video 'oncoding' part, we removed any audio tracks, subtitles and irrelevant metadata before optimising it for web delivery.
After encoding the video we converted the video into frames. Converting to frames requires FFmpeg to be runnable from PowerShell. This is a cross-platform solution to record, convert and stream audio and video.
Alternatively, installing certain applications like Handbrake may also install FFmpeg. Finally, we took the generated JPEG frames and optimised them for web as well using online tools like Tinyping, which stripped around 30% per frame from its size.
Download frames
Downloading a large amount of images in the same UI thread was a challenge. In order to be able to efficiently do this, we off-loaded the image download logic into multiple separate threads using WebWorkers.
Building the scrubbing mechanism
The scrubbing mechanism for Sitecore was built with the following core parameters in mind:
- videoDuration: Required for setting up the scroll scene height and general trigger points for the slides in-view
- scrubVelocity: How fast the video scrub experience is perceived by the user during scrolling
- easing: Easing effect for rendered frames in view
The scrubbing mechanism logic was fairly easy to achieve. We suggest using this technique for videos that are fairly small in size as this will deliver optimum performance and an improved user experience.
At Codehouse we're constantly exploring new technologies and challenges that will improve the visitor experience. We continually work on exciting design and build projects often on CMS like Sitecore.
