Computer Science students improve UCT lecture recording

16 Nov 2017 - 13:45
Trak4K lecture recording UTC

Image: Lecturer tracking by Charles and Tanweer's project TRACK4K  

Lecture recording is becoming increasingly popular at universities and with good reason. It reduces the need for extensive note-taking in class, which allows students to focus and engage better without the stress of missing vital information, and can serve as a useful recapping tool for students who may have missed crucial content during a lecture. Recordings can be particularly helpful for students with learning disabilities, and provide a valuable safety net for all students.

CILT is responsible for UCT’s rapidly growing lecture recording service, and over the last two years has been working with a group of UCT Computer Science students to solve some tricky problems using video and audio processing algorithms.

Masters students Charles Fitzhenry and Tanweer Khatieb are attempting to mimic a human camera operator with the use of a 4K camera.

In larger venues, lecture videos need to show a close-up view so that students can read the blackboards clearly. Traditionally this problem has been solved using PTZ (pan, tilt, zoom) cameras, which follow the lecturer in real-time. However, this solution is expensive, and doesn’t work well in all circumstances.

Tanweer and Charles believe they can achieve a better result by using less expensive high-resolution 4K fixed cameras to record the whole teaching area, and then processing the video afterwards using complex algorithms that identify the best field of view to show to students at any time.

Charles’ component applies computer vision algorithms to identify features and motion inside video frames, to determine for example where the lecturer is, and whether the lecturer is writing on a blackboard.

Tanweer’s component uses this recognition information to create a virtual cinematographer, which decides where to focus the viewer’s attention by simulating the movements of a camera operator.

The result is a new video stream that is made by cropping out a small piece of the larger video. This is no small feat, as the system needs to extract a cropping window for every single video frame in order to build the final video. As an example, a one hour lecture typically has about 108000 frames.

Supervisor Associate Professor Patrick Marais explains, “one of the requirements is to do this in a computationally efficient manner, which constrains the algorithms that can be used. Lecture recording systems may process hundreds of videos every day, so the requirement from CILT was for a system which takes no more than three times the duration of the video to run.”

The Masters research builds on a simpler solution developed by Charles, Tanweer and fellow student Max Hahn in 2016 for their Computer Science Honours year project, Track4K.

By contrast, Masters student and ICTS staff member Devan Govender has focused on audio analysis to automate the process of trimming lecture videos.

At present, every recorded lecture is edited by staff at ICTS to remove “dead time” in the video before the lecture starts and after the lecture ends. However, this manual processing can slow down the publication of recordings, and as volumes increase, a mostly automated system became an essential requirement.

Devan’s research trained an audio classification system to identify which parts of the video soundtrack are silence, background noise, audience chatter, and a lecturer speaking. The system is then able to make a decision about which parts of the video to include and which to exclude.

Tests thus far have been positive with a 90 percent accuracy rate in distinguishing between speech and chatter. Devan hopes to run a trial before the end of 2017.

For Devan, a self-professed geek, the core motivation for doing this research is to streamline the process, reduce turnaround time and just keep the students happy.

Such collaborations and innovative approaches to improving lecture recording at UCT are exactly what is needed to improve the student experience. 

Bottom right image: Charles Fitzhenry, Stephen Marquard (CILT), Professor Patrick Marais, Tanweer Khatieb and Devan Govender.