Predicting Vehicle Motion and Direction using Optical Flow
In this post we’ll explore the topic of optical flow. Optical flow has many useful application in computer vision such as structure from motion and video compression. Let’s see if it’s possible to use optical flow to predict a vehicle’s motion from a dash camera feed.
Thanks for the video: driving dash cam
Optical Flow Algorithm
I decide to use Lucas–Kanade to calculate optical flow.
I decided to use the ORB(Oriented FAST and rotated BRIEF) algorithm to collect features (or keypoints), from a video frame. ORB is the child of two different algorithms, which I’ll discuss briefly. Later, we will track the collected keypoints from ORB and use an optical flow algorithm called Lucas–Kanade to predict the vehicle’s motion and direction.
ORB uses FAST(Features from accelerated segment test ) to detect features; as you could have guessed from the name. The FAST feature detection algorithm is used to collect keypoints (i.e pixel regions that contain corners). I chose FAST because it preforms well with real-time video. FAST looks at a Bresenham circular region of 16 pixels and is able to detect corners quickly within a given threshold. It provides similar accuracy to the SIFT algorithm, but it accomplish it’s task in less time because it “checks” less pixels.
ORB uses BRIEF(Binary Robust Independent Elementary Features) to find feature descriptions.
Why combine FAST and BRIEF?
I’ll leave to explanation to Ethan Rublee
Predicting Motion and Direction
If we want to predict motion and direction, we’ll first need to find the mean angle of all the keypoints between two frames. After we have found the mean angle we can determine the direction from circular motion.
Calculating Mean Angle
We’ll use arctan to calculate the angle between keypoints.
When collecting angles between keypoints we will only collect points that are within a defined threshold. The threshold being, the distance between keypoints.
Defining Circular Direction and Motion
Now that we have found the mean angle, we can determine the motion and direction of the vechile. I will admit, this is going to be a bit hand-wavy.
Give the mean angle, we will define stationary motion to have 0 degrees. Forward motion will be greater than 0 degress and less than or equal to 180 degrees.
Try to imagine an object moving directly towards you. The angle between objects motion and the background plane is 90 degrees.
Using this intuition, we can determine if the object’s motion is more to the left or the right.
If the mean angle is greater than 0 degrees and less 90 degress, than the objects direction is more right.
If the mean angle is greater than 90 degrees and less 180 degress, than the objects direction is more left.
This is by no means a perfect system for determining a vehicle’s motion, but it’s not a bad start.
- Calculate dense optical flow and only use keypoints found at the centroid of high density regions.
- Try to predict the vehicle’s speed from the mean angle. In other words, compute the angular acceleration.
- Find a motion dataset and train a spatiotemporal convolutional neural network, than compare the network’s prediction with the proposed optical flow algorithm.