Extract Motion Vectors from Video Using libvpx

This article explains how to extract motion vectors from a VP8 or VP9 video stream using the libvpx library. It covers configuring the library for frame inspection, setting up the necessary decoding callback functions, and parsing the block-level motion vector structures to retrieve spatial movement information.

Step 1: Build libvpx with Inspection Enabled

By default, the standard production builds of libvpx do not expose block-level motion vector data. To access this metadata, you must compile the libvpx library from source with the inspection API enabled.

Run the configure script with the following flag:

./configure --enable-inspection --enable-vp9-decoder
make
make install

This configuration flag exposes the inspection headers and structures, such as vpx_inspect.h, which are required to register callbacks that capture decoder state data.

Step 2: Register the Inspection Callback

Once your library is compiled with inspection support, you must configure the VP9 decoder context to use an inspection callback. This callback is executed for every decoded frame, providing access to the macroblock and block-level decisions, including motion vectors.

Include the inspection header in your application:

#include "vpx/vpx_decoder.h"
#include "vpx/vp8dx.h"
#include "vpx/vpx_frame_buffer.h"

Next, define the callback function matching the vpx_inspect_cb signature:

void on_frame_decoded(void *user_priv, const void *inspect_data) {
    const vpx_inspect_data *data = (const vpx_inspect_data *)inspect_data;
    
    int width = data->x_count;   // Number of blocks horizontally
    int height = data->y_count;  // Number of blocks vertically
    
    for (int y = 0; y < height; y++) {
        for (int x = 0; x < width; x++) {
            // Index to access the block's inspection data
            int block_idx = y * width + x;
            vpx_inspect_block block = data->blocks[block_idx];
            
            // Check if the block uses inter-prediction (has motion vectors)
            if (block.ref_frame[0] > 0) { 
                int16_t mv_x = block.mv[0].as_mv.col;
                int16_t mv_y = block.mv[0].as_mv.row;
                
                // Process or store the motion vector coordinates
                printf("Block [%d,%d]: MV_X = %d, MV_Y = %d\n", x, y, mv_x, mv_y);
            }
        }
    }
}

Step 3: Configure the Decoder to Trigger the Callback

After initializing the decoder context, associate your custom callback function with the decoder using the vpx_codec_control API.

vpx_codec_ctx_t decoder;
vpx_codec_dec_cfg_t cfg = {0};

// Initialize VP9 decoder
if (vpx_codec_dec_init(&decoder, vpx_codec_vp9_dx(), &cfg, 0)) {
    fprintf(stderr, "Failed to initialize decoder\n");
    return;
}

// Prepare the inspection configuration structure
vpx_inspect_init inspect_config;
inspect_config.inspect_cb = on_frame_decoded;
inspect_config.inspect_priv = NULL; // Pass custom user data pointer here if needed

// Set the control to register the inspection callback
if (vpx_codec_control(&decoder, VP9_SET_INSPECTION_CALLBACK, &inspect_config)) {
    fprintf(stderr, "Failed to configure inspection callback\n");
}

Step 4: Decode the Stream

Feed the compressed VP9 stream packets into the decoder normally using vpx_codec_decode. For every frame processed, the decoder automatically executes the on_frame_decoded callback, allowing you to stream or log the block coordinates and corresponding motion vector values in real-time.

while (get_next_compressed_packet(&packet)) {
    vpx_codec_decode(&decoder, packet.data, packet.size, NULL, 0);
    
    // The inspection callback is triggered internally during the decode call.
    vpx_codec_iter_t iter = NULL;
    vpx_image_t *img = vpx_codec_get_frame(&decoder, &iter);
    while (img) {
        // Handle output frame buffer if rendering or saving
        img = vpx_codec_get_frame(&decoder, &iter);
    }
}

Alternative Method: Using FFmpeg wrapper for libvpx

If writing raw libvpx C code is not required for your workflow, you can extract motion vectors using FFmpeg’s libvpx wrapper. This approach exposes motion vector data via the export_side_data flag.

Run the following command to export frame motion vectors to a readable side data format:

ffmpeg -flags2 +export_mvs -c:v libvpx-vp9 -i input.webm -vf codecview=mv=pf+bf+bb output.mp4

In an FFmpeg-based C application, you can read this data directly from decoded frames by querying the AV_FRAME_DATA_MOTION_VECTORS side data type.