Video Understanding Model Enablement

1. Why Do We Need AI Models That Understand Video?

As platforms like OTT, short-form, and live streaming continue to proliferate, the volume of video being produced is exploding. This has created three real-world challenges in the field:

  • Increasing time spent on platform-specific editing and repurposing
  • The burden of meta-tagging for search and classification
  • Growing time spent analyzing video content

Why Does Video Analysis Take So Long?

Video presents two fundamental challenges that other data types do not.

First, there is the sheer volume of data. Even a 10-second clip consists of 300 individual images at 30fps. As videos get longer, the data that must be processed grows linearly, creating fundamental constraints on memory and computation time.

Second, there is the challenge of temporal understanding. A frame showing someone “lifting a cup” and one showing them “setting a cup down” are visually indistinguishable in isolation. Understanding the context of motion across time is essential.

The Limits of Traditional Approaches

Conventional methods have relied on manual input — tagging, system metadata, subtitles — entered by people. This was manageable when video volumes were small, but is simply not feasible at today’s scale.

The Direction of Analysis Technology

Frame-based analysis
→ Analyzes individual images; cannot understand temporal context

Shot-based analysis
→ Analyzes at scene-cut boundaries; partial context reflected

Multimodal embedding-based analysis (Marengo)
→ Understands visuals, audio, and dialogue simultaneously; captures temporal context as well

2. Key Terms and Concepts

1. Embedding

  • The process of converting data into a numeric array (vector) that preserves semantic meaning
  • Difference from simple vectorization: simple vectorization assigns numbers by rule without capturing meaning, whereas embeddings are learned by an AI model so that semantic relationships are encoded in the numbers
  • The more similar the meaning, the more similar the resulting vectors
  • Not exclusive to image processing — a general-purpose concept used across AI, including text, recommendation systems, and RAG
  • Derived from the word “embed” — the idea of planting meaning into a vector space

2. Cosine Similarity

  • Measures how closely two vectors point in the same direction
  • Value range: -1 to 1 (closer to 1 means more similar)
  • Compares direction only, not vector magnitude
  • Core principle behind video search: compare a query vector against clip vectors and return the most similar clip

3. Marengo

  • TwelveLabs’ embedding model
  • Splits a video into clips and converts each into a 512-dimensional vector
  • Supports 3 modality embeddings: visual, audio, and transcription
  • Input types: text, image, text+image (synchronous API); video, audio (asynchronous API)
  • Text, images, and video are all mapped into the same 512-dimensional space, enabling cross-modal search
  • Does not generate video — it is a model for understanding video

4. Pegasus

  • TwelveLabs’ generative model
  • Watches a video and produces text descriptions, summaries, and question-answering
  • Does not create vectors, so direct search is not possible
  • However, text generated by Pegasus can be embedded via Marengo for indirect use in search
  • Does not generate video — it is a model for understanding video

3. From Video Search to Pipelines

image.png

image.png

1. Three Search Strategies

Background: Using only the visual modality leads to poor search quality for queries based on dialogue or sound. How the three modalities are combined is the key.

Fused Embedding

  • Combines all 3 modalities into a single vector via weighted average and stores it
  • Simple and fast, but requires regenerating all embeddings if weights change
  • No way to tell which modality contributed to a given result

Multi-Vector Fixed Weights

  • Stores all 3 modalities in separate indexes
  • At search time, queries each index individually and aggregates the scores
  • Changing weights in code takes effect immediately without regenerating embeddings (flexible)
  • Two score aggregation methods:
    • Score Fusion: multiplies each score by its weight and sums them; score ranges may differ across modalities
    • RRF (Reciprocal Rank Fusion): aggregates based on rank rather than raw score; fair combination even when score ranges differ; k=60 is the conventional constant
  • Weights are still fixed, so they cannot be automatically adjusted per query

Intent-based Dynamic Routing (= Dynamic Routing, same concept)

  • Analyzes query intent and automatically determines modality weights dynamically
  • Users simply make requests in natural language; the system adjusts weights internally
  • The most sophisticated approach

2. VOD Pipeline (1-3)

  • Upload video to S3 → EventBridge detects event → 3 Lambda functions run in parallel
    • Marengo Lambda: generates embeddings (asynchronous)
    • Pegasus Lambda: generates video summary
    • Metadata Lambda: saves to DynamoDB
  • Automates what was done manually in section 1-1 as an event-driven, serverless pipeline

3. Model Fragmentation and Orchestration

  • Reality: specialized models for video generation, embedding, text generation, speech, etc. continue to multiply
  • Two directions coexist: specialization (higher performance but higher cognitive overhead) vs. consolidation (more convenient but performance still lacking)
  • Solution direction: an orchestration layer
    • A general-purpose model (e.g., Claude) analyzes the prompt and dynamically calls the appropriate specialized model
    • Users and developers no longer need to manage individual models directly
  • MCP: not about routing model calls, but a standardized interface for connecting to external tools and services (S3, Slack, databases, etc.)
    • Distinct from model invocation (Bedrock API)