Skip to content
Huzaifa Qadeer
Back to all work

Source IT

One API for footage and asset sourcing across licensed and public-domain libraries.

Role
Solo — API, queueing layer, front end
Status
FinishedHosted
Called by
Context

Why it exists

Sourcing b-roll by hand for one video took close to a day: several libraries searched in turn, licenses checked one at a time, credits tracked manually. Source IT collapses that into a single request, and it is now the acquisition layer under both pipelines.

Architecture

How it is put together

Three layers. Adapters speak each library’s own API, a scheduling and deduplication core decides what is worth fetching, and one normalized surface is what MotionGen AI and the front end call.
  1. 01

    Provider adapters

    Licensed and public-domain sources

    • Pexels
    • Pixabay
    • Licensed search-engine results
    • Internet Archive
    • U.S. National Archives (NARA)
  2. 02

    Core

    Queueing, deduplication, fetching

    • Per-provider rate limiting and backoff
    • Priority queue with retries
    • Cross-provider deduplication
    • Partial fetching to bound bandwidth and storage
  3. 03

    API surface

    One normalized response

    • Same response shape regardless of source
    • Attribution metadata per clip
    • Consumed by MotionGen AI and by the standalone front end
Decisions

Trade-offs

  • Extract sourcing into its own service with its own API

    Instead of Keep it as a module inside MotionGen AI

    Manual sourcing cost about a day of b-roll work per video, and both pipelines needed the same thing. The cost is a second service to deploy, monitor and hold provider credentials for, plus a network hop on every asset request.

  • Fetch only part of a candidate file before deciding on it

    Instead of Download every candidate in full and evaluate it locally

    Bandwidth and storage stay bounded as request volume grows. The cost is that a candidate is judged on partial data, so some clips are rejected on evidence a full download would have contradicted.

  • One normalized response shape across every provider

    Instead of Pass each library’s native response straight through to the caller

    Callers never branch on which library a clip came from. The cost is that provider-specific fields are dropped in normalization, and a new library cannot be used at all until an adapter is written for it.

Constraints

Limits and how they are handled

Every provider enforces its own rate limits, and they do not agree with each other.
Per-provider backoff behind a priority queue, with retries spread rather than bunched.
A clip cannot be used without its license and credit line.
Attribution metadata is part of the response, attached to each clip.
Stack

Built with

  • HTTP API
  • AWS
Links

Where to look next

The repository is private. Ask for a walkthrough.