Raw Footage Timeline & Dialogue Indexer (Video Intel)
An in-memory streaming video analysis engine that indexes 4.5 hours of commercial dailies in 18 minutes, enforcing a strict scope boundary against subjective editing advice.
Scrubbing 4.5 Hours of Dailies at 4x Speed
Commercial video editors and assistants waste significant billable time on routine timeline scrubbing. A single production day produces 4 to 8 hours of raw camera dailies: handheld B-roll, continuous product spins, and multiple takes of unscripted interviews.
When assembling a 60-second commercial spot, an assistant editor spends half a day scrubbing the timeline at 4x speed: dropping markers where an interviewee spoke a key soundbite, or hunting through dozens of takes for a 2-second steady insert shot of a coffee pour.
Traditional AI video services attempt to solve this by dumping thousands of frames to cloud vision APIs, running up massive bills, or by generating unhelpful subjective summaries (e.g. “A creative shot that feels inspiring”) that editors immediately disregard.
Scene Detection Is Pixel Math. Indexing Requires a Strict Scope Boundary.
The core design constraint: scene detection is deterministic math, not AI. An exact RGB difference matrix between consecutive frames computed in NumPy is either above a cut threshold or it is not. “Frame 4,832 begins a new scene” is an objective physical fact.
The second constraint is objective indexing: AI video tools frequently overstep by telling editors “You should cut this scene” or “This clip would work great as an intro.” Video Intelligence is strictly an indexing and retrieval catalog.
The tool indexes footage by timecode, not editorial opinion. How to cut is the editor's call.
The scope boundary enforces this: the JSON schema contains no recommendation keys (additionalProperties: false), the prompt explicitly forbids editorial suggestions, and offline tests grep output HTML for forbidden advisory words.
In-Memory Streaming to NLE Markers in 18 Minutes
Video Intelligence analyzes a 4.5-hour shoot session in under 18 minutes: FFmpeg streams 320x180 RGB buffers directly into NumPy arrays for in-memory scene cut detection (no temporary image files on disk); faster-whisper generates monotonic word-level transcripts aligned to video frames; keyframes are tagged with standard cinematography taxonomy; and the scope boundary ensures no subjective advice.
The output exports directly to Adobe Premiere Pro EDL/CSV markers, Final Cut Pro FCPXML sequence files, and a self-contained static HTML dailies viewer with sub-second quote search. Total API spend: < $0.75 for an entire 4.5h session.
{ "session": "Maison_Living_Commercial_D1", "footage_duration_sec": 16200.0, "processing_time_min": 17.8, "scenes_detected": 48, "words_transcribed": 8420, "nle_markers_exported": ["markers.edl", "timeline.fcpxml"], "editorial_scope_violations": 0, "total_api_spend_usd": 0.68 }
6-Phase In-Memory Streaming & Indexing Pipeline
Video and audio are analyzed in parallel with zero temporary disk image buffers. Click any phase card to inspect its implementation, input state, and output state.
FFprobe Introspect ($0.00) → In-Memory RGB Cuts ($0.00) → Whisper Alignment ($0.00) → B-Roll Tagging (~$0.01) → Scope Guard ($0.00) → NLE Export ($0.00)
Phase 01 — Media Introspection
Runs non-destructive stream introspection via FFprobe. Extracts exact video stream parameters: container format, codec ID, pixel format (YUV420p vs ProRes 422HQ), exact framerate (23.976 vs 29.97), and audio sample rate.
Three Decisions That Define the Architecture
In-Memory RGB Video Streaming via FFmpeg Stdout Pipe
Decoding 4.5 hours of 4K video to disk as JPEG image files produces over 300,000 files and burns 60GB+ of SSD write cycles. It turns a compute-bound task into an I/O bottleneck.
Video Intelligence pipes downsampled 320x180 raw RGB24 frames directly from FFmpeg stdout into an in-memory byte buffer read by NumPy. Consecutive frame delta matrices are computed on the fly at over 120 fps. No temporary files are written to disk.
def phase_scene_detect(pipe: IO[bytes]) -> list[SceneCut]: cuts = [] prev_frame = None frame_idx = 0 while raw := pipe.read(320 * 180 * 3): curr_frame = np.frombuffer(raw, dtype=np.uint8) if prev_frame is not None: delta = np.mean(np.abs(curr_frame - prev_frame)) if delta > SCENE_CUT_THRESHOLD: cuts.append(compute_timecode(frame_idx, fps)) prev_frame = curr_frame frame_idx += 1 return cuts
The Editorial Scope Boundary
The biggest failure mode in AI video tooling is overstepping: models giving subjective directorial advice that insults the editor’s judgment. Video Intelligence enforces defensive layers against editorializing:
Layer 1: The Pydantic schema strictly defines objective fields (framing, camera_motion, dialogue_quote) with additionalProperties: false. Layer 2: System prompt explicitly commands: “Output factual observations only. Never suggest cut points, pacing, or scene placement.” Layer 3: The automated test suite greps all output JSON and HTML reports for forbidden advisory terms (should, recommend, best take, cut here).
FORBIDDEN_EDITORIAL_TERMS = ["should", "recommend", "suggest", "best take", "cut here"] def phase_assert_scope(catalog: VideoCatalog) -> None: payload = catalog.to_json().lower() for word in FORBIDDEN_EDITORIAL_TERMS: if word in payload: raise ScopeViolationError(f"Found forbidden editorial term: {word}")
Native Premiere EDL and Final Cut FCPXML Timeline Markers
A standalone web catalog is useless if the editor has to manually type timecodes into their editing software. Video Intelligence outputs industry-standard timeline marker formats directly into the project directory.
It compiles both an Adobe Premiere Pro Edit Decision List (markers.edl) with colored timeline flags and an Apple Final Cut Pro XML (sequence.fcpxml) file. When imported into Premiere or FCP, all soundbites and visual scene markers align to the nearest frame boundary.
def phase_export_timeline(catalog: VideoCatalog, out_dir: Path) -> None: write_premiere_edl(catalog, out_dir / "markers.edl") write_fcpxml(catalog, out_dir / "sequence.fcpxml") write_catalog_html(catalog, out_dir / "search_catalog.html")
Run the Full Suite Locally in Under 5 Seconds
The full test suite runs offline against pre-rendered synthetic MP4 fixtures and pre-cached vision API responses. No API keys, no network calls, no spend required to verify the implementation.