Automated YouTube & Media Asset Pipeline: FastAPI, DuckDB, Polars & Local LLMs

Overview

Publishing technical engineering walkthroughs on YouTube demands heavy post-production metadata generation: extracting accurate timecodes, generating chapter summaries, drafting code-syntax-aware descriptions, and formatting SEO tags.

Rather than relying on manual copy-pasting or generic ChatGPT prompts with token truncation, I engineered an automated, local-first media asset processing pipeline.

Architecture Blueprint

Why Polars + DuckDB Over Pandas?

Traditional Python pipelines rely on Pandas for tabular data manipulation. In this media pipeline:

1. Zero-Copy Memory Footprint: Transcripts with word-level timestamps generate tens of thousands of event rows. Polars leverages Arrow columnar memory with multithreaded SIMD operations, parsing timestamps 8x faster than Pandas.
2. DuckDB In-Memory OLAP: Embedded DuckDB allows fast analytical SQL queries over transcript chunks (e.g., finding the longest pauses, topic transition boundaries, and query-by-keyword) directly in memory with zero database daemon overhead.

import polars as pl
import duckdb

def analyze_transcript_pacing(words_df: pl.DataFrame) -> pl.DataFrame:
    """Calculates words per minute and identifies topic transition pauses."""
    return words_df.with_columns([
        (pl.col("start_time").diff()).alias("pause_duration"),
        (pl.col("word_count") / (pl.col("duration") / 60.0)).alias("wpm")
    ]).filter(pl.col("pause_duration") > 2.5)

Structured Output Enforcement via Pydantic

To prevent LLM hallucination in chapter timecodes, outputs are strictly bounded using Pydantic schemas:

from pydantic import BaseModel, Field
from typing import List

class ChapterMarker(BaseModel):
    timestamp: str = Field(pattern=r"^\d{2}:\d{2}(:\d{2})?$", description="MM:SS or HH:MM:SS format")
    title: str = Field(min_length=5, max_length=60)
    summary: str

class VideoMetadataPackage(BaseModel):
    video_title: str
    seo_tags: List[str]
    chapters: List[ChapterMarker]
    github_references: List[str]

Key Results & Takeaways

• Time Saved per Video: Reduced post-recording metadata and companion blog generation from 2 hours to under 4 minutes.
• Completely Local & Free: Runs locally on Apple Silicon using Ollama and local models, maintaining 100% privacy and $0 API spend.
• YouTube Companion Integration: Generates the exact timestamps and markdown guides published on this portfolio.