The Problem
Long meeting recordings and talks hold decisions and action items that nobody has time to recover. Manually scrubbing hours of audio to find who owns what, and what was actually decided, is slow, error-prone work that never happens.
The Solution
A self-hosted pipeline that turns any YouTube URL or recording into a structured knowledge base: audio is downloaded and chunked, transcribed with Whisper (or Sarvam AI for Hinglish audio, translated to English), then Mistral produces a session title, map-reduce summary, action items with owners and deadlines, key decisions, and open questions. A Chroma + MiniLM RAG layer lets users chat with the transcript and get grounded answers with a strict 'not in the transcript' fallback.
How the workflow runs
01 · Ingest
Download YouTube audio or convert any local file to 16 kHz mono WAV via pydub/FFmpeg.
02 · Chunk
Split audio into 10-minute chunks; Hinglish pieces are further sliced to ≤25s to satisfy Sarvam's 30s API limit.
03 · Transcribe
Route each chunk to Whisper (English) or Sarvam speech-to-text-translate (Hinglish → English) and join the transcript.
04 · Analyse
LangChain LCEL chains over Mistral generate the title, map-reduce summary, action items, key decisions, and open questions.
05 · Index
Transcript is split (500 chars, 50 overlap), embedded with MiniLM, and persisted in a local Chroma collection.
06 · Chat
User questions retrieve the top-4 chunks and stream grounded answers — or an honest 'not found' when context is missing.
AI layer
- Whisper (local)
- Configurable model size (tiny → large-v3) for offline English transcription.
- Sarvam AI STT-translate
- Transcribes Hinglish speech and translates to English in one pass, with 25s slicing to respect API limits.
- Mistral Small via LCEL
- Single composable chain style across summarisation, extraction, and chat — temperature-tuned per task.
- RAG pipeline
- Chroma + all-MiniLM-L6-v2 embeddings with a strict grounded-answer system prompt to prevent hallucination.
Automation layer
- Format-agnostic intake
- One entry point detects URL vs file path and handles download, conversion, and normalisation automatically.
- Chunked batch processing
- Long audio processed chunk-by-chunk with live pipeline status in the UI — no manual steps between stages.
- Persistent vector store
- Chroma collection persisted to disk so transcripts stay queryable without re-embedding.
Technical challenges
Sarvam's sync API rejects audio longer than 30 seconds, breaking Hinglish transcription on real meetings.
Pre-slice each chunk into 25-second pieces with a 5s safety margin, send sequentially, and join transcripts — with temp-file cleanup guaranteed in a finally block.
LLM summaries degrade on hour-long transcripts that exceed the context window.
Map-reduce summarisation: RecursiveCharacterTextSplitter (3000/200) produces partial summaries that are combined into one professional brief.
RAG answers risk hallucinating details that were never said in the meeting.
System prompt constrains answers to retrieved context only, with an explicit fallback message when the answer isn't present.
Outcome
- Hours of meeting audio reduced to a one-screen brief: title, summary, actions with owners, decisions, and open questions.
- Transcript becomes queryable — 'what did we decide about X?' returns a grounded, quotable answer in seconds.
- Runs fully self-hosted: no transcript data leaves the machine except the LLM calls.
Lessons learned
- “Real-world STT APIs fail on edge cases (length limits, codecs) — defensive chunking and cleanup matter more than the happy path.”
- “Grounding prompts need explicit refusal language; 'only use the context' alone is not enough to stop filler answers.”
- “A dual interface (Streamlit + CLI) keeps the same pipeline testable end-to-end without a browser.”