Video Pipeline — Case Study

Oct 20, 2026 · @Gary

Overview

A Python pipeline that turns a shot list into a finished, narrated and captioned animated video for Thai children. It replaces an all-in-one AI video site with a pipeline where only two steps cost money, video generation and the narration voice; everything else runs locally on Apple Silicon.

Built for	A Thai children's content creator
Runs on	16GB M2 MacBook (an M1 Mac mini until October 5)
Stack	Python, ffmpeg, SDXL (local, mps) with IP-Adapter, Seedance 1 Pro image-to-video and ElevenLabs v4 voice on fal.ai, whisper.cpp
Output	1280x720, 24fps, Thai narration with burned-in Thai captions
Role	Research, architecture, implementation

The problem

The creator wanted full-motion, 3D-cartoon style episodes, and was looking at subscription sites like everygen.ai that hide script, image and video generation behind one prompt. Those sites charge a margin on every step, including steps a laptop can do for free.

Three constraints shaped the build:

• Quality. The reference samples were full 3D animation, not narrated stills. Local video models on a 16GB Mac cannot reach that, so motion has to come from a hosted model such as Seedance, Kling or Veo.
• Cost. Generated video is billed per second, and a careless re-run pays again. Spend had to be visible before it happened.
• Language. Thai is tonal, and TTS and lip-sync support for it was unproven. The first design kept a human narrator and added Thai captions; that changed once the creator heard current Thai TTS voices (below).

Architecture

[embedded content: pipeline flow · two paid steps]

Each shot goes either to a paid hosted model or to a free local zoom, and the narration voice is a small paid call; frames, captions and assembly never leave the Mac.

Engineering decisions

Estimate before you spend. --estimate reads the episode, counts paid seconds and prints the expected cost, then stops. The four-shot sample episode came to $2.42 per attempt at the fal rate at the time. Pricing came from the live fal dashboard, not documentation, after the documented Seedance rate turned out to be stale. The first real episode, 17 shots and 160 seconds, estimates at $8.75 a final attempt and $3.50 a draft, against a budget of about $20 per finished video, retries included.

Never pay twice. Every stage skips output that already exists. The raw clip downloaded from fal is stored apart from the re-encoded clip, so changing resolution, trim or encoding never triggers a new paid call. Retrying a shot is a deliberate act: delete its two clip files.

A cost dial in the shot list. Each shot is "generated" (paid AI motion) or "still" (a free ffmpeg pan and zoom over the start frame). Establishing shots and narration-heavy moments work as stills, so the creator trades cost against motion per shot, in the JSON, without touching code. In practice the creator wants movement in every shot, so the first episode is all generated and the cost lever moved to the choice of model.

Human override at every step. A PNG dropped into the frames folder replaces the SDXL frame for that shot, and a hand-edited captions.srt is used as is. This matters because plain SDXL prompts do not keep a character consistent between shots; curated frames and reference images do (below).

One machine, kept awake. Work moved to the M2 MacBook on October 5, and every stage, SDXL included, now runs there. Clearing the MPS cache and tiling the VAE hold SDXL at about 8.7GB, where it had grown to 15GB and swapped. Long runs start under caffeinate -i: the first one on the MacBook stalled for 15 to 30 minutes at a time with the lid closed.

Hand-off built in. The project ships with a CLAUDE.md covering setup, architecture, cost rules and exactly what is untested, so work continues in Claude Code with no lost context. The code is pycodestyle clean.

Characters drawn from references. Plain SDXL redrew the ant in every frame, sometimes with wings or as a crowd. Each shot now names its characters and is drawn with IP-Adapter from a reference image of the character alone on a plain background. Getting there took four rounds:

• Adapter scale 0.25 for the ant; 0.5 for the more detailed butterfly, whose face and wings drifted at 0.25.
• One subject per frame: an ant beside a caterpillar or butterfly merged into one creature on every seed, so the narration carries the second subject.
• "wings" and "other ants" in the ant shots' negative prompts.
• Three seeds per failing shot, then pick; the chosen seed is stored in the shot.

A model picked by the budget. Motion in every shot at about three minutes ruled out Seedance 2.0 Fast, at $0.2419 a second or about $43 an episode. Seedance 1 Pro on fal costs $0.054 a second at 720p. It makes 2 to 12 seconds per clip, so a longer shot is chained from the last frame of the clip before. Every motion prompt is drafted first on Seedance 1 Pro Fast at $0.0216 a second.

A voice chosen by ear. Thai listeners recognised the free macOS Thai voice as Siri, and the creator was not confident recording her own narration. She compared six Thai samples from MiniMax, ElevenLabs and Gemini and chose ElevenLabs v4 "Sarah", about $0.12 an episode. Each spoken line is cached with the text and voice that made it, so an unchanged line is never paid for twice.

Captions from the script, not from speech. whisper.cpp heard "หมด" for "มด" (ant) and "นอน" for "หนอน" (caterpillar) throughout the Thai narration. Synthesised narration is now captioned straight from its text, timed by each line's spoken length; whisper stays for a real recording.

Timing follows the voice. Each shot runs for its spoken line plus about 1.5 seconds, so switching to Sarah took the episode from 177 to 160 seconds. A clip paid for at an older, shorter timing now stops the run rather than leaving its shot short.

Status and next

Every stage has now run on the Macs. The four-shot sample episode ran end to end on October 4 through Seedance 2.0 Fast: a 20-second 1280x720 file with narration and burned-in captions, with Thai tone marks and stacked vowels drawn correctly in Thonburi.

The first real episode, a 17-shot, 160-second story about an ant and a caterpillar, has its characters and frame seeds settled and its narration made in Sarah's voice. Its earlier drafts ran on Seedance 1 Pro Fast at the old Kanya timing. Next is a draft at Sarah's timing (about $3.50), then the final run (about $8.75).