I came back from a trip with my phone full of memories. Photos, videos, random moments, everything was there.

The next day I wanted to turn them into a reel. But editing meant going through every clip, finding the right moments, syncing everything to the music, adding captions, adjusting colours and getting the pacing right. It felt like another project on my to-do list, and the reel didn't happen.

So I asked a different question: what if an AI agent made the first cut for me? That became Reel Agent, and the reel below was made with it from my raw trip clips and a song.

What It Does

You upload raw clips, photos and a song. Then you describe the reel you want, or pick a quick action: auto-edit, stronger hook, cut silences, faster pacing, beat sync, cinematic look. The agent returns a complete 9:16 timeline with:

  • A hook in the first seconds and pacing that fits the song
  • Cuts snapped to the music's beat grid
  • Word-by-word captions from a local Whisper transcript
  • Text overlays, transitions and a colour look
  • A caption and hashtags for the post

AI Does the First 80%, You Control the Last 20%

I didn't want a black box that spits out an MP4. Every edit the agent makes lands in a real timeline editor as one undoable step. You can trim, reorder, split, restyle captions and drag text around the preview, or ask the agent for another pass: "make the hook stronger", "cut the silences". The conversation is remembered per project, so each request builds on the last one.

How the Agent Sees Your Footage

A language model can't watch a video file, so every upload is turned into things it can read. ffmpeg probes the file, then builds a thumbnail, a 4×2 contact sheet of frames and a 720p proxy for smooth scrubbing. Whisper produces word-level timestamps for anything spoken. For the song, the server works out the tempo, the beat grid and its strongest 15 seconds.

On every turn, Claude Code gets all of it: durations, orientation, transcripts, BPM and beats. It opens the contact sheets with its Read tool to actually look at the footage, then answers with the whole timeline as structured JSON. The server validates that JSON against a schema before the editor applies it.

How the server calls the agent
claude -p \
  --output-format stream-json \
  --json-schema '<timeline schema>' \
  --append-system-prompt '<reel playbook>' \
  --tools Read --allowedTools Read

The agent only gets the Read tool: no shell, no web, no write access. It can look at your footage and propose an edit, nothing else. How it edits lives in one system prompt (the reel playbook) and one output schema, so changing its style is a text edit.

An Export That Matches the Preview

Captions and text are drawn by the same canvas code in the preview and in the export, one image per distinct frame, and ffmpeg composites them into a 1080×1920 MP4. What you see in the editor is what you post.

The Stack

  • Claude Code: the agent's brain, one CLI session per project
  • Whisper (whisper.cpp): local speech-to-text with word timestamps
  • ffmpeg: probing, proxies and the final render
  • Next.js + Tailwind: the timeline editor
  • Node.js + Express: the API

Try It

Reel Agent is open source under the MIT license and runs locally with your own Claude Code account: github.com/umeshmore45/reel-agent.

Sometimes the best side projects don't start with a business idea. They start with a problem you personally want to solve. Mine was a phone full of clips I never edited.