Clip Trimmer:
From 0 to 55 Tests in One Month

The Mechanical Part of Video Editing

Short-form vertical video has a specific format that does not vary: 9:16 aspect ratio, face centred, captions on screen, one or two moments where the camera punches in for emphasis. If you're producing more than a few clips, you will spend a significant amount of time doing the same things in the same order.

I am not a video editor. I do not enjoy editing video. I wanted a tool I could point at a raw recording and get a usable vertical clip out of, with the mechanical work handled automatically. That is where Clip Trimmer started — not as a product, not as something to ship, but as something I needed to stop doing by hand.

The Pipeline

The tool takes a landscape recording and produces a portrait clip. The full pipeline is: detect face position → compute crop window → transcribe audio → generate word-level timestamps → identify high-energy moments → apply punch zoom → burn captions → output. In practice each of those steps has at least one failure mode I did not anticipate.

Whisper Timestamps: Useful, But They Drift

I use an open-source speech recognition model for transcription. It can return word-level timestamps — not just what was said, but when each word was spoken. That is exactly what active-word caption highlighting requires.

The problem is timestamp drift. Whisper's word-level timestamps are computed relative to an internal audio segmentation that does not always align with the real audio clock. On longer clips, the reported timestamps can drift noticeably from where the words actually land — enough to make captions feel subtly out of sync without being obviously broken.

The fix I use is to align reported timestamps against the actual audio waveform: when a segment boundary and an energy boundary in the audio are close, I trust the measured audio over the model's estimate. It handles most drift cases. Background noise situations can produce false positives — those remain unsolved.

Face Tracking: Simpler Than I Thought, Then Harder

I use a face detection library to find the face position in each frame, then compute a 9:16 crop window centred on the face. For the first few clips this worked immediately and I thought this was going to be the easy part.

It is not the easy part. The problems I ran into:

• Face partially out of frame. When the speaker leans sideways the detected face box can extend past the video edge. The crop window needs to be clamped, but clamping it too aggressively makes the face off-centre. There is a tradeoff here that I tuned by hand.
• No face detected. Looking down at notes, turning sideways briefly, reaching for something. The tool needs to hold the last known position and drift back to centre gradually, not snap or freeze.
• Processing speed. Running face detection on every frame of a long clip is slow. I sample frames at intervals and interpolate positions in between. This is fast enough and smooth enough that the interpolation is not visible.

The crop coordinates go into an FFmpeg crop filter with the computed x,y values. FFmpeg takes it from there.

Punch Zoom: Timing Is Everything

Punch zoom is a brief scale-up applied at a moment of emphasis. In manually edited video, the editor chooses those moments by feel. I automate it using audio energy: a peak in the audio energy envelope above a threshold triggers a zoom, with a cooldown period between triggers to prevent rapid-fire stacking.

This works most of the time. It breaks when someone speaks quietly throughout (no energy peaks, no zooms) or when there are ambient sounds (a knock on a desk triggers a zoom on the wrong frame). Neither of those has a clean automated solution. The current version does not try to solve them — it just flags in the output log when a zoom trigger looked suspicious, so you can review it.

ASS Captions: The Format Nobody Loves

The caption format I use is ASS — Advanced SubStation Alpha. It is a 1990s subtitle format with an XML-era aesthetic that has not aged gracefully. The documentation is scattered across forum posts from 2004. The parameter names are inconsistent. The colour encoding is &H00FFFFFF& in BGRA order, which is backwards from every other colour format in existence.

I use it anyway because it is the only widely-supported format that can highlight the currently spoken word in a different colour while the rest of the caption stays on screen. SRT cannot do this. VTT cannot do this. ASS can, via per-word colour overrides embedded inline. The FFmpeg ass filter burns it into the video.

It works, it looks good, and every time I look at the raw ASS file I feel a small amount of regret about the format choice. There is no better option.

The 55 Tests

The test suite did not start with a goal of 55 tests. It started with one test: does the face detection return a bounding box given this image? The suite grew because things broke in ways I did not expect, and each breakage became a test.

A few examples of failures that became tests:

• FFmpeg exits with code 0 on some errors. An invalid filter chain, on certain versions of FFmpeg, returns exit code 0 and produces a zero-byte output file. The first time this happened I thought the clip had exported successfully. Now I check output file size and duration.
• Whisper returns an empty segments list on silence. A recording that starts with 3 seconds of silence before speech confuses the VAD (voice activity detection) in some Whisper configurations. The caption writer must handle an empty input without crashing.
• Face crop coordinates must be even numbers. FFmpeg's crop filter requires even-number dimensions for YUV encoding. A face at an odd pixel position produces a 1-pixel rounding error that causes a warning on some codec configs and an error on others. Now all crop values are rounded down to even.
• Punch zoom at the very end of a clip. A zoom triggered near the end of a clip produces an FFmpeg filter chain error because the output would extend past the clip duration. Now zoom triggers too close to the end are discarded.

CI/CD: Catching Things I Miss

The test suite runs on every push via a CI pipeline. The pipeline installs dependencies, runs the full test suite against a set of fixture clips, and fails the build if anything breaks.

Before I had CI, I broke the caption writer twice by refactoring the timestamp alignment code and not realising the change affected caption timing. Both times I only noticed when I watched an exported clip and the captions were wrong. With CI, those failures show up as red in the push — before the clip is ever rendered.

Three bugs that CI caught before they shipped:

1. A change to the energy envelope normalisation that made zoom triggers fire on nearly every frame.
2. A path handling bug on paths with spaces that only failed on my second test machine.
3. A Whisper model version mismatch where the output dictionary structure changed between model versions.

None of those are interesting bugs. They are exactly the kind of dull, boring regression that a test suite exists to catch so I do not have to catch them manually.

What Is Next

V2 is complete. The pipeline works end to end. The test suite covers the known failure modes. The E2E test on real production clips is still in progress — it is the last thing before I consider this genuinely reliable for regular use.

V3 is real work. Smart tracking means following the active speaker even when they move significantly — current tracking loses the face if the person walks out of the initial frame. AI-guided cuts means identifying automatically which 60 seconds of a 10-minute recording are worth clipping — that requires understanding content, not just audio energy.

Both of those are months away. I am not building them until V2 is proven stable on real content. Building features on top of a shaky foundation is how you end up with a complicated tool that does not work reliably.