Lab · Tutorial

Video to notes: what is said, and what is shown

By the team at LARGE FORMAT · October 30, 2026

A lot of what matters in a video is never said out loud. A tutorial shows the command instead of reading it. A reel puts its one key line in a caption. A demo flashes the price on screen while the voice-over talks about something else. If you turn videos into notes from the audio alone, all of that is lost.

We built a small open-source tool, video2notes, that keeps two tracks. The spoken track is the transcript, fetched with Supadata's official Python SDK. The on-screen track is the text in the frames, read with ffmpeg and tesseract. Both go into one markdown note, side by side but never mixed. This page walks through how it works, with the code.

Walkthrough coming soonA short screen recording of the tool in use. It will open on YouTube.

Nothing from YouTube loads on this page.

1. The problem

Speech-to-text is a solved problem for most public videos. Supadata returns the captions when a video has them, and can generate a transcript when it does not. But a transcript is only half of a video.

In our sample run on a 90-second explainer, the captions heard the creator's name as "JJ Lair". The title card on screen spelled it correctly: JJ Allaire. Code, file names, version numbers and URLs do even worse in captions, because nobody reads them out.

The fix is simple to state. Read the screen too, and keep the two sources apart, so you always know where a line came from.

2. Setup

You need Python 3.10 or newer, ffmpeg, tesseract and a Supadata API key.

brew install ffmpeg tesseract          # or: sudo apt install ffmpeg tesseract-ocr
git clone https://github.com/lrgfrmt/video2notes.git
cd video2notes
python -m venv .venv && source .venv/bin/activate
pip install -e .
export SUPADATA_API_KEY=your-key-here

The key is read from the environment and never printed. The repo's .env.example holds the variable name and nothing else. Job polling, used below, arrived in version 1.7.0 of the SDK. Until that version is on PyPI, the project installs the SDK from Supadata's GitHub repository at a pinned commit.

3. Fetching a transcript with the SDK

One call does most of the work. Supadata answers in one of two ways. If the video has captions, you get the transcript straight away. If it has to generate one, you get a job ID instead, and you poll until the job is done. The SDK returns a different object in each case, so check for job_id:

import os, time
from supadata import Supadata, SupadataError

supadata = Supadata(api_key=os.environ["SUPADATA_API_KEY"])

result = supadata.transcript(url=url, lang="en", text=False, mode="auto")

if getattr(result, "job_id", None):          # async: a transcript is being generated
    while True:
        job = supadata.transcript.get_job_status(result.job_id)
        if job.status == "completed":
            result = job.result
            break
        if job.status == "failed":
            raise RuntimeError(job.error)
        time.sleep(2)                          # still "queued" or "active"

for chunk in result.content:                   # offset is in milliseconds
    print(chunk.offset / 1000, chunk.text)

The mode argument decides what you pay for. native only uses captions that already exist. generate always transcribes the audio. auto tries captions first. Each chunk carries its start time in milliseconds. We convert it to seconds so it lines up with the frame timestamps later. The real code also sets a timeout, so a stuck job fails with a clear message instead of hanging, and wraps SupadataError in a readable error.

4. Pulling on-screen text

The second track runs entirely on your machine and uses no Supadata credits. It has four steps.

Download. yt-dlp fetches the video at up to 720p. We only need the picture, because the audio is already covered by the transcript.

Pick frames worth reading. Reading every frame would be slow and repetitive. Instead, ffmpeg keeps a frame whenever the picture changes a lot, plus at least one frame every two seconds, so slow fades and static slides are not missed:

ffmpeg -i video.mp4 \
  -vf "select='gt(scene,0.3)+isnan(prev_selected_t)+gte(t-prev_selected_t,2)',showinfo" \
  -fps_mode vfr frames/frame_%05d.png

The showinfo filter logs each kept frame's timestamp. We read those times from ffmpeg's log and pair them, in order, with the saved images.

Read each frame. Raw OCR on video frames is noisy. Textures, icons and motion blur come back as letters. Asking tesseract for TSV output gives a confidence score for every word. We rebuild each line and keep it only when its average confidence is 75 or higher:

proc = subprocess.run(["tesseract", frame, "stdout", "--psm", "3", "tsv"],
                      capture_output=True, text=True)
lines = {}
for row in proc.stdout.splitlines()[1:]:
    cols = row.split("\t")
    if len(cols) < 12 or cols[0] != "5" or not cols[11].strip():
        continue                                  # level 5 = one word
    key = tuple(cols[1:5])                        # page, block, paragraph, line
    lines.setdefault(key, []).append((cols[11], float(cols[10])))

kept = [" ".join(w for w, _ in words) for words in lines.values()
        if sum(c for _, c in words) / len(words) >= 75]

On our sample, an unfiltered first pass returned 573 lines. The filtered pass returned 220, and almost all of the junk was gone.

Remove repeats. A caption that stays up for ten seconds would otherwise appear five times. We normalise each line, compare it with what we have already kept, and drop near matches and partial reads:

def normalise(text):
    return re.sub(r"[^a-z0-9]+", " ", text.lower()).strip()

kept, seen = [], []
for line in sorted(lines, key=lambda l: l.start):
    key = normalise(line.text)
    if any((len(key) >= 8 and key in other)
           or difflib.SequenceMatcher(None, key, other).ratio() >= 0.85
           for other in seen):
        continue
    kept.append(line)
    seen.append(key)

Each line keeps the time it first appeared, which is the moment that matters.

5. Merging into one note

The note keeps the two tracks apart on purpose. Spoken text goes under ## Spoken, grouped into paragraphs of about 30 seconds, each with a timestamp. A divider follows. Then comes ## On screen, a timestamped list with lines from the same frame on one row. Both tracks use the same clock, so you can line up what was said at 0:30 with what was on screen at 0:30.

The whole thing runs as one command:

video2notes "https://www.youtube.com/watch?v=_20US068pzk" -o notes.md --mode native

Use --no-onscreen for a transcript-only note. --scene-threshold and --floor-seconds tune how many frames are read.

6. Sample output

Here is the start of the real note for a Creative Commons video, quarto in 100 seconds by xvzf. The full file is in the repo's examples folder.

# quarto in 100 seconds

Source: <https://www.youtube.com/watch?v=_20US068pzk>

## Spoken

**[00:00]** quarto a scientific and Technical publishing system built on pandok the universal file converter was announced July 28 2022 created by JJ Lair team and open source contributors at positive formerly known as our studio ...

---

## On screen

- **[00:02]** a universal document cc
- **[00:04]** a universal document converter
- **[00:11]** JJ Allaire
- **[00:26]** neovim · Jupyter
- **[00:30]** title: "matplotlib demo" · format: · code-fold: true · jupyter: python3 ...

From examples/sample-output.md, trimmed

Read the two sections against each other. The captions say "JJ Lair", "pandok" and "positive formerly known as our studio". The screen says JJ Allaire, and shows the actual code and command-line help that the voice only mentions. Neither track is complete alone. Together they are a usable note.

7. Limits and costs

Public videos only. There is no support for logins, cookies, or private or members-only videos, by design.
OCR is not perfect. Stylised fonts, motion blur and busy backgrounds cause misreads. The note says so under its On screen heading.
Credits. Each run makes one transcript request to Supadata, plus status checks if a transcript has to be generated. Credits per call are listed on Supadata's pricing page. Use --mode native if you only want existing captions.
Time. The download and OCR run locally and cost nothing but time. Our 86-second sample produced 50 frames and finished in about 25 seconds on a laptop.

8. Links

The code, MIT licensed: github.com/lrgfrmt/video2notes
Supadata: supadata.ai (referral link)
Supadata Python SDK docs: docs.supadata.ai/integrations/python
yt-dlp: github.com/yt-dlp/yt-dlp
Tesseract OCR: github.com/tesseract-ocr/tesseract

Questions, or a video workflow you want built? Start a project with the team at LARGE FORMAT.

Lab · Video-to-notes.txt