How auto-sync works · Score to Video

Timing a score-follow video by hand means listening all the way through and tapping at the start of every line. For a long piece, that’s a hundred taps or more, and every one of them has to land in the right place. Auto-sync does that work for you. It reads your score, listens to your recording, and works out where each line starts, usually before you’ve even reached the sync step.

Here’s what happens behind the scenes, and why it works as well as it does.

Step 1: Learn what the music should sound like

To follow a recording, auto-sync first needs to know which notes to expect and when. It gets them from the score in one of two ways:

From a MusicXML file, if you added one. This is the most accurate source, because it holds every pitch and rhythm exactly as written.
From the PDF itself, if it was exported from notation software like MuseScore, Dorico, or a recent version of Finale or Sibelius. Those PDFs quietly carry the shape and position of every note, and the studio can read them straight off the page.

A scan or a photo of a printed score doesn’t carry that information, and neither do PDFs made with some older music fonts. That’s why auto-sync is available for some PDFs and not others.

Auto-sync also accounts for transposing instruments. A written C on a B♭ clarinet actually sounds a B♭, so auto-sync works with the notes you’ll really hear, not the notes on the page. That way a band or orchestra score lines up with its recording.

Step 2: Count the bars on the page

Next, the studio looks at each line of music you cropped and counts the bar lines in it. Then it compares that count with the score. If the page says there are 212 bars and the score says there are 212 bars, the two almost certainly describe the same music.

If they’re off by a bar or two, the studio works out which lines it most likely miscounted and corrects them. If they’re off by a lot, something is wrong: maybe the PDF and the MusicXML file are different editions, or different pieces entirely. In that case auto-sync stops and tells you, rather than making a video that’s quietly wrong.

Step 3: Find the music in the recording

Recordings rarely start right on the downbeat. There might be a few seconds of room noise, a breath, some tuning, or a round of applause at the end.

Before it tries to match anything, auto-sync scans the recording to find where the music actually starts and stops. It tells applause apart from music by its texture, not just its volume: clapping is noisy and has no pitch, while even the quietest music has notes in it. That way the opening bars aren’t stretched to cover the silence before them.

Step 4: Turn both into the same “fingerprint”

A recording is sound and a score is symbols, so auto-sync can’t compare them directly. Instead, it turns both into the same simple description: at each moment, which of the twelve notes (C, C♯, D, and so on up to B) are sounding, and how strongly.

This fingerprint deliberately ignores a lot:

Which octave a note is in, so a piano reduction still matches a full orchestra.
How loud it is, so a soft passage matches as well as a loud one.
Which instrument plays it, so a clarinet and a violin playing the same melody look the same.

What’s left is the harmony and melody, the part of the music that stays the same no matter who plays it or how it was recorded. It’s a bit like recognizing a tune someone hums. You don’t need the original instrument to know the song.

Step 5: Line them up

Now auto-sync has two fingerprints: one for how the score should sound, and one for how the recording does sound. The job is to line them up.

The tricky part is that real performers don’t play like a metronome. They speed up, slow down, stretch a phrase, and hold a fermata. So auto-sync doesn’t just slide one timeline along the other. It stretches and squeezes the score’s timeline, a little here and a little there, until every moment matches the recording as closely as possible. Picture pinning one end of a rubber band to the start of the piece and the other end to the finish, then pulling each point in between until it sits over the right spot in the recording.

It does this in two passes. The first is a quick, rough match of the whole piece to find the overall route. The second zooms in along that route and refines every moment. Working this way lets it handle a twenty-minute recording in a browser tab.

Once every bar has a time, the start of each line is simply the time of that line’s first bar.

Step 6: Double-check its own work

Auto-sync doesn’t just hand you an answer. It checks how sure it is about each line.

The main check is simple: it runs the match twice, once from the start of the piece to the end and once from the end back to the start. If both runs agree on where a line begins, it’s very likely right. If they land in different places, or the match there looks weak, the line gets flagged.

That’s what Review flagged in the sync step walks you through. Instead of checking every line, you only check the handful auto-sync wasn’t sure about.

Repeats

If your score has repeat signs, the performers might take them or skip them. Auto-sync tries each possibility, matches every version against the recording, and picks the one that fits best. When two versions fit almost equally well, it doesn’t guess. It asks you whether the repeat was taken.

When it needs a hand

Auto-sync follows the music by its notes and bars, so it struggles with passages that have neither:

Cadenzas printed without bar lines leave nothing to count.
Measure-repeat signs (the slash that means “play the bar again”) have no written notes to match.

Neither is difficult music, but both are blank to auto-sync. Mark passages like these with the Unsure button. Auto-sync then skips that stretch, lines up the music on either side of it, and estimates the times inside it from the score’s tempo. Those estimates look different on the timeline, so you know to give them a quick check.

If the whole recording matches the score poorly, auto-sync tells you that too, and lets you choose between placing its best guess and syncing by hand.

What this means for you

The sync step becomes a quick review. Most lines arrive already placed.
Your changes stick. Any line you move by hand stays exactly where you put it, even if you run auto-sync again.
Better input, better results. A MusicXML file gives the most accurate timing, and a PDF exported from notation software is next best.

Want to see it in action? Open the studio with a score and a recording, and watch the lines fall into place.