What ASR means for Chinese video subtitles

A modern recogniser does not listen for words the way a person does. It converts short slices of audio into acoustic features, predicts which sound units are present, and then searches for the most likely sequence of characters given both the acoustics and a language model. The language model is why output reads fluently even when it is wrong: it prefers combinations that are common in text.

That design produces a characteristic failure mode. The recogniser is good at producing a sentence that sounds like something a person would say, and bad at producing exactly the words this particular person said. So the output reads naturally and is wrong exactly where the content is specific — a brand name, a person’s name, a regional expression, a number.

It also explains why the error rate changes with the recording. Clear speech in a quiet room with one speaker gives the model plenty of signal. Music, overlapping speakers, room echo, or a phone call compress the signal until the model is guessing, and fluent guesses look much more confident than they are.

Mandarin has a small syllable inventory and a large character inventory. Thousands of characters map onto roughly four hundred base syllables, multiplied by tone. A recogniser that hears the syllable correctly still has to choose among characters that sound identical, and that choice depends on context the model may not have.

The practical consequence is that homophone errors cluster in exactly the places where a subtitle is judged: names, titles, technical terms, and set phrases. A viewer reading a subtitle will notice one wrong character in a name immediately, even if the surrounding sentence is perfect, because they are matching the text against what they already know about the subject.

Tone adds a second layer. Mandarin tones are carried partly by pitch contour, and pitch is the first thing damaged by compression, music beds, and cheap microphones. A recogniser working from a compressed audio track is effectively working with the tone information half removed, which is one reason a phone recording produces more character errors than a studio recording of the same sentences.

The start and end times attached to each segment are estimates, usually derived from the same acoustic features that produced the words. They are typically good to a fraction of a second, which is fine for reading and noticeable in music, comedy, and anything where a line has to land on a beat.

Two systematic problems show up often. First, segments tend to start slightly late and end slightly early, because the model is conservative about edges; the result reads as a subtitle that lags. Second, a long unbroken stretch of speech gets split at arbitrary points, so a single spoken sentence becomes two subtitle lines with a break in the middle of a phrase.

The fix is editorial, not decorative: nudge boundaries so that lines begin when the sound begins and end when the phrase ends, and merge or split segments so that each subtitle line corresponds to a unit a reader can take in at once. This is the step people skip, and it is the step that makes output feel professional.

Start with the audio, not the text: if there is music, crosstalk, or a section where the recording drops, mark those minutes first. They will contain the densest errors, and reviewing them once with attention is cheaper than patching the same spots three times.

Then read for names and numbers. Write down every one you correct as you go; a repeated name will be wrong the same way every time, and a search-and-replace pass at the end is faster than catching each occurrence by ear.

Then read the text on its own, without the audio, at reading speed. Errors that survive the audio pass often show up here, because the sentence stops making sense when you are not hearing the intended words. Finally, watch the whole thing once with sound to catch timing problems you cannot see on a transcript.

Recognition gives you text with approximate boundaries. Alignment assigns exact times to units smaller than a segment — a word, a syllable, a character — by matching the known text against the audio again. It is the step that makes word-level highlighting, karaoke-style highlighting, and character-aligned pinyin possible.

Alignment only works when the text is already correct, which is why the order matters. Aligning an unreviewed transcript produces precise times for the wrong characters: the highlight lands on the right moment and the wrong word. In practice this is why tools that promise "perfect alignment" still need you to fix the text first.

Once the text is corrected, alignment is what lets a bilingual layer behave properly. Each Chinese character can carry its own pinyin, and each pinyin syllable can be highlighted when it is sung or spoken, instead of the whole line lighting up at once.

It should be used as-is when the purpose is search, rough navigation, or a first look at a long recording. A draft transcript that is 95 percent right still lets you find the five minutes you need, and the cost of the remaining errors is near zero because nobody is publishing it.

It should not be used as-is for anything a third party will read as a record: published subtitles, legal or medical notes, interview quotes, or classroom material. In those cases the transcript is a claim about what someone said, and an unreviewed claim is a liability rather than a convenience.

The practical middle ground is a review budget. Decide in advance how much time the file deserves and spend it on the parts that carry meaning: names, numbers, the sentences that will be quoted, and the first and last minute where a viewer decides whether to keep watching.

Recognition runs on the audio of the video you upload, producing timed Chinese segments. The review step is in the editor: you edit a line and the timing stays attached, so corrections are local rather than a rebuild of the file. Character-aligned pinyin is generated from the corrected text, not from the raw output, so a fixed character fixes its pinyin too.

We are explicit that the automatic pass is a draft. The product exists to make the correction step fast and local — jump to a segment, hear it, fix it, move on — because that step is where the difference between a usable subtitle file and a plausible-looking wrong one is decided.