You have two streams at mismatched sample rates: high-frequency audio frames (one every 20 ms, hundreds per sentence) and a low-frequency character sequence (a few dozen chars). The hard part: they're different lengths, and nothing tells you which frames map to which char—like joining two distributed streams whose clocks are unsynced, with no join key. CTC's trick resembles stream-processing's debounce + collapse-dedup: each frame emits one symbol (possibly "blank"), then you fold away consecutive repeats and blanks to get clean text.
The hardest part of speech recognition (STT / ASR, automatic speech recognition, turning speech into text) isn't "understanding" but alignment: training data only gives you "this audio = this sentence," without labeling which frames are "hel" and which are "lo." Hand-labeling alignments is prohibitively expensive. The genius of CTC (Connectionist Temporal Classification, Graves et al. 2006) is: it needs no alignment labels—it "integrates out" all possible alignments at once.
How: add a special blank (⊘) to the character set. The network emits a character distribution for every frame (possibly ⊘). A "frame→symbol" path becomes text via a collapse rule: first merge adjacent repeats, then delete all ⊘. So the text "hi" can be produced by countless frame-level paths, e.g. hh⊘i, ⊘hii, h⊘⊘i… CTC's loss is maximizing the summed probability of all paths that collapse to the correct text:
P(text | audio) = Σ_{all paths π that collapse to the text} P(π | audio)
Symbol by symbol: π is one frame-level path; collapse compresses it to text; Σ sums the probabilities of millions of valid paths. Why sum? Because we don't care which frame maps to which char, only that the final text is right—so we marginalize out alignment, the latent variable we don't care about. This sum looks exponential but is computed efficiently by dynamic programming (forward-backward), same lineage as HMMs. Blank also solves two real problems: distinguishing genuinely repeated characters ("hello"'s two l's need a ⊘ between them or they collapse into one) and what to output during silence/pauses.
CTC is one school (frame-independent, non-autoregressive, fast). Another is the encoder-decoder path of Whisper (Radford et al. 2022): the encoder compresses audio into features, the decoder autoregressively generates text char by char like translation, with alignment learned implicitly by attention. A third is wav2vec 2.0 (Baevski et al. 2020)—first self-supervised pretraining (a "fill-in-the-blank" on massive unlabeled audio, masking a stretch of waveform and predicting it contrastively), then fine-tuning on a little labeled data, letting the pretrained representation chew through half the alignment problem first.
# Transcribe once with official Whisper; then see CTC's collapse rule import whisper # pip install openai-whisper model = whisper.load_model("base") # local model, no API key needed result = model.transcribe("audio.mp3", language="en") print(result["text"]) # encoder-decoder autoregressive output # The CTC collapse rule itself is only a few lines — this is its core def ctc_collapse(frames, blank="⊘"): out, prev = [], None for ch in frames: # frames: per-frame argmax symbols if ch != prev and ch != blank: # skip repeats, drop blanks out.append(ch) prev = ch return "".join(out) print(ctc_collapse(list("hh⊘ii"))) # → hi
Modern TTS is like a two-stage compiler: the acoustic model is the front end, compiling text into an intermediate representation (IR)—a mel-spectrogram (a "musical score for sound," time on the x-axis, frequency energy on the y-axis); the vocoder is the back end, "code-generating" that IR into a playable waveform. The benefit is exactly a compiler's: separation of concerns—the front end handles content and prosody, the back end handles audio quality and timbre, each upgradeable independently.
Generating a 16 kHz waveform directly from text means the model must emit sixteen thousand highly-correlated samples per second—a search space too vast to learn. The breakthrough is dimensional relay: first generate the far lower-density mel-spectrogram (a few dozen frames per second, keeping only the frequency energy the ear is sensitive to, discarding phase detail), then have a vocoder fill the waveform back in.
① Acoustic model: text → mel-spectrogram. Tacotron 2 (Shen et al. 2018) is the classic: seq2seq + attention maps a character sequence to spectrogram frames. Here lies TTS's core difficulty—duration: text is short, speech is long, so how many frames does one character get? Where to stress or pause? Autoregressive models solve this implicitly via attention but are prone to skipping/repeating/breaking down; later FastSpeech-family models switched to an explicit duration predictor with parallel generation—more stable, faster.
② Vocoder: mel-spectrogram → waveform. The pioneer WaveNet (van den Oord et al. 2016)—autoregressive, generating sample by sample, stunning quality but too slow to be practical (16k steps for 1 second). Successors used GANs (HiFi-GAN etc.) and diffusion vocoders to parallelize it, speeding up by orders of magnitude without losing quality.
③ New paradigm: codec language model. VALL-E (Wang et al. 2023) treats TTS as "language modeling"—first compress speech into discrete tokens with a neural codec (the star of the next section), then train an LLM to autoregressively predict those audio tokens, and finally decode back to waveform. It blurs the acoustic-model/vocoder split and brings a striking capability: 3 seconds of target audio suffices for zero-shot voice cloning.
# Run a two-stage TTS via HuggingFace's official pipeline (Bark/VITS similar) from transformers import pipeline import soundfile as sf tts = pipeline("text-to-speech", model="microsoft/speecht5_tts") # speaker vector: the "identity fingerprint" of timbre; swap for a cloned voice import torch spk = torch.zeros((1, 512)) # placeholder; use a real speaker embedding out = tts("Hello, this is a speech synthesis demo.", forward_params={"speaker_embeddings": spk}) sf.write("out.wav", out["audio"], out["sampling_rate"]) # Inside: text→mel-spectrogram(acoustic)→waveform(vocoder), both stages wrapped
Quantizing continuous audio into discrete codes is essentially a database's dictionary encoding: rather than storing a long string of high-precision floats, learn a codebook and map each small chunk of sound to an integer ID ("which codeword"). And RVQ (residual vector quantization) is more like staged delta encoding: the first level roughly approximates, the second encodes only "the residual (error) the first didn't capture," the third refines the residual again… approaching layer by layer, precision stacking up.
This section is the key hub for plugging audio into the LLM world. The whole LLM machinery (Transformers, next-token prediction) is built on discrete tokens, but audio is continuous. A neural audio codec is that "audio tokenizer": the encoder compresses the waveform into low-frequency features, the quantizer snaps each continuous feature vector to the nearest codeword in the codebook and emits an integer ID, and the decoder reconstructs the waveform from IDs. The whole chain trains end-to-end, aiming for "reconstructed quality as close to the original as possible."
The catch: a single codebook isn't enough. To cover rich timbres the codebook would need to be huge (hundreds of thousands of entries), unstable to train and wasteful. RVQ's answer is divide and conquer—a cascade of small codebooks:
This way N small codebooks of K entries each yield Kᴺ expressive power while storing only N·K codewords—exponential expressiveness, linear storage. SoundStream (Zeghidour et al. 2021) pioneered this end-to-end RVQ codec; EnCodec (Défossez et al. 2022) pushed it to higher fidelity. Their significance goes beyond compression: audio thereby becomes a "discrete token sequence"—so VALL-E can "write" speech like writing sentences, and music and sound effects can be LLM-generated. This is precisely the bedrock that lets multimodal LLMs "hear and speak."
# Encode a waveform into discrete tokens with official EnCodec, then decode from transformers import EncodecModel, AutoProcessor import torch model = EncodecModel.from_pretrained("facebook/encodec_24khz") proc = AutoProcessor.from_pretrained("facebook/encodec_24khz") wav = torch.randn(1, 24000) # 1s fake audio; load a real waveform instead inp = proc(raw_audio=wav, sampling_rate=24000, return_tensors="pt") enc = model.encode(inp["input_values"], inp["padding_mask"]) codes = enc.audio_codes # discrete codes: integer IDs, shape [layers, frames] print(codes.shape, codes.min().item(), codes.max().item()) # These integer IDs can serve as "audio tokens" fed into a language model rec = model.decode(enc.audio_codes, enc.audio_scales, inp["padding_mask"])[0] # IDs → reconstructed waveform
Offline recognition is like batch processing: wait for the whole audio to arrive, see all context, compute the optimum slowly. Streaming recognition is like stream processing: results come out as data flows in, and you can't wait for the future. This forces a hard constraint—causality: the output at time t can only depend on inputs at t and before. The cost is exactly what you meet in streaming systems: you can't see future context, so results can only be suboptimal—unless you buffer a bit more (add latency) to buy quality.
Live captions, voice assistants, and simultaneous interpretation all require output while speaking, not after the sentence ends. But deep models love "seeing globally": standard attention lets every frame see the whole audio, and bidirectional convolutions read future frames too. To make them streaming, you must perform causality surgery—three mechanisms:
① Causal convolution / causal attention: force each frame to see only the past, not the future (mask out future frames), same lineage as an autoregressive LLM's causal mask. Fully streaming, zero latency, but losing future context drops accuracy.
② Chunk-based: cut audio into small chunks (e.g. 320 ms), allowing full bidirectional view within a chunk (local future) while keeping causality across chunks. This is the compromise knob between latency and quality: bigger chunks give more context and higher quality, but you wait one chunk longer—more latency.
③ Limited look-ahead: let the model peek at a small stretch of future (e.g. 200 ms) before emitting the current result—a little fixed latency to substantially recover the accuracy lost to causality.
Structurally, non-autoregressive models like CTC are naturally suited to streaming (each frame independent, compute as frames arrive), whereas Whisper-style autoregressive decoders need extra design to stream—which is why low-latency settings favor CTC/Transducer families.
# The essence of streaming: a sliding buffer, compute per chunk, past-only import numpy as np class StreamingASR: def __init__(self, chunk=0.32, lookahead=0.2, sr=16000): self.buf = np.zeros(0, dtype=np.float32) # accumulating buffer self.chunk = int(chunk * sr) # samples per chunk self.la = int(lookahead * sr) # look-ahead window (buys quality) def feed(self, audio): # mic streams in continuously self.buf = np.concatenate([self.buf, audio]) while len(self.buf) >= self.chunk + self.la: # take "current chunk + a bit of look-ahead", emit only the chunk's result window = self.buf[: self.chunk + self.la] yield self.decode_causal(window) # causal decode: no revising finalized text self.buf = self.buf[self.chunk:] # slide, drop what's processed # chunk/lookahead are your two "latency↔quality" knobs