Weave

Internals

Figure 3. One word, from the air to its color turn. The turn lands at 1.75 s, while the speaker is still provisional and nearly three seconds before the durable decision.

The contract

Field	Holds
`words[]` — what was said	`text` · `start` · `end` — the word, and the moment it begins and ends
`words[]` — who said it	the speaker, and whether that is settled or still a guess
`words[]` — how it was said	loudness and pitch, each placed within that speaker’s own range
`speakers`	S1 … S12, each given a color from the palette below

Figure 4. What one word carries, from schema.py. start and end are the pair everything else is keyed to: prosody, the motion clock, the read-ahead floor and the haptic cue all read them, which is why a recognizer that cannot supply them is no use here.

The palette, by order of first appearance
S1	#E5E517	S4	#E517E5
S2	#17E517	S5	#E51717
S3	#17E5E5	S6	#E58017
S7+	twelve supporting hues	—	unknown

The event stream

Eight kinds of event travel the stream; these are the four a renderer acts on. A corrected word comes back under the same id, so it replaces the old one rather than adding to it.

Event	Carries
`word`	text, span, speaker, loudness, pitch, and the salience flags the haptics read
`verification`	a finished phrase, re-recognized and corrected word by word
`sound`	a non-speech sound — laughter, applause, music, the room itself
`level`	input level, noise floor, gain, and the bearing when an array supplies one

Rule	Figure	Explanation
The playhead trails the acoustic clock	1.75 s	The color turn runs 1.75 s behind the voice, so recognized words are on screen well before they are colored.
Read-ahead is a per-word floor	420 ms	Words arrive in bursts, so a fixed delay guarantees nothing. Every word gets at least 420 ms on screen before its color turns.
The turn is never gated on the speaker	—	Text arrives long before a durable speaker decision, so the turn happens on time in whatever color is known.
Gain is applied to the recognizer's copy only	—	Loudness is measured from the original signal. Amplify it first and a whisper and a shout end up the same size.

Motion

asas eacheach wordword isis spokenspoken

$$\Delta t_i \;=\; \frac{i}{n-1}\cdot 80\ \mathrm{ms}$$

The 80 ms is split evenly across the letters, so the color sweeps from the first to the last at a constant rate. The last letter turns exactly 80 ms after the first, however long the word is.

$i$: the letter's position in the word, counted from zero
$n$: how many letters the word has

Color turn. Read-ahead ink until the spoken onset, then wiped across the letters rather than switched.

mostmost wordswords onlyonly liftlift THISTHIS popspops

$$s_{\mathrm{pop}} = 1 + 0.076\,\bigl(0.5 + 0.5\,e\bigr),\qquad e = \frac{s_{\mathrm{crest}}-1}{0.425}$$

Half the pop is given to every word outright and the other half is earned by how much that word swelled. A word with no growth still moves 1.038×, and the most emphatic one reaches the full 1.076×.

$s_{\mathrm{pop}}$: the size the word reaches at the top of its pop
$e$: how far this word's crest sits up the reachable range, 0 at no growth and 1 at the largest the system allows

Pop and lift. One animation. Every word rises 0.16 em and every word grows, in proportion to its crest.

normalnormal oror louderlouder

$$s_{\mathrm{crest}} = 1 + \mathrm{PWL}(L),\qquad L = \operatorname{clip}\!\left(\frac{db - lo}{hi - lo} + E,\; 0,\; 1\right)$$$$E = 0.75\,\tau + 0.25\,\bar{\tau} + 20\log_{10}\frac{F_0}{\tilde{F_0}}$$

The fraction places this word’s level between that speaker’s quiet and loud ends — 0 at one, 1 at the other — with emphasis added on top and clip holding the result inside them. PWL turns that into how much to grow: nothing at all below 0.21, and at most 0.425 above it.

$s_{\mathrm{crest}}$: the size the word swells to: 1.000 at no growth, 1.425 at the largest the system allows
$L$: loudness: where this word sits in that speaker's recent range, 0 to 1
$db$: the level of this word's own span, taken as the 90th percentile of its frame RMS in dB
$lo,\ hi$: the 10th and 95th percentile of that speaker's recent levels; the two are held at least 18 dB apart so a monotone passage is not stretched
$E$: emphasis, added on top of the level; only a score above 0.72 is added at all
$\tau$: spectral tilt: how hard the voice is pressed. It rises when someone speaks with force, even without getting louder
$\bar{\tau}$: the mean tilt of recent words, which keeps one loud word from setting the baseline by itself
$F_0$: the fundamental frequency of this word, that is its pitch
$\tilde{F_0}$: the median pitch that speaker has been producing lately
$\mathrm{PWL}$: a piecewise-linear curve fitted to the reference film, through six control points, topping out at 0.425
$\operatorname{clip}$: clips the value to the range that follows it; anything outside becomes the nearest end

Crest. A real font-size, not a transform — and the row does not move. Upward only, to 1.425×.

aa highhigh voicevoice aa lowlow oneone

$$w = \operatorname{clip}\bigl(400 + r'\,\Delta_w + p,\; 200,\; 900\bigr)$$$$r = r_v + k\,(r_w - r_v)$$$$p = 460\,\operatorname{clip}\!\left(\frac{L - 0.222}{1 - 0.222},\; 0,\; 1\right)$$$$r' = \begin{cases} r, & r \ge 0 \\ r\,\bigl(1 - p/460\bigr), & r < 0\end{cases}$$

Weight starts at the neutral 400 and is moved by two pushes that add: where the pitch sits, and how loud this word is right now. Pitch alone cannot tell a habitually high voice from one that is shouting, which is why it is split in two.

$w$: the rendered font weight, between 200 and 900
$r$: where this word’s pitch sits, −1…1: +1 at 80 Hz and below, 0 right across the neutral 175–190 Hz band, −1 at 250 Hz and above
$r_v$: the same mapping applied to the speaker’s running median pitch — the voice rather than the word
$r_w$: …and applied to this one word’s own pitch
$k$: how far the word is allowed to leave its speaker, 0.70. At 0 one talker is one weight however they modulate; at 1 the speaker stops showing at all
$r'$: r reduced: the pull toward Light weakens as the word gets louder, and vanishes entirely at full p. A pressed voice is not an airy one
$\Delta_w$: how far there is to go from 400 to the end: 500 going heavier, 200 going lighter. The light half is the shorter one, which is why it is what limits how much of this channel you can see
$p$: this word’s own loudness, counted upward from ordinary speaking level only: 0 at or below it, adding at most 460 of weight at the loudest. It never subtracts
$L$: the same 0-to-1 loudness the crest reads; 0.222 is where ordinary speaking level sits on it
$\operatorname{clip}$: clips the value to the range that follows it; anything outside becomes the nearest end

Weight. The speaker’s habitual pitch sets where a word starts — the lower the voice, the heavier — this word’s own pitch moves it from there, and a loud or pressed word climbs further.

thinthin andand reedyreedy fullfull andand warmwarm

$$W = 100 + \operatorname{clip}\bigl(0.7\,t + 0.3\,h,\; -1,\; 1\bigr)\,\Delta_W,\qquad h = \operatorname{clip}\bigl(2(0.5 - \theta),\, -1,\, 1\bigr)$$

Seven tenths of the width comes from the word’s own pitch and three tenths from breathiness: the lower the pitch the wider, the breathier the voice the narrower. It never leaves 82 to 124 percent.

$W$: the rendered font width, in percent, between 82 and 124
$t$: the pitch of this word itself, not the speaker’s median: +1 when low, −1 when high
$\theta$: how much air escapes with the voice. Whispering, or speaking on a sigh, leaves the folds open enough that the sound drifts from a steady vibration; that drift is seven tenths of the reading and how much high-frequency energy there is the other three
$h$: breathiness on a −1 to 1 scale: the more air in the voice, the closer to −1
$\Delta_W$: how far there is to go from the neutral 100% to the end of the range

Width. Driven by the word’s own pitch, with a breathiness reading blended in. The quietest of the eight channels.

asas eacheach wordword isis spokenspoken

$$\ell = 0.525\ \mathrm{em}$$

A constant. The only decision this channel makes is whether it applies at all, and that depends on silence sitting on both sides of the word.

$\ell$: how far the word leaves the line, in em, when there is silence on both sides of it

Hold lift. Silence either side lifts a word, and it lands on its turn. Isolation, not sustain.

animationanimation

$$\Delta t_g \;=\; \frac{g}{G-1}\cdot 80\ \mathrm{ms}$$

The same 80 ms as the color turn, split across syllable groups instead of letters. A two-syllable word therefore takes two steps where a six-letter word would take six.

$g$: the syllable group's position in the word, counted from zero
$G$: how many syllable groups the word splits into

Character wave. One syllable at a time — se+en, but+ton. Never one letter.

soso wewe stoppedstopped andand lookedlooked hehe nevernever saidsaid anythinganything

All of it, on one line. Every channel together, on the studio’s own clocks — a swelling word overlaps its neighbors rather than shoving them along.

LiveLive captionscaptions thatthat showshow who,who, whenwhen andand how.how.

What it is

The pipeline

Internals

Constraints

Motion

The studio