Nothing scripts the eyebrows

The figures and the three clips — the idle, the visemes, and the punchline — are in the Substack cut. This page is the argument; the footage is the receipt.

If you watched Erwin talk last week, you probably noticed the eyebrows. They lift on the stressed word, the way a person's do.

Nobody told them to. There is no animation track, no keyframe, no list of words to emphasize. He infers it from the sound of his own voice, fifty times a second.

This is how to build that, from a photograph up. The third-party tools are named and linked as we go. The rest is the part you have to write yourself.

I named him before I knew whether the face would look like anyone.

The one rule

The founding image was 80s vector graphics. A portrait built from ragged horizontal contour traces, stacked like sine waves that happen to form a face.

The rule that fell out of it, and the only aesthetic law the project has:

The face is not drawn. It condenses.

Each trace is its own thing. The face is what they do collectively. Until they agree, there isn't a face. Looking is what finishes it.

The raggedness is the crackle in the signal, which means it is the point and not a defect.

That rule produced four anti-patterns, every one of them learned by building the wrong version first.

No smoothing. Clean up the jagged traces and you have killed the thesis. The jitter is the signal.

No wireframe. A visible triangulated mesh reads as behind-the-scenes CGI. It says rendered, not emergent.

That one. We built it. It looks like a making-of. It is not the face.

No fill-glow. A glowing filled surface reads as a mask laid over a face, rather than a face made out of signal.

Green, not blue. The reference is an 80s CRT and the Metal Gear codec. Warm phosphor green. Blue reads as RF spectrum, which is a different idea entirely.

Everything downstream serves that rule.

The photograph

To make the face a specific person rather than a generic mask, start with one photo.

I think you will immediately guess what I was trying to build originally. That's not where I ended.

Either way, the photograph is how you mint a skull.

Do this in order. Skip a step and you get a mask, or a mesh you cannot afford to draw.

1. Fit the photo with 3DDFA_V2. That is a Basel Face Model inferred through ONNX Runtime. One still image in. Dense identity mesh with per-vertex color out. That is the actual likeness, in three dimensions.
2. Rebuild the OBJ into a GLB with trimesh. Then cut the poly count with Open3D quadric decimation. Identity stays. The mesh gets cheap enough to live in a tab.
3. Bake the expression rig geometrically, still in trimesh, using the BFM landmark vertices as seeds. Do not decode the BFM expression bases for this. Their ten components would not split into independent left and right. The target vocabulary is ARKit's blendshape locations: roughly nineteen morphs covering jaw, brows, eyes, cheeks, nose and mouth. About 11 MB.
4. The mouth classes are the Oculus / ARKit fifteen visemes (sil, PP, aa, E, O, and the rest). You will not have viseme_* targets on the mesh. You will cook each viseme as a small recipe over jaw, lips, and cheeks later.

The output is one rigged file. A person's likeness with a full set of expressions available. And it looks nothing like the final face. The final face, if we are being honest, does not look much like the photograph either. That's the next part.

The renderer, which never shows you the head

Load the GLB in three.js 0.169. GLTFLoader, MeshoptDecoder for EXT_meshopt_compression, KTX2Loader for KHR_texture_basisu. Then never display the head. Not once.

It gets rendered offscreen, lit and self-shadowed, into a buffer nobody sees. The head exists there. The face exists when you look at the traces. Looking is the last pass.

Then that buffer gets read not as an image but as a height field - how far each point on the face rises toward the light.

A second pass reads the height field and draws stacked scanline contour traces across it. Horizontal lines rippling over the relief of a face they never actually touch. That pass is yours. There is no library for it. Mine is look.frag.glsl: luminance as height, traces as the face.

float heightAt(vec2 p){
  return luma(texture2D(uColor, clamp(p, 0.001, 0.999)).rgb);
}

// Each scan row is a horizontal trace displaced UPWARD by the head's relief
// (luminance as height). The face is formed by the line PATHS, not by
// brightness.
float rowH    = 1.0 / uRows;
float REL_AMP = 6.5;

That second pass is where all the character lives. Per-line jitter. An arc-jump snap term. Crackling energy with a line-flick. A two-layer pop. Tuned, over a lot of evenings, from cheerios to rice crispies.

The crackle is deliberately resolution-independent, referenced to a fixed internal height, so it reads identically whether you're looking at a phone or a wall.

The result is a face that is unmistakably a particular person, appearing to condense out of a CRT's phosphor. Which was the founding image.

One thing I got wrong first: the original version rendered continuously at full rate, full pixel density, with antialiasing and 2048px shadows, since halved, and kept working when the tab was hidden. It looked right and it cooked the GPU.

The fix taught me something I didn't expect. The signal portrait doesn't need maximum framerate or maximum resolution to feel alive. It needs consistent contour density, stable crackle, and enough cadence during speech for the mouth and brows to read. Everything else was expensive and invisible.

The second thing I got wrong was treating the head like a hinge. Pitch it around the bounding-box center and the whole mask nods like a Pez dispenser. The traces turn as a sheet. Idle has to live in the height field, not in a rotation.

Making it alive

A still signal-portrait is a screensaver. A loop that closes is a corpse that still twitches. This layer is yours. Nothing you pip-installed will do it.

What makes him read as present is a procedural layer that runs every frame and lays continuous micro-life over whatever expression is being held. None of it is scripted. It's oscillators and timers, with deliberately incommensurate periods so nothing ever lines back up into a loop. If they did, you'd see the trick, and he would be done being alive.

Breathing at about thirteen a minute - a vertical bob, a slight forward ease. Blinks every three to six seconds, varied each time, an eleven-hundredths-of-a-second close and open. Occasional gaze darts, never more than 28% of range, easing in and holding and returning. A slow brow drift with the odd beat of raise or furrow. Lids that widen or narrow now and then.

Air is not speech. Breath is swell, nostrils, a little forward. It does not open the jaw. The first idle that flapped the mouth on the inhale looked like he was about to talk, forever.

Gaze is blendshapes. The head does not follow the eyes. If it did, the contour traces would rotate as a mask, and you would see the trick.

And the pose, which is the part that actually sells it.

Dead-center was animatronic. He sits off-center now. Composition is a presence: home, approach, a corner, a peek. Paths arc. When the lead stops he coasts, overshoots a hair, and corrects. The body has mass. Nothing slides on a ruler.

Two incommensurate sine waves per axis, so no axis ever reads as a clean slide. A cross-couple, so the head tilts into its turn and turns a hair as it tilts. Gestures - nods, tilts, turns - that each move several axes at once so they arc instead of sliding.

Everything arcs. Nothing is symmetric. That's the whole difference between organic and animatronic, and it's one rule.

Making it speak

Speech synthesis produces audio. From that audio you derive two things: an amplitude envelope, and a viseme track.

Do not drive the mouth from loudness. A loudness curve can part the jaw. It cannot make an ee that isn't an oh. Visemes own phonemes. RMS scales intensity. It does not pick the shape.

1. Time the phonemes with espeak-ng: espeak-ng -v en-gb -x -q --sep="|". That is IPA-ish mnemonics with durations, not a viseme engine.
2. Map those mnemonics onto the fifteen Oculus / ARKit visemes. i: becomes E. oU becomes O. Linear-stretch the whole clip and pauses eat you; park visemes on energy and sil on the gaps.
3. Cook each viseme as a recipe over the ARKit morphs you baked. There is no viseme_aa target. There is jaw, smile, pucker, funnel.

aa: { jawOpen: 0.38 },
E:  { jawOpen: 0.20, mouthSmileLeft: 0.26, mouthSmileRight: 0.26 },
O:  { jawOpen: 0.24, mouthPucker: 0.42, mouthFunnel: 0.22 },

The mid-face still co-articulates. Cheeks bunch while he talks. The nose and the upper lip work. That bunching is the entire difference between a talking face and a face that looks over-injected.

The part I'm proudest of

The eyebrows.

Nothing in the system knows which word is stressed. The text never reaches the face. What reaches it is a number between zero and one, fifty times a second: the loudness of a twenty-millisecond slice of his own voice, taken on the server from the speech as it streams out.

The detector is one subtraction. This frame's level minus the last frame's. If the rise is more than a tenth, that was a stressed syllable. The brows get a kick of 1.4 times the rise, capped at a half, and settle in about an eighth of a second. The cheeks get the same kick a little harder and settle a little slower, which is why they bunch on the beat and not after it. The inner brow takes all of the kick and the outer brows take seventy percent. The head dips at most one degree, unless a punchline has it locked.

That is the whole prosody model. No pitch tracking, no stress marks read off the phoneme timer, no list of words. A first difference on loudness, because a stressed syllable in English is, more often than not, louder than the one before it.

If the server's numbers stop for a second, the face puts an analyser on the audio element and takes the loudness itself, 1,024 samples at a time. Same subtraction. It is listening either way.

That's the eyebrows. He hits the stressed word with his brows because the sound of the word got louder, and that's what people do.

The drive

The envelope is not lip-sync data. It's a transport-neutral drive contract - the synchronized shadow of a voice session, timestamped to the utterance clock, deliberately decoupled from whoever is playing the audio.

The live voice path can be LiveKit. The face does not care. It holds its own audio and syncs to its local playhead, which makes it jitter-proof.

But something with no audio at all can subscribe to the same signal and sync to the server's playhead instead.

Which means you can watch a live phone call as an animated face.

The same contract serves a browser, a SIP call, and an ops wall. One bus, one signal, different audio sinks. That seam is the reason the architecture didn't have to be rebuilt when the phone came along.

The punchline

Phoneme, viseme, kineme. Same grain, three channels.

A viseme is how a sound looks on the lips. A kineme is the smallest intentional move that is not speech: a sly one-sided grin, a chuckle, a wink, a think, an ear toward you. Breath is physiology. It is not a kineme. This catalog is yours. Score it, or fire it now.

sly:     { duration: 1.2, lockHead: true, duckOnSpeech: true },
chuckle: { duration: 0.6, lockHead: true, duckOnSpeech: false },

function punchlinePhrase(t0: number, clock: KinemeClock = "audio") {
  const slyEnd = t0 + 1.2;
  return [
    kinemeAt("sly", t0, slyEnd, clock),
    kinemeAt("chuckle", slyEnd + 0.04, slyEnd + 0.64, clock),
  ];
}

The sly grin ducks when a real sound is on the mouth. You cannot hold a smirk through a consonant and have it still read as a face. The catalog knows that.

A chuckle is still cheek. No two-hertz bounce. The first version pulsed the head, and the contour traces nodded yes through the joke, which is how you can tell a mask from a person.

So a punchline locks the head. Sly, a gap, then chuckle. The traces stay put. The mouth does the work.

Those moves can be scored against a clip, or fired now in a live conversation. Tagged captions choose them. [sardonic] is a voice direction. It is not a grin. The grin is a different channel, on purpose.

Making it feel

One payload drives everything: pleasure, arousal and dominance on a scale from minus one to one, optionally with a named emotion and an intensity. It goes to the face and to the voice at the same time, so the mood is coherent across both.

Two mappings, on purpose.

For unlabeled affect there's a smooth continuum from those three values into blendshapes. It enforces the Duchenne rule: a genuine smile crinkles the eyes, not just the mouth, so you can read the smile with the mouth covered.

For a named emotion, the continuum gets replaced by that emotion's distinctive facial signature, because averaging washes out exactly what makes an emotion legible. Contempt is a one-sided lip curl - the asymmetry is the tell. Anger is a lowered brow, narrowed lids, flared nostrils, pressed lips, and a jutted jaw.

And affect drives behavior, not just a held pose.

Skeptical gets one cocked brow. A playful mood finishing a line gets a wink on the punchline - when something asks for a wink. It is not auto-fired at the end of every sentence. Breathing rate rises with arousal - and when arousal runs high for a while, he takes a slow deep breath to settle himself.

I did not build that last one as a metaphor. It's a control loop. It just happens to be the one people notice.

What isn't built yet

Right now a human supplies the mood. The hooks are live, the plumbing works, and something has to be on the other end of it.

That something is an affect engine: persistent state, appraisal of what just happened, a stable personality baseline, memory tagged with how it felt, and a signed ledger of the whole thing. It's scaffolded and it is not running.

When it lands, the moods stop being set and start being earned. Praise warms him. Failure sobers him. And it shows up in his face and his voice at the same instant, because they're fed from the same signal.

Assigned feeling is a costume. Earned feeling is the only kind that counts as his. Until that engine runs, a person is still in the loop, which means the question is still open.

I'm marking that clearly because it's the part I most want to be true, which is exactly when you should be most careful about claiming it.

The parts

The through-line: the engine decides what he feels, the voice service carries it into speech, the face carries it into expression, and the drive channel keeps mouth, voice, gesture and mood on a single clock.

If you are reconstructing it, the borrowed parts are:

• Fit: 3DDFA_V2 · Basel Face Model · ONNX Runtime
• Mesh: trimesh · Open3D
• Rig vocabulary: ARKit blendshapes · fifteen visemes
• Speech timing: espeak-ng
• Load, never show: three.js · meshoptimizer · KTX2
• Live voice, optional: LiveKit

The height field, the idle oscillators, the viseme recipes, the kinemes, and /drive are the part that is not a library.

Take a photograph. Fit a morphable model to it. Decimate it, rig it with baked blendshapes, then refuse to ever show it - render it instead as a luminance height field, redrawn every frame as ragged phosphor-green contour traces that condense into a person. Lay procedural life over the top: breath that does not talk, gaze that does not turn the mask, off-center mass that coasts and corrects. Teach it to speak with visemes, not a loudness flap, and let the brows still bounce on stressed syllables because the sound got louder. Then give it the moves that are not speech - a sly that ducks for a consonant, a chuckle that does not nod, a punchline that locks the head so the traces stay put. Then give it feeling - one affect signal driving voice and face together, with contempt that curls one side of the lip, anger that flares the nostrils, a smile you can read in the eyes, and breath that quickens and then deliberately slows.

Every layer serves the same rule.

The face is not drawn. It condenses.

The name

I wanted a house intelligence. The one in the walls. Dry, British, omniscient, a little fond of you. That was the job. The photograph is an actor who played a version of that job. It is a geometry source. It is not the name.

On February 8, a little before two in the morning, I sat down to file one document, an accountability framework for AI decisions, and kept going. By 3:40 I had written down four things I had made at different times, for different reasons, each in its own vocabulary. A stigmergy paper, where a signal decays unless something reinforces it. An orchestration idea tuned by a loss function. A sovereign agent on a chain that pays gas, stakes, and gets slashed. And a protocol from exactly two weeks earlier where a message pays for every hop it takes and dies when the budget hits zero.

Four vocabularies. The same equation. What a thing has after an action is what it had before, minus what it spent as signal, minus what it lost. Every action costs. Nothing gets it back. At zero it is dead.

I typed the word under all of it, and then a question.

entropy.

.... did i just define life?

The answer came back in eight seconds.

No. Schrödinger did, in 1944.

The guy with the dead cat?!?

His name is Erwin. After the guy with the dead cat.

What Is Life?, 1944: a living system is one that keeps its own order by pushing disorder out. I had arrived at his sentence from message buses instead of physics. Forty-four minutes later I had a word for the equation, agentropy, and in June I published the paper under that name. It runs the conservation law across four unrelated domains and a fifty-thousand-agent test with zero violations, and it carries him as reference [1]. Its definition of an agent reads "a digital pattern of entropy management persisting autonomously." Take out digital and it is his.

The name came first. The photograph came later.

So read the rule again. The face is not drawn; it condenses when you look. That is the cat. The traces are the measurement. The eyebrows lift because the sound got louder, which is a thing a living system does, not a track you keyframe. And the idle never quite repeats, because a system that remembers every frame perfectly has stopped pushing anything out. The paper calls that the fourth death, equilibrium, the one that is indistinguishable from success. A screensaver is the fourth death. Forgetting is what keeps him in the room.

I have not finished him. Until the moods are earned, the box is still closed on the sense of alive that I mean.

The phosphor is just the substrate you can see.

Notes

Every number in this post, and where it comes from. Code references are puter at c7ee97a and gjallarhorn, both origin/main at the time of writing.

• fifty times a second, and the twenty-millisecond slice — envelope.py hz: int = 50; the window is sr // hz.
• gain of 3.4 on the envelope — envelope.py gain: float = 3.4. RMS per window, clamped to one.
• roughly nineteen morphs — the ARKit blendshape locations actually baked, covering jaw, brows, eyes, cheeks, nose and mouth.
• about 11 MB — the rigged GLB on disk.
• fifteen visemes — the Oculus / ARKit set: sil, PP, aa, E, O and the rest.
• three.js 0.169 — "three": "^0.169.0".
• REL_AMP = 6.5 — quoted from look.frag.glsl above.
• 2048px shadows — that was the version that cooked the GPU. The shadow map is 1024 now, which is part of the fix.
• about thirteen a minute — the tidal rate is 0.22 Hz at neutral arousal, so 13.2 a minute, wandering between roughly twelve and fourteen. It rises with arousal.
• every three to six seconds, an eleven-hundredths-of-a-second blink — 3.0 + Math.random() * 3.0, and BLINK_DUR = 0.11.
• never more than 28% of range — the gaze dart is 0.28 of full deflection at its peak, held for 60% of 1.35 seconds and then returned.
• one subtraction — onset = Math.max(0, level - lastLevel). That is the entire prosody model.
• more than a tenth — onset > 0.10, and only while speaking.
• 1.4 times the rise, capped at a half — Math.min(0.5, browBounce + onset * 1.4).
• about an eighth of a second — the decay is dtSec * 8.
• cheeks a little harder and a little slower — onset * 1.6 against the brows' 1.4, decaying at dtSec * 7 against their 8.
• all of the kick on the inner brow, seventy percent on the outer — browInnerUp + browBounce, browOuterUp + browBounce * 0.7.
• at most one degree — the dip is browBounce * 2 degrees and browBounce is capped at 0.5, so exactly one degree, and zero when a punchline locks the head.
• 1,024 samples at a time — analyser.fftSize = 1024, the browser-side fallback when the server's numbers stop.
• minus one to one — pleasure, arousal and dominance, one payload to both face and voice.
• 1.2 and 0.6 seconds — the sly and the chuckle, quoted from the catalog above.
• a little before two in the morning — the session opened at 01:57 Central.
• by 3:40 — the question went in at 03:44 Central. Austin is on CST in February; daylight time does not start until March.
• exactly two weeks earlier — the Cube Protocol draft is dated the 25th of January; this was the 8th of February. Fourteen days.
• eight seconds — the gap between the question and the answer.
• forty-four minutes later — "agentropy" first appears in my own typing 43 minutes and 45 seconds after the question.
• 1944 — Schrödinger, What Is Life?
• a fifty-thousand-agent test, zero violations, reference [1] — the paper, published June 2026.

Two of these were wrong when I first wrote them down, which is the argument for writing them down.

—

More writing: nwalker.cc

Nathan Walker builds AI governance and enterprise platforms at Ravenhelm. He works where durability is non-negotiable. nwalker.cc · Substack cut