The front window of our house watches the street this Halloween. Three eyes, one to a pane, rear-projected onto the glass by a laser projector on the living room floor. They sleep when the walkway is empty. When a trick-or-treater starts up the path, all three snap open, and each one follows its visitor to the door. Once the porch empties, they close one at a time and go back to sleep.

The division of labor was clean. I contributed the imagination, the picture of a person standing on that walkway while something huge watches them. My AI assistant contributed the engineering: the pipeline, the tracking, the tests, the discipline of finishing a real-time system in three weeks. There is an earlier essay about teaching this assistant to reach and to remember. This is what that investment looks like when the deliverable is a show.

The Idea and the Deadline

The first plan, dated September 24, was a different show: a projected skeleton that mirrored your movements as you walked up. Two weeks later I shelved it. A skeleton needs a whole body to read from the street, and a body needs a rendering pipeline there was no time to make good. An eye only has to look at you, and everyone on the walkway knows what that means.

The deadline did most of the architectural work. Show night is Saturday, October 31, dark by six. Silence is part of the design, and so is a dark porch. Dark to us, that is: an infrared flood under the eaves lights the walkway for the camera alone. The cast is a fisheye camera already hanging above the door, a Raspberry Pi with a Hailo AI accelerator, a spare MacBook Air, and a projector I had to buy.

The Shape of the System

There are five moving parts, and each one talks to the next in the simplest way that could work.

Diagram of the show: sees Pi, show Mac, Chrome, projector, window, notebook dashboard, joined by labeled arrows.

The components and the communication paths between them.

The sees Pi, that camera Raspberry Pi, owns the seeing. It runs a YOLOv8s pose model about 20 times a second. A plain HTTP request gets back a small JSON snapshot: each person as a bounding box plus 17 body keypoints (shoulders, ankles, face). Keypoints, not video, cross the Wi-Fi. Video would be megabits; a snapshot with one person on the walkway is a few hundred bytes.

The Mac polls that snapshot 40 times a second, twice the camera’s rate, on a fixed grid timed from the start of each poll. A slow wireless round trip never stretches the interval. Dumber than a stream, which is what a wireless link in the dark wants. When the Pi stops answering, the bridge backs off exponentially and heals on the next good frame.

One thread on the Mac owns everything that thinks. Thirty times a second it drains the polled frames through a perception layer, a tracker, and a mind, then publishes one snapshot of the whole show. Every eye’s mode, gaze target, mood levels, and any one-shot action in flight ride in that snapshot. It is complete rather than incremental, so a renderer that reconnects needs nothing else, and a slow client gets skipped, never waited on.

The renderer is Chrome in kiosk mode, pointed at the projector: a plain canvas page drawing at animation-frame speed. Measured on site it held 120 frames per second with a 99th percentile frame time of 9.4 milliseconds. My assistant ported the motion model from Adafruit’s M4_Eyes, the animated eyes for their Monster M4sk board (Burgess, 2019). Saccades, the quick jumps a real eye makes, travel 83 to 166 milliseconds. Blinks close in 36 to 72. Nobody present means a wandering gaze. We kept Phil Burgess’s timing numbers; they were tuned by someone who wanted exactly what I want: an eye that looks alive.

The wire between the mind and the page is Server-Sent Events, the browser’s built-in push channel: one message per publish. Here is a real one, from a synthetic visitor walked through the actual mind code, abridged to the left eye:

{
  "v": 1, "seq": 241, "t_show": 1012.05, "blackout": false,
  "eyes": [
    {
      "id": "left", "mode": "track",
      "target": { "yaw": 17.042, "pitch": -6.0,
                  "vyaw": 0.352, "vpitch": 0.0 },
      "mood": { "open": 0.2212, "scowl": 0, "squint": 0, "smug": 0,
                "pupil": 1.0, "red": 0.25, "tremble": 0,
                "blink_rate": 1.0 },
      "action": { "kind": "surprise", "id": 1,
                  "t_start": 1012.05, "delay_ms": 98,
                  "dur_ms": 1291 }
    }
  ]
}

The middle and right eyes carry the same action with their own ids and delays of 80 and 9 milliseconds. All three open on the same tick without moving in lockstep. Angles are degrees in the unmirrored view from outside the window, positive yaw to the viewer’s right. The whole snapshot is 899 bytes with all three eyes aboard. Liveness is judged by that sequence number advancing, not by the connection staying open, so a stalled mind settles the eyes to sleep like a dropped stream does.

Where You Are Is the Hard Part

Drawing the eye was never the hard problem. The hard problem is knowing where a person is when your only witness is a fisheye lens eight feet up. Image coordinates mean nothing to an eye that has to look at a real person on a real sidewalk.

So we fitted the ground instead of assuming it. We taped the walkway at known positions, photographed the marks through the fisheye, and fit a mapping from image to floor that recovers all eight marks to within 2.7 inches. Every detection’s feet go through that map, then into a door distance and a radial speed (a least-squares slope over the last 0.8 seconds of track), and those are the numbers the mind reads.

One measured detail earned its keep. A person’s ground position can come from confident ankle keypoints or from the bottom of the bounding box, and the two estimates flip back and forth when ankle confidence sits at the threshold. On synthetic flicker that oscillation was worth 8.5 degrees of gaze jitter near the door. The fix is a learned offset that follows the confident ankles, which removes the jump at its cause instead of smoothing over the symptom.

What the Mind Decides

Flowchart of the mind: asleep, stir, surprise, watch, the door visit, a wink or smug look, settle.

What the mind does with one visitor, from the walkway to the door and back.

The mind is a small set of rules about people, most of them there to keep the show from being wrong in front of an audience. A person counts only after three sightings within half a second, so a single flickering detection never wakes anything. A confirmed person moves through phases (arriving, at the door, leaving) judged by radial speed rather than position, so standing on the doormat reads as standing still. Standing within 1.2 meters of the door for a second marks the visit as done.

The mind assigns attention so the gazes never cross: the visitors who matter most are matched to the panes left to right, an eye keeps its person for at least a second and a half, and the nearest visitor takes any spare eye.

The gaze itself is geometry. Yaw is the real angle from a pane to its person, scaled by a gain. Its default is one half: zero is the Mona Lisa, one is a real eyeball. Pitch is a fixed slight downward look. A walking visitor’s gaze leads them by their own speed, which covers part of the pipeline’s latency. The gain is a live dashboard control, tuned on the real walk.

Then the repertoire. Asleep is the default. Idle life is a stir every 10 to 20 seconds after 10 seconds of quiet: one or two eyes crack open and look around, never all three, because a porch that is already awake cannot surprise the next person who walks up. The first visitor after ten quiet seconds gets a Surprise, and the gaze snaps to them. Watch follows, and the scowl builds with door distance.

Three bloodshot eyes with slit pupils, all open, looking down and to the left.

Watch: every eye tracks its visitor.

The same three eyes, heavy-lidded and squinting.

The full grouch, for standing too close to the door.

A second visitor gets a double take. A leaving visitor gets a wink. A turn-back gets a smug look. A wave from a visitor within 2.5 meters of the door calms the whole show: all three drain shut while still watching, hold a while, and wake when another group arrives. That behavior shipped behind a dashboard switch, off by default, because a gesture detector that has never seen a dark Halloween crowd has not earned the right to fire on its own.

The three eyes flung wide open under raised brows.

Surprise: the first visitor after ten quiet seconds.

The Projector and the Mirror

The projector is a ViewSonic LS740HD: 1080p DLP, laser phosphor, instant on and off (ViewSonic, n.d.). Instant on matters more than any other spec, because the show stops and starts constantly during tuning. ProjectorCentral measured the brightness at 4,783 ANSI lumens (Kahl, 2024), about eight times more light than a 1.73-meter image through window glass needs; against the neighbor’s porch lights it still keeps its contrast. Input lag is 49.5 milliseconds in every mode (Kahl, 2024), on top of a pipeline that runs about 220 milliseconds end to end, and nothing in the show moves fast enough to miss it.

The deepest bug of the project was a flip. Rear projection needs exactly one horizontal mirror of the picture: either the projector flips it or the page does, never both and never neither. Our dry run, one week before the show, ran with no flip at all, and a gaze flip remembered from an earlier test made it stranger: the eyes tracked the wrong way. The launcher now states the gaze flip explicitly on every start, the page owns the whole-picture flip, and the saved alignment records which flip it was aligned under. Software cannot read the projector’s menu. The only real test is to walk the walkway yourself and watch what the eyes do.

Testing a Show, Not a Demo

Nothing here can be validated by watching it work once. The failure modes are crowds, darkness, a camera that wedges, a Pi that reboots, a browser tab that throttles. Every recorded walk on the real walkway replays through the real mind at the real event rate, so a behavior change is judged on the same tape that will play on Saturday night.

Under that sits a catalog of named synthetic scenes in eight families, replayed through the real perception and mind code, and a runner whose exit code is zero only when every scene still behaves. The low-frame-rate blind spot and the flickering phantom each have scenes of their own, because those are the failure shapes of a dark porch.

The renderer’s modules are pure functions over plain state, so they run under node: 139 JavaScript tests with coverage floors. The Python side is 452 tests, and the last full run before the show passed at 94 percent coverage. A three-hour soak drove the real stack against a replayed walk with the show’s own client load, plus injected faults: the Pi vanishing, lag spikes, dark frames. The pipeline recovered from all of them, the rate held at 30 hertz, and the watchdog never had to fire.

A dashboard on a notebook polls the show’s health once a second, with strips for the display, the renderer, the camera and the lights, and chimes when something goes red. Nothing auto-restarts Chrome: a human is on hand all night, and the dashboard’s job is to make a failure visible fast, not to survive it.

The Night Before

I am writing this on the Friday before the show, with the dry run a week behind us and one setup evening left. The eyes are aligned to their panes, the wave detector is measured, and the runbook is printed. My assistant built the machine. I held the gates.

Halloween is the one night a year a house is allowed to look back. Tomorrow evening the porch light goes off, the window will wake, and three eyes will settle into their panes to wait. They will know the moment the first costume turns up the walk. They always look.

References