FingerIO (Remote)

FingerIO was a phone system: its whole point was that a phone already carries the speaker, the microphone and the compute to turn air into a touchscreen. This page puts it back on one. The phone emits the OFDM symbol, records its own echoes, and runs the entire channel-estimation pipeline on board, sending only the finished picture here over WebRTC. The maths is explained on the single-device page; this is the same code with the front end moved to where your hand is.

Pair your phone

Code — — — —

Generating session…

On the phone, open sensingstudio.org/fingerio-remote and enter this code. Both devices need to be online; the audio never leaves the phone.

Parameters

Band low 18.0 kHz

Lower edge of the inaudible OFDM band. Phone speakers are small and generally happier up here than a laptop’s; if you can hear a faint hiss from across the room, raise it.

Band high 22.0 kHz

Upper edge. Wider band → finer time resolution. Phone microphones usually run to 20 kHz honestly and roll off above; if the impulse response looks like mush, bring this down to 20 kHz and give up a little resolution.

Symbol 5.3 ms / 0.91 m

Symbol length N. Longer symbols span more range bins (max range = c·N/2f_s) but update the CIR less often. 256 at 48 kHz is the FingerIO sweet spot: 91 cm of unambiguous range refreshed at ~190 Hz.

Background τ 0.60 s

Time constant of the running average that estimates static clutter. Short τ tracks slow movement (and erases it from the display); long τ lets every wave of the hand glow but holds onto stale clutter — including the phone being picked up.

Volume 0.30

Output amplitude on the phone. The band is above the audible range, so loud is fine — but a phone earpiece distorts sooner than a laptop tweeter. If the impulse response looks clipped and flat, back off.

Display range 50 cm

Maximum range plotted, and the number of range bins the phone actually bothers to send. Most of the action is within the first 30 cm of the phone.

Run

Pair a phone, press Start, then tap the phone once to let it play and record.

Lay the phone flat on the desk, screen up, and leave it there — then wave your hand 5–25 cm above it. Echo cancellation, AGC and noise suppression are all disabled on the phone so it captures the sonar honestly. Keep the phone’s screen awake: most phones suspend audio and timers within seconds of the screen going dark.

Why the phone runs the whole pipeline

The obvious split — phone streams its microphone, laptop does the signal processing — cannot work, and it is worth knowing why. A WebRTC audio track is Opus-coded and resampled through a jitter buffer. Opus is a perceptual codec: it keeps what you hear and throws away the rest, and the jitter buffer stretches and shrinks the stream by fractions of a sample to hold two clocks together. Both are harmless for speech and fatal here. The measurement is sample timing: one range bin is one sample, 3.57 mm at 48 kHz, and the symbol-boundary search that keeps the circular-convolution trick honest resolves to a single sample. A jitter buffer that quietly inserts one is a reflector that jumped 3.6 cm.

So the phone keeps its samples. It plays the OFDM loop, records at the native rate, re-syncs the symbol grid twice a second, and runs the same fingerio-core.js the single-device page runs. What crosses the network is the finished picture: the impulse response and the clutter estimate quantised to a byte per range bin, the heatmap columns the phone has produced since the last packet, and the tracked-distance samples thinned to about 40 Hz. Fifteen packets a second, a few tens of kB/s.

The CIR-rate readout is the phone’s, not the laptop’s — it counts channel estimates the phone actually finished. It should sit near f_s/N (about 190 Hz at the defaults). If it sags, the phone is the bottleneck: close its other apps, or go to a longer symbol.

Things to try

Wave a flat palm horizontally above the phone at 5, 15, 25 cm. Watch the bright streak ride up and down the y-axis while you stand at the laptop — two devices, one measurement.
Walk away with the laptop. The link is peer-to-peer, so the sonar keeps running as long as both ends have a network; the range×time strip becomes a record of a room you are no longer standing in.
Hold still for a few seconds. The running average eats the streak — that is τ doing its job. Move again and it lights right back up.
Drop the upper band edge to 19 kHz. Range resolution gets worse (peaks widen in n) because removing high frequencies spreads the inverse-FFT impulse. Bandwidth is resolution.
Switch the symbol to 1024. The CIR refresh slows to ~47 Hz but the unambiguous range stretches to 3.6 m — enough to see someone crossing the room the phone is sitting in.
Compare the same gesture on the phone and on the laptop’s own speaker. The phone’s speaker and mic are centimetres apart and both point the same way; a laptop’s are metres of plastic apart. The direct-path bin at the left edge is the difference.

What this isn’t

The original FingerIO uses the phone’s two-microphone array to triangulate a finger in 2D, plus a finite-state tracker that follows one peak. The browser hands out a single mixed channel, so this page does the depth dimension only — the per-symbol CIR pipeline the rest of the system rests on. Add a second microphone and the lateral coordinate is the same trick run twice.

Sonar endpoint

Enter the code shown on the laptop to turn this phone into the sonar.