Fathom

Community-first research

Three things from the blind and low-vision community shaped everything that followed. First, experienced navigators don't want a running commentary. They want what a good human guide gives them: attention, and silence until something actually matters. That became Lookout's whole principle, that silence means safety. Second, trust runs both ways. People have to trust what the app says and what it doesn't say, and an app that cries wolf loses them faster than one that misses the occasional obstacle. For this audience the tolerance for false alarms is basically zero. Third, vision loss isn't one thing, so every interaction needs a visual, an audio, and a haptic version. Nothing can live in a single channel.

System prompts as design artifacts

When the AI's voice is the interface, the system prompts stop being engineering detail and become the most important design artifact in the project. I worked them the way I'd work a component library, with explicit rules, anti-patterns, and a defined personality. The Lookout prompt spells out exactly what earns speech (a step, a head-height obstacle, someone on a collision course) and what doesn't (a wall, furniture off to the side, a sound the user can already hear). It flatly forbids narrating a clear, safe walk. The Snapshot prompt lays every description out in the same spatial order, space type first, then ahead, left, and right, then landmarks and signage, so the mental map comes out the same shape every time.

Mode architecture through user feedback

Early concepts had five modes. Community feedback cut that down to a learnable core: Lookout and Go, with Snapshot as something you can fire from anywhere or off the iPhone's Action Button. Getting it under a minute to learn mattered, because people experience this entirely through audio and haptics. Later, as the app matured, I added Task, which turns a spoken goal into a step-by-step plan, and an Assistant you can just talk to. Every addition had to clear the same bar: could a blind user understand it and run it by ear in under a minute? The shape of the modes follows how blind users actually move through the world, not how the AI happens to be built.

FathomUI — designing beyond the visual

FathomUI is the design system I built for this. The visual layer uses a warm grayscale with no pure black or white, because pure extremes cause halation for low-vision eyes. Contrast clears WCAG AAA on the main surfaces, and a Contrast Boost mode pushes primary text past 19:1 for people who need it. The harder and more interesting work was extending the system into sound and touch. Each mode change has its own sound. Each hazard level maps to a haptic intensity. Silence is a defined state rather than an absence: when Lookout is on and the path is clear, you hear nothing, on purpose. The app is meant to be just as complete by ear or by touch as it is by sight.

Choosing Swift and building native

Why native? The iOS accessibility stack (VoiceOver, AVSpeechSynthesizer, ARKit, LiDAR, Core ML) only behaves correctly in native code. React Native and Flutter put a layer between you and the platform that fights VoiceOver in particular, and Fathom needed fine control over the OS accessibility APIs. So, Swift. I started knowing almost none of it and ended up with roughly 50,000 lines across 200+ files. I didn't get there by handing everything to Claude, which leaves you code you can't debug later. I got there by learning enough of Swift's type system, its concurrency model, and SwiftUI to review what came back, catch the architectural mistakes, and make the calls the AI couldn't. Swift's protocol-oriented style turned out to fit the abstraction Fathom needed; the AIVisionProvider protocol reads like idiomatic Swift, not a bolt-on.

AI architecture and model selection

Fathom runs two AI channels plus an on-device safety net. A Gemini REST channel handles the timed scene narration behind Lookout and Go. Running a cloud model on a live camera feed gets expensive fast, so cost was a design constraint from the start. I needed a cap to keep spend from running away, and I engineered the REST narration to be cheap and efficient underneath it: the model is adaptive, dropping Lookout (by far the highest-volume caller) to an economy tier; the system prompt is cached instead of re-sent every second; and frames go out only as often as they're actually useful. Task and the Assistant lean on more advanced, more expensive model calls, which is why they sit behind Fathom Plus while Lookout and Snapshot stay free. The second channel is Gemini Live over a WebSocket, powering the real-time back-and-forth in the Assistant and Live Task. Both go through my own Supabase Edge Function proxy, authenticated with App Attest, so the API key never ships in the clear and I can change models on the server. Under all of it, Core ML running YOLO11 does on-device object detection as a fallback and a safety net. Because everything sits behind the AIVisionProvider protocol, the rest of the app doesn't know or care which model is live, and moving to a better one is a config change rather than a rewrite.

Privacy designed at the architecture level

Sending a blind user's camera frames to a cloud AI is a real privacy responsibility, and Apple treats it that way too. Fathom asks for explicit consent before anything reaches a third-party AI, and I learned to enforce that at the network boundary instead of on a single screen, so there's simply no code path where a frame leaves the device without it. Voice input is transcribed on-device with Apple's Speech framework, and only the resulting text goes out, so raw microphone audio never leaves the phone. There are no accounts and no personal data collected, just anonymous analytics you can switch off. Privacy here was a design problem as much as an engineering one, and it turned out to be central to passing review.

LiDAR, grounding, and mobile hardware

Before Fathom I knew iPhone Pros had LiDAR, but not how ARKit exposed it or what it was good for. It builds a real-time depth map from infrared pulses, timed by how long they take to bounce back, and ARKit hands you that through ARFrame's depthMap. It's plenty for catching steps and drop-offs, and for something subtler: grounding. A cloud model working off a single frame will guess a distance ("the doorway's about ten feet ahead") and sometimes be wrong, which is dangerous here. So Fathom feeds the real forward LiDAR distance into the prompt and the AI states a measured number instead of a guess, gated on freshness so it never invents a wall in an open hallway. The headache was false positives at furniture edges and doorways; temporal filtering cleared most of them out. Running ARKit and Gemini at once is expensive, so the frame-rate and thermal logic are tuned to keep the phone usable on a long walk.

Version control and a crew of agents

I wrote roughly 50,000 lines of production Swift solo, with Claude Code as the coding partner. The loop was simple: I write the PRD, the feature specs, and the system prompts, then Claude and I scaffold the architecture, build the services, and iterate on the Swift in real time. I also built my own crew of specialized agents for the parts that wanted an expert in the room: one for iOS architecture reviews, one for the ML and computer-vision pipeline, one for Swift code review, others for debugging and the build-and-release grind. Each carried the standards and context for its slice of the app, so I could hand off a focused problem and get back work that already knew the house rules. Git mattered more than I expected for a team of one. A branch per capability. Commit messages still legible months later. The room to back out of an architecture decision that turned out wrong, of which there were several: branches abandoned after two days, service layers torn down and rebuilt once the first cut couldn't handle the edge cases. The git log is a more honest account of how this got built than any case study. And the spec was how I told a partner that knew Swift but nothing about blind users what to actually build. Without it, Claude writes correct code for the wrong problem.

Real-device testing and performance optimization

Everything got tested on a physical iPhone Pro: LiDAR step detection calibrated in real stairwells, VoiceOver checked on every screen, haptics tuned for a one-handed grip with the phone held forward at chest height. Performance problems don't show up in code review. They show up twenty minutes in, when the phone is warm. ARFrame retention, where captured depth frames weren't being released fast enough, only appeared under sustained load, and fixing it meant actually understanding Swift's reference-counting model. VoiceOver brought its own class of conflict: it and the speech synthesizer both want the audio channel, so a hazard alert can get cut off or stuck behind an announcement. The only way to design around that is to test with VoiceOver genuinely on, which is a different app than the one you get with it off, and which surfaced interactions you'd never catch reading the docs.

App Review and launch

Getting onto the App Store took three rejections, and each one made the app better. The first was for a subscription flow missing its Terms of Use and Privacy Policy links. The second was the one that mattered: Apple wants explicit in-app consent before any data goes to a third-party AI, spelling out what's sent and to whom, and a privacy policy on its own doesn't cut it. That's what pushed the consent check down to the network chokepoint instead of a screen. The third was a completeness bug, where denying camera access bounced the user straight to Settings instead of explaining and letting them choose. Fixing all of it meant reworking the consent architecture, the permission flow, and a pile of App Store Connect metadata, right down to a demo script a sighted reviewer could follow. The fourth build passed. Review isn't a gate at the end; it's part of the design.