Sketching Interfaces

Ben and I had a vision: sketch an interface on a whiteboard; watch the computer understand your intent; generate production code. Simple, right?

This was 2017. Years before GPT made multimodal reasoning obvious. Years before anyone thought you could bridge the gap between analog gesture and digital precision without massive friction.

We borrowed every webcam IT would give us. Spent months at our desks drawing rectangles with Sharpies, training CV models on sketches of Airbnb's design system. People walking past thought we'd lost it.

The architecture: a CV model detected sketched components and generated an abstract syntax tree. That AST fed into Canon, my system for maintaining design-system-as-single-source-of-truth and rendering across platforms. The sketch became React Native code, live, as you drew it.

I prototyped with Gene Kogan's open-source OpenCV classification work, then dove into academic papers reimplementing detection with R-CNN and YOLO—the tools we had before transformers. The developer experience of AI & ML engineering in 2017 was... not great. I have deep appreciation for today's abstractions.

Looking back, object detection wasn't the right architecture—diffusion models and LLMs would make this interaction pattern trivial a few years later. But we couldn't wait for the future. We saw what we wanted and built it with what we had.

Everyone thought we were insane. Then the demo worked. Sketch a screen, watch it become a working prototype using Airbnb's actual design system components. Idea to implementation in seconds.

A few years later, this interaction pattern started showing up everywhere. Figma announced AI features with similar concepts. Other tools followed. We weren't insane. Just early.