Sketch to 3D: Fast surface and depth synthesis from rough hand drawings
Executive Summary
Translating coarse 2D line sketches into 3D geometry traditionally requires hours of manual 3D modeling or 2â6 minutes of iterative Score Distillation Sampling (SDS). We developed a single-pass, multi-scale latent conditioning framework that predicts continuous surface normal fields, continuous depth maps, and physically based rendering (PBR) materials in under 3.9 seconds on consumer GPU clusters.
1. The Challenge of Sketch-to-3D Modeling
Hand sketching is the fastest way for industrial designers, architects, and digital artists to capture spatial ideas. However, hand drawings are inherently ambiguous: strokes have variable thickness, contours are often incomplete or self-overlapping, and there is zero explicit depth or lighting data.
Prior generative approaches relied on 3D NeRF optimization (e.g., DreamFusion), which requires running hundreds of diffusion denoising steps. This takes 3â5 minutes per asset, breaking creative flow and making live, interactive design exploration impossible.
2. Our Technical Approach
We reformulated the 3D synthesis pipeline into two synchronized neural streams operating in a single forward pass:
Stroke-Aware Attention Encoder (SAE)
A hierarchical tokenizer extracting multi-resolution stroke contours (1/4 to 1/32 scale) with sinusoidal position embeddings, remaining invariant to pen jitter and line gaps.
Dual-Stream Geometric Alignment
Jointly estimates metric depth and surface normal vectors while enforcing differential tangent constraints, preventing surface distortion or hollow geometry.
3. Quantitative Benchmarks
Evaluated across 1,200 diverse sketch-geometry pairs on the Objaverse-Sketch and QuickDraw-3D datasets:
| Method | Normal Cosine (â) | Depth RMSE (â) | Inference Time | VRAM |
|---|---|---|---|---|
| DreamFusion (SDS) | 0.742 | 0.184 | 240.0s | 18.2 GB |
| Zero-1-to-3 + NeRF | 0.789 | 0.142 | 180.0s | 14.5 GB |
| TripoSR (Feedforward) | 0.821 | 0.116 | 6.8s | 11.0 GB |
| Our Framework (Draw3D) | 0.946 | 0.052 | 3.84s | 6.2 GB |
4. Production Deployment on Draw3D
This architecture serves as the core rendering backbone of draw3d.online. Using TensorRT FP16 quantization with custom fused CUDA operators, production latency was reduced from 7.2s to 3.84s while cutting GPU compute costs by 62%.
Cite This Work & Download Paper
You can download the full academic PDF or copy the BibTeX citation for academic publications:
@article{bhuiyan2026sketch3d,
title={Sketch to 3D: Fast Surface and Depth Synthesis from Rough Hand Drawings},
author={Bhuiyan, Jabed and Kites 3D Vision Research Lab},
journal={Kites Technical Reports},
volume={TR-2026-01},
year={2026},
doi={10.6084/m9.figshare.33416719},
url={https://doi.org/10.6084/m9.figshare.33416719}
}