A Single-shot Fourier Camera and Its Application to Multipath ToF Imaging

Contributed by Peyman Fayyaz Shahandashti and co-authors, based on the IEEEXplore® article, “Simultaneous Multifrequency Demodulation for Single-Shot Multiple-Path ToF Imaging”, published in the IEEE Transactions on Computational Imaging in January 2024, and the SPS Webinar of the same name, available on the SPS Resource Center.

Introduction

As computational imaging advances, specialized image sensor designs and optimized processing algorithms are becoming essential for accelerating tasks such as signal processing, feature extraction, and deep learning inference. A common challenge across many domains is capturing highly concentrated time-domain responses efficiently. This challenge arises in a wide range of applications, including multipath time-of-flight imaging, transient and ultrafast photography, high-energy particle detection, and non-line-of-sight (NLOS) imaging. To address it, we combine a novel sensor design with efficient signal processing to recover sparse macroscopic impulse response functions (IRFs) from modulated light. The sensor enables a single-shot Fourier camera and supports fast reconstruction of sparse IRFs using optimal Fourier-based sensing kernels. Conventional time-domain sampling based on Shannon-Nyquist criteria requires prohibitively high sampling rates. In contrast, our Fourier camera effectively condenses highly concentrated time-domain signals into a few Fourier coefficients. This approach is optimal, as it maximizes incoherence between the sensing functions and the signal’s representation basis.

We focus particularly on resolving multipath interference (MPI) in indirect time-of-flight (iToF) sensors. iToF sensors offer the benefits of compact integration, high spatial resolution, and efficient hardware implementations. However, they still struggle with the MPI issue. MPI occurs when light reflects off multiple surfaces or scatters through translucent materials before returning to the sensor, leading to overlapping signals and results in erroneous depth measurements. The aim is to capture the IRF of a macroscopic scene, which is often sparse or highly compressible. By leveraging multifrequency data, the Fourier camera can effectively distinguish direct and indirect light paths, thereby improving the accuracy and reliability of depth information at high speed, ensuring efficient and accurate 3D imaging in real-time applications.

Proposed Method: Single-Shot Multifrequency Sensing

Effectively addressing MPI requires sufficient frequency-domain bandwidth, yet most current iToF architectures aren't built with this in mind and cannot handle MPI natively. Commercial iToF cameras typically rely on two or more modulation frequencies to tackle MPI, phase wrapping, and depth ambiguity. However, they acquire these frequencies sequentially, which introduces motion artifacts and reduces frame rates. Our approach offers a novel alternative: simultaneous multifrequency measurement within an iToF sensor, enabling single-shot MPI resolution [1]. This is achieved through a hardware-software co-design that combines specialized circuit design with robust signal processing.

Figure 1(a) shows the single-shot multifrequency concept. Instead of cycling through frequencies, our sensor captures all of them at once by using macro-pixels, each made up of subpixels modulated at distinct frequencies. This results in a set of rich Fourier-domain measurements acquired in a single exposure, ideal for separating overlapping light paths. As illustrated in Figure 1(b), unlike conventional ToF sensors, we illuminate the scene with ultrashort, periodic pulses, approximating a Dirac delta, and demodulate each pixel at a unique frequency.

Spectral Signal Processing for Multipath Recovery:

With multiple frequency responses captured in a single frame, we reformulate MPI correction as a spectral estimation problem. Each light path induces a different phase shift, appearing as distinct frequency components in the Fourier domain. We explore two algorithms with low computational costs to retrieve path-specific depths from multifrequency measurements and compare with a mono-frequency baseline approach, the four-phase algorithm [2].

We used the matrix pencil method, a non-iterative, closed-form algorithm that models each path as a complex exponential and recovers them using rank constraints [3], and inverse discrete Fourier transform (IDFT) with time-domain super-resolution, a fast method that reconstructs the scene’s temporal response using frequency-domain samples [4]. To push accuracy further, we proposed data refinement techniques that leverage self-compensation using calibration data, scene-informed priors using distance probability density functions, and denoising based on low-rank structure (Cadzow filtering).

Fig. 1.  Conventional and simultaneous multifrequency measurements concept (a),  the proposed iToF imaging setup with multifrequency demodulation and impulse illumination (b), and reconstructed 3D image with the baseline, matrix pencil, and IDFT methods (c). The matrix pencil and IDFT methods resolve MPI  within our single-shot iToF system.

Testing and Results:

We present the results of a series of experiments aimed at evaluating the performance of the proposed joint design of the sensor architecture and depth retrieval algorithm. We evaluated our approach using electrical simulations of the full sensor pipeline and standard MPI benchmarks from the iToF2dToF dataset [5].

The system was tested for different cases and was able to accurately resolve up to three separate light paths in a single shot, maintaining centimeter-level precision even under low SNR conditions. Both methods yield significant improvements over conventional four-phase ToF processing, showing better robustness in complex scenes with multiple bounces. Fig. 1(c) shows the retrieved depth maps. As highlighted in the marked areas, the proposed system and algorithm provide robust results even in MPI. In comparison with state-of-the-art techniques, the depth error is in par with other methods, especially in challenging scenarios including MPI.

Conclusion:

We introduce a new joint hardware–software design that addresses the MPI challenge in iToF depth imaging through single-shot, on-chip multifrequency demodulation. The system captures multiple Fourier coefficients of the scene response function in one exposure and leverages robust algorithms to solve the inverse problem through parametric closed-form depth reconstruction.

For a detailed description of the hardware design, algorithmic implementation, and refinement methods, along with experimental evaluations on both reflective and diffusive MPI, please see our original manuscript: Simultaneous Multifrequency Demodulation for Single-Shot Multiple-Path ToF Imaging  [1].

References:

[1] Shahandashti, Peyman Fayyaz, et al. "Simultaneous multifrequency demodulation for single-shot multiple-path ToF imaging." IEEE Transactions on Computational Imaging 10 (2024): 54-68, doi: https://dx.doi.org/10.1109/TCI.2023.3348758.

[2] Lange, Robert, and Peter Seitz. "Solid-state time-of-flight range camera." IEEE Journal of Quantum Electronics 37.3 (2001): 390-397.

[3] Hua, Yingbo, and Tapan K. Sarkar. "Matrix pencil method for estimating parameters of exponentially damped/undamped sinusoids in noise." IEEE Transactions on Acoustics, Speech, and Signal Processing 38.5 (1990): 814-824.

[4] P. F. Shahandashti, et al. “Fast time-domain super-resolution for single-shot multi-path ToF imaging,” in Proc. 29th IEEE Int. Conf. Electron., Circuits, Syst., 2022, pp. 1–4.

[5] F. Gutierrez-Barragan, et al. “iToF2dToF: A robust and flexible representation for data-driven time-of-flight imaging,” IEEE Transactions on Computational Imaging, vol. 7, pp. 1205–1214, 2021.