# Image Acquisition with an iToF Sensor
This page will help you get the in's and out's of the iToF principle to help you configure the Camera in an optimal way.
It will first address the iToF principle and explain some good imaging practices depending on the scene.

```{contents} Topics
:local:
:depth: 2
```

## Quick guide
This summary is a quick overview of which settings have which effects. Refer to the [](#concepts) and [](#itof-technology) below to understand those effects.
```{table}
| Setting                | Positive Effects when increasing                                       | Negative Effects when increasing                                                                                                                                                                                                                                                                                                  |
| ---------------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Integration Time       | + Higher Amplitude<br>+ Better SNR<br>+ Less illumination power needed | - Higher [](#motion-blur)<br>- Higher [](#motion-shift)<br>- Lower Max. Framerate<br>- More prone to Multipath effects<br>- Saturation Effects more likely<br>- Increased Thermal Managment Requirements<br>- Higher Shot Noise<br>- Increased Thermal Managmenet Requirements for the Illumination<br>- Has Impact on Eye Safety |
| Modulation Frequency   | + Lower Distance Noise<br>+ Better Demodulation Contrast<br>           | - Lower [Unambiguity Range](#itof-technology)                                                                                                                                                                                                                                                                                     |
| Framerate              | + Less [](#motion-blur)<br>+ Less [](#motion-shift)                    | - Increased Thermal Managment<br>- Increased Power Supply Requirements<br>- Lower Max. Integration Time<br>- Has Impact on Eye Safety                                                                                                                                                                                             |
| LED Illumination Power | + Higher Amplitude<br>+ Better SNR+ Less integration time needed       | - Saturation Effects more likely<br>- Increased Power Supply Requirements<br>- Increased Thermal Managment Requirements<br>- Has Impact on Eye Safety                                                                                                                                                                             |
| LED FOV                | + Larger Area illuminated                                              | - Less Irradiance on Objects                                                                                                                                                                                                                                                                                                      |
| Lense FOV              | + Larger Area visible                                                  | - Less [Angular Resolution](#resolution)<br>- More Sensitive to Ambient Light Effects                                                                                                                                                                                                                                             |
```

## iToF Technology
The following section will introduce the iToF technology and explain following terms in the context of iToF imaging:
- DCS
- Unambiguity Range
- Amplitude

The time of flight technologies refer to measuring an objects distance by using the time it takes from emission to reception of the reflected light.
In case of iToF it is indirect as we do not directly measure the time it takes directly but extract it through **autocorrelation** of modulation and demodulation. Doing this allows us to extract the phase shift from sample images referred to as a **DCS**. In theory it is possible to extract the phase with 2 DCS only but in practice too many artifact render this method too unreliable. It is therefore preferred to use 4 DCS which allow to compensate said artifacts.

```{figure} ../_static/images/itof_phase_shift.svg
:name: itof-phase-shift
:alt: iToF phase shift example
:align: center
Pixel signal (blue bar) for different DCS demodulation phases
```

This method inherently creates an ambiguity as it is not possible to differentiate two objects which yield the same phase but are not in the same period. For example in the [figure above](itof-phase-shift), object A has the same phase shift as Object C. Even though object C is a full period further away.

The range in which object can be positioned unambiguously is called the **unambiguity range**. A range can be converted into a **modulation frequency** with following formula:
```{math} D_{unambiguity} = \dfrac{c}{2}*T_{mod}
```
Note that the division by two is required since the light travels to the object and back.\
It is very important to be aware of the desired maximum range when operating in iToF as it partially determines following:
- **Distance Resolution** of one 3D image
- **Illumination power and/or integration time** required to see the objects

## Advanced Imaging modes
In standard operation each 4 DCS are taken for each pair of distance, amplitude to calculate the distance, amplitude.
```{figure} ../_static/images/itof_std_mode.drawio.svg
:name: std-acquisition
:alt: Standard acquisition mode frame readout showing possible calculation timings
:align: left
:figwidth: 100%
4 DCS Standard Mode Acquisition
```
This yields a constant stream of data that does not need further interpretation.

### Rolling Acquisition
It is possible to create a shifting window for the calculation, which allows to caluculate a dsitance and an amplitude after each DCS acquisition and increase the frame rate by a factor of 4.
```{figure} ../_static/images/itof_rolling_mode.drawio.svg
:name: fig-rolling-acquisition
:alt: Rolling acquisition mode frame readout showing possible calculation timings
:align: left
:figwidth: 100%
4 DCS Rolling Mode Acquisition
```
This mode is only recommended in cases where a fast continous stream of frames is of interest as a lower rate leads to uncorrelated DCS resulting in nonsensical image data. This increases the requirement on the processing side for it to take full advantage of this mode.

### HDR Aqcuisition
The HDR mode consist of taking multiple images with different integration times. This allows to detect a wider range of object reflectivity or distance without under or overexposing the image.
```{figure} ../_static/images/itof_hdr_mode.drawio.svg
:figwidth: 100%
4 DCS HDR Mode Acquisition
```
The images can be combined to one using a quality criteria (see [](#amplitude)).

### Multi-Modulation Frequency Acquisition
The modulation frequency determines the [unambiguity range](#itof-technology). This means that for an application with a longer range the distance resolution gets lower. The multi-modulation frequiency acquisition allows to keep the similar accuracy by combining multiple images with different modulation frequency. Each additional frequency adds additional information about the position.
```{figure} ../_static/images/multi-modulation.drawio.svg
:name: multi-modulation
:alt: multi modulation example
:align: left
:figwidth: 100%
Disambiguate higher frequency(2nd) using a lower frequency(1st)
```

In the [figure above](multi-modulation) two frames with two modulation frequencies are acquired. In this example the higher frequency was chosen to be exactly 4 times the lower (Note that it is not a requirement). This equates to saying that the full period of {math}`f_{mod,2}` lies within a quarter of {math}`f_{mod,1}` which, in turn, allows to determine the required shift to be added to the phase extracted with the {math}`f_{mod,2}` modulation frame.
Additionally, the resulting distances can be averaged to increase the accuracy even further.

### 2-DCS Acquisition
As hinted int the [](#itof-technology) section, it is also possible to calculate the distance based on 2 DCS at the cost of error inducing artifacts.
```{figure} ../_static/images/itof_2dcs_mode.drawio.svg
```
The acquisition modes [](#rolling-acquisition), [](#hdr-aqcuisition) and [](#multi-modulation-frequency-acquisition) are easily combined with this mode.

## Concepts
This section will explain various concepts which are helpful to know for iToF imaging.

### Amplitude
In standard 2-D imaging, the quality of an image exposure is directly correleated to the image values. In iToF imaging this is more complex as a DCS does not contain direct information on illlumination intensity. This is where the signal **Amplitude** comes in.
The amplitude can be used as a **metric** of the signal **quality**. Allowing following classification depending on the amplitude optimized for noise as reference:
```{table} \* The optimal amplitude may vary depending on the readout architechture
| {math}`A/A_{Optimal}`\* | Classification         | Description                                                                          | Improvment Option                                   |
| ----------------------- | ---------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------- |
| < 2%                    | Invalid Distance value | Very inaccurate distance measurement<br>May provide information with temporal filter | Increase Integration Time and/or Illumination Power |
| 2%...5%                 | Valid Distance value   | High distance noise                                                                  | Increase Integration Time and/or Illumination Power |
| 5%...100%               | Good Distance value    | Low distance noise                                                                   | No change required                                  |
| > 100%                  | Overexposed            | Overflow                                                                             | Decrease Integration Time and/or Illumination Power |
```

The amplitude scales linearly with integration time.

### Ambient Light Suppression, Overexposure and Saturation
In a simplified version the ESPROS Ambient Light Suppression technology is composed of two charger holder. The pixel and the floating diffusion. The pixel is photosensitive and converts as much incoming light to charge as possible. This charge then contains the useful signal but also dark current and ambient light, hence its high capacity.
```{figure} ../_static/images/itof_overexposed.drawio.svg
:width: 90%
Possible states of pixe and floating diffusion which result in erronous data
```
The charge is then transferred to the floating diffusion with a lower capacity in a way that only the useful signal charge is transfered.
In case the **pixel** exceeds it's capacity we refer to it as **saturation**. Whe the **floating diffusion** contains a signal which exceeds the range of the ADC it is refered to as **overexposed**.

In addition, traditional methods such as band-pass filtering are used to block ambient light that falls outside the emitter’s wavelength range. Hence, only the ambient light in the dedicated sensor's emitter wavelength has to be canceled with ESPROS Ambient Light Suppression technology.

### Resolution
Two sets of resolution need to differnetiated when talking about iToF resolution. The angular resolution, which tells which angle is resolved from one pixel to another and the distance resolution, which tells the lowest distance from the camera between two objects at which it is possible to see the difference.

The angluar resolution is determined by the pixel pitch and the receiver lense.

The distance resolution is determined by signal noise and modulation frequency.

### Motion Blur
The motion blur happens during the integration time of a single image. The capture of an object which is in movement during this period is going blur its capture in the direction of its movement, leaving a trail of phantom images which combine during integration. For very fast moving objects this may render detection complicated.
```{figure} ../_static/images/motion_blur.drawio.svg
:align: center
Motion blur of moving object within one integration
```
The blurring effect is increased when:
- an object has a higher speed
- the integration time is longer

### Motion Shift
The motion shift happens when combining images which were acquired sequentially.
```{figure} ../_static/images/motion_shift.drawio.svg
:align: center
Motion shift of moving object when combining 4 frames.
```
It is typical to have moving objects in the scene watched by a camera and as described in the previous section [](#itof-technology) multiple DCS are required to extract the distance. Since these images are produced sequentially an object may appear at different places within a combined image.

The motion shift effect is increased when:
- an object has a higher speed
- The integration time is longer
- The readout & transmission time is longer

### Ambient Light
Any light which is not emitted intentionally for the 3-D image is part of the ambient light. Usual sources are:
- the sun
- any kind of lamp
- other IR-emitters
This light is parasitic and may overweigh the signal light and may lead to saturation. The ambient light charge scales linearly with integration time.

### Dark Current
Dark current is a thermal effect that affects all silicon-based electronics. It is a statistical phenomenon where electrons get randomly "generated". Usually this charge will dissipate however the pixel will keep this charge.
The amount of charge appearing scales exponentially with the temperature and linearly with integration time.

For ESPROS iToF sensors this effect is only noticeable in the grayscale mode, as the 3-D mode has internal differentiation

### Multipath
Multipath are an issue especially in closed spaces, multiple cylindrical reflective objects and/or multiple diffuse parallel planes.
The effects when the light can travel multiple light paths that land on the same pixel. The phase of the incident light is not the same when the light paths have different length, meaning that they will mix up in the pixel.
```{figure} ../_static/images/itof_multipath.drawio.svg
:align: center
Multiple light paths may land on the same pixel which lead to errors in the resulting phase
```
The image above is an example of the multipath issue however, often, the light can travel through plenty of ways to enter the same pixel as the primary path.

### Lense Distortion
When using manufactured spherical lenses there is always an associated distortion.
Every lens-design has its own distortion pattern which can be compensated.

### Demodulation Contrast
Since the it is not possible to demodulate perfectly there will always be a small overlap when splitting the charge. This overlap is constant and therefore negligable for small modulation frequencies but the higher the frequencies the higher the effect.
```{figure} ../_static/images/itof_demodulation_contrast.drawio.svg
:align: center
Demodulation Contrast acts similarily to a  1st order low pass filter
```