Comparing basic camera specifications such as frame rate, resolution and interface is easy. Comparing imaging performance of cameras such as quantum efficiency, temporal dark noise and saturation capacity is not. First we need to understand what these various measurements really mean.
What is quantum efficiency and is it measured at the peak or at a specific wavelength? How is signal to noise ratio different from dynamic range? This white paper will address these questions and will explain how to compare and select cameras based on the imaging performance data following the EMVA1288 standard.
EMVA1288 is a standard that defines what aspects of camera performance to measure, how to measure them and how to present the results in a unified method. The first section of the white paper will help to understand the various aspects of imaging performance of an imaging sensor. It will outline the basic concepts that are important to understand when considering how an image sensor converts light into a digital image and ultimately defines the performance of the sensor. Figure 1 presents a single pixel and highlights these concepts.
First we need to understand the noise inherent in the light itself. Light consists of discrete particles, Photons, generated by a light source. Because a light source generates photons at random times, there will be noise in the perceived intensity of the light. The physics of light states that the noise observed in the intensity of light is equivalent to the square root of the number of photons generated by the light source. This type of noise is called Shot Noise.
It should be noted that the number of photons observed by a pixel will depend on the exposure times and the light intensity. This article will consider the number of photons as a combination of exposure time and light intensity. Similarly, Pixel Size has a non-linear effect on the light collection ability of the sensor because it needs to be squared to determine the light sensitive area. This will be discussed in more detail in the next article in the context of comparing performance of two cameras.
The first step in digitizing the light is to convert the photons to electrons. This article does not get into how sensors do this, rather it presents the measure of the efficiency of the conversion. The ratio of electrons generated during the digitization process to photons is called Quantum Efficiency (QE). The example sensor in Figure 1 one has QE of 50% because 3 electrons are generated when 6 photons “fall” on the sensor.
Before electrons are digitized they are stored within the pixel, referred to as the Well. The number of electrons that can be stored within the well is called Saturation Capacity or Well Depth. If the well receives more electrons than the saturation capacity additional electrons will not be stored.
Once light collection is completed by the pixel, the charge in the well is measured and this measurement is called the Signal. The measurement of the signal in Figure 1 is represented by an arrow gage. The error associated with this measurement is called Temporal Dark Noise or Read Noise.
Finally, Grey Scale is determined by converting the signal value, expressed in electrons, into a 16-bit Analog to Digital Units (ADU) pixel value. The ratio between the analog signal value to digital grey scale value is referred to as Gain and is measured in electrons per ADU. The gain parameter as defined by the EMVA1288 standard should not be confused with the gain of the “analog to digital” conversion process.
When evaluating camera performance, it is very common to refer to Signal to Noise Ratio andDynamic Range. These two measures of camera performance consider the ratio of noise observed by the camera versus the signal. The difference is that Dynamic Range considers only the Temporal Dark Noise, while Signal to Noise Ratio includes the root mean square (RMS) summation of the Shot Noise as well.
Absolute sensitivity threshold is the number of photons needed to get a signal equivalent to the noise observed by the sensor. This is an important metric because it represents the theoretical minimum amount of light needed to observe any meaningful signal at all. Details of this measurement will be covered in more detail in the following articles.
To help compare sensors and cameras based on the EMVA1288 standard, Point Grey created an industry-first comprehensive study of imaging performance of more than 70 camera models. Seewww.ptgrey.com/sensor-review
For the purpose of this white paper, we will consider applications such as license plate recognition (LPR) or optical character recognition (OCR) where monochrome imaging is commonly used and the amount of light that camera can collect may be limited due to short exposure times. It is fairly straight forward to determine the resolution, frame rate and field of view required to solve an imaging problem, however deciding whether the camera will have sufficient imaging performance can be more difficult.
This challenge is typically solved by trial and error. Let’s consider an example where a vision system designer determines that a VGA camera with ?’’ CCD running at 30 FPS is sufficient in the application. The initial tests may show that the camera has sufficient sensitivity at exposure times of 10 ms when the object is stationary. See Figure 2 showing a simple example with characters B,8, D and 0 that can be easily confused by a vision algorithm. The top left image taken with a ?’’ CCD camera produces images suitable for image processing.
However, when the object starts to move, exposure times need to be reduced and the camera is not able to provide useful information because the letter “B” and “D” cannot be distinguished from numbers “8” and “0”. Images in the middle and bottom left of Figure 2 show deterioration of image quality. In particular ?’’ CCD at 2.5 ms exposure time produces images unsuitable for image processing.
For the purposes of this example, the assumption is that large depth of field is not required and therefore the minimum F-number of the lens is acceptable. In other words, it is not possible to collect more light by opening the shutter of the lens.
So, the designer needs to consider a different camera. The question is whether a different camera has a chance to improve the performance of the system. Using a larger sensor has generally been accepted as a good way of solving low light performance problems, so a ?’’ sensor could be a good choice. But instead of continuing with trial and error, considering the EMVA 1288 imaging performance of the camera can be useful.
Looking at the EMVA 1288 data, it can be observed that ?’’ sensor has better quantum efficiency and lower noise, but that ?’’ CCD has a larger pixel and larger saturation capacity. This article shows how to determine whether the ?’’ camera will perform better.
Figure 3 compares the cameras by plotting the signal value versus the light density (photons/ ?m2). The signal as a function of light density is determined using the following formula:
An important assumption made by this article is that the lenses have the same field of view, same F-number and same camera settings.
The figure shows that for the same light density, the ?’’ sensor will generate higher signal. It can also be observed that saturation occurs at a similar light density level of 700 photons/?m2, however the ?’’ sensor has significantly higher saturation capacity.
In the application that is considered in this white paper, the comparison of cameras needs to be done at low light level. Therefore considering the noise levels becomes particularly important.
Figure 4 shows the signal and noise at low lighting levels. The noise presented in the figure is an RMS summation of Temporal Dark Noise and Shot Noise which was calculated using the following formula:
The graph shows that absolute sensitivity threshold (the light level at which signal is equal to the noise) is reached by the ?’’ sensor at a slightly lower level than that of ?’’ sensor. The more important measure needed to determine which camera will perform better in low light applications is the signal to noise ratio (SNR).
Figure 5 shows the SNR of the two cameras as a function of lighting level.
Based on the higher signal to noise ratio of the ?’’ sensor, theory suggests that the ?’’ cameras should perform better than ?’’ camera at low light levels.
From images in Figure 2, it can be seen that at 2.5 ms exposure time, the ?’’ sensor preserves the shape of the characters at all exposure times, while the ?’’ sensor makes it difficult to distinguish between characters. The ?’’ sensor therefore performs better and the practical results are in-line with the theory.
Point Grey has done an extensive study of cameras and has published the EMVA 1288 imaging performance results. This information can be used to compare the performance of different camera models. While camera implementation does influence the imaging performance, this study can be generally useful when comparing any two cameras with sensors that are covered in the document.
Point Grey offers specific camera comparison documents. Please contact firstname.lastname@example.org to request a comparison between Point Grey’s camera models.
It should be noted that the method outlined in this white paper is useful to get a general idea of how well a camera will perform compared to another. This method can assist in ruling out cameras that are not likely to improve the required performance, however, the ultimate test of the performance of the camera is in the actual application.
Now we will compare the performance of a traditional CCD sensor versus a modern CMOS sensor in low light imaging conditions and in a scene with a wide range of lighting conditions.
In the previous section we showed that a camera with the Sony ICX414, a ?’’ VGA CCD, performs better in low light conditions than a camera with the Sony ICX618, a ?’’ VGA CCD. Now we will compare the ?’’ VGA CCD with the new Sony Pregius IMX249, 1/1.2’’ 2.3Mpix global shutter CMOS sensor.
At first glance, this may seem like comparing “apples to oranges”, however the cost of the cameras with these two sensors is comparable at approximately ?400, a VGA region of interest in the CMOS camera is actually closer to the optical size of the ?’’ camera and the frame rates are also similar at the VGA resolution.
The EMVA 1288 data for the cameras shows that the IMX249 CMOS sensor has significantly better quantum efficiency, lower noise and higher saturation capacity. On the other hand the ICX414 CCD sensor has a larger pixel, which was the critical parameter in the example presented in the previous article.