Smartphone digital image processing

Back in 2005, the Nokia N90 with swivel screen clamshell design ushered in an era of competition between digital cameras and smartphones.

In those days, most experts characterized the 2-megapixel camera of this smartphone with such epithets as fantastic, unprecedented, gorgeous, etc. Of course, today the picture quality of Nokia N90 evoke other emotions. Probably professional photographers smiled condescendingly, looking at this picture with a low resolution, a minimum dynamic range and a very high noise level even in bright sunshine.

Indeed, the black silhouette of a tree without gradations of shadows and the indistinguishable car number could hardly excite them in terms of possible competition.

But after a few years the situation has changed radically. The sales schedule very convincingly illustrates this situation.

Of course, the market for expensive DSLR cameras has specific features. Unlike phones, they perfectly serve their owners for 10 years or more. In addition, new models provide an extremely small increase in quality, further limiting the motivation for replacement. But, of course, cameraphones have become one of the main reasons of the stagnation in this segment. In fact, today only semi-professional and professional DSLR cameras continue to maintain their market position.

A modern cameraphone vs a professional camera for $ 7500?

Recently, the popular Matti Haapoja photographer presented a side-by-side comparison of the picture quality taken with the iPhone 11 Pro and the expensive Canon EOS-1D X Mark II, costing $ 7,500.

Probably, an experienced photographer with sharp vision will be able to correctly determine that the A-picture on the left belong to the Canon EOS-1D X Mark II. But given the difference in price ($ 1,000 vs $ 7,500), in weight (6.63 ounces (188 grams) vs 1.5 kg) and dimensions, this advantage looks very doubtful.

No less convincing comparison of the iPhone 11 Pro vs. The Canon 7D Mk II in portrait mode was introduced on YouTube by Computer Clan.

Of course, many people ask a very obvious question: how does a relatively inexpensive smartphone with miniature optics ensure the quality of an expensive professional DSLR camera.

Sensor Dimensions + AI algorithms

Of course, the sizes of modern matrices in smartphone cameras and their specs are radically different from prototypes. For example, Samsung introduced the unprecedented 108-megapixel ISOCELL Bright HMX sensor, which is already used in the Samsung Galaxy S20 5G, Xiaomi Mi Note 10 and Mi 10 Pro.

Video demonstrates this innovative sensor.

But, of course, this example is an exception to the rule. Usually, the size of the sensors in DSLR cameras is about 20-30 times larger compared to phones.

However, smartphones provide excellent quality even at night in low ambient light.

However, this reality is consistent with the general trend in the consumer electronics segment. In fact, the device’s platform has ceased to be a major factor in image quality.

Today, the lens and sensor provide only the first part of the task, collecting image data and transforming them into electrical impulses. Digital image processing algorithms do the second part of the job. Probably, the contribution of these components can be expressed in percent and numbers. But the growing influence of the processing algorithms is beyond doubt. Of course, they require tremendous performance processor, which solves these tasks. But the latest 8-core Qualcomm Snapdragon 855+, HiSilicon Kirin 990, Exynos 9611, etc cope with them very effectively. Moreover, the performance of even mid-range SoC is excessive in many cases.

In fact, a modern smartphone collects an image from individual pixels, calculating and adjusting the value of each of them. In addition, AI of phone actively uses pixel interpolation, generating them if necessary.

AI algorithms

Some algorithms well illustrate the capabilities of modern smartphones. Probably, many saw photos with the popular bokeh effect.

The corresponding algorithm in phone works as follows. First, the camera must separate the objects in the frame to snap them to the foreground and background. But for this, the algorithm must estimate the distances to each object in the frame.

Two cameras, for example, on the back of the phone, use the traditional principle of determining depth. In this case, each of the two cameras takes one picture. Accordingly, the angle of shots will be slightly different.

Then the images are superimposed on each other, visualizing the displacement of objects in the frame.

This method uses the parallax effect. As known, parallax is a change in the apparent position of the object relative to the distant background, depending on the viewing angle. For small angles, the distance to the object is equal to the ratio of the distance between the view points (centers of cameras lenses) to the displacement angle (d = l / α).

But parallax does not work for selfies due to the lack of a second camera. In this case, the camera can use semantic segmentation, dividing the photo into two layers with pixels of the main object and background. Then the algorithm will defocus the background layer, providing a bokeh effect. Unlike a DSLR camera, it does not provide a smooth change in the depth of field. But this quality is more than enough for a selfie.

HDR and auto semantic segmentation

Of course, these are not the most difficult tasks. High-quality bokeh effect with a single main camera requires more sophisticated technologies, but better illustrates the possibilities of digital image processing.

Firstly, the camera forms an image with a high dynamic range (HDR), providing high detail in dark and light parts of the frame. For this, the smartphone takes a series of shots, selects the best fragments from each and forms one picture of the highest quality.

This technology is very similar to the creation of Frankenstein, but from the best legs and arms.

Further, the camera begins to use machine learning and neural networks. It automatically detects all objects in the foreground and background using semantic segmentation. For this purpose, the neural network uses a huge database. For example, to identify people, it uses various photos with their images in various poses, clothes, with different hairstyles, etc. As a result, the neural network will create something like this silhouette with a cup in his hands.

Next, the camera will defocus the black area, providing a bokeh effect.

PDAF with Dual Pixel technology

Further quality improvement requires the formation of a depth map for smooth blurring the background. To solve this problem, the camera uses PDAF (Phase Detection Autofocus) with Dual Pixel technology.

Simplified, it works as follows. Each pixel of the sensor includes two independent photodiodes.

In turn, a tiny lens is placed above each pixel, which divides the light flux into two parts.

As a result, one photodiode receives the light flux from the left half of the lens, and the other from the right.

In fact, this technology provides a parallax effect, allowing the algorithm to calculate the stereo effect and build a depth map. As a result, even one smartphone camera provides a bokeh effect with smooth change in the depth of field.

Conclusion

Of course, image processing algorithms for creating bokeh effect make up only a small part of the capabilities of modern smartphones. But they well illustrate the pronounced trend of rapid development of the AI-technologies in this segment.

As a result, this causes a philosophical discussion about naturalness, realism, etc. Probably, many will prefer imperfect realism instead of the splendor of an artificial colorful image.

In fact, something similar happens on television with Filmmaker Mode. As known, Tom Cruise urged viewers to abandon some playback optimizations in TVs to preserve the original intent of the filmarkers, including the directors, cameramans, etc. In fact, TV in this mode will play movies with the original color reproduction, aspect ratio and frame rate. Today LG, Panasonic and Vizio, already supported the creation of this mode, implementing its support in their models.

In any case, photographers with professional DSLR cameras will continue to delight fans of large colorful photographs without the intervention of AI technologies. But the segment of budget and mid-budget cameras is hopelessly losing to smartphones.