Sony introduces first combined image sensor and AI chip – TechCrunch

Sony has developed an interesting new hybrid technology: An image sensor with hardware-integrated AI processing system, making it a single integrated system. The benefits and applications of this are potentially enormous as imagery and code continue to merge.

The idea is quite simple in concept. You take a traditional CMOS image sensor like you’ll find in any phone or camera today and pile it onto a logic chip that’s designed not only to extract pixels from the sensor, but also to make run a machine learning model that extracts information from these pixels. .

The result is a unique electronic assembly that can perform many interesting processes on a photo before that photo is ever sent somewhere else, such as a main logic board, GPU, or the cloud.

To be clear, image sensors already have companion processors that do the usual work of sorting pixels, compressing them into JPEGs, and so on. But they are very focused on doing a handful of common tasks very quickly.

The Sony chip, as the company explains, is capable of more sophisticated processes and outputs. For example, if the exposure is of a dog in a field, the chip could immediately scan it for objects and, instead of sending the full image, just report “dog”, “grass ” and everything she recognizes.

He could also make mostly improvised edits, such as cropping everything in the photo except the parts he recognizes and has been asked to point out – only the flowers, but never the stems, for example.

The advantage of such a system is that it can remove all kinds of unnecessary or unwanted data before that data enters the storage or processing pipeline of the main device. This means that less CPU power is used, on the one hand, but it can also be safer and more secure.

Cameras in public places could preemptively blur faces or license plates. Smart home devices could recognize individuals without ever saving or sending image data. Multiple exposures could be merged to form heat or frequency maps of the camera’s field of view.

You might expect higher power consumption or latency from a chip with built-in AI processes, but companies like Xnor (recently acquired by Apple) have shown that such tasks can be done very quickly. and at extremely low cost.

While more complex processing would always be the responsibility of larger and more powerful chips, this type of first pass is capable of producing a wide variety of valuable data and, properly designed, could prove more robust against attacks or attacks. abuse.

At present, Sony’s “intelligent vision sensor” is still only a prototype, available to order for testing, but not for production. But as Sony is one of the world’s leading image sensor suppliers, this is likely to find its way into a number of devices in one form or another.

Michael C. Garrison