[ITmedia News] AI x naked eye 3D display? Converting 2D images to 3D on the spot and following the viewer's eyes - confirmed on site

When I stood in front of the display, a 3D image of osechi cuisine appeared before my eyes. No 3D glasses required. If you move your body from side to side, the angle at which you can see the food will also change. This is because the AI ​​is tracking the viewer's eye position through the camera. What's more, the original 2D video is converted into 3D video in real time by AI.

On November 12th, Japanese restaurant chain operator Tonden (Eniwa City, Hokkaido), Softbank, and SB Creative unveiled to the press at the Tonden Shirahata store (Saitama City) a demonstration experiment of digital signage that allows users to experience 3D images with the naked eye. Will ``Floating Osechi'' be enough to stop customers coming to the store? Compiled based on local experience.

This demonstration is an attempt to utilize SoftBank's technology, which uses AI to convert existing 2D videos into 3D in real time, to digital signage for restaurants. This is the first time the company has demonstrated this 3D display in Japan.

The three companies will test whether naked-eye 3D images can attract customers' attention better than regular 2D images, increasing their interest in food and their desire to purchase food. We will also test whether existing 2D videos can be converted to 3D and easily distributed and operated without the need to create special materials.

I tried 3D Osechi

When the reporter stood in front of the display, he could clearly see the osechi ryori protruding from the screen. When you move your body from side to side, the angle at which you can see the food changes accordingly. It's hard to tell in the photo, but we were able to confirm that it looks three-dimensional even without special glasses. How does it work? At the briefing session, Yuji Kumon of SoftBank's Service Planning Division introduced three technologies.

The first is a lenticular lens that enables 3D viewing with the naked eye. Lined up with long, slender, semi-cylindrical lenses, it uses light refraction to deliver different images to the left and right eyes. This allows images to be viewed in three dimensions without the need for special glasses.

The second is a ``depth estimation AI model'' that converts regular 2D images into 3D images. AI estimates the depth of objects in the video and generates images for the right and left eyes in real time. For example, if a video shows Mt. Fuji in the background of cherry blossoms, the AI ​​will instantly read the context and convert it so that the cherry blossoms in the foreground stand out. Previously, it was necessary to create videos specifically for 3D displays, but with this technology, existing 2D videos can be used as is.

“The main point is that by using depth estimation AI, we no longer have to create images for 3D display.” (Kumon)

The third feature is ``AI eye tracking,'' which uses AI to detect the position of the viewer's eyes from images from a camera mounted on the display. The image is adjusted according to the position of the viewer's viewpoint to display clearer 3D images. Even if you are slightly off the front, stereoscopic vision is possible as long as the eye tracking functions.

However, there are some points that caught my attention. AI eye tracking can detect one person's viewpoint per display. Therefore, when viewed by multiple people, it does not appear to everyone as 3D. In fact, when I watched other journalists' experiences from a distance, the screen appeared distorted, like a mirage, depending on the angle. In restaurants where many customers come and go, it seems likely that the issue will be how many people can receive 3D images.

In the future, SB Creative plans to incorporate this 3D conversion technology into its cloud-based digital signage distribution system ``BeyondCAST.'' BeyondCAST is the company's service that allows you to collectively distribute and manage videos, etc. to displays at multiple locations. If realized, companies will be able to stream 3D images on 3D-compatible displays simply by registering their 2D videos.

What is the status of the demonstration?

In this demonstration, we will confirm the effectiveness through surveys and customer reactions. The survey was conducted by placing a board next to the display and asking participants to answer three questions with two choices. Have the appropriate person put a sticker on it.

As of 3 p.m. on the 12th, all of the stickers had answered positively to the questions, ``Did you see the New Year's dishes pop out in 3D?'' and ``If you saw a 3D image in a store, would you be tempted to look at it?'' On the other hand, when asked, ``Did your interest in osechi increase after watching the 3D video?'' the majority answered ``It increased,'' but there were also a few votes for ``No change.''

SoftBank and SB Creative plan to consider commercialization based on the final results.