[ITmedia Business Online] AI function that is “better at PC work than humans” released OpenAI “Dots” will change the future of devices

An engineer from OpenAI (OpenAI) appeared on the popular podcast "Latent Space" and explained the computer use functions of the new personal AI "Dots". What was shown was that AI is beginning to use PCs better than humans.

Evolution of PC operation AI that is “faster than humans”

Computer use is a technology in which an AI agent operates a PC on behalf of a human. Open a website, press a button, fill out a form, and find the information you need. AI will now perform tasks that have traditionally been performed by humans using a mouse or keyboard.

For computer use, there is also a method of letting AI operate the user's own PC. On the other hand, with Dots, each agent is given a Linux virtual computer in the cloud. The AI ​​can operate not only the browser but also desktop apps.

Until now, computer use has had problems such as failures during operations or slow processing times. However, Ari Weinstein, head of product engineering for computer use agents at OpenAI, says performance has improved significantly in recent months.

"In many cases, we can probably complete tasks faster than the average human," he said of Dots' current computer use. He gave the example of a meal delivery service where customers can specify even the amount of ingredients. The task of setting up detailed order details for multiple meals would have taken about two hours if done by hand, but was completed in just 15 minutes using a computer. The next goal is to operate the software as quickly as, or faster than, experienced computer users.

Two factors in which AI has an “advantage” over humans

However, speed is not the only advantage that AI has over humans.

Where a human would proceed in sequence, ``click, wait, look at the screen, click the next'', Dots writes JavaScript as necessary and executes a series of operations all at once. Furthermore, OpenAI's new API introduces a mechanism to proceed with processing without waiting for one tool to finish processing another.

Furthermore, Dots have another strength that humans do not have.

With a system called Appshot, Dots receives more than just screen images. The internal structure of the app or web page is also passed along.

For example, when a human looks at a website, they cannot see the link destination URL set behind the text displayed on the screen. However, Dots can even pass that information. You can also get event names that are only partially displayed on the calendar screen.