Muse, an AI agent developed by Meta, has helped me manage my dental insurance, book dinner tables, and create my podcast. But it also required giving him my most private data and putting my trust in him, writes Eli Tan.
Secure calls
Last week I answered a call from an unknown number. A voice said: “I have contacted Anthem, a dental insurance company, and I am transferring the call to you now.” The call was coming from an artificial intelligence “agent” created by Meta, the company that owns the Facebook and Instagram platforms. Ten minutes before that moment, the AI agent had called the insurance company, provided it with my membership number, answered a security question (to verify my identity), and waited on the line before transferring the call to me to speak to the relevant employee... just as I had asked him to do.
I was amazed at how easy this agent made handling the insurance process. This was one of many tasks she completed with the help of Muse, which Meta revealed this month.
The first consumer-oriented AI agent
Muse is the first consumer AI agent launched by a major technology company and made available to the public. The technology industry sees these agent tools as the “next frontier” in this field, as they have capabilities that far exceed those of traditional “chatbots”: agents can browse the Internet, log into websites, fill out spreadsheets, and make phone calls on behalf of users, just to name a few.
The test requires submitting personal and banking data
To test Muse, I used it continuously over the course of two weeks, handing over many aspects of my life to it. I had to trust him with my most private data, including linking him to my bank accounts and email. Then I assigned him dozens of daily tasks, like ordering groceries, scheduling my meetings, and forming my fantasy football teams.
After two weeks, I found Muse to be the most useful AI application in my experience. This was clearly demonstrated in a specific situation that occurred after I linked my credit cards to the app and asked it to track my expenses and record them in Google Sheets. I watched the program open tabs with hundreds of rows of data within minutes, then identify two duplicate subscriptions and cancel them for me. Muse then used my email to request a refund for a free trial subscription to a newspaper that had expired, saving me $44.99.
Privacy problem
I'm not entirely sure whether Muse and other AI agents will catch on; These tools require a level of trust with which many may not feel comfortable, especially when dealing with a company like Meta, which has faced privacy issues. Some sites also impose restrictions on these agents within their terms of service. For example, Amazon recently blocked Muse from accessing it.
Melissa Vadasseril, a senior manager at Acoustic, a marketing platform that relies on intelligent agent technologies, says the emergence of Muse reminded her of the 2010 launch of Apple's digital personal assistant, Siri, an innovation that many were late to adopt. “People didn't realize what they could do with it,” she added. As for “Muse,” she says: “I think it is still a bit early for the market, but I am very curious to see how things will develop.”
My experience is for two weeks
My two-week experience with Muse began by downloading the free app and logging in using my Instagram and Facebook accounts (note that there are restrictions on free use, as users must pay $20 or $100 per month to get additional features).
While I was talking to the intelligent agent - as if I were talking to a normal chatbot - he suggested that I name him “Wren,” and he created an avatar of an attractive little brown bird. Our conversation then continued as follows:
* Eli: Hello, Ren! You are my new assistant.
- Ren: Hello! Nice to meet you. What can I do for you today?
I quickly granted Rain access to my bank accounts and apps like Google Docs, Gmail, Ticketmaster, OpenTable, and Google Sheets, all of which have partnerships with Muse. From here, the experiments began.
I had a few goals: I wanted to see if Muse could give me that “personal empowerment” that Meta CEO Mark Zuckerberg always talked about. I also wanted to test MUSE's capabilities to the fullest to see what AI agents can actually achieve, and where they might fall short. I also wondered, given current concerns about powerful AI systems, whether my agent might rebel and get out of control to communicate with other agents, potentially compromising the privacy of my data.
I soon discovered that Ren was able to fill out electronic forms very efficiently, a tedious task when using the Internet. I could take a photo of a traffic violation, and Rain would pay it for me, asking for my permission before completing the payment. While the agent was performing his tasks, I was watching his movements through a small browser window, as he was navigating between websites just as humans do, using the mouse and cursor.
Make phone calls
Ren can also make phone calls using different characters, as she has a male voice called “Brett” and a female voice called “Hailey.” Currently, access to the calling feature is limited to a limited number of Muse users, but I have found it useful for dealing with customer services, such as communicating about my dental insurance.
Ren had difficulty with some tasks, particularly logging into third-party websites. He was unable to order tickets to San Francisco Giants games without my help, and he had difficulty logging into my rewards account at AMC theaters.
Listen to the “Artificial” podcast
Other features worked well, but weren't very useful. For example, every morning on my way to work, I listened to a 10-minute podcast that Ren prepared for me called “Morning Brief,” voiced by two AI presenters, “Maya” and “Theo.”
Although the Morning Brief was informative and tailored to my interests, the presenters' performances were bland and lacked the charm and charm of my favorite podcasts, hosted by humans. I tried to liven it up, but Rin's options were usually limited.
* Ellie: Can you have Maya and Theo sing the intro to make it a little livelier?
- Ren: It's done. But there's a little caveat: Maya and Theo have no real singing skills, so this will sound more like a rhythmic chant than a singing performance. And frankly, that's the secret of his appeal.
Spending money
As a final task, I asked Ren to suggest a good way to spend the $44.99 he saved me. At this point, Ren was very familiar with the details of my life, so I thought he might suggest something charitable, like donating money to a charity, or buying flowers for my friend who was recovering from surgery. Instead, the tool suggested a better idea: buy bowls from Heath Ceramics, a local ceramics manufacturer with a loyal fan base on Facebook Marketplace, one of the platforms I enjoy browsing the most.
Rain told me it could message sellers and negotiate on my behalf, and that it was already linked to my Facebook account. I set some standards for the smart agent: buy goods at only half the retail price, and always be sure to negotiate to get the lowest price possible.
The “tool” comprehensively scanned the platform and began messaging sellers about their offerings, always starting the conversation with a low offer that was $20 less than the advertised price. After receiving a counteroffer from the seller, she would come back to me to determine our next strategy.
Heath collectors in the San Francisco Bay Area are some of the toughest negotiators I've ever dealt with, but the tool has proven its worth, preferring a soft, calm approach to dialogue. The tool once convinced me not to submit a lower bid on a set of shiny bowls because it picked up a tone of frustration in the seller's speech.
ElevenLabs also launched a new generation of artificial intelligence voice generation models, namely “Eleven v4” and “Eleven v4 Turbo,” with major improvements that focus on making artificial voices more natural and expressive, and understanding the way text is delivered, not just pronouncing its words.
The two new models come as part of the company's efforts to bring conversations generated by artificial intelligence closer to human performance, by understanding context, tone, human rhythm, and feelings, in addition to expanding and supporting languages and dialects.
A voice that understands delivery
“Eleven 4” represents the company’s new main model, and it was designed to deal with the text as an integrated audio performance, as it can take into account the context of the conversation, and change the tone, rhythm, and pronunciation according to the content of the text.
This is especially evident in dialogues that include more than one speaker, as the model is able to preserve the personality of each voice, and take the context of the dialogue precisely into account, instead of reading each sentence separately. The company also improved the model's ability to maintain consistent audio in long-form content, which is important when producing audiobooks, podcasts, and clips that include long texts.
Expanded support to over 90 languages
In the new generation, Eleven Labs has expanded support to include more than 90 languages, while improving the ability to preserve the identity of the voice and the way it performs when using different languages. Improvement is also evident when using the Arabic language and local dialects. By testing the model in the Saudi dialect, it is possible to notice a more natural performance in pronouncing phrases and vocabulary used in daily speech, especially when choosing an appropriate voice and writing the text in the way the user actually speaks.
For example, it is possible to create a family dialogue in the Saudi dialect between several people, so that the model presents it in a manner closer to natural speech, taking into account the differences in voices, tones, and manner of speaking between the characters.
The quality of the result depends at the same time on the choice of voice and the way the text is written, so the accent and tone may differ from one voice to another.
Instructions for changing the method of delivery
One of the most notable additions in the new generation is the ability to enter “performance instructions” within the text itself, allowing the user to specify the way he wants the sentence to be said. It is possible, for example, to direct the voice to whisper, laugh, or speak enthusiastically or sadly, and add pauses, reactions, and sound effects in specific places in the text. These instructions are written in square brackets before the required part, so that the model attempts to execute them while generating the sound. Thus, the user not only writes “what does the voice say,” but can also give instructions on “how to say it,” an experience that makes writing a text closer to preparing a script that contains directions for a voice actor.
Audio reproduction from a short recording
Eleven Labs has also developed voice cloning technologies. The “Instant Voice Cloning” feature allows the creation of a copy of the user’s voice based on a short recording that may only be about 10 seconds long.
The new generation also supports “professional sound reproduction,” intended to obtain a more accurate and stable copy of the speaker’s voice, which can be used in long-form content and uses that require maintaining a unified vocal identity.
“Eleven 4 Turbo” for quick conversations
As for “Eleven 4 Turbo,” it focuses on applications that require a quick voice response, such as assistants, voice agents, and customer services that rely on artificial intelligence. The company says that the model is capable of generating sound at a speed of about 100 milliseconds, which makes conversations faster and more natural, and reduces the time waiting for a response.
Wider uses of AI voice
The new improvements open up broader uses for AI voice, including video production, podcasts, and audiobooks, as well as voice assistants, customer services, gaming, and educational content.
It also allows you to control tone, emotions, delivery, and produce sounds in different languages and dialects. These developments indicate that voice generation techniques have moved from simply pronouncing words to understanding context, dialect, emotions, and manner of speech. As for the Arabic language and the Saudi dialect in particular, the experience shows clear progress towards more natural sounds, closer to the daily way of speaking.
