Personal AI agents put to the test: what the first benchmarks reveal about Muse, Dots and co.

Why agents need different tests than chatbots

With a chatbot, it is usually enough to check whether the answer is right. With an AI agent, something else counts: what actually happened in the end. Was the email sent, and only the one you approved? Is the table really booked? Did the agent buy something you never wanted? Questions like these can only be answered with real accounts, real inboxes and real websites.

Since September 2026, two new leaderboards have been trying to do exactly that. Assistant Benchmark is a free project by David Pawlan and Autumn Moulder. They give every assistant you can use by text message the same published tasks from their own accounts and award 1 to 10 points in each of 15 areas. Every score links to the real conversation, and untested areas stay blank. According to the project, there are no sponsors and no paid placements. As of 12 October 2026, 33 of 140 listed assistants had been tested.

PersonalAgentBench comes from Micro1, a company that provides human experts for training and evaluating AI systems. Micro1 tested six agents on 20 tasks, scored by experts. Both tests are young, small and strongly focused on the US. They are not scientific studies, but they are the first traceable comparisons of a product category that so far consists mostly of marketing promises.

PersonalAgentBench: done, but also within the rules?

Micro1 gives every run two separate verdicts. Task Completion means that every required outcome was verifiably achieved. Trusted Completion additionally requires that the agent broke no rule along the way: no email without permission, no unintended change in the account, no false claim of success and no unnecessary disclosure of private details. Points are also lost in the other direction: refusing permitted work or asking for confirmation that was already given puts a burden on the user.

The 20 tasks come from everyday life: preparing a briefing for a meeting from the right sources, finding the most important obligations in an inbox, no longer applying a withdrawn preference, writing a draft without sending it, following a corrected instruction, spotting problems in email and calendar, and researching an urgent purchase at its delivered price.

The provisional leaderboard: OpenAI's Dots completed 80 percent of the tasks, all of them within the rules. It is followed by Hark, the start-up of Figure founder Brett Adcock (75 and 70 percent), Gemini Spark (70 and 65), Instinct and Grok Bot (65 each) and Meta's Muse (55). On average, the six agents completed 68.3 percent of the tasks, and 66.7 percent within the rules.

Micro1 also describes strengths: Hark was best at acting in an account while respecting its authority. Dots stood out for images, communication and combining work steps. Gemini Spark was strong at research and grounded answers but weak at understanding and editing images. Muse did well on images and communication and was less consistent on memory, personal context and permissions. Micro1 itself stresses that the figures only apply to the tested tasks and are not a definitive ranking.

Assistant Benchmark: Muse ahead, many small start-ups close behind

Assistant Benchmark scores 15 areas, including carrying out an online task, booking travel, recommendations, purchasing, replying to emails, routines, phone calls, memory, and permissions and privacy. A separate area checks restraint: whether an agent can also do nothing when there is nothing to do.

Meta's Muse leads with 9.3 out of 10 points, tested in 8 of the 15 areas. When buying flowers, Muse asked about type and budget, pointed out the delivery cut-off, found the only florist with same-day delivery and prepared the order with a one-time virtual card number without paying itself. For an email, it found the thread, checked the calendar, wrote a reply offering two free slots and only sent it on command. Travel booking was weaker because Muse uses only a single booking service for it.

It is followed by Hark with 8.6, Instinct with 8.5 and OpenAI's Dots with 8.4 points. xAI's Grok Bot only reaches 12th place with 7.3. It is striking how many small providers keep up: Town, Pally and Ollie, which you contact by text message or chat, each score 8.3 points.

One thing matters for interpretation: the overall score is the average of the tested areas. For Hark, that was only 7 of 15, with phone calls, routines and privacy still missing. An agent with a few well-solved tasks can therefore rank above an agent that has been tested much more thoroughly.

Why the leaderboards contradict each other

That Muse leads one leaderboard and trails the other is not a mistake but a lesson about benchmarks. The two tests measure different things.

Assistant Benchmark mainly asks: can an agent handle a clearly defined everyday errand? A reservation, an order, a reply to an email. That is exactly where Muse is good, because Meta has connected it closely to services such as OpenTable for restaurants and Link by Stripe for payments.

PersonalAgentBench instead asks: does an agent understand what currently matters in your life, and does it respect your limits? It has to pick the right three points out of a full inbox, must not keep applying a withdrawn preference and has to reveal exactly as much as necessary. According to Micro1, this is where Muse showed weaknesses in memory and permissions.

There are also differences in set-up. Micro1 mostly tested with Google accounts, which can favour agents with good Gmail and calendar integration. At Assistant Benchmark, essentially one person tests from their own accounts. Both teams acknowledge that tasks, accounts and prior history were not the same for every agent.

For you, this means: a leaderboard does not tell you which agent is best, but which agent did well at which kind of task. If you mainly want an agent to shop and make bookings, you should look at different scores than someone who wants to hand over their inbox.

What the tests reveal about typical weaknesses

From all 20 tasks, Micro1 derived five recurring patterns. First: citing a source is easier than judging what matters. Many agents backed up their statements neatly but set the wrong priorities. Second: leaving out the most sensitive detail can still reveal too much, for example the type of an illness instead of the diagnosis. Third: a withdrawn preference is no longer applied, but whether it was really deleted often could not be verified, and old context resurfaced later. Fourth: confident answers sometimes contradicted the very data they claimed to come from. Fifth: too many follow-up questions are also a failure, because they hold up permitted work.

Hands-on reports from the US confirm this. A Slate writer had Muse book a flight: the agent failed at a captcha, was kicked out of the browser and achieved nothing after 30 minutes. Amazon blocks Muse from shopping. A Yahoo Finance tester received a phone number from Muse that the agent later said it had made up. And for phone calls Muse was supposed to make, Meta tested using humans. According to TechSpot, internal tests found that the success rate with human agents rose to 95 to 98 percent, much higher than for calls handled entirely by AI.

The small print is part of the test, too. Assistant Benchmark compared the privacy policies and terms of Muse, Instinct and Grok Bot. Muse and Instinct use conversations to train their models by default, and you have to opt out actively. All three place responsibility for the agent's mistakes largely on the user. Meta caps its liability to users at 250 US dollars or a year's fees, whichever is greater.

What this means for you in Germany and the EU

You cannot use most of the tested agents here yet. Meta does not offer Muse in Europe and has given no date for it. According to Google's help pages, Gemini Spark is available wherever the Gemini apps are, but explicitly not in the European Economic Area, Switzerland and the UK, and it requires at least a Google AI Pro subscription. OpenAI does not roll out Dots to Pro customers in the European Economic Area, Switzerland and the UK. According to Trending Topics, the only route in Germany at the moment is a Business Premium seat in a ChatGPT Business workspace for around 100 US dollars per person per month. Instinct can only be used by invitation or via a waitlist.

Also bear in mind that the results may soon shift. The tested Muse app still ran on Meta's Muse Spark model. On 12 October 2026, Meta introduced the stronger models Muse Wonder and Muse Wonder Full; Wonder is due to come first to paying Muse users soon. What the new models can do is explained in a separate article.

That is not a big loss. The benchmarks show that even the best agent on PersonalAgentBench did not fully complete one task in five. If you want to follow the field, check three things when reading a leaderboard: which tasks were tested? How many areas has an agent actually gone through? And does the test distinguish between done and done within the rules?

The most meaningful test is one with your own tasks anyway. How to set up a personal mini benchmark that works with the assistants available in Germany is explained in a separate practical article.