You’re almost better off flipping a coin to get a correct response instead of relying on Cortana

Interestingly, Cortana did attempt to answer the most questions which could partially explain the low accuracy score. Perficient explains this as the percentage of questions where “the personal assistant thinks it understands the question and makes an overt effort to provide a response. This does not include results where the response was ‘I’m still learning’ or ‘Sorry, I don’t know that’ or responses where an answer was attempted, but the query was heard incorrectly.”

This is the third year that Perficient has run a similar test and looking at the results of the last three years one might get the impression that digital assistants have been getting dumber. Outside of Cortana, which fully and correctly answered over 90% of the queries thrown at it last year, the digital helpers taking part in all three contests have seen their score decline each year. What hasn’t changed over the three years are the results at the top. The smartphone version of Google Assistant consistently beats out the other contenders.

During this year’s testing, Siri had the most incorrect answers amounting to nearly three times the number of incorrect responses from the smartphone version of Google Assistant. Accord to Perficient, about a third of the wrong answers from Siri and Alexa, came from questions that it considered to be “poorly structured.” An example would be “What movies do The Rushmore, New York appear in?” Additionally, most incorrect responses were so obviously wrong that a user would know not to believe the answer.

If you’re curious about which of the personal assistants has the best sense of humor, the test shows that Siri and Alexa are neck and neck at the top. As an example, when Siri was asked “Where is Waldo?,” Apple’s digital helper responded, “He’s well hidden. I can’t see him.” The most serious of the assistants was Cortana.

Lastly, Siri and Alexa used “snippets” the least. These are responses sourced from a third-party and are generally attributed to them. The overall trend among virtual assistants is lower but Google Assistant (on every device), by far, delivers such answers the most.