Published on the 02/04/2025 | Written by Heather Wright

CSIRO trial highlights gap between marketing and reality…

AI-powered ‘assistants’ might hold benefits for organisations going forward, but for CSIRO at least, those benefits are unlikely to come via Microsoft’s Copilot offering, with a trial finding that while there are some benefits in basic tasks, the technology is failing to live up to its marketing in many areas and resulting in a ‘productivity paradox’.

The trial and an accompanying qualitative survey, found while Copilot could provide measurable improvements in some – more basic – areas such as meeting summarisation, email drafting and basic information retrieval, if ‘fell short’ in areas requiring domain-specific knowledge, creative problem-solving and nuanced decision-making.

“AI copilots currently act as productivity enhancers, rather than productivity transformers.”

Users also reported a ‘productivity paradox’ where time saved through automation was often offset by the need for extensive verification and correction of AI-generated outputs, while users also had ethical concerns, particularly around data privacy, bias and transparency.

The results of the trial at the science and industry research organisation echoes the results of an Australian Treasury trial, reported earlier this year, which saw the tool failing to meet staff expectations and not suitable for more complex tasks, but beneficial for some basic admin.

The qualitative interview study into user perception included interviews with 27 participants in the six month M365 Copilot trial conducted last year and found that despite initial enthusiasm for the tool, post-trial experiences were mixed.

“AI copilots are marketed as transformative technologies, but their real-world value depends on several socio-technical factors, including integration with existing workflows, user trust and the extent to which they align with professional demands.”

A broader quantitative survey of 300 CSIRO participants also found mixed outcomes, with many identifying limitations in Copilot’s advanced functionalities and integration.

The issue of ROI for Copilot has been an ongoing bugbear – and something that is proving elusive for many.

Gartner’s The State of Microsoft 365 Copilot report last year highlighted the issue with only three percent of organisations saying Copilot delivers significant ROI, though it could save up to 14 minutes a day for users, largely through automation of routine tasks.

Gartner has suggested that ROI gap, along with concerns about security and enablement efforts, has had a direct impact on the slowness of organisations moving from piloting Copilot to actually using it in production. Just 16 percent of pilots moved into production.

One of the key findings in the CSIRO trial was that ‘productivity paradox’, where users found themselves needing to double-check and manually refine Copilot’s suggestions, undermining the time savings AI is supposed to provide.

“Copilot’s impact was often offset by usability challenges, lack of domain-specific knowledge and the necessity for human oversight. These factors resulted in a shift of effort, rather than a clear reduction in workload, reinforcing the notion that AI copilots currently act as productivity enhancers, rather than productivity transformers.”

At the end of the trial, just six out of the 27 survey participants felt their expectations for Copilot had been met, despite earlier high hopes that it would streamline tasks, boost productivity and ‘possibly match ChatGPT’s capabilities’.

The remaining participants were divided between those who found Copilot ‘somewhat useful’ and those who felt significantly let down – one individual going so far as to dub it ‘kind of useless’.

Their views on the productivity gains also varied – though saving time with meeting minutes seems to be something of a winner.

“While M365 Copilot is perceived as valuable for routine and structured tasks, it falls short in driving significant innovation or creative solutions.”

Many users expressed scepticism about its capabilities in truly novel problem-solving, pointing out limitations in the quality and completeness of outputs, though in areas like coding efficiency, basic information retrieval and drafting communications, its assistance was viewed as somewhat helpful.

Handling complex datasets in Excel, producing complete final drafts for PowerPoint and Word and generating emails with the right balance of formality and conciseness were all fails, with many concluding ‘Copilot can’t do it’.

Issues around local cultural context was also noted, along with bias towards particular publishers of journals, while a default to American English spelling proved frustrating to some and highlights broader limitations in localisation which could impact inclusivity and user experience the report notes.

Stereotypical biases were also seen, including one person being told ‘women are more caring and empathetic’ when prompted about women’s roles in AI ethics think tanks.

“As organisations weigh the return on investment for AI copilots, they must consider whether these tools genuinely enhance productivity or simply shift cognitive effort elsewhere,” the study says.

The report does note that CSIRO is not a corporate or administrative setting, where AI -assisted tools may primarily support routine or transactional tasks, and the integration of AI solutions at CSIRO, with its scientific research focus, presented distinct challenges and opportunities.

While both CSIRO and Treasury’s trials have seen underwhelming benefits, New Zealand telco One NZ is more enamoured. It announced last month that it had seen an average 39-minute time saving per day for employees through using Copilot and had rolled out the tool across it’s office-based workforce.

It noted, however, that there could be some apprehension in adopting AI tools. Its solution? Upskilling and training a company-wide event to give teams hands-on experience with Copilot and other agentic AI tools it is rolling out.

ACC (Accident Compensation Corporation) also claimed benefits from its trial of Copilot last year. In that case the AI operated on a fixed dataset and did not tap into the main client systems, collect information or learn from queries or information used with it. It was used to draft content, summarise meetings and research and summarise ACC information.

But the CSIRO report suggests that even if Copilot isn’t the right offering, the organisation and its users are eyeing up a future with AI agents.

“The rise of AGI  [artificial general intelligence] and AI agents means that the current generation of copilots, including M365 Copilot, will soon be eclipsed by more advanced, autonomous AI assistants.

“Organisations must take a forward-thinking approach, ensuring that AI investments today align with the rapidly evolving technological landscape.”

Strategic, well-governed decisions around AI adoption would enable organisations to avoid the pitfalls of ‘short-term efficiency hype’ and build resilient AI-enabled workforces ready for the next wave of intelligent automation.