Most articles about AI governance start with a framework. This one starts with the failures, because the failures are no longer hypothetical. Professional firms — law firms, accountancy practices, consultancies — have now been publicly embarrassed by AI in enough distinct ways that we can name the patterns, and the interesting thing about those patterns is that each one points to a specific, fairly ordinary governance control that would have prevented it.

Having spent ten years in operations in regulated financial services, and now building AI systems that run under governance every day, I find the professional-services picture both familiar and instructive. The regulators have not written a new rulebook. The courts have not invented new duties. What has changed is the ease with which a competent professional can produce confident, polished, wrong work — and the speed at which that work reaches a client, a court, or the front page.

The four ways AI has embarrassed professional firms in public

1. Fabricated authority

In June 2025 the High Court handed down judgment in the joined cases of Ayinde v London Borough of Haringey and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin). In the first, grounds for judicial review contained five citations to cases that did not exist, together with a misstatement of a statutory provision; in the second, a witness statement referred to authorities that were fictitious or did not say what they were claimed to say. The Divisional Court, exercising its Hamid jurisdiction over lawyers’ duties to the court, was blunt: putting fake cases in a pleading is “wholly improper”, and describing them afterwards as “minor citation errors” made matters worse[1].

These were not isolated lapses. A public database maintained by researcher Damien Charlotin has tracked court decisions worldwide involving AI-fabricated citations or content; by mid-2026 it had recorded well over a thousand such cases, up from roughly two hundred a year earlier. The failure mode is well understood — large language models generate plausible text, and a plausible-looking citation is exactly what they produce when no real one exists — yet filings containing invented authority keep arriving in front of judges.

2. Over-reliance without review

Accountancy and consulting have their own headline incident. In 2025, Deloitte Australia agreed to partially refund the Australian government’s roughly AU$440,000 fee for an assurance review of a welfare compliance system after the published report was found to contain fabricated references and a quotation wrongly attributed to a federal court judge. A revised version was reissued with a disclosure that generative AI had been used in its preparation. The report had been through a professional firm’s production process and delivered to a government client — which is precisely the point. The failure was not that AI was used; it was that AI-generated content reached the deliverable without a review rigorous enough to catch invented sources.

3. Confidentiality leakage

The third failure mode is quieter, because its victims rarely find out. Consumer-grade AI tools may use what you type into them for further training, and the Bar Council’s guidance for barristers spells out the consequence: material entered into such systems could resurface in outputs to other users, which is “plainly problematic” where the input is confidential or subject to legal professional privilege[2]. The Law Society’s guidance for solicitors is similarly direct — it is generally advisable not to feed confidential information into generative AI tools where you lack direct control and oversight over how the tool is developed and deployed[3]. For an accountant holding client financial data, or a consultant holding commercially sensitive board papers, the analysis is the same even if the word “privilege” is not.

4. Shadow AI

The fourth failure mode is what makes the other three hard to control: AI use the firm does not know about. UpGuard’s research on shadow AI found that more than 80% of workers use unapproved AI tools in their jobs, and that around three-quarters of those using shadow AI admitted to sharing potentially sensitive information with those tools — customer data and internal documents among the most common. Strikingly, senior decision-makers were more than twice as likely as the staff they manage to use unapproved tools. A firm that has banned ChatGPT has not stopped its people using ChatGPT; it has stopped them telling anyone they use it. And a policy nobody follows is worse than no policy, because it creates the illusion of control.

The duties were already there

None of this required new regulation, and largely none has arrived. What the professional bodies have done instead is restate existing obligations through an AI lens.

The Law Society’s position is that a solicitor’s professional duties — to the court and to the client — apply to work regardless of whether AI was used to assist with it, and regardless of whether the AI was used by the solicitor personally or by someone under their supervision[3]. The Bar Council reaches the same destination: there is “nothing inherently improper about using reliable AI tools for augmenting legal services, but they must be properly understood by the individual practitioner and used responsibly”[2]. The Solicitors Regulation Authority’s Risk Outlook work on AI in the legal market notes rapid adoption — around three-quarters of the largest firms were already using AI — and frames the risks (accuracy, bias, confidentiality) as matters for existing standards rather than new ones[4].

Accountancy has followed the same pattern. ICAEW’s guidance treats AI as an assistant, not an authority, and stresses that professional scepticism — the discipline auditors already apply to management’s claims — must extend to AI output[5]. The Professional Conduct in Relation to Taxation guidance has been updated to interpret the five fundamental principles through an AI lens, explicitly not to add new rules[6].

The through-line is worth stating plainly: AI changes the medium, not the duty. Competence, confidentiality and supervision attach to the professional, not the tool. A partner who would never let a trainee’s first draft go to a client unread has no basis for treating a model’s first draft differently — the model is, in effect, a very fast, very confident junior with no professional obligations of its own and no fear of consequences.

Governance that maps to the failure modes

Because each failure mode is specific, the controls can be too. A useful test for any AI policy: for each of the four failures above, can you point to the clause that would have stopped it?

  • Verification gates on anything citable. Any AI-assisted output containing a citation, a statistic, a quotation or a reference to authority gets source-verified by a human before it leaves the firm — checked against the primary source, not against the model’s own assurances. For law firms this is now close to a survival requirement; the Ayinde judgment makes clear that “I didn’t know the tool did that” is not a defence the courts will entertain[1].
  • Named reviewers and review depth matched to stakes. The Deloitte episode shows that a generic review step is not enough — the review has to be designed to catch AI-specific defects, which look different from human ones. Human errors cluster around carelessness and gaps; model errors are fluent, internally consistent and confidently sourced to things that do not exist. Reviewers need to be told this, trained on examples, and given time to check references rather than prose quality alone.
  • An approved-tools list, not a ban. Specify which tools are permitted, under which subscription tiers (enterprise agreements with no-training clauses and data-residency terms, not personal accounts), and for which categories of data. Pair it with a clear line on client transparency: the SRA suggests firms consider telling clients when AI is used on their matter[4], and the cleanest mechanism is a short clause in the engagement letter describing the firm’s use of approved AI tools and the safeguards around client data. Firms that were embarrassed retrospectively disclosed; firms with governance disclose up front.
  • Make the approved route better than the workaround. People use unapproved tools because the approved ones are absent, slow or worse. A realistic tools list, fast evaluation of requests to add new ones, and training that treats staff as adults — here is what these tools do well, here is where they fail, here is what must never be pasted into them — will do more than any prohibition. Given that executives are the heaviest shadow-AI users, the training has to reach the partnership, not just the staff.

And one conversation many firms are deferring: insurance. Professional indemnity insurers are actively working out their position on AI. Some carriers have introduced broad AI exclusions; others are probing “silent AI” exposure in existing wordings, and brokers report insurers asking specifically about governance, training and oversight regimes at renewal. A firm that can show an approved-tools list, review gates and training records is having a very different renewal conversation from one that cannot — and a firm that says “we don’t use AI” while 80% of its people quietly do is arguably misrepresenting its risk.

Where the formal frameworks fit

Two frameworks dominate the AI governance conversation, and it helps to be honest about who they are for.

ISO/IEC 42001 is the first certifiable international standard for AI management systems — the AI counterpart to ISO 27001, covering policy, risk assessment, impact assessment, lifecycle controls and continual improvement, with accredited certification available from bodies such as BSI[7]. The NIST AI Risk Management Framework is a voluntary US framework built on four functions — Govern, Map, Measure, Manage — with a 2024 Generative AI Profile that catalogues risks such as confabulation and data privacy and maps suggested actions against each[8]. Firms operating in or serving the EU should also note the AI Act’s Article 4 AI-literacy duty, applicable since February 2025 to deployers of AI systems at every risk level, with national enforcement beginning in August 2026.

For a ten-partner practice, pursuing ISO 42001 certification this year is probably the wrong project. The certification makes sense when clients start asking for it in tenders, when you build AI into services you sell, or when your scale means informal oversight has genuinely stopped working. What a small firm should take from these frameworks is their shape, not their paperwork: someone accountable, an inventory of what tools are used for what, risks assessed before adoption rather than after the incident, and periodic review. That fits on one page. A one-page policy that is followed — approved tools, forbidden data, review gates, who to ask — beats a forty-page framework binder that is not.

The direction of travel seems clear enough. Courts have shown they will name and refer professionals who file fabricated authority; clients who read about the Deloitte refund will ask their own advisers harder questions; insurers are already asking them. Financial advice firms face a parallel but distinct version of this picture under FCA supervision, which I’ve covered separately in AI governance for UK financial advisers. For everyone else in professional services, the governance required is not exotic. It is the same discipline the professions have always applied to junior work — applied, deliberately, to a new and tireless junior.

This article is for informational purposes only and does not constitute legal advice or a compliance opinion. Consult a qualified professional for advice specific to your firm.

Sources

[1] Courts and Tribunals Judiciary, ‘Ayinde v London Borough of Haringey; Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin)’, 6 June 2025. Available at: https://www.judiciary.uk/wp-content/uploads/2025/06/Ayinde-v-London-Borough-of-Haringey-and-Al-Haroun-v-Qatar-National-Bank.pdf

[2] Bar Council, ‘Considerations when using ChatGPT and generative AI software based on large language models’, updated November 2025. Available at: https://www.barcouncilethics.co.uk/documents/considerations-when-using-chatgpt-and-generative-ai-software-based-on-large-language-models/

[3] The Law Society, ‘Generative AI — the essentials’, first published 7 August 2024, since updated. Available at: https://www.lawsociety.org.uk/topics/ai-and-lawtech/generative-ai-the-essentials

[4] Solicitors Regulation Authority, ‘Risk Outlook report: The use of artificial intelligence in the legal market’. Available at: https://www.sra.org.uk/sra/research-publications/artificial-intelligence-legal-market/

[5] ICAEW, ‘Why professional scepticism is so crucial to using AI’, January 2025. Available at: https://www.icaew.com/insights/viewpoints-on-the-news/2025/jan-2025/why-professional-scepticism-is-so-crucial-to-using-ai

[6] ICAEW, ‘AI and accountants: the rules and guidance you need to follow’, 2025. Available at: https://www.icaew.com/technical/practice-resources/practice-news/ai-and-accountants

[7] BSI, ‘ISO/IEC 42001 — AI Management System’. Available at: https://www.bsigroup.com/en-US/products-and-services/standards/iso-42001-ai-management-system/

[8] NIST, ‘AI Risk Management Framework’ (including the Generative AI Profile, NIST AI 600-1, July 2024). Available at: https://www.nist.gov/itl/ai-risk-management-framework