AI-powered criminals, before the ITU

The ITU-T workshop on securing AI, run by the UN’s telecommunications standards body, dealt mainly with protecting AI-based systems and agents. Nicolas Thomas, our founder, took the subject from the other end: what criminals already do with those same tools.

His talk, “AI-powered criminals, a new frontier in cybersecurity”, is available on video, in English with subtitles.

Tools built for fraud

Criminals use language models and generative AI to produce convincing fraudulent content. Specialised tools exist, sold under explicit names, to forge identity documents that are hard to tell from the originals.

A barrier to entry that has collapsed

This is the most concerning part: for around a hundred euros a month, someone with no technical skill can reach tools that produce undetectable voice clones. Skill is no longer the limiting factor.

One real case

In 2024 an employee at the Arup group joined a video call in which the other participants were deepfakes, including one of their own executive. The attack ended in a 25 million dollar transfer. Hong Kong police summed the case up in one phrase: everyone was fake.

The limits of detectors

Detection tools produce false positives, including on texts written long before generative AI existed. A defence resting entirely on them inherits their errors.

Prove the real rather than hunt the fake

The proposal put to the workshop was to change target. Rather than trying to recognise generated content, build systems that let a person demonstrate their digital legitimacy at the moment they act. That also calls for partnerships between organisations, to exchange information that has already been verified.

The talk closed on protecting personal identity data: it is what feeds the frauds once it circulates.