AI extremism tool: OpenAI, Anthropic partner explores new safeguards
Synopsis
New AI tool aims to detect extremism and redirect users to real-world support systems
A crisis-response contractor working with OpenAI and Anthropic is developing a new AI-driven tool to detect early signs of violent extremism and redirect users to support systems, marking a major step in tackling safety risks linked to generative AI platforms.
Key highlights
- ThroughLine developing AI tool to counter violent extremism
- Backed by work with OpenAI, Anthropic and Google
- Collaboration underway with The Christchurch Call
- Hybrid model to combine chatbot intervention with human support
Inside the new AI intervention model
New Zealand-based startup ThroughLine is building a system that identifies users showing extremist tendencies on platforms like ChatGPT and guides them toward deradicalisation resources.
The tool is expected to combine chatbot-based engagement with referrals to real-world mental health and support services, creating a hybrid intervention model.
Founder Elliot Taylor said the technology is currently being tested, though no official launch timeline has been set.
Why AI platforms are under pressure
The initiative comes as AI companies face increasing scrutiny over their role in preventing harmful behaviour.
OpenAI previously faced pressure from Canadian authorities after a school shooting suspect had been banned from its platform without law enforcement being notified.
A growing number of lawsuits globally have also accused AI firms of failing to prevent, or even enabling, violent outcomes.
Expanding beyond mental health support
ThroughLine already works with major AI companies to redirect users at risk of self-harm, domestic violence and eating disorders.
The company operates a network of over 1,600 helplines across 180 countries, connecting users to human-run services when risks are detected.
Now, it aims to expand that scope to include online radicalisation, a category that has grown alongside the rise of AI chatbot usage.
Global collaboration to tackle online extremism
ThroughLine is in discussions with The Christchurch Call, a global effort launched after New Zealand’s 2019 terror attack to curb online extremism.
The partnership would see the organisation provide expertise while ThroughLine develops the technical solution.
The tool could also be adapted for moderators of gaming platforms, and parents and caregivers monitoring online behaviour.
Experts flag both promise and challenges
Analysts say the concept addresses a key gap in AI safety, the relational aspect of online radicalisation.
However, experts caution that success will depend heavily on the effectiveness of follow-up support systems, along with the quality of real-world interventions users are directed to.
Concerns also remain around whether escalation measures, such as alerting authorities, could trigger unintended consequences.
Australia angle: What it means for local policy and tech
For Australia, the development is closely relevant:
- Policymakers are already debating stricter AI safety frameworks
- Universities and research bodies are studying online radicalisation risks
- Tech firms operating locally may need to adopt similar intervention systems
The move could accelerate regulatory discussions around AI accountability and platform responsibility in Australia.
Now What?
The tool remains in development, with testing ongoing and no confirmed rollout date.
As AI adoption grows, platforms like OpenAI and Google are expected to invest further in safety infrastructure to manage rising risks tied to extremism and harmful behaviour.
FAQs
Q1: What is the new AI extremism tool?
It is a system being developed to detect extremist tendencies in users and redirect them to support services using both chatbots and human intervention.
Q2: Who is building this tool?
New Zealand-based startup ThroughLine, which already works with OpenAI, Anthropic and Google.
Q3: Why is this tool needed?
Because AI platforms face growing criticism for failing to prevent harmful behaviour, including extremism.
Q4: Will the tool involve human support?
Yes, it will combine automated chatbot responses with referrals to real-world helplines and services.
Follow Inspirepreneur Magazine for the business news.
I write about markets, money, and the macro forces that move them. Passionate about turning complex economic trends into sharp, easy-to-understand stories. Off the clock, it’s hip hop, rock, reggae -- and a mix of cricket and basketball.
