cross-posted from: https://lemmy.dbzer0.com/post/76486627
What just happened? Another incident has taken place that illustrates the need to be careful what you tell AI. A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to “shoot up” the Sheriff’s office. After a human reviewer examined the statements, they were reported to police.
According to the arrest report, Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she would attack the Sheriff’s office. She later said that she uses Anthropic’s chatbot like a “diary.”
Claude’s safety systems flagged the entry and it was escalated to a human reviewer. After deciding it was a credible threat, the reviewer reported it to law enforcement.
The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.
Deputies identified Heller and visited her home. She was detained without incident before an LCSO intelligence detective took over the investigation.
Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Anthropic isn’t going to be taking any chances when it comes to anything it deems a potential threat. Last month, it was reported that OpenAI and Sam Altman are being sued by British Columbia over claims that the company could have prevented a mass shooting in the Canadian province.
The shooter, eighteen-year-old former pupil Jesse Van Rootselaar, had previously been flagged by OpenAI’s safety team for her conversations about gun violence, but OpenAI never alerted police because the conversations did not meet the threshold for legal referral.
In June, Florida also sued OpenAI and Altman, alleging that ChatGPT had contributed to real-world harms, including the 2025 Florida State University shooting.
The latest incident is another reminder to think before you enter something into a chatbot that could get you into trouble. It’s certainly not a private diary whose contents are for your eyes only.
Reports last month revealed that human contractors reviewing Microsoft Copilot’s image editor can see users’ prompts, uploaded photos and AI-generated edits. Documents show that some of those assignments contain sexual, disturbing or potentially illegal material, though the reviewers are not there to flag the content – only to assess whether the output is accurate.
gen-ai chatbots have been advising people in how best to kill themselves with no one who owns a gen-ai company being held legally responsible, but when police are potentially at risk, suddenly there are security measures in place & legal consequences
Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.
Emphasis mine.
I think any message that you type to an AI chatbot has the expectation that it will not be viewed by another person. This should be a quick case. Way to go Anthropic, you’ve done nothing except let people know that you will hand them over to the police, and given this woman another reason to not like her local police department.
The rest of that law states it is against the law to transmit it. She transmitted it because that’s how talking to a non-local LLM works. She broke the law, the question is, did she know that she did not have the expectation of privacy.
Dude, how is talking to AI transmitting the message? If you have kidnapped someone and about to ask for ransom, would you a) call his family or b) tell ChatGPT to notify his family???
I’m on her side but how do you think her message got to the server lol
Which server? Is the victim to be owning the AI server and constantly reading all messages sent to it?
…Do you think she’s hosting Claude locally…?
The text she wrote was transmitted to a server to be processed. That is how every cloud LLM works.
Again I don’t support her arrest but that’s what transmitting means in the context of the internet
No, apparently she wasn’t hosting any server.
And yes, I also don’t think that constitute as transmitting.
How do you think her messages got to Claude? Perhaps TCP??
Because it doesn’t process the request or token on machine. So it was transmitted over the Internet to the model that processed it. She didn’t intend to transmit it (probably). But unfortunately she did exactly that. Claude isn’t a local model and so far as I can tell unless you opt out of a lot of stuff (such as allowing Anthropic to train on your LLM chat data), you will be transmitting whatever you put into it over the internet.
If she had not transmitted it anywhere then Anthropic would not have been able to report her to the police.
Thought crime! Thought crime!
FLORIDA WOMAN
