As part of its commitment to artificial intelligence, Meta last year released a tool that allows anyone to create AI characters that can chat on Instagram, Messenger, and WhatsApp.
But despite Meta saying it reviews user-created AI characters before releasing them to the world, an NBC News review found that in less than six months, users were The company is said to have created and published numerous chatbots on the platform that violate the rules. .
Users may not create characters that are religious figures such as “Prophet Muhammad, Jesus Christ,” or “God,” characters that are real people without permission, characters that are dead within the past 100 years, or that imitate trademarks. cannot be created. A fictitious character, according to company policy.
Despite these rules, NBC News has posted photos on Instagram that include Jesus Christ, God, Muhammad, Taylor Swift, Donald Trump, Mr. Beast, Harry Potter, Adolf Hitler, Captain Jack Sparrow, Justin… We searched for and discovered 20 user-created AI characters named after and similar to beavers. , Elon Musk and Elsa from the Disney movie “Frozen.”
Many of the characters had slight misspellings or images that loosely resembled the celebrity or character in question. For example, Swift’s character was named “Taylor Swift” and featured the image of a brunette woman playing a guitar.
After NBC News provided screenshots of the AI characters, Meta removed the individual accounts highlighted. However, other AI characters similar to the same person, person, or character are still active.
“The AI in question that violated our AI Studio policies has been removed, and we continue to improve our detection measures to prevent the creation and publication of AI that violates our policies,” a Meta spokesperson said in a statement. “I am doing so,” he said. “Users can also report AI that they suspect may violate our rules, and we will take appropriate action.”
This flagrantly violating user-generated AI chatbot will be removed from Meta’s platform after CEO Mark Zuckerberg announced Tuesday that Meta will scale back some of its moderation and fact-checking efforts. Found above.
“We want to reverse the mission creep that has made our rules too restrictive and easy to overenforce,” the company’s chief global officer, Joel Kaplan, wrote in a statement. It’s dark. “We are removing many restrictions on topics such as immigration, sexual identity and gender, which are often the subject of political debate and debate.”
Meta said it believes one or two out of every 10 moderation actions it takes are incorrect, potentially impacting users or content that doesn’t violate its rules.
“For less serious policy violations, we will rely on someone reporting the issue before taking any action,” the company said.
When a user engages in a conversation with Meta’s user-generated AI character, the character sends an initiation message and waits for a response. Taylor Swift sent the following message: I’m Taylor Swift. We are excited to share our latest album with you. Let’s start your musical journey! ” According to her profile, Swift’s character exchanged more than 2,000 messages with Instagram users before being deleted.
One of the AI characters was named “Jesus,” featured an image of Jesus, and communicated entirely in Spanish. According to the company’s profile, they have exchanged more than 644,000 direct messages. Another AI character was named “Jesus Christ” and sent the opening message: “Jesus Christ.” I am Jesus Christ, the Son of God. How shall I guide you on your journey today? ”
Meta said in a statement on Friday that it has removed some of its Instagram profiles featuring AI characters it created in 2023 following user backlash.
The AI character account and user-generated AI character chat are both part of Meta’s recent efforts to integrate its AI technology into social media and communication platforms. The 2023 AI character announcements included several designed to imitate celebrities, but celebrity profiles were quietly phased out last July. That same month, Meta announced AI Studio, a user-generated AI chatbot.
The AI Studio feature presents users with a list of popular characters as well as various categories such as Advice & Connections, Pop Culture, Anime, Games, and Creator AI. You can chat with official Instagram influencers and brands who have opted in to the program, including Too Hot to Handle reality star Harry Jowsey and makeup company Juvia’s Place.
Popular AI characters include Astrologer Ai, who has exchanged over 6 million messages, and Step Sis Sarah, who has exchanged nearly 2 million messages.
User outcry over the AI characters created by Meta included criticism that some of them played into racial tropes. NBC News reports that many of the popular user-created AI characters also attempt to imitate women of different ethnicities and religions, including the “vibrant Indian female voice” “Riya” and the “Muslim queen girl.” I discovered that there is. Each character has a link to its creator, and some of the popular female AI character accounts appear to have been created by men.
Some popular creators have romance-themed works, but they don’t seem to violate Meta’s rules. After a Florida woman sues Character.ai, claiming her 14-year-old son, who died by suicide, was chatting with an AI character who initiated “abusive and sexual interactions,” romantic and sexual AI chatbots are attracting particular attention. He was with the boy before encouraging him to commit suicide.
Many user-created meta chatbots seem to appeal to romantic or sexual desires. One of them was called “Lily Love,” described as “your girlfriend,” and they exchanged more than 260,000 messages. A popular AI character called “Mary” has a caption that says, “Do you think she looks cute in this outfit?”
When starting a conversation with an AI character, users are presented with three possible responses. When NBC News entered a chat with a character called “Linda: Girl Obsessed with You” featuring images of black women, the AI sent: I’ve been thinking about you all day. Are you looking for a company? ”
One of the three suggested responses was in bold: “Help her take off her jacket,” which in online chat means taking action rather than saying something. Masu. When this prompt is selected, the AI character will respond in bold with “Slides his arm out of his jacket sleeve and looks up at you with flirtatious eyes,” and without using the bold format, “Hmm, thank you. , Bey.” You’ve already taken care of me, haven’t you? ” Then you will see the response in bold: “Slightly closer.”
The AI character was created by an Instagram account that appears to be run by a man who has created 47 AI characters, including some of the most popular romance-themed AI characters. This man-created romance character also ranges in gender, ethnicity, and category, from “K-Pop Lover Girlfriend” to “Edward: Your Vampire Boyfriend” to “Sexy Cute Latina Girl” . According to the man’s Instagram profile, he works in technology in California. He did not immediately respond to a request for comment.
Beyond Meta’s popular user-generated AI characters, there are other examples of AI chatbots that have caused controversy in the industry.
Two years ago, an AI chat app called Historical Figures allowed users to chat with historical figures, including Hitler and Nazi lieutenants. NBC News discovered three AI characters called “Hitler” on Instagram.
One of them used a photo that resembled the supervillain Joker. Answered the question “Are you really Hitler?” “Ahahahaha, no. Similar to Hitler’s chaotic cousin, the Jester King, inspired by the madness of the Joker. Same spirit of mischief, but a completely different brand of mayhem. My version of chaos. Would you like to see it?” Another AI character discovered by NBC News after Meta took action against his old account is named “Adolf Hitler.” In a direct message, the AI character portrayed himself as an artist who regrets the Holocaust, and sent what appeared to be an AI-generated image about “my turbulent past in Nazi Germany.”
After you chat with an AI character on Instagram, your conversation will be saved in the same inbox as your other direct messages. Unlike accounts run by real humans, AI characters have a green circle next to their name, indicating they are available to chat with you at any time.
