A Business Insider investigation into Instagram's new TV service found it recommending a crop of graphic and disturbing videos, whose content appeared to include child exploitation and genital mutilation.
Business Insider spent nearly three weeks monitoring IGTV, the Facebook-owned video service that launched in June as part of Instagram's attempts to muscle in on rivals like YouTube and Snapchat. IGTV's algorithm recommended questionable content during that time, including sexually suggestive footage of young girls and an explicit video of a mutilated penis.
Two of the videos discovered by Business Insider were reported to the police by the National Society for the Prevention of Cruelty to Children, a British children's charity. Instagram removed them five days after Business Insider reported them through the app's official reporting channel.
Instagram apologized to users who saw the videos and said it wanted to make IGTV a safe space for young people.
The findings come at a time when Facebook is under extraordinary scrutiny over inappropriate content on its platforms. Facebook and Instagram share a community-operations team, and Mark Zuckerberg's company has hired an army of 7,500 moderators and is using artificial intelligence to snuff out posts that break its guidelines.
Earlier this year, a former Facebook moderator told Business Insider that she had to review 8,000 posts a day and that the work made her numb to child abuse. But despite the intense oversight, and the resources Facebook is devoting to policing its platforms, disturbing content appears to be slipping through the net and, as in the case of IGTV, is even being suggested to users.
IGTV's content-recommendation machine
Instagram launched IGTV in June, a move many viewed as Facebook pushing into YouTube's territory. It allows users to set up their own channels and upload video lasting up to an hour. Anyone with an Instagram account can make a channel, and users swipe through them much as they would flick through channels on a television.
IGTV recommends content in three ways: a For You tab, which plays videos as soon as you open IGTV; a Popular section; and a Following menu, which offers videos from people you follow.
Instagram did not answer Business Insider's questions on how IGTV's algorithm recommends certain videos and why videos were suggested that appeared to show child exploitation. But it appears that the For You section recommends things users will like, possibly based on past activity. The Popular tab seems to gather trending content from across IGTV.
Users can scroll through the recommended videos by swiping left, or IGTV will automatically play the next video. It is clearly designed to encourage scrolling and continued viewing, in much the same way the YouTube algorithm recommends content through its Up Next bar.
Disturbing videos of young girls
Business Insider monitored the For You and Popular tabs for almost three weeks to establish what kinds of content IGTV's algorithm was serving up for users.
We did so in two ways: first through the account of this author and other Business Insider journalists and then with an anonymous login set up as a child's account. This second account had no activity history on Instagram and a user age set to 13, which is the earliest people can officially sign up on the app.
Within days of monitoring IGTV through Business Insider accounts, a video appeared in the For You section titled "Hot Girl Follow Me." It showed a young girl, we speculate to have been 11 or 12, in a bathroom. She glanced around before moving to take her top off. Just as she's about to remove her clothing, the video ends.
The video, uploaded by a user Business Insider is not naming for legal reasons, also appeared under the Popular tab on IGTV. It was also one of the first videos recommended under the For You section to the child account set up by Business Insider, which had no prior history of activity on Instagram.
The same user who uploaded the "Hot Girl Follow Me" video posted another video, titled "Patli Kamar follow me guys plzz," which was also recommended to our child Instagram account under the For You section. It featured another clearly underage girl exposing her belly and pouting for the camera.
The same two videos were separately uploaded by a different user, whom again Business Insider has chosen not to identify. The video named "Hot Girl Follow Me" was called "Follow me guys" by this second user and was also circulating on IGTV's suggested posts.
Comments on the videos show they were being recommended to other IGTV users. They were also being interpreted by other users as sexually suggestive.
Some condemned the videos and questioned why they had been suggested. "BRO SHE'S LIKE FUCKING 10 WHY THE FUCK IS THIS IN MY INSTAGRAM RECOMMENDED," one user said, commenting on the "Hot Girl Follow Me" video.
Others were more predatory in tone. "Superb," one user commented on the "Patli Kamar follow me guys plzz" video. "Sexy grl," another added.
The National Society for the Prevention of Cruelty to Children, which is frequently involved in law-enforcement activities around child abuse, reviewed the videos and reported them to the police. It was concerned that they could constitute illegal indecent images under UK law because they appeared to feature footage of erotic posing.
"This is yet another example of Instagram falling short by failing to remove content that breaches its own guidelines," a spokeswoman for the group said.
Business Insider reported the videos through Instagram's official reporting function. Because there were no obvious criteria for alerting the company to potential child exploitation, they were logged as "nudity or pornography."
The videos remained online for five days. It was only after Business Insider contacted Instagram's press office that the content was removed. By this time, the two videos — and other versions uploaded by the second user — had more than 1 million views.
Instagram left the accounts that posted the videos active, however. Business Insider asked why the accounts were left up, as Instagram has a "zero-tolerance policy" on child abuse. Instagram said the policy applied to the content and not to the account uploading it.
"Instagram's zero-tolerance policy towards child-abuse content is the right one, and it must make sure its policy is enforced in practice," a spokeswoman for the National Society for the Prevention of Cruelty to Children said. "Where Instagram has removed child-abuse content from an account, we would expect that account to be reviewed by a moderator to establish whether the account should also be suspended."
As of this week, the two accounts remain active. They continue to post sexually suggestive content, but not of the same nature as the "Hot Girl Follow Me" and "Patli Kamar follow me guys plzz" videos.
A graphic video of genital mutilation
Potential child exploitation was not the only questionable content being recommended by IGTV's algorithm.
One of the first videos recommended to Business Insider's anonymous account, registered with an age of 13, was graphic footage of a penis undergoing an operation involving a motorized saw.
The penis appeared to have a metal lug nut affixed around its middle, above which it was extremely swollen and dark red. The bolt was being removed with a round electric saw by what appeared to be a medic.
The video was quickly wiped from IGTV after being reported by Business Insider as nudity, though the account that uploaded it remained live.
Another recommended video showed a baby lying on the floor, wailing inconsolably, with a monkey standing over and touching the infant. Adults were standing around in a circle, shouting while filming the scene on their phones as the monkey occasionally lashed out at them.
Instagram found that the video did not breach of guidelines, but it learned the account that uploaded it was linked to a different account that had been taken down for breaching community guidelines. For this reason, Instagram took the account down.
IGTV's algorithm pushed a multitude of other questionable recommendations. Examples included a video in which a group of men deceived a sex worker into thinking she was going to be arrested, a video of a woman pulling something long and bloody out of her nose, and various sexually suggestive scenes.
Business Insider presented its investigation to MP Damian Collins, the British lawmaker who is leading an inquiry into deceptive information and Facebook's data breach involving Cambridge Analytica. He described the findings as "very disturbing" and said big tech companies needed to sink more investment into enforcing their rules.
"It's a question of the responsibility of the companies to monitor the content that's on their platforms," he said. "A lot of the problematic content is already in breach of the community guidelines of these services, but what it shows is that there's not effective enforcement."
An Instagram spokeswoman said: "We care deeply about keeping all of Instagram — including IGTV — a safe place for young people to get closer to the people and interests they care about.
"We have community guidelines in place to protect everyone using Instagram and have zero tolerance for anyone sharing explicit images or images of child abuse. We have removed the videos reported to us and apologize to anyone who may have seen them.
"We take measures to proactively monitor potential violations of our community guidelines and just like on the rest of Instagram, we encourage our community to report content that concerns them. We have a trained team of reviewers who work 24/7 to remove anything which violates our terms."
'It's not so different from where YouTube was 10 years ago'
It's no secret that social media has a problem with disturbing and illegal material, and tech giants like Facebook have come under fire recently for failing to effectively moderate content at scale. For IGTV, however, the problem isn't just that this material exists but rather that the algorithm is actively suggesting it.
Mike Henry, the CEO of the video-analytics firm OpenSlate, which works with big technology firms such as YouTube, noted that IGTV was still a young service. "While Instagram is relatively mature, IGTV is a brand-new social video platform and will need time to develop its policies and technology," he said. "It's not so different from where YouTube was 10 years ago."
YouTube's child-safety policy is broader than Instagram's, for example. Instagram's report function is limited to child nudity, while YouTube's endangerment policy bans "sexualization of minors," which allows for reporting images users suspect of being child exploitation.
Henry also said Instagram would have to figure out how to better filter its new platform if it hoped to monetize it, especially considering IGTV was touted as a space for influencers.
"Influencers make great video producers with compelling economics and, at scale, a viable canvas for video ad dollars," he said. "With the right policies and infrastructure, IGTV has the potential to become a major player."
For Collins, IGTV's early missteps are evidence that governments need to do more to regulate tech firms.
"These companies are ad services — they make money out of understanding every single thing you could ever want to know about your users so you can target them with advertising," he said. "That same technology should surely very easily be able to root out harmful content as well.
"They don't do it because there's not been a commercial incentive for them to do it, so they've just not bothered. But what we have to do through regulation is create that incentive, to say, 'You've actually got an obligation to do it and if you don't do it, then there will be costs for you for not complying, so you need to invest in doing this now.'"