As a means of stopping other bots from exploiting postings for AI training, Reddit announced that it will begin limiting the majority of automated bots’ access to the platform’s public data.
Although this has technically always been Reddit policy, the business is now putting it into practice by altering the robots.txt file, a crucial component of the internet that controls which websites web crawlers are permitted access to.
According to the firm, crawlers and other bots without a prior agreement with the company would still be blocked and have their rate limited.
“Good faith actors,” such as the Internet Archive and scholars, should not be impacted by the modifications, it stated.
Newsng gathered that Reddit prohibits outside parties from crawling its data without authorization, and it will quickly take action against any such attempts.
Some Reddit collaborators and partners, including The Internet Archive, are already able to do so because of agreements they have, thus their likes will not be impacted by the future change.
In a statement, a Reddit spokesperson said that it wasn’t targeting a particular company. “This update isn’t meant to single any one entity out; it’s meant to protect Reddit while keeping the internet open,” the spokesperson said.
“In the next few weeks, we’ll be updating our robots.txt instructions to be as clear as possible: if you are using an automated agent to access Reddit, regardless of what type of company you are, you need to abide by our terms and policies, and you need to talk to us.
“We believe in the open internet, but we do not believe in the misuse of public content.”
Search engines will need to arrange a licensing agreement to use Reddit as a data source for AI model training and other commercial uses after the protocol has been changed.
Websites and media are currently entering into agreements with several AI businesses, including Google and OpenAI.
We earlier reported that the social media platform and its investors are hoping to generate up to $748 million through its Initial Public Offering, IPO.
