Reddit has a warning for AI corporations and different scrapers: play by our guidelines or get blocked. The corporate stated in that it plans to replace its Robots Exclusion Protocol (robots.txt file), which permits it to dam automated scraping of its platform.
The corporate stated it can additionally proceed to dam and rate-limit crawlers and different bots that don’t have a previous settlement with the corporate. The modifications, it stated, shouldn’t have an effect on “good religion actors,” just like the Web Archive and researchers.
Reddit’s discover comes shortly after a number of experiences that Perplexity and different AI corporations recurrently web sites’ robots.txt protocol, which is utilized by publishers to inform internet crawlers they don’t need their content material accessed. Perplexity’s CEO, in a current with Quick Firm, stated that the protocol is “not a authorized framework.”
In a press release, a Reddit spokesperson instructed Engadget that it wasn’t concentrating on a specific firm. “This replace isn’t meant to single anyone entity out; it’s meant to guard Reddit whereas retaining the web open,” the spokesperson stated. “Within the subsequent few weeks, we’ll be updating our robots.txt directions to be as clear as doable: if you’re utilizing an automatic agent to entry Reddit, no matter what kind of firm you’re, it is advisable to abide by our phrases and insurance policies, and it is advisable to discuss to us. We imagine within the open web, however we don’t imagine within the misuse of public content material.”
It’s not the primary time the corporate has taken a tough line with regards to information entry. The corporate cited AI corporations’ use of its platform when it started charging for final 12 months. Since then, it has struck licensing offers with some AI corporations, together with and . The agreements enable AI companies to coach their fashions on Reddit’s archive and have been a major income for the newly-public Reddit. The “discuss to us” a part of that assertion is probably going a not-so-subtle reminder that the corporate is now not within the enterprise of handing out its content material without spending a dime.
