- Google and Reddit entered into a licensing agreement in February 2024 that allowed the search giant to use Reddit's content for AI training.
- Turns out, the deal was much more than that. Reddit is now blocking other search engines including Bing and DuckDuckGo from crawling its content.
- Reportedly, Google is paying $60 million per year to Reddit.
In February 2024, it was widely reported that Reddit had struck a deal with Google. The contract allowed Google to train its AI model on Reddit’s content. The deal is reported to cost Google $60 million per year. While initially, the deal appeared like a licensing agreement for AI training, now it seems the deal has gone beyond the remit of the contract.
404 Media reports that thanks to the AI licensing deal between Google and Reddit, Google is the only search engine allowed to crawl Reddit’s new posts and content. This is a significant development as it demonstrates Google’s anti-competitive behavior in the search engine space. Google is also routing massive traffic to Reddit. Google Search alternatives like Bing, DuckDuckGo, Mojeek, or Qwant are unable to crawl new and recent Reddit posts.
Now, Search Engine Land has confirmed with Microsoft that Reddit is indeed blocking other search engines except Google. The report says that Reddit modified its “robot.txt” file on July 1, 2024, blocking many search engines and AI bots from crawling its content. Surprisingly, Google is the only major search engine that can crawl Reddit’s content.
I did try to find new Reddit posts on Bing and DuckDuckGo, but they kept throwing older posts from a month ago. Search engines are respecting the “robots.txt” standard and have stopped crawling new Reddit posts.
It raises several questions about Google’s monopolistic behavior. Google is already being investigated by the US Department of Justice over its antitrust practices. This development may make things worse for Google.
