Dear news publishers
What if we told you that you are being used as a scapegoat in the era of generative AI?
What if we told you that your website traffic is going to reduce drastically thereby impacting e-advertising efforts?
Sounds scary but if you do not stay vigilant, it may turn into reality. ChatGPT and Google’s Bard have resorted to deploying a scheme that can be a game-changer for web users, content creators and researchers but might push you out of the business in the future. Bots from these generative AI tools are crawling news publisher sites to add fresh content to their database. While ChatGPT does not provide information after September 2021, it is being said that the plus users might soon gain access to updated information from premium news publishers along with the source, resulting from the crawling activities.
In this blog, we will delve deeper into how AI bots can impact the future of news publishers, the importance of blocking them and what’s the current scenario around this topic
AI bots vs news publishers – How it began?
The world is increasingly becoming AI-adaptive. To ensure that news publishers do not stay behind in the race, search engine giants Google and Bing announced policies regarding how the content on news publisher websites is used in the era of AI. Moreover, as a part of these policies, publishers wished to exercise greater control over the content usage which was guaranteed by these platforms.
In fact, in September 2022, Bing launched their bot Bing Chat into search results by assuring publishers that exclusive content carrying a NOCACHE or NOARCHIVE tag will continue to appear in their search results without impacting their rankings in the SERP.
Similarly, Google also announced Google Extended, a tool for publishers to be able to control access to content on their sites and decide whether they “help improve” Google Bard, its AI-driven chat tool.
Despite these consistent efforts from the search engine giants, what’s the problem if the AI training continues for the news publishers? What has led to the outrage among news publishers?
News publisher’s view – It’s the future dealings at stake
The UK’s Independent Publishers Alliance has recommended its members block ChatGPT from crawling their sites as soon as possible. This comes due to the following reasons-
- Increased cost- This involves monetary cost due to a higher hosting bracket employment if the number of bot visits increases to a small publisher’s site. Moreover, additional costs might be incurred for the prevention of plagiarism arising from the parasitic activities of the generative AI tools.
- Less negotiating power-In case these bots are permitted and not blocked, any information would be readily available to a web user using generative AI tools. As a result, the publishers would be left with less negotiating power to charge the AI tools for their content. Similarly, they may not be able to charge a premium price from the users to gain long-term loyalty.
- Self-harm with the entry of regulators- When publishers are given an option to block the bots, they create a disadvantage for themselves by allowing them. In future dealings, when regulators are involved, publishers may not be able to charge the AI tools as the latter may argue saying ‘Why should they pay for information that is available for free’ in case publishers do not block them currently.
What does the future look like for news publishers?
Since August, 44% of 1,123 news publishers monitored in a continual survey by open-source archiving system homepages.news have blocked out of ChatGPT’s crawls using robots.txt – the code that tells crawlers what parts of a website they are allowed to access.
UK publishers reported by homepages.news to have blocked GPT bot include the Daily Mail, The Sun, The Guardian, Belfast Telegraph, Daily Herald, Newsquest’s Daily Echo, The Economist and The National.
In the US, major players like ABC News, Axios, New York Times and sister title The Athletic, Bloomberg, Boston Globe, CBS News, CNBC, CNN, New York Daily News, Deadline, E, ESPN, Gawker, the Hollywood Reporter, Los Angeles Times, NBC News, New Yorker, Semafor, Slate, USA Today, Wall Street Journal, Washington Post and many local titles have all blocked GPTBot.
While there are advantages to it as mentioned before, publishers need to be careful while blocking the bots. For example, in the case of Google Extended which is a part of Google bots, you need to be wary of the bots you allow and block. In case you block Google bots completely, you are at risk of making your site disappear from the search results altogether.
From a publisher's perspective, it is important to be mindful and not allow generative AI bots to take over your business without a concrete negotiation technique in place. As for the future, it is always advantageous to strike a deal with these tools to provide access to your content. However, the need of the hour is to play safe and block these bots with much-needed precautions.
