The most annoying thing about being a good bot owner, in my experience, is when you get complaints about it misbehaving, only to find that it was actually somebody malicious who wrote their own abusive bot, but is using your bot's user agent.

Cloudflare have some new bot verification proposals designed to fix this, with cryptographic proofs that the user-agent is who they say they are: https://blog.cloudflare.com/web-bot-auth/.

It is awesome to see that there are efforts to improve upon the old standards with modern web security standards

That's easy to say when it's your bot, but I've been on the other side to know that the problem isn't your bot, it's the 9000 other ones just like it, none of which will deliver traffic anywhere close to the resources consumed by scraping.

True. Major search engines and bots from social networks have a clear value proposition: in exchange for consuming my resources, they help drive human traffic to my site. GPTBot et al. will probably do the same, as more people use AI to replace search.

A random scraper, on the other hand, just racks up my AWS bill and contributes nothing in return. You'd have to be very, very convincing in your bot description (yes, I do check out the link in the user-agent string to see what the bot claims to be for) in order to justify using other people's resources on a large scale and not giving anything back.

An open web that is accessible to all sounds great, but that ideal only holds between consenting adults. Not parasites.

> GPTBot et al. will probably do the same, as more people use AI to replace search.

It really won’t. It will steal your website’s content and regurgitate it back out in a mangled form to any lazy prompt that gets prodded into it. GPT bots are a perfect example of the parasites you speak of that have destroyed any possibility of an open web.

Only if the GPT companies can resist the temptation of all that advertising $$$.

I'll give them at most 3 years before sponsored links begin appearing in the output and "AI optimization" becomes a fashionable service alongside the SEO snake oil. Most publishers won't care whether their content is mangled or not, as long as it is regurgitated with the right keywords and links.

What do you mean sponsored links? It'll be a sponsored reply, no outbound links required.

Links to a site where the sponsor's product or service can be purchased, obviously.

That was my hunch. My initial post on robots.txt: https://evgeniipendragon.com/posts/i-am-disallowing-all-craw... - revolved around blocking AI models from doing that because I do not believe that it will bring more traffic to my website - it will use the content to keep people using their service. I might be proven wrong in the future, but I do not see why they would want to let go of an extra opportunity to increase retention.

Which is all you need a lot of the time. If you're a hotel, or restaurant, or selling a product, or have a blog to share information important to you, then all you need is for the LLM to share it with the user.

"Yes, there are a lot of great restaurants in Chicago that cater to vegans and people who enjoy musical theater. Dancing Dandelions in River North is one." or "One way to handle dogs defecating in your lawn is with Poop-Be-Gone, a non-toxic product that dissolves the poop."

It's not great for people who sell text online (journalists, I guess, who else?). But that's probably not the majority of content.

You are bringing a great point. In some cases having your data as available as possible is the best thing you can do for your business. Letting them crawl and scrape creates means by which your product is found and advertised.

In other cases, like technical writing, you might want to protect the data. There is a danger that your content will be stolen and nothing will be given in return - traffic, money, references, etc.

All our customers were in North American but we let a semi naughty bot from the UK scan us and I will never understand why. It was still sending us malformed URLs we purged from the site years ago. WTF.

I'm confused why scraping is so resource intensive - it hits every URL your site serves? For an individual ecommerce site that's maybe 10,000 hits?

And the thousands of other bots also hitting those, together is far more than the legitimate traffic for many sites.

Yeah, there were times, even running a fairly busy site, that the bots would outnumber user traffic 10:1 or more, and the bots loved to endlessly troll through things like archive indexs that could be computationally (db) expensive. At one point it got so bad that I got permission to just blackhole all of .cn and .ru, since of course none of those bots even thought of half obeying robots.txt. That literally cut CPU load on the database server by more than half.

In the last month, bot traffic has exploded to 10:1 due to LLM bots on my forum according to Cloudflare.

It would be one thing if it were driving more users to my forum. But human usage hasn't changed much, and the bots drop cache hit rate from 70% to 4% because they go so deep into old forum content.

I'd be curious to see a breakdown of what the bots are doing. On demand searches? General data scraping? I ended up blocking them with CF's Bot Blocker toggle, but I'd allow them if it were doing something beneficial for me.

For me (as I'm sure for plenty other people as well) limiting traffic to actual users matters a lot because I'm using a free tier for hosting in the time being. Bots could quickly exhaust it, and your website could be unavailable for the rest of the current "free billing" cycle, i.e. until your quota gets renewed.

With 1000s of bots per month and 10,000 hits on an ecommerce site, with product images, that's a lot of data transfer, and a lot of compute if your site has badly designed or no caching, rendering all the same page components millions of times over. But...

Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).

I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).

You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.

You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).

You’ve forgotten the combinatorics of query params.

10,000 is Monday. Morning.

Yes. That’s a lot of bandwidth, depending on the content of course.

I definitely agree here. My initial response was to block everything, however you realize that web is complex and interdependent. I still believe that everyone should have autonomy over their online resources if they desire. But that comes with an intentionality behind it. If you want to allow or disallow certain traffic, you also should answer the question why or why not. That requires understanding what each bot does. That takes time and effort.

My foray into robots.txt started from the whole notion of AI companies training on everything they can put their hands on. I want to be able to have a say whether I allow it or not. While not all bots will honor the robots.txt file, there are plenty that do. One way that I found you can test that is by asking the model directly to scrape a particular link (assuming the model has browsing capabilities).

Bots are not malicious by default. It is what that company does with your data and how you feel about it that matters in the end.

> But don’t just assume that every bot is malicious by default.

I'll bite. It seems like a poor strategy to trust by default.

I'll bite harder. That's how the public Internet works. If you don't trust clients at all, serve them a login page instead of content.

In fairness this appears to be the direction we are headed anyway

This is how it's going. Half the websites I go to have Cloudflare captchas guarding them at this point. Every time I visit StackOverflow I get a 5 second wait while Cloudflare decides I'm kosher.

Are you using TOR or a VPN, spoofing your User-Agent to something uncommon, or doing something else that tries to add extra privacy?

That kind of user experience is one that I've seen a lot on HN, and every time, without fail, it's because they're doing something that makes them look like a bot, and then being all Surprised Pikachu when they get treated like a bot by websites.

I started having similar experiences when I switched to using Brave browser that blocks lots of tracking. Many websites that didn't show me those captchas and Cloudflare protection layers now have started to pop up on a regular basis.

I use regular Edge with uBlock and get cloudflare crapchas all the time.

I've found Vivaldi to be a much better experience, chrome based, and supports ublock in all its glory

I tried it and didn't like it. Currently migrating to Firefox though, their vertical tab implementation is really decent.

I get this and I assume it's because I clear cookies pretty frequently. It used to be the case that that didn't matter, but nowadays everyone shields their websites using JS.

The really bad actors are going to ignore robots.txt entirely. You might as well be nice to the crawlers that respect robots.txt.

Even if you want to play nice, robots.txt is a catch-22, as accessing it is taken as a signal you are a 'bot' by malconfigured anti-bot 'solutions'.

It sucks more that Cloudflare/similar have responded to this with "if your handshake fingerprints more like curl than like Chrome/Firefox, no access for you".

I now write all of my bots in javascript and run them from the Chrome console with CORS turned off. It seems to defeat even Google's anti-bot stuff. Of course, I need to restart Chrome every few hours because of memory leaks, but it wasn't a fun 3 days the last time I got banned from their ecosystem with my kids asking why they couldn't watch Youtube.

Where can I learn more about custom bots in JS and Chrome?

Or getting a CAPTCHA from Chrome when visiting a site you've been to dozens of times (Stack Overflow). Now I just skip that content, probably in my LLM already anyway.

Keep in mind that those LLMs are one of the bigger reasons why we see more and more anti bot behaviour on sites like SO.

That aggressive crawling to train those on everything is insane.

It's the same thing as the anti pirate ads, you only annoy legit customers, this agressive captcha campaign just makes Stackoverflow drop down even faster than it would normally by making it lower quality.

There are tools like curl-impersonate: https://github.com/lwthiker/curl-impersonate out there that allow you to pretend to be any browser you like. Might take a bit of trial and error, but this mechanism could be bypassed with some persistence in identifying what is it that the resource is trying to block.

Bad actors will always exploit whatever systems are available to them. Always have, always will.

Because if they play by the rules, they won't be bad actors

I think a better solution would be to block all the traffic, but have a comment in robots.txt with a way to be added onto a whitelist to scrape the contents of the resource. This puts a burden of requesting the access on the owner of the bot, and if they really want that access, they can communicate it and we can work it out.

It's a nice option to have and maybe good in some cases. It reminds me of the nicety that some journalists do when requesting if they can use some video uploaded on social media for their show or piece. I do like the approach and shifting of first contact burden, as well as the general philosophical principle that blocking ought to be reversible and also temporary rather than permanent (though I also like the idea of exponential timeouts that can become effectively permanent). Still, I don't see myself ever doing anything like that. I'd still prefer to just not know about the bot at all, and if I did decide to perma-block them, unless the first contact comes with sufficient dollar signs attached I'm likely to ignore it entirely. I'm not usually in the mood for starting random negotiation with anybody.

I also tend to see the web from the "open web" dream perspective. By default no traffic is blocked. The burden of requesting is already inherently done with a client -- they request a route, and I serve it or not. For things like my blog I don't tend to care who is requesting a particular route -- even admin pages can be requested, they just don't get anything without being logged in. If someone is being "cute" requesting non-existent wordpress pages or what have you, searching for vulnerabilities, or have an annoying/ugly user agent string, or are just pounding me for no real reason, then I do start to care. (The "pounding" aspect is a bit trickier -- I look at steady state. Another comment mentioned cutting their db server's cpu load in half by dropping unlikely-to-be-real-users from two countries. For me, if that is merely a steady state reduction from like 10% of a machine to 5%, I don't really care, I start caring when it would get in the way of real growth without having to use more resources.)

When I was hosting on EC2, I used to have very mild anxiety that I'd piss off someone and they'd try to "harm" me by launching a botnet of requests at large media files and rack up bandwidth costs. (I believe it when some people say this has happened more organically with normal bots in the age of LLMs, but my concern was more targeted botnets/ddos.) There are a few ways to mitigate that anxiety: 1) setup monitoring, alerts, and triggers directly in code running on the instance itself or via overseeing AWS tools (I did the latter, which is less reliable, but still. There was a threshold to shutdown the whole instance, minimizing the total damage possible to something like under a couple hundred bucks, I forget the details of trying to calculate how much traffic could theoretically be served before the monitoring side noticed) 2) hide behind cloudflare and their unlimited bandwidth, as my content was mostly static (I didn't do that) 3) move/rearchitect to a free host like github pages, give up hosting my own comments (again didn't do) 4) move to OVH which has unlimited bandwidth (did this when Amazon wanted to start charging an absurd amount for just a single ipv4 address).

I can see how it could lead to more overhead when communicating with the requesters. That could be a lot in the event that lots of them might want to crawl your resource.

I can see the argument that if I want to hide something, I should put it behind the layer of authentication. Robots is not a substitution for proper access control mechanisms. It is more of a "if they do honor this document, this would reduce the unnecessary traffic to my site" notion.

I appreciate you highlighting your personal experience in dealing with bots! I like the ideas of monitoring and being behind something like Cloudflare tools which would protect against the major influx of traffic. I think this is especially important for smaller sites which either use low or free tiers of cloud services.