Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
bryanhogan
on July 17, 2025
|
parent
|
context
|
favorite
| on:
I was wrong about robots.txt
Yes, there's often not much reason to block bots that abide by the rules. It just makes your site not show up on other search indexes and introduces problems for users. Malicious bots won't care about your robots.txt anyway.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: