IP = 142.251.46.142
robots.txtUser-agent: *
Disallow: /hosting/search
Disallow: /p/*/issues/csv
Disallow: /p/*/source/diff
Disallow: /p/*/people/detail
Disallow: /u/*
Disallow: /a/
Allow: /a/eclipselabs.org/
Allow: /a/apache-extras.org/
Disallow: /a/*/hosting/search
Disallow: /a/*/p/*/issues/csv
Disallow: /a/*/p/*/source/diff
Disallow: /p/blackout-m7/
Crawl-delay: 120
Look up this url in the url tool
https://code.google.com/.well-known/acme-challenge: 404 text/html; charset=UTF-8
https://code.google.com/.well-known/csvm: 404 text/html; charset=UTF-8
https://code.google.com/.well-known/nostr.json: 404 text/html; charset=UTF-8
https://code.google.com/.well-known/security.txt: 404 text/html; charset=UTF-8
https://code.google.com/.well-known/traffic-advice: 404 text/html; charset=UTF-8