Search

Matched domain: habr.com

`robots.txt`

User-agent: Yandex
Disallow: /search/
Disallow: /ru/search/
Disallow: /en/search/
Disallow: /ru/companies/*/fans/
Disallow: /en/companies/*/fans/
Disallow: /ru/companies/*/workers/
Disallow: /en/companies/*/workers/
Disallow: /ru/users/*/followers/
Disallow: /en/users/*/followers/
Disallow: /ru/users/*/following/
Disallow: /en/users/*/following/
Clean-param: utm_source&utm_medium&utm_term&utm_campaign /

User-agent: Googlebot
Disallow: /search/
Disallow: /ru/search/
Disallow: /en/search/
Disallow: /*?*utm_source=
Disallow: /*?*utm_medium=
Disallow: /*?*utm_term=
Disallow: /*?*utm_campaign=
Disallow: /ru/companies/*/fans/
Disallow: /en/companies/*/fans/
Disallow: /ru/companies/*/workers/
Disallow: /en/companies/*/workers/
Disallow: /ru/users/*/followers/
Disallow: /en/users/*/followers/
Disallow: /ru/users/*/following/
Disallow: /en/users/*/following/

User-agent: Slurp
Crawl-delay: 8
Disallow: /search/
Disallow: /ru/search/
Disallow: /en/search/
Disallow: /*?*utm_
Disallow: /ru/companies/*/fans/
Disallow: /en/companies/*/fans/
Disallow: /ru/companies/*/workers/
Disallow: /en/companies/*/workers/
Disallow: /ru/users/*/followers/
Disallow: /en/users/*/followers/
Disallow: /ru/users/*/following/
Disallow: /en/users/*/following/

User-agent: *
Crawl-delay: 10
Disallow: /search/
Disallow: /ru/search/
Disallow: /en/search/
Disallow: /*?*utm_
Disallow: /ru/companies/*/fans/
Disallow: /en/companies/*/fans/
Disallow: /ru/companies/*/workers/
Disallow: /en/companies/*/workers/
Disallow: /ru/users/*/followers/
Disallow: /en/users/*/followers/
Disallow: /ru/users/*/following/
Disallow: /en/users/*/following/

Look up this url in the url tool https://habr.com/.well-known/acme-challenge: 404 text/html; charset=utf-8
https://habr.com/.well-known/csvm: 404 text/html; charset=utf-8
https://habr.com/.well-known/nostr.json: 404 text/html; charset=utf-8
https://habr.com/.well-known/security.txt: 404 text/html; charset=utf-8
https://habr.com/.well-known/traffic-advice: 404 text/html; charset=utf-8