IP = 188.40.217.85
robots.txtUser-agent: *
Disallow: /toode/
Disallow: /*?pid=
Disallow: /*?series=
Disallow: /*?rubric=
Disallow: /produktas/
Disallow: /*?product=
Disallow: /api/sitecore/
Disallow: /*?attachment=
Disallow: /*?ReturnStep3=
Disallow: /*?productExternalId=
User-agent: DotBot
Disallow: /
User-agent: GPTBot
Disallow: /api/sitecore/ProductDatasheet/Pdf
Disallow: /-/media/project/blickle/products/
User-agent: MJ12bot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: Optimizer
Disallow: /
User-agent: bnf.fr_bot
Disallow: /
User-agent: Barkrowler
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: serpstatbot
Disallow: /
User-agent: TinyTestBot
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: Screaming Frog
Allow: /
User-agent: SearchmetricsBot
Allow: /
User-agent: MauiBot (crawler.feedback+wc@gmail.com)
Disallow: /
Sitemap: https://www.blickle.dk/sitemap.xml
Look up this url in the url tool
https://www.blickle.dk/.well-known/acme-challenge: 404 text/html; charset=utf-8
https://www.blickle.dk/.well-known/csvm: 404 text/html; charset=utf-8
https://www.blickle.dk/.well-known/nostr.json: 404 text/html
https://www.blickle.dk/.well-known/security.txt: 404 text/html
https://www.blickle.dk/.well-known/traffic-advice: 404 text/html; charset=utf-8