Search

## FreightUtils robots.txt
## Site-wide policy: open for legitimate search and AI crawlers,
## blocked for SEO-scraper bots that don't add value back to operators.
## /dop is an unlisted, passcode-gated private demo — disallowed for every
## crawler. Named groups override `User-agent: *`, so the disallow is
## repeated in each allowed-bot group below.

User-agent: *
Allow: /
Disallow: /dop
Crawl-delay: 5

## SEO scrapers — blocked. They consume bandwidth and feed paid
## competitor-intelligence products without contributing to indexing.
User-agent: AhrefsBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: MJ12bot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: BLEXBot
Disallow: /

## AI crawlers — explicitly allowed. We *want* FreightUtils to be
## reachable from LLM training and inference (e.g. ChatGPT browsing,
## Claude search, Bing/Copilot indexing, OpenAI search) so users can
## find freight reference data through those surfaces.
User-agent: GPTBot
Allow: /
Disallow: /dop

User-agent: OAI-SearchBot
Allow: /
Disallow: /dop

User-agent: ChatGPT-User
Allow: /
Disallow: /dop

User-agent: ClaudeBot
Allow: /
Disallow: /dop

User-agent: anthropic-ai
Allow: /
Disallow: /dop

User-agent: CCBot
Allow: /
Disallow: /dop

User-agent: PerplexityBot
Allow: /
Disallow: /dop

User-agent: Google-Extended
Allow: /
Disallow: /dop

User-agent: Applebot
Allow: /
Disallow: /dop

User-agent: Amazonbot
Allow: /
Disallow: /dop

Sitemap: https://www.freightutils.com/sitemap.xml

Contact: mailto:contact@freightutils.com
Expires: 2027-05-02T00:00:00.000Z
Preferred-Languages: en
Canonical: https://www.freightutils.com/.well-known/security.txt
Policy: https://www.freightutils.com/for-it