Search

Matched domain: urlpipe.dev

`robots.txt`

# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file

# Allow all crawlers, including AI search bots (GPTBot, ChatGPT-User,
# PerplexityBot, ClaudeBot, anthropic-ai, Google-Extended, Bingbot).
#
# The disallowed paths are signed-in areas that only redirect to the login
# page. /auth stays crawlable: its pages carry the noindex tag, and a crawler
# only sees that tag on a page it is allowed to fetch.
User-agent: *
Allow: /
Disallow: /admin
Disallow: /app
Disallow: /jobs
Disallow: /onboarding

Sitemap: https://urlpipe.dev/sitemap.xml

Look up this url in the url tool https://urlpipe.dev/.well-known/acme-challenge: 404 text/html; charset=utf-8
https://urlpipe.dev/.well-known/csvm: 404 text/html; charset=utf-8
https://urlpipe.dev/.well-known/nostr.json: 404 text/html; charset=utf-8
https://urlpipe.dev/.well-known/security.txt: 404 text/html; charset=utf-8
https://urlpipe.dev/.well-known/traffic-advice: 404 text/html; charset=utf-8