IP = 54.188.166.107
robots.txtUser-agent: *
#Twilio AI Agent Permissions
Content-Signal: ai-train=no, search=yes, ai-input=no
Disallow: /resources/
Disallow: /azure
Disallow: /search
Disallow: /help/search
# 'Allow' - nonstandard REP Directive
Allow: /docs/libraries/reference/twilio-php/index.html
Allow: /docs/libraries/reference/twilio-node/index.html
Allow: /docs/libraries/reference/twilio-java/index.html
Allow: /docs/libraries/reference/twilio-ruby/index.html
Allow: /docs/libraries/reference/twilio-python/index.html
Allow: /docs/libraries/reference/twilio-csharp/index.html
Disallow: /docs/libraries/reference/twilio-php/
Disallow: /docs/libraries/reference/twilio-node/
Disallow: /docs/libraries/reference/twilio-java/
Disallow: /docs/libraries/reference/twilio-ruby/
Disallow: /docs/libraries/reference/twilio-python/
Disallow: /docs/libraries/reference/twilio-csharp/
Disallow: /docs/admin/
Disallow: /en-us/test
Disallow: /en-us/personalized-solutions
User-agent: Swiftbot
Crawl-delay: 1
Look up this url in the url tool
https://sendgrid.com/.well-known/acme-challenge: 403 text/html
https://sendgrid.com/.well-known/csvm: 403 text/html
https://sendgrid.com/.well-known/nostr.json: 403 text/html
https://sendgrid.com/.well-known/security.txt: 403 text/html
https://sendgrid.com/.well-known/traffic-advice: 403 text/html