Search

Matched domain: www.gov.uk

`robots.txt`

User-agent: *
Disallow: /*/print$
# Don't allow indexing of site search
Disallow: /search/all*
Sitemap: https://www.gov.uk/sitemap.xml

# The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models.
# It results in timeouts from Vertex that back up requests from users making genuine searches
User-agent: meta-externalagent
Disallow: /search/all*

# https://ahrefs.com/robot/ crawls the site frequently
User-agent: AhrefsBot
Crawl-delay: 10

# https://www.deepcrawl.com/bot/ makes lots of requests. Ideally we'd slow it
# down rather than blocking it but it doesn't mention whether or not it
# supports crawl-delay.
User-agent: deepcrawl
Disallow: /

# Complaints of 429 'Too many requests' seem to be coming from SharePoint servers
# (https://social.msdn.microsoft.com/Forums/en-US/3ea268ed-58a6-4166-ab40-d3f4fc55fef4)
# The robot doesn't recognise its User-Agent string, see the MS support article:
# https://support.microsoft.com/en-us/help/3019711/the-sharepoint-server-crawler-ignores-directives-in-robots-txt
User-agent: MS Search 6.0 Robot
Disallow: /

Look up this url in the url tool https://www.gov.uk/.well-known/acme-challenge: 404 text/html; charset=utf-8
https://www.gov.uk/.well-known/csvm: 404 text/html; charset=utf-8
https://www.gov.uk/.well-known/nostr.json: 404 text/html; charset=utf-8

`https://www.gov.uk/.well-known/security.txt`

Policy: https://www.gov.uk/help/report-vulnerability

Contact: https://hackerone.com/44c348eb-e030-4273-b445-d4a2f6f83ba8/embedded_submissions/new
Contact: https://www.gov.uk/contact/govuk

Acknowledgments: https://vdp.cabinetoffice.gov.uk/thanks.txt

Hiring: https://www.civilservicejobs.service.gov.uk/

Last-Updated: 2026-07-20 03:40:05+00:00

Expires: 2026-10-20 03:40:05+00:00

Generated at: https://github.com/CO-Cyber-Security/security.txt

https://www.gov.uk/.well-known/traffic-advice: 404 text/html; charset=utf-8