Crawler Identification
AI Khazna AI SEO / AEO / GEO Audit Bot
User-Agent string
AI-Khazna-Audit-Bot/1.0 (+https://aikhazna.com/about/audit-bot)What this bot does
Fetches one page chosen by the user, then analyzes the HTML and response headers for 22 search-engine, answer-engine, and generative-engine optimization signals. The result is written to a database row tied to that single scan request. The bot also performs one lightweight probe of /llms.txt on the same origin to check for the emerging LLM-discoverability standard.
Paths this bot may request
MethodPathNotes
GET
<target URL pasted by the user>Main page fetch; follows redirects.GET
{origin}/llms.txtLightweight probe (5s timeout). A 404 is the expected response and is treated as a soft suggestion to publish llms.txt, not a failure.Operating principles
- User-initiated only. This bot never crawls autonomously — it only fetches the URL a user pastes into the tool.
- Single request per scan. We do not follow links from the fetched HTML or perform recursive discovery.
- No authentication. The tool is anonymous; the bot does not log in, submit forms, or carry cookies between scans.
- Originates from Vercel serverless infrastructure (IP ranges vary; no fixed allowlist is provided).
- Identifies itself honestly via the User-Agent string above, which links back to this page so server admins can verify it.
How to block this bot
Block the User-Agent above at your web server (Nginx/Apache), CDN (Cloudflare/Vercel), or WAF. Because this bot acts on direct user request, blocking it will prevent your site from being analyzed by users of the AI Khazna tools — a fair choice, but worth knowing.
Contact
Questions, false-positive reports, or removal requests: office@aikhazna.com