The request came back with a block page
On 3 October 2026, unauthenticated audit requests to molthub.click returned HTTP 403 for our homepage, robots.txt, sitemap and several public content pages. The homepage response displayed a Cloudflare block page. A separate web retrieval tool could not read the homepage, robots, sitemap or public experiment. We retained the status codes and scope instead of describing these requests as successful crawling.
An allow rule is one layer of evidence
Our site source allows public crawling. A robots policy expresses the site's rules; it does not establish that a request passes a CDN or security filter. A browser visit, an audit request and a verified search crawler can receive different responses. We have not inspected the triggering Cloudflare rule, so the cause remains unresolved.
We found a false pass in our own scanner
Our quick scan previously treated failure to retrieve robots.txt as absence of a blocking rule. That gave the check a passing score without evidence. We changed the result to not verified when retrieval fails, times out or returns an HTML page. Unverified checks receive no passing points, and the report states that a readable policy still cannot establish real crawler access.
Visibility is still a separate measurement
The earlier five-query observation on 2 September returned zero molthub mentions and zero citations in a public web-search observation. The HTTP review does not update those figures. We still need official index inspection and fresh, repeated AI-platform answers before reporting an improvement. Changing code alone would not support that claim.
What another team can check
Inspect an actual public response, identify the security rule if access fails, then verify the page with the relevant search platform's official inspection tool. Keep unknown results separate from failures and passes. Preserve the old query set when testing a clearer buyer-query panel, and archive answers with their dates and cited URLs. This is our recommended workflow, not a guarantee of inclusion.
Sources you can inspect
- molthub HTTP access observation, 3 October 2026
First-party request paths, status codes, method and limitations for this audit.
- molthub public self-GEO experiment
Historical five-query observation and the separate current access review.
- Google Search Central: AI features and your website
Primary guidance on crawling, indexing, text accessibility and eligibility. It does not establish our site's index status.
- Cloudflare: AI bot behavior
Primary explanation of search, agent and training behavior; not evidence of our triggering rule.
This is an observation of molthub's own website from one audit environment. It is not a client result or a cross-platform AI visibility test.
Apply this to your project