User-agent: * Allow: / # Real bug fixed (2026-09-01, admin/checkout/account.html -> .php # migration, full-site SEO/security audit follow-up): admin.html, # checkout.html, and account.html are now redirect stubs — the real pages # live at admin.php/checkout.php/account.php, which weren't covered by # any Disallow rule at all until this line. Keeping both old and new # blocked: the old .html paths still exist as real, fetchable URLs (just # redirect stubs now) so there's no harm leaving them listed too. Disallow: /admin.html Disallow: /admin.php # ── /api/ (rewritten 2026-09-24 — this was actively harming indexing) ── # A blanket `Disallow: /api/` looks like good hygiene and was anything but. # Every catalogue page on this site builds its product grid client-side # from these endpoints, and Googlebot's renderer obeys robots.txt for the # fetches a page makes while rendering. Blocking /api/ therefore blocked # Google from the very data the page needs, so it rendered an empty grid # and scored working pages as Soft 404s — confirmed when a manual indexing # request for hot-wheels-india.php (a page showing 49 real products in a # browser) was rejected as one. # # So: the read-only endpoints a page needs in order to render itself are # now crawlable, and everything carrying personal data, session state or # write actions stays blocked. Crawlable is not the same as indexable — # every /api/ response also sends `X-Robots-Tag: noindex, nofollow` (see # cors_headers() in api/helpers.php), so Google may fetch this JSON to # render a page but will never list it as a search result. That header is # what makes this safe, and it is why the earlier "/api/index.php/products # — Crawled, currently not indexed" entries won't come back. # # Google resolves conflicts by longest match, so each Allow below beats # the broad Disallow for its own path. Disallow: /api/ # Private endpoints, listed explicitly so the intent survives anyone # editing the Allow list below (2026-10-11, GSC crawled # /api/index.php/otp/* and /customers/*). The broad Disallow above already # blocks them — none of the Allow prefixes below matches these paths — but # the longest-match rule means a future Allow like /api/index.php/c would # silently re-open them, so they are pinned shut here too. Disallow: /api/index.php/otp Disallow: /api/index.php/customers Disallow: /api/index.php/auth Disallow: /api/index.php/orders Disallow: /api/index.php/cart Disallow: /api/index.php/admin Allow: /api/index.php/products Allow: /api/index.php/categories Allow: /api/index.php/brands Allow: /api/index.php/reviews Allow: /api/index.php/settings # content/ = story, testimonials and community posts, which the homepage # builds client-side (added 2026-10-16: they were blocked for the renderer). Allow: /api/index.php/content # flash-sales: ONLY the public banner. The bare /flash-sales route is the # admin list and answers 401 to everyone else (2026-10-16) — it was in this # Allow list, so Googlebot could fetch it and Search Console reported a # "Blocked due to unauthorized request (401)". The pages only ever call # /flash-sales/banner. Allow: /api/index.php/flash-sales/banner # Admin-only routes that sit UNDER an Allow prefix above and answer 401 to # anyone not signed in to the admin. Longer than their Allow, so these win. Disallow: /api/index.php/products/export Disallow: /api/index.php/products/*/audit Disallow: /api/index.php/reviews?all Disallow: /checkout.html Disallow: /checkout.php Disallow: /account.html Disallow: /account.php Disallow: /order-success.html # Pre-order balance payment page (2026-10-22) — private, per-order, emailed # to the customer. Also sends X-Robots-Tag: noindex and a robots meta tag. Disallow: /pay-balance.php Disallow: /invoice.html # Liveness probe for uptime monitoring (2026-09-24) — an operational # endpoint that returns "ok"/"db", not a page. It also sends # X-Robots-Tag: noindex, so this is the belt to that braces. Disallow: /health.php # Error page template (2026-10-11): always answers 404 + noindex; nothing # links to it, and it must never be fetched as if it were content. Disallow: /404.php # _templates/ (2026-09-16, GSC audit follow-up) — private raw markup # sources for shop.php/drops.php/index.php/category-page.php, never meant # to be a real page. _templates/.htaccess already Denies direct HTTP # access, but that's an unconfirmed mechanism on this host (see its own # comment) — this is the second, independent layer: even if the .htaccess # Deny doesn't fire, nothing here should ever be crawled or indexed. Disallow: /_templates/ # admin-guide assets (2026-09-16) — screenshots of the admin portal # (customer PII already blurred, but no reason for this to be crawlable or # show up in search either) and the admin guide PDF, both embedded in the # Admin → Help Guide tab. Nothing on any public page links to these. Disallow: /assets/img/admin-guide/ Disallow: /assets/docs/ # The one sitemap (2026-10-16). /sitemap.xml is the public URL — the # .htaccess rewrites it to sitemap.php, and a direct request for # /sitemap.php is 301'd to /sitemap.xml, so there is exactly one. Only clean, # canonical, 200, indexable URLs are listed (see sitemap.php's header). # (History: on 2026-08-31 this line pointed at sitemap.php because the # sitemap.xml rewrite was not being honoured on this host and returned the # homepage; that is confirmed working now — check with # `curl -sI https://minibeesandwheels.com/sitemap.xml` (200, application/xml).) Sitemap: https://minibeesandwheels.com/sitemap.xml