arrow_back Back to Insights
SEO Engineering · 6 Min Read

XML Sitemaps and robots.txt: Getting the Basics Right

Two small files control how search engines and AI crawlers see your entire website. Here's how to set up sitemap.xml and robots.txt correctly, and the mistakes that quietly block indexing.

XML Sitemaps and robots.txt: Getting the Basics Right — Cover image

Two of the simplest files on your website carry a disproportionate amount of control over what search engines and AI crawlers actually see. Get them wrong, and even a perfectly built website can go effectively unindexed.

What robots.txt Does

robots.txt sits at your domain root (yourdomain.com/robots.txt) and tells crawlers which parts of your site they're allowed to access. It's a request, not an enforcement mechanism — well-behaved crawlers (Google, Bing) respect it; malicious bots often ignore it entirely.

A basic, safe example:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://www.spwebsmiths.com/sitemap.xml

The Mistake That Silently Kills SEO

The single most damaging robots.txt error is an accidental blanket disallow left over from a staging environment:

User-agent: *
Disallow: /

This blocks crawlers from your entire site. It's shockingly common — a developer sets this during development to prevent an unfinished site from being indexed, then forgets to remove it at launch. We check for this on every audit, because it's the difference between "ranking poorly" and "not existing in search results at all."

What sitemap.xml Does

Where robots.txt restricts, sitemap.xml invites — it's a structured list of every URL you want indexed, along with metadata like last-modified dates. It doesn't guarantee indexing, but it makes discovery far more reliable, especially for pages with few internal links pointing to them.

A well-formed sitemap entry looks like:

<url>
  <loc>https://www.spwebsmiths.com/services/website-development/</loc>
  <lastmod>2026-07-02</lastmod>
  <changefreq>monthly</changefreq>
</url>

Common Sitemap Mistakes

  • Including pages that are also blocked by robots.txt (contradictory signals)
  • Including redirected or 404'd URLs that no longer exist
  • Never updating it as new pages are added — a stale sitemap undersells a growing site
  • Forgetting to submit it to Bing Webmaster Tools, not just Google Search Console (this matters more than ever given Bing's role in feeding ChatGPT Search)

Submission Checklist

  1. Generate sitemap.xml (most CMS platforms and static site generators can automate this)
  2. Reference it in robots.txt
  3. Submit it manually in Google Search Console → Sitemaps
  4. Submit it in Bing Webmaster Tools as well
  5. Re-check submission status after a few days to confirm no errors

The Bottom Line

These two files take minutes to set up correctly, but errors in either one are among the most common reasons we find that a well-designed website simply isn't showing up in search. Check both before assuming a ranking problem is about content or backlinks — sometimes it's this simple.

Want us to check your robots.txt and sitemap for free? Get in touch.