Generate a Sitemap.xml for Better Search Indexing

You shipped a site, Google indexed the homepage, and forty other pages sat untouched for three weeks. That is the usual reason people go looking for how to generate a sitemap.xml for better search indexing. A sitemap does not force Google to index anything, but it removes the guesswork about which URLs exist, which ones changed, and which ones you consider canonical.

The rest of this post covers the file format, how to build one by hand and by script, how to validate sitemap.xml before submitting, and where robots.txt fits.

What Is a Sitemap.xml File

A sitemap.xml file is a list of URLs you want crawlers to know about, wrapped in a specific XML schema. Nothing more exotic than that.

Here is a minimal, valid example:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-07-21</lastmod>
  </url>
  <url>
    <loc>https://example.com/pricing</loc>
    <lastmod>2026-07-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/first-post</loc>
    <lastmod>2026-06-30</lastmod>
  </url>
</urlset>

Four rules govern the file, and breaking any of them costs you indexing.

The xmlns declaration must match the sitemaps.org schema URL exactly. A typo there invalidates the whole document.

Every <loc> must be a fully qualified, absolute URL. The parser rejects relative paths.

One sitemap holds at most 50,000 URLs and 50 MB uncompressed. Past that, split it and use a sitemap index.

Special characters in URLs need XML escaping. An ampersand in a query string becomes & or the parser fails.

Google ignores <priority> and largely ignores <changefreq>. Skip both. <lastmod> does get used, but only if it is accurate. Stamping every URL with today's date on every build teaches the crawler to distrust the field.

Generate a Sitemap.xml for Better Search Indexing in Three Steps

For a site under a few hundred pages, a browser tool is faster than wiring up a build step.

Open the XML Sitemap Generator and enter your root domain. The tool crawls the reachable pages and assembles the urlset for you.

Review the URL list before you export. Look specifically for staging hosts, URLs with tracking parameters, paginated archive pages, and tag or category pages that duplicate content you already list. Remove them.

Export the file, drop it at your web root so it resolves at https://example.com/sitemap.xml, and confirm it loads in a browser as XML rather than downloading as plain text. A wrong Content-Type header will not block Google, but it makes debugging harder.

A quick sanity check that catches most problems: count the URLs in the sitemap and compare against the number of pages you expect to rank. If your site has 40 real pages and the sitemap lists 380, something is generating URL variants you did not intend.

Sitemap.xml vs Robots.txt: Two Different Jobs

People conflate these constantly, so be precise about it.

Robots.txt tells crawlers where they may not go. It is a set of exclusion rules, and it is the wrong place to hide anything sensitive since the file is public.

Sitemap.xml tells crawlers what exists and is worth fetching. It is an inclusion hint, not a command.

The two interact in one way that trips people up. If robots.txt blocks a URL from your sitemap, it won't get indexed with content. Google sees the URL, respects the block, and may show a bare listing with no description. Check the robot.txt file for SEO conflicts whenever your sitemap URLs go unindexed.

Point crawlers at the sitemap from robots.txt with one line:

User-agent: *
Disallow: /admin/
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

The Sitemap: directive is absolute and sits outside any User-agent group. Run your file through the robots.txt checker to confirm the directive parses and none of your sitemap URLs fall under a Disallow rule.

Validate Sitemap.xml Before Submitting

Search Console rejects malformed sitemaps with terse errors like "Couldn't fetch" that tell you nothing about the actual cause. Catch problems locally first.

Run the file through the sitemap XML validator. It checks schema conformance, absolute URL format, escaping, and the 50,000 URL ceiling.

Four failures account for most rejections.

Unescaped ampersands from query strings. ?a=1&b=2 must be written ?a=1&b=2.

Mixed protocols or hosts. If your canonical is https://www.example.com, every <loc> must use that exact host, not the apex domain or http.

A byte order mark before the XML declaration. Some editors add one invisibly, and strict parsers reject it.

Malformed lastmod values. Use 2026-07-21 or a full W3C datetime. 07/21/2026 fails.

For general XML shape problems, the XML formatter will pretty print the document and surface the exact line where a tag went unclosed.

How to Submit Your Sitemap to Google Search Console

Once the file validates and resolves publicly, submission takes under a minute.

Open Search Console and select the property that matches your sitemap's protocol and host. A https://www. property will not accept a sitemap served from the apex domain.

Go to Indexing, then Sitemaps. Enter the path, usually sitemap.xml, and submit.

Check back in 24 to 72 hours. Status "Success" with a discovered URL count means the file parsed. That count is URLs read, not URLs indexed. For indexing progress, use the Pages report instead.

Bing Webmaster Tools accepts the same file at the same path. Submit there too; it costs nothing.

You can also ping search engines from CI after a deploy:

curl -sS "https://www.google.com/ping?sitemap=https://example.com/sitemap.xml"

Treat that as a nudge rather than a guarantee. Google has deprecated the ping endpoint for most purposes and relies on regular recrawls of the sitemap URL it already knows about.

Generate a Sitemap.xml for Better Search Indexing at Scale

Once your page count moves and changes often, generate the file during your build instead of clicking through a UI.

import { writeFileSync } from "node:fs";

const BASE = "https://example.com";

// Replace with your real route source: filesystem walk, CMS query, DB rows.
const pages = [
  { path: "/", updated: "2026-07-21" },
  { path: "/pricing", updated: "2026-07-14" },
  { path: "/blog/first-post", updated: "2026-06-30" },
];

const esc = (s) =>
  s.replace(/&/g, "&")
   .replace(/</g, "&lt;")
   .replace(/>/g, "&gt;")
   .replace(/"/g, "&quot;");

const body = pages
  .map(({ path, updated }) =>
    `  <url>\n    <loc>${esc(BASE + path)}</loc>\n    <lastmod>${updated}</lastmod>\n  </url>`
  )
  .join("\n");

writeFileSync(
  "public/sitemap.xml",
  `<?xml version="1.0" encoding="UTF-8"?>\n` +
  `<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n${body}\n</urlset>\n`
);

console.log(`wrote ${pages.length} urls`);

Two details in that script matter more than the rest. The esc helper prevents the ampersand failure described above. And updated comes from real page data, such as a git commit date or a CMS modified field, rather than new Date(). Accurate lastmod values are what make the field useful to a crawler.

Past 50,000 URLs, emit multiple files and a sitemap index that references them, then submit only the index.

Keep the Sitemap Honest After Launch

A stale sitemap is worse than none, because it burns crawl budget on URLs that 404 or redirect.

Regenerate on every deploy that changes routes. Drop URLs the moment you delete or redirect a page. Exclude anything noindex, since listing a page you tell Google not to index is a contradiction. Review the Search Console sitemap report monthly for a growing error count.

Generate a sitemap.xml for better search indexing, validate it before you submit, and keep robots.txt out of its way. That combination fixes the majority of "Google will not index my pages" problems that have nothing to do with content quality.

Ready to build yours? Use the free XML Sitemap Generator to create the file in your browser, then run it through the sitemap XML validator before you submit it to Search Console. No account, no upload, no waiting.