The method to find them is simple: get the set of URLs you say exist (the sitemap), get the set of URLs your site actually links to (a crawl), and compare.

Why orphans matter

• Discovery: crawlers find most pages by following links. A sitemap entry helps, but a page with no internal links is a weak candidate for crawling and ranking.
• Signals: internal links pass context and importance. An orphan gets none.
• Waste: some orphans are pages that should have been removed, like an old pricing page with outdated numbers that's still indexed.

Step 1: collect sitemap URLs

import xml.etree.ElementTree as ET, requests

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
UA = {"User-Agent": "orphan-finder/1.0"}

def sitemap_urls(url):
    root = ET.fromstring(requests.get(url, headers=UA, timeout=20).content)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            yield from sitemap_urls(loc.text.strip())
    else:
        for loc in root.findall("sm:url/sm:loc", NS):
            yield loc.text.strip()

Step 2: crawl internal links

A small breadth-first crawler starting at the homepage, staying on your host:

from collections import deque
from html.parser import HTMLParser
from urllib.parse import urljoin, urldefrag, urlparse
import time

class Links(HTMLParser):
    def __init__(self):
        super().__init__(); self.hrefs = []
    def handle_starttag(self, tag, attrs):
        if tag == "a":
            href = dict(attrs).get("href")
            if href: self.hrefs.append(href)

def normalize(u):
    u, _ = urldefrag(u)
    p = urlparse(u)
    path = p.path or "/"
    return f"{p.scheme}://{p.netloc.lower()}{path}" + (f"?{p.query}" if p.query else "")

def crawl(start, limit=2000):
    host = urlparse(start).netloc
    seen, linked = {normalize(start)}, set()
    queue = deque([start])
    while queue and len(seen) <= limit:
        url = queue.popleft()
        try:
            r = requests.get(url, headers=UA, timeout=20)
        except requests.RequestException:
            continue
        if "text/html" not in r.headers.get("Content-Type", ""):
            continue
        p = Links(); p.feed(r.text)
        for href in p.hrefs:
            abs_url = normalize(urljoin(r.url, href))
            if urlparse(abs_url).netloc != host:
                continue
            linked.add(abs_url)
            if abs_url not in seen:
                seen.add(abs_url); queue.append(abs_url)
        time.sleep(0.3)   # be polite to your own server
    return linked

For larger sites, also respect robots.txt and skip obvious non-pages (/cart, /login, search result URLs with query strings) to keep the crawl focused.

Step 3: compare

import sys

sitemap = {normalize(u) for u in sitemap_urls(sys.argv[1])}
start = sys.argv[2]
linked = crawl(start)

orphans = sorted(sitemap - linked - {normalize(start)})
unlisted = sorted(u for u in linked - sitemap if "?" not in u)

print(f"{len(sitemap)} in sitemap, {len(linked)} linked internally\n")
print("Orphans (in sitemap, never linked):")
for u in orphans: print("  ", u)
print("\nLinked but missing from sitemap:")
for u in unlisted[:50]: print("  ", u)

Run it:

python orphans.py https://example.com/sitemap.xml https://example.com/

The second list is a useful bonus: pages your site links to that aren't in the sitemap, often new sections the sitemap generator doesn't know about.

Normalization is everything

Most false orphans come from URL mismatches, not real problems. If the sitemap lists https://example.com/blog/post/ and your links use /blog/post, a naive comparison calls it an orphan. Before trusting the results:

• Match the trailing-slash style your site uses.
• Lowercase the host (paths are case-sensitive, so leave them alone).
• Decide whether http:// links count; ideally your internal links use HTTPS already.
• Strip tracking parameters such as utm_* from crawled links.

If you see a large number of orphans, compare a few by hand first; it's usually a normalization rule.

What to do with real orphans

Go through the list and sort each page into one of three buckets:

1. Valuable and current: link to it from a relevant hub page, category or related articles. One or two contextual links from pages that themselves get traffic are enough.
2. Outdated or duplicate: redirect it to its replacement with a 301, and remove it from the sitemap.
3. Intentionally unlinked (campaign landing pages, thank-you pages): remove them from the sitemap, and add noindex if they shouldn't appear in search.

JavaScript navigation caveat

This crawler reads server-rendered HTML. If your menus or related-post widgets are rendered client-side, links that exist for visitors may be invisible here, producing false orphans. Spot-check a few with the browser, or render pages with a headless browser for the crawl.

Make it a habit

Orphans appear gradually, so a quarterly check works well. For the faster-moving signals on the pages themselves, SEOtto finds pages through your XML sitemap or internal links and alerts you when status codes, titles, canonicals or indexing directives change: seotto.com.

Summary

• Orphans = sitemap URLs with no internal links pointing to them.
• Crawl from the homepage, collect internal links, and diff against the sitemap.
• Normalize URLs carefully to avoid false positives.
• Link, redirect, or de-list each real orphan.

Related reading

• How to check HTTP to HTTPS and www redirects are set up right
• How to check Open Graph tags so link previews don't break
• Find duplicate and missing title tags with a Python crawl script

This guide covers the tags that matter, a script to audit them, and the caching quirks that make fixes look like they didn't work.

The tags that matter

<meta property="og:title" content="How to check HTTPS redirects">
<meta property="og:description" content="Test all four host and scheme variants in one minute.">
<meta property="og:image" content="https://example.com/og/https-redirects.png">
<meta property="og:url" content="https://example.com/blog/https-redirects">
<meta property="og:type" content="article">
<meta name="twitter:card" content="summary_large_image">

Rules that trip people up:

• og:image must be an absolute URL. /og/image.png doesn't work in most consumers.
• The image must be publicly reachable: no login, no hotlink protection, no robots.txt block on the image path.
• Size: 1200×630 pixels is the common recommendation for large cards. Very small images may be shown as a thumbnail or not at all.
• og:url should match the canonical URL, otherwise shares of the same page get split across URLs.
• One of each. Duplicate og:title tags (one from the theme, one from a plugin) leave the result up to each platform.

X (Twitter) falls back to Open Graph for title, description and image, but needs twitter:card to choose the large image layout.

Common failures

• Every page has the same og:title and og:image because the tags are hard-coded in the layout.
• Article pages have tags, but category or product pages fall back to the homepage values.
• The image URL points to a staging domain or a CDN path that changed.
• Tags are injected by JavaScript. Most preview crawlers don't run JavaScript, so they see nothing.
• The image file is several megabytes and times out in some crawlers.

Audit tags with a script

This script fetches the raw HTML (what preview crawlers see), reads the tags, and checks the image:

# og_check.py  usage: python og_check.py urls.txt
import sys, requests
from html.parser import HTMLParser

UA = {"User-Agent": "Mozilla/5.0 (og-check)"}
REQUIRED = ["og:title", "og:description", "og:image", "og:url"]

class Meta(HTMLParser):
    def __init__(self):
        super().__init__(); self.tags = {}; self.canonical = None
    def handle_starttag(self, tag, a):
        a = dict(a)
        if tag == "meta":
            key = a.get("property") or a.get("name")
            if key and key.startswith(("og:", "twitter:")):
                self.tags.setdefault(key, []).append(a.get("content", ""))
        if tag == "link" and a.get("rel") == "canonical":
            self.canonical = a.get("href")

def check(url):
    issues = []
    r = requests.get(url, headers=UA, timeout=20)
    m = Meta(); m.feed(r.text)
    for key in REQUIRED:
        values = m.tags.get(key, [])
        if not values or not values[0].strip():
            issues.append(f"missing {key}")
        elif len(values) > 1:
            issues.append(f"duplicate {key} ({len(values)}x)")
    img = (m.tags.get("og:image") or [""])[0]
    if img:
        if not img.startswith("http"):
            issues.append(f"og:image not absolute: {img}")
        else:
            ir = requests.get(img, headers=UA, timeout=20, stream=True)
            size = int(ir.headers.get("Content-Length") or 0)
            ctype = ir.headers.get("Content-Type", "")
            ir.close()
            if ir.status_code != 200:
                issues.append(f"og:image returns {ir.status_code}")
            elif not ctype.startswith("image/"):
                issues.append(f"og:image content-type is {ctype}")
            elif size > 5_000_000:
                issues.append(f"og:image is {size // 1_000_000} MB")
    og_url = (m.tags.get("og:url") or [""])[0]
    if m.canonical and og_url and og_url != m.canonical:
        issues.append(f"og:url {og_url} != canonical {m.canonical}")
    return issues, (m.tags.get("og:title") or [""])[0]

titles = {}
for line in open(sys.argv[1]):
    u = line.strip()
    if not u: continue
    issues, title = check(u)
    titles.setdefault(title, []).append(u)
    print(f"{u}\t{'; '.join(issues) or 'ok'}")

for title, urls in titles.items():
    if title and len(urls) > 1:
        print(f"\nSame og:title on {len(urls)} pages: '{title}'")

The last block flags pages sharing an identical og:title, which usually means a template is using a site-wide default.

Run it on one page per template

python og_check.py urls.txt

https://example.com/	ok
https://example.com/blog/https-redirects	ok
https://example.com/products/mug	missing og:description; og:image returns 404

Same og:title on 2 pages: 'Example Store'

Why fixes don't show up right away

Platforms cache previews, sometimes for days. After fixing tags:

• Facebook: use the Sharing Debugger and click "Scrape Again".
• LinkedIn: the Post Inspector refreshes its cache for the URL.
• Slack and Discord: cache per URL; adding a harmless query string (?v=2) shows the new preview for testing.

If the platform's debugger still shows old values, check that it's not hitting a cached version of your page at the CDN.

Keep previews from regressing

Theme updates and plugin changes are the usual culprits when previews break months after they were fixed. SEOtto includes Open Graph among the checks it runs on each crawl of your site, alongside titles, descriptions and canonicals, and alerts you when they change: seotto.com.

Summary

• Set og:title, og:description, absolute og:image, og:url and twitter:card.
• Render tags server-side; preview crawlers rarely run JavaScript.
• Check image status, type and size, and avoid duplicate tags.
• Use platform debuggers to clear cached previews.

Related reading

• How to check hreflang tags for errors (with a script)
• How to check HTTP to HTTPS and www redirects are set up right
• A weekly technical SEO check routine for agency client sites
• Why your XML sitemap suddenly has fewer URLs
• Did your meta descriptions change? Why it's worth noticing
• How to find orphan pages by comparing your sitemap to internal links

This guide shows how to test all four variants quickly, what correct looks like, and how to fix the common problems.

What "correct" looks like

Pick a canonical host (with or without www, it doesn't matter which) and make sure:

• All three non-canonical variants return 301 (or 308) directly to the canonical HTTPS URL.
• Each redirect is a single hop, not http://www → https://www → https://.
• The path and query string are preserved: http://example.com/pricing?plan=pro lands on https://example.com/pricing?plan=pro, not the homepage.
• The canonical URL returns 200.

Test all four variants with curl

#!/usr/bin/env bash
# redirect-check.sh  usage: ./redirect-check.sh example.com /some/path
domain=$1; path=${2:-/}
for url in "http://$domain$path" "http://www.$domain$path" "https://$domain$path" "https://www.$domain$path"; do
  echo "== $url"
  curl -s -o /dev/null -L --max-redirs 5 \
    -w '%{num_redirects} hop(s) -> %{url_effective} [%{http_code}]\n' "$url"
  curl -sI "$url" | grep -iE '^(HTTP|location)' | head -2
done

Example output for a healthy site:

== http://example.com/pricing
1 hop(s) -> https://example.com/pricing [200]
HTTP/1.1 301 Moved Permanently
Location: https://example.com/pricing
== http://www.example.com/pricing
1 hop(s) -> https://example.com/pricing [200]
...
== https://example.com/pricing
0 hop(s) -> https://example.com/pricing [200]

Always test with a real path, not just /. A common bug is a redirect rule that sends every variant to the homepage.

Problems you'll find

Two hops instead of one. http://www.example.com → https://www.example.com → https://example.com. It works, but each hop costs time and crawl budget. Usually the HTTPS redirect and the host redirect live in different layers (CDN and server) and run in sequence.

302 instead of 301. A temporary redirect says "the old URL is still the real one". For a permanent host or scheme change, use 301 or 308.

A variant serves content. https://www.example.com returns 200 with the full site instead of redirecting. Now two hosts serve identical pages. A canonical tag helps search engines choose, but a redirect is the clear signal.

Certificate errors on the non-canonical host. If the certificate only covers example.com, then https://www.example.com fails with a TLS error before any redirect can happen. Visitors who type www see a warning page. The certificate must cover every hostname you redirect from.

Path dropped. All variants land on /. Old links to deep pages lose their target.

Fix it in Nginx

Handle every non-canonical variant in one server block that redirects directly:

# Redirect all HTTP, and HTTPS www, to https://example.com
server {
    listen 80;
    listen [::]:80;
    server_name example.com www.example.com;
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name www.example.com;
    ssl_certificate     /etc/ssl/example.com/fullchain.pem;   # must include www
    ssl_certificate_key /etc/ssl/example.com/privkey.pem;
    return 301 https://example.com$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    http2 on;
    server_name example.com;
    # ... the real site ...
}

$request_uri keeps the path and query string. Using return rather than rewrite is simpler and faster.

Fix it in Apache (.htaccess)

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} ^www\. [NC]
RewriteRule ^ https://example.com%{REQUEST_URI} [L,R=301]

The two conditions combined with [OR] produce a single redirect for any non-canonical request. Behind a proxy or CDN that terminates TLS, %{HTTPS} is always off; check %{HTTP:X-Forwarded-Proto} instead or you'll create a loop.

If a CDN is in front

Decide which layer owns the redirects. If the CDN handles "always use HTTPS" and your origin handles www removal, you get two hops. Either put both rules in the CDN (a single redirect rule matching all variants) or both at the origin. Avoid duplicating rules in both places.

Add HSTS once it's stable

After the redirects are correct, an HSTS header tells browsers to use HTTPS directly on future visits, skipping the HTTP hop:

add_header Strict-Transport-Security "max-age=31536000" always;

Start with a short max-age while testing; browsers remember the setting, so mistakes are sticky.

Re-check after infrastructure changes

Redirect setups break when you move hosts, change CDNs, renew certificates or touch server configs. Running the script after such changes takes a minute. SEOtto also checks redirects (https, www/non-www) and SSL certificates on every crawl of your site and alerts when something changes: seotto.com.

Summary

• Test all four scheme and host variants, with a real path.
• Expect one 301 hop straight to the canonical HTTPS URL, path preserved.
• Make sure the certificate covers every hostname.
• Keep redirect rules in one layer, then add HSTS.

Related reading

• Find duplicate and missing title tags with a Python crawl script
• How to check hreflang tags for errors (with a script)
• A weekly technical SEO check routine for agency client sites
• Why your XML sitemap suddenly has fewer URLs
• Did your meta descriptions change? Why it's worth noticing
• How to find orphan pages by comparing your sitemap to internal links

Why a broken sitemap matters

Google can find pages through links, so a broken sitemap rarely deindexes a site. What it does is slow discovery: new pages, updated pages and pages with few internal links get found later, or not at all. On large or fast-changing sites, that delay costs traffic.

First, a quirk: "Couldn't fetch" is sometimes just "pending"

When you submit a new sitemap, Search Console sometimes shows "Couldn't fetch" while it simply hasn't tried yet. If the sitemap is brand new and passes every check below, wait a day or two before digging deeper. If it persists, or an existing sitemap flips to it, work through the list.

1. Does the URL return 200?

curl -sI https://example.com/sitemap.xml

You want HTTP/2 200 (or HTTP/1.1 200 OK). Common failures:

• 404 after a plugin change or migration. The sitemap moved, say from /sitemap.xml to /sitemap_index.xml, and the old URL is what you submitted.
• 301/302 to another URL. Google follows redirects, but redirect chains, especially to another host, cause trouble. Submit the final URL.
• 5xx or a timeout. Large generated sitemaps can time out on slow servers. Cache them as static files.
• 403 from a firewall or bot protection that treats Googlebot as a bot to block. Check your CDN or WAF rules.

2. Is it blocked by robots.txt?

If robots.txt disallows the sitemap's path, Google won't fetch it. Check for rules like Disallow: /sitemap or Disallow: /*.xml$. While you're there, add the sitemap location:

Sitemap: https://example.com/sitemap.xml

3. Is it really XML?

Open the sitemap URL with curl rather than a browser, which may hide problems:

curl -s https://example.com/sitemap.xml | head -5

You should see something like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

Red flags:

• An HTML page instead of XML (a login page, a cookie wall, a "not found" template served with 200).
• Whitespace or a PHP warning before <?xml, which makes the file invalid.
• Unescaped & in URLs. It must be written &.

4. Is it within the limits?

A single sitemap may contain at most 50,000 URLs and be at most 50 MB uncompressed. Bigger sites need a sitemap index that points to several sitemaps. Each child sitemap must pass the same checks, and a single broken child shows up as an error on the index.

5. Are the URLs in it the right ones?

A sitemap that loads fine can still be wrong:

• URLs on the wrong host or protocol (http:// or a staging domain).
• URLs that redirect, return 404, or are noindexed.
• URLs whose canonical points elsewhere.

Google treats these as noise. A sitemap should list only canonical, indexable pages that return 200.

6. The silent one: a sitemap that shrinks

This failure never shows an error. A deploy or plugin update changes how the sitemap is generated, and it goes from 4,000 URLs to 400, or to an empty <urlset>. The file still returns 200 and validates, so Search Console reports "Success". Only the "Discovered URLs" count tells the story, if anyone looks.

A quick way to count URLs:

curl -s https://example.com/sitemap.xml | grep -c "<loc>"

Record that number and compare it after each release.

After the fix

1. Confirm the URL returns 200 and valid XML.
2. In Search Console, open Sitemaps, remove the old entry if the URL changed, and submit the correct one.
3. Check back in a day or two for "Success" and a sensible "Discovered URLs" number.

Automate the boring part

Sitemaps break the same way robots.txt does: quietly, during a deploy. If you'd rather not run curl by hand after every release, an SEO monitor can load the sitemap on every crawl and count its URLs. SEOtto, for example, alerts when a sitemap fails to load, comes back empty, or loses 20% or more of its URLs. Whichever route you take, checking status code, format and URL count covers almost every "Couldn't fetch" you'll ever see.

Related reading

• Why your XML sitemap suddenly has fewer URLs
• How to check hreflang tags for errors (with a script)
• How to check Open Graph tags so link previews don't break
• Did your meta descriptions change? Why it's worth noticing