How to Extract URLs in Bulk From a Site That May Be Paywalled

How to Extract URLs in Bulk From a Site That May Be Paywalled You need a list of article URLs, product pages, or document links from a site — but the content might be behind a paywall, login, or access restriction. Here's how to get the URLs (even if not the full content) without hitting paywalls on every request. Strategy: Sitemap First, Crawl Second Most sites publish their URL structure in sitemaps even when the content is paywalled. This is free — no login needed: import requests from xml.etree import ElementTree import urllib.parse def get_sitemap_urls(domain): """Extract all URLs from a site's sitemap""" urls = [] # Try common sitemap locations sitemap_paths = [ "/sitemap.xml", "/sitemap_index.xml", "/sitemaps/sitemap.xml", "/news-sitemap.xml", ] headers = {"User-Agent": "Mozilla/5.0 (compatible; SitemapBot/1.0)"} for path in sitemap_paths: url = f"https://{domain}{path}" r = requests.get(url, headers=headers, timeout=10) if r.status_code != 200: continue # Parse XML sitemap root