How to Find All Existing and Archived URLs on a web site

How to Find All Existing and Archived URLs on a web site

Blog Article

There are plenty of explanations you may need to seek out all the URLs on a website, but your exact purpose will decide what you’re looking for. For instance, you might want to:

Identify each and every indexed URL to analyze issues like cannibalization or index bloat
Collect current and historic URLs Google has viewed, specifically for web site migrations
Discover all 404 URLs to Get better from post-migration errors
In Each and every situation, a single Software gained’t Offer you every little thing you'll need. Unfortunately, Google Research Console isn’t exhaustive, and a “web-site:example.com” search is limited and tough to extract facts from.

During this post, I’ll wander you through some equipment to construct your URL record and right before deduplicating the information employing a spreadsheet or Jupyter Notebook, determined by your site’s sizing.

Previous sitemaps and crawl exports
If you’re on the lookout for URLs that disappeared from your Stay site not too long ago, there’s a chance a person with your staff could possibly have saved a sitemap file or maybe a crawl export prior to the alterations had been made. For those who haven’t previously, look for these data files; they're able to normally give what you need. But, in case you’re looking at this, you almost certainly didn't get so lucky.

Archive.org
Archive.org
Archive.org is a useful Resource for Search engine optimisation duties, funded by donations. In the event you search for a website and choose the “URLs” alternative, you are able to entry up to 10,000 shown URLs.

However, There are several limitations:

URL limit: It is possible to only retrieve as many as web designer kuala lumpur 10,000 URLs, that's insufficient for greater websites.
Quality: Lots of URLs could possibly be malformed or reference useful resource information (e.g., visuals or scripts).
No export alternative: There isn’t a crafted-in solution to export the listing.
To bypass The dearth of the export button, make use of a browser scraping plugin like Dataminer.io. Having said that, these limits necessarily mean Archive.org might not give a complete Alternative for larger sized web-sites. Also, Archive.org doesn’t show regardless of whether Google indexed a URL—however, if Archive.org discovered it, there’s a fantastic opportunity Google did, way too.

Moz Professional
Whilst you may perhaps usually use a link index to locate exterior internet sites linking for you, these tools also explore URLs on your internet site in the process.


Ways to use it:
Export your inbound inbound links in Moz Professional to get a swift and straightforward listing of concentrate on URLs from the site. Should you’re working with a huge Web page, consider using the Moz API to export information past what’s workable in Excel or Google Sheets.

It’s imperative that you Observe that Moz Pro doesn’t affirm if URLs are indexed or found out by Google. Nonetheless, due to the fact most internet sites apply precisely the same robots.txt rules to Moz’s bots because they do to Google’s, this technique usually is effective nicely for a proxy for Googlebot’s discoverability.

Google Look for Console
Google Look for Console provides numerous precious resources for developing your list of URLs.

Hyperlinks reviews:


Comparable to Moz Professional, the Links area gives exportable lists of concentrate on URLs. Regretably, these exports are capped at one,000 URLs each. You can utilize filters for particular pages, but since filters don’t implement towards the export, you would possibly must rely on browser scraping applications—limited to 500 filtered URLs at any given time. Not perfect.

General performance → Search Results:


This export provides you with an index of internet pages getting research impressions. Though the export is limited, You can utilize Google Research Console API for much larger datasets. Additionally, there are cost-free Google Sheets plugins that simplify pulling a lot more in depth info.

Indexing → Webpages report:


This section provides exports filtered by concern kind, although these are generally also limited in scope.

Google Analytics
Google Analytics
The Engagement → Pages and Screens default report in GA4 is a wonderful supply for accumulating URLs, that has a generous limit of a hundred,000 URLs.


Better yet, you may use filters to generate distinct URL lists, efficiently surpassing the 100k Restrict. One example is, if you need to export only blog URLs, observe these actions:

Stage 1: Incorporate a segment on the report

Step 2: Simply click “Develop a new segment.”


Step three: Define the phase that has a narrower URL pattern, such as URLs that contains /website/


Note: URLs present in Google Analytics may not be discoverable by Googlebot or indexed by Google, but they supply useful insights.

Server log data files
Server or CDN log files are Probably the final word Instrument at your disposal. These logs capture an exhaustive list of each URL route queried by people, Googlebot, or other bots during the recorded period.

Factors:

Info dimension: Log files can be large, lots of web pages only retain the last two months of data.
Complexity: Analyzing log data files is often tough, but various resources can be found to simplify the method.
Combine, and great luck
As soon as you’ve gathered URLs from each one of these resources, it’s time to combine them. If your web site is small enough, use Excel or, for larger datasets, applications like Google Sheets or Jupyter Notebook. Guarantee all URLs are regularly formatted, then deduplicate the checklist.

And voilà—you now have a comprehensive list of latest, aged, and archived URLs. Good luck!