If I want to see all the third-party requests included with a webpage, I have a few options (that I know of):
- Look at the code in the
<head>and before the
- Use a browser extension like Ghostery to see what’s being served
- Utilize services like Calibre or SpeedCurve for in-depth analysis
- Inspect the page—use developer tools (network & sources panel)
But if I want to itemize that information and make it portable (e.g., paste it into a spreadsheet for filtering or further analysis), the process is not as straightforward.
Save as HAR
From inside some browsers’ web inspectors (Chrome, Firefox, Edge), you can save a HAR (HTTP Archive format) file (further reading). Among other things, this captures all the requests a site is making. Note that HAR files can contain sensitive data, so save and share with care.
Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet.
You can learn a lot poking around the app.
Unfortunately, I’ve yet to find a way to export or copy this info into text or a table. (Update)
This approach (and level of analysis) gives me a good perception upgrade for what’s happening in a browser when third-party scripts, iframes, etc. load with a webpage. Next, I’ll be taking chunks of this information for multiple sites and running some comparisons to see what scripts/services are most common.
I’m figuring all this out as I go, so if you have advice or better ideas, let me know!
My friend Matt Weinberg from Vector Media Group informed me that you can export a full session from Charles app as a CSV file (File->Export Session->CSV), which is ideal. He also pointed out that “a HAR file is just a JSON file with inlined content. So any tool that can parse JSON can do whatever you need to with a HAR.” Bingo. Thanks, Matt!