ChromeDriver for Data Scraping

· 2 min

Selenium

Selenium is my go-to for automated frontend tests. It gives your code an interface for driving a browser.

ChromeDriver implements Selenium's WebDriver protocol for Chrome. Think of it as three blocks:

There are implementations for other browsers, such as Firefox and Edge, which is handy for cross-browser testing.

But for data scraping?

By data scraping, I mean getting information from web pages programmatically: parsing the HTML for that sweet data. RESTful APIs mean we need to do it less often, but it's still needed from time to time.

Usually you request a page, get the markup back and parse it, which is trivial in most languages. In C#:

using(var wc = new WebClient())
{
    var html = wc.DownloadString("http://somewebsite.com/some-page.html");
}

Then use a library such as HtmlAgilityPack for parsing and selectors.

However, many web apps now use JavaScript frameworks such as Angular, React and Vue to load their content client-side, using the browser's JavaScript engine (V8, in Chrome). Without running the JavaScript, you get the markup without the content. You're visiting a house, but no one is home.

Some popular websites, such as Amazon, also deliberately obfuscate their markup and rely on JavaScript to build the page, to stop their data being mined.

Solving the problem

This is where the automated testing tool comes in. ChromeDriver drives a real Chrome, so in theory it can render the target page, JavaScript included, and let us read the data we want.

Trying it out

This example uses amazon.co.uk. Data scraping is a grey area and likely against their terms of service, but it's a good educational example of otherwise unobtainable data.

  1. Install Chrome if you don't have it.
  2. Download ChromeDriver.

Next, create a simple console application:

Add the Selenium.WebDriver package from the NuGet window:

Use Selenium to start ChromeDriver and tell Chrome to browse to a product page:

Pass the ID of the element holding the data you want to FindElementById. The finished demo:

The result:

Starting Chrome and rendering every page adds a lot of overhead, so this isn't better than a code-only approach in general. For heavily dynamic web apps, though, it's sometimes the only way in.