How to collect network traffic automatically with Selenium WebDriver

If you need to collect performance data and http web traffic between your browser and web server. You can use following 2 methods to make the process automatically with Selenium WebDriver.

  1. Log http traffic data into har files using Firebug and netExport Firefox add-on.
  2. Use BrowserMob Proxy as proxy server to route web page request and log http traffic data into har files.

No mater which method you choose, you need to know

What is har file?

HAR is abbreviation of HTTP Archive. You can make http web traffic visualized with this format files created by http communication tracking tools. This file include detail information about the communication between web browser and web server. You can also make analysis with all these information.

The har file format is similar with json, you can use  jackson or gson libraries to parse the http web communication data in the log har file.

Use Firebug and netExport to capture web traffic
  1. Set up java project with eclipse if you do not create one. Please refer following topic.
    How to create java project with myeclipse
    How to fix java Unsupported major.minor version 51.0 Exception
  2. Add selenium webdriver jar library into your java project if you do not do this. Please refer following topic.
    How to add selenium server standalone jar file into your java projectI prefer selenium-server-standalone-2.53.0.jar and Firefox 45.0.1.
  3.  Download and add Firebug add-on in Firefox 45.0.1.
    After add Firebug.1) Input following command in windows start run menu.
    %APPDATA%\Mozilla\Firefox\Profiles\find Firefox profile folder2) Copy the Firebug xpi file from your Firefox profile folder like following.
    C:\Users\zhaosong\AppData\Roaming\Mozilla\Firefox\Profiles\yn80ouvt.default\extensions\[email protected]
    Firebug xpi file3) Save the xpi file into C:\Workspace\\Lib.
  4. Download netExport Firefox extension.
    I prefer netExport-0.9b3.xpiSave the download file to C:\Workspace\\Lib

Now the preparation is done, we will show you how to coding in java.

  1. Create the Firefox profile which will be used by Selenium WebDriver and add Firebug and netExport extension to the code to create firefox profile and add firebug netExport add-on in webdriver
  2. Set Firebug and netExport preferences to enable Firebug extension to capture http web traffic and save data into har formated log file.
    java code to set firebug and netExport add-on's preferences
  3. Initial Firefox browser using Selenium code to initialize firefox webdriver
  4. Capture http web traffic to har log file under C:\\Workspace\\\\ code to capture http web traffic into har format log file
Use BrowserMob Proxy to capture web traffic

It is very easy to collect http communication information from browsers use BrowserMob Proxy. It can be used as a standalone or embedded proxy server. The proxy server also save the communication data in HAR format. The code below will use BrowserMob as a standalone proxy server to capture http traffic.

  1. Download
  2. Unzip the downloaded zip file to C:\Workspace\\Lib.
  3. Add browsermob-dist-2.1.4.jar from following directory to your java project libraries.
    C:\Workspace\\Lib\browsermob-proxy-2.1.4\lib\ browsermob-dist-2.1.4.jar
    add browsermob-dist-2.1.4.jar to java project
  4. Go to C:\Workspace\\Lib\browsermob-proxy-2.1.4\bin in command console, and start BrowserMobProxy server by run browsermob-proxy.bat.
    start browsermob proxy server use command line
  5.  Create a new class
    creat a new java class CaptureTrafficWithBrowserMobProxy
  6. Initiate proxy server
    initiate proxy server
  7. Create firefox webdriver using the running proxy server
    create firefox webdriver using the running proxy server
  8. Use firefox webdriver browse page and log the traffic in har format.
    use firefox webdriver browse page and log the traffic in har format

How to  visualize the har file.

You can use chrome extension har file viewer to visualize the content in the generated har file.
har file in google har file viewer extension

[download id=”202″]


2 thoughts on “How to collect network traffic automatically with Selenium WebDriver”

  1. Hi, I downloaded and ran your code, however the result I see in the har file is very very slim!
    Only 1 request to the website and then response is 301 – and no further resources… I get this both when using your original code / resetting the driver.get to any other site (example: driver.get(“”)).

    {“log”:{“version”:”1.2″,”creator”:{“name”:”BrowserMob Proxy”,”version”:”2.1.4-legacy”,”comment”:””},”pages”:[{“id”:”12″,”startedDateTime”:”2020-01-12T21:34:49.517Z”,”title”:”12″,”pageTimings”:{“comment”:””},”comment”:””}],”entries”:[{“pageref”:”12″,”startedDateTime”:”2020-01-12T21:34:49.558Z”,”request”:{“method”:”GET”,”url”:””,”httpVersion”:”HTTP/1.1″,”cookies”:[],”headers”:[],”queryString”:[],”headersSize”:425,”bodySize”:0,”comment”:””},”response”:{“status”:302,”statusText”:”Found”,”httpVersion”:”HTTP/1.1″,”cookies”:[],”headers”:[],”content”:{“size”:231,”mimeType”:”text/html; charset=UTF-8″,”comment”:””},”redirectURL”:””,”headersSize”:377,”bodySize”:231,”comment”:””},”cache”:{},”timings”:{“comment”:””,”blocked”:7,”dns”:57,”connect”:48,”send”:0,”wait”:79,”receive”:37,”ssl”:-1},”serverIPAddress”:”″,”comment”:””,”time”:230}],”comment”:””}}

    Any idea what’s going on?

  2. Hi While I am using browser mob proxy and executing class getting error message as “ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.”

    Please help me on this

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.