How to collect network traffic automatically with Selenium WebDriver

If you need to collect performance data and http web traffic between your browser and web server. You can use following 2 methods to make the process automatically with Selenium WebDriver.

  1. Log http traffic data into har files using Firebug and netExport Firefox add-on.
  2. Use BrowserMob Proxy as proxy server to route web page request and log http traffic data into har files.

No mater which method you choose, you need to know

What is har file?

HAR is abbreviation of HTTP Archive. You can make http web traffic visualized with this format files created by http communication tracking tools. This file include detail information about the communication between web browser and web server. You can also make analysis with all these information.

The har file format is similar with json, you can use  jackson or gson libraries to parse the http web communication data in the log har file.

Use Firebug and netExport to capture web traffic
  1. Set up java project with eclipse if you do not create one. Please refer following topic.
    How to create java project with myeclipse
    How to fix java Unsupported major.minor version 51.0 Exception
  2. Add selenium webdriver jar library into your java project if you do not do this. Please refer following topic.
    How to add selenium server standalone jar file into your java projectI prefer selenium-server-standalone-2.53.0.jar and Firefox 45.0.1.
  3.  Download and add Firebug add-on in Firefox 45.0.1.
    https://addons.mozilla.org/en-US/firefox/addon/firebug/
    After add Firebug.1) Input following command in windows start run menu.
    %APPDATA%\Mozilla\Firefox\Profiles\find Firefox profile folder2) Copy the Firebug xpi file from your Firefox profile folder like following.
    C:\Users\zhaosong\AppData\Roaming\Mozilla\Firefox\Profiles\yn80ouvt.default\extensions\[email protected]
    Firebug xpi file3) Save the xpi file into C:\Workspace\dev2qa.com\Lib.
  4. Download netExport Firefox extension.
    https://getfirebug.com/releases/netexport/
    I prefer netExport-0.9b3.xpiSave the download file to C:\Workspace\dev2qa.com\Lib
READ :   How to use Firefox webdriver to launch Firefox browser automatically in java

Now the preparation is done, we will show you how to coding in java.

  1. Create the Firefox profile which will be used by Selenium WebDriver and add Firebug and netExport extension to the profile.java code to create firefox profile and add firebug netExport add-on in webdriver
  2. Set Firebug and netExport preferences to enable Firebug extension to capture http web traffic and save data into har formated log file.
    java code to set firebug and netExport add-on's preferences
  3. Initial Firefox browser using Selenium WebDriver.java code to initialize firefox webdriver
  4. Capture http web traffic to har log file under C:\\Workspace\\dev2qa.com\\NetworkTraffic.java code to capture http web traffic into har format log file
Use BrowserMob Proxy to capture web traffic

It is very easy to collect http communication information from browsers use BrowserMob Proxy. It can be used as a standalone or embedded proxy server. The proxy server also save the communication data in HAR format. The code below will use BrowserMob as a standalone proxy server to capture http traffic.

  1. Download browsermob-proxy-2.1.4-bin.zip
  2. Unzip the downloaded zip file to C:\Workspace\dev2qa.com\Lib.
  3. Add browsermob-dist-2.1.4.jar from following directory to your java project libraries.
    C:\Workspace\dev2qa.com\Lib\browsermob-proxy-2.1.4\lib\ browsermob-dist-2.1.4.jar
    add browsermob-dist-2.1.4.jar to java project
  4. Go to C:\Workspace\dev2qa.com\Lib\browsermob-proxy-2.1.4\bin in command console, and start BrowserMobProxy server by run browsermob-proxy.bat.
    start browsermob proxy server use command line
  5.  Create a new class CaptureTrafficWithBrowserMobProxy.java
    creat a new java class CaptureTrafficWithBrowserMobProxy
  6. Initiate proxy server
    initiate proxy server
  7. Create firefox webdriver using the running proxy server
    create firefox webdriver using the running proxy server
  8. Use firefox webdriver browse page and log the traffic in har format.
    use firefox webdriver browse page and log the traffic in har format

How to  visualize the har file.

You can use chrome extension har file viewer to visualize the content in the generated har file.
har file in google har file viewer extension

Download “Source code for how to capture web traffic with webdriver” Source-code-for-how-to-capture-web-traffic-with-webdriver-.zip – Downloaded 299 times – 2 KB

 

(Visited 7,618 times, 25 visits today)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.