Ndtyjky

I'm trying to get browser network logs using selenium to debug request/responses. Could you please help me to find out a way.

And I'm using selenium 3.14.0 and latest Chrome browser.

Using Python and ChromeDriver

To get network logs, you need to install BrowserMobProxy as well along with selenium in python

pip install browsermob-proxy

You need to start browser proxy and configure the proxy in chrome option of chrome driver,

from browsermobproxy import Server

from selenium import webdriver



server = Server("path/to/browsermob-proxy")

server.start()

proxy = server.create_proxy()



# Configure the browser proxy in chrome options

chrome_options = webdriver.ChromeOptions()

chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))

browser = webdriver.Chrome(chrome_options = chrome_options)



#tag the har(network logs) with a name

proxy.new_har("google")

Then you can navigate to page using selenium

driver.get("http://www.google.co.in")

After navigation, you can get the network logs in json format from the proxy

print(proxy.har) # returns a Network logs (HAR) as JSON 

Also before quitting the driver, stop the proxy server as well at the end,

server.stop()

browser.quit()

To get only the network logs up until the page has finished loading (no ajax/async network logs during the main usage of the page), you can get the Performance Log: http://chromedriver.chromium.org/logging/performance-log

To enable the Performance Logging for the ChromeDriver, for example,

DesiredCapabilities cap = DesiredCapabilities.chrome();

LoggingPreferences logPrefs = new LoggingPreferences();

logPrefs.enable(LogType.PERFORMANCE, Level.ALL);

cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);

RemoteWebDriver driver = new RemoteWebDriver(new URL("http://127.0.0.1:9515"), cap);

The chromium performance-log page also links to this complete example https://gist.github.com/klepikov/5457750 which has Java and python code to get the Performance Logs.

Again, it's important to keep in mind that this will only get the network requests up until the point that the page is finished loading. After that, the driver will only return the same performance logs until the page reloads.

If you want to get network logs asynchronously throughout the usage of the page, you can use BrowserMobProxy to act as a proxy server for your Selenium driver and capture all those network requests. Then, you can get those captured requests from BrowserMobProxy's generated HAR file: https://github.com/lightbody/browsermob-proxy#using-with-selenium

// start the proxy

BrowserMobProxy proxy = new BrowserMobProxyServer();

proxy.start(0);



// get the Selenium proxy object

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);



// configure it as a desired capability

DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);



// start the browser up

WebDriver driver = new FirefoxDriver(capabilities);



// enable more detailed HAR capture, if desired (see CaptureType for the complete list)

proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);



// create a new HAR with the label "yahoo.com"

proxy.newHar("yahoo.com");



// open yahoo.com

driver.get("http://yahoo.com");



// get the HAR data

Har har = proxy.getHar();

Once you have the HAR file, it is a JSON like list of network events that you can work with.