Trying to scrape display URL's from Google SERPs

Support
#1

I cannot seem to scrape the display URL from the google SERPs using XPathOnURL. I know that I can use the Google Search Connector, but I don't see any way to upload a bulk list of keywords to check for - I can only do one keyword at a time. Any suggestions or help would be appreciated.

I tried using Google "inspect" tool to get the Xpath, but that just returns an empty cell. I've also tried using the chrome extension called "XPath Helper Wizard", but its the same situation - blank cell for the results.

#2

Hi Tom,

You can insert the requests from Google Search Connector as formula mode and change the reference to the cells containing your keywords.

Google is tricky to scrape because it alters your results depending on different factors such as your location. In the Connector we've added HTTP headers to make it more reliable. You can look in the GoogleSearch.xml file for the xpaths and the headers used if you want to scrape manually.

#3

Thank you! I will try this out.

#4

you get blank cells because Google bans your ip