Unable to scrape SimilarWeb Total Visits data with XPathOnURL

Support
#1

I'm using this connector in Excel (where the A column is my list of domains I want total visit data for): =Dump(XPathOnUrl("https://www.similarweb.com/website/"&A3,"//span[contains(@class,'engagementInfo-value engagementInfo-value--large u-text-ellipsis')]","",,"text"))

It should be scraping this from SimilarWeb:

Pasted image1472×414 107 KB

The thing is this has worked previously at much higher rates than it is now. I was able to scrape total visit data with about a 75% success rate a few weeks or a month ago. Now, I'm returning barely 1% of results.

Any suggestions for mining SimilarWeb data?

#2

Hi Jack,

Your formula works 100% of the time for me. Perhaps your success rate has decreased because SimilarWeb is temporarily blocking your IP due to high activity to prevent scraping?

Another trick is to use the Transpose parameter combined with Dump if you want the values separated:
=dump(XPathOnUrl("https://www.similarweb.com/website/"&A1,"//span[@class='engagementInfo-valueNumber js-countValue']","",,"text"),TRUE)

bild.png1429×302 25.9 KB

1 Like
#3

That is what I was figuring too. Thank you so much for the help!