i'm trying to extract Google search results hrefs. but the maximum i can only download is 100 even though I've added &num=500 in the URL. How do I increase the limit for the xpathonurl dump?
I'm fairly certain 100 is the maximum number of hits per page. I've tried to increase the number beyond 100 and see in-browser if Google returned the corresponding number of results but it still caps at 100.
You could try the "&start=n" parameter (n=offset number) but beware of Google enabling CAPTCHA if abused.
1 Like