Xpath on Google Local Path Help Requested

Chris · July 26, 2017, 10:12pm

Hello,

I am using this Xpath to pull Google Local Pack results.

//div[1]/div[*]/div/div/a[2]

It mostly works great with a few exceptions
For example:

when using the formula:
=IF(M25="","",Dump(XPathOnUrl(R25,$AO$1,"href",HttpSettings(TRUE,,,"3000|6000|Host"),"text"),TRUE))

if $AO$1 is always:
//div[1]/div[*]/div/div/a[2]

I get good result urls.

but sometimes I don't

http://www.google.com/search?pws=0&gl=EN&q=Nashville DUI Accidents Injuries Lawyer&num=10 generates 4 resutls versus the standard 3 map pack resutls. The odd result (intead of a normal url) is: /aclk?sa=l&ai=DChcSEwi1i6CzgqfVAhUMF4EKHR9GCeEYABAFGgJ5dw&sig=AOD64_0t0y0OuSRq9Xkv9JqRuupx9IKEOw&q=&ctype=99&ved=0ahUKEwigyZuzgqfVAhUJ4yYKHfOmDLsQhKwBCFI&adurl=

another example is:
http://www.google.com/search?pws=0&gl=EN&q=Nashville What is the difference between compensatory and punitive damages Lawyer&num=10
which generates:
/search?pws=0&gl=EN&tbm=bks&q=inauthor:%22Henry+Philip+Farnham%22&sa=X&ved=0ahUKEwiurpikg6fVAhVHOCYKHaicDrMQ9AgIRDAG

3 good urls

and another is:
http://www.google.com/search?pws=0&gl=EN&q=Nashville Nashville Personal Injury Office Lawyer&num=10
which generates:
/aclk?sa=l&ai=DChcSEwiQ8ZHAgqfVAhWRPIEKHWUDCH8YABAOGgJ5dw&sig=AOD64_07krbeJbY9Vqs8VfFXWRorQozbow&q=&ctype=99&ved=0ahUKEwjiko7AgqfVAhWFQSYKHQmCBmEQhKwBCFQ&adurl=

3 good urls.

any insight/ help on the proper xpath to generate ONLY the 3 normal pack results would be appreciated.

Thanks!
Chris

Chris · July 27, 2017, 11:51am

I mistakenly referenced M25. I meant R25.

diskborste · July 27, 2017, 6:36pm

Hi Chris,

It would help if you could be more specific about exactly what you need to extract and supply a specific url. Do you mean these three links?

Chris · July 27, 2017, 7:05pm

Thanks very much for the quick reply.

here is an example. This search
http://www.google.com/search?pws=0&gl=EN&q=Nashville Automobile Accidents Lawyer&num=10
generates the 3 pack (red boxed)
https://www.screencast.com/t/apisdd9z085

The formula to generate the desired results is;

=Dump(XPathOnUrl("http://www.google.com/search?pws=0&gl=EN&q=Nashville Automobile Accidents Lawyer&num=10","//div[1]/div[*]/div/div/a[2]","href",HttpSettings(TRUE,,,"3000|6000|Host"),"text"),TRUE)

I've tried this successfully with the list of "for example urls" previously posted, where it lists 3 domains scraped from the Google 3-pack.

BUT... when trying the same formula using the urls from the list under "sometimes I don't :(" I get the non domain result in my example.

does that help clarify?
any insight/help would be appreciated.
Thanks!
Chris

diskborste · July 27, 2017, 7:30pm

The URL you provided generates the page in the screen dump in my previous post. Google doesn't return the same results for identical search URLs because of your location etc. This might also be affected by the header settings and they may be changed when sending a request via SeoTools.

This makes it very hard to test xpaths based on your urls and it might also be the reason you have random success.

Chris · July 27, 2017, 8:30pm

Hi again,

if you paste this...
http://www.google.com/search?pws=0&gl=EN&q=Nashville Automobile Accidents Lawyer&num=10
It's the same as this...
http://www.google.com/search?pws=0&gl=EN&q=Nashville+Automobile+Accidents+Lawyer&num=10

because of the way the forum creates the link you are not getting the whole url.

you are only navigating to...
http://www.google.com/search?pws=0&gl=EN&q=Nashville

you should be using http://www.google.com/search?pws=0&gl=EN&q=Nashville+Automobile+Accidents+Lawyer&num=10

Does that help?

diskborste · July 27, 2017, 9:41pm

Yes, that works. Will take a look at it and see if i can find a reliable way of scraping the location info.

Chris · August 2, 2017, 5:30pm

Hi Victor - Just checking in on this. Were you able to find anything that could help?
Any insights would be appreciated.
Thanks
Chris

diskborste · August 4, 2017, 6:41pm

Hi Chris,

Sorry for a late response. How about this:

Your xpaths didn't work for me and had issues finding an isolated path with titles which were not auto-generated. However, this seems to do the trick!