I'm trying to get some of the first page results for a list of keywords, and I've been noticing some instances where it isn't returning the correct results. This can even be seen in the sample file for XPathOnURL that comes bundled with SEOToolsForExcel. I've included a screenshot below:
Why are some of the resulting URLs "/search&q=dogs&..." ? instead of the http/https URL for that SERP position?
For a freshly loaded SERP page, your Xpath should return the destination URL for each SERP listing. It appears as though it's extracting Google's tracking URL instead.
Whenever a SERP link is clicked, a JavaScript event rewrites the original @href value with an internal Google URL that tracks the click before redirecting you to the desired page. The JS function also adds a new attribute called @data-href that contains the original URL. You can visualize this by hovering over a link, then comparing the URL shown in the status bar before and after right-clicking the link. Here's what happens to the <a> tag when that JavaScript fires:
Before click:
Same listing, After click:
Normally, the tracking URL begins with /url?.... I believe the /search?... instance you're getting may be the Related Searches at the bottom of the SERP?
Hopefully this is the right track.
-Tim
