First URL for keyword search

General
#1

Hey, guys.

I've been trying to get the URL for the first result from a keyword search, but I'm going through some issues.

Say, if I search for Apple, I want to get something like www.apple.com/

I was doing it by Google, but then they blocked me after a a while and I use an static IP, so that's a problem. That was the way I was doing it:

XPathOnUrl("http://www.google.com.br/search?q="&B2;"//div/div/div[1]/div/div/div/div/cite")

I tried using Bing and DuckDuckGo but I couldn't make it work. I guess I couldn't find the correct XPath, or the results are somewhat protected, idk.

Would anyone have a better solution for this? Bing and DuckDuckGo are still an option for me if I can make them work.

Thanks in advance! (:

#2

Try the Bing Search connector in the latest release.

#3

I've had best results using the following Xpath to get Google US's 1st Organic listing (ignoring Answer Boxes & other SERP features):
(//div[@id='rso']//div[@class='g' and descendant::div[@class='rc']][not(ancestor::div[@class='xpdopen'])])[1]//cite

I just tried it on google.com.br and it worked fine there as well. Note that this will give you the display address (usually in green) as in your example. If you want the actual URL from the href, replace …//cite with …//h3/a/@href. I usually collect both elements so I can identify whenever Google chooses to override such things (especially with mobile search).

When scraping Google SERPs, make sure you set the following in your HTTP settings (at least a 1sec delay):
<IntervalBetweenRequests RandomFrom="1000" RandomTo="2500" IfSame="Host" />

If you're getting banned, it might be time to consider using proxies. Hope this helps!

#4

Thanks! It works fine when I choose "Value" as output, but when I choose "Formula" it returns blank. Do you know why is that? I need the formula so I can select a cell to input the Query value and make it scalable.

#5

Thanks for your detailed answer! I tried it all, but it isn't working for me, unfortunately. I get an exception error saying that the XPath has an invalid token. Do you have any idea what that could mean?

#6

Can you provide the exact formula you are getting an error from?

#7

That's the formula I get:

=Connector("BingSearch.SearchResults";"Saigon Dragon Studios";;;"Url";VERDADEIRO;1)
  • VERDADEIRO is the portuguese for TRUE.
  • My intention is to change the query value for a cell reference.
  • When I choose "Values" as the Output it returns the following URL: http://saigondragonstudios.com/

Thanks for taking the time and effort in helping me (:

#8

Hmmm... Maybe try using double quotes rather than single quotes?

(//div[@id="rso"]//div[@class="g" and descendant::div[@class="rc"]][not(ancestor::div[@class="xpdopen"])])[1]//cite

#9

We had some issues with formula inserts in the previous release. Can you try this update?

#10

Tried both options of quotes, double double quotes, tried putting them on a cell and referencing it.... also tried the new release of SeoTools.... and it still isn't working :frowning:

XPath still has invalid token.
Formula still gives blank output, while Values gives the correct URL.

Thanks again. I appreciate you all trying to help.