I have two question regarding the Spider:
I want to scrape websites for contact emails and only want the URLs that contain an email to be listed. Is this possible?
Is it possible to scrape websites for keywords? For example, I would like only URLs with content containing careers/jobs and a keyword to be listed.
Thanks !
This is not possible directly in the Spider, but you can extract the text/html and apply filters on emails/keywords afterwards.
You mean with gettextonurl ?
I want to scrape websites for contact emails and only want the URLs that contain an email to be listed. Is this possible?
Is this possible ? Thanks !
Yes, or XPathOnUrl
I will try it out. Are there any tips on how to keep the Excel sheet as "light" as possible?
I want to crawl a list of websites and open up the content with Chatgpt.
By the way: Is it possible to use the spider for a list of domains ?
Yes, but since "Depth" does not appear I assume that the page is not crawled or only one page is spidered ?
Only the specific page in the range, which means depth = 0
