Hi I am trying to scrape 5 different text over 50 URLs using XPathOnURL.
However the process is incredible slow - some metrics can be crawl within seconds - some take over 30 minutes also not showing.
How should I improve this process?
Hmm, it sounds strange that some metrics takes longer than other. Do you have Cache enabled? This means you only need to fetch each URL one time and the requests are made against the cache. Otherwise each metric will be made as a new HTTP request.
Yes I do - does it mean its better I open each webpage for it to crawl faster?
One thing Im not sure will it affect or not is Im using proxy - should I disable it?
If you are using a Cache then it should take close to the same amount of time for five values compared to one.
Yes, please try.