Crawl password protected website with Spider

Support
#1

Hello,

Is it possible to crawl password protected website with Spider ?

In HTTPSettings, there are fields for user/password authentification. I filled out these but I still get a HTTP 401 error so I'm wondering.

thanks for your help,
Fred

#2

I believe the username/password fields in Global HTTP Settings are meant for basic HTTP authentication. That is, not regular login fields.

Unfortunately, the spider is not ideal for this task. A better alternative would be a headless browser library such as Selenium. If you want to stick with Excel then VBA and the IE Browser also works, although fewer sites supports it today.