Working with Infinite Scroll???
Hey, any idea how to run a query on an page with infinite scroll?
I'm trying to pull posts from a user's profile on a mobile app where users share memes. Yeah. MEMES!
The site uses infinite scrolling, so I have to scroll to the bottom as far as many posts on that page exist to be able to view the entire html contents.
Right now, my setup is using an iPhone 7 user-agent in the
Global HTTP Settings.
The query I'm using is a simple
Dump of a
CsQueryOnUrl whose href tag contains a substring, and then returns the href value:
The query returns the first set of results only. In my case, exactly 10.
Right now, since I don't really know how to use SeoTools with VBA, which I'm assuming would be my best bet, my work-around is to use a UA Spoofer in chrome & an Autoscroll extension to let the entire page contents load. THENNNᴺᴺNNNNᴺ - I have to save the entire raw html page locally with an
xml extension, and point the url to it. But the last time I used this work-around, I was using
XpathOnUrl. So using
CsQueryOnUrl won't work with just a local html file, right? That's fine. I can switch to Xpath queries. I just need to figure out this Infinite Scroll issue.
Thanks in advance for any help! I'm open to any suggestions
Additional similar request: ↴
Some of these pages I'm pulling may contains over 10K posts. If each scroll loads 10 posts, that's alot of html code.
Any ideas on how to speed things up? Or maybe using the autoscroll extension I use in Chrome, can I tell Chrome somehow to load a page without actually "Loading" the page contents?
I'm not sure if that makes sense or if that's how html works, but if I disable the loading of actual images on the page but load the code only, it should process the code much faster no? I don't actually need the images just yet. I just need the hrefs for them.
These posts are all image based.