Hi SeoTools Community,
I am currently working on a large scale twitter scraping project and am running into issues scraping more than 800 tweets from any one user. Is there a way to scrape more than 800 tweets using SeoTools.
Thank you,
Eric
Are you using the official SeoTools Twitter Connector? You should be able to get 3000 followers according to the API limits (15 requests / 15 minutes | 200 followers / request).
You can either scrape manually or set a really slow deley 1 minute / request (or save the cursor generated by the API which is used in the pagination process).
Hi Victor- I was referring to this comment yesterday. Apologies for sounding foolish, but Im not entirely sure how to save the cursor and apply that to the next search, or to set the delay time as it appears to revert to original settings when used. vielen dank
Hi,
Yes, I understand that. I've updated the Twitter Connector with Custom Cursor support. First, copy the following code and replace the old code in Twitter.xml. This file is located in the install directory/Connectors.
Next, select the NextCursor field and extract how many followers you want until you reaches the limit. Then copy the latest Cursor and add it to the "Custom Cursor" field. The parsing will continue from the latest pagination. You will probably have to wait 15 or so minutes.
2 Likes
Thank you so much Victor, Your help is greatly appreciated. I'm very excited to start using all of the software's capabilities.
One more thing-was there an xml code snippet that i missed?
Yes, my bad 
https://raw.githubusercontent.com/nielsbosma/SeoTools-for-Excel-Connectors/master/Twitter.xml
No worries- if I haventt broken something by breakfast, its not a normal day
Sorry to be a problem here but after replacing the twitter.xml file, I have no custom cursor box, nor do i have an option to check a box for the NextCursor as pictured above. ( I have restarted excel) The xml file appears to have the correct inputs and callouts for the nextCUrsor etc, but does not appear within the interface as it does in yours.
Never mind. it is working correctly now. In explanation, I had renamed the twitter.xml original file and copied in the new version to the components folder, but for some reason the plugin continued to use the old version of the file. When i removed it from the folder and restarted xl again it finally used to the code from the new file. Go figure. Apologies for the question, though, in fairness, that's not a normal behaviour.