Setting delay for large scale twitter scrape

If Im to understand correctly, the twitter API limits calls to 3000 and this can be exceeded only if a delay is set. Where exactly is this set and what is the standard, is it in milliseconds, seconds or minutes?

No, the limits depend on which endpoint you are calling in the Twitter API. You can look at the various endpoints and their limits here:
https://dev.twitter.com/rest/reference

For example, Tweet Search has a limit of 180 requests within a 15 minute window:

What are you planning on extracting from Twitter? For Tweets or User Ids I greatly recommend doing bulk requests, which allows you to extract many items in a single request. There are also built-in support for this in the Twitter Connector called "Tweets Lookup Batch" etc.

If you still need to set a delay, you can do it from Settings -> Global HTTP Settings ->

These inputs are in milliseconds.

1 Like

Thanks for the quick reply Dick! I wasn't aware of the bulk request function yet, this may be helpful. Could it be used to grab all followers in one swath or will it need to be parsed in 3k chunks?

Initially the need is to evaluate my current followers for a number of needs and have it within XL for ease of use- hash tag search etc, as i have 11.5k followers.

But i will need to transition to some more automated uses and harvesting other's followers and eliminating eggs and identifiable shills to feed follow lists and identifying influencers etc. I've been doing all by hand until now and its too cumbersome.

Also-apologies for getting your name wrong Victor :slight_smile: trying to work too quickly never turns out well