When I scrape tweets using a hashtag search, I am only getting 25-200 tweets. I need to get closer to 10,000. How can I do this?
Are you using the Twitter Connector "Tweets Search" function? It is restricted by the rules of the Twitter API. Seven day historical search, 180 requests per 15 minutes and pagination with 20 tweets per page.
There are other methods, such as paying Twitter/GNIP for historical data, or doing manual scraping via their search page.
I have another tool through google docs, its an addon, that will pull upto 15k tweets going through Twitter API. Can the tool scrape through search? Because maybe that is what the other tool is doing.
What addon is that? Do you connect your Twitter account via OAuth authorization. If then it should use be bound by the same rules as SeoTools. it is also possible to scrape a large number of tweets with delayed requests to handle the time limit. It may scrape through search with some kind of infinity scrolling method, but then retweet id:s and a lot of other meta information are lost.
Tweet Archiver Premium. It is a pretty obscure plugin. I think it must be scraping from search. It does use oauth. It takes a few minutes but it scrapes generally between 5K-15K tweets. It has all the data, user, links, follower counts, tweet link, full tweet text, geo, etc.
I looked at the description of Tweet Archiver and think we misunderstood each other I thought you meant historical data. TA pulls tweets posted in the last hour (15 min for Premium) and it does this continually. The SeoTools Twitter Connector extracts historical tweets from the archive and it is not designed to continually scrape in the background.