Stopping after 300 results? - Printable Version +- ScrapeBox Forum (https://www.scrapeboxforum.com) +-- Forum: ScrapeBox Main Discussion (https://www.scrapeboxforum.com/Forum-scrapebox-main-discussion) +--- Forum: Scrapebox Footprints (https://www.scrapeboxforum.com/Forum-scrapebox-footprints) +--- Thread: Stopping after 300 results? (/Thread-stopping-after-300-results) |
Stopping after 300 results? - rightintwo - 07-14-2019 Hi Guys, I am trying to scrape LinkedIn profiles for a certain job titles. When I run it, it stops at 300 results every time. The search that I am trying to scrape Google says has 2 million results, but I can only get to 300 reults total. Here are the details: URL I am scraping: site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlalaries -inurl:dir -inurl:jobs "Director of IT" Note: the forum post is converting "colon salaries" to an emoji. Proxies: Using proxies harvested through SB proxy manager. Currently have 168 proxies that passed the Google Proxy test. I've tried using the detailed harvest and the custom harvester. Same result. I also reloaded the custom harvester settings to see if anything had changed. Anyone have any suggestions? RE: Stopping after 300 results? - loopline - 07-16-2019 Google soft caps at 300 to 600 results for more advanced queries, clearly 300 with a query like that. So just add on keywords like site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" a site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" b site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 1 site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" 2 site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" purple site:linkedin.com (inurl:in OR inurl:pub) -intitle:directory -inurlConfusedalaries -inurl:dir -inurl:jobs "Director of IT" car etc.. then just remove duplicates when your done. This makes google return different sets of results from its database. RE: Stopping after 300 results? - rightintwo - 07-16-2019 Oh wow- didn't know that. Good to know! Thank you Loopline! RE: Stopping after 300 results? - loopline - 07-17-2019 Your welcome, have a great day! |