05-25-2011, 09:52 PM
Hi all,
I'm scraping a bunch of web directories/article directories and I want to get as many as possible, so I got my footprints ready and now trying to figure out what are the best keywords I should use to get as many unique web directories/article directories as possible? I already tried using top 2,000 english words and also a list of category names found on most directories. I'm not very happy with the perecentage of unique domains I'm getting - 99.9% non unique out of about 500,000.
Are there some ways to increase the % of unique domains gathered by using some specific types of keywords? Also, does anyone know if specific search queries like allintitle, allinurl, allintext etc...(or combination of these) yeilds more unique domains when doing large scrapes like this?
Thanks.
I'm scraping a bunch of web directories/article directories and I want to get as many as possible, so I got my footprints ready and now trying to figure out what are the best keywords I should use to get as many unique web directories/article directories as possible? I already tried using top 2,000 english words and also a list of category names found on most directories. I'm not very happy with the perecentage of unique domains I'm getting - 99.9% non unique out of about 500,000.
Are there some ways to increase the % of unique domains gathered by using some specific types of keywords? Also, does anyone know if specific search queries like allintitle, allinurl, allintext etc...(or combination of these) yeilds more unique domains when doing large scrapes like this?
Thanks.