Crawlers can retrieve data much quicker and in greater depth than human searchers, so bad scraping practices can have some impact on the performance of the site.
Needless to say, if a single crawler is performing multiple requests per second and/or downloading large files, a under powered server would have a hard time keeping up with requests from multiple crawlers.
Since spiders don’t bring direct organic traffic and seemingly affect the performance of the site, most site admins hate spiders and do their best to prevent them.
Lets go through how websites detect and block spiders and also know the techniques to overcome those barriers.
Most websites don’t have anti scraping mechanisms since it would affect the user experience, but some sites do not believe in open data access.
Before going through this article always keep in mind that
A GOOD SPIDER MUST OBEY A WEBSITE’S CRAWLING POLICIES.
HOW DOES DETECTING ‘SPIDER ACTIVITY’ WORK?
A web server can use different mechanisms to detect a spider from a normal user. Here are some methods used by a site to detect a spider:
• Unusual traffic/high download rate especially from a single client/or IP address within a short time span raises a bot alert.
• Repetitive tasks done on website based on an assumption that a human user won’t perform the same repetitive tasks all the time.
• The site has honeypot traps inside their pages, these honeypots are usually links which aren’t visible to a normal user but only to a spider . When a scraper/spider tries to access the link, the alarms are tripped.
Spend some time and investigate the anti-scraping mechanisms used by a site and build the spider accordingly, it will provide a better outcome in the long run and increase the longevity and robustness of your work.
EASIEST WAY TO FIND IF A SITE HATES BOTS
Check the robots.txt file if it contains line like these, It means the site doesn’t like bots. However, since most sites want to be on Google (arguably the largest scraper of websites globally ;-)) they do allow access to bots and spiders.
User-agent: *
Disallow: /
This line is for preventing well-behaved bots or the bots which respect robots.txt.
Another way is CAPTCHAs irritating presence in the sites other than in authentication page.
WHAT HAPPENS WHEN YOU GET BANNED
There are two ways to ban a webspider, either by banning all accesses from a particular IP or by banning all accesses that use a specific id to access the server (most browsers and web spiders identify themselves whenever they request a page by user agents. Chrome browser for example uses Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36
The banning can be temporary or permanent. Temporary blocks can last minutes or hours.
HOW DO WE KNOW A SITE HAS BLOCKED US?
If any of the following symptoms appear on the site that you are crawling, it is a sign of being blocked or banned.
• Showing CAPTCHA pages
• Unusual content delivery delay
• Frequent response with 404,301,500 errors,
also frequent appearance of these status codes are also indication of blocking.
• 401 Unauthorized
• 403 Forbidden
• 404 Not Found
• 408 Request Timeout
• 429 Too Many Requests
WEB CRAWLING BEST PRACTICES
These are the best practices we can follow to overcome the detection.
1. MAKE CRAWLING SLOWER, DO NOT DDoS THE SERVER, TREAT THEM NICELY
Use auto throttling mechanisms, which will automatically throttle crawling speed based on the load on both spider and the website, you are crawling and also adjust the spider to optimum crawling speed. The faster you crawl, the worse it is for everyone.
Put some random sleeps in between requests, Add some delays after crawled number of pages. Choose the lowest number of concurrent requests possible. These techniques make the spider looks like a human being.
2. DISGUISE YOUR REQUESTS BY ROTATING IP/PROXY
A server can easily detects a bot by checking the requests from a single IP address, So we use different IPs for making request to a server and detection rate become lesser. Make a pool of IPs that you can use and use random ones for each request.
There are several methods can be used to change the IP. Services like VPN ,shared proxies, TOR can help and some third parties are also provides services for IP rotation.
3. USER-AGENT SPOOFING
Since every request made from a client end contains a user-agent header ,Using the same useragent multiple times leads to the detection of a bot. User agent spoofing is the best solution for this. Spoof the User agent by making a list of user agents and pick a random one for each request.
Websites do not want to block genuine users so you should try to look like one. Set your user-agent to a common web browser instead of using the library default (such as wget/version or urllib/version). You could even pretend to be the Google Bot: Googlebot/2.1; (http://www.google.com/bot.html)
You can check your user-agent string here:
http://www.whatsmyuseragent.com/
A good user-agent string list can be found here:
http://www.useragentstring.com/pages/useragentstring.php
4. BE AWARE OF HONEYPOTS
Some site designers put honeypot traps inside websites to detect web spiders, They may be links that normal user can’t see and a spider can.
When following links always take care that the link has proper visibility with no nofollow tag. Some honeypot links to detect spiders will be have the CSS style display:none or will be color disguised to blend in with the page’s background color.
5. DO NOT ALWAYS FOLLOW THE SAME CRAWLING PATTERN
Only robots follow the same crawling pattern,Sites that have intelligent anti-crawling mechanisms can easily detect spiders from finding pattern in their actions. Humans wont perform repetitive tasks a lot of times. Incorporate some random clicks on the page, mouse movements and random actions that will make a spider looks like a human client.
6. ALWAYS RESPECT THE robots.txt
All web spiders are supposed to follow rules that you place in a robots.txt file in a website, such as how frequently they are allowed to request pages, and from what directories they are allowed to crawl through. They should also be supplying a consistent valid User-Agent string that identifies the requests as a bot request.
Source: http://learn.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/
Needless to say, if a single crawler is performing multiple requests per second and/or downloading large files, a under powered server would have a hard time keeping up with requests from multiple crawlers.
Since spiders don’t bring direct organic traffic and seemingly affect the performance of the site, most site admins hate spiders and do their best to prevent them.
Lets go through how websites detect and block spiders and also know the techniques to overcome those barriers.
Most websites don’t have anti scraping mechanisms since it would affect the user experience, but some sites do not believe in open data access.
Before going through this article always keep in mind that
A GOOD SPIDER MUST OBEY A WEBSITE’S CRAWLING POLICIES.
HOW DOES DETECTING ‘SPIDER ACTIVITY’ WORK?
A web server can use different mechanisms to detect a spider from a normal user. Here are some methods used by a site to detect a spider:
• Unusual traffic/high download rate especially from a single client/or IP address within a short time span raises a bot alert.
• Repetitive tasks done on website based on an assumption that a human user won’t perform the same repetitive tasks all the time.
• The site has honeypot traps inside their pages, these honeypots are usually links which aren’t visible to a normal user but only to a spider . When a scraper/spider tries to access the link, the alarms are tripped.
Spend some time and investigate the anti-scraping mechanisms used by a site and build the spider accordingly, it will provide a better outcome in the long run and increase the longevity and robustness of your work.
EASIEST WAY TO FIND IF A SITE HATES BOTS
Check the robots.txt file if it contains line like these, It means the site doesn’t like bots. However, since most sites want to be on Google (arguably the largest scraper of websites globally ;-)) they do allow access to bots and spiders.
User-agent: *
Disallow: /
This line is for preventing well-behaved bots or the bots which respect robots.txt.
Another way is CAPTCHAs irritating presence in the sites other than in authentication page.
WHAT HAPPENS WHEN YOU GET BANNED
There are two ways to ban a webspider, either by banning all accesses from a particular IP or by banning all accesses that use a specific id to access the server (most browsers and web spiders identify themselves whenever they request a page by user agents. Chrome browser for example uses Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.149 Safari/537.36
The banning can be temporary or permanent. Temporary blocks can last minutes or hours.
HOW DO WE KNOW A SITE HAS BLOCKED US?
If any of the following symptoms appear on the site that you are crawling, it is a sign of being blocked or banned.
• Showing CAPTCHA pages
• Unusual content delivery delay
• Frequent response with 404,301,500 errors,
also frequent appearance of these status codes are also indication of blocking.
• 401 Unauthorized
• 403 Forbidden
• 404 Not Found
• 408 Request Timeout
• 429 Too Many Requests
WEB CRAWLING BEST PRACTICES
These are the best practices we can follow to overcome the detection.
1. MAKE CRAWLING SLOWER, DO NOT DDoS THE SERVER, TREAT THEM NICELY
Use auto throttling mechanisms, which will automatically throttle crawling speed based on the load on both spider and the website, you are crawling and also adjust the spider to optimum crawling speed. The faster you crawl, the worse it is for everyone.
Put some random sleeps in between requests, Add some delays after crawled number of pages. Choose the lowest number of concurrent requests possible. These techniques make the spider looks like a human being.
2. DISGUISE YOUR REQUESTS BY ROTATING IP/PROXY
A server can easily detects a bot by checking the requests from a single IP address, So we use different IPs for making request to a server and detection rate become lesser. Make a pool of IPs that you can use and use random ones for each request.
There are several methods can be used to change the IP. Services like VPN ,shared proxies, TOR can help and some third parties are also provides services for IP rotation.
3. USER-AGENT SPOOFING
Since every request made from a client end contains a user-agent header ,Using the same useragent multiple times leads to the detection of a bot. User agent spoofing is the best solution for this. Spoof the User agent by making a list of user agents and pick a random one for each request.
Websites do not want to block genuine users so you should try to look like one. Set your user-agent to a common web browser instead of using the library default (such as wget/version or urllib/version). You could even pretend to be the Google Bot: Googlebot/2.1; (http://www.google.com/bot.html)
You can check your user-agent string here:
http://www.whatsmyuseragent.com/
A good user-agent string list can be found here:
http://www.useragentstring.com/pages/useragentstring.php
4. BE AWARE OF HONEYPOTS
Some site designers put honeypot traps inside websites to detect web spiders, They may be links that normal user can’t see and a spider can.
When following links always take care that the link has proper visibility with no nofollow tag. Some honeypot links to detect spiders will be have the CSS style display:none or will be color disguised to blend in with the page’s background color.
5. DO NOT ALWAYS FOLLOW THE SAME CRAWLING PATTERN
Only robots follow the same crawling pattern,Sites that have intelligent anti-crawling mechanisms can easily detect spiders from finding pattern in their actions. Humans wont perform repetitive tasks a lot of times. Incorporate some random clicks on the page, mouse movements and random actions that will make a spider looks like a human client.
6. ALWAYS RESPECT THE robots.txt
All web spiders are supposed to follow rules that you place in a robots.txt file in a website, such as how frequently they are allowed to request pages, and from what directories they are allowed to crawl through. They should also be supplying a consistent valid User-Agent string that identifies the requests as a bot request.
Source: http://learn.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/
No comments:
Post a Comment