How to bypass bot detection

Websites can have a different level of anti-bot mechanisms depending on the sensitivity of their data and budget. If your automations about being blocked, setting these three flags will bypass most anti-bot mechanisms.

Flags to bypass bot detection

Use the &headless=false flag

Most bot detectors will check your user-agent, which by default explicitly claims you're running headless chrome. This is a dead giveaway. It can be changed by setting a specific user-agent but we highly recommend you use the &headless=false flag instead, which changes your user-agent to a more credible one.

Use the &stealth flag

The stealth flag implements Puppeteer's puppeteer-extra-plugin-stealth plugin which applies various techniques to make detection of headless puppeteer harder. This flag may backfire and be easily detected by some sites, so consider avoiding it as well.

Use our in-built proxy or use a third-party proxy

Finally, the hardest sites to crack down check your IP address; there are two type of bocks that can occur, those based on type of IP, and those based on frequency of requests (rate-limits).

• Sites checking the type of IP address will detect your data-center IP addresses when using Browserless. To overcome this, using a proxy with residential IP addresses will be the best option.
• Sites that work the first few times and then stop working, are probably rate-limiting and it's not the residential part of it that blocks us. For these cases, you don't necessarily need a residential proxy and data-center IP addresses that rotate should be enough.

Additional strategies

If none of these do the trick, get in touch with us at support@browserless.io>.

We have more trick up our sleeves we can show you, such as captcha solving and changing viewport sizes, especially using our enterprise features.