TIL: Using PTT and Cloudflare to Block Web Crawlers?

title: [TIL] PTT + Cloudflare to Block Web Crawlers?
published: false
date: 2022-05-25 00:00:00 UTC
tags: 
canonical_url: http://www.evanlin.com/ptt-cloudflare/
---

![image-20220525172234806](http://www.evanlin.com/images/2021/image-20220525172234806.png)

## Problem with Web Crawlers

I originally set up a [PTT crawler](https://github.com/kkdai/linebot-ptt-beauty) to scrape some data, but recently I found that I couldn't see any data. Upon investigation, I discovered that CloudFlare seemed to be blocking it.

## Error Message

2022-05-25T08:45:46.856731+00:00 app[web.1]:

2022-05-25T08:45:46.856731+00:00 app[web.1]: DDoS protection by Cloudflare
2022-05-25T08:45:46.856731+00:00 app[web.1]:

2022-05-25T08:45:46.856731+00:00 app[web.1]: Ray ID: 710cff2fcebb1962

I'll have to put it aside for now and check it out later. I found some articles to keep for now.

## Related Links

- [kkdai](https://github.com/kkdai)/**[linebot-ptt-beauty](https://github.com/kkdai/linebot-ptt-beauty)**
- [How can I get html from page with cloudflare ddos portection?](https://stackoverflow.com/questions/32425973/how-can-i-get-html-from-page-with-cloudflare-ddos-portection)
- [Understanding Cloudflare DDoS protection](https://support.cloudflare.com/hc/en-us/articles/200172676-Understanding-Cloudflare-DDoS-protection)
- [How to customize your HTTP DDoS protection settings](https://blog.cloudflare.com/http-ddos-managed-rules/)