1. Wrong ban

robots in Baidu.On the update of txt, if you click "detect and update" many times, you will often be able to update, but often can not update the problem。That way: Things that should not be included in robots.txt on the prohibition is included, and it is normal to delete。So what's the problem？It's not that the servers are overloaded, but that the firewall has mistakenly blacklisted some Baiduspider。

2. The server is abnormal

The conventional server will not say, we all know that the general good。But there are some special servers, presumably the vast majority of webmasters do not know it？For example, the "RTHK server" of Western Digital is very interesting, is it really RTHK？Its own computer room in China, what is Hong Kong and Taiwan？In order to avoid filing and use a Hong Kong and Taiwan IP, the data are all in China。

What's wrong with that？We will find: the server of the site is through the CDN, even if you upload a picture, will be displayed as "302 status code", the access speed is improved, but this is conducive to SEO？

3. The real IP address cannot be obtained

Larger sites generally use CDN acceleration, but some sites use CDN acceleration not only for "devices" but also for spiders。What's the result？If the CDN node is unstable, then the problem can be fatal to the website spider。

The reason why many large sites open CDN is that it is easy to be attacked, and it can be imagined if you do not do "spider back to the source" at this time。Does your site have a CDN？Please log in Baidu webmaster platform to see if spider can crawl the real IP address!