How to Block PetalBot

What is PetalBot?

PetalBot is an intelligent web crawler developed by Petal Search to collect data from websites and help inform search engine optimization (SEO) efforts. It has been designed to be lightweight on web servers while providing detailed information about each page it visits.

Petal Search powers both Huawei Assistant and AI Search services.

What is PetalBot’s user agent?

PetalBot’s user agent is slightly different when visiting websites on a computer versus a mobile device.

• Computer: Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
• Mobile device: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

How much pressure does PetalBot put on a web server?

PetalBot was designed to cause minimal pressure on the web servers it is accessing. Still, it is recommended that webmasters enable compression for the pages that are visited by the bot in order to minimize the amount of pressure put on the server during its visits. 

How to block PetalBot?

Webmasters who wish to block PetalBot from visiting their website should add ‘petalbot’ or ‘petal bot’ to their robots.txt file and/or use a web application firewall (WAF) to block the user agent from accessing the server. If you have a bot management solution like DataDome, you can also make sure that PetalBot is on the block list.

Some people across the internet have noted that PetalBot does not fully adhere to the rules set in a robots.txt file. If you’re worried about PetalBot traffic, you should use a bot management software to block it entirely.

Conclusion

In conclusion, Petal Bot is an effective and efficient web crawler developed by Petal Search that is lightweight on web servers while providing valuable data for SEO efforts. Webmasters can take advantage of the bot’s features if they desire, or choose to block it from visiting their website with the appropriate measures. 

With the right measures in place, websites can take advantage of PetalBot’s features without sacrificing their security or privacy. This will help ensure that website owners have control over who visits their pages and what data is collected. Ultimately, it is up to webmasters to determine whether or not they will utilize this helpful resource on their websites.