Price Scraping Prevention: Best Practices for Securing Your Data
Price scraping is prevalent across the web, particularly affecting the e-commerce, travel, and finance industries. It is so prevalent that many are investing in e-commerce protection. The mass numbers of bots used in scraping attacks can slow your website down significantly, remove your competitive edge, and overall reduce your revenue. In this article, we will cover:
What is price scraping and why is it a problem?
Web scraping price monitoring takes place when automated bots conduct pricing data scraping from your websites, mobile apps, and APIs without permission. Price scraping is a particularly dangerous threat for price-sensitive industries that have many products and list their prices publicly, such as retail, travel, and hospitality. Price scrapers can:
- Undermine your competitive edge.
- Skew your marketing analytics.
- Slow down your website.
- Increase your operating costs.
- Reduce your revenue.
How does price scraping work?
Price scraping can happen in two ways: First, as part of a bot attack that scrapes everything on your website. These bots scrape not only your prices, but also your reviews, product listings, ratings, blog content, delivery times, product availability, etc. Their goal is to gather as much information as possible, all of which they store in a database to either sell or use for malicious purposes.
Second, scraping prices from websites can happen as part of a bot attack that’s focused specifically on prices. These bots only go after your prices and they tend to scrape prices so frequently they notice within seconds when one of your prices change. The more advanced bots then use that information to automatically undercut your prices on a competitor or comparison website.
Both the full web scrapers and the price scrapers can be built either manually or through a bots-as-a-service company that makes launching a bot attack incredibly easy.
Who carries out price scraping and why?
Just like you’re interested in your competitor’s prices, your competitor is interested in yours. If they’re interested enough, they might decide to automate the price discovery process and create a bot that constantly scans your prices. Even worse, they can then use that info to automatically lower their prices when you do.
Another party interested in your prices is the consumer. If you offer a product that’s similar or identical to a competitor’s product, the consumer will want to know which one is cheaper. But they might not want to do the research themselves, so they rely on comparison websites that use price scrapers to scrape your and your competitor’s prices.
This is what happened to Kurt Geiger. The DevOps Manager noticed that scraper bots were stealing their content, including their prices. They did the same competitor price scraping with their competitors and used that pricing information to make comparisons between products. The scraper bots were so aggressive it frequently overloaded Kurt Geiger’s backend system, resulting in website slowdowns. If they had invested in a web scraping data protection strategy (or a web scraping prevention software), this problem could have been prevented.
Examples of Price Scraping in Different Industries
Competitors and consumers are most interested in your prices, but many others might want to scrape your prices for varying reasons depending on the industry in question.
- E-commerce: To offer their similar product for just a little lower than your price, and automatically update when your price changes.
- Finance: To evaluate your pricing model and build something competitive.
- Travel: To sell their plane tickets, hotel stays, etc. for a little less than you charge, and automatically update when your price changes.
- Marketing: To create a report on pricing trends.
- Startups: As part of a product they’re building.
How to Mitigate the Risks of Price Scraping
Today’s scraping bots are sophisticated. They can fill out forms to discover your pricing, rotate quickly through thousands of IPs, and simulate human behavior. A WAF doesn’t protect against these bots, because they don’t carry the usual attack signatures a WAF protects against. Additionally, a WAF is IP-centric when bots no longer are.
Preventing Price Scraping with Bot Protection
The below common methods of preventing price scraping (and scraping prevention in general) are useful, but not fully effective when used on their own. A bot protection solution is specifically designed to identify malicious scraper bots, including price scrapers, among human traffic. Bot management software should detect bots in real-time without slowing your site down, and shouldn’t keep humans out.
Implementing CAPTCHA
A CAPTCHA can provide another barrier to scraper bots before they can reach your website. Look for a CAPTCHA connected with a powerful bot management solution that uses the challenge to gather more signals about suspected bots. Humans should almost never see a CAPTCHA challenge, and bots should be stopped by one—if they aren’t already stopped by bot detection software.
Session Tracking
Session tracking involves tracking a user’s journey through your website by cataloging all of their requests. Tracking the journey can locate suspicious activity. For example, a human user is likely to navigate to the site’s home page first and then browse through. A bot, on the other hand, is likely to navigate to one specific URL and possibly jump around with full URLs. Session tracking can locate price scrapers based on how they’re interacting with your website.
IP-based Blocking
When an IP address is determined to be a threat (through something like session tracking), you can block it. This ensures that any bot attempting to use the same IP address to make multiple price scraping attacks on your website will be stopped unless they use a proxy to change their IP.
Detecting and Responding to Price Scraping Attempts
As part of detecting web scraping, warning signs of possible price scraping on your website include:
- Competitors’ prices updating almost as quickly as yours do.
- High-volume traffic on your e-commerce pages that doesn’t translate into sales.
- Slow website and poor server performance due to the volume of traffic.
- User accounts with high activity on your site but no purchases.
As soon as you detect price scraping, you will want to put a stop to it. A good software will identify suspect user behavior and block bad bots before they can steal your prices. The best way to deal with scraping is to prevent it from the start, rather than reacting only when it’s become a problem.
Protecting Your Business From Price Scraping with DataDome
The best way to protect against any form of scraping is through advanced bot protection. DataDome CAPTCHA is a bot management solution that detects and blocks scraping threats in real-time. It takes only minutes to install on any web infrastructure and runs on auto-pilot. Try out our free 30-day trial today to see how many bots are currently scraping your website.
FAQs
Is price scraping legal?
Technically, price scraping is legal. If a price is publicly available, it’s legal to scrape. Although good terms and conditions can help, it’s unlikely you’ll go to court against every scraping bot sent your way. But that doesn’t mean you should just let bots scrape your prices. On the contrary, it’s vitally important to protect your business against any form of scraping.
Related posts
How BPX Protected Revenue-Critical APIs from 6.6M Price Scraping Attacks
Tell me more
Etix Stops Ticket Scalping Without Slowing Down Real Fans
Tell me more
What Is an Anti-Bot Solution and How Does It Protect Your Business?
Tell me more
How DataDome Blocked an 80M-Request Scraping Attack on a Leading Review Platform
Tell me more
