During one week in October an e-commerce website that deploys 51Degrees Human Confidence saw daily visits increase by about 50%. Happy days, especially with Black Friday around the corner.
But when we drilled into these additional sessions we could see the increase came from Low confidence devices that are very unlikely to be shoppers.
Those Low confidence visitors loaded pages like a browser and read the pricing page every day. From 24 October they started running searches as well. People read prices, that's the point of a pricing page, but software coming back every day?
Human Confidence rates how likely it is that a person is reading the page. Low means almost certainly no person.
What the data shows
The chart is from the site's analytics.
The green High confidence sessions hardly changed, nearly all the extra traffic is Low confidence red.
The pricing page, on the right in the table, got a lot of attention over the last four days.
| Human Confidence | All pages | Pricing page |
|---|---|---|
| Low | 53% | 45% |
| Medium | 31% | 11% |
| High | 16% | 44% |
High was nearly three times as common on the pricing page as across the site, and Low still accounted for 45%. Medium is harder to do much with because it's sessions we can't call either way. That is where a retailer needs to decide on the treatment. More on that later.
What a rival scraper could learn from your website
A retailer has thousands of prices that change every week. Nobody knows exactly what the scraper came for, so the right-hand column is our best guess.
| What the website saw | What it would mean for a retailer |
|---|---|
| Low visits read the pricing page every day of the week. | A rival's repricing tool could match your next price cut the same day. |
| They were still at it at the end of the week. | Keep that up for a year and they have your sale calendar. |
| From 24 October the site search suddenly got busy, and only 1% of the visits using it were High. | What's sold out, and when. |
| 97% of visits from China were Low confidence, compared with 12% from the UK. | Which countries you charge more in. |
Who could you turn away?
Leave search engines alone. They send your prospective customers. But a rival copying your price list should do its own homework. Both use crawlers to fetch pages in bulk. Polite crawlers read robots.txt, a plain text file from 1994 where a website says what they may do. But a scraper is a crawler pretending to be a browser with a human behind it, so no list of crawlers catches it.
At a different website, a travel business operating in several countries, AI crawlers and agents went from under 1% of new visitors' requests in June to about 90% in August. A few looked like agents planning trips for real people, judging by where they came from and what they read. Most seemed to be checking prices and availability.
Agents are different. An AI assistant planning someone's trip is working for that person, who may be about to buy. Block every agent and you turn away the people who sent them too. We wouldn't decide that for you, and you might make a different choice in November and February.
Before you show a price
Ask for Human Confidence before showing a price. High gets the prices. For Low we'd strip the prices out and leave the page there, because a few real people might get rated Low and a missing price is less of a shock than a blocked page. Medium, how about a captcha where they have to rotate the animal or find the traffic lights? All add to the cost of scraping.
Better analysis
You don't have to start by blocking anything. The website in that chart just recorded the ratings in analytics at first and blocked nothing. Put Human Confidence in your analytics and then segment by Low, Medium, and High. Make important decisions from the actions of the High confidence sessions only. A smaller, but more reliable sample.
Human Confidence is in Pro for the cloud service and Bespoke for your own servers. Crawler detection and a robots.txt built for you are both free. Pro you can sign up for online. For Bespoke, talk to us.