Bots and humans

Human Confidence tells you, in one word and in real time, how sure you can be that a person is viewing the page. Keep the bots out of your analytics and your ad spend, and let the humans through.

Crawlers that announce themselves are easy. Our crawler data names them and says what they collect pages for, so your robots.txt can decide which ones to serve.

Create a Resource Key

Your crawler

{
  "iscrawler": true,
  "crawlername": "Common Crawl CCBot",
  "crawlerusage": ["Archiving","Search","Train"],
  "isartificialintelligence": "True",
  "humanconfidence": "Low"
}

https://cloud.51degrees.com/api/v4/json?resource=YOUR_KEY&user-agent=CCBot/2.0 (http … crawl.org/faq/)

Legacy

Legacy

Psst. Your bot list only catches the crawlers that say who they are. What about the ones posing as people? 😫

Single feature. Missing the whole picture.

51Degrees

51Degrees

36 million downloads, 1.6 million websites, and 142,000 downloads in the last week.

Plans and prices are on the pricing page, and every property is in the Property Dictionary.

What you need to know

🔗

A crawler collects pages in bulk for its operator. A scraper hides by posing as a browser. An AI agent works for one person on one task. Human Confidence returns High, Medium, or Low confidence that a person is reading the page. Add the value to your analytics before deciding what each visit can access.

The panel at the top of this page shows canned requests and the cloud's answer for each. They follow the public HumanConfidence derived property, which you can read in full and try in the script tester. Declared crawlers and browsers driven by a web driver are Low, and the answer is Unknown until the 51Degrees JavaScript has run. Otherwise it depends on whether the page was in view, how recent the browser is, and what the IP address says about the people who use it.

Cloud Human Confidence, Node.js
const CLOUD = 'https://cloud.51degrees.com/api/v4/';
const KEY = process.env.RESOURCE_KEY;

const headers = await cloudHeaders();
const evidence = new URLSearchParams();
for (const [name, value] of Object.entries(req.headers)) {
  if (headers.has(name)) evidence.set(name, value);
}
for (const [name, value] of Object.entries(req.body ?? {})) {
  if (name.startsWith('51D_')) evidence.set(name, value);
}
evidence.set('client-ip', ip);
const response = await fetch(CLOUD + KEY + '.json', {
  method: 'POST', body: evidence,
  signal: AbortSignal.timeout(TIMEOUT_MS)
});
const json = await response.json();
if (!response.ok) throw new Error(JSON.stringify(json.errors));
answer = json.derived?.humanconfidence ?? 'Unknown';

if (human.answer === 'High') return next();
if (human.answer === 'Low')
    return res.redirect('/automated-access');
res.redirect('/challenge');

The full Node.js cloud example

Developers start with the Human Confidence overview and its quick start. Retailers can read what it means for e-commerce. Human Confidence comes with the Pro plan, our cloud service, or with the Bespoke plan on your own servers.

Read the Human Confidence quick start

🔗

To start identifying bot traffic, you will need our IsCrawler property. Our CrawlerName property can provide additional information on the crawler.

Googlebot and Bingbot collect pages for search indexes, and other crawlers collect them to train AI models. CrawlerUsage says what each one collects pages for, so you can decide which uses of your content you allow.

Tell crawlers your decision in your robots.txt. Our robots.txt generator writes the file from the uses you allow, and emails you an update when crawlers or their uses change.

See the crawler properties

🔗

Decide which crawlers you want. A claimed crawler name alone is not proof of who sent the request, so verify the search engine crawlers you admit before you allow access, as our post on how likely your traffic is to be from real humans explains.

Bots posing as people sometimes claim an unusual device or a very old browser. Combine 51Degrees properties into a bot filter tailored to your audience, as our guide to filtering bots from your analytics with User-Agent data shows.

  • Location data and device detection can inform your access rules. Neither identifies a fraudster on its own.
  • Use Human Confidence to decide whether to serve content, request a login, or restrict a visit. The value is a confidence, so choose a policy suited to your audience.
  • Segment analytics by declared crawler and Human Confidence. Review those segments before treating every visit as a person.

See how our customers use real-time data

🔗

We support cloud and on-premise deployment. For cloud use, read about the Resource Key, then create a Resource Key with your chosen properties. Add your License Key when your plan requires one.

We have documentation for each API language to get you started with on-premise device detection. You can access a free trial by contacting us for more information.

Read the device detection quick start

🔗
  • Various deployment options in a range of API languages, all provided open-source and free via GitHub.
  • We're an ISO 9001 and 27001 certified company, so you can trust in our high-quality service.
  • Our Data team update our device database of 1,290,488 combinations daily. We source our data from real devices, vendors, or optional usage sharing. We don't cut corners by scraping the web.

See the crawler database

Pick one to find out more

Each heading opens a short answer, with code or an example, and a link to the documentation for more.

Read the Human Confidence overview
What is a web crawler?

What is a web crawler?

Crawlers

Crawlers collect pages in bulk for whoever runs them, for example to build a search index or train an AI model.

To detect bot activity, you can use the User-Agent HTTP header or IP intelligence tool. Our dedicated Data team regularly monitors the web for bots to add to our crawler database.

People, agents and scrapers

A crawler collects pages in bulk for its operator. A scraper hides by posing as a browser. An AI agent works for one person on one task.

4 ways to use Bots and humans

🔗

Is that a person, or an AI agent browsing? Human Confidence answers before the page is served

Some AI agents are shopping for a real person. Scrapers aren't shopping at all, they're copying. Both drive a real browser, so neither looks like a crawler, and your server should treat them differently. Give High the prices and keep them from Low. What Medium gets is your decision, and a sign-in is the usual answer when the scraping hurts.

Nine bots and one shopper show up in your logs as ten browsers. Human Confidence picks out the shopper, and your traffic report starts counting buyers.

Who it serves. Retailers, and anyone else whose website should serve people first.

Talk to us about this

Pairs well with Suspicious traffic management, for the full traffic investigation.

Integration time. 30 minutes in any Pipeline API integration.

Ingredients, 5

Bots and Humans
  • HumanConfidence
  • IsCrawler
  • CrawlerName
  • CrawlerUsage
  • IsArtificialIntelligence

Hero ingredient. HumanConfidence Scraper or shopper? Adapt to every request.

  1. Read Human Confidence on every page view
  2. High means you can be confident a person is reading the page. Serve the prices.
  3. Low gets no prices, or no page.
  4. Decide which AI agents you welcome. Some are shopping for a real person.

Chef's tip. Keep the rating in the session for half an hour, so you aren't asking the cloud on every page.

🔗

Strain the bots out before they cost you money

Crawlers that announce themselves are easy. The crawler properties name them and say what they're for, and your robots.txt handles them. The hard ones drive a real browser or come through a data center, a VPN, or a proxy, and most analytics count every one as a visitor. Human Confidence rates whatever is left once the named crawlers are out, and the network properties say where it came from.

Who it serves. Anyone whose analytics or ad spend leaks to bots.

Talk to us about this

Pairs well with Spot the fake, to verify the device behind every suspicious request.

Integration time. 30 minutes in any Pipeline API integration.

Ingredients, 10

Bots and Humans
  • IsCrawler
  • CrawlerName
  • CrawlerUsage
  • CrawlerProductTokens
  • IsArtificialIntelligence
  • HumanConfidence
IP Intelligence
  • IsVPN
  • IsProxy
  • IsHosted
  • HumanProbability

Hero ingredient. HumanConfidence Know who is looking at your prices before you show them.

  1. Pass every request through IsCrawler
  2. Strain out the bots that say who they are, by name and usage
  3. Add IsArtificialIntelligence for the AI scrapers and agents
  4. Read Human Confidence for whatever is left

Chef's tip. Check IsCrawler first. It's free on every plan and names most of the bots before Human Confidence has to rate anything.

🔗

High-value audiences, no cookies required

The device in someone's hand says whether they're a premium, mid-market, or budget buyer, and the network says where they are. Segment by price band, device age, and technology adoption, then add the country and town from the IP address. There's no behavioral data to collect or defend.

Who it serves. Media buyers targeting premium users after the third-party cookie.

Talk to us about this

Pairs well with Bidstream enrichment, to turn enriched bid requests into premium audience targeting.

Integration time. One afternoon with the Pipeline API.

Ingredients, 12

Device Detection
  • PriceBand
  • ReleaseAge
  • ReleaseYear
  • HardwareRank
  • HardwareVendor
  • DeviceType
  • BrowserName
  • PlatformName
IP Intelligence
  • Country
  • Town
Bots and Humans
  • HumanConfidence

Hero ingredient. PriceBand It separates premium, mid-market, and budget consumers without needing any behavioral data.

  1. Reduce your traffic down to devices in a premium price band.
  2. Add ReleaseAge to find the flagships released this year.
  3. Season with country and town for market-level targeting.
  4. Serve premium creative to people who actually buy premium. Cap and measure with 51Did.

Chef's tip. Someone on a brand-new flagship and someone on a 2019 mid-ranger are not the same audience. Stop bidding like they are.

🔗

Tell each crawler which uses of your pages you allow

Crawlers collect pages for different uses. Some build a search index that sends you visitors, and others train AI models that answer your visitors' questions without them. robots.txt tells each crawler what you allow, but only for the crawlers it names, and new ones appear every month.

Our robots.txt generator writes the file from the uses you allow rather than from a list of names you have to keep up, and emails you a new file when crawlers or their uses change.

Who it serves. Publishers, retailers, and any website whose pages crawlers collect.

Talk to us about this

Pairs well with Suspicious traffic management, for the bots that ignore robots.txt.

Integration time. 10 minutes, with no code.

Ingredients, 4

Bots and Humans
  • IsCrawler
  • CrawlerName
  • CrawlerUsage
  • CrawlerProductTokens

Hero ingredient. CrawlerUsage Allow search. Refuse training. Your choice.

  1. Choose the uses you allow on the robots.txt generator
  2. Publish the file at the root of your website
  3. Replace it when we email you an update
  4. Check CrawlerUsage on each request for crawlers that ignore it

Chef's tip. Keep the email address current, because the generator sends a new file when crawlers or their uses change.

Pick the one closest to your business challenge

The story is for you and the recipe for your engineers, with the properties they need and a link to create their Resource Key.