Sellton

SelltonBot

SelltonBot is the web crawler of Sellton. It reads public company websites so that Sellton's business data says accurately what each company does, where it operates and whether it is hiring.

It identifies itself on every request

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SelltonBot/1.0; +https://sellton.ai/bot) Chrome/124.0.0.0 Safari/537.36

What it reads

  • Public pages of a company's own website: the homepage, and the pages that describe the business: about, products and services, pricing, customers and partners, careers, contact, and the legal notice or imprint. A visit requests at most 25 pages of a website, besides robots.txt and the sitemaps, and keeps at most 15.
  • Your robots.txt and the sitemaps it lists.
  • The public job listings your careers page links to on a job-board service.

What it never does

  • It never logs in, submits a form or reads anything behind a password.
  • It never solves a CAPTCHA and never tries to get past bot protection. A site that turns it away is left alone.
  • It carries no cookies from one website to another.
  • SelltonBot is looking for information about companies, not about individuals.

How it behaves on your site

  • One page at a time, and at least one second between two requests to the same host. When a page needs its JavaScript run, SelltonBot loads it as a browser would, without images, video or fonts.
  • It backs off when asked. A 429 Too Many Requests is honoured: it waits the time your Retry-After names. If that wait is long, the visit ends and SelltonBot comes back another day.
  • How often: a website is visited about every six weeks. A site that turned SelltonBot away is not tried again for a month. A site that did not answer is tried again after a week, then after a month. After three failures in a row, it is not tried for at least nine months.

How to control it with robots.txt

SelltonBot reads robots.txt at the start of every visit, on every host it requests, and obeys it. A group addressed to SelltonBot is obeyed instead of the general * group.

Keep SelltonBot out of your whole site:

User-agent: SelltonBot
Disallow: /

Keep it out of part of your site:

User-agent: SelltonBot
Disallow: /internal/

Your change applies from SelltonBot's next visit.

What we do with it

What SelltonBot reads becomes business information in Sellton: a description of the business, its products and services, its locations and its open roles. We use what a company publishes about itself and its business.

How Sellton handles personal data, including the lawful bases we rely on and the rights you have, is set out in our Privacy Policy. If you think something about your company should not be there, write to us and we will look at it.

Contact

Questions, or anything you want us to change about how SelltonBot visits your site: write to admin@sellton.ai. A person reads every message. Please include your domain.