Crawler policy
Esplanda-PriceCheck, our price check crawler
If you found this user-agent in your server logs, it was reading public prices for Esplanda’s free price check. Here is what it reads, how fast it goes, and how to block it.
The user-agent on every request
Esplanda-PriceCheck/1.0 (+https://esplanda.com/price-check-crawler)- robots.txt token
Esplanda-PriceCheck- Strings
- One, never rotated or disguised
- Reads
- Public prices only
What it reads
Public prices, and nothing else
The free price check compares what nearby businesses publish in public. The crawler reads exactly that, and keeps as little as the comparison needs.
What it reads
- Public prices and product names, the way any shopper would see them.
- Published feeds first: marketplace feeds and public storefront pages, then Google local inventory where a store publishes it, then online menus.
- A store’s own website last, only the pages that list products, and only after its robots.txt says yes.
- What it keeps about a business: the name, the address, the public listing link, the price, when it was seen and the kind of source. Nothing more.
What it never reads
- Personal data. No names, emails or phone numbers of people.
- Reviews. No review text, reviewer names or ratings.
- Anything behind a login. No accounts, cookies, member prices, loyalty prices or paywalls.
- Copies of your pages. A page is read for its product titles and prices; the page itself is not kept.
- Your business numbers. No cost, margin, sales, stock or traffic, and no score about your business.
How fast it reads
Slow, polite and easy to spot
Every limit below is fixed in the crawler’s code and shared across every report running at once, so a popular store never gets hit by several checks together.
- First
robots.txt is read before any other request. A no is final. If it can’t be reached or read, the answer is no.
- 1 / sec
One request per second per website. One connection at a time.
- 200
Requests per website per report, at most. No more than 200 pages, three links deep.
- 1,000
Requests per website in any 24 hours. Counted across every report.
- 90 sec
The longest a visit to one website lasts. No subdomains and no other sites.
- Weekly
The most often a store’s own website or menu is read again. Marketplace listings are read again every three days.
- Slower when you ask. A Crawl-delay longer than a second is honored; a shorter one doesn’t speed it up. A Retry-After header is honored exactly.
- It leaves when told twice. Two “too many requests” answers in one run and it leaves your site for that run, then goes at half speed for the next 24 hours.
- It follows your page tags. noindex and nofollow are respected.
- No way around a block. No CAPTCHA solving, no getting around bot protection, no proxies and no retrying under another name.
- A 403 ends the visit. So does a bot check or a login page. It is recorded and not retried.
- One code path. Every request goes through the same checks: robots.txt, the speed limit and the user-agent.
How to block it
Two lines in robots.txt
Add these to the robots.txt file at the root of your website. That’s all it takes.
robots.txt
User-agent: Esplanda-PriceCheck
Disallow: /- Read first, never stale. Esplanda-PriceCheck reads robots.txt before any other request to your site and never relies on a copy more than a day old. There is no setting anywhere that overrides it.
- Blocking all robots blocks us too. A “User-agent: *” group that disallows everything applies to Esplanda-PriceCheck as well. If you have a group for us and one for everyone, ours is followed.
- Never held against you. A report lists your website as a source that said no, and nothing more: no penalty, no score and no remark about your business.
- Your website only. robots.txt covers your own site. Prices you publish on a marketplace are read from that marketplace’s own published feed. To keep your business out of price checks entirely, ask for removal.
Removal and corrections
Asking is faster than blocking
Blocking stops future reading of your website. Removal takes your business out of price checks, wherever its prices come from.
Remove a business
Any business can be removed within one business day, customer or not, and we stop collecting its prices. No retention offer and no argument. Remove a business from price checks.
Correct a price
Each dispute is reviewed within two business days and ends in a correction or a removal. Dispute a price.
See how prices are used
How comparable businesses and products are chosen, and when a price is shown. How the free price check works.
Anything else about the crawler? Contact us and name the website it visited.
Questions
Crawler questions website owners ask
What is Esplanda-PriceCheck?
Esplanda-PriceCheck is the crawler behind Esplanda’s free price check. It reads the public prices that liquor stores, groceries and restaurants publish, so a store owner can compare their prices with comparable businesses nearby. It identifies itself on every request with one user-agent string that links to this page.
How do I block the Esplanda price check crawler?
Add two lines to the robots.txt file at the root of your website: User-agent: Esplanda-PriceCheck, then Disallow: /. The crawler reads robots.txt before any other request and never reads a site that says no. If your robots.txt can’t be reached or read, it treats that as a no too.
Does blocking the crawler count against my business?
No. A blocked website is recorded as a source that said no, and nothing more. It produces no penalty, no score and no statement about your business in any report.
How fast does the crawler read a website?
One request per second per website at most, shared across every report running at once, with one connection at a time. It makes no more than 200 requests to a website for one report and 1,000 in any 24 hours, and it honors Crawl-delay and Retry-After.
How do I get my business removed from price checks?
Ask for removal on the removal page. Any business can be removed within one business day, customer or not, and we stop collecting its prices. There is no retention offer and no argument.
Run a store? See what nearby stores charge
Get your free price check, or book a demo and we’ll walk through the prices near a store like yours.
Free. No credit card. No connection to your register. About three minutes. Your first 5 products show free, straight away; sign up free to see them all.
