DutyFreeProfessorBot

The reader behind Duty Free Professor, a price-comparison guide for airport duty-free shopping. This page says what it does, how it announces itself, how it behaves, and how to limit or block it.

What it does

It reads product pages and price listings that airport duty-free shops publish openly on their websites, so that travellers can see how the same bottle or fragrance is priced from one airport to the next. Each price is stored with the date it was read and shown with that date.

• It reads only public pages. It never logs in, never adds anything to a basket, never buys, and never bypasses an access control, a paywall, a challenge page or a login.
• It collects prices, product names, sizes, barcodes and product images that the shop itself publishes. It collects no personal data of any kind.
• It reads a small share of a catalogue at a time, and most shops are revisited no more than once a day when refreshes are running.

How it behaves

• It announces itself by name in every request, with a link to this page.
• It obeys robots.txt: rules addressed to DutyFreeProfessorBot by name, wildcard rules, and any Crawl-delay a site publishes. A site whose robots file refuses the reader itself is treated as a refusal.
• It paces itself: at least one second between requests to a host, more when a site asks for it, one connection per host at a time.
• A refusal is a stop. If a site answers 401 or 403, or serves a challenge page, the reader stops for that site and does not try another way. It does not rotate addresses, change its identity, solve challenges or use any paid service to get through.
• Some pages are read in a real browser, because a few shops draw their prices only with scripts. That browser sends the same name, follows the same robots rules, loads only what the page needs to function, and blocks advertising and analytics beacons so nothing is reported to third parties.

How to limit or block it

Add a group for the reader to your robots.txt. It reads the file before every run and applies the change on the next visit.

User-agent: DutyFreeProfessorBot
Disallow: /

To slow it down rather than stop it:

User-agent: DutyFreeProfessorBot
Crawl-delay: 10

Or write to bot@dutyfreeprofessor.com with the hostname and what you would like, and it is done by hand. The same address is the place to tell us about a mistaken price, to ask for your shop to be included, or to arrange a direct data feed instead of page reads.