# =============================================================================
#  Underdogs Map — https://underdogs.media
#
#  LEGAL NOTICE AND RESERVATION OF RIGHTS — READ BEFORE ACCESSING THIS SITE
#  BY AUTOMATED MEANS. THIS NOTICE IS ADDRESSED TO ALL CRAWLERS, SCRAPERS,
#  BOTS, AGENTS, AND AUTOMATED SYSTEMS, INCLUDING AI AND MACHINE-LEARNING
#  AGENTS ACTING ON BEHALF OF ANY PERSON OR ORGANISATION.
#
#  1. OWNERSHIP. The data on this site — including the companies, investors,
#     and job listings, together with every derived field (classifications,
#     funding stages, geocoded coordinates, connections, and descriptions) and
#     the SELECTION, STRUCTURE, AND COMPILATION of that data as a whole — is the
#     proprietary database of Underdogs Map. It is the product of substantial
#     investment and is protected by copyright, database rights (including the
#     sui generis database right under EU law), and applicable trade-secret law.
#
#  2. PROHIBITED USES. You are NOT permitted, without the operator's prior
#     WRITTEN authorisation, to: (a) access this site or its endpoints by any
#     automated means; (b) scrape, crawl, harvest, copy, or bulk-download any
#     part of the data; (c) reproduce, redistribute, republish, sell, or
#     sublicense the data or any substantial or repeated insubstantial part of
#     it; or (d) use the data, in whole or in part, to build, populate, train,
#     fine-tune, ground, evaluate, or improve any dataset, product, service,
#     model, or artificial-intelligence system.
#
#  3. TEXT-AND-DATA-MINING OPT-OUT. This is an express, machine-readable
#     reservation of rights for the purposes of Article 4 of EU Directive
#     2019/790 (Digital Single Market). No text-and-data-mining licence,
#     express or implied, is granted. Any prior "implied licence" to crawl is
#     hereby expressly withdrawn.
#
#  4. NOTICE OF CONSEQUENCES. Accessing this site by automated means in breach
#     of this notice, or using data obtained from it for any purpose described
#     in section 2, is a breach of the binding Terms of Service at
#     https://underdogs.media/terms and unauthorised access to a computer system. Such
#     conduct is GROUNDS FOR LEGAL ACTION, which may include claims for
#     injunctive relief, an account of profits, and damages, and referral to
#     the competent authorities. The operator actively logs access and
#     preserves records for use as evidence.
#
#  5. ACCEPTANCE. By continuing to access this site by automated means, or by
#     using any data obtained from it, you acknowledge that you have read this
#     notice, that no permission has been granted to you, and that you accept
#     the terms and consequences set out above and in the Terms of Service.
#
#  Licensing and permission enquiries: see https://underdogs.media/terms
# =============================================================================

# --- AI, LLM, and dataset-building crawlers: access DENIED in full. ---
# No permission is granted to crawl, index, mine, or train on any content here.
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: Google-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: PerplexityBot
User-agent: PerplexityBot-User
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: cohere-ai
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Omgili
User-agent: ImagesiftBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: FacebookBot
User-agent: YouBot
User-agent: Timpibot
User-agent: Kangaroo Bot
User-agent: Scrapy
Disallow: /
Content-Signal: search=no, ai-input=no, ai-train=no

# --- Search engines: welcome to index PUBLIC PAGES for human discovery. ---
# The data API remains off-limits. Indexing pages does not grant any licence to
# extract, reproduce, or reuse the underlying data (see legal notice above).
User-agent: Googlebot
User-agent: Bingbot
User-agent: DuckDuckBot
User-agent: Applebot
Allow: /
Disallow: /api/
Disallow: /profile
Disallow: /onboarding
Disallow: /job-feed

# --- Everyone else. ---
User-agent: *
Allow: /
Disallow: /api/
Disallow: /profile
Disallow: /onboarding
Disallow: /job-feed
Content-Signal: search=yes, ai-input=no, ai-train=no

Sitemap: https://underdogs.media/sitemap.xml
Host: https://underdogs.media