# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: User-Agent: * # Disallow: / Sitemap: https://www.winealign.com/sitemaps/index.xml.gz Disallow: /wine_agents/*/update_wines Disallow: /wineries/*/update_wines Disallow: /profile/*/update Disallow: /top/update Disallow: /wines/locate/* Disallow: /m/v1/locate/* Disallow: /wines/*/pdf/*