User-agent: * Crawl-delay: 10 # dont allow access to these folders on this server Disallow: /main/css/ Disallow: /main/includes/ Disallow: /main/_baks/ Disallow: /main/_notes/ Disallow: /main/documents/ Disallow: /main/search/ Disallow: /main/Templates/ Disallow: /Intranet/ # To disallow a specific file # Disallow: /directory/file.html #This example will Disallow anything in #/folder1/ except /folder1/myfile.html, since #the latter will match first. In case of #Google, though, the order is not important. # Allow: /folder1/myfile.html # Disallow: /folder1/