# to protect the site from web-crawlers (search engine components) from indexing the contents in any files User-agent: * Disallow: /