Google has updated its information on specific subjects. it does get quite techo so I will try to make not tech help here:
Robots.txt file
This is plain text file main use is to keep Googles digital spider (the tools that finds your site) out of certain parts of your website. It talks about the commands in [...]