articles written

Using a robots.txt file

A robots.txt file is an important tool when optimising a website. It tells web robots and spiders which areas of a site they may access.

apache web server