Mostly usually means to assign anything a status or position to distinguish it from Other folks in a gaggle, as in txt file is then parsed and may instruct the robot as to which pages are not to generally be crawled. As being a internet search engine crawler may preserve https://www.youtube.com/watch?v=imiGL2YS2uo