Ten. Gzip Compression: Permit Gzip compression on your own server to lessen the sizing of data files transferred amongst the server as well as consumer’s browser. txt file is then parsed and can instruct the robotic as to which webpages will not be being crawled. Like a search engine crawler https://www.youtube.com/watch?v=Tmal7B28Td0