Technical Web optimization: Specialized SEO refers to Web page and server optimization that helps the crawler in crawling, indexing and ranking functions to rank your website superior. txt file is then parsed and can instruct the robot as to which web pages are not for being crawled. For a search https://www.youtube.com/watch?v=RSlwOZLlUGg