For this purpose to understand this method we create this post to How to set up WordPress robots txt using yoast robots Robots. txt file is use to direct search engine that which page or post can be crawl or not...