The robots.txt file tells a search engine where it is allowed to go on your website. You can edit the robots.txt file through Yoast SEO and Yoast SEO Premium. In this article, we』ll show you how to go about this!
If the robots.txt file is not writable or your WordPress install has disabled file editing, the following steps may fail. Optionally, you can edit at the server level.
Do you want to know more about what the robots.txt file is and what it does? Check out our Ultimate guide to robots.txt.
WordPress defaults
By default, WordPress will generate a robots.txt file with the following contents:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
When you create or customize your robots.txt file, this content is replaced.
Create or edit robots.txt in the WordPress Dashboard
The easiest way to create or edit the robots.txt file is through Yoast SEO in the WordPress Dashboard. To do this, follow the steps below.
Log in to your WordPress website. When you』re logged in, you will be in your 『Dashboard』. Click on 『SEO』. On the left-hand side, you will see a menu. In that menu, click on 『SEO』. Click on 『Tools』. The 『SEO』 settings will expand providing you with additional options. Click on 『Tools』. Click on 『File Editor』. This menu will not appear if your WordPress install has disabled file editing. Please enable file editing or edit the file through FTP. If you are not sure how to use FTP, your host provider can help. Make the changes to your file. Save your changes.
Learn how to use Yoast SEO to its full extent with the Yoast SEO for WordPress training
Create or edit on your server
If the robots.txt file is not writable or your WordPress install has disabled file editing, creating or editing robots.txt through the WordPress Dashboard may fail. In that case, you can edit at the server level. WordPress generates a virtual robots.txt file if the site root does not contain a physical file. To override the virtual file, please follow these steps to create a physical robots.txt file.
Use your favorite text editor and create a text file.Save the empty file with the name robots.txt.Upload the file to your server. If you』re unsure where on your server to upload the file, please contact your webhost.
If WordPress was blocking access to the virtual file, you should be able to edit the physical file from our plugin. If not, you can always edit the robots.txt directly on your server using FTP or a server file manager.
Please contact your web host for assistance if you have trouble uploading or editing files on your server.