Library·SOP·SEO
Compare the public robots.txt file with Rank Math's editor, keep the current rules and any existing Sitemap line, make one evidence-backed narrow crawl change, then read the public file and key URLs again. robots.txt controls crawler access; it does not protect private information or reliably remove URLs from search results.
Last reviewed: · Editorial review
Clarity, structure and references reviewed in this version. This is not a test in a live account.
Compare the public robots.txt file with Rank Math's editor, keep the current rules and any existing Sitemap line, make one evidence-backed narrow crawl change, then read the public file and key URLs again. robots.txt controls crawler access; it does not protect private information or reliably remove URLs from search results.
This SOP covers:
This does not cover:
First confirm that a crawl rule is the right tool, then record the file that crawlers can actually fetch before opening Rank Math.
Open the supplied Technical SEO Action Plan, make your own copy, and use Start here to enter the website and today's review date.
Use robots.txt only to manage crawler access to a public URL path. It is public text, not a password: do not list secrets or rely on it to protect customer files. If the goal is to hide private data or remove a page from Google, stop and ask the site owner to use access controls or the correct noindex/removal method.
In a browser, open https://your-site.example/robots.txt using the same host as the public pages you care about. Replace the example host with your site's host; www and non-www, and http and https, can have different robots files.
In Action plan, add a row for this robots.txt URL. Copy the entire public text into What you found and record when you read it.
If /robots.txt is missing, shows an error, or returns a normal web page instead of rules, write down what you saw and ask the WordPress or hosting owner to confirm what crawlers receive. Do not assume the Rank Math editor controls the live file yet.
In WordPress, open Rank Math SEO. On its Dashboard, switch to Advanced Mode at the top right if needed, then choose General Settings → Edit robots.txt.
Look at the text field. If it contains text, Rank Math has a saved custom value. If it is blank and gray defaults appear, those words are placeholders and no custom value is saved in Rank Math. Record the editor state in Action plan → What you found.
If the field contains saved text, compare its rules with the public file. If the field is blank, use the public response as the baseline. A general note that physical files can override Rank Math is not proof this site has one.
If Rank Math says the file is locked, its saved text does not match the public response, or the public URL shows an error, stop. Record the difference and ask the WordPress or hosting owner which file is live. Do not delete or edit a physical server file as part of this guide.
A robots rule changes whether crawlers may fetch matching URLs. Choose one exact reason and path, then check what else the rule would match before saving it.
Read the live rules and identify one real problem: either narrow or remove one rule that blocks an important public page, or add one rule for a specific low-value path the owner confirms crawlers do not need. Disallow tells the named crawlers not to fetch matching paths. If the review finds no clear problem, record the reason in Action plan, set Status to No change needed, and skip the edit.
Check what lives at the target path. Make sure it does not contain important pages, images, stylesheets, or scripts Google needs to understand those pages. Never block CSS or JavaScript paths as a shortcut.
A rule path starts at the site's root and is case-sensitive. For example, Disallow: /catalog/previews/ matches that path and its descendants, but not /Catalog/previews/ or the slashless /catalog/previews. User-agent names the crawler group. Google selects the most-specific matching group, so a specific group may not inherit * rules. Within that group, the most-specific path wins; Google allows the URL if equally specific paths conflict. Ask the technical owner if groups or paths make the result unclear.
Keep all public rules except the single rule you plan to change. This includes existing admin, AJAX, and WooCommerce rules. If there is a Sitemap line, keep its full URL exactly as written.
In Action plan, put the robots.txt URL and target path in Page or issue, the full public baseline and reason in What you found, and the exact proposed line in Next action. Explain why this path should change in Why this comes first, name the WordPress owner, and put today's date in Review date. In Result / how to undo, record the editor's original state: blank, or the complete saved text. If text is too long for the cell, use an approved recovery note and record where it is. Set Status to In progress before editing.
Make only the recorded change, then check the public file and important paths. The public readback—not the editor's save message—is the proof that crawlers can receive the intended rules.
If you have recorded the public file, the editor is editable, and the proposed path is safe, prepare only the one rule recorded in Action plan. Use the matching existing User-agent group. If the editor is locked or there is no clear matching group, stop and ask the WordPress owner.
If the editor was blank with defaults shown only as gray placeholder text, paste the complete public baseline you saved, then add the one planned rule. Include the full baseline so your saved copy keeps the site’s existing rules. If the editor already held saved text, change only the planned line and keep the rest.
Click Save Changes once. Open the public /robots.txt URL in a fresh tab or reload it to read the actual response.
If the public text is missing the rule, changed other lines, or blocks an important path, restore the editor's original state: clear the field if it was originally blank, or restore its complete original saved text if it was nonempty. Click Save Changes and read /robots.txt again. If the public response does not match the original baseline, ask the WordPress or hosting owner to finish the rollback.
Compare the rule with the target URL path and the important page/resource path you recorded. Confirm the target is matched as intended and the important path does not match a disallow rule. Open the important public page and confirm it still loads and looks normal.
Record the public readback and URL checks in Result / how to undo. Set Status to Done only if the file and protected-page checks pass. If the editor and public file disagree or a check fails, set Waiting and name the owner who must resolve it. If no rule was justified, record the decision and set Status to No change needed.
Google may cache robots.txt for up to 24 hours or longer if it cannot refresh the file. A successful public readback confirms the site response now; it does not confirm when Google will next crawl. If the goal is to change whether a page appears in Search, use the proper indexing method with the site owner. A robots rule alone does not guarantee removal.
