googlebot

Have you received a message recently in regards to “Googlebot cannot access css and js files” for your WordPress website? Here’s a simple way to rectify that issue.

On the morning of July 28, 2015 Google made a change in the way it views WordPress CSS and JS files and now adamantly wants access to the wp-admin and wp-includes folders. To remove this warning you will need:

Access to Webmaster Tools for your site (which you should already have if you are receiving the Googlebot warning.
FTP access for the website.

When you login to Webmaster Tools and look at the message for the site in question, you will see the following list:

To see what the issue is (and trust us they’re the same for most WordPress sites), follow the steps:

1. Fetch as Google
Click on “Fetch as Google” and you’ll be taken to the fetch page. Make sure you click on “FETCH AND RENDER” as you won’t get the proper report from “FETCH”.

After a few minutes you should see the results of the Fetch, and the status will most likely be “Partial”

Click on the line for the “Path” and you’ll be taken to the results page, which will show you the blocked resources that Googlebot wants access to (you may need to scroll down in your window to find it):

You can see that Googlebot wants access to the wp-includes folder and for some reason it can’t. Click on “robots.txt Tester” to see why.

2. Robots.txt Tester

Now we see the issue in raging red:

The robots.txt is blocking access.

WordPress Blocks Folders By Default and the Fix

In the near future we can expect a WordPress update to deal with this issue, but for the short term, here’s the fix.

First, if you access your site’s root folder in FTP, you’ll notice that there is no robots.txt file. This is because WordPress has a default block of the /wp-admin/ and /wp-includes/ folders. So, we need to change that. The answer lies within the /wp-includes/ folder itself. Navigate inside that folder and find the “functions.php” file. Open that file to edit it (we use Textmate for Mac).

Do a search for “robot” in the functions.php file. Around line 1094 you’ll find the following:

Do you see the lines that “Disallow” the /wp-admin/ and /wp-includes/ directories? That’s the issue. Remove those two lines (making sure you leave no blank lines) and save the file back to the server:

Problem Solved!

Now, go back to Webmaster Tools and do a “Fetch and Render” and check the results with the “robots.txt Tester”. You should find that Google is now happy with you once again!