Around 3 weeks ago I installed a new wordpress blog, on a new domain in order to play around with some software (which will be written about here at a later date). I spent some time setting up a search engine optimized site with a ton of pages. I then paid for some social bookmarking, got some link building done and waited. The site got “indexed” very quickly into Google but the listing looked like this:
www.domain.com
That was it. I figured it might be a new way that Google pre-indexes sites (without any meta data). I gave it about a week and then grew impatient that the bot still hadn’t crawled beyond the first page nor added Meta data to the listing. I started digging around and determined that something must be corrupt in the robots.txt. I checked what the default robots file looked like and it looked like this:
To the best of my knowledge the / indicates that the search engine should not crawl past the first page. I edited the robots.txt to this
Taking out the / should theoretically let the search engine go past the index page. I waited a few more days and still no dice.
Today, I was looking through the source of the page to see if anything jumped out at me as an obvious problem and I saw this in the source:
I thought that was pretty strange, especially considering I updated the robots.txt to not disallow anything. I did some more digging and found that in wordpress under the privacy setting their are 2 options:
1) I would like my blog to be visible to everyone, including search engines (like Google, Sphere, Technorati) and archivers
2) I would like to block search engines, but allow normal visitors
Guess which one was checked!
I know that i’m not possibly stupid enough to check that box, so I installed WordPress on a fresh domain. Guess what? It automatically checked thae box to only allow normal visitors. I’m not sure if this is a default setting for WordPress or just on my host but it’s kind of obnoxious.
The moral of this story is that if you are installing WordPress, especially for SEO purposes, check your privacy settings.
Isn’t this one of the first options you get asked when installing a new blog?
I have a couple of recent blogs that have received jack shit in the way of search engine traffic. Off to check them now…eep.
I’ve built a site before with htaccess only allowing my IP in; when I was done & happy with the site, I ran some PPC campaigns to the site, before removing the IP restriction! d’oh! Luckily, I caught it quickly, and didn’t waste too much money. Now I ask someone else to check the site, before I do PPC
@Finch – I installed this through Fantastico in Cpanel, so you arent asked anything.
@Eric – That’s a pretty interesting one.
Did you do a manual install or use an auto installer like Fantastico?
I noticed this setting as well when I used Fantastico to setup blogs instead of doing it myself via FTP.
Yea, when you install WP manually, the 3rd install optinon (Blog name then email) is that checkbox.
@Nick – Fantastico should add that question rather than just checking it. Also if they were just going to select something by default you would think it would be allow everyone.
I guess I gotta start just manually installing WordPress. I have to admit, a 1 click installation is pretty alluring though.
What is this i see about Link Building with out my help? Tsk Tsk 😉
Hmm I have used Fantastico before too… Guess I’d better check it.
Use seo-blog plugin that applies auto tweaks after fantastico… also may look into plugin-central.
I did this on 100 domains recently. I couldn’t figure out why they weren’t indexed. Robots.txt was to blame.
Ran into the same thing a few weeks ago. now I don’t feel so bad. I used fantastico to install wordpress.
Same deal here… and yes I also used Fantastico. Took me about a week to realize I had a problem… Suck. Ah well pinged the ol Goog once I had it fixed.
HAHAHAHAAHA.
We discovered THE SAME THING with our blog today. It looks like this got turned on the Word Press default for some reason.
Before, it used to have the “search friendly” option selected for me but now it automatically selects “Don’t index pages, allows normal visitors only” on my manual installs.
Maybe WordPress changed it or something.
None the less, hopefully changing it will get you indexed finally.
so is there anyone using Drupal? i find myself using WP more for things, but Drupal is still in my toolbox too.
more on topic, i think i recall having to check the box in one of the early screens.
Google is crawling that site like crazy now. Good lesson learned here 🙂
Sup Brandon, you mentioned outsourcing the linkbuilding process. How did that service perform? If you liked it could you recommend them(feel free to email me). I might need help in that area for a project. Your post always rock!
I got same problem a few times when I did this job by fantastico, now this thing is first thing when I setup a wordpress blog.
@Ad Hustler “@Finch – I installed this through Fantastico in Cpanel, so you arent asked anything.”
That’s probably one reason why manual install is preferred over Fantastico.
that and i heard the auto-installs make updates a little sticky sometimes. i raise my hand for manual install too, though. doesn’t take that much longer…
Yikes, that sucks! Maybe you should just make a bot that installs wordpress, all the required plugins / activates them, permalink structure and makes sure the “se blocking” is unchecked. 😛
Thanks for the heads up. I am now going to check my blogs to make sure this is not checked off on any of them. i have about a dozen blogs. It would really suck big time if that was the case and those things were check off allowing only normal visitors.