W3C Link Checker and Robots.txt Exclusion

I was ready. I had just finished updating the last few links (and some content) on my clients website and I was ready to check for broken links. I opened the trusty link checker over at W3C Schools and got the following error:

Error: 403 Forbidden by robots.txt

This particular site is a WordPress site and I use the Google XML Sitemap plugin to handle all that stuff, which, in this case defaults to using WordPress to generate the robots.txt. I could, in the WordPress settings, allow search engines access but this was the staging server and I couldn’t allow robots to start searching and indexing it.

What I needed was an exception to allow the W3C Link Checker to search the site.

To solve this problem I first copied the exact text from the dynamically generated robots.txt file. I did this by navigating to the file at the root: http://yourclientsdomain/robots.txt. Next, I created a temporary robots.txt file in the root of the site. In it I pasted the exact text from the first step. Then from the W3C Link Checker documentation I added a robots exclusion rule for only their site. Like so:

User-Agent: *
Disallow: /

User-Agent: W3C-checklink

This allows the link checker access to the site, but continues to block all other robots. Once finished I renamed the robots.txt file in the root to robots_backup.txt. I wanted to keep it for the time being. I will be link checking again once we go live. Eventually I will delete it. Happy coding.

Posted in: Development  |  Tagged with: , ,  |  Leave a comment

Leave a Reply

Most recent work

  • Reductress Education Online satire workshops from the one and only Reductress.
  • Douglass Interior Products Since 1978, Douglass Interior Products has been providing exceptional materials for the aircraft, marine, automotive and interior design industries.
  • Betty Mills Betty Mills is an e-commerce store offering cleaning, sanitary & medical supplies.
  • Reductress I crank out on-going web projects for Reductress. Super fun as you can imagine.

Need hosting?

  • Media Temple My go to hosting provider. You get rockstar hosting and I make a little cash. No pressure, just clean honest fun.

Past work

  • How To Win At Feminism The badasses over at Reductress are at it again with their new book How To Win At Feminism.
  • Nice Manners Music Nice Manners is an LA based recording studio offering their clients all sorts of digital audio services. This is a demo of an internal genre based music search & player I built for them. I am ecstatic about launching this project—check it out.
  • Our Restroom Our Restroom is an (inter)national crowdsourced campaign who’s goal is simple: compel businesses to make their single occupancy restrooms gender neutral. Championed by Kristin Russo, designed by Allison Weiss, built by me on WordPress.
  • Get Your Shit Together This is a site dedicated to helping you get your shit together before an unexpected tragedy, like the loss of a loved one.
  • IHG Instagram Sweepstakes An awesome Instagram driven sweepstakes site for IHG.
  • Trivial Beersuit A site that provides professional pub trivia to bars and restaurants.
  • Audi YouTube Channel This site was a quick turn-around hair pulling adventure for the Super Bowl: revamping Audi’s YouTube Channel.
  • Everyone Is Gay Everyone Is Gay is an indispensable question and answer resource for the LGBTQ community. Hell, it’s great advice for anyone with a body with an emotion or two. I made the site responsive among other things. Give it a look see.
  • Teach.org The new Teach.org. Providing tools to help students become teachers.
  • Valhalla DSP A one man Seattle based company, writing pretty sick professional audio plugins. Check ’em out!
  • Christine Chaney Creative The online presence of Seattle artist Christine Chaney. Specializing in architecture, art and apparel.
  • Holiday Inn Tumblr A quick, fully responsive update to the Holiday Inn Tumblr page.
  • Microsoft's Art of Touch An artful HTML5 Microsoft website. Sadly, this site is no longer live.
  • Microsoft Hardware Healthy Computing Tool A useful little tool to help you find the perfect Microsoft mouse or keyboard.
  • Pella Professional Pella manufactures high quality doors and windows. This is their professional site.
  • Imaging the World A non-profit dedicated to providing ultrasound services in remote areas of the world.
  • Microsoft Visual Studio 2012 Launch This sweet new Visual Studio 2012 Launch website sports a mobile first, responsive web design architecture and comes in 14 different languages.
  • Ginny Ruffner: A not so still life A site for a documentary film about Seattle artist Ginny Ruffner.