Tag Archives: MindSpider

How Do .Gov Sites Use Robots.txt

photo credit: ph0t0 {is on the move}
Looking back at some older post I revisited the SEOmoz post on government susceptibility to cross site scripting. The results of their test over 2 years ago make me curious how government sites interact with the search engines today.
So I went to Google and searched: “robots.txt” “disallow:” filetype:txt [...]