# robots.txt for http://svs.gsfc.nasa.gov/ # our own link checking code User-agent: LinkCheckerFromOstia Disallow: User-agent: * Disallow: /search/ Disallow: /temp/ Disallow: /prepub/ Disallow: /preview/