Examine your robots tags, in general you should use
<meta name="robots" content="robots-terms">
The content=”robots-terms” is a comma separated list used in the Robots META Tag that may contain one or more of the following commands without regard to case: noindex, nofollow, all, index and follow. If you decide to include Robots Meta Tag, use the following code to allow all spiders to index and follow the page:
<meta name="robots" content="index, follow">

For Printer-Friendly pages, you may want to use the following command to disallow spiders from indexing the page to prevent Duplicate Content
<meta name="robots" content="noindex, follow">

domain.com to www.domain.com
Options +FollowSymlinks
RewriteEngine on
rewritecond %{http_host} ^domain.com [nc]
rewriterule ^(.*)$ http://www.domain.com/$1 [r=301,nc]

example:
http://www.domain-name.com
http://www.domain-name.com/index.html

http://www.domain-name.com/news/
http://www.domain-name.com/news/index.html

The best way is always redirect URLs with file extensions to URLs ending with slash, this is will also save you future headache when switching languages or platforms. To do so add the following code into your “.htaccess” file
RewriteCond %{THE_REQUEST} ^[^/]*/index\.html [NC]
RewriteRule . / [R=301,L]

Each site must have a comprehensive HTML site map page that contains hard coded direct links to all pages on site.
-
This allows spiders easily find and crawl all your pages,
-
It also makes all your pages 2 clicks away from the home page
- And it prevents
of having orphaned pages on the site with no incoming links.