7. Be careful with dynamic URLs and frames. Dynamic pages are created as a result of user input (like entering a query into a search box). Searchbots can't input queries so they generally can't access dynamic content. Framed websites also present a problem because an individual page may be indexed outside its frame and lack navigation or other important page elements.
8. Generate a sitemap. A text-based sitemap is particularly important for websites that use some sort of dynamic menu system because spiders may have difficulty following links inside JavaScript menus, Flash files, or applets. Rules:
- Every page contains a text link to the sitemap.
- The sitemap is constantly updated to reflect website changes.
9. Verify your robots.txt file. A search spider is also called a "robot" and it relies on the robots.txt file to list pages that are ok to crawl. Check with Google, Yahoo and MSN for correct code.
10 Avoid spam. Your website rank may be penalized or you may just get ejected from the index. |