Crawler Configuration
The Site Search Crawler can use web technologies to improve the granularity and scope of a crawl.
But there are things that the Crawler cannot understand, like IFrames or client side JavaScript...
Use the sidebar or explore the links below:
- Advanced settings
- Canonical Crawling
- Constant Crawl
- Content Inclusion & Exclusion
- Crawling Password Protected or Private Content
- IFrames
- Image Crawling
- Language Optimization
- Meta Tags
- PDF Crawling
- robots.txt
- rel="nofollow"
- RSS and Atom
- Sitemap XML
- URL Inspector
- Video Crawling
Stuck? Looking for help? Contact support or check out the Site Search community forum!