Class DataSourceConfigurationWebCrawlerConfigurationUrlsSeedUrlConfiguration


  • public final class DataSourceConfigurationWebCrawlerConfigurationUrlsSeedUrlConfiguration
    extends java.lang.Object
    • Method Detail

      • seedUrls

        public java.util.List<java.lang.String> seedUrls()
        Returns:
        List of seed or starting point URLs of the websites you want to crawl. The list can include a maximum of `100` seed URLs. Array Members: Minimum number of `0` items. Maximum number of `100` items. Length Constraints: Minimum length of `1`. Maximum length of `2048`.
      • webCrawlerMode

        public java.util.Optional<java.lang.String> webCrawlerMode()
        Returns:
        Default mode is set to `HOST_ONLY`. Valid values are `EVERYTHING`, `HOST_ONLY`, and `SUBDOMAINS`. `HOST_ONLY` crawls only the website host names. For example, if the seed URL is `"abc.example.com"`, then only URLs with host name `"abc.example.com"` are crawled. `SUBDOMAINS` crawls the website host names with subdomains. For example, if the seed URL is `"abc.example.com"`, then `"a.abc.example.com"` and `"b.abc.example.com"` are also crawled. `EVERYTHING` crawls the website host names with subdomains and other domains that the webpages link to.