GEODI can also use web pages and RSS news sources as content. Web pages can have very variable structures. To support this variability, the Web Page data source offers many options.
We want to inform you about some important issues related to web page indexing. Indexing web pages may require intensive internet usage. Some sites may think it is a DDOS attack and ban your IP. Also, some page content, due to legal terms, may not be processed and used outside the site. Our sole responsibility is to provide software, and DECE can not be held responsible for any consequences.
Addresses
You can provide a single address or multiple addresses. Domain restriction settings will work independently for each address.
Level
With level=0 only the given page is indexed. The level must be large enough to access all pages. For cases with paging, the level value can be 1000+.
Parameters to Ignore
Many web pages use URL parameters. By default, GEODI creates content for each unique URL. But in some cases, a parameter may not change the content of the page. In such situations, you may ignore such parameters to get a better index result.
For example:
https://sample.com
https://sample.com?backtomail=true
Pages to Ignore
Some websites may have social media links, advertising pages, or similar pages that you don't want in to the index.
You may list as many pages as you need to exclude. Each page must be separated with “;”.
Wildcards are allowed. *adds* ; *last.html ignores any files containing adds and ends with last.html.
Render like Browsers
Normally, The page content is extracted from the HTML. But some web page content is dynamically created. In such cases, check this option. The indexing will be slower but the results will be better.
The alternative web browser module must be installed for the option to work.
How page content names are generated?
Page names are formed using the order of og:title → title → page URL
You can query pages using doc: ,