Versions Compared

Version Old Version 7 New Version Current
Changes made by

serdar ak

serdar ak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Indexing is what GEODI crawls all data and runs the the result of discovery. GEODI creates brief information about the data and uses this brief to answer searches, reports, and all othersother requests. GEODI will not need the original data unless you want to open it in a viewer or start indexing.

Table of Contents
stylenone

Info

Continuous Discovery

Once you index data sources, the process will repeat automatically for new data(rows, files, emails, etc.). You do not need to intervene; just tell GEODI the recheck period. This is done in Project Wizard.

Info

Index Storage

  1. The index needs some storage. The size will be much smaller than the data, but it is unpredictable. You may assume 1/10 generally.

  2. Options like sampling mode or similarity indexing., affect the index size.

  3. A backup space for the index should also be reserved for uninterrupted service.

Info

Indexing Time

The time very depends on the CPU, Memory, Disk and other resources of the Server

Indexing Speed

  1. GEODI has an indexing speed setting. We suggest that you set it to maximum.

  2. The first indexing may take time, but continuous discovery does not. GEODI indexes only new or changed content. The default is rescans per day, but you may set it to any period.

  3. There are other factor that affects indexing speed:

    1. The time depends on the CPU, memory, disk, and other server resources that GEODI runs on.

    2. Data throughput of the Network and Disk of data sources

    are

    1. is also very important.

    2. Options like OCR or FacePro greatly affect the performance.

  4. The GEODI indexing engine multitasks and processes multiple documents simultaneously. The Indexing Speed parameter mentioned in the Settings tab affects this speed. If the server has no task other than GEODI, we suggest that you set it to maximum. GEODI will use the CPU as much as possible but leave at least a core for user interaction.

Image Removed

  1. Sampling is a great way to increase speed. While it gives partial results, it may be enough to provide an insight into the data.

  2. Use file content filtering to eliminate “unnecessary” content. Default rules eliminate many file types not related to GDPR or PCI/DSS.

Info

Options

The beginning should be “Index all Content”Content.” After that, you may use all other options. If you set scheduled indexing and have periodic backups, you will not need to use options other than maintenance needs.

image-20240728-074018.png

...

Info

Monitoring Indexing

GEODI informs you about the progress Once you start indexing, the process can be monitored.

  1. The progress bar shows the approximate percentage of indexing. Please be careful about the progress bar, which is not

lineer

  1. linear.

That is

  1. GEODI

can

  1. does not know how much time it will require to index

for

  1. future documents, so the progress bar is only an estimate using the previous document indexing time.

Sampling

Sampling is possible for both, structured and unstructured data. Each data source asks you the sampling values. Sampling saves great time for discovery projects. We suggest you always use sampling for DB discovery. For unstructured data sampling is also a good starting point. Start with sampled mode and see what is in data, are there any unnecessary types or are there any permission problem.

  1. This area shows the numbers. The graphics show trends in terms of documents/second.

  2. This shows if any error or warning was generated. You can click and download the report, which resides in the % appdata%logs folder.

  3. Error reports and detailed reports about the project.

image-20240921-092936.pngImage Added

Info

Index Storage

  1. The index needs some storage. The size will be much smaller than the data, but it is unpredictable. You can generally assume 10% to 20%.

  2. Options like sampling mode or similarity indexing affect the index size.

  3. A backup space for the index should also be reserved for uninterrupted service.

File Content Filtering

Any corpus contains various file types. Some may not be necessary for the project scope, and some may be too large to disrupt for the network or unwanted at allthe system.

GEODI comes ready with default rules to avoid some file types and place some size limitations, collected from the best practices of many Discovery projects. Here, we documented the rules and how you can modify them.

These filters apply to all files, whether they come from a folder, from an email attachment, from GDE, or embedded into a database.

Filters (%100)

Explanation

IgnoreRules

Ignore rules contain file extensions, directory names, and some patterns.

Default The default list contains *.DLL, *.SYS, “programm “program files” and similar.

Any file matches the ignore rules are not indexed and logged at all.

Settings are in:

  • <geodi>Settings\IgnoreFileTypes

  • <geodi>\Settings\IgnoreFolders

(warning) If you need to override defauls defaults please do it in %appdata%

(info) GDE has some additional ignorerules documented in the related page.

KnownFiles

These files are the ones GEODI has a reader like PDF, DOCX etc. The full fu l list is Supported Formats

These files are processed as expected unless there is an IgnoreRule or ProtectRule. Ignorule will set the file type invisible. Protectrule Protec rule may set so size limitation.

UnknownFiles

By default unknownfile , unknown file types are ignored.

You may override this settings from Project Wizard use advanced settings.

If you use “only name and date” then all unknown extensions will be indexed.

You may add any unwanted to ignore list but these actions requires to run discovery all over again.

image-20240919-115040.png

ProtectRules

These rules is to protects protect the system and network again too against large files. Protect Protec rules apply to known and unknown files.

The content are is grouped as local and far. There is 's no limitation for local content, which resides in local folders and network folders. Far means me ns files from GDE, e-mail email attachments, and files from web pages.

By default, Far content is filtered as any file greater than 100Mb, and Compressed files greater than 500Mb are indexed as name only. You will wi l know these files but not their content.

Settings are in:

<geodi>\Settings\Engine\ResourceBalancing

(warning) If you need to override defauls defaults, please do it in %appdata%

Query of Indexed Content

GEODI provides a rich query language to query from coıntent, duplicates, date ranges, and more. Here are unique queries to help you query erroneous content and more.

  1. status:OnlyName → gives the content with only the name and date. These files come from UnknownFiles and ProtectRules.

  2. status:HasScanError → Files unread by file error, encryption, or other reasons are marked by an ! at the name.  

  3. status:IsContainer → Files within a folder or ZIP/RAR.

  4. status:IsCompletedIndex → Content succesful indexed.

  5. status:Crashed → In case of an index recovery after a system crash, GEODI will recover the index. This query shows the unsuccessful content. To avoid these, you should use an index backup.

  6. status:PartialRead → Content partially read by protectRules.

  7. GEODICryptedContent → encrypted content. 

  8. GEODICryptedContentPart → Partly encrypted content

Troubleshooting

Expand
titleIndexing is slow

The GEODI discovery engine is one of the fastest among other discovery engines. Slow indexing may depend on the machine, settings, or enviromentenvironment.

  1. Check the indexing speed and be sure that the speed is high

  2. Check engine errors, . If a source is throwing too many errors, this may slow down the indexing

  3. Another task may be using too much many resources,

  4. Too many recognizers may slow down a the indexing

  5. Slow A slow disk may slow down indexing. Consider dividing the index and putting some part to on a fast disk, like an SSD.

  6. Use sampling mode if you need a quciker quicker result.

Expand
titleToo much CPU is used

High CPU usage for an engine like GEODI should be expected. The CPU usage of GEODI never goes to an unresponsive machine state. GEODI always leaves one core to other tasks.

  1. CPU usage may be temporary, just ; wait to see if it drops

  2. If a consistent CPU usage is observed, try decreasing the indexing speed

  3. OCR or FacePro needs CPU, if consumes more CPU resources. If you are using these options, decrease the indexing speed or wait.

Expand
titleIndex size is too high

GEODI compressed to index as much as possible. Index size upto %20 up % to 20 percent of corpus size should be expected. If it looks too high to you, then you may try the following.

  1. Similarity index may be open. This index need disk.

  2. Some files may hav too much information (logs, csv CSV, etc.) may contain too much information, so you may exlude want to exclude them.

Expand
titleIndex errors

GEODI generates error logs during indexing. These logs mostly are about concern content and shuld should be considered as warning warnings or infoinformation. There may be real erros about system , errors; you will be informed about them.

Most of them are

  1. Unreadable content

  2. Encyrpted Encrypted content

  3. Unreachable content (because of permissions)

FAQ

Expand
titleMyDocument count and GEODI content count does not match

The GEODI content count includes all folders and files in compressed files like ZIP and RAR, so it's typical for the count not to match.

Some file types may be in ignore lists.

Other than that be sure that GEODI covers all.