elasticsearch-analysis-annotation

Analysis plugin for ElasticSearch providing capability for processing inline
annotations in documents.

Description

Inline annotations are considered to be simple semantic informations inlined in
source text, that are removed from the indexed text and injected as synonyms at
positions of the words they are related to.

This plugin provides analyzer AnnotationAnalyzer as well as filter
InlineAnnotationFilter.
AnnotationAnalyzer is composed of WhitespaceTokenizer, LowerCaseFilter and
InlineAnnotationFilter (with default settings).
More sophisticated analyzers (equivalent to StandardAnalyzer or SnowballAnalyzer)
can be configure via configuration file elasticsearch.yml or web API.

Example

Let’s say we have this documents

"Mozart[artist] was born[lifeEvent] in Salzburg[city;Austria]"

If we parse this with StandardAnalyzer equivalent with annotation analysis added to it
we get these tokens - some are omitted due to used StopFilter.

                           | [austria]
[artist] | | [lifeevent] | |  [city]
 mozart  | |    born     | | salzburg

If we use StandardAnalyzer the result would be

mozart | artist | | born | lifeevent | | salzburg | city | austria

Installation

This plugin follows conventions for elasticsearch plugins, thus can be installed
in a standard manner - see http://www.elasticsearch.org/guide/reference/modules/plugins/

Using this plugin

To use those custom analyzers/filters you need to either modify elasticsearch.yml
configuration file - see http://www.elasticsearch.org/guide/reference/index-modules/analysis/ or specify
index mapping via elasticsearch API.

The following example configuration contains definitions for analyzers based on behaviour of
StandardAnalyzer and SnowballAnalyzer.

Please note that standard_annotation and snowball_annotation analyzers use standard tokenizer,
which removes all non-alphanumeric characters and thus makes it impossible to process inline
annotations marked with [,],; (which are used in default behaviour of InlineAnnotationFilter).

For this purpose we need to use mapping char filter, which remaps those special characters to
their equivalent, which will be accepted by standard tokenizer as part of the token.

 index :
     analysis :
         char_filter : 
             annotation_remap : 
                 type : mapping
                 mappings : ["[=>__annotation_start__", "]=>__annotation_end__",";=>__annotation_delimiter__"]
         analyzer :                
             standard_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop]
             snowball_annotation :
                 type : custom
                 tokenizer : standard
                 char_filter : annotation_remap
                 filter : [standard, lowercase, annotation_filter, stop, snowball]
         filter :
             annotation_filter :
                 type : annotation_filter
                 start : __annotation_start__
                 end : __annotation_end__
                 delimiter : __annotation_delimiter__

To test the analyzer you can query the following
http://localhost:9200/test/_analyze?analyzer=annotation&text=“Mozart[city;Salzburg]”

Limitation

Another thing to keep in mind is that you can’t use word-delimiting characters inside annotations.
The whole string would be treated as two tokens which would result in unexpected behaviour.

Customization

The InlineAnnotationFilter can be slightly customized.

List of supported options

• start - start delimiter for inline annotation
• end - end delimiter for inline annotation
• prefix - string to be prepended to synonym, that is created from inline annotation
• suffix - string to be apended to synonym, that is created from inline annotation
• token-type - token type of synonym
• delimiter - delimiter for multiple inline annotations

Example providing default values

index :
    analysis :
        analyzer :                
            annotation :
                type : annotation
                start : [
                end : ]
                prefix : [
                suffix : ]
                token-type: synonym
                delimiter : ;

Elasticsearch version

This plugin was successfuly tested on elasticsearch version 0.90.2