[id='search_api']
=  Indexing and Searching

{brandname} provides a search API that lets you index and search cache values stored as Java POJOs or as objects encoded as link:https://developers.google.com/protocol-buffers/[Protocol Buffers].

== Overview

Searching is possible both in link:#query_library[library] and link:#query_remote[client/server mode] (for Java, C#, Node.js and other clients), and {brandname} can index data using link:http://lucene.apache.org/[Apache Lucene], offering an efficient link:https://en.wikipedia.org/wiki/Full-text_search[full-text] capable search engine in order to cover a wide range of data retrieval use cases.

Indexing configuration relies on a schema definition, and for that {brandname} can use annotated Java classes when in library mode, and protobuf schemas for remote clients. By standardizing on protobuf, {brandname} allows full interoperability between Java and non-Java clients.

{brandname} has its own query language called link:#query_ickle[Ickle], which is string-based and adds support for full-text searching. Ickle support searches over indexed data, partially indexed
data or non-indexed data.

Finally, {brandname} has support for link:#query_continuous[Continuous Queries], which works in a reverse manner to the other APIs: instead of creating, executing a query
and obtain results, it allows a client to register queries that will be evaluated continuously as data in the cluster changes, generating notifications
whenever the changed data matches the queries.

== Indexing Entry Values

Indexing entry values in {brandname} caches dramatically improves search performance and allows you to perform full-text queries. However, indexing can degrade write throughput for {brandname} clusters. For this reason you should plan to use strategies to optimise query performance, depending on the cache mode and your use case. More information on link:#query_performance[query performance guide].

=== Configuration

To enable indexing via XML, you need to add the `<indexing>` element to your cache configuration, specify the entities that are indexed and optionally pass additional properties.

[NOTE]
====
The presence of an `<indexing>` element which omits the `enabled` attribute will auto-enable indexing for your
convenience, even though the default value of the `enabled` attribute is defined as `"false"` in the XSD schema.
In the programmatic config, `enabled()` must be used.
====

.Declaratively
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing.xml[]
----

.Programmatically
[source,java]
----
import org.infinispan.configuration.cache.*;

ConfigurationBuilder cacheCfg = ...
cacheCfg.indexing().enable()
            .addIndexedEntity(Book.class)
      .addProperty("property name", "propery value")
----

=== Specifying Indexed Entities

It is recommended to declare the indexed types, as they will be mandatory in the next {brandname} version.

.Declaratively
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexed_entities.xml[]
----

.Programmatically
[source,java]
----
 cacheCfg.indexing()
       .addIndexedEntity(Car.class)
       .addIndexedEntity(Truck.class)

----

When the cache is storing protobuf, the indexed types should be the `Message` declared in the protobuf schema.
For example, for the schema below:

[source,proto]
----
include::config_examples/library.proto[]
----

The config should be:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexed_entities_proto.xml[]
----

[[query_index_storage]]
=== Index Storage

{brandname} can store indexes in the file system or in memory (`local-heap`). File system is the recommended and the default configuration, and memory indexes should only be used for small to medium indexes that don't need to survive restart.

.Configuration for file system indexes:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_filesystem.xml[]
----

.Configuration for memory indexes:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_local_heap.xml[]
----

[[query_index_manager]]
=== Index Manager

{brandname} uses internally a component called "Index Manager" to control how new data is applied to the index and
when the data is visible to searches.

The default Index Manager `directory-based` writes to the index as soon as the data is written to the cache. The downside is it can slow down considerably cache writes specially under heavy writing scenarios, since it needs to do constant costly operations called "flushes" on the index.

The `near-real-time` index manager is similar to the default index manager but takes advantage of the Near-Real-Time features of Lucene. It has better write performance because it flushes the index to the underlying store less often.
The drawback is that unflushed index changes can be lost in case of a non-clean shutdown. Can be used in conjunction with `local-heap` or `filesystem`.

Example with `local-heap`:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_near_real_time.xml[]
----

Example with `filesystem`:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_near_real_time_fs.xml[]
----

[[query_massindexer]]
=== Rebuilding Indexes

Rebuilding an index reconstructs it from data stored in the cache. You need to rebuild indexes if you change things like
definitions of indexed types or Analyzers. Likewise you might need to rebuild indexes if they are deleted for some reason.
Beware it might take some time as it needs to reprocess all data in the grid!

[source,java]
----
Indexer indexer = Search.getIndexer(cache);
CompletionStage<Void> future = index.run();
----

//TODO
//[[query_indexless]]
//==== Indexless

//TODO
//[[query_hybrid]]
//==== Hybrid

[[searching_ickle]]
== Searching

Create relational and full-text queries in both Library and Remote Client-Server mode with the Ickle query language.

To use the API, first obtain a QueryFactory to the cache and then call the `.create()` method, passing in the string
to use in the query. Each `QueryFactory` instance is bound to the same `Cache` instance as the `Search`, but it is
otherwise a stateless and thread-safe object that can be used for creating multiple queries in parallel.

For instance:

[source,java,tile="Using Ickle"]
----
// Remote Query, using protobuf
QueryFactory qf = org.infinispan.client.hotrod.Search.getQueryFactory(remoteCache);
Query q = qf.create("from sample_bank_account.Transaction where amount > 20");

// Embedded Query using Java Objects
QueryFactory qf = org.infinispan.query.Search.getQueryFactory(cache);
Query q = qf.create("from com.acme.Book where price > 20");

// Execute the query
QueryResult<Book> queryResult = q.execute();
----

[NOTE]
====
A query will always target a single entity type and is evaluated over the contents of a single cache. Running a
query over multiple caches or creating queries that target several entity types (joins) is not supported.
====

Executing the query and fetching the results is as simple as invoking the `run()` method of the `Query` object. Once
executed, calling `run()` on the same instance will re-execute the query.

=== Pagination

You can limit the number of returned results by using  the `Query.maxResults(int maxResults)`. This can be used in
conjunction with `Query.startOffset(long startOffset)` to achieve pagination of the result set.

[source,java]
----
// sorted by year and match all books that have "clustering" in their title
// and return the third page of 10 results
Query<Book> query = queryFactory.create("FROM com.acme.Book WHERE title like '%clustering%' ORDER BY year").startOffset(20).maxResults(10)
----

=== Number of Hits

The `QueryResult` object has the `.hitCount()` method to return the total number of results of the query, regardless
of any pagination parameter. The hit count is only available for indexed queries for performance reasons.


=== Iteration

The `Query` object has the `.iterator()` method to obtain the results lazily. It returns an instance of `CloseableIterator` that must be closed after usage.

[NOTE]
====
The iteration support for Remote Queries is currently limited, as it will first fetch all entries to the client
before iterating.
====

=== Using Named Query Parameters

Instead of building a new Query object for every execution it is possible to include named parameters in the query which
can be substituted with actual values before execution. This allows a query to be defined once and be efficiently
executed many times. Parameters can only be used on the right-hand side of an operator and are defined when the query is
created by supplying an object produced by the `org.infinispan.query.dsl.Expression.param(String paramName)` method to
the operator instead of the usual constant value. Once the parameters have been defined they can be set by invoking either
`Query.setParameter(parameterName, value)` or `Query.setParameters(parameterMap)` as shown in the examples below.
⁠
[source,java,tile="Using Named Parameters"]
----
QueryFactory queryFactory = Search.getQueryFactory(cache);
// Defining a query to search for various authors and publication years
Query<Book> query = queryFactory.create("SELECT title FROM com.acme.Book WHERE author = :authorName AND publicationYear = :publicationYear").build();

// Set actual parameter values
query.setParameter("authorName", "Doe");
query.setParameter("publicationYear", 2010);

// Execute the query
List<Book> found = query.list();
----

Alternatively, you can supply a map of actual parameter values to set multiple parameters at once:
⁠
[source,java,title="Setting multiple named parameters at once"]
----
Map<String, Object> parameterMap = new HashMap<>();
parameterMap.put("authorName", "Doe");
parameterMap.put("publicationYear", 2010);

query.setParameters(parameterMap);
----

[NOTE]
====
A significant portion of the query parsing, validation and execution planning effort is performed during the first
execution of a query with parameters. This effort is not repeated during subsequent executions leading to better
performance compared to a similar query using constant values instead of query parameters.
====

[[query_ickle]]
=== Ickle Query Language Parser Syntax

The Ickle query language is small subset of the link:https://en.wikipedia.org/wiki/Java_Persistence_Query_Language[JPQL]
query language, with some extensions for full-text.

The parser syntax has some notable rules:

* Whitespace is not significant.
* Wildcards are not supported in field names.
* A field name or path must always be specified, as there is no default field.
* `&&` and `||` are accepted instead of `AND` or `OR` in both full-text and JPA predicates.
* `!` may be used instead of `NOT`.
* A missing boolean operator is interpreted as `OR`.
* String terms must be enclosed with either single or double quotes.
* Fuzziness and boosting are not accepted in arbitrary order; fuzziness always comes first.
* `!=` is accepted instead of `<>`.
* Boosting cannot be applied to `>`,`>=`,`<`,`<=` operators. Ranges may be used to achieve the same result.

==== Filtering operators

Ickle support many filtering operators that can be used for both indexed and non-indexed fields.

[options="header"]
|==============================================================================
| Operator | Description | Example
| in | Checks that the left operand is equal to one of the elements from the Collection of values given as argument.
|FROM Book WHERE isbn IN ('ZZ', 'X1234')
| like | Checks that the left argument (which is expected to be a String) matches a wildcard pattern that follows the JPA rules.| FROM Book WHERE title LIKE '%Java%'
|=| Checks that the left argument is an exact match of the given value         | FROM Book WHERE name = 'Programming Java'
|!=| Checks that the left argument is different from the given value            | FROM Book WHERE language != 'English'
|>| Checks that the left argument is greater than the given value.             | FROM Book WHERE price > 20
|>=| Checks that the left argument is greater than or equal to the given value. | FROM Book WHERE price >= 20
|<| Checks that the left argument is less than the given value.                | FROM Book WHERE year < 2012
|<=| Checks that the left argument is less than or equal to the given value.   | FROM Book WHERE price  <= 50
|between| Checks that the left argument is between the given range limits.  | FROM Book WHERE price BETWEEN 50 AND 100
|==============================================================================

==== Boolean conditions

Combining multiple attribute conditions with logical conjunction (`and`) and disjunction (`or`) operators in order to
create more complex conditions is demonstrated in the following example. The well known operator precedence rule for
boolean operators applies here, so the order of the operators is irrelevant. Here `and`
operator still has higher priority than `or` even though `or` was invoked first.

[source,sql]
----
# match all books that have "Data Grid" in their title
# or have an author named "Manik" and their description contains "clustering"

FROM com.acme.Book WHERE title LIKE '%Data Grid%' OR author.name = 'Manik' AND description like '%clustering%'
----

Boolean negation has highest precedence among logical operators and applies only to the next simple attribute condition.

[source,sql]
----
# match all books that do not have "Data Grid" in their title and are authored by "Manik"
FROM com.acme.Book WHERE title != 'Data Grid' AND author.name = 'Manik'

----

==== Nested conditions
Changing the precedence of logical operators is achieved with parenthesis:

[source,sql]
----
# match all books that have an author named "Manik" and their title contains
# "Data Grid" or their description contains "clustering"
FROM com.acme.Book WHERE author.name = 'Manik' AND ( title like '%Data Grid%' OR description like '% clustering%')
----

==== Selecting attributes
In some use cases returning the whole domain object is overkill if only a small subset of the attributes are actually
used by the application, especially if the domain entity has embedded entities. The query language allows you to specify
a subset of attributes (or attribute paths) to return - the projection. If projections are used then the `QueryResult.list()`
will not return the whole domain entity but will return a `List` of `Object[]`, each slot in the array corresponding to
a projected attribute.

//TODO document what needs to be configured for an attribute to be available for projection.

[source,sql]
----
# match all books that have "Data Grid" in their title or description
# and return only their title and publication year
SELECT title, publicationYear FROM com.acme.Book WHERE title like '%Data Grid%' OR description like '%Data Grid%'
----

==== Sorting
Ordering the results based on one or more attributes or attribute paths is done with the `ORDER BY` clause. If multiple sorting criteria
are specified, then the order will dictate their precedence.


//TODO document what needs to be configured for an attribute to be available for sorting.

[source,sql]
----
# match all books that have "Data Grid" in their title or description
# and return them sorted by the publication year and title
FROM com.acme.Book WHERE title like '%Data Grid%' ORDER BY publicationYear DESC, title ASC
----

==== Grouping and Aggregation

{brandname} has the ability to group query results according to a set of grouping fields and construct aggregations of
the results from each group by applying an aggregation function to the set of values that fall into each group.
Grouping and aggregation can only be applied to projection queries (queries with one or more field in the SELECT clause).

The supported aggregations are: avg, sum, count, max, min.

The set of grouping fields is specified with the `GROUP BY` clause and the order used for defining grouping fields is
not relevant. All fields selected in the projection must either be grouping fields
or else they must be aggregated using one of the grouping functions described below. A projection field can be
aggregated and used for grouping at the same time. A query that selects only grouping fields but no aggregation fields
is legal.
⁠
Example: Grouping Books by author and counting them.
[source,sql]
----
SELECT author, COUNT(title) FROM com.acme.Book WHERE title LIKE '%engine%' GROUP BY author
----

[NOTE]
====
A projection query in which all selected fields have an aggregation function applied and no fields are used for
grouping is allowed. In this case the aggregations will be computed globally as if there was a single global group.
====

==== Aggregations

The following aggregation functions can be applied to a field:


* `avg()` - Computes the average of a set of numbers. Accepted values are primitive numbers and instances of `java.lang.Number`. The result is represented as `java.lang.Double`. If there are no non-null values the result is `null` instead.
* `count()` - Counts the number of non-null rows and returns a `java.lang.Long`. If there are no non-null values the result is `0` instead.
* `max()` - Returns the greatest value found. Accepted values must be instances of `java.lang.Comparable`. If there are no non-null values the result is `null` instead.
* `min()` - Returns the smallest value found. Accepted values must be instances of `java.lang.Comparable`. If there are no non-null values the result is `null` instead.
* `sum()` - Computes the sum of a set of Numbers. If there are no non-null values the result is `null` instead. The following table indicates the return type based on the specified field.

.Table sum return type
|===
|Field Type |Return Type

|Integral (other than BigInteger)
|Long

|Float or Double
|Double

|BigInteger
|BigInteger

|BigDecimal
|BigDecimal
|===

==== Evaluation of queries with grouping and aggregation

Aggregation queries can include filtering conditions, like usual queries. Filtering can be performed in two stages: before
and after the grouping operation. All filter conditions defined before invoking the `groupBy()` method will be applied
before the grouping operation is performed, directly to the cache entries (not to the final projection). These filter
conditions can reference any fields of the queried entity type, and are meant to restrict the data set that is going to
be the input for the grouping stage. All filter conditions defined after invoking the `groupBy()` method will be applied to
the projection that results from the projection and grouping operation. These filter conditions can either reference any
of the `groupBy()` fields or aggregated fields. Referencing aggregated fields that are not specified in the select clause
is allowed; however, referencing non-aggregated and non-grouping fields is forbidden. Filtering in this phase will
reduce the amount of groups based on their properties. Sorting can also be specified similar to usual queries. The
ordering operation is performed after the grouping operation and can reference any of the `groupBy()` fields or aggregated
fields.

==== Using Full-text search

===== Fuzzy Queries

To execute a fuzzy query add `~` along with an integer, representing the distance from the term used, after the term.
For instance

[source,sql,tile="Fuzzy Queries in Ickle"]
----
FROM sample_bank_account.Transaction WHERE description : 'cofee'~2
----

===== Range Queries

To execute a range query define the given boundaries within a pair of braces, as seen in the following example:

[source,sql,tile="Range queries with Ickle"]
----
FROM sample_bank_account.Transaction WHERE amount : [20 to 50]
----

===== Phrase Queries

A group of words can be searched by surrounding them in quotation marks, as seen in the following example:

[source,sql,tile="Phrase queries with Ickle"]
----
FROM sample_bank_account.Transaction WHERE description : 'bus fare'
----

===== Proximity Queries

To execute a proximity query, finding two terms within a specific distance, add a `~` along with the distance after the phrase.
For instance, the following example will find the words canceling and fee provided they are not more than 3 words apart:

[source,sql,tile="Proximity queries with Ickle"]
----
FROM sample_bank_account.Transaction WHERE description : 'canceling fee'~3
----

===== Wildcard Queries

To search for "text" or "test", use the `?` single-character wildcard search:

[source,sql,tile="Single-character wildcard queries with Ickle"]
----
FROM sample_bank_account.Transaction where description : 'te?t'
----

To search for "test", "tests", or "tester", use the `*` multi-character wildcard search:

[source,sql,tile="Multi-character wildcard queries with Ickle"]
----
FROM sample_bank_account.Transaction where description : 'test*'
----

===== Regular Expression Queries

Regular expression queries can be performed by specifying a pattern between `/`. Ickle uses Lucene’s regular expression syntax, so to search for the words `moat` or `boat` the following could be used:

[source,sql,tile="Regular Expression queries with Ickle"]
----
FROM sample_library.Book  where title : /[mb]oat/
----

===== Boosting Queries

Terms can be boosted by adding a `^` after the term to increase their relevance in a given query, the higher the boost factor the more relevant the term will be. For instance to search for titles containing beer and wine with a higher relevance on beer, by a factor of 3, the following could be used:

[source,sql,tile="Boosting queries with Ickle"]
----
FROM sample_library.Book WHERE title : beer^3 OR wine
----

[[query_library]]
== Embedded Search

Embedded searching is available when {brandname} is used as a library. No protobuf mapping is required, and both indexing and searching are done on top of Java objects.

=== Quick example

We're going to store `Book` instances in an {brandname} cache called "books". `Book` instances will be indexed, so we enable indexing for the cache:

{brandname} configuration:

.infinispan.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/infinispan_distributed_cache_books.xml[]
----

Obtaining the cache:

[source,java]
----
import org.infinispan.Cache;
import org.infinispan.manager.DefaultCacheManager;
import org.infinispan.manager.EmbeddedCacheManager;

EmbeddedCacheManager manager = new DefaultCacheManager("infinispan.xml");
Cache<String, Book> cache = manager.getCache("books");

----

Each `Book` will be defined as in the following example; we have to choose which properties are indexed, and for each property we can optionally choose advanced indexing options using the annotations defined in the Hibernate Search project.

[source,java]
.Book.java
----
import org.hibernate.search.annotations.*;
import java.util.Date;
import java.util.HashSet;
import java.util.Set;

//Values you want to index need to be annotated with @Indexed, then you pick which fields and how they are to be indexed:
@Indexed
public class Book {
   @Field String title;
   @Field String description;
   @Field @DateBridge(resolution=Resolution.YEAR) Date publicationYear;
   @IndexedEmbedded Set<Author> authors = new HashSet<Author>();
}

----

[source,java]
.Author.java
----

public class Author {
   @Field String name;
   @Field String surname;
   // hashCode() and equals() omitted
}

----

Now assuming we stored several `Book` instances in our {brandname} `Cache` , we can search them for any matching field as in the following example.

[source,java]
.QueryExample.java
----
include::code_examples/QueryExample.java[]
----

Apart from `list()` you have the option for obtaining on `iterator()`, or use pagination.

[[mapping_embedded]]
=== Mapping Entities

{brandname} relies on the rich API of link:http://hibernate.org/search/[Hibernate Search] in order to define fine grained configuration for indexing at entity level.
This configuration includes which fields are annotated, which analyzers should be used, how to map nested objects and so on.
Detailed documentation is available at link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#search-mapping[the Hibernate Search manual].

==== @DocumentId
Unlike Hibernate Search, using `@DocumentId` to mark a field as identifier does not apply to {brandname} values; in {brandname} the identifier for all `@Indexed` objects is the key used to store the value. You can still customize how the key is indexed using a combination of `@Transformable` , custom types and custom `FieldBridge` implementations.

==== @Transformable keys
The key for each value needs to be indexed as well, and the key instance must be transformed in a `String`. {brandname} includes some default transformation routines to encode common primitives, but to use a custom key you must provide an implementation of `org.infinispan.query.Transformer` .

[small]*Registering a key Transformer via annotations*

You can annotate your key class with `org.infinispan.query.Transformable` and your custom transformer implementation
will be picked up automatically:

[source,java]
----

@Transformable(transformer = CustomTransformer.class)
public class CustomKey {
   ...
}

public class CustomTransformer implements Transformer {
   @Override
   public Object fromString(String s) {
      ...
      return new CustomKey(...);
   }

   @Override
   public String toString(Object customType) {
      CustomKey ck = (CustomKey) customType;
      return ...
   }
}

----

[small]*Registering a key Transformer via the cache indexing configuration*

Use the `key-transformers` xml element in both embedded and server config:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_key_transformer.xml[]
----

Alternatively, use the Java configuration API (embedded mode):

[source,java]
----

   ConfigurationBuilder builder = ...
   builder.indexing().enable()
         .addKeyTransformer(CustomKey.class, CustomTransformer.class);

----

[[query_configuration_api]]
==== Programmatic mapping

Instead of using annotations to map an entity to the index, it's also possible to configure it programmatically.

In the following example we map an object `Author` which is to be stored in the grid and made searchable on two properties but without annotating the class.

[source,java]
----
import org.apache.lucene.search.Query;
import org.hibernate.search.cfg.Environment;
import org.hibernate.search.cfg.SearchMapping;
import org.hibernate.search.query.dsl.QueryBuilder;
import org.infinispan.Cache;
import org.infinispan.configuration.cache.Configuration;
import org.infinispan.configuration.cache.ConfigurationBuilder;
import org.infinispan.configuration.cache.Index;
import org.infinispan.manager.DefaultCacheManager;
import org.infinispan.query.CacheQuery;
import org.infinispan.query.Search;
import org.infinispan.query.SearchManager;

import java.io.IOException;
import java.lang.annotation.ElementType;
import java.util.Properties;

SearchMapping mapping = new SearchMapping();
mapping.entity(Author.class).indexed()
       .property("name", ElementType.METHOD).field()
       .property("surname", ElementType.METHOD).field();

Properties properties = new Properties();
properties.put(Environment.MODEL_MAPPING, mapping);
properties.put("hibernate.search.[other options]", "[...]");

Configuration infinispanConfiguration = new ConfigurationBuilder()
        .indexing().index(Index.NONE)
        .withProperties(properties)
        .build();

DefaultCacheManager cacheManager = new DefaultCacheManager(infinispanConfiguration);

Cache<Long, Author> cache = cacheManager.getCache();
SearchManager sm = Search.getSearchManager(cache);

Author author = new Author(1, "Manik", "Surtani");
cache.put(author.getId(), author);

QueryBuilder qb = sm.buildQueryBuilderForClass(Author.class).get();
Query q = qb.keyword().onField("name").matching("Manik").createQuery();
CacheQuery cq = sm.getQuery(q, Author.class);
assert cq.getResultSize() == 1;
----


[[query_remote]]
== Remote Search

Remote search is very similar to embedded with the notable difference that data must use
link:http://code.google.com/p/protobuf/[Google Protocol Buffers] as an encoding for both over-the-wire and storage.
Furthermore, it's necessary to write (or generate from Java classes) a protobuf schema defining the data structure and indexing elements instead of relying on Hibernate Search annotations.

The usage of protobuf allows remote query to work not only for Java, but for REST, C# and Node.js clients.

[[remote_query_example]]
=== A remote query example

We are going to revisit the Book Sample from embedded query, but this time using the Java Hot Rod client and the
Infinispan server.
An object called `Book` will be stored in a Infinispan cache called "books". Book instances will be indexed, so we enable
indexing for the cache:

.infinispan.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexed_entities_proto.xml[]
----

Alternatively, if indexing the cache is not indexed, we configure the `<encoding>` as `application/x-protostream` to make sure
the storage is queryable:

.infinispan.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/non_indexed_entities_proto.xml[]
----

Each `Book` will be defined as in the following example: we use `@Protofield` annotations to identify the
protocol buffers message fields and the `@ProtoDoc` annotation on the fields to configure indexing attributes:

[source,java, title=Book.java]
----
include::code_examples/BookIndexed.java[]
----

The annotations above will generate during compilation the artifacts necessary to read, write  and query `Book` instances. To enable this generation, use the `@AutoProtoSchemaBuilder` annotation in a newly created class with empty constructor or interface:

[source,java, title=RemoteQueryInitializer.java]
----
include::code_examples/RemoteQueryInitializer.java[]
----

After compilation, a file `book.proto` file will be created in the configured `schemaFilePath`, along with an implementation
`RemoteQueryInitializerImpl.java` of the annotated interface. This concrete class can be used directly in the Hot Rod client
code to initialize the serialization context.

Putting all together:

[source,java, title=RemoteQuery.java]
----
include::code_examples/RemoteQuery.java[]
----


[[enable_indexing]]
=== Indexing of Protobuf encoded entries

As seen in link:#remote_query_example[Remote Query example], one step necessary to query protobuf entities
is to provide the client and server with the relevant metadata about entities (.proto file).

The descriptors are stored in a dedicated cache on the server named `___protobuf_metadata`.
Both keys and values in this cache are plain strings. Registering a new schema is therefore as simple as performing a `put()` operation on this cache using the schema's name as key and the schema file itself as the value.

Alternatively you can use the CLI (via the `cache-container=*:register-proto-schemas()` operation),
the Management Console, the REST endpoint `/rest/v2/schemas` or the `ProtobufMetadataManager` MBean via JMX.
Be aware that, when security is enabled, access to the schema cache via the remote protocols requires
that the user belongs to the pass:['___schema_manager'] role.

[NOTE]
====
Even if indexing is enabled for a cache no fields of Protobuf encoded entries will be indexed unless you use
the `@Indexed` and `@Field` inside protobuf schema documentation annotations `(@ProtoDoc)`  to specify what fields need to get indexed.
====

[[analysis]]
=== Analysis
Analysis is a process that converts input data into one or more terms that you can index and query.
While in link:#mapping_embedded[Embedded Query] mapping is done through link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#_analysis[Hibernate Search annotations], that supports
a rich set of Lucene based analyzers, in client-server mode the analyzer definitions are declared in a platform neutral way

==== Default Analyzers
{brandname} provides a set of default analyzers for remote query as follows:

[%header,cols=2*]
|===

| Definition
| Description

| `standard`
| Splits text fields into tokens, treating whitespace and punctuation as delimiters.

| `simple`
| Tokenizes input streams by delimiting at non-letters and then converting all letters to lowercase characters. Whitespace and non-letters are discarded.

| `whitespace`
| Splits text streams on whitespace and returns sequences of non-whitespace characters as tokens.

| `keyword`
| Treats entire text fields as single tokens.

| `stemmer`
| Stems English words using the Snowball Porter filter.

| `ngram`
| Generates n-gram tokens that are 3 grams in size by default.

| `filename`
| Splits text fields into larger size tokens than the `standard` analyzer, treating whitespace as a delimiter and converts all letters to lowercase characters.

|===

These analyzer definitions are based on Apache Lucene and are provided "as-is".
For more information about tokenizers, filters, and CharFilters, see the
appropriate Lucene documentation.

==== Using Analyzer Definitions

To use analyzer definitions, reference them by name in the `.proto` schema file.

. Include the `Analyze.YES` attribute to indicate that the property is analyzed.
. Specify the analyzer definition with the `@Analyzer` annotation.

The following example shows referenced analyzer definitions:

[source,protobuf,options="nowrap"]
----
/* @Indexed */
message TestEntity {

    /* @Field(store = Store.YES, analyze = Analyze.YES, analyzer = @Analyzer(definition = "keyword")) */
    optional string id = 1;

    /* @Field(store = Store.YES, analyze = Analyze.YES, analyzer = @Analyzer(definition = "simple")) */
    optional string name = 2;
}
----

If using Java classes annotated with `@ProtoField`, the declaration is similar:

[source,java,options="nowrap"]
----
@ProtoDoc("@Field(store = Store.YES, analyze = Analyze.YES, analyzer = @Analyzer(definition = \"keyword\"))")
@ProtoField(number = 1)
final String id;

@ProtoDoc("@Field(store = Store.YES, analyze = Analyze.YES, analyzer = @Analyzer(definition = \"simple\"))")
@ProtoField(number = 2)
final String description;
----


==== Creating Custom Analyzer Definitions
If you require custom analyzer definitions, do the following:

. Create an implementation of the
`ProgrammaticSearchMappingProvider` interface packaged in a `JAR` file.
. Provide a file named `org.infinispan.query.spi.ProgrammaticSearchMappingProvider` in the
`META-INF/services/` directory of your `JAR`. This file should contain the fully qualified class name of your implementation.
. Copy the `JAR` to the `lib/` directory of your {brandname} installation.
+
[IMPORTANT]
====
Your jar must be available to the {brandname} server during startup. You cannot add it if the server is already running.
====
+
The following is an example implementation of the
`ProgrammaticSearchMappingProvider` interface:
+
[source,java,options="nowrap"]
----
import org.apache.lucene.analysis.core.LowerCaseFilterFactory;
import org.apache.lucene.analysis.core.StopFilterFactory;
import org.apache.lucene.analysis.standard.StandardFilterFactory;
import org.apache.lucene.analysis.standard.StandardTokenizerFactory;
import org.hibernate.search.cfg.SearchMapping;
import org.infinispan.Cache;
import org.infinispan.query.spi.ProgrammaticSearchMappingProvider;

public final class MyAnalyzerProvider implements ProgrammaticSearchMappingProvider {

   @Override
   public void defineMappings(Cache cache, SearchMapping searchMapping) {
      searchMapping
            .analyzerDef("standard-with-stop", StandardTokenizerFactory.class)
               .filter(StandardFilterFactory.class)
               .filter(LowerCaseFilterFactory.class)
               .filter(StopFilterFactory.class);
   }
}
----


[[query_continuous]]
== Continuous Query

Continuous Queries allow an application to register a listener which will receive the entries that currently match a
query filter, and will be continuously notified of any changes to the queried data set that result from further cache
operations. This includes incoming matches, for values that have joined the set, updated matches, for matching values
that were modified and continue to match, and outgoing matches, for values that have left the set. By using a Continuous
Query the application receives a steady stream of events instead of having to repeatedly execute the same query to
discover changes, resulting in a more efficient use of resources. For instance, all of the following use cases could
utilize Continuous Queries:

* Return all persons with an age between 18 and 25 (assuming the Person entity has an `age` property and is updated by
the user application).
* Return all transactions higher than $2000.
* Return all times where the lap speed of F1 racers were less than 1:45.00s (assuming the cache contains Lap entries and
that laps are entered live during the race).

=== Continuous Query Execution

A continuous query uses a listener that is notified when:

* An entry starts matching the specified query, represented by a `Join` event.
* A matching entry is updated and continues to match the query, represented by an `Update` vent.
* An entry stops matching the query, represented by a `Leave` event.

When a client registers a continuous query listener it immediately begins to receive the results currently matching the
query, received as `Join` events as described above. In addition, it will receive subsequent notifications when other
entries begin matching the query, as `Join` events, or stop matching the query, as `Leave` events, as a consequence of
any cache operations that would normally generate creation, modification, removal, or expiration events. Updated cache
entries will generate `Update` events if the entry matches the query filter before and after the operation. To
summarize, the logic used to determine if the listener receives a `Join`, `Update` or `Leave` event is:

. If the query on both the old and new values evaluate false, then the event is suppressed.
. If the query on the old value evaluates false and on the new value evaluates true, then a `Join` event is sent.
. If the query on both the old and new values evaluate true, then an `Update` event is sent.
. If the query on the old value evaluates true and on the new value evaluates false, then a `Leave` event is sent.
. If the query on the old value evaluates true and the entry is removed or expired, then a `Leave` event is sent.

[NOTE]
====
Continuous Queries can use all query capabilities except: grouping, aggregation, and sorting operations.
====

=== Running Continuous Queries

To create a continuous query, do the following:

. Create a Query object. See link:#searching_ickle[the searching section]
. Obtain the ContinuousQuery (`org.infinispan.query.api.continuous.ContinuousQuery` object of your cache by calling
the appropriate method:
 * `org.infinispan.client.hotrod.Search.getContinuousQuery(RemoteCache<K, V> cache)` for remote mode
 * `org.infinispan.query.Search.getContinuousQuery(Cache<K, V> cache)` for embedded mode

. Register the query and a continuous query listener (`org.infinispan.query.api.continuous.ContinuousQueryListener`) as follows:

[source,java]
----
continuousQuery.addContinuousQueryListener(query, listener);
----

The following example demonstrates a simple continuous query use case in embedded mode:
⁠
[source,java,title="Registering a Continuous Query"]
----
import org.infinispan.query.api.continuous.ContinuousQuery;
import org.infinispan.query.api.continuous.ContinuousQueryListener;
import org.infinispan.query.Search;
import org.infinispan.query.dsl.QueryFactory;
import org.infinispan.query.dsl.Query;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

[...]

// We have a cache of Persons
Cache<Integer, Person> cache = ...

// We begin by creating a ContinuousQuery instance on the cache
ContinuousQuery<Integer, Person> continuousQuery = Search.getContinuousQuery(cache);

// Define our query. In this case we will be looking for any Person instances under 21 years of age.
QueryFactory queryFactory = Search.getQueryFactory(cache);
Query query = queryFactory.create("FROM Person p WHERE p.age < 21");

final Map<Integer, Person> matches = new ConcurrentHashMap<Integer, Person>();

// Define the ContinuousQueryListener
ContinuousQueryListener<Integer, Person> listener = new ContinuousQueryListener<Integer, Person>() {
    @Override
    public void resultJoining(Integer key, Person value) {
        matches.put(key, value);
    }

    @Override
    public void resultUpdated(Integer key, Person value) {
        // we do not process this event
    }

    @Override
    public void resultLeaving(Integer key) {
        matches.remove(key);
    }
};

// Add the listener and the query
continuousQuery.addContinuousQueryListener(query, listener);

[...]

// Remove the listener to stop receiving notifications
continuousQuery.removeContinuousQueryListener(listener);
----

As Person instances having an age less than 21 are added to the cache they will be received by the listener and will be
placed into the `matches` map, and when these entries are removed from the cache or their age is modified to be greater
or equal than 21 they will be removed from `matches`.

=== Removing Continuous Queries
To stop the query from further execution just remove the listener:

[source,java]
----
continuousQuery.removeContinuousQueryListener(listener);
----

=== Notes on performance of Continuous Queries

Continuous queries are designed to provide a constant stream of updates to the application, potentially resulting in a
very large number of events being generated for particularly broad queries. A new temporary memory allocation is made
for each event. This behavior may result in memory pressure, potentially leading to `OutOfMemoryErrors` (especially in
remote mode) if queries are not carefully designed. To prevent such issues it is strongly recommended to ensure that
each query captures the minimal information needed both in terms of number of matched entries and size of each match
(projections can be used to capture the interesting properties), and that each `ContinuousQueryListener` is designed
to quickly process all received events without blocking and to avoid performing actions that will lead to the generation
of new matching events from the cache it listens to.

== Statistics

Query link:http://docs.jboss.org/hibernate/search/5.7/api/org/hibernate/search/stat/Statistics.html[Statistics]
can be obtained from the `SearchManager`, as demonstrated in the following code snippet.

[source,java]
----
SearchManager searchManager = Search.getSearchManager(cache);
org.hibernate.search.stat.Statistics statistics = searchManager.getStatistics();
----

TIP: This data is also available via JMX through the link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#_statisticsinfombean[Hibernate Search StatisticsInfoMBean]
registered under the name `org.infinispan:type=Query,manager="{name-of-cache-manager}",cache="{name-of-cache}",component=Statistics`.
Please note this MBean is always registered by {brandname} but the statistics are collected only if statistics collection is enabled at cache level.

WARNING: Hibernate Search has its own configuration properties `hibernate.search.jmx_enabled` and `hibernate.search.generate_statistics`
for JMX statistics as explained link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#search-monitoring[here].
Using them with {brandname} Query is forbidden as it will only lead to duplicated MBeans and unpredictable results.

[[query_performance]]
== Performance Tuning

=== Batch writing in SYNC mode

By default, the link:#query_index_manager[Index Managers] work in sync mode, meaning when data is written to {brandname}, it will perform the indexing operations synchronously.
This synchronicity guarantees indexes are always consistent with the data (and thus visible in searches), but can slowdown write operations since it will also perform a commit to the index.
Committing is an extremely expensive operation in Lucene, and for that reason, multiple writes from different nodes can be automatically batched into a single commit to reduce
the impact.

So, when doing data loads to {brandname} with index enabled, try to use multiple threads to take advantage of this batching.

If using multiple threads does not result in the required performance, an alternative is to load data with indexing temporarily disabled and run
 a link:#query_massindexer[re-indexing] operation afterwards. This can be done writing data with the `SKIP_INDEXING` flag:

[source,java]
----
cache.getAdvancedCache().withFlags(Flag.SKIP_INDEXING).put("key","value");
----

=== Writing using async mode

If it's acceptable a small delay between data writes and when that data is visible in queries, an index manager can be configured to work in *async mode*.
The async mode offers much better writing performance, since in this mode commits happen at a configurable interval.

Configuration:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_async.xml[]
----

=== Index reader async strategy

Lucene internally works with snapshots of the index: once an `IndexReader` is opened, it will only see the index changes up to the point it was opened;
further index changes will not be visible until the `IndexReader` is refreshed. The Index Managers used in {brandname} by default will check the
freshness of the index readers before every query and refresh them if necessary.

It is possible to tune this strategy to relax this freshness checking to a pre-configured interval by using the `reader.strategy` configuration set as `async`:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/indexing_async_reader_strategy.xml[]
----

=== Lucene Options

It is possible to apply tuning options in Lucene directly. For more details, see the link:++https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#_lucene_configuration++[Hibernate Search manual].
