[[integrations]]
= Integrations
{brandname} can be integrated with a number of other projects, as detailed below.

== Apache Spark

{brandname} provides an link:http://spark.apache.org[Apache Spark] connector capable of exposing caches as an RDD, allowing batch and stream jobs to be run against data stored in {brandname}. For further details, see the link:https://github.com/infinispan/infinispan-spark/blob/master/README.md[{brandname} Spark connector documentation].
Also check the link:https://github.com/infinispan/infinispan-spark/tree/master/examples/twitter/README.md[Docker based Twitter demo].

== Apache Hadoop

The {brandname} Hadoop connector can be used to expose {brandname} as a Hadoop compliant data source and sink that implements link:https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/InputFormat.html[InputFormat]/link:https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/OutputFormat.html[OutputFormat].
For further details, refer to the full link:https://github.com/infinispan/infinispan-hadoop/blob/master/README.md[documentation].

[[integrations_lucene_directory]]
== Apache Lucene
{brandname} includes a highly scalable distributed link:http://lucene.apache.org[Apache Lucene Directory] implementation.

This directory closely mimics the same semantics of the traditional filesystem and RAM-based directories, being able to work as a drop-in replacement for existing applications using Lucene and providing reliable index sharing and other features of {brandname} like node auto-discovery, automatic failover and rebalancing, optionally transactions, and can be backed by traditional storage solutions as filesystem, databases or cloud store engines.

The implementation extends Lucene's _org.apache.lucene.store.Directory_ so it can be used to _store_ the index in a cluster-wide shared memory, making it easy to distribute the index. Compared to rsync-based replication this solution is suited for use cases in which your application makes frequent changes to the index and you need them to be quickly distributed to all nodes. Consistency levels, synchronicity and guarantees, total elasticity and auto-discovery are all configurable; also changes applied to the index can optionally participate in a JTA transaction, optionally supporting XA transactions with recovery.

Two different _LockFactory_ implementations are provided to guarantee only one _IndexWriter_ at a time will make changes to the index, again implementing the same semantics as when opening an index on a local filesystem. As with other Lucene Directories, you can override the _LockFactory_ if you prefer to use an alternative implementation.

=== Lucene compatibility
Apache Lucene versions 5.5.x

=== Maven dependencies
All you need is the following:

.pom.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::dependencies_maven/lucene_directory.xml[]
----

=== How to use it

See the below example of using the {brandname} Lucene Directory in order to index and query a single Document:

[source,java]
----
include::code_examples/LuceneDirectoryExampleQuery.java[]
----

The _indexName_ in the _DirectoryBuilder_ is a unique key to identify your index. It takes the same role as the path did on filesystem based indexes: you can create several different indexes giving them different names. When you use the same _indexName_ in another instance connected to the same network (or instantiated on the same machine, useful for testing) they will join, form a cluster and share all content. Using a different _indexName_ allows you to store different indexes in the same set of Caches.

The _metadataCache_, _dataCache_ and _lockCache_ are the caches that will store the indexes. More details provided below.

New nodes can be added or removed dynamically, making the service administration very easy and also suited for cloud environments: it's simple to react to load spikes, as adding more memory and CPU power to the search system is done by just starting more nodes.

=== Configuration
{brandname} can be configured as LOCAL clustering mode, in which case it will disable clustering features and serve as a cache for the index, or any clustering mode. A transaction manager is not mandatory, but when enabled the changes to the index can participate in transactions.

Batching was required in previous versions, it's not strictly needed anymore.

As pointed out in the javadocs of link:{javadocroot}/org/infinispan/lucene/directory/DirectoryBuilder.html[DirectoryBuilder], it's possible for it to use more than a single cache, using specific configurations for different purposes. Each cache is explained below:

==== Lock Cache
The lock cache is used to store a single entry per index that will function as the directory lock. Given the small storage requirement this cache is usually configured as REPL_SYNC. Example of declarative configuration:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/lucene_indexes_locking.xml[]
----

==== Metadata Cache

The metadata cache is used to store information about the files of the directory, such as buffer sizes and number of chunks. It uses more space than the Lock Cache, but not as much as the Data Cache, so using a REPL_SYNC cache should be fine for most cases.
Example of configuration:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/lucene_indexes_metadata.xml[]
----

==== Data Cache

The {brandname} Lucene directory splits large (bigger than the chunkSize configuration) files into chunks and stores them in the Data cache.
This is the largest of the 3 index caches, and both DIST_SYNC/REPL_SYNC cache modes can be used.
Usage of REPL_SYNC offers lower latencies for queries since each node holds the whole index locally; DIST_SYNC, on the other hand, will affect query latency due to remote calls to fetch for chunks, but offers better scalability.

Example of configuration:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/lucene_indexes_data.xml[]
----

=== Using a CacheLoader
Using a CacheLoader you can have the index content backed up to a permanent storage; you can use a shared store for all nodes or one per node, see cache passivation for more details.

When using a CacheLoader to store a Lucene index, to get best write performance you would need to configure the CacheLoader with _async=true_ .

=== Storing the index in a database
It might be useful to store the Lucene index in a relational database; this would be very slow but {brandname} can act as a cache between the application and the JDBC interface, making this configuration useful in both clustered and non-clustered configurations. When storing indexes in a JDBC database, it's suggested to use the _JdbcStringBasedCacheStore_ , which will need the `key-to-string-mapper` attribute to be set to `org.infinispan.lucene.LuceneKey2StringMapper`:

[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/lucene_key_string_mapper.xml[]
----

=== Loading an existing Lucene Index

The _org.infinispan.lucene.cachestore.LuceneCacheLoader_ is an {brandname} CacheLoader able to have {brandname} directly load data from an existing Lucene index into the grid. Currently this supports reading only.

[options="header"]
|===============
|Property|Description|Default
| _location_ |The path where the indexes are stored. Subdirectories (of first level only) should contain the indexes to be loaded, each directory matching the index name attribute of the {brandname} Directory constructor.|none (mandatory)
| _autoChunkSize_ |A threshold in bytes: if any segment is larger than this, it will be transparently chunked in smaller cache entries up to this size.|32MB

|===============

It's worth noting that the IO operations are delegated to Lucene's standard _org.apache.lucene.store.FSDirectory_ , which will select an optimal approach for the running platform.

Implementing write-through should not be hard: you're welcome to try implementing it.

=== Architectural limitations
This Directory implementation makes it possible to have almost real-time reads across multiple nodes. A fundamental limitation of the Lucene design is that only a single IndexWriter is allowed to make changes on the index: a pessimistic lock is acquired by the writer; this is generally ok as a single IndexWriter _instance_ is very fast and accepts update requests from multiple threads. When sharing the Directory across {brandname} nodes the IndexWriter limitation is not lifted: since you can have only one instance, that reflects in your application as having to apply all changes on the same node. There are several strategies to write from multiple nodes on the same index:

.Index write strategies
* One node writes, the other delegate to it sending messages
* Each node writes on turns
* You application makes sure it will only ever apply index writes on one node

The _{brandname} Lucene Directory_ protects its content by implementing a distributed locking strategy, though this is designed as a last line of defense and is not to be considered an efficient mechanism to coordinate multiple writes: if you don't apply one of the above suggestions and get high write contention from multiple nodes you will likely get timeout exception.

=== Suggestions for optimal performance

==== JGroups and networking stack
JGroups manages all network IO and as such it is a critical component to tune for your specific environment. Make sure to read the link:http://jgroups.org/manual-3.x/html/index.html[JGroups reference documentation], and play with the performance tests included in JGroups to make sure your network stack is setup appropriately. Don't forget to check also operating system level parameters, for example buffer sizes dedicated for networking. JGroups will log warning when it detects something wrong, but there is much more you can look into.

==== Using a CacheStore
Currently all CacheStore implementations provided by {brandname} have a significant slowdown; we hope to resolve that soon but for the time being if you need high performance on writes with the Lucene Directory the best option is to disable any CacheStore; the second best option is to configure the CacheStore as _async_ . If you only need to load a Lucene index from read-only storage, see the above description for _org.infinispan.lucene.cachestore.LuceneCacheLoader_ .

==== Apply standard Lucene tuning
All known options of Lucene apply to the {brandname} Lucene Directory as well; of course the effect might be less significant in some cases, but you should definitely read the link:http://lucene.apache.org/core/index.html[Apache Lucene documentation] .

==== Disable batching and transactions
Early versions required {brandname} to have batching or transactions enabled. This is no longer a requirement, and in fact disabling them should provide little improvement in performance.

==== Set the right chunk size
The chunk size can be specified using the link:{javadocroot}/org/infinispan/lucene/directory/DirectoryBuilder.html[DirectoryBuilder] fluent API. To correctly set this variable you need to estimate what the expected size of your segments is; generally this is trivial by looking at the file size of the index segments generated by your application when it's using the standard FSDirectory. You then have to consider:

* The chunk size affects the size of internally created buffers, and large chunk sizes will cause memory usage to grow. Also consider that during index writing such arrays are frequently allocated.
* If a segment doesn't fit in the chunk size, it's going to be fragmented. When searching on a fragmented segment performance can't peak.

Using the _org.apache.lucene.index.IndexWriterConfig_ you can tune your index writing to _approximately_ keep your segment size to a reasonable level, from there then tune the chunksize, after having defined the chunksize you might want to revisit your network configuration settings.

=== Demo

There is a simple command-line demo of its capabilities distributed with {brandname} under demos/lucene-directory; make sure you grab the "Binaries, server and demos" package from download page, which contains all demos.

Start several instances, then try adding text in one instance and searching for it on the other. The configuration is not tuned at all, but should work out-of-the box without any changes. If your network interface has multicast enabled, it will cluster across the local network with other instances of the demo.

=== Additional Links
* Issue tracker: link:https://jira.jboss.org/browse/ISPN/component/12312732[]
* Source code: link:https://github.com/infinispan/infinispan/tree/master/lucene/lucene-directory/src/main/java/org/infinispan/lucene[]

[[integrations_directory_provider]]
=== Directory Provider for Hibernate Search
Hibernate Search applications can use {brandname} as a directory provider, taking advantage of {brandname}'s distribution and low latency capabilities to store the Lucene indexes.

=== Maven dependencies

.pom.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::dependencies_maven/hibernate_directory_provider.xml[]
----

=== How to use it

The directory provider alias is _"infinispan"_, and to enable it for an index, the following property should be in the link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/#configuration[Hibernate Search configuration]:

----
hibernate.search.MyIndex.directory_provider = infinispan
----

to enable it by default for all indexes:

----
hibernate.search.default.directory_provider = infinispan
----

The {brandname} cluster will start with a link:https://github.com/infinispan/infinispan/blob/master/lucene/directory-provider/src/main/resources/default-hibernatesearch-infinispan.xml[default configuration], see below how to override it.

=== Configuration

Optional properties allow for a custom {brandname} configuration or to use an existent _EmbeddedCacheManager_:

[options="header"]
|===============
|Property|Description|Example value
|`hibernate.search.infinispan.configuration_resourcename`| Custom configuration for {brandname} | config/infinispan.xml
|`hibernate.search.infinispan.configuration.transport_override_resourcename`| Overrides the JGroups stack in the {brandname} configuration file | jgroups-ec2.xml
|`hibernate.search.infinispan.cachemanager_jndiname`| Specifies the JNDI name under which the _EmbeddedCacheManager_ to use is bound. Will cause the properties above to be ignored when present| `java:jboss/infinispan/container/hibernate-search`
|===============

=== Architecture considerations

The same limitations presented in the Lucene Directory apply here, meaning the index will be shared across several nodes and only one _IndexWriter_ can have the lock.

One common strategy is to use Hibernate Search's JMS Master/Slave or JGroups backend together with the {brandname} directory provider: instead of sending updates directly to the index, they are sent to a JMS queue or JGroups channel and a single node applies all the changes on behalf of all other nodes.

Refer to the link:https://docs.jboss.org/hibernate/stable/search/reference/en-US/html_single/[Hibernate Search documentation] for instructions on how to setup JMS or JGroups backends.

include::integrations_hibernate_cache.adoc[]

[[integrations_hibernate_ogm]]
==  JPA / Hibernate OGM

Hibernate can perform CRUD operations directly on an {brandname} cluster.

Hibernate OGM is an extension of the popular Hibernate ORM project which makes
the Hibernate API suited to interact with NoSQL databases such as {brandname}.

When some of your object graphs need high scalability and elasticity, you can
use Hibernate OGM to store these specific entities into {brandname} instead of
your traditional RDBMS.
The drawback is that {brandname} - not being a relational database - can not run
complex relational queries.

Hibernate OGM allows you to get started with {brandname} in minutes, as:

 - the JPA API and its annotations are simple and well known
 - you don't need to learn Protobuf or Externalizer encoding formats
 - no need to learn the {brandname} API
 - the Hot Rod client is also setup and managed for you

It will still be beneficial to eventually learn how to configure {brandname} for
top performance and learn about all capabilities it has, but you can get a proof
of concept application done quickly with the example configuration.

Hibernate OGM also gives you several more benefits; being designed and
implemented in collaboration with the {brandname} team it incorporates experience
and deep understanding of how to best perform some common operations.

For example a common mistake for people new to {brandname} is to "serialize"
Java POJOs for long term storage of important information; the {brandname} API
allows this as it's useful for short lived caching of metadata, but you wouldn't
be able to de-serialize your data when you make any changes to your model.
You wouldn't want to wipe your database after any and each update of your
application, would you?

In the best of cases such an encoding wouldn't be very efficient; in some worse
scenarios your team might not have thought such details though and you get stuck
into a complex migration on your live data.

Just like when using Hibernate ORM with a relational database, data stored over
Hibernate OGM is easy to recover even using other tools as it's encoded using a
well defined Protobuf schema.

Being able to "map" new domain objects by simply adding a couple of annotations
is going to make you more productive than re-inventing such error-prone encoding
techniques, or figuring out how to best store object graphs and relations
into {brandname}.

Finally, using Hibernate OGM allows you to use all existing framework
integration points, such as injecting an `EntityManager` as usual: it's not
yet another tool but it's the real Hibernate, so inheriting all well known
integrations: this will work in Java EE, Spring, Grails, Jhipster, ... and all
other technologies integrating with Hibernate.

It's booted like any Hibernate instance: compared to using it with an RDBMS
you just have to change some configuration properties, and of course omit the
`DataSource` as {brandname} won't use one.

For more details, check the link:http://hibernate.org/ogm/[Hibernate OGM project]
and the link:https://docs.jboss.org/hibernate/stable/ogm/reference/en-US/html_single/#ogm-infinispan[Hibernate OGM / {brandname}]
section of the documentation.

include::integrations_spring.adoc[]

ifndef::productized[]
= {brandname} modules for WildFly / EAP

As the {brandname} modules shipped with link:http://wildfly.org/[WildFly] / link:https://www.redhat.com/en/technologies/jboss-middleware/application-platform[EAP] are tailored to its internal usage, it is recommend to install separate modules
if you want to use {brandname} in your application that is deployed to WildFly / EAP. By installing these modules, it is possible to deploy user applications without packaging the {brandname} JARs within the deployments (WARs, EARs, etc), thus minimizing their size.
Also, there will be no conflict with WildFly / EAP's internal modules since the slot will be different.
endif::productized[]

ifdef::productized[]
= {brandname} modules for EAP

As the {brandname} modules shipped with link:https://www.redhat.com/en/technologies/jboss-middleware/application-platform[EAP] are tailored to its internal usage, it is recommend to install separate modules if you want to use {brandname} in your application that is deployed to EAP. By installing these modules, it is possible to deploy user applications without packaging the {brandname} JARs within the deployments (WARs, EARs, etc), thus minimizing their size.
Also, there will be no conflict with EAP's internal modules since the slot will be different.
endif::productized[]

[[modules_installation_section]]
== Installation
ifndef::productized[]
The modules for WildFly / EAP are available in the link:http://infinispan.org/download/[downloads] section of our site.
After extracting the zip, copy the contents of the `modules` directory to the `WILDFLY_HOME/modules` directory, so that
for example the {brandname} core module would be under `WILDFLY_HOME/modules/system/add-ons/{moduleprefix}/org/infinispan/core`.
endif::productized[]

ifdef::productized[]
The modules for EAP are available from the link:https://access.redhat.com/downloads/[Red Hat Customer Portal].

After extracting the zip, copy the contents of the `modules` directory to the `${EAP_HOME}/modules` directory. For example, the {brandname} core module should be under `${EAP_HOME}/modules/system/add-ons/{moduleprefix}/org/infinispan/core`.
endif::productized[]

== Application Dependencies

If you are using Maven to build your application, mark the {brandname} dependencies as _provided_ and configure your artifact archiver to generate the appropriate MANIFEST.MF file:

.pom.xml
[source,xml,options="nowrap",subs=attributes+]
----
include::dependencies_maven/eap_dependencies.xml[]
----

The next section illustrates the manifest entries for different types of {brandname}'s dependencies.

=== {brandname} core

In order expose only {brandname} core dependencies to your application, add the follow to the manifest:

.MANIFEST.MF
[source,subs=attributes]
----
Manifest-Version: 1.0
Dependencies: org.infinispan:{infinispanslot} services
----

=== Remote

If you need to connect to remote {brandname} servers via Hot Rod, including execution of remote queries, use the module `org.infinispan.remote` that exposes the needed dependencies conveniently:

.MANIFEST.MF
[source,subs=attributes]
----
Manifest-Version: 1.0
Dependencies: org.infinispan.remote:{infinispanslot} services
----

=== Embedded Query

For embedded querying, including the {brandname} Query DSL, Lucene and Hibernate Search Queries, add the following:

.MANIFEST.MF
[source,subs=attributes]
----
Manifest-Version: 1.0
Dependencies: org.infinispan:{infinispanslot} services, org.infinispan.query:{infinispanslot} services
----

=== Lucene Directory

Lucene users who wants to simple use {brandname} as a _org.apache.lucene.store.Directory_ don't need to add the query module, the entry below is sufficient:

.MANIFEST.MF
[source,subs=attributes]
----
Manifest-Version: 1.0
Dependencies: org.infinispan.lucene-directory:{infinispanslot}
----

=== Hibernate Search directory provider for {brandname}

The Hibernate Search directory provider for {brandname} is also contained within the {brandname} modules zip. It is not necessary to add an entry to the manifest file since the Hibernate Search module already has an optional dependency to it.
When choosing the {brandname} module zip to use, start by checking which Hibernate Search is in use, more details below.

==== Usage with {wildflybrandname} Internal Hibernate Search Modules

The Hibernate Search module present in {wildflybrandname}
ifndef::productized[]
10.x
endif::productized[]
ifdef::productized[]
7.2
endif::productized[]
has slot "5.5", which in turn has an optional dependency to `org.infinispan.hibernate-search.directory-provider:for-hibernatesearch-5.5`.
This dependency will be available once the {brandname} modules are link:#modules_installation_section[installed].

==== Usage with other Hibernate Search modules

The module `org.hibernate.search:{infinispanslot}` distributed with {brandname} is to be used together with {brandname} Query only (querying data from caches), and should not be used by Hibernate ORM applications.
To use a Hibernate Search with a different version that is present in {wildflybrandname}, please consult the link:https://docs.jboss.org/hibernate/search/5.6/reference/en-US/html_single/#search-configuration-deploy-on-wildfly[Hibernate Search documentation].

Make sure that the chosen Hibernate Search optional slot for `org.infinispan.hibernate-search.directory-provider` matches the one distributed with {brandname}.

=== Usage
There are two possible ways for your application to utilize {brandname} within {wildflybrandname}, embedded mode and server mode.

=== Embedded Mode
All CacheManagers and cache instances are created in your application logic.  The lifecycle of your EmbeddedCacheManager
is tightly coupled with your application's lifecycle, resulting in any manager instances created by your application being destroyed with your application.

==== Server Mode
In server mode, it is possible for cache containers and caches to be created before runtime as part of {wildflybrandname}'s
standalone/domain.xml configuration. This allows cache instances to be shared across multiple applications, with the
lifecycle of the underlying cache container being independent of the deployed application.

===== Configuration
To enable server mode, make the following additions to your {wildflybrandname} configuration in `standalone/domain.xml`:

. Add the {brandname} extensions to your `<extensions>` section.
+
ifndef::productized[]
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/wfly_extensions_community.xml[]
----
endif::productized[]
ifdef::productized[]
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/wfly_extensions_product.xml[]
----
endif::productized[]
+
. Configure the {brandname} subsystem and your required containers and caches in the server profile that requires {brandname}.
+
[IMPORTANT]
====
Be sure that you define the `module` attribute to load the correct {brandname} classes.
====
+
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/wfly_subsystem.xml[]
----
+
ifndef::productized[]
. Define the {wildflybrandname} https://docs.jboss.org/author/display/WFLY10/Interfaces+and+ports[socket-bindings] required by the endpoint and/or JGroup subsystems
endif::productized[]
ifdef::productized[]
. Define the required {wildflybrandname} socket-bindings for JGroup subsystems.
endif::productized[]
+
ifndef::productized[]
. Configure any endpoints that you require via the endpoint subsystem:
+
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/wfly_endpoints.xml[]
----
endif::productized[]
+
. Define the JGroups transport, ensuring that you define the `model` attribute for each protocol.
+
[NOTE]
====
You do not neet to define a JGroups transport for local cache configurations. The JGroups transport is required for clustered cache configurations only.
====
+
[source,xml,options="nowrap",subs=attributes+]
----
include::config_examples/wfly_jgroups.xml[]
----

==== Accessing Containers and Caches
Once a container has been defined in your server's configuration, it is possible to inject an instance of a CacheContainer
or Cache into your application using the `@Resource` JNDI lookup. A container is accessed using the following string
`java:jboss/datagrid-infinispan/container/<container_name>` and similarly a cache is
accessed via `java:jboss/datagrid-infinispan/container/<container_name>/cache/<cache_name>`.

The example below shows how to inject the CacheContainer
called "infinispan_container" and the distributed cache "namedCache" into an application.

[source,java]
----
include::code_examples/ExampleApplicationInjectCache.java[]
----

== Troubleshooting

=== Enable logging

Enabling trace on `org.jboss.modules` can be useful to debug issues like `LinkageError` and `ClassNotFoundException`.
To enable it at runtime using the {wildflybrandname} CLI:

----
bin/jboss-cli.sh -c '/subsystem=logging/logger=org.jboss.modules:add'
bin/jboss-cli.sh -c '/subsystem=logging/logger=org.jboss.modules:write-attribute(name=level,value=TRACE)'
----

=== Print dependency tree

The following command can be used to print all dependencies for a certain module. For example, to obtain the tree for the module `org.infinispan:{infinispanslot}`, execute from `WILDFLY_HOME`:

[source,bash,subs=attributes+]
----
$ java -jar jboss-modules.jar -deptree -mp modules/ "org.infinispan:{infinispanslot}"
----
