[[persistence]]
== Persistence
Persistence allows configuring external (persistent) storage engines complementary to the default in memory storage offered by {brandname}.
An external persistent storage might be useful for several reasons:

* Increased Durability. Memory is volatile, so a cache store could increase the life-span of the information store in the cache.

* Write-through. Interpose {brandname} as a caching layer between an application and a (custom) external storage engine.

* Overflow Data. By using eviction and passivation, one can store only the "hot" data in memory and overflow the data that is less frequently used to disk.

The integration with the persistent store is done through the following SPI: CacheLoader, CacheWriter, AdvancedCacheLoader and AdvancedCacheWriter (discussed in the following sections).

These SPIs allow for the following features:

* Alignment with link:http://jcp.org/en/jsr/detail?id=107[JSR-107]. The link:{javadocroot}/org/infinispan/persistence/spi/CacheWriter.html[CacheWriter] and link:{javadocroot}/org/infinispan/persistence/spi/CacheLoader.html[CacheLoader] interface are similar to the the loader and writer in JSR 107. This should considerably help writing portable stores across JCache compliant vendors.

* Simplified Transaction Integration. All necessary locking is handled by {brandname} automatically and implementations don’t have to be concerned with coordinating concurrent access to the store. Even though concurrent writes on the same key are not going to happen (depending locking mode in use), implementors should expect operations on the store to happen from multiple/different threads and code the implementation accordingly.

* Parallel Iteration. It is now possible to iterate over entries in the store with multiple threads in parallel.

* Reduced Serialization. This translates in less CPU usage. The new API exposes the stored entries in serialized format. If an entry is fetched from persistent storage for the sole purpose of being sent remotely, we no longer need to deserialize it (when reading from the store) and serialize it back (when writing to the wire). Now we can write to the wire the serialized format as read from the storage directly.


=== Configuration
Stores (readers and/or writers) can be configured in a chain. Cache read operation looks at all of the specified `CacheLoader` s, in the order they are configured, until it finds a valid and non-null element of data. When performing writes all cache `CacheWriter` s are written to, except if the `ignoreModifications` element has been set to true for a specific cache writer.

.Implementing both a CacheWriter and CacheLoader

[IMPORTANT]
====
Store providers should implement both the `CacheWriter` and the `CacheLoader` interfaces. Stores that do this are considered both for reading and writing (assuming `read-only=false`) data.
====
//Exclude the following configuration from productized docs
ifndef::productized[]
[source,xml]
----
This is the configuration of a custom(not shipped with infinispan) store:
   <local-cache name="myCustomStore">
      <persistence passivation="false">
         <store
            class="org.acme.CustomStore"
            fetch-state="false" preload="true" shared="false"
            purge="true" read-only="false" singleton="false" segmented="true">

            <write-behind modification-queue-size="123" thread-pool-size="23" />

            <property name="myProp">${system.property}</property>
         </store>
      </persistence>
   </local-cache>
----
endif::productized[]
//Include the following configuration in productized docs
ifdef::productized[]
[source,xml]
----
This is the configuration of a custom(not shipped with infinispan) store:
   <local-cache name="myCustomStore">
      <persistence passivation="false">
         <store
            class="org.acme.CustomStore"
            fetch-state="false" preload="true" shared="false"
            purge="true" read-only="false" singleton="false">

            <write-behind modification-queue-size="123" thread-pool-size="23" />

            <property name="myProp">${system.property}</property>
         </store>
      </persistence>
   </local-cache>
----
endif::productized[]

Parameters that you can use to configure persistence are as follows:

`connection-attempts`::
Sets the maximum number of attempts to start each configured
CacheWriter/CacheLoader. If the attempts to start are not successful, an
exception is thrown and the cache does not start.

`connection-interval`::
Specifies the time, in milliseconds, to wait between connection attempts on
startup. A negative or zero value means no wait between connection attempts.

`availability-interval`::
Specifies the time, in milliseconds, between availability checks to determine
if the PersistenceManager is available. In other words, this interval sets how
often stores/loaders are polled via their
`org.infinispan.persistence.spi.CacheWriter#isAvailable` or
`org.infinispan.persistence.spi.CacheLoader#isAvailable` implementation. If a
single store/loader is not available, an exception is thrown during cache
operations.

`passivation`::
Enables passivation. The default value is `false` (boolean).
+
This property has a significant impact on {brandname} interactions with the loaders. See link:#cache_passivation[Cache Passivation] for more information.

`class`::
Defines the class of the store and must implement `CacheLoader`, `CacheWriter`,
or both.

`fetch-state`::
Fetches the persistent state of a cache when joining a cluster. The default
value is `false` (boolean).
+
The purpose of this property is to retrieve the persistent state of a cache and
apply it to the local cache store of a node when it joins a cluster. Fetching
the persistent state does not apply if a cache store is shared because it
accesses the same data as the other stores.
+
This property can be `true` for one configured cache loader only. If more than
one cache loader fetches the persistent state, a configuration exception is
thrown when the cache service starts.

`preload`::
Pre-loads data into memory from the cache loader when the cache starts. The
default value is `false` (boolean).
+
This property is useful when data in the cache loader is required immediately
after startup to prevent delays with cache operations when the data is loaded
lazily. This property can provide a "warm cache" on startup but it impacts
performance because it affects start time.
+
Pre-loading data is done locally, so any data loaded is stored locally in the node only. The pre-loaded data is not replicated or distributed. Likewise, {brandname} pre-loads data only up to the maximum configured number of entries in link:#eviction_anchor[eviction].

`shared`::
Determines if the cache loader is shared between cache instances. The default value is `false` (boolean).
+
This property prevents duplicate writes of data to the cache loader by
different cache instances. An example is where all cache instances in a cluster
use the same JDBC settings for the same remote, shared database.

//Exclude segmented cache stores from productized docs.
ifndef::productized[]
`segmented`::
Configures a cache store to segment data. The default value is `false`
(boolean).
+
If `true` the cache store stores data in buckets. The `hash.numSegments` property configures how many buckets there are for storing data.
+
Depending on the cache store implementation, segmenting data can cause slower
write operations. However, performance improves for other cache operations. See
link:#segmented_stores[Segmented Stores] for more information.
endif::productized[]

`read-only`::
Prevents new data from being persisted to the cache store. The default value is
`false` (boolean).

`purge`::
Empties the specified cache loader at startup. The default value is `false`
(boolean). This property takes effect only if the `read-only` property is set
to `false`.

`max-batch-size`::
ifndef::productized[]
Sets the maximum size of a batch to insert of delete from the cache store. The
default value is #{AbstractStore-maxBatchSize}.
endif::productized[]
ifdef::productized[]
Sets the maximum size of a batch to insert of delete from the cache store. The
default value is 100.
endif::productized[]
+
If the value is less than `1`, no upper limit applies to the number of
operations in a batch.

`write-behind`::
Asynchronously persists data to the cache store. The default value is `false`
(boolean). See link:#write_behind_asynchronous[Asynchronous Write-Behind] for
more information.

`singleton`::
Enables one node in the cluster, the coordinator, to store modifications. The
default value is `false` (boolean).
+
Whenever data enters a node, it is replicated or distributed to keep the
in-memory state of the caches synchronized. The coordinator is responsible for
pushing that state to disk.
+
If `true` you must set this property on all nodes in the cluster. Only the
coordinator of the cluster persists data. However, all nodes must have this
property enabled to prevent other nodes from persisting data.
+
You cannot enable the `singleton` property if the cache store is shared.

[NOTE]
====
You can define additional attributes in the `properties` section to configure
specific aspects of each cache loader, such as the `myProp` attribute in the previous example.

Other cache loaders with more complex configurations also include additional
properties. See the following JDBC cache store configuration for examples.
====

The preceding configuration applies a generic cache store implementation.
However, the default {brandname} store implementation has a more complex
configuration schema, in which the `properties` section is replaced with XML
attributes:

[source,xml]
----
<persistence passivation="false">
   <!-- note that class is missing and is induced by the fileStore element name -->
   <file-store
           shared="false" preload="true"
           fetch-state="true"
           read-only="false"
           purge="false"
           path="${java.io.tmpdir}">
      <write-behind thread-pool-size="5" />
   </file-store>
</persistence>

----

The same configuration can be achieved programmatically:

[source,java]
----
   ConfigurationBuilder builder = new ConfigurationBuilder();
   builder.persistence()
         .passivation(false)
         .addSingleFileStore()
            .preload(true)
            .shared(false)
            .fetchPersistentState(true)
            .ignoreModifications(false)
            .purgeOnStartup(false)
            .location(System.getProperty("java.io.tmpdir"))
            .async()
               .enabled(true)
               .threadPoolSize(5)
            .singleton()
               .enabled(true)
               .pushStateWhenCoordinator(true)
               .pushStateTimeout(20000);
----

[[cache_passivation]]
=== Cache Passivation
A CacheWriter can be used to enforce entry passivation and activation on eviction in a cache. Cache passivation is the
process of removing an object from in-memory cache and writing it to a secondary data store (e.g., file system, database)
on eviction. Cache activation is the process of restoring an object from the data store into the in-memory cache when
it's needed to be used. In order to fully support passivation, a store needs to be both a CacheWriter and a CacheLoader.
In both cases, the configured cache store is used to read from the loader and write to the data writer.

When an eviction policy in effect evicts an entry from the cache, if passivation is enabled, a notification that the
entry is being passivated will be emitted to the cache listeners and the entry will be stored. When a user attempts to
retrieve a entry that was evicted earlier, the entry is (lazily) loaded from the cache loader into memory. When the
entry has been loaded a notification is emitted to the cache listeners that the entry has been activated. In order to
enable passivation just set passivation to true (false by default). When passivation is used, only the first cache loader
configured is used and all others are ignored.

==== Limitations

Due to the unique nature of passivation, it is not supported with some other store configurations.

* Transactional store - Passivation writes/removes entries from the store
outside the scope of the actual Infinispan commit boundaries.
* Shared store - Shared store requires entries always being in the store for
other owners. Thus passivation makes no sense as we can't remove the entry from
the store.

==== Cache Loader Behavior with Passivation Disabled vs Enabled
When passivation is disabled, whenever an element is modified, added or removed, then that modification is persisted in
the backend store via the cache loader. There is no direct relationship between eviction and cache loading.
If you don't use eviction, what's in the persistent store is basically a copy of what's in memory. If you do use eviction,
what's in the persistent store is basically a superset of what's in memory (i.e. it includes entries that have been
evicted from memory). When passivation is enabled, and with an unshared store, there is a direct relationship between
eviction and the cache loader.
Writes to the persistent store via the cache loader only occur as part of the eviction process. Data is deleted from the
persistent store when the application reads it back into memory. In this case, what's in memory and what's in the
persistent store are two subsets of the total information set, with no intersection between the subsets.
With a shared store, entries which have been passivated in the past will continue to exist in the store, although they
may have a stale value if this has been overwritten in memory.

The following is a simple example, showing what state is in RAM and in the persistent store after each step of a 6 step process:

[options="header"]
|===============
|Operation|Passivation Off|Passivation On, Shared Off|Passivation On, Shared On
|Insert keyOne|*Memory:* keyOne +
*Disk:* keyOne|*Memory:* keyOne +
*Disk:* (none)|*Memory:* keyOne +
*Disk:* (none)
|Insert keyTwo|*Memory:* keyOne, keyTwo +
*Disk:* keyOne, keyTwo|*Memory:* keyOne, keyTwo +
*Disk:* (none)|*Memory:* keyOne, keyTwo +
*Disk:* (none)
|Eviction thread runs, evicts keyOne|*Memory:* keyTwo +
*Disk:* keyOne, keyTwo|*Memory:* keyTwo +
*Disk:* keyOne|*Memory:* keyTwo +
*Disk:* keyOne
|Read keyOne|*Memory:* keyOne, keyTwo +
*Disk:* keyOne, keyTwo|*Memory:* keyOne, keyTwo +
*Disk:* (none)|*Memory:* keyOne, keyTwo +
*Disk:* keyOne
|Eviction thread runs, evicts keyTwo|*Memory:* keyOne +
*Disk:* keyOne, keyTwo|*Memory:* keyOne +
*Disk:* keyTwo|*Memory:* keyOne +
*Disk:* keyOne, keyTwo
|Remove keyTwo|*Memory:* keyOne +
*Disk:* keyOne|*Memory:* keyOne +
*Disk:* (none)|*Memory:* keyOne +
*Disk:* keyOne
|===============

=== Cache Loaders and transactional caches
When a cache is transactional and a cache loader is present, the cache loader won't be enlisted in the transaction in which the cache is part.
That means that it is possible to have inconsistencies at cache loader level: the transaction to succeed applying the in-memory state but (partially) fail applying the changes to the store.
Manual recovery would not work with caches stores.

===  Write-Through And Write-Behind Caching
{brandname} can optionally be configured with one or several cache stores allowing it to store data in a persistent location such as shared JDBC database, a local filesystem, etc. {brandname} can handle updates to the cache store in two different ways:


* Write-Through (Synchronous)
* Write-Behind (Asynchronous)

==== Write-Through (Synchronous)
In this mode, which is supported in version 4.0, when clients update a cache entry, i.e. via a Cache.put() invocation, the call will not return until {brandname} has gone to the underlying cache store and has updated it. Normally, this means that updates to the cache store are done within the boundaries of the client thread.

The main advantage of this mode is that the cache store is updated at the same time as the cache, hence the cache store is consistent with the cache contents. On the other hand, using this mode reduces performance because the latency of having to access and update the cache store directly impacts the duration of the cache operation.

Configuring a write-through or synchronous cache store does not require any particular configuration option. By default, unless marked explicitly as write-behind or asynchronous, all cache stores are write-through or synchronous. Please find below a sample configuration file of a write-through unshared local file cache store:

[source,xml]
----
<persistence passivation="false">
   <file-store fetch-state="true"
               read-only="false"
               purge="false" path="${java.io.tmpdir}"/>
</persistence>

----

[[write_behind_asynchronous]]
==== Write-Behind (Asynchronous)
In this mode, updates to the cache are asynchronously written to the cache
store. {brandname} puts pending changes into a modification queue so that it can quickly store changes.

The configured number of threads consume the queue and apply the modifications
to the underlying cache store. If the configured number of threads cannot consume the modifications fast enough, or if the underlying store becomes unavailable, the modification queue becomes full. In this event, the cache store becomes write-through until the queue can accept new entries.

This mode provides an advantage in that cache operations are not affected by
updates to the underlying store. However, because updates happen
asynchronously, there is a period of time during which data in the cache store
is inconsistent with data in the cache.

The write-behind strategy is suitable for cache stores with low latency and
small operational cost; for example, an unshared file-based cache store that is
local to the cache itself. In this case, the time during which data is
inconsistent between the cache store and the cache is reduced to the lowest
possible period.

The following is an example configuration for the write-behind strategy:

[source,xml]
----
<persistence passivation="false">
   <file-store fetch-state="true"
               read-only="false"
               purge="false" path="${java.io.tmpdir}">
   <!-- start write-behind configuration -->
   <write-behind modification-queue-size="123" thread-pool-size="23" />
   <!-- end write-behind configuration -->
   </file-store>
</persistence>
----

//Exclude segmented cache stores from productized docs.
ifndef::productized[]
[[segmented_stores]]
==== Segmented Stores
You can configure stores so that data resides in segments to which keys map.
See link:#key_ownership[Key Ownership] for more information about segments and
ownership.

Segmented stores increase read performance for bulk operations; for example,
streaming over data (`Cache.size`, `Cache.entrySet.stream`), pre-loading the
cache, and doing state transfer operations.

However, segmented stores can also result in loss of performance for write
operations. This performance loss applies particularly to batch write
operations that can take place with transactions or write-behind stores. For
this reason, you should evaluate the overhead for write operations before you
enable segmented stores. The performance gain for bulk read operations might
not be acceptable if there is a significant performance loss for write
operations.

[IMPORTANT]
====
Loss of data can occur if the number of segments in a cache store are not
changed gracefully. For this reason, if you change the `numSegments` setting in the store configuration, you must migrate the existing store to use the new configuration.

The recommended method to migrate the cache store configuration is to perform a
rolling upgrade. The store migrator supports migrating a non-segmented cache
store to a segmented cache store only. The store migrator does not currently
support migrating from a segmented cache store.
====

[NOTE]
====
Not all cache stores support segmentation. See the appropriate section for each
store to determine if it supports segmentation.

If you plan to convert or write a new store to support segmentation, see the
following SPI section that provides more details.
====
endif::productized[]

=== Filesystem based cache stores

A filesystem-based cache store is typically used when you want to have a
cache with a cache store available locally which stores data that has
overflowed from memory, having exceeded size and/or time restrictions.

WARNING: Usage of filesystem-based cache stores on shared filesystems like NFS,
Windows shares, etc. should be avoided as these do not implement proper
file locking and can cause data corruption. File systems are inherently
not transactional, so when attempting to use your cache in a transactional
context, failures when writing to the file (which happens during the commit
phase) cannot be recovered.

include::persistence_sfs.adoc[]
include::persistence_sifs.adoc[]
include::persistence_jdbc.adoc[]
include::persistence_remote.adoc[]
include::persistence_cluster.adoc[]
include::persistence_cli.adoc[]
include::persistence_rocksdb.adoc[]
include::persistence_leveldb.adoc[]
include::persistence_jpa.adoc[]
include::persistence_custom.adoc[]
include::persistence_storemigrator.adoc[]

Following sections describe the SPI and also discuss the SPI implementations that {brandname} ships out of the box.

=== SPI

The following class diagram presents the main SPI interfaces of the persistence API:

image::images/Figure2_1_persistence_API.png[align="center", title="Persistence SPI"]

Some notes about the classes:

* link:{javadocroot}/org/infinispan/commons/io/ByteBuffer.html[ByteBuffer] - abstracts the serialized form of an object

* link:{javadocroot}/org/infinispan/persistence/spi/MarshalledEntry.html[MarshalledEntry] - abstracts the information held within
  a persistent store corresponding to a key-value added to the cache. Provides method for reading this information both in serialized (link:{javadocroot}/org/infinispan/commons/io/ByteBuffer.html[ByteBuffer]) and deserialized (Object) format. Normally data read from the store is kept in serialized format and lazily deserialized on demand, within the
  link:{javadocroot}/org/infinispan/persistence/spi/MarshalledEntry.html[MarshalledEntry] implementation

* link:{javadocroot}/org/infinispan/persistence/spi/CacheWriter.html[CacheWriter] and link:{javadocroot}/org/infinispan/persistence/spi/CacheLoader.html[CacheLoader] provide basic methods for reading and writing to a store

* link:{javadocroot}/org/infinispan/persistence/spi/AdvancedCacheLoader.html[AdvancedCacheLoader] and link:{javadocroot}/org/infinispan/persistence/spi/AdvancedCacheWriter.html[AdvancedCacheWriter] provide operations to manipulate the underlaying storage in bulk: parallel iteration and purging of expired entries, clear and size.

//Exclude segmented cache stores from productized docs.
ifndef::productized[]
* link:{javadocroot}/org/infinispan/persistence/spi/SegmentedAdvancedLoadWriteStore.html[SegmentedAdvancedLoadWriteStore] provide all the various operations that deal with segments.

A cache store can be segmented if it does one of the following:

* Implements the
link:{javadocroot}/org/infinispan/persistence/spi/SegmentedAdvancedLoadWriteStore.html[SegmentedAdvancedLoadWriteStore] interface. In this case only a single
store instance is used per cache.

* Has a configuration that extends the link:{javadocroot}/org/infinispan/configuration/cache/AbstractSegmentedConfiguration.html[AbstractSegmentedConfiguration] abstract class. Doing this requires you to implement the `newConfigurationFrom` method where it is expected
that a new `StoreConfiguration` instance is created per invocation. This
creates a store instance per segment to which a node can write. Stores might
start and stop as data is moved between nodes.
endif::productized[]

A provider might choose to only implement a subset of these interfaces:

* Not implementing the  link:{javadocroot}/org/infinispan/persistence/spi/AdvancedCacheWriter.html[AdvancedCacheWriter] makes the given writer not usable for purging expired entries or clear

* If a loader does not implement the link:{javadocroot}/org/infinispan/persistence/spi/AdvancedCacheLoader.html[AdvancedCacheLoader]
inteface, then it will not participate in preloading nor in cache iteration
(required also for stream operations).

If you're looking at migrating your existing store to the new API or to write a new store implementation, the link:https://github.com/infinispan/infinispan/blob/master/core/src/main/java/org/infinispan/persistence/file/SingleFileStore.java[SingleFileStore] might be a good starting point/example.

=== More implementations
Many more cache loader and cache store implementations exist.
Visit link:http://infinispan.org/cache-store-implementations[this website] for more details.
