==  Management Tooling
Management of Infinispan instances is all about exposing as much relevant statistical information that allows administrators to get a view of the state of each Infinispan instance. Taking in account that a single installation could be made up of several tens or hundreds Infinispan instances, providing clear and concise information in an efficient manner is imperative. The following sections dive into the range of management tooling that Infinispan provides.

=== JMX
Over the years, link:http://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html[JMX] has become the de facto standard for management and administration of middleware and as a result, the Infinispan team has decided to standardize on this technology for the exposure of management and statistical information.

==== Understanding The Exposed MBeans
By connecting to the VM(s) where Infinispan is running with a standard JMX GUI such as link:http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html[JConsole] or link:http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/[VisualVM] you should find the following MBeans:

*  For CacheManager level JMX statistics, without further configuration, you should see an MBean called _org.infinispan:type=CacheManager,name="DefaultCacheManager"_ with link:http://docs.jboss.org/infinispan/{infinispanversion}/apidocs/jmxComponents.html#CacheManager[properties specified by the CacheManager MBean] .

*  Using the cacheManagerName attribute in globalJmxStatistics XML element, or using the corresponding GlobalJmxStatisticsConfigurationBuilder.cacheManagerName(String cacheManagerName) call, you can name the cache manager in such way that the name is used as part of the JMX object name. So, if the name had been "Hibernate2LC", the JMX name for the cache manager would have been: _org.infinispan:type=CacheManager,name="Hibernate2LC"_ . This offers a nice and clean way to manage environments where multiple cache managers are deployed, which follows link:http://www.oracle.com/technetwork/java/javase/tech/best-practices-jsp-136021.html[JMX best practices] .

*  For Cache level JMX statistics, you should see several different MBeans depending on which configuration options have been enabled. For example, if you have configured a write behind cache store, you should see an MBean exposing properties belonging to the cache store component. All Cache level MBeans follow the same format though which is the following: `org.infinispan:type=Cache,name="${name-of-cache}(${cache-mode})",manager="${name-of-cache-manager}",component=${component-name}` where:

*  ${name-of-cache} has been substituted by the actual cache name. If this cache represents the default cache, its name will be `___defaultCache`.

*  ${cache-mode} has been substituted by the cache mode of the cache. The cache mode is represented by the lower case version of the possible enumeration values shown link:http://docs.jboss.org/infinispan/{infinispanversion}/apidocs/org/infinispan/configuration/cache/CacheMode[here].

*  ${name-of-cache-manager} has been substituted by the name of the cache manager to which this cache belongs. The name is derived from the _cacheManagerName_ attribute value in `globalJmxStatistics` element.

*  ${component-name} has been substituted by one of the JMX component names in the link:http://docs.jboss.org/infinispan/{infinispanversion}/apidocs/jmxComponents.html[JMX reference documentation] .

For example, the cache store JMX component MBean for a default cache configured with synchronous distribution would have the following name: `org.infinispan:type=Cache,name="___defaultcache(dist_sync)",manager="DefaultCacheManager",component=CacheStore`

Please note that cache and cache manager names are quoted to protect against illegal characters being used in these user-defined names.

==== Enabling JMX Statistics
The MBeans mentioned in the previous section are always created and registered in the MBeanServer allowing you to manage
your caches but some of their attributes do not expose meaningful values unless you take the extra step of enabling
collection of statistics. Gathering and reporting statistics via JMX can be enabled at 2 different levels:

.CacheManager level
The CacheManager is the entity that governs all the cache instances that have been created from it.
Enabling CacheManager statistics collections differs depending on the configuration style:


* If configuring the CacheManager via XML, make sure you add the following XML under the `<cache-container />` element:

 <cache-container statistics="true"/>

* If configuring the CacheManager programmatically, simply add the following code:

 GlobalConfigurationBuilder globalConfigurationBuilder = ...
 globalConfigurationBuilder.globalJmxStatistics().enable();

.Cache level
At this level, you will receive management information generated by individual cache instances.
Enabling Cache statistics collections differs depending on the configuration style:

* If configuring the Cache via XML, make sure you add the following XML under the one of the top level cache elements, such as `<local-cache />`:

 <local-cache statistics="true"/>

* If configuring the Cache programmatically, simply add the following code:

 ConfigurationBuilder configurationBuilder = ...
 configurationBuilder.jmxStatistics().enable();

==== Monitoring cluster health

It is also possible to monitor Infinispan cluster health using JMX. On CacheManager there's an additional object called `CacheContainerHealth`. It contains the following attributes:

* cacheHealth - a list of caches and corresponding statuses (HEALTHY, UNHEALTHY or REBALANCING)
* clusterHealth - overall cluster health
* clusterName - cluster name
* freeMemoryKb - Free memory obtained from JVM runtime measured in KB
* numberOfCpus - The number of CPUs obtained from JVM runtime
* numberOfNodes - The number of nodes in the cluster
* totalMemoryKb - Total memory obtained from JVM runtime measured in KB

==== Multiple JMX Domains
There can be situations where several CacheManager instances are created in a single VM, or Cache names belonging to different CacheManagers under the same VM clash.

Using different JMX domains for multi cache manager environments should be last resort. 
Instead, it's possible to name a cache manager in such way that it can easily be identified and used by monitoring tools such as RHQ. For example:


* Via XML:

[source,xml]
----
<cache-container statistics="true" name="Hibernate2LC"/>
----


* Programmatically:

[source,java]
----
GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
    .enable()
    .cacheManagerName("Hibernate2LC");

----

Using either of these options should result on the CacheManager MBean name being: `org.infinispan:type=CacheManager,name="Hibernate2LC"`

For the time being, you can still set your own jmxDomain if you need to and we also allow duplicate domains, or rather duplicate JMX names, but these should be limited to very special cases where different cache managers within the same JVM are named equally.

==== Registering MBeans In Non-Default MBean Servers
Let's discuss where Infinispan registers all these MBeans. By default, Infinispan registers them in the link:http://docs.oracle.com/javase/6/docs/api/java/lang/management/ManagementFactory.html#getPlatformMBeanServer()[standard JVM MBeanServer platform] . However, users might want to register these MBeans in a different MBeanServer instance. For example, an application server might work with a different MBeanServer instance to the default platform one. In such cases, users should implement the link:http://docs.jboss.org/infinispan/{infinispanversion}/apidocs/org/infinispan/jmx/MBeanServerLookup.html[MBeanServerLookup interface] provided by Infinispan so that the link:http://docs.jboss.org/infinispan/{infinispanversion}/apidocs/org/infinispan/jmx/MBeanServerLookup.html#getMBeanServer()[getMBeanServer() method] returns the MBeanServer under which Infinispan should register the management MBeans. You can find an example in the default link:http://anonsvn.jboss.org/repos/infinispan/tags/4.0.0.FINAL/core/src/main/java/org/infinispan/jmx/PlatformMBeanServerLookup.java[PlatformMBeanServerLookup class] used by Infinispan. So, once you have your implementation ready, simply configure Infinispan with the fully qualified name of this class. For example:


* Via XML:

[source,xml]
----
<cache-container statistics="true">
   <jmx mbean-server-lookup="com.acme.MyMBeanServerLookup" />
</cache-container>
----


* Programmatically:

[source,java]
----
GlobalConfigurationBuilder globalConfigurationBuilder = ...
globalConfigurationBuilder.globalJmxStatistics()
    .enable()
    .mBeanServerLookup(new com.acme.MyMBeanServerLookup());

----

==== MBeans added in Infinispan 5.0
There has been a couple of noticeable additions in Infinispan 5.0 in terms of exposed MBeans:

.  MBeans related to Infinispan servers are now available that for the moment focus on the transport layer. A new MBean named `org.infinispan:type=Server,name={Memcached|HotRod},component=Transport` offers information such as: host name, port, bytes read, byte written, number of worker threads, etc.

.  When global JMX statistics are enabled, the JGroups channel MBean is also registered automatically under the name `org.infinispan:type=channel,cluster={name-of-your-cluster}`, so you can get key information of the group communication transport layer that's used to cluster Infinispan instances. To find out more about the information provided, check the link:http://community.jboss.org/docs/10938[JGroups JMX documentation].

=== RHQ
The preferred way to manage multiple Infinispan instances spread across different servers is to use RHQ, which is JBoss' enterprise management solution. Thanks to RHQ's agent and auto discovery capabilities, monitoring both Cache Manager and Cache instances is a very simple task. With RHQ, administrators have access to graphical views of key runtime parameters or statistics and can also be notified be these exceed or go below certain limits. The Infinispan specific statistics shown by RHQ are a reflection of the JMX information exposed by Infinispan which has been formatted for consumption by RHQ. Please follow these steps to get started with RHQ and have Infinispan instances monitored with it:

. Firstly, download and install an RHQ server and install and start at least one RHQ agent. The job of the RHQ agent is to send information about the Infinispan instance back to the server which is the one that shows the information via a nice GUI. You can find detailed information on the installation process in link:https://docs.jboss.org/author/display/RHQ/Installation[RHQ's installation guide] and you can find information on how to run an agent in the link:https://docs.jboss.org/author/display/RHQ/Running+the+RHQ+Agent[Running the RHQ Agent] .
+
.Careful with H2 database installation
WARNING: If you're just building a demo or testing RHQ server, you can avoid the need to install a fully fledged database and use an in-memory H2 database instead.
However, you might encounter issues after testing database connection as shown link:https://lists.fedorahosted.org/pipermail/rhq-users/2010-June/000045.html[here].
Simply repeating the installation avoiding testing the connection should work. 
+
.Where do I install the RHQ agent?
TIP: The most common set up is to have the RHQ agent installed in the same machine where Infinispan is running.
If you have multiple machines, an agent can be installed in each machine.

. By now, you should have an RHQ server and agent running. It's time now to download the latest Infinispan binary distribution (*-bin.zip or *-all.zip should do) from the link:http://infinispan.org/download/[downloads] section and locate the RHQ plugin jar file which should be named something like infinispan-rhq-plugin.jar. This is located under the modules/rhq-plugin directory.

. The link:https://docs.jboss.org/author/display/RHQ/Agent+Plugin+Administration[adding and updating plugins section] on the RHQ guide contains some more detailed information on how to update both RHQ servers and agents with new plugins, but essentially, this process involves uploading a new plugin to the RHQ server and then pushing the plugin to one, or several, RHQ agents.
+
.Speeding up plugin installation
TIP: If you're simply demoing or testing and you only have a single agent, once the plugin has been uploaded to the server, simply go to the agent command line interface and type: plugins update .This will force the agent to retrieve the latest plugins from the server. Doing this can be considerably faster than some of the other alternatives.

. At this point, RHQ is ready to start monitoring Infinispan instances, but before firing them up, make sure you start them with the following system properties so that RHQ agents can discover them:
+
 -Dcom.sun.management.jmxremote.port=6996 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
+
.Remote JMX port value
TIP: The actual port value used does not really matter here, but what matters is that a port is given, otherwise Infinispan instances cannot be located. So, you can easily start multiple Infinispan instances in a single machine, each with a different remote JMX port, and a locally running agent will be able to discover them all without any problems.

. Once Infinispan instances have been discovered, you should see a new resource for each of the cache manager running appearing in the link:https://docs.jboss.org/author/display/RHQ/Initial+Auto-discovery+and+Import[Inventory/Discovery Queue] of the RHQ server. Simply import it now and you should see each cache manager appearing with as many child cache resources as caches are running in each cache manager. You're now ready to monitor Infinispan!

==== RHQ monitoring tips
This section focuses on the lessons learned while developing the Infinispan RHQ plugin that are likely to be useful to anyone using RHQ.

* By default, at least in version 2.3.1 of RHQ, the RHQ agent sends an availability report of any managed resources every 5 minutes. The problem with this is that if you're testing whether your Infinispan instance is automatically discovered by the RHQ server, it can take up to 5 minutes to do so! Also, it can take 5 minutes for the RHQ server to figure out that you've shutdown your Infinispan instance. You can change this setting by the following property (default value is 300 seconds) in `rhq-agent/conf/agent-configuration.xml`. For example, if you wanted the availability to be sent every 1 minute, simply change the value to 60:


[source,xml]
----
<entry key="rhq.agent.plugins.availability-scan.period-secs" value="60"/>
----

.Careful with agent configuration changes
WARNING: Please bear in mind the instructions given in the link:https://docs.jboss.org/author/display/RHQ/RHQ+Agent+Installation[RHQ agent installation] and more specifically the paragraph below with regards to changes made to properties in agent-configuration.xml:

[quote]
____
Once the agent is configured, it persists its configuration in the Java Preferences backing store. Once this happens, agent-configuration.xml is no longer needed or used. Editing agent-configuration.xml will no longer have any effect on the agent, even if you restart the agent. If you want the agent to pick up changes you make to agent-configuration.xml, you must either restart the agent with the "--cleanconfig" command line option or use the "config --import" agent prompt command.
____

=== Hawt.io
link:http://hawt.io[Hawt.io], a slick, fast, HTML5-based open source management console, also has support for Infinispan.
Refer to link:http://hawt.io/plugins/infinispan/[Hawt.io's documentation] for information regarding this plugin.

=== Writing plugins for other management tools
As mentioned in the previous section, RHQ consumes the JMX data exposed by Infinispan, and in similar fashion, plugins could be written for other 3rd party management tools that were able to transform these data into the correct representation in these tools, for example graphs, etc.

