  <chapter id="ch-advanced-resources"><title>Advanced Resource Types</title>
    <section id="group-resources">
      <title>Groups - A Syntactic Shortcut</title>
      <para>
	One of the most common elements of a cluster is a set of resources that need to be located together, start sequentially and stop in the reverse order.
	To simplify this configuration we support the concept of groups.
      </para>
      <example>
	<title>An example group</title>
	<programlisting>
<![CDATA[
  <group id="shortcut">
   <primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat">
    <instance_attributes id="params-public-ip">
       <nvpair id="public-ip-addr" name="ip" value="1.2.3.4"/>
    </instance_attributes>
   </primitive>
   <primitive id="Email" class="lsb" type="exim"/>
  </group>
]]>
	</programlisting>
      </example>
      <para>
	Although the example above contains only two resources, there is no limit to the number of resources a group can contain.
	The example is also sufficient to explain the fundamental properties of a group:
      </para>
      <itemizedlist>
	<listitem><para>Resources are started in the order they appear in (Public-IP first, then Email)</para></listitem>
	<listitem><para>Resources are stopped in the reverse order to which they appear in (Email first, then Public-IP)</para></listitem>
	<listitem>
	  <para>If a resource in the group can't run anywhere, then nothing after that is allowed to run</para>
	  <itemizedlist>
	    <listitem><para>If Public-IP can’t run anywhere, neither can Email</para></listitem>
	    <listitem><para>If Email can’t run anywhere, this does not affect Public-IP in any way</para></listitem>
	  </itemizedlist>
	</listitem>
      </itemizedlist>
      <para>The group above is logically equivalent to writing:</para>
      <example>
	<title>How the cluster sees a group resource</title>
	<programlisting>
<![CDATA[
  <configuration>
   <resources>
    <primitive id="Public-IP" class="ocf" type="IPaddr" provider="heartbeat">
     <instance_attributes id="params-public-ip">
        <nvpair id="public-ip-addr" name="ip" value="1.2.3.4"/>
     </instance_attributes>
    </primitive>
    <primitive id="Email" class="lsb" type="exim"/>
   </resources>
   <constraints>
      <rsc_colocation id="xxx" rsc="Email" with-rsc="Public-IP" score="INFINITY"/>
      <rsc_order id="yyy" first="Public-IP" then="Email"/>
   </constraints>
  </configuration>
]]>
	</programlisting>
      </example>
      <para>Obviously as the group grows bigger, the reduced configuration effort can become significant.</para>
      <section>
	<title>Properties</title>
	<table frame="all">
	  <title>Properties of a Group Resource</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead>
	    <tbody>
	      <row>
		<entry>id</entry>
		<entry>Your name for the group</entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Options</title>
	<para>Options inherited from <link linkend="s-resource-options">simple resources</link>: <literal>priority, target-role, is-managed</literal></para>
      </section>
      <section>
	<title>Using Groups</title>
	<section>
	  <title>Instance Attributes</title>
	  <para>Groups have no instance attributes, however any that are set here will be inherited by the group's children.</para>
	</section>
	<section>
	  <title>Contents</title>
	  <para>
	    Groups may only contain a collection of <link linkend="s-resource-supported">primitive</link> cluster resources.
	    To refer to the child of a group resource, just use the child's id instead of the group's.
	  </para>
	</section>
	<section>
	  <title>Constraints</title>
	  <para>Although it is possible to reference the group's children in constraints, it is usually preferable to use the group's name instead.</para>
	  <example>
	    <title>Example constraints involving groups</title>
	    <programlisting>
<![CDATA[
  <constraints>
    <rsc_location id="group-prefers-node1" rsc="shortcut" node="node1" score="500"/>
    <rsc_colocation id="webserver-with-group" rsc="Webserver" with-rsc="shortcut"/>
    <rsc_order id="start-group-then-webserver" first="Webserver" then="shortcut"/>
  </constraints>
]]>
	    </programlisting>
	  </example>
	</section>
	<section>
	  <title>Stickiness</title>
	  <para>Stickiness, the measure of how much a resource wants to stay where it is, is additive in groups.
Every active member of the group will contribute its stickiness value to the group's total.
So if the default <literal>resource-stickiness</literal> is 100 a group has seven members, five of which are active, then the group as a whole will prefer its current location with a score of 500.</para>
	</section>
      </section>
    </section>
    <section id="s-resource-clone">
      <title>Clones - Resources That Should be Active on Multiple Hosts</title>
      <para>
	Clones were initially conceived as a convenient way to start N instances of an IP resource and have them distributed throughout the cluster for load balancing.
	They have turned out to quite useful for a number of purposes including integrating with Red Hat's DLM, the fencing subsystem and OCFS2.
      </para>
      <para>You can clone any resource provided the resource agent supports it.</para>
      <para>Three types of cloned resources exist.</para>
      <itemizedlist>
	<listitem><para>Anonymous</para></listitem>
	<listitem><para>Globally Unique</para></listitem>
	<listitem><para>Stateful</para></listitem>
      </itemizedlist>
      <para>
	Anonymous clones are the simplest type.
	These resources behave completely identically everywhere they are running.
	Because of this, there can only be one copy of an anonymous clone active per machine.
      </para>
      <para>
	Globally unique clones are distinct entities.
	A copy of the clone running on one machine is not equivalent to another instance on another node.
	Nor would any two copies on the same node be equivalent.
      </para>
      <para>Stateful clones are covered later in <xref linkend="s-resource-multistate"/>.</para>
      <example>
	<title>An example clone</title>
	<programlisting>
<![CDATA[
  <clone id="apache-clone">
    <meta_attributes id="apache-clone-meta">
       <nvpair id="apache-unique" name="globally-unique" value="false"/>
    </meta_attributes>
    <primitive id="apache" class="lsb" type="apache"/>
  </clone>
]]>
	</programlisting>
      </example>
      <section>
	<title>Properties</title>
	<table frame="all">
	  <title>Properties of a Clone Resource</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead><tbody><row>
		<entry>id</entry>
		<entry>Your name for the clone</entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Options</title>
	<para>Options inherited from <link linkend="s-resource-supported">simple resources</link>: <literal>priority, target-role, is-managed</literal></para>
	<table frame="all">
	  <title>Clone specific configuration options</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead>
	    <tbody>
	      <row>
		<entry>clone-max</entry>
		<entry>How many copies of the resource to start. Defaults to the number of nodes in the cluster.</entry>
              </row>
              <row>
		<entry>clone-node-max</entry>
		<entry>How many copies of the resource can be started on a single node. Defaults to 1.</entry>
              </row>
              <row>
		<entry>notify</entry>
		<entry>When stopping or starting a copy of the clone, tell all the other copies beforehand and when the action was successful. Allowed values: true, <emphasis>false</emphasis>
		  </entry>
              </row>
              <row>
		<entry>globally-unique</entry>
		<entry>Does each copy of the clone perform a different function? Allowed values: <emphasis>true</emphasis>, false</entry>
              </row>
              <row>
		<entry>ordered</entry>
		<entry>Should the copies be started in series (instead of in parallel). Allowed values: true, <emphasis>false</emphasis>
		  </entry>
              </row>
              <row>
		<entry>interleave</entry>
		<entry>Changes the behavior of ordering constraints (between clones/masters) so that instances can start/stop as soon as their peer instance has (rather than waiting for every instance of the other clone has). Allowed values: true, <emphasis>false</emphasis>
		  </entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Using Clones</title>
	<section>
	  <title>Instance Attributes</title>
	  <para>Clones have no instance attributes, however any that are set here will be inherited by the clone's children.</para>
	</section>
	<section>
	  <title>Contents</title>
	  <para>Clones must contain exactly one group or one regular resource.</para>
	  <warning>
	    <para>
	      You should never reference the name of a clone's child.
	      If you think you need to do this, you probably need to re-evaluate your design.
	    </para>
	  </warning>
	</section>
	<section>
	  <title>Constraints</title>
	  <para>
	    In most cases, a clone will have a single copy on each active cluster node.
	    However if this is not the case, you can indicate which nodes the cluster should to preferentially assign copies to with resource location constraints.
	    These constraints are written no differently to those for regular resources except that the clone's id is used.
	  </para>
	  <para>
	    Ordering constraints behave slightly differently for clones.
	    In the example below, apache-stats will wait until all copies of the clone that need to be started have done so before being started itself.
	    Only if no copies can be started will apache-stats be prevented from being active.
	    Additionally, the clone will wait for apache-stats to be stopped before stopping the clone.
	  </para>
	  <para>
	    Colocation of a regular (or group) resource with a clone means that the resource can run on any machine with an active copy of the clone.
	    The cluster will choose a copy based on where the clone is running and the rsc resource's own location preferences.
	  </para>
	  <para>
	    Colocation between clones is also possible.
	    In such cases, the set of allowed locations for the rsc clone is limited to nodes on which the with clone is (or will be) active.
	    Allocation is then performed as-per-normal.
	  </para>
	  <example>
	    <title>Example constraints involving clones</title>
	    <programlisting>
<![CDATA[
  <constraints>
    <rsc_location id="clone-prefers-node1" rsc="apache-clone" node="node1" score="500"/>
    <rsc_colocation id="stats-with-clone" rsc="apache-stats" with="apache-clone"/>
    <rsc_order id="start-clone-then-stats" first="apache-clone" then="apache-stats"/>
  </constraints>
]]>
	    </programlisting>
	  </example>
	</section>
	<section>
	  <title>Stickiness</title>
	  <para>
	    To achieve a stable allocation pattern, clones are slightly sticky by default.
	    If no value for <literal>resource-stickiness</literal> is provided, the clone will use a value of 1.
	    Being a small value, it causes minimal disturbance to the score calculations of other resources but is enough to prevent Pacemaker from needlessly moving copies around the cluster.
	  </para>
	</section>
	<section>
	  <title>Resource Agent Requirements</title>
	  <para>
	    Any resource can be used as an anonymous clone as it requires no additional support from the resource agent.
	    Whether it makes sense to do so depends on your resource and its resource agent.
	  </para>
	  <para>
	    Globally unique clones do require some additional support in the resource agent.
	    In particular, it must only respond with <envar>${OCF_SUCCESS}</envar> if the node has that exact instance active.
	    All other probes for instances of the clone should result in <envar>${OCF_NOT_RUNNING}</envar>.
	    Unless of course they are failed, in which case they should return one of the other OCF error codes.
	  </para>
	  <para>Copies of a clone are identified by appending a colon and a numerical offset. Eg. apache:2</para>
	  <para>Resource agents can find out how many copies there are by examining the <envar>OCF_RESKEY_CRM_meta_clone_max</envar> environment variable and which copy it is by examining <envar>OCF_RESKEY_CRM_meta_clone</envar>.</para>
	  <para>
	    You should not make any assumptions (based on <envar>OCF_RESKEY_CRM_meta_clone</envar>) about which copies are active.
	    In particular, the list of active copies will not always be an unbroken sequence, nor always start at 0.
	  </para>
	</section>
	<section>
	  <title>Notifications</title>
	  <para>
	    Supporting notifications requires the <literal>notify</literal> action to be implemented.
	    Once supported, the notify action will be passed a number of extra variables which, when combined with additional context, can be used to calculate the current state of the cluster and what is about to happen to it.
	  </para>
	  <table frame="all">
	    <title>Environment variables supplied with Clone notify actions</title>
	    <tgroup cols="2">
              <thead>
		<row>
		  <entry>Variable</entry>
		  <entry>Description</entry>
		</row>
              </thead>
	      <tbody>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_type</entry>
		  <entry>Allowed values: pre, post</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_operation</entry>
		  <entry>Allowed values: start, stop</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_start_resource</entry>
		  <entry>Resources to be started</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_stop_resource</entry>
		  <entry>Resources to be stopped</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_active_resource</entry>
		  <entry>Resources the that are running</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_inactive_resource</entry>
		  <entry>Resources the that are not running</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_start_uname</entry>
		  <entry>Nodes on which resources will be started</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_stop_uname</entry>
		  <entry>Nodes on which resources will be stopped</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_active_uname</entry>
		  <entry>Nodes on which resources are running</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_inactive_uname</entry>
		  <entry>Nodes on which resources are not running</entry>
		</row>
	      </tbody>
	    </tgroup>
	  </table>
	  <para>The variables come in pairs, such as <envar>OCF_RESKEY_CRM_meta_notify_start_resource</envar> and <envar>OCF_RESKEY_CRM_meta_notify_start_uname</envar> and should be treated as an array of whitespace separated elements.</para>
	  <para>Thus in order to indicate that clone:0 will be started on sles-1, clone:2 will be started on sles-3, and clone:3 will be started on sles-2, the cluster would set</para>
	  <para><envar>OCF_RESKEY_CRM_meta_notify_start_resource</envar>="clone:0 clone:2 clone:3"</para>
	  <para><envar>OCF_RESKEY_CRM_meta_notify_start_uname</envar>="sles-1 sles-3 sles-2"</para>
	  <para>Example notification variables</para>
	</section>
	<section>
	  <title>Proper Interpretation of Notification Environment Variables</title>
	  <para>Pre-notification (stop)</para>
	  <itemizedlist>
	    <listitem>
              <para>Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	    </listitem>
	    <listitem>
              <para>Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	    </listitem>
	    <listitem>
              <para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para>
	    </listitem>
	    <listitem>
              <para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	    </listitem>
	  </itemizedlist>
	  <para/>
	  <para>Post-notification (stop) / Pre-notification (start)</para>
	  <itemizedlist>
	    <listitem>
              <para>Active resources:</para>
	    </listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource </para>
	  <itemizedlist>
	    <listitem>
              <para>Inactive resources:</para>
	    </listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_stop_resource </para>
	  <itemizedlist>
	    <listitem>
              <para>Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource</para>
	    </listitem>
	    <listitem>
              <para>Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	    </listitem>
	  </itemizedlist>
	  <para/>
	  <para>Post-notification (start)</para>
	  <itemizedlist>
	    <listitem>
              <para>Active resources:</para>
	    </listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem>
              <para>Inactive resources: </para>
	    </listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem>
              <para>Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource</para>
	    </listitem>
	    <listitem>
              <para>Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	    </listitem>
	  </itemizedlist>
	</section>
      </section>
    </section>
    <section id="s-resource-multistate">
      <title>Multi-state - Resources That Have Multiple Modes</title>
      <para>
	Multi-state resources are a specialization of Clones (please ensure you understand the section on clones before continuing) that allow the instances to be in one of two operating modes.
	These modes are called Master and Slave but can mean whatever you wish them to mean.
	The only limitation is that when an instance is started, it must come up in the Slave state.
      </para>
      <section>
	<title>Properties</title>
	<table frame="all">
	  <title>Properties of a Multi-State Resource</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead><tbody><row>
		<entry>id</entry>
		<entry>Your name for the multi-state resource</entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Options</title>
	<para>Options inherited from <link linkend="s-resource-supported">simple resources</link>: <literal>priority, target-role, is-managed</literal></para>
	<para>Options inherited from <link linkend="s-resource-clone">clone resources</link>: <literal>clone-max, clone-node-max, notify, globally-unique, ordered, interleave</literal></para>
	<table frame="all">
	  <title>Multi-state specific resource configuration options</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead><tbody><row>
		<entry>master-max</entry>
		<entry>How many copies of the resource can be promoted to master status. Defaults to 1.</entry>
              </row>
              <row>
		<entry>master-node-max</entry>
		<entry>How many copies of the resource can be promoted to master status on a single node. Defaults to 1.</entry>
          </row></tbody></tgroup>
	</table>
	</section>
      <section>
	<title>Using Multi-state Resources</title>
	<section>
	  <title>Instance Attributes</title>
	  <para>Multi-state resources have no instance attributes, however any that are set here will be inherited by the master's children.</para>
	</section>
	<section>
	  <title>Contents</title>
	  <para>Masters must contain exactly one group or one regular resource.</para>
	  <warning>
	    <para>
	      You should never reference the name of a master's child.
	      If you think you need to do this, you probably need to re-evaluate your design.
	    </para>
	  </warning>
	</section>
	<section>
	  <title>Monitoring Multi-State Resources</title>
	  <para>
	    The normal type of monitor actions you define are not sufficient to monitor a multi-state resource in the Master state.
	    To detect failures of the master instance, you need to define an additional monitor action with <literal>role="Master"</literal>.
	  </para>
	  <important><para>It is crucial that every monitor operation has a different interval</para></important>
	  <example>
	    <title>Monitoring both states of a multi-state resource</title>
	    <programlisting>
<![CDATA[
  <master id="myMasterRsc">
   <primitive id="myRsc" class="ocf" type="myApp" provider="myCorp">
    <operations>
     <op id="public-ip-slave-check" name="monitor" interval="60"/>
     <op id="public-ip-master-check" name="monitor" interval="61" role="Master"/>
    </operations>
   </primitive>
  </master>
]]>
	    </programlisting>
	  </example>
	</section>
	<section>
	  <title>Constraints</title>
	  <para>
	    In most cases, a multi-state resources will have a single copy on each active cluster node.
	    However if this is not the case, you can indicate which nodes the cluster should to preferentially assign copies to with resource location constraints.
	    These constraints are written no differently to those for regular resources except that the master's id is used.
	  </para>
	  <para>
	    When considering multi-state resources in constraints, for most purposes it is sufficient to treat them as clones.
	    The exception is when the <literal>rsc-role</literal> and/or <literal>with-rsc-role</literal> (for colocation constraints) and <literal>first-action</literal> and/or <literal>then-action</literal> (for ordering constraints) are used.
	  </para>
	  <table frame="all">
	    <title>Additional constraint options relevant to multi-state resources</title>
	    <tgroup cols="2">
              <thead>
		<row>
		  <entry>Field</entry>
		  <entry>Description</entry>
		</row>
              </thead><tbody><row>
		  <entry>rsc-role</entry>
		  <entry>
		    <para>An additional attribute of colocation constraints that specifies the role that rsc must be in. </para>
		    <para>Allowed values: <emphasis>Started</emphasis>, Master, Slave</para>
		  </entry>
		</row>
		<row>
		  <entry>with-rsc-role</entry>
		  <entry>
		    <para>An additional attribute of colocation constraints that specifies the role that with-rsc must be in.</para>
		    <para>Allowed values: <emphasis>Started</emphasis>, Master, Slave</para>
		  </entry>
		</row>
		<row>
		  <entry>first-action</entry>
		  <entry>
		    <para>An additional attribute of ordering constraints that specifies the action that the first resource must complete before executing the specified action for the then resource. </para>
		    <para>Allowed values: <emphasis>start</emphasis>, stop, promote, demote</para>
		  </entry>
		</row>
		<row>
		  <entry>then-action</entry>
		  <entry>
		    <para>An additional attribute of ordering constraints that specifies the action that the then resource can only execute after the first-action on the first resource has completed.</para>
		    <para>Allowed values: start, stop, promote, demote. Defaults to the value (specified or implied) of first-action</para>
		  </entry>
		</row>
	      </tbody>
	    </tgroup>
	  </table>
	  <para>
	    In the example below, myApp will wait until one of database copies has been started and promoted to master before being started itself.
	    Only if no copies can be promoted will apache-stats be prevented from being active.
	    Additionally, the database will wait for myApp to be stopped before it is demoted.
	  </para>
	  <para>
	    Colocation of a regular (or group) resource with a multi-state resource means that it can run on any machine with an active copy of the clone that is in the specified state (Master or Slave).
	    In the example, the cluster will choose a location based on where database is running as a Master, and if there are multiple Master instances it will also factor in myApp‘s own location preferences when deciding which location to choose.
	  </para>
	  <para>
	    Colocation with regular clones and other multi-state resources is also possible.
	    In such cases, the set of allowed locations for the rsc clone is (after role filtering) limited to nodes on which the <literal>with-rsc</literal> clone is (or will be) in the specified role.
	    Allocation is then performed as-per-normal.
	  </para>
	  <example>
	    <title>Example constraints involving multi-state resources</title>
	    <programlisting>
<![CDATA[
  <constraints>
   <rsc_location id="db-prefers-node1" rsc="database" node="node1" score="500"/>
   <rsc_colocation id="backup-with-db-slave" rsc="backup" with-rsc="database" with-rsc-role="Slave"/>
   <rsc_colocation id="myapp-with-db-master" rsc="myApp" with-rsc="database" with-rsc-role="Master"/>
   <rsc_order id="start-db-before-backup" first="database" then="backup"/>
   <rsc_order id="promote-db-then-app" first="database" first-action="promote" then="myApp" then-action="start"/>
  </constraints>
]]>
	    </programlisting>
	  </example>
	</section>
	<section>
	  <title>Stickiness</title>
	  <para>
	    To achieve a stable allocation pattern, clones are slightly sticky by default.
	    If no value for <literal>resource-stickiness</literal> is provided, the clone will use a value of 1.
	    Being a small value, it causes minimal disturbance to the score calculations of other resources but is enough to prevent Pacemaker from needlessly moving copies around the cluster.
	  </para>
	</section>
	<section>
	  <title>Which Resource Instance is Promoted</title>
	  <para>
	    During the start operation, most Resource Agent scripts should call the <command>crm_master</command> utility.
	    This tool automatically detects both the resource and host and should be used to set a preference for being promoted.
	    Based on this, <literal>master-max</literal>, and <literal>master-node-max</literal>, the instance(s) with the highest preference will be promoted.
	  </para>
	  <para>The other alternative is to create a location constraint that indicates which nodes are most preferred as masters.</para>
	  <example>
	    <title>Manually specifying which node should be promoted</title>
	    <programlisting>
<![CDATA[
  <rsc_location id="master-location" rsc="myMasterRsc">
    <rule id="master-rule" score="100" role="Master">
      <expression id="master-exp" attribute="#uname" operation="eq" value="node1"/>
    </rule>
  </rsc_location>
]]>
	    </programlisting>
	  </example>
	</section>
	<section>
	  <title>Resource Agent Requirements</title>
	  <para>
	    Since multi-state resources are an extension of cloned resources, all the requirements of Clones are also requirements of multi-state resources.
	    Additionally, multi-state resources require two extra actions demote and promote.
	    These actions are responsible for changing the state of the resource.
	    Like start and stop, they should return <envar>OCF_SUCCESS</envar> if they completed successfully or a relevant error code if they did not.
	  </para>
	  <para>
	    The states can mean whatever you wish, but when the resource is started, it must come up in the mode called Slave.
	    From there the cluster will then decide which instances to promote into a Master.
	  </para>
	  <para>
	    In addition to the Clone requirements for monitor actions, agents must also <emphasis>accurately</emphasis> report which state they are in.
	    The cluster relies on the agent to report its status (including role) accurately and does not indicate to the agent what role it currently believes it to be in.
	  </para>
	  <table frame="all">
	    <title>Role implications of OCF return codes</title>
	    <tgroup cols="2">
              <thead>
		<row>
		  <entry>Monitor Return Code</entry>
		  <entry>Description</entry>
		</row>
              </thead><tbody><row>
		  <entry>OCF_NOT_RUNNING</entry>
		  <entry>Stopped</entry>
		</row>
		<row>
		  <entry>OCF_SUCCESS</entry>
		  <entry>Running (Slave)</entry>
		</row>
		<row>
		  <entry>OCF_RUNNING_MASTER</entry>
		  <entry>Running (Master)</entry>
		</row>
		<row>
		  <entry>OCF_FAILED_MASTER</entry>
		  <entry>Failed (Master)</entry>
		</row>
		<row>
		  <entry>Other</entry>
		  <entry>Failed (Slave)</entry>
            </row></tbody></tgroup>
	  </table>
	</section>
	<section>
	  <title>Notifications</title>
	  <para>
	    Like with clones, supporting notifications requires the <literal>notify</literal> action to be implemented.
	    Once supported, the notify action will be passed a number of extra variables which, when combined with additional context, can be used to calculate the current state of the cluster and what is about to happen to it.
	  </para>
	  <table frame="all">
	    <title>Environment variables supplied with Master notify actions
	      <footnote><para>Variables in bold are specific to Master resources and all behave in the same manner as described for Clone resources.</para></footnote>
	    </title>
	    <tgroup cols="2">
              <thead>
		<row>
		  <entry>Variable</entry>
		  <entry>Description</entry>
		</row>
              </thead><tbody><row>
		  <entry>OCF_RESKEY_CRM_meta_notify_type</entry>
		  <entry>Allowed values: pre, post</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_operation</entry>
		  <entry>Allowed values: start, stop</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_active_resource</entry>
		  <entry>Resources the that are running</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_inactive_resource</entry>
		  <entry>Resources the that are not running</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_master_resource</emphasis></entry>
		  <entry>Resources that are running in Master mode</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_slave_resource</emphasis></entry>
		  <entry>Resources that are running in Slave mode</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_start_resource</entry>
		  <entry>Resources to be started</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_stop_resource</entry>
		  <entry>Resources to be stopped</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_promote_resource</emphasis></entry>
		  <entry>Resources to be promoted</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_demote_resource</emphasis></entry>
		  <entry>Resources to be demoted</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_start_uname</entry>
		  <entry>Nodes on which resources will be started</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_stop_uname</entry>
		  <entry>Nodes on which resources will be stopped</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_promote_uname</emphasis></entry>
		  <entry>Nodes on which resources will be promoted</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_demote_uname</emphasis></entry>
		  <entry>Nodes on which resources will be demoted</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_active_uname</entry>
		  <entry>Nodes on which resources are running</entry>
		</row>
		<row>
		  <entry>OCF_RESKEY_CRM_meta_notify_inactive_uname</entry>
		  <entry>Nodes on which resources are not running</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_master_uname</emphasis></entry>
		  <entry>Nodes on which resources are running in Master mode</entry>
		</row>
		<row>
		  <entry><emphasis>OCF_RESKEY_CRM_meta_notify_slave_uname</emphasis></entry>
		  <entry>Nodes on which resources are running in Slave mode</entry>
            </row></tbody></tgroup>
	  </table>
	</section>
	<section>
	  <title>Proper Interpretation of Notification Environment Variables</title>
	  <para>Pre-notification (demote)</para>
	  <itemizedlist>
	    <listitem><para>Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource</para></listitem>
	    <listitem><para>Master resources: $OCF_RESKEY_CRM_meta_notify_master_resource</para></listitem>
	    <listitem><para>Slave resources: $OCF_RESKEY_CRM_meta_notify_slave_resource</para></listitem>
	    <listitem><para>Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource</para></listitem>
	    <listitem><para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources to be promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources to be demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	  </itemizedlist>
	  <para/>
	  <para>Post-notification (demote) / Pre-notification (stop)</para>
	  <itemizedlist>
	    <listitem><para>Active resources: $OCF_RESKEY_CRM_meta_notify_active_resource</para></listitem>
	    <listitem><para>Master resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_master_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_demote_resource </para>
	  <itemizedlist>
	    <listitem><para>Slave resources: $OCF_RESKEY_CRM_meta_notify_slave_resource</para></listitem>
	    <listitem><para>Inactive resources: $OCF_RESKEY_CRM_meta_notify_inactive_resource</para></listitem>
	    <listitem><para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources to be promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources to be demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	    <listitem><para>Resources that were demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	  </itemizedlist>
	  <para/>
	  <para>Post-notification (stop) / Pre-notification (start)</para>
	  <itemizedlist>
	    <listitem><para>Active resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource </para>
	  <itemizedlist>
	    <listitem><para>Master resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_master_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_demote_resource </para>
	  <itemizedlist>
	    <listitem><para>Slave resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_slave_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource </para>
	  <itemizedlist>
	    <listitem><para>Inactive resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_stop_resource </para>
	  <itemizedlist>
	    <listitem><para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources to be promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources to be demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	    <listitem><para>Resources that were demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	  </itemizedlist>
	  <para>Post-notification (start) / Pre-notification (promote)</para>
	  <itemizedlist>
	    <listitem><para>Active resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem><para>Master resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_master_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_demote_resource </para>
	  <itemizedlist>
	    <listitem><para>Slave resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_slave_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem><para>Inactive resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem><para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources to be promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources to be demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	    <listitem><para>Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources that were demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	  </itemizedlist>
	  <para/>
	  <para>Post-notification (promote)</para>
	  <itemizedlist>
	    <listitem><para>Active resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_active_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem><para>Master resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_master_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_demote_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_promote_resource </para>
	  <itemizedlist>
	    <listitem><para>Slave resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_slave_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_start_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_promote_resource </para>
	  <itemizedlist>
	    <listitem><para>Inactive resources: </para></listitem>
	  </itemizedlist>
	  <para>    $OCF_RESKEY_CRM_meta_notify_inactive_resource</para>
	  <para>        plus $OCF_RESKEY_CRM_meta_notify_stop_resource</para>
	  <para>        minus $OCF_RESKEY_CRM_meta_notify_start_resource </para>
	  <itemizedlist>
	    <listitem><para>Resources to be started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources to be promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources to be demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources to be stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	    <listitem><para>Resources that were started: $OCF_RESKEY_CRM_meta_notify_start_resource</para></listitem>
	    <listitem><para>Resources that were promoted: $OCF_RESKEY_CRM_meta_notify_promote_resource</para></listitem>
	    <listitem><para>Resources that were demoted: $OCF_RESKEY_CRM_meta_notify_demote_resource</para></listitem>
	    <listitem><para>Resources that were stopped: $OCF_RESKEY_CRM_meta_notify_stop_resource</para></listitem>
	  </itemizedlist>
	</section>
      </section>
    </section>
  </chapter>
