  <chapter id="ch-constraints"><title>Resource Constraints</title>
    <section id="s-scores">
      <title>Scores</title>
      <para>
	Scores of all kinds are integral to how the cluster works.
	Practically everything from moving a resource to deciding which resource to stop in a degraded cluster is achieved by manipulating scores in some way.
      </para>
      <para>
	Scores are calculated on a per-resource basis and any node with a negative score for a resource can't run that resource.
	After calculating the scores for a resource, the cluster then chooses the node with the highest one.
      </para>
      <section id="s-scores-infinity">
	<title>Infinity Math</title>
	<para><literal>INFINITY</literal> is currently defined as 1,000,000 and addition/subtraction with it follows the following 3 basic rules:</para>
	<itemizedlist>
	  <listitem><para>Any value + <literal>INFINITY</literal> = <literal>INFINITY</literal></para></listitem>
	  <listitem><para>Any value - <literal>INFINITY</literal> = -<literal>INFINITY</literal></para></listitem>
	  <listitem><para><literal>INFINITY</literal> - <literal>INFINITY</literal> = -<literal>INFINITY</literal></para></listitem>
	</itemizedlist>
      </section>
    </section>
    <section id="s-resource-location">
      <title>Deciding Which Nodes a Resource Can Run On</title>
      <para>
	There are two alternative strategies for specifying which nodes a resources can run on.
	One way is to say that by default they can run anywhere and then create location constraints for nodes that are not allowed.
	The other option is to have nodes "opt-in"...
	to start with nothing able to run anywhere and selectively enable allowed nodes.
      </para>
      <section>
	<title>Options</title>
	<table frame="all">
	  <title>Options for Simple Location Constraints</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead><tbody><row>
		<entry>id</entry>
		<entry>A unique name for the constraint</entry>
              </row>
              <row>
		<entry>rsc</entry>
		<entry>A resource name</entry>
              </row>
              <row>
		<entry>node</entry>
		<entry>A node's uname</entry>
              </row>
              <row>
		<entry>score</entry>
		<entry>Positive values indicate the resource can run on this node. Negative values indicate the resource can not run on this node. Values of +/- <literal>INFINITY</literal> change "can" to "must".</entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Asymmetrical "Opt-In" Clusters</title>
	<para>To create an opt-in cluster, start by preventing resources from running anywhere by default</para>
	<para><command>crm_attribute --attr-name symmetric-cluster --attr-value false</command></para>
	<para>
	  Then start enabling nodes.
	  The following fragment says that the web server prefers sles-1, the database prefers sles-2 and both can failover to sles-3 if their most preferred node fails.
	</para>
	<example>
	  <title>Example set of opt-in location constraints</title>
	  <programlisting>
<![CDATA[
  <constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-3" score="0"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-2" score="200"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-3" score="0"/>
  </constraints>
]]>
	  </programlisting>
	</example>
      </section>
      <section>
	<title>Symmetrical "Opt-Out" Clusters</title>
	<para>To create an opt-out cluster, start by allowing resources to run anywhere by default</para>
	<para><command>crm_attribute --attr-name symmetric-cluster --attr-value true</command></para>
	<para>
	  Then start disabling nodes.
	  The following fragment is the equivalent of the above opt-in configuration.
	</para>
	<example>
	  <title>Example set of opt-out location constraints</title>
	  <programlisting>
<![CDATA[
  <constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="200"/>
    <rsc_location id="loc-2-dont-run" rsc="Webserver" node="sles-2" score="-INFINITY"/>
    <rsc_location id="loc-3-dont-run" rsc="Database" node="sles-1" score="-INFINITY"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="200"/>
  </constraints>
]]>
	  </programlisting>
	</example>
	<para>
	  Whether you should choose opt-in or opt-out depends both on your personal preference and the make-up of your cluster.
	  If most of your resources can run on most of the nodes, then an opt-out arrangement is likely to result in a simpler configuration.
	  On the other-hand, if most resources can only run on a small subset of nodes an opt-in configuration might be simpler.
	</para>
      </section>
      <section id="node-score-equal">
	<title>What if Two Nodes Have the Same Score</title>
	<para>
	  If two nodes have the same score, then the cluster will choose one.
	  This choice may seem random and may not be what was intended, however the cluster was not given enough information to know what was intended.
	</para>
	<example>
	  <title>Example of two resources that prefer two nodes equally</title>
	  <programlisting>
<![CDATA[
  <constraints>
    <rsc_location id="loc-1" rsc="Webserver" node="sles-1" score="INFINITY"/>
    <rsc_location id="loc-2" rsc="Webserver" node="sles-2" score="INFINITY"/>
    <rsc_location id="loc-3" rsc="Database" node="sles-1" score="500"/>
    <rsc_location id="loc-4" rsc="Database" node="sles-2" score="300"/>
    <rsc_location id="loc-5" rsc="Database" node="sles-2" score="200"/>
  </constraints>
]]>
	  </programlisting>
	</example>
	<para>
	  In the example above, assuming no other constraints and an inactive cluster, Webserver would probably be placed on sles-1 and Database on sles-2.
	  It would likely have placed Webserver based on the node's uname and Database based on the desire to spread the resource load evenly across the cluster.
	  However other factors can also be involved in more complex configurations.
	</para>
      </section>
    </section>
    <section id="s-resource-ordering">
      <title>Specifying the Order Resources Should Start/Stop In</title>
      <para>The way to specify the order in which resources should start is by creating <literal>rsc_order</literal> constraints.</para>
      <table frame="all">
	<title>Properties of an Ordering Constraint</title>
	<tgroup cols="2">
          <thead>
            <row>
	      <entry>Field</entry>
	      <entry>Description</entry>
            </row>
          </thead>
	  <tbody>
	    <row>
	      <entry>id</entry>
	      <entry>A unique name for the constraint</entry>
            </row>
            <row>
	      <entry>first</entry>
	      <entry>The name of a resource that must be started before the then resource is allowed to. </entry>
            </row>
            <row>
	      <entry>then</entry>
	      <entry>The name of a resource. This resource will start after the first resource.</entry>
            </row>
            <row>
	      <entry>score</entry>
	      <entry>If greater than zero, the constraint is mandatory. Otherwise it is only a suggestion. Default value: <emphasis>INFINITY</emphasis>
	      </entry>
            </row>
            <row>
	      <entry>symmetrical</entry>
	      <entry>If true, which is the default, stop the resources in the reverse order. Default value: <emphasis>true</emphasis></entry>
            </row>
	  </tbody>
	</tgroup>
      </table>
      <section>
	<title>Mandatory Ordering</title>
	<para>
	  When the then resource cannot run without the first resource being active, one should use mandatory constraints.
	  To specify a constraint is mandatory, use a scores greater than zero.
	  This will ensure that the then resource will react when the first resource changes state.
	</para>
	<itemizedlist>
	  <listitem><para>If the first resource was running and is stopped, the then resource will also be stopped (if it is running)</para></listitem>
	  <listitem><para>If the first resource was not running and cannot be started, the then resource will be stopped (if it is running)</para></listitem>
	  <listitem><para>If the first resource is (re)started while the then resource is running, the then resource will be stopped and restarted</para></listitem>
	</itemizedlist>
      </section>
      <section>
	<title>Advisory Ordering</title>
	<para>
	  On the other-hand, when <literal>score="0"</literal> is specified for a constraint, the constraint is considered optional and only has an effect when both resources are stopping and or starting.
	  Any change in state by the first resource will have no effect on the then resource.
	</para>
	<example>
	  <title>Example of an optional and mandatory ordering constraint</title>
	  <programlisting>
<![CDATA[
  <constraints>
    <rsc_order id="order-1" first="Database" then="Webserver" />
    <rsc_order id="order-2" first="IP" then="Webserver" score="0"/>
  </constraints>
]]>
	  </programlisting>
	</example>
	<para>Some additional information on ordering constraints can be found in the document <ulink url="http://www.clusterlabs.org/mediawiki/images/d/d6/Ordering_Explained.pdf">Ordering Explained</ulink>
	</para>
      </section>
    </section>
    <section id="s-resource-colocation">
      <title>Placing Resources Relative to other Resources</title>
      <para>When the location of one resource depends on the location of another one, we call this colocation.</para>
      <para>
	There is an important side-effect of creating a colocation constraint between two resources, that it affects the order in which resources are assigned to a node.
	If you think about it, its somewhat obvious.
	You can't place A relative to B unless you know where B is
	<footnote>
          <para>While the human brain is sophisticated enough to read the constraint in any order and choose the correct one depending on the situation, the cluster is not quite so smart. Yet.</para>
	</footnote>.
	So when you are creating colocation constraints, it is important to consider whether you should colocate A with B or B with A.
      </para>
      <para>
	Another thing to keep in mind is that, assuming A is collocated with B, the cluster will also take into account A's preferences when deciding which node to choose for B.
	For a detailed look at exactly how this occurs, see the <ulink url="http://www.clusterlabs.org/mediawiki/images/6/61/Colocation_Explained.pdf">Colocation Explained</ulink> document.
      </para>
      <section>
	<title>Options</title>
	<table frame="all">
	  <title>Properties of a Collocation Constraint</title>
	  <tgroup cols="2">
            <thead>
              <row>
		<entry>Field</entry>
		<entry>Description</entry>
              </row>
            </thead>
	    <tbody>
	      <row>
		<entry>id</entry>
		<entry>A unique name for the constraint</entry>
              </row>
              <row>
		<entry>rsc</entry>
		<entry>The colocation source. If the constraint cannot be satisfied, the cluster may decide not to allow the resource to run at all.</entry>
              </row>
              <row>
		<entry>with-rsc</entry>
		<entry>The colocation target. The cluster will decide where to put this resource first and then decide where to put the resource in the rsc field</entry>
              </row>
              <row>
		<entry>score</entry>
		<entry>Positive values indicate the resource should run on the same node. Negative values indicate the resources should not run on the same node. Values of +/- <literal>INFINITY</literal> change "should" to "must".</entry>
              </row>
	    </tbody>
	  </tgroup>
	</table>
      </section>
      <section>
	<title>Mandatory Placement</title>
	<para>
	  Mandatory placement occurs any time the constraint's score is <literal>+INFINITY</literal> or <literal>-INFINITY</literal>.
	  In such cases, if the constraint can't be satisfied, then the rsc resource is not permitted to run.
	  For <literal>score=INFINITY</literal>, this includes cases where the with-rsc resource is not active.
	</para>
	<para>If you need resource1 to always run on the same machine as resource2, you would add the following constraint: </para>
	<example>
	  <title>An example colocation constraint</title>
	  <programlisting>
  &lt;rsc_colocation id="colocate" rsc="resource1" with-rsc="resource2" score="INFINITY"/>
	  </programlisting>
	</example>
	<para>Remember, because <literal>INFINITY</literal> was used, if resource2 can't run on any of the cluster nodes (for whatever reason) then resource1 will not be allowed to run.</para>
	<para>
	  Alternatively, you may want the opposite...
	  that resource1 cannot run on the same machine as resource2.
	  In this case use <literal>score="-INFINITY"</literal>
	</para>
	<example>
	  <title>An example anti-colocation constraint</title>
	  <programlisting>
  &lt;rsc_colocation id="anti-colocate" rsc="resource1" with-rsc="resource2" score="-INFINITY"/> 
	  </programlisting>
	</example>
	<para>
	  Again, by specifying <literal>-INFINTY</literal>, the constraint is binding.
	  So if the only place left to run is where resource2 already is, then resource1 may not run anywhere.
	</para>
      </section>
      <section>
	<title>Advisory Placement</title>
	<para>
	  If mandatory placement is about "must" and "must not", then advisory placement is the "I'd prefer if" alternative.
	  For constraints with scores greater than <literal>-INFINITY</literal> and less than <literal>INFINITY</literal>, the cluster will try and accommodate your wishes but may ignore them if the alternative is to stop some of the cluster resources.
	</para>
	<para>Like in life, where if enough people prefer something it effectively becomes mandatory,  advisory colocation constraints can combine with other elements of the configuration to behave as if they were mandatory.</para>
	<example>
	  <title>An example advisory-only colocation constraint</title>
	  <programlisting>
  &lt;rsc_colocation id="colocate-maybe" rsc="resource1" with-rsc="resource2" score="500"/>
	  </programlisting>
	</example>
      </section>
    </section>
    <section id="s-resource-sets-ordering">
      <title>Ordering Sets of Resources</title>
      <para>A common situation is for an administrator to create a chain of ordered resources, such as:</para>
      <example>
	<title>A chain of ordered resources</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_order id="order-1" first="A" then="B" />
    <rsc_order id="order-2" first="B" then="C" />
    <rsc_order id="order-3" first="C" then="D" />
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	<figure id="fig-ordered-set">
	  <title>Ordered Set</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/resource-set.png" width="16cm" depth="2.5cm" align="center"/> 
	    </imageobject>
	    <caption>Visual representation of the four resources' start order for the above constraints</caption>
	  </mediaobject>
	</figure>
      </para>
      <para>To simplify this situation, there is an alternate format for ordering constraints</para>
      <example>
	<title>A chain of ordered resources expressed as a set</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
  </constraints>
]]>
	</programlisting>
      </example>
      <note><para>Resource sets have the same ordering semantics as groups.</para></note>
      <example>
	<title>A group resource with the equivalent ordering rules</title>
	<programlisting>
<![CDATA[
  <group id="dummy">
    <primitive id="A" .../>
    <primitive id="B" .../>
    <primitive id="C" .../>
    <primitive id="D" .../>
  </group>
]]>
	</programlisting>
      </example>
      <para>
	While the set-based format is not less verbose, it is significantly easier to get right and maintain.
	It can also be expanded to allow ordered sets of (un)ordered resources.
	In the example below, rscA and rscB can both start in parallel, as can rscC and rscD, however rscC and rscD can only start once <emphasis>both</emphasis> rscA <emphasis>and</emphasis> rscB are active.</para>
      <example>
	<title>Ordered sets of unordered resources</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_order>
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	<figure id="fig-ordered-set-two">
	  <title>Two Sets of Unordered Resources</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/two-sets.png" width="13cm" depth="7.5cm" align="center"/> 
	    </imageobject>
	    <caption>Visual representation of the start order for two ordered sets of unordered resources</caption>
	  </mediaobject>
	</figure>
      </para>
      <para>Of course either or both sets of resources can also be internally ordered (by setting <literal>sequential="true"</literal>) and there is no limit to the number of sets that can be specified.</para>
      <example>
	<title>Advanced use of set ordering - Three ordered sets, two of which are internally unordered </title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_order id="order-1">
      <resource_set id="ordered-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="ordered-set-2" sequential="true">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
      <resource_set id="ordered-set-3" sequential="false">
        <resource_ref id="E"/>
        <resource_ref id="F"/>
      </resource_set>
    </rsc_order>
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	<figure id="fig-ordered-set-three">
	  <title>Three Resources Sets</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/three-sets.png" width="16cm" depth="7.5cm" align="center"/> 
	    </imageobject>
	    <caption>Visual representation of the start order for the three sets defined above</caption>
	  </mediaobject>
	</figure>
      </para>
    </section>
    <section id="s-resource-sets-collocation">
      <title>Collocating Sets of Resources</title>
      <para>
	Another common situation is for an administrator to create a set of collocated resources.
	Previously this possible either by defining a resource group (See <xref linkend="group-resources"/>) which could not always accurately express the design; or by defining each relationship as an individual constraint, causing a constraint explosion as the number of resources and combinations grew.
      </para>
      <example>
	<title>A chain of collocated resources</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_colocation id="coloc-1" rsc="B" with-rsc="A" score="INFINITY"/>
    <rsc_colocation id="coloc-2" rsc="C" with-rsc="B" score="INFINITY"/>
    <rsc_colocation id="coloc-3" rsc="D" with-rsc="C" score="INFINITY"/>
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	To make things easier, we allow an alternate form of colocation constraints using resource_sets.
	Just like the expanded version, a resource that can't be active also prevents any resource that must be collocated with it from being active.
	For example if B was not able to run, then both C (and by inference D) must also remain stopped.
      </para>
      <example>
	<title>The equivalent colocation chain expressed using resource_sets</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-example" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
  </constraints>
]]>
	</programlisting>
      </example>
      <note><para>Resource sets have the same colocation semantics as groups.</para></note>
      <example>
	<title>A group resource with the equivalent colocation rules</title>
	<programlisting>
<![CDATA[
  <group id="dummy">
    <primitive id="A" .../>
    <primitive id="B" .../>
    <primitive id="C" .../>
    <primitive id="D" .../>
  </group>
]]>
	</programlisting>
      </example>
      <para>
	This notation can also be used in this context to tell the cluster that a set of resources must all be located with a common peer, but have no dependencies on each other.
	In this scenario, unlike the previous on, B would be allowed to remain active even if A or C (or both) were inactive.
      </para>
      <example>
	<title>Using colocation sets to specify a common peer. </title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-1" sequential="false">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
        <resource_ref id="C"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="true">
        <resource_ref id="D"/>
      </resource_set>
    </rsc_colocation>
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	Of course there is no limit to the number and size of the sets used.
	The only thing that matters is that in order for any member of set N to be active, all the members of set N+1 must also be active (and naturally on the same node), and that if a set has <literal>sequential="true"</literal>, then in order for member M to be active, member M+1 must also be active.
	You can even specify the role in which the members of a set must be in using the set's role attribute.
      </para>
      <example>
	<title>A colocation chain where the members of the middle set have no inter-dependencies and the last has master status.</title>
	<programlisting>
<![CDATA[
  <constraints>
    <rsc_colocation id="coloc-1" score="INFINITY" >
      <resource_set id="collocated-set-1" sequential="true">
        <resource_ref id="A"/>
        <resource_ref id="B"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="false">
        <resource_ref id="C"/>
        <resource_ref id="D"/>
        <resource_ref id="E"/>
      </resource_set>
      <resource_set id="collocated-set-2" sequential="true" role="Master">
        <resource_ref id="F"/>
        <resource_ref id="G"/>
      </resource_set>
    </rsc_colocation>
  </constraints>
]]>
	</programlisting>
      </example>
      <para>
	<figure>
	  <title>Another Three Resources Sets</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/three-sets-complex.png" width="20cm" depth="9cm" align="center"/> 
	    </imageobject>
	    <caption>Visual representation of a colocation chain where the members of the middle set have no inter-dependencies</caption>
	  </mediaobject>
	</figure>
      </para>
    </section>
  </chapter>
