  <chapter id="ch-basics"><title>Configuration Basics</title>
    <section id="s-config-layout">
      <title>Configuration Layout</title>
      <para>The cluster is written using XML notation and divided into two main sections; configuration and status.</para>
      <para>
	The status section contains the history of each resource on each node and based on this data, the cluster can construct the complete current state of the cluster.
	The authoritative source for the status section is the local resource manager (lrmd) process on each cluster node and the cluster will occasionally repopulate the entire section.
	For this reason it is never written to disk and admin's are advised against modifying it in any way.
      </para>
      <para>
	The configuration section contains the more traditional information like cluster options, lists of resources and indications of where they should be placed.
	The configuration section is the primary focus of this document.
      </para>
      <para>The configuration section itself is divided into four parts:</para>
      <itemizedlist>
	<listitem><para>Configuration options (called <literal>crm_config</literal>)</para></listitem>
	<listitem><para>Nodes</para></listitem>
	<listitem><para>Resources</para></listitem>
	<listitem><para>Resource relationships (called <literal>constraints</literal>)</para></listitem>
      </itemizedlist>
      <example>
	<title>An empty configuration</title>
	<programlisting>
<![CDATA[
  <cib generated="true" admin_epoch="0" epoch="0" num_updates="0" have-quorum="false">
     <configuration>
        <crm_config/>
        <nodes/>
        <resources/>
        <constraints/>
     </configuration>
     <status/>
  </cib>
]]>
	</programlisting>
      </example>
    </section>
    <section id="s-cluster-status">
      <title>The Current State of the Cluster</title>
      <para>
	Before one starts to configure a cluster, it is worth explaining how to view the finished product.
	For this purpose we have created the <command>crm_mon</command> utility that will display the current state of an active cluster.
	It can show the cluster status by node or by resource and can be used in either single-shot or dynamically-updating mode.
	There are also modes for displaying a list of the operations performed (grouped by node and resource) as well as information about failures.
      </para>
      <para>Using this tool, you can examine the state of the cluster for irregularities and see how it responds when you cause or simulate failures.</para>
      <para>Details on all the available options can be obtained using the <command>crm_mon --help</command> command.</para>
      <figure id="fig-output-crm_mon">
	<title>Sample output from crm_mon</title>
	<screen>
  <command># crm_mon</command>
  ============
  Last updated: Fri Nov 23 15:26:13 2007
  Current DC: sles-3 (2298606a-6a8c-499a-9d25-76242f7006ec)
  3 Nodes configured.
  5 Resources configured.
  ============

  Node: sles-1 (1186dc9a-324d-425a-966e-d757e693dc86): online
      192.168.100.181    (heartbeat::ocf:IPaddr):    Started sles-1
      192.168.100.182    (heartbeat:IPaddr):        Started sles-1
      192.168.100.183    (heartbeat::ocf:IPaddr):    Started sles-1
      rsc_sles-1    (heartbeat::ocf:IPaddr):    Started sles-1
      child_DoFencing:2    (stonith:external/vmware):    Started sles-1
  Node: sles-2 (02fb99a8-e30e-482f-b3ad-0fb3ce27d088): standby
  Node: sles-3 (2298606a-6a8c-499a-9d25-76242f7006ec): online
      rsc_sles-2    (heartbeat::ocf:IPaddr):    Started sles-3
      rsc_sles-3    (heartbeat::ocf:IPaddr):    Started sles-3
      child_DoFencing:0    (stonith:external/vmware):    Started sles-3</screen>
      </figure>
      <figure id="fig-output-crm_mon-node">
	<title>Sample output from crm_mon -n</title>
	<screen>
  <command># crm_mon -n</command>
  ============
  Last updated: Fri Nov 23 15:26:13 2007
  Current DC: sles-3 (2298606a-6a8c-499a-9d25-76242f7006ec)
  3 Nodes configured.
  5 Resources configured.
  ============

  Node: sles-1 (1186dc9a-324d-425a-966e-d757e693dc86): online
  Node: sles-2 (02fb99a8-e30e-482f-b3ad-0fb3ce27d088): standby
  Node: sles-3 (2298606a-6a8c-499a-9d25-76242f7006ec): online

  Resource Group: group-1
    192.168.100.181    (heartbeat::ocf:IPaddr):    Started sles-1
    192.168.100.182    (heartbeat:IPaddr):        Started sles-1
    192.168.100.183    (heartbeat::ocf:IPaddr):    Started sles-1
  rsc_sles-1    (heartbeat::ocf:IPaddr):    Started sles-1
  rsc_sles-2    (heartbeat::ocf:IPaddr):    Started sles-3
  rsc_sles-3    (heartbeat::ocf:IPaddr):    Started sles-3
  Clone Set: DoFencing
    child_DoFencing:0    (stonith:external/vmware):    Started sles-3
    child_DoFencing:1    (stonith:external/vmware):    Stopped
    child_DoFencing:2    (stonith:external/vmware):    Started sles-1</screen>
      </figure>
      <para>
	The DC (Designated Controller) node is where all the decisions are made and if the current DC fails a new one is elected from the remaining cluster nodes.
	The choice of DC is of no significance to an administrator beyond the fact that its logs will generally be more interesting.
      </para>
    </section>
    <section id="s-config-updates">
      <title>How Should the Configuration be Updated?</title>
      <para>There are three basic rules for updating the cluster configuration:</para>
      <itemizedlist>
	<listitem><para>Rule 1 - Never edit the cib.xml file manually. Ever. I'm not making this up.</para></listitem>
	<listitem><para>Rule 2 - Read Rule 1 again.</para></listitem>
	<listitem><para>Rule 3 - The cluster will notice if you ignored rules 1 &amp; 2 and refuse to use the configuration.</para></listitem>
      </itemizedlist>
      <para>Now that it is clear how NOT to update the configuration, we can begin to explain how you should.</para>
      <para>
	The most powerful tool for modifying the configuration is the cibadmin command which talks to a running cluster.
	With cibadmin, the user can query, add, remove, update or replace any part of the configuration and all changes take effect immediately so there is no need to perform a reload-like operation.
      </para>
      <para>The simplest way of using cibadmin is to us it to save the current configuration to a temporary file, edit that file with your favorite text or XML editor and then upload the revised configuration.</para>
      <figure id="fig-config-editor">
	<title>Safely using an editor to modify the cluster configuration</title>
	<programlisting>
  <command>cibadmin --query > tmp.xml</command>
  <command>vi tmp.xml</command>
  <command>cibadmin --replace --xml-file tmp.xml</command>
	</programlisting>
      </figure>
      <para>
	Some of the better XML editors can make use of a Relax NG schema to help make sure any changes you make are valid.
	The schema describing the configuration can normally be found in /usr/lib/heartbeat/pacemaker.rng on most systems.
      </para>
      <para>If you only wanted to modify the resources section, you could instead do </para>
      <figure id="fig-config-replace">
	<title>Safely using an editor to modify a subsection of the cluster configuration</title>
	<programlisting>
  <command>cibadmin --query --obj_type resources > tmp.xml</command>
  <command>vi tmp.xml</command>
  <command>cibadmin --replace --obj_type resources --xml-file tmp.xml</command>
	</programlisting>
      </figure>
      <para>to avoid modifying any other part of the configuration.</para>
    </section>
    <section id="s-config-delete">
      <title>Quickly Deleting Part of the Configuration</title>
      <para>Identify the object you wish to delete. eg.</para>
      <figure id="fig-config-search">
	<title>Searching for STONITH related configuration items</title>
	<screen>
  <command># cibadmin -Q | grep stonith</command>
<![CDATA[
   <nvpair id="cib-bootstrap-options-stonith-action" name="stonith-action" value="reboot"/>
   <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="1"/>
  <primitive id="child_DoFencing" class="stonith" type="external/vmware">
   <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:1" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:2" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:0" type="external/vmware" class="stonith">
   <lrm_resource id="child_DoFencing:3" type="external/vmware" class="stonith">
]]></screen>
      </figure>
      <para>
	Next identify the resource's tag name and id (in this case we'll chose <literal>primitive</literal> and <literal>child_DoFencing</literal>).
	Then simply execute: 
      </para>
      <para><command>cibadmin --delete --crm_xml ‘&lt;primitive id="child_DoFencing"/>'</command></para>
    </section>
    <section id="s-config-tools">
      <title>Updating the Configuration Without Using XML</title>
      <para>Some common tasks can also be performed with one of the higher level tools that avoid the need to read or edit XML.</para>
      <para>To enable stonith for example, one could run:</para>
      <para><command>crm_attribute --attr-name stonith-enabled --attr-value true</command></para>
      <para>Or to see if somenode is allowed to run resources, there is:</para>
      <para><command>crm_standby --get-value --node-uname somenode</command></para>
      <para>Or to find the current location of my-test-rsc one can use:</para>
      <para><command>crm_resource --locate --resource my-test-rsc</command></para>
    </section>
    <section id="s-config-sandboxes">
      <title>Making Configuration Changes in a Sandbox</title>
      <para>
	Often it is desirable to preview the effects of a series of changes before updating the configuration atomically.
	For this purpose we have created <command>crm_shadow</command> which creates a "shadow" copy of the configuration and arranges for all the command line tools to use it.
      </para>
      <para>
	To begin, simply invoke <command>crm_shadow</command> and give it the name of a configuration to create
	<footnote>
          <para>Shadow copies are identified with a name, making it possible to have more than one</para>
	</footnote> and be sure to follow the simple on-screen instructions. 
	<warning><para>Read the above carefully, failure to do so could result in you destroying the cluster's active configuration</para></warning>
      </para>
      <figure id="fig-config-sandbox">
	<title>Creating and displaying the active sandbox</title>
	<programlisting>
 # <command>crm_shadow --create test</command>
 Setting up shadow instance
 Type Ctrl-D to exit the crm_shadow shell
 shadow[test]: 
 shadow[test] # <command>crm_shadow --which</command>
 test</programlisting>
      </figure>
      <para>
	From this point on, all cluster commands will automatically use the shadow copy instead of talking to the cluster's active configuration.
	Once you have finished experimenting, you can either commit the changes, or discard them as shown below.
	Again, be sure to follow the on-screen instructions carefully.
      </para>
      <para>For a full list of <command>crm_shadow</command> options and commands, invoke it with the <parameter>--help</parameter> option.</para>
      <example>
	<title>Using a sandbox to make multiple changes atomically</title>
	<programlisting>
 shadow[test] # <command>crm_failcount -G -r rsc_c001n01</command>
  name=fail-count-rsc_c001n01 value=0
 shadow[test] # <command>crm_standby -v on -n c001n02</command>
 shadow[test] # <command>crm_standby -G -n c001n02</command>
 name=c001n02 scope=nodes value=on
 shadow[test] # <command>cibadmin --erase --force</command>
 shadow[test] # <command>cibadmin --query</command>
<![CDATA[
 <cib cib_feature_revision="1" validate-with="pacemaker-1.0" admin_epoch="0" crm_feature_set="3.0" have-quorum="1" epoch="112"
      dc-uuid="c001n01" num_updates="1" cib-last-written="Fri Jun 27 12:17:10 2008">
    <configuration>
       <crm_config/>
       <nodes/>
       <resources/>
       <constraints/>
    </configuration>
    <status/>
 </cib>
]]>
  shadow[test] # <command>crm_shadow --delete test --force</command>
  Now type Ctrl-D to exit the crm_shadow shell
  shadow[test] # <command>exit</command>
  # <command>crm_shadow --which</command>
  No shadow instance provided
  # <command>cibadmin -Q</command>
<![CDATA[
 <cib cib_feature_revision="1" validate-with="pacemaker-1.0" admin_epoch="0" crm_feature_set="3.0" have-quorum="1" epoch="110"
       dc-uuid="c001n01" num_updates="551">
    <configuration>
       <crm_config>
          <cluster_property_set id="cib-bootstrap-options">
             <nvpair id="cib-bootstrap-1" name="stonith-enabled" value="1"/>
             <nvpair id="cib-bootstrap-2" name="pe-input-series-max" value="30000"/>
]]>
	</programlisting>
	<para>Making changes in a sandbox and verifying the real configuration is untouched</para>
      </example>
    </section>
    <section id="s-config-testing-changes">
      <title>Testing Your Configuration Changes</title>
      <para>
	We saw previously how to make a series of changes to a "shadow" copy of the configuration.
	Before loading the changes back into the cluster (eg. <command>crm_shadow --commit mytest --force</command>), it is often advisable to simulate the effect of the changes with ptest. Eg.
      </para>
      <para><command>ptest --live-check -VVVVV --save-graph tmp.graph --save-dotfile <filename>tmp.dot</filename></command></para>
      <para>
	The tool uses the same library as the live cluster to show what it would have done given the supplied input.
	It's output, in addition to a significant amount of logging, is stored in two files <filename>tmp.graph</filename> and <filename>tmp.dot</filename>, both are representations of the same thing -- the cluster's response to your changes.
	In the graph file is stored the complete transition, containing a list of all the actions, their parameters and their pre-requisites.
	Because the transition graph is not terribly easy to read, the tool also generates a Graphviz dot-file representing the same information.
      </para>
      <para>
	<figure id="fig-transition-small">
	  <title>Small Cluster Transition</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/Policy-Engine-small.png" width="16cm" depth="6cm" align="center"/> 
	    </imageobject>
	    <caption>An example transition graph as represented by Graphviz</caption>
	  </mediaobject>
	</figure>
      </para>
      <para>Interpreting the Graphviz output</para>
      <itemizedlist>
	<listitem><para>Arrows indicate ordering dependencies</para></listitem>
	<listitem><para>Dashed-arrows indicate dependencies that are not present in the transition graph</para></listitem>
	<listitem><para>Actions with a dashed border of any color do not form part of the transition graph</para></listitem>
	<listitem><para>Actions with a green border form part of the transition graph</para></listitem>
	<listitem><para>Actions with a red border are ones the cluster would like to execute but are unrunnable</para></listitem>
	<listitem><para>Actions with a blue border are ones the cluster does not feel need to be executed</para></listitem>
	<listitem><para>Actions with orange text are pseudo/pretend actions that the cluster uses to simplify the graph</para></listitem>
	<listitem><para>Actions with black text are sent to the LRM</para></listitem>
	<listitem><para>Resource actions have text of the form <replaceable>rsc</replaceable>_<replaceable>action</replaceable>_<replaceable>interval</replaceable> <replaceable>node</replaceable></para></listitem>
	<listitem><para>Any action depending on an action with a red border will not be able to execute. </para></listitem>
	<listitem><para>Loops are <emphasis>really</emphasis> bad. Please report them to the development team. </para></listitem>
      </itemizedlist>
      <para>
	In the above example, it appears that a new node, <literal>node2</literal>, has come online and that the cluster is checking to make sure <literal>rsc1</literal>, <literal>rsc2</literal> and <literal>rsc3</literal> are not already running there (Indicated by the <literal>*_monitor_0</literal> entries).
	Once it did that, and assuming the resources were not active there, it would have liked to stop <literal>rsc1</literal> and <literal>rsc2</literal> on <literal>node1</literal> and move them to <literal>node2</literal>.
	However, there appears to be some problem and the cluster cannot or is not permitted to perform the stop actions which implies it also cannot perform the start actions.
	For some reason the cluster does not want to start <literal>rsc3</literal> anywhere.
      </para>
      <para>For information on the options supported by ptest, use <command>ptest --help</command></para>
      <para>
	<figure id="fig-transition-large">
	  <title>Complex Cluster Transition</title>
	  <mediaobject>
	    <imageobject>
	      <imagedata fileref="images/Policy-Engine-big.png" width="16cm" depth="10.5cm" align="center"/> 
	    </imageobject>
	    <caption>Another, slightly more complex, transition graph that you're not expected to be able to read</caption>
	  </mediaobject>
	</figure>
      </para>
    </section>
    <section id="s-config-synchronization">
      <title>Do I Need to Update the Configuration on all Cluster Nodes?</title>
      <para>No. Any changes are immediately synchronized to the other active members of the cluster.</para>
      <para>To reduce bandwidth, the cluster only broadcasts the incremental updates that result from your changes and uses MD5 sums to ensure that each copy is completely consistent.</para>
    </section>
  </chapter>
