Next: 2.6 Managing a Job Up: 2. Users' Manual Previous: 2.4 Running a Job: Contents Index
A job is submitted for execution to HTCondor using the condor_submit command. condor_submit takes as an argument the name of a file called a submit description file. This file contains commands and keywords to direct the queuing of jobs. In the submit description file, HTCondor finds everything it needs to know about the job. Items such as the name of the executable to run, the initial working directory, and command-line arguments to the program all go into the submit description file. condor_submit creates a job ClassAd based upon the information, and HTCondor works toward running the job.
The contents of a submit description file have been designed to save time for HTCondor users. It is easy to submit multiple runs of a program to HTCondor with a single submit description file. To run the same program many times on different input data sets, arrange the data files accordingly so that each run reads its own input, and each run writes its own output. Each individual run may have its own initial working directory, files mapped for stdin, stdout, stderr, command-line arguments, and shell environment; these are all specified in the submit description file. A program that directly opens its own files will read the file names to use either from stdin or from the command line. A program that opens a static file, given by file name, every time will need to use a separate subdirectory for the output of each run.
The condor_submit manual page
is on page ![[*]](crossref.png)
and
contains a complete and full description of how to use condor_submit.
It also includes descriptions of all the many commands that may be placed
into a submit description file.
In addition, the index lists entries for each command under the
heading of Submit Commands.
In addition to the examples of submit description files given here, there are more in the condor_submit manual page.
An executable is compiled to run on a specific platform. Since this submit description file does not specify a platform, HTCondor will use its default, which is to run the job on a machine which has the same architecture and operating system as the machine where condor_submit is run to submit the job.
Standard input for this job will come from file inputfile, as specified by the input command, and standard output for this job will go to file outputfile, as specified by the output command. HTCondor expects to find these files in the current working directory, as this job is submitted, and the system will take care of getting the input file to where it needs to be when the job is executed, as well as bring back the output results after job execution.
A log file, myexe.log, will also be produced that contains events the job had during its lifetime inside of HTCondor. When the job finishes, its exit conditions will be noted in the log file. This file's contents are an excellent way to figure out what happened to submitted jobs.
####################
#
# Example 1
# Simple HTCondor submit description file
#
####################
Executable = myexe
Log = myexe.log
Input = inputfile
Output = outputfile
Queue
####################
#
# Example 2
# Standard universe submit description file
#
####################
Executable = foo
Universe = standard
Log = foo.log
Queue
Example 3 queues two copies of the program mathematica. The first copy will run in directory run_1, and the second will run in directory run_2 due to the initialdir command. For each copy, stdin will be test.data, stdout will be loop.out, and stderr will be loop.error. Each run will read input and write output files within its own directory. Placing data files in separate directories is a convenient way to organize data when a large group of HTCondor jobs is to run. The example file shows program submission of mathematica as a vanilla universe job. The vanilla universe is most often the right choice of universe when the source and/or object code is not available.
The request_memory command is included to ensure that the mathematica jobs match with and then execute on pool machines that provide at least 1 GByte of memory.
####################
#
# Example 3: demonstrate use of multiple
# directories for data organization.
#
####################
executable = mathematica
universe = vanilla
input = test.data
output = loop.out
error = loop.error
log = loop.log
request_memory = 1 GB
initialdir = run_1
queue
initialdir = run_2
queue
The submit description file for Example 4 queues 150 runs of program foo which has been compiled and linked for Linux running on a 32-bit Intel processor. This job requires HTCondor to run the program on machines which have greater than 32 MiB of physical memory, and the rank command expresses a preference to run each instance of the program on machines with more than 64 MiB. It also advises HTCondor that this standard universe job will use up to 28000 KiB of memory when running. Each of the 150 runs of the program is given its own process number, starting with process number 0. So, files stdin, stdout, and stderr will refer to in.0, out.0, and err.0 for the first run of the program, in.1, out.1, and err.1 for the second run of the program, and so forth. A log file containing entries about when and where HTCondor runs, checkpoints, and migrates processes for all the 150 queued programs will be written into the single file foo.log.
#################### # # Example 4: Show off some fancy features including # the use of pre-defined macros. # #################### Executable = foo Universe = standard requirements = OpSys == "LINUX" && Arch =="INTEL" rank = Memory >= 64 image_size = 28000 request_memory = 32 error = err.$(Process) input = in.$(Process) output = out.$(Process) log = foo.log queue 150
A wide variety of job submissions can be specified with extra information to the queue submit command. This flexibility eliminates the need for a job wrapper or Perl script for many submissions.
The form of the queue command defines variables and expands values, identifying a set of jobs. Square brackets identify an optional item.
queue [<int expr>]
queue [<int expr>] [<varname>] in [slice] <list of items>
queue [<int expr>] [<varname>] matching [files | dirs] [slice] <list of items with file globbing>
queue [<int expr>] [<list of varnames>] from [slice] <file name> | <list of items>
All optional items have defaults:
The list of items uses syntax in one of two forms. One form is a comma and/or space separated list; the items are placed on the same line as the queue command. The second form separates items by placing each list item on its own line, and delimits the list with parentheses. The opening parenthesis goes on the same line as the queue command. The closing parenthesis goes on its own line. The queue command specified with the key word from will always use the second form of this syntax. Example 3 below uses this second form of syntax.
The optional slice specifies a subset of the list of items using the Python syntax for a slice. Negative step values are not permitted.
Here are a set of examples.
transfer_input_files = $(filename) arguments = -infile $(filename) queue filename matching files *.datThe use of file globbing expands the list of items to be all files in the current directory that end in .dat. Only files, and not directories are considered due to the specification of files. One job is queued for each file in the list of items. For this example, assume that the three files initial.dat, middle.dat, and ending.dat form the list of items after expansion; macro filename is assigned the value of one of these file names for each job queued. That macro value is then substituted into the arguments and transfer_input_files commands. The queue command expands to
transfer_input_files = initial.dat arguments = -infile initial.dat queue transfer_input_files = middle.dat arguments = -infile middle.dat queue transfer_input_files = ending.dat arguments = -infile ending.dat queue
queue 1 input in A, B, CVariable input is set to each of the 3 items in the list, and one job is queued for each. For this example the queue command expands to
input = A queue input = B queue input = C queue
queue input,arguments from (
file1, -a -b 26
file2, -c -d 92
)
Using the from form of the options,
each of the two variables specified is given a value from the list of items.
For this example the queue command expands to
input = file1 arguments = -a -b 26 queue input = file2 arguments = -c -d 92 queue
Here is an example of a queue command for which the values of these automatic variables are identified.
queue 3 in (A, B)
Externally defined submit commands can be incorporated into the submit description file using the syntax
include : <what-to-include>
The <what-to-include> specification may specify a single file,
where the contents of the file will be incorporated
into the submit description file at the point within the file
where the include is.
Or, <what-to-include> may cause a program to be executed,
where the output of the program is incorporated
into the submit description file.
The specification of <what-to-include> has the bar character
(|) following the name of the program to be executed.
The include key word is case insensitive. There are no requirements for white space characters surrounding the colon character.
Included submit commands may contain further nested include specifications, which are also parsed, evaluated, and incorporated. Levels of nesting on included files are limited, such that infinite nesting is discovered and thwarted, while still permitting nesting.
Consider the example
include : list-infiles.sh |In this example, the bar character at the end of the line causes the script list-infiles.sh to be invoked, and the output of the script is parsed and incorporated into the submit description file. If this bash script contains
echo "transfer_input_files = `ls -m infiles/*.dat`"then the output of this script has specified the set of input files to transfer to the execute host. For example, if directory infiles contains the three files A.dat, B.dat, and C.dat, then the submit command
transfer_input_files = infiles/A.dat, infiles/B.dat, infiles/C.datis incorporated into the submit description file.
Conditional if/else semantics
are available in a limited form.
The syntax:
if <simple condition>
<statement>
. . .
<statement>
else
<statement>
. . .
<statement>
endif
An else key word and statements are not required,
such that simple if semantics are implemented.
The <simple condition> does not permit compound conditions.
It optionally contains the exclamation point character (!)
to represent the not operation,
followed by
defined keyword followed by the name of a
variable.
If the variable is defined, the statement(s) are
incorporated into the expanded input.
If the variable is not defined, the statement(s) are
not incorporated into the expanded input.
As an example,
if defined MY_UNDEFINED_VARIABLE
X = 12
else
X = -1
endif
results in X = -1, when MY_UNDEFINED_VARIABLE is
not yet defined.
version keyword, representing the version number of
of the daemon or tool currently reading this conditional.
This keyword is followed by an HTCondor version number.
That version number can be of the form x.y.z or x.y.
The version of the daemon or tool is compared to the specified
version number.
The comparison operators are
== for equality. Current version 8.2.3 is equal to 8.2.
>= to see if the current version number is greater than or
equal to. Current version 8.2.3 is greater than 8.2.2,
and current version 8.2.3 is greater than or equal to 8.2.
<= to see if the current version number is less than or
equal to. Current version 8.2.0 is less than 8.2.2,
and current version 8.2.3 is less than or equal to 8.2.
if version >= 8.1.6
DO_X = True
else
DO_Y = True
endif
results in defining DO_X as True if the current
version of the daemon or tool reading this if statement is 8.1.6
or a more recent version.
True or yes or the value 1.
The statement(s) are incorporated.
False or no or the value 0
The statement(s) are not incorporated.
$(<variable>) may be used where the
immediately evaluated value is a simple boolean value.
A value that evaluates to the empty string is considered False,
otherwise a value that does not evaluate to a simple boolean value
is a syntax error.
The syntax
if <simple condition>
<statement>
. . .
<statement>
elif <simple condition>
<statement>
. . .
<statement>
endif
is the same as syntax
if <simple condition>
<statement>
. . .
<statement>
else
if <simple condition>
<statement>
. . .
<statement>
endif
endif
Here is an example use of a conditional in the submit description file. A portion of the sample.sub submit description file uses the if/else syntax to define command line arguments in one of two ways:
if defined X
arguments = -n $(X)
else
arguments = -n 1 -debug
endif
Submit variable X is defined on the condor_submit command line with
condor_submit X=3 sample.subThis command line incorporates the submit command X = 3 into the submission before parsing the submit description file. For this submission, the command line arguments of the submitted job become
-n 3
If the job were instead submitted with the command line
condor_submit sample.subthen the command line arguments of the submitted job become
-n 1 -debug
A set of predefined functions increase flexibility. Both submit description files and configuration files are read using the same parser, so these functions may be used in both submit description files and configuration files.
Case is significant in the function's name, so use the same letter case as given in these definitions.
A = $ENV(HOME)binds A to the value of the HOME environment variable.
"%d" is used
as the format specifier.
$RANDOM_CHOICE(0,1,2,3,4,5,6,7,8)
min and max, inclusive,
is selected.
The optional step parameter
controls the stride within the range, and it defaults to the value 1.
For example, to randomly chose an even integer in the range 0-8 (inclusive):
$RANDOM_INTEGER(0, 8, 2)
"%16G" is used
as a format specifier.
name and returns a substring of it.
The first character of the string is at index 0.
The first character of the substring is at index start-index.
If the optional length is not specified,
then the substring includes characters up to the end of the string.
A negative value of start-index works back from the end of the
string.
A negative value of length eliminates use of characters
from the end of the string.
Here are some examples that all assume
Name = abcdef
Here are example uses of the function macros in a submit description file. Note that these are not complete submit description files, but only the portions that promote understanding of use cases of the function macros.
$(Process) are desired.
MyIndex = $(Process) + 1 initial_dir = run-$INT(MyIndex, %04d)Assuming that there are three jobs queued, such that
$(Process) becomes 0, 1, and 2,
initial_dir will evaluate to the directories
run-0001, run-0002, and run-0003.
Values = $(Process) * 10 Extension = $INT(Values, %03d) input = X.$(Extension)Assuming that there are four jobs queued, such that
$(Process) becomes 0, 1, 2, and 3,
Extension will evaluate to 000, 010, 020, and 030,
leading to files defined for input of
X.000, X.010, X.020, and X.030.
arguments = $Fnx(FILE)
transfer_input_files = $(FILE)
queue FILE MATCHING (
samplerun/*.dat
)
Assume that two files that end in .dat, A.dat and B.dat,
are within the directory samplerun.
Macro FILE expands to samplerun/A.dat and samplerun/B.dat
for the two jobs queued.
The input files transferred are
samplerun/A.dat and samplerun/B.dat
on the submit host.
The $Fnx() function macro expands to the complete file name
with any leading directory specification stripped,
such that the command line argument for one of the
jobs will be A.dat and the command line argument for the
other job will be B.dat.
The requirements and rank commands in the submit description file are powerful and flexible. Using them effectively requires care, and this section presents those details.
Both requirements and rank need to be specified
as valid HTCondor ClassAd expressions, however, default values are set by the
condor_submit program if these are not defined in the submit description file.
From the condor_submit manual page and the above examples, you see
that writing ClassAd expressions is intuitive, especially if you
are familiar with the programming language C. There are some
pretty nifty expressions you can write with ClassAds.
A complete description of ClassAds and their expressions
can be found in section 4.1 on
page .
All of the commands in the submit description file are case insensitive, except for the ClassAd attribute string values. ClassAd attribute names are case insensitive, but ClassAd string values are case preserving.
Note that the comparison operators
(<, >, <=, >=, and ==)
compare strings
case insensitively. The special comparison operators
=?= and =!=
compare strings case sensitively.
A requirements or rank command in
the submit description file may utilize attributes
that appear in a machine or a job ClassAd.
Within the submit description file (for a job) the
prefix MY. (on a ClassAd attribute name)
causes a reference to the job ClassAd attribute,
and the prefix TARGET. causes a reference to
a potential machine or matched machine ClassAd attribute.
The condor_status command displays statistics about machines within the pool. The -l option displays the machine ClassAd attributes for all machines in the HTCondor pool. The job ClassAds, if there are jobs in the queue, can be seen with the condor_q -l command. This shows all the defined attributes for current jobs in the queue.
A list of defined ClassAd attributes for job ClassAds
is given in the unnumbered Appendix on
page .
A list of defined ClassAd attributes for machine ClassAds
is given in the unnumbered Appendix on
page .
When considering the match between a job and a machine, rank is used to choose a match from among all machines that satisfy the job's requirements and are available to the user, after accounting for the user's priority and the machine's rank of the job. The rank expressions, simple or complex, define a numerical value that expresses preferences.
The job's Rank expression evaluates to one of three values. It can be UNDEFINED, ERROR, or a floating point value. If Rank evaluates to a floating point value, the best match will be the one with the largest, positive value. If no Rank is given in the submit description file, then HTCondor substitutes a default value of 0.0 when considering machines to match. If the job's Rank of a given machine evaluates to UNDEFINED or ERROR, this same value of 0.0 is used. Therefore, the machine is still considered for a match, but has no ranking above any other.
A boolean expression evaluates to the numerical value of 1.0 if true, and 0.0 if false.
The following Rank expressions provide examples to follow.
For a job that desires the machine with the most available memory:
Rank = memory
For a job that prefers to run on a friend's machine on Saturdays and Sundays:
Rank = ( (clockday == 0) || (clockday == 6) )
&& (machine == "friend.cs.wisc.edu")
For a job that prefers to run on one of three specific machines:
Rank = (machine == "friend1.cs.wisc.edu") ||
(machine == "friend2.cs.wisc.edu") ||
(machine == "friend3.cs.wisc.edu")
For a job that wants the machine with the best floating point performance (on Linpack benchmarks):
Rank = kflopsThis particular example highlights a difficulty with Rank expression evaluation as currently defined. While all machines have floating point processing ability, not all machines will have the kflops attribute defined. For machines where this attribute is not defined, Rank will evaluate to the value UNDEFINED, and HTCondor will use a default rank of the machine of 0.0. The Rank attribute will only rank machines where the attribute is defined. Therefore, the machine with the highest floating point performance may not be the one given the highest rank.
So, it is wise when writing a Rank expression to check if the expression's evaluation will lead to the expected resulting ranking of machines. This can be accomplished using the condor_status command with the -constraint argument. This allows the user to see a list of machines that fit a constraint. To see which machines in the pool have kflops defined, use
condor_status -constraint kflopsAlternatively, to see a list of machines where kflops is not defined, use
condor_status -constraint "kflops=?=undefined"
For a job that prefers specific machines in a specific order:
Rank = ((machine == "friend1.cs.wisc.edu")*3) +
((machine == "friend2.cs.wisc.edu")*2) +
(machine == "friend3.cs.wisc.edu")
If the machine being ranked is friend1.cs.wisc.edu, then the
expression
(machine == "friend1.cs.wisc.edu")is true, and gives the value 1.0. The expressions
(machine == "friend2.cs.wisc.edu")and
(machine == "friend3.cs.wisc.edu")are false, and give the value 0.0. Therefore, Rank evaluates to the value 3.0. In this way, machine friend1.cs.wisc.edu is ranked higher than machine friend2.cs.wisc.edu, machine friend2.cs.wisc.edu is ranked higher than machine friend3.cs.wisc.edu, and all three of these machines are ranked higher than others.
If vanilla, java, or parallel universe jobs are submitted without using the File Transfer mechanism, HTCondor must use a shared file system to access input and output files. In this case, the job must be able to access the data files from any machine on which it could potentially run.
As an example, suppose a job is submitted from blackbird.cs.wisc.edu, and the job requires a particular data file called /u/p/s/psilord/data.txt. If the job were to run on cardinal.cs.wisc.edu, the file /u/p/s/psilord/data.txt must be available through either NFS or AFS for the job to run correctly.
HTCondor allows users to ensure their jobs have access to the right shared files by using the FileSystemDomain and UidDomain machine ClassAd attributes. These attributes specify which machines have access to the same shared file systems. All machines that mount the same shared directories in the same locations are considered to belong to the same file system domain. Similarly, all machines that share the same user information (in particular, the same UID, which is important for file systems like NFS) are considered part of the same UID domain.
The default configuration for HTCondor places each machine in its own UID domain and file system domain, using the full host name of the machine as the name of the domains. So, if a pool does have access to a shared file system, the pool administrator must correctly configure HTCondor such that all the machines mounting the same files have the same FileSystemDomain configuration. Similarly, all machines that share common user information must be configured to have the same UidDomain configuration.
When a job relies on a shared file system, HTCondor uses the requirements expression to ensure that the job runs on a machine in the correct UidDomain and FileSystemDomain. In this case, the default requirements expression specifies that the job must run on a machine with the same UidDomain and FileSystemDomain as the machine from which the job is submitted. This default is almost always correct. However, in a pool spanning multiple UidDomains and/or FileSystemDomains, the user may need to specify a different requirements expression to have the job run on the correct machines.
For example, imagine a pool made up of both desktop workstations and a dedicated compute cluster. Most of the pool, including the compute cluster, has access to a shared file system, but some of the desktop machines do not. In this case, the administrators would probably define the FileSystemDomain to be cs.wisc.edu for all the machines that mounted the shared files, and to the full host name for each machine that did not. An example is jimi.cs.wisc.edu.
In this example, a user wants to submit vanilla universe jobs from her own desktop machine (jimi.cs.wisc.edu) which does not mount the shared file system (and is therefore in its own file system domain, in its own world). But, she wants the jobs to be able to run on more than just her own machine (in particular, the compute cluster), so she puts the program and input files onto the shared file system. When she submits the jobs, she needs to tell HTCondor to send them to machines that have access to that shared data, so she specifies a different requirements expression than the default:
Requirements = TARGET.UidDomain == "cs.wisc.edu" && \
TARGET.FileSystemDomain == "cs.wisc.edu"
WARNING: If there is no shared file system, or the HTCondor pool administrator does not configure the FileSystemDomain setting correctly (the default is that each machine in a pool is in its own file system and UID domain), a user submits a job that cannot use remote system calls (for example, a vanilla universe job), and the user does not enable HTCondor's File Transfer mechanism, the job will only run on the machine from which it was submitted.
HTCondor works well without a shared file system. The HTCondor file transfer mechanism permits the user to select which files are transferred and under which circumstances. HTCondor can transfer any files needed by a job from the machine where the job was submitted into a remote scratch directory on the machine where the job is to be executed. HTCondor executes the job and transfers output back to the submitting machine. The user specifies which files and directories to transfer, and at what point the output files should be copied back to the submitting machine. This specification is done within the job's submit description file.
To enable the file transfer mechanism, place two commands in the job's submit description file: should_transfer_files and when_to_transfer_output. By default, they will be:
should_transfer_files = IF_NEEDED when_to_transfer_output = ON_EXIT
Setting the should_transfer_files command explicitly enables or disables the file transfer mechanism. The command takes on one of three possible values: