PyCondor API¶
The main functionality of PyCondor is contained within the Job
and Dagman
objects.
Job¶
-
class
pycondor.
Job
(name, executable, error=None, log=None, output=None, submit=None, request_memory=None, request_disk=None, request_cpus=None, getenv=None, universe=None, initialdir=None, notification=None, requirements=None, queue=None, extra_lines=None, dag=None, arguments=None, retry=None, verbose=0)[source]¶ Job object consisting of an executable to be run, potentially with a series of different command-line arguments.
Note that the
submit
,error
,log
, andoutput
parameters can be explicitly given or configured by settingPYCONDOR_SUBMIT_DIR
,PYCONDOR_ERROR_DIR
,PYCONDOR_LOG_DIR
, andPYCONDOR_OUTPUT_DIR
environment variables. An explicitly given value will be used over an environment variable, while an environment variable will be used over a default value.Parameters: - name : str
Name of the Job instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Job.
- executable : str
Path to corresponding executable for Job.
- error : str or None, optional
Path to directory where condor Job error files will be written (default is None, will not be included in Job submit file).
- log : str or None, optional
Path to directory where condor Job log files will be written (default is None, will not be included in Job submit file).
- output : str or None, optional
Path to directory where condor Job output files will be written (default is None, will not be included in Job submit file).
- submit : str, optional
Path to directory where condor Job submit files will be written (defaults to the directory was the Job was submitted from).
- request_memory : str or None, optional
Memory request to be included in submit file.
- request_disk : str or None, optional
Disk request to be included in submit file.
- request_cpus : int or None, optional
Number of CPUs to request in submit file.
New in version 0.1.0.
- getenv : bool or None, optional
Whether or not to use the current environment settings when running the job (default is None).
- universe : str or None, optional
Universe execution environment to be specified in submit file (default is None).
- initialdir : str or None, optional
Initial directory for relative paths (defaults to the directory was the job was submitted from).
- notification : str or None, optional
E-mail notification preference (default is None).
- requirements : str or None, optional
Additional requirements to be included in ClassAd.
- queue : int or None, optional
Integer specifying how many times you would like this job to run.
- extra_lines : list or None, optional
List of additional lines to be added to submit file.
- dag : Dagman, optional
If specified, Job will be added to dag (default is None).
- arguments : str or iterable, optional
Arguments with which to initialize the Job list of arguments (default is None).
- retry : int or None, optional
Option to specify the number of retries for all Job arguments. This can be superseded for arguments added via the add_arg() method. Note: this feature is only available to Jobs that are submitted via a Dagman (default is None; no retries).
- verbose : int, optional
Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).
Examples
>>> import pycondor >>> job = pycondor.Job('myjob', 'myscript.py') >>> job.build_submit()
Attributes: - args : list
List of arguments for this Job instance.
- parents : list
Only set when included in a Dagman. List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Job is submitted to HTCondor.
- children : list
Only set when included in a Dagman. List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Job has completed.
Methods
add_arg
(arg[, name, retry])Add argument to Job add_args
(args)Adds multiple arguments to Job add_child
(node)Adds node to children list add_children
(nodes)Adds nodes to the children list add_parent
(node)Adds node to parents list add_parents
(nodes)Adds nodes to the parents list build
([makedirs, fancyname])Build and saves the submit file for Job build_submit
(**kwargs)Calls build and submit sequentially haschildren
()Checks for any children nodes hasparents
()Checks for any parent nodes submit_job
(**kwargs)Submits Job to condor -
add_arg
(arg, name=None, retry=None)[source]¶ Add argument to Job
Parameters: - arg : str
Argument to append to Job args list.
- name : str or None, optional
Option to specify a name related to this argument. If a name is specified, then a separate set of log, output, and error files will be generated for this particular argument (default is
None
).New in version 0.1.2.
- retry : int or None, optional
Option to specify the number of times to retry this node. Default number of retries is 0. Note: this feature is only available to Jobs that are submitted via a Dagman.
New in version 0.1.2.
Returns: - self : object
Returns self.
-
add_args
(args)[source]¶ Adds multiple arguments to Job
Parameters: - args : iterable
Iterable of arguments to append to the arguments list
Returns: - self : object
Returns self.
-
add_child
(node)¶ Adds node to children list
Parameters: - node : BaseNode
Job or Dagman to append to the children list.
Returns: - self : object
Returns self.
-
add_children
(nodes)¶ Adds nodes to the children list
Parameters: - nodes : list or tuple
List of nodes to append to the children list
Returns: - self : object
Returns self.
-
add_parent
(node)¶ Adds node to parents list
Parameters: - node : BaseNode
Job or Dagman to append to the parents list.
Returns: - self : object
Returns self.
-
add_parents
(nodes)¶ Adds nodes to the parents list
Parameters: - nodes : list or tuple
List of nodes to append to the parents list
Returns: - self : object
Returns self.
-
build
(makedirs=True, fancyname=True)[source]¶ Build and saves the submit file for Job
Parameters: - makedirs : bool, optional
If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is
True
).- fancyname : bool, optional
Appends the date and unique id number to error, log, output, and submit files. For example, instead of
jobname.submit
the submit file becomesjobname_YYYYMMD_id
. This is useful when running several Jobs of the same name (default isTrue
).
Returns: - self : object
Returns self.
-
build_submit
(**kwargs)[source]¶ Calls build and submit sequentially
Parameters: - makedirs : bool, optional
If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is
True
).- fancyname : bool, optional
Appends the date and unique id number to error, log, output, and submit files. For example, instead of
jobname.submit
the submit file becomesjobname_YYYYMMD_id
. This is useful when running several Jobs of the same name (default isTrue
).- submit_options : str, optional
Options to be passed to
condor_submit
for this Job (see the condor_submit documentation for possible options).
Returns: - self : object
Returns self.
-
haschildren
()¶ Checks for any children nodes
Returns: - bool
Returns whether or not this node has any child nodes.
-
hasparents
()¶ Checks for any parent nodes
Returns: - bool
Returns whether or not this node has any parent nodes.
-
submit_job
(**kwargs)[source]¶ Submits Job to condor
Parameters: - submit_options : str, optional
Options to be passed to
condor_submit
for this Job (see the condor_submit documentation for possible options).
Returns: - self : object
Returns self.
Examples
>>> import pycondor >>> job = pycondor.Job('myjob', 'myscript.py') >>> job.build() >>> job.submit_job(submit_options='-maxjobs 1000 -interactive')
Dagman¶
-
class
pycondor.
Dagman
(name, submit=None, extra_lines=None, dag=None, verbose=0)[source]¶ Dagman object consisting of a series of Jobs and sub-Dagmans to manage.
Note that the
submit
parameter can be explicitly given or configured by setting thePYCONDOR_SUBMIT_DIR
environment variable. An explicitly given value forsubmit
will be used over the environment variable, while the environment variable will be used over a default value.Parameters: - name : str
Name of the Dagman instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Dagman.
- submit : str
Path to directory where condor dagman submit files will be written (defaults to the directory was the Dagman was submitted from).
- extra_lines : list or None, optional
List of additional lines to be added to submit file.
New in version 0.1.1.
- dag : Dagman, optional
If specified, Dagman will be added to dag as a subdag (default is None).
- verbose : int, optional
Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).
Attributes: - jobs : list
The list of jobs for this Dagman instance to manage.
- parents : list
List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Dagman is submitted to HTCondor.
- children : list
List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Dagman has completed.
Methods
add_child
(node)Adds node to children list add_children
(nodes)Adds nodes to the children list add_job
(job)Add job to Dagman add_parent
(node)Adds node to parents list add_parents
(nodes)Adds nodes to the parents list add_subdag
(dag)Add dag to Dagman build
([makedirs, fancyname])Build and saves the submit file for Dagman build_submit
(**kwargs)Calls build and submit sequentially haschildren
()Checks for any children nodes hasparents
()Checks for any parent nodes submit_dag
(**kwargs)Submits Dagman to condor visualize
([filename])Visualize Dagman graph -
add_child
(node)¶ Adds node to children list
Parameters: - node : BaseNode
Job or Dagman to append to the children list.
Returns: - self : object
Returns self.
-
add_children
(nodes)¶ Adds nodes to the children list
Parameters: - nodes : list or tuple
List of nodes to append to the children list
Returns: - self : object
Returns self.
-
add_job
(job)[source]¶ Add job to Dagman
Parameters: - job : Job
Job to append to Dagman jobs list.
Returns: - self : object
Returns self.
-
add_parent
(node)¶ Adds node to parents list
Parameters: - node : BaseNode
Job or Dagman to append to the parents list.
Returns: - self : object
Returns self.
-
add_parents
(nodes)¶ Adds nodes to the parents list
Parameters: - nodes : list or tuple
List of nodes to append to the parents list
Returns: - self : object
Returns self.
-
add_subdag
(dag)[source]¶ Add dag to Dagman
Parameters: - dag : Dagman
Subdag to append to Dagman jobs list.
Returns: - self : object
Returns self.
-
build
(makedirs=True, fancyname=True)[source]¶ Build and saves the submit file for Dagman
Parameters: - makedirs : bool, optional
If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is
True
).- fancyname : bool, optional
Appends the date and unique id number to error, log, output, and submit files. For example, instead of
dagname.submit
the submit file becomesdagname_YYYYMMD_id
. This is useful when running several Dags/Jobs of the same name (default isTrue
).
Returns: - self : object
Returns self.
-
build_submit
(**kwargs)[source]¶ Calls build and submit sequentially
Parameters: - makedirs : bool, optional
If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is
True
).- fancyname : bool, optional
Appends the date and unique id number to error, log, output, and submit files. For example, instead of
dagname.submit
the submit file becomesdagname_YYYYMMD_id
. This is useful when running several Dags/Jobs of the same name (default isTrue
).- submit_options : str, optional
Options to be passed to
condor_submit_dag
for this Dagman (see the condor_submit_dag documentation for possible options).
Returns: - self : object
Returns self.
-
haschildren
()¶ Checks for any children nodes
Returns: - bool
Returns whether or not this node has any child nodes.
-
hasparents
()¶ Checks for any parent nodes
Returns: - bool
Returns whether or not this node has any parent nodes.
-
submit_dag
(**kwargs)[source]¶ Submits Dagman to condor
Parameters: - submit_options : str, optional
Options to be passed to
condor_submit_dag
for this Dagman (see the condor_submit_dag documentation for possible options).
Returns: - self : object
Returns self.