PyCondor API¶

The main functionality of PyCondor is contained within the Job and Dagman objects.

Job¶

class pycondor.Job(name, executable, error=None, log=None, output=None, submit=None, request_memory=None, request_disk=None, request_cpus=None, getenv=None, universe=None, initialdir=None, notification=None, requirements=None, queue=None, extra_lines=None, dag=None, arguments=None, retry=None, verbose=0)[source]¶

Job object consisting of an executable to be run, potentially with a series of different command-line arguments.

Note that the submit, error, log, and output parameters can be explicitly given or configured by setting PYCONDOR_SUBMIT_DIR, PYCONDOR_ERROR_DIR, PYCONDOR_LOG_DIR, and PYCONDOR_OUTPUT_DIR environment variables. An explicitly given value will be used over an environment variable, while an environment variable will be used over a default value.

Parameters:

name : str: Name of the Job instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Job.
executable : str: Path to corresponding executable for Job.
error : str or None, optional: Path to directory where condor Job error files will be written (default is None, will not be included in Job submit file).
log : str or None, optional: Path to directory where condor Job log files will be written (default is None, will not be included in Job submit file).
output : str or None, optional: Path to directory where condor Job output files will be written (default is None, will not be included in Job submit file).
submit : str, optional: Path to directory where condor Job submit files will be written (defaults to the directory was the Job was submitted from).
request_memory : str or None, optional: Memory request to be included in submit file.
request_disk : str or None, optional: Disk request to be included in submit file.
request_cpus : int or None, optional: Number of CPUs to request in submit file.

New in version 0.1.0.
getenv : bool or None, optional: Whether or not to use the current environment settings when running the job (default is None).
universe : str or None, optional: Universe execution environment to be specified in submit file (default is None).
initialdir : str or None, optional: Initial directory for relative paths (defaults to the directory was the job was submitted from).
notification : str or None, optional: E-mail notification preference (default is None).
requirements : str or None, optional: Additional requirements to be included in ClassAd.
queue : int or None, optional: Integer specifying how many times you would like this job to run.
extra_lines : list or None, optional: List of additional lines to be added to submit file.
dag : Dagman, optional: If specified, Job will be added to dag (default is None).
arguments : str or iterable, optional: Arguments with which to initialize the Job list of arguments (default is None).
retry : int or None, optional: Option to specify the number of retries for all Job arguments. This can be superseded for arguments added via the add_arg() method. Note: this feature is only available to Jobs that are submitted via a Dagman (default is None; no retries).
verbose : int, optional: Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build_submit()

Attributes:

args : list: List of arguments for this Job instance.
parents : list: Only set when included in a Dagman. List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Job is submitted to HTCondor.
children : list: Only set when included in a Dagman. List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Job has completed.

Methods

`add_arg`(arg[, name, retry])	Add argument to Job
`add_args`(args)	Adds multiple arguments to Job
`add_child`(node)	Adds node to children list
`add_children`(nodes)	Adds nodes to the children list
`add_parent`(node)	Adds node to parents list
`add_parents`(nodes)	Adds nodes to the parents list
`build`([makedirs, fancyname])	Build and saves the submit file for Job
`build_submit`(**kwargs)	Calls build and submit sequentially
`haschildren`()	Checks for any children nodes
`hasparents`()	Checks for any parent nodes
`submit_job`(**kwargs)	Submits Job to condor

add_arg(arg, name=None, retry=None)[source]¶

Add argument to Job

Parameters:

arg : str: Argument to append to Job args list.
name : str or None, optional: Option to specify a name related to this argument. If a name is specified, then a separate set of log, output, and error files will be generated for this particular argument (default is None).

New in version 0.1.2.
retry : int or None, optional: Option to specify the number of times to retry this node. Default number of retries is 0. Note: this feature is only available to Jobs that are submitted via a Dagman.

New in version 0.1.2.

Returns:

self : object: Returns self.

add_args(args)[source]¶

Adds multiple arguments to Job

Parameters:	args : iterable Iterable of arguments to append to the arguments list
Returns:	self : object Returns self.

add_child(node)¶

Adds node to children list

Parameters:	node : BaseNode Job or Dagman to append to the children list.
Returns:	self : object Returns self.

add_children(nodes)¶

Adds nodes to the children list

Parameters:	nodes : list or tuple List of nodes to append to the children list
Returns:	self : object Returns self.

add_parent(node)¶

Adds node to parents list

Parameters:	node : BaseNode Job or Dagman to append to the parents list.
Returns:	self : object Returns self.

add_parents(nodes)¶

Adds nodes to the parents list

Parameters:	nodes : list or tuple List of nodes to append to the parents list
Returns:	self : object Returns self.

build(makedirs=True, fancyname=True)[source]¶

Build and saves the submit file for Job

Parameters:

makedirs : bool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancyname : bool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).

Returns:

self : object: Returns self.

build_submit(**kwargs)[source]¶

Calls build and submit sequentially

Parameters:

makedirs : bool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancyname : bool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).
submit_options : str, optional: Options to be passed to condor_submit for this Job (see the condor_submit documentation for possible options).

Returns:

self : object: Returns self.

haschildren()¶

Checks for any children nodes

Returns:	bool Returns whether or not this node has any child nodes.

hasparents()¶

Checks for any parent nodes

Returns:	bool Returns whether or not this node has any parent nodes.

submit_job(**kwargs)[source]¶

Submits Job to condor

Parameters:	submit_options : str, optional Options to be passed to `condor_submit` for this Job (see the condor_submit documentation for possible options).
Returns:	self : object Returns self.

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build()
>>> job.submit_job(submit_options='-maxjobs 1000 -interactive')

Dagman¶

class pycondor.Dagman(name, submit=None, extra_lines=None, dag=None, verbose=0)[source]¶

Dagman object consisting of a series of Jobs and sub-Dagmans to manage.

Note that the submit parameter can be explicitly given or configured by setting the PYCONDOR_SUBMIT_DIR environment variable. An explicitly given value for submit will be used over the environment variable, while the environment variable will be used over a default value.

Parameters:

name : str: Name of the Dagman instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Dagman.
submit : str: Path to directory where condor dagman submit files will be written (defaults to the directory was the Dagman was submitted from).
extra_lines : list or None, optional: List of additional lines to be added to submit file.

New in version 0.1.1.
dag : Dagman, optional: If specified, Dagman will be added to dag as a subdag (default is None).
verbose : int, optional: Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Attributes:

jobs : list: The list of jobs for this Dagman instance to manage.
parents : list: List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Dagman is submitted to HTCondor.
children : list: List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Dagman has completed.

Methods

`add_child`(node)	Adds node to children list
`add_children`(nodes)	Adds nodes to the children list
`add_job`(job)	Add job to Dagman
`add_parent`(node)	Adds node to parents list
`add_parents`(nodes)	Adds nodes to the parents list
`add_subdag`(dag)	Add dag to Dagman
`build`([makedirs, fancyname])	Build and saves the submit file for Dagman
`build_submit`(**kwargs)	Calls build and submit sequentially
`haschildren`()	Checks for any children nodes
`hasparents`()	Checks for any parent nodes
`submit_dag`(**kwargs)	Submits Dagman to condor
`visualize`([filename])	Visualize Dagman graph

add_child(node)¶

Adds node to children list

Parameters:	node : BaseNode Job or Dagman to append to the children list.
Returns:	self : object Returns self.

add_children(nodes)¶

Adds nodes to the children list

Parameters:	nodes : list or tuple List of nodes to append to the children list
Returns:	self : object Returns self.

add_job(job)[source]¶

Add job to Dagman

Parameters:	job : Job Job to append to Dagman jobs list.
Returns:	self : object Returns self.

add_parent(node)¶

Adds node to parents list

Parameters:	node : BaseNode Job or Dagman to append to the parents list.
Returns:	self : object Returns self.

add_parents(nodes)¶

Adds nodes to the parents list

Parameters:	nodes : list or tuple List of nodes to append to the parents list
Returns:	self : object Returns self.

add_subdag(dag)[source]¶

Add dag to Dagman

Parameters:	dag : Dagman Subdag to append to Dagman jobs list.
Returns:	self : object Returns self.

build(makedirs=True, fancyname=True)[source]¶

Build and saves the submit file for Dagman

Parameters:

makedirs : bool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancyname : bool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).

Returns:

self : object: Returns self.

build_submit(**kwargs)[source]¶

Calls build and submit sequentially

Parameters:

makedirs : bool, optional: If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).
fancyname : bool, optional: Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).
submit_options : str, optional: Options to be passed to condor_submit_dag for this Dagman (see the condor_submit_dag documentation for possible options).

Returns:

self : object: Returns self.

haschildren()¶

Checks for any children nodes

Returns:	bool Returns whether or not this node has any child nodes.

hasparents()¶

Checks for any parent nodes

Returns:	bool Returns whether or not this node has any parent nodes.

submit_dag(**kwargs)[source]¶

Submits Dagman to condor

Parameters:	submit_options : str, optional Options to be passed to `condor_submit_dag` for this Dagman (see the condor_submit_dag documentation for possible options).
Returns:	self : object Returns self.

visualize(filename=None)[source]¶

Visualize Dagman graph

Parameters:	filename : str or None, optional File to save graph diagram to. If `None` then no file is saved. Valid file extensions are ‘png’, ‘pdf’, ‘dot’, ‘svg’, ‘jpeg’, ‘jpg’.