PyCondor API

The main functionality of PyCondor is contained within the Job and Dagman objects.

Job

class pycondor.Job(name, executable, error=None, log=None, output=None, submit=None, request_memory=None, request_disk=None, request_cpus=None, getenv=None, universe=None, initialdir=None, notification=None, requirements=None, queue=None, extra_lines=None, dag=None, arguments=None, retry=None, verbose=0)[source]

Job object consisting of an executable to be run, potentially with a series of different command-line arguments.

Note that the submit, error, log, and output parameters can be explicitly given or configured by setting PYCONDOR_SUBMIT_DIR, PYCONDOR_ERROR_DIR, PYCONDOR_LOG_DIR, and PYCONDOR_OUTPUT_DIR environment variables. An explicitly given value will be used over an environment variable, while an environment variable will be used over a default value.

Parameters:
name : str

Name of the Job instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Job.

executable : str

Path to corresponding executable for Job.

error : str or None, optional

Path to directory where condor Job error files will be written (default is None, will not be included in Job submit file).

log : str or None, optional

Path to directory where condor Job log files will be written (default is None, will not be included in Job submit file).

output : str or None, optional

Path to directory where condor Job output files will be written (default is None, will not be included in Job submit file).

submit : str, optional

Path to directory where condor Job submit files will be written (defaults to the directory was the Job was submitted from).

request_memory : str or None, optional

Memory request to be included in submit file.

request_disk : str or None, optional

Disk request to be included in submit file.

request_cpus : int or None, optional

Number of CPUs to request in submit file.

New in version 0.1.0.

getenv : bool or None, optional

Whether or not to use the current environment settings when running the job (default is None).

universe : str or None, optional

Universe execution environment to be specified in submit file (default is None).

initialdir : str or None, optional

Initial directory for relative paths (defaults to the directory was the job was submitted from).

notification : str or None, optional

E-mail notification preference (default is None).

requirements : str or None, optional

Additional requirements to be included in ClassAd.

queue : int or None, optional

Integer specifying how many times you would like this job to run.

extra_lines : list or None, optional

List of additional lines to be added to submit file.

dag : Dagman, optional

If specified, Job will be added to dag (default is None).

arguments : str or iterable, optional

Arguments with which to initialize the Job list of arguments (default is None).

retry : int or None, optional

Option to specify the number of retries for all Job arguments. This can be superseded for arguments added via the add_arg() method. Note: this feature is only available to Jobs that are submitted via a Dagman (default is None; no retries).

verbose : int, optional

Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build_submit()
Attributes:
args : list

List of arguments for this Job instance.

parents : list

Only set when included in a Dagman. List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Job is submitted to HTCondor.

children : list

Only set when included in a Dagman. List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Job has completed.

Methods

add_arg(arg[, name, retry]) Add argument to Job
add_args(args) Adds multiple arguments to Job
add_child(node) Adds node to children list
add_children(nodes) Adds nodes to the children list
add_parent(node) Adds node to parents list
add_parents(nodes) Adds nodes to the parents list
build([makedirs, fancyname]) Build and saves the submit file for Job
build_submit(**kwargs) Calls build and submit sequentially
haschildren() Checks for any children nodes
hasparents() Checks for any parent nodes
submit_job(**kwargs) Submits Job to condor
add_arg(arg, name=None, retry=None)[source]

Add argument to Job

Parameters:
arg : str

Argument to append to Job args list.

name : str or None, optional

Option to specify a name related to this argument. If a name is specified, then a separate set of log, output, and error files will be generated for this particular argument (default is None).

New in version 0.1.2.

retry : int or None, optional

Option to specify the number of times to retry this node. Default number of retries is 0. Note: this feature is only available to Jobs that are submitted via a Dagman.

New in version 0.1.2.

Returns:
self : object

Returns self.

add_args(args)[source]

Adds multiple arguments to Job

Parameters:
args : iterable

Iterable of arguments to append to the arguments list

Returns:
self : object

Returns self.

add_child(node)

Adds node to children list

Parameters:
node : BaseNode

Job or Dagman to append to the children list.

Returns:
self : object

Returns self.

add_children(nodes)

Adds nodes to the children list

Parameters:
nodes : list or tuple

List of nodes to append to the children list

Returns:
self : object

Returns self.

add_parent(node)

Adds node to parents list

Parameters:
node : BaseNode

Job or Dagman to append to the parents list.

Returns:
self : object

Returns self.

add_parents(nodes)

Adds nodes to the parents list

Parameters:
nodes : list or tuple

List of nodes to append to the parents list

Returns:
self : object

Returns self.

build(makedirs=True, fancyname=True)[source]

Build and saves the submit file for Job

Parameters:
makedirs : bool, optional

If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).

fancyname : bool, optional

Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).

Returns:
self : object

Returns self.

build_submit(**kwargs)[source]

Calls build and submit sequentially

Parameters:
makedirs : bool, optional

If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).

fancyname : bool, optional

Appends the date and unique id number to error, log, output, and submit files. For example, instead of jobname.submit the submit file becomes jobname_YYYYMMD_id. This is useful when running several Jobs of the same name (default is True).

submit_options : str, optional

Options to be passed to condor_submit for this Job (see the condor_submit documentation for possible options).

Returns:
self : object

Returns self.

haschildren()

Checks for any children nodes

Returns:
bool

Returns whether or not this node has any child nodes.

hasparents()

Checks for any parent nodes

Returns:
bool

Returns whether or not this node has any parent nodes.

submit_job(**kwargs)[source]

Submits Job to condor

Parameters:
submit_options : str, optional

Options to be passed to condor_submit for this Job (see the condor_submit documentation for possible options).

Returns:
self : object

Returns self.

Examples

>>> import pycondor
>>> job = pycondor.Job('myjob', 'myscript.py')
>>> job.build()
>>> job.submit_job(submit_options='-maxjobs 1000 -interactive')

Dagman

class pycondor.Dagman(name, submit=None, extra_lines=None, dag=None, verbose=0)[source]

Dagman object consisting of a series of Jobs and sub-Dagmans to manage.

Note that the submit parameter can be explicitly given or configured by setting the PYCONDOR_SUBMIT_DIR environment variable. An explicitly given value for submit will be used over the environment variable, while the environment variable will be used over a default value.

Parameters:
name : str

Name of the Dagman instance. This will also be the name of the corresponding error, log, output, and submit files associated with this Dagman.

submit : str

Path to directory where condor dagman submit files will be written (defaults to the directory was the Dagman was submitted from).

extra_lines : list or None, optional

List of additional lines to be added to submit file.

New in version 0.1.1.

dag : Dagman, optional

If specified, Dagman will be added to dag as a subdag (default is None).

verbose : int, optional

Level of logging verbosity option are 0-warning, 1-info, 2-debugging (default is 0).

Attributes:
jobs : list

The list of jobs for this Dagman instance to manage.

parents : list

List of parent Jobs and Dagmans. Ensures that Jobs and Dagmans in the parents list will complete before this Dagman is submitted to HTCondor.

children : list

List of child Jobs and Dagmans. Ensures that Jobs and Dagmans in the children list will be submitted only after this Dagman has completed.

Methods

add_child(node) Adds node to children list
add_children(nodes) Adds nodes to the children list
add_job(job) Add job to Dagman
add_parent(node) Adds node to parents list
add_parents(nodes) Adds nodes to the parents list
add_subdag(dag) Add dag to Dagman
build([makedirs, fancyname]) Build and saves the submit file for Dagman
build_submit(**kwargs) Calls build and submit sequentially
haschildren() Checks for any children nodes
hasparents() Checks for any parent nodes
submit_dag(**kwargs) Submits Dagman to condor
visualize([filename]) Visualize Dagman graph
add_child(node)

Adds node to children list

Parameters:
node : BaseNode

Job or Dagman to append to the children list.

Returns:
self : object

Returns self.

add_children(nodes)

Adds nodes to the children list

Parameters:
nodes : list or tuple

List of nodes to append to the children list

Returns:
self : object

Returns self.

add_job(job)[source]

Add job to Dagman

Parameters:
job : Job

Job to append to Dagman jobs list.

Returns:
self : object

Returns self.

add_parent(node)

Adds node to parents list

Parameters:
node : BaseNode

Job or Dagman to append to the parents list.

Returns:
self : object

Returns self.

add_parents(nodes)

Adds nodes to the parents list

Parameters:
nodes : list or tuple

List of nodes to append to the parents list

Returns:
self : object

Returns self.

add_subdag(dag)[source]

Add dag to Dagman

Parameters:
dag : Dagman

Subdag to append to Dagman jobs list.

Returns:
self : object

Returns self.

build(makedirs=True, fancyname=True)[source]

Build and saves the submit file for Dagman

Parameters:
makedirs : bool, optional

If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).

fancyname : bool, optional

Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).

Returns:
self : object

Returns self.

build_submit(**kwargs)[source]

Calls build and submit sequentially

Parameters:
makedirs : bool, optional

If Job directories (e.g. error, output, log, submit) don’t exist, create them (default is True).

fancyname : bool, optional

Appends the date and unique id number to error, log, output, and submit files. For example, instead of dagname.submit the submit file becomes dagname_YYYYMMD_id. This is useful when running several Dags/Jobs of the same name (default is True).

submit_options : str, optional

Options to be passed to condor_submit_dag for this Dagman (see the condor_submit_dag documentation for possible options).

Returns:
self : object

Returns self.

haschildren()

Checks for any children nodes

Returns:
bool

Returns whether or not this node has any child nodes.

hasparents()

Checks for any parent nodes

Returns:
bool

Returns whether or not this node has any parent nodes.

submit_dag(**kwargs)[source]

Submits Dagman to condor

Parameters:
submit_options : str, optional

Options to be passed to condor_submit_dag for this Dagman (see the condor_submit_dag documentation for possible options).

Returns:
self : object

Returns self.

visualize(filename=None)[source]

Visualize Dagman graph

Parameters:
filename : str or None, optional

File to save graph diagram to. If None then no file is saved. Valid file extensions are ‘png’, ‘pdf’, ‘dot’, ‘svg’, ‘jpeg’, ‘jpg’.