doit is all about automating task dependency management and execution. Tasks can execute external shell commands/scripts or python functions (actually any callable). So a task can be anything you can code :)
Tasks are defined in plain python module with some conventions. A function that starts with the name task_ defines a task-creator recognized by doit. These functions must return (or yield) dictionaries representing a task. A python module/file that defines tasks for doit is called dodo file (that is something like a Makefile for make).
Note
You should be comfortable with python basics. If you don’t know python yet check Python tutorial.
Take a look at this example (file dodo.py):
def task_hello():
"""hello"""
def python_hello(targets):
with open(targets[0], "a") as output:
output.write("Python says Hello World!!!\n")
return {
'actions': [python_hello],
'targets': ["hello.txt"],
}
When doit is executed without any parameters it will look for tasks in a file named dodo.py in the current folder and execute its tasks.
$ doit
. hello
On the output it displays which tasks were executed. In this case the dodo file has only one task, hello.
Every task must define actions. It can optionally defines other attributes like targets, file_dep, verbosity, doc ...
Actions define what the task actually do. A task can define any number of actions. The action “result” is used to determine if task execution was successful or not.
There 2 basic kinds of actions: cmd-action and python-action.
If action is a python callable or a tuple (callable, *args, **kwargs) - only callable is required. The callable must be a funcion, method or callable object. Classes and built-in funcions are not allowed. args is a sequence and kwargs is a dictionary that will be used as positional and keywords arguments for the callable. see Keyword Arguments.
The result of the task is given by the returned value of the action function. So it must return a boolean value True, None, a dictionary or a string to indicate successful completion of the task. Use False to indicate task failed. If it raises an exception, it will be considered an error. If it returns any other type it will also be considered an error but this behavior might change in future versions.
def task_hello():
"""hello py """
def python_hello(times, text, targets):
with open(targets[0], "a") as output:
output.write(times * text)
return {'actions': [(python_hello, [3, "py!\n"])],
'targets': ["hello.txt"],
}
The function task_hello is a task-creator, not the task itself. The body of the task-creator function is always executed when the dodo file is loaded.
Note
The body of task-creators are executed even if the task is not going to be executed. The body of task-creators should be used to create task metadata only, not execute tasks! From now on when the documentation says that a task is executed, read “the task’s actions are executed”.
If action is a string it will be executed by the shell.
The result of the task follows the shell convention. If the process exits with the value 0 it is successful. Any other value means the task failed.
Note that the string must be escaped acoording to python string formatting.
def task_hello():
"""hello cmd """
msg = 3 * "hi! "
return {
'actions': ['echo %s ' % msg + ' > %(targets)s',],
'targets': ["hello.txt"],
}
It is easy to include dynamic (on-the-fly) behavior to your tasks with python code from the dodo file. Let’s take a look at another example:
Note
The body of the task-creator is always executed, so in this example the line msg = 3 * “hi! “ will always be executed.
For complex commands it is also possible to pass a callable that returns the command string. In this case you must explicit import CmdAction.
from doit.action import CmdAction
def task_hello():
"""hello cmd """
def create_cmd_string():
return "echo hi"
return {
'actions': [CmdAction(create_cmd_string)],
'verbosity': 2,
}
It is possible to create other type of actions, check tools.InteractiveAction as an example.
By default a task name is taken from the name of the python function that generates the task. For example a def task_hello would create a task named hello.
It is possible to explicit set a task name with the parameter basename.
def task_hello():
return {
'actions': ['echo hello']
}
def task_xxx():
return {
'basename': 'hello2',
'actions': ['echo hello2']
}
$ doit
. hello
. hello2
When explicit using basename the task-creator is not limited to create only one task. Using yield it can generate several tasks at once. It is also possible to yield a generator that genrate tasks. This is useful to write some generic/reusable task-creators.
def gen_many_tasks():
yield {'basename': 't1',
'actions': ['echo t1']}
yield {'basename': 't2',
'actions': ['echo t2']}
def task_all():
yield gen_many_tasks()
$ doit
. t2
. t1
Most of the time we want to apply the same task several times in different contexts.
The task function can return a python-generator that yields dictionaries. Since each sub-task must be uniquely identified it requires an additional field name.
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
yield {'name': filename,
'actions': ["touch %s" % filename]}
$ doit
. create_file:file0.txt
. create_file:file1.txt
. create_file:file2.txt
One of the main ideas of doit (and other build-tools) is to check if the tasks/targets are up-to-date. In case there is no modification in the dependencies and the targets already exist, it skips the task execution to save time, as it would produce the same output from the previous run.
i.e. In a compilation task the source file is a file_dep, the object file is a target.
def task_compile():
return {'actions': ["cc -c main.c"],
'file_dep': ["main.c", "defs.h"],
'targets': ["main.o"]
}
doit automatically keeps track of file dependencies. It saves the signature (MD5) of the dependencies every time the task is completed successfully.
So if there are no modifications to the dependencies and you run doit again. The execution of the task’s actions is skipped.
$ doit
. compile
$ doit
-- compile
Note the -- (2 dashes, one space) on the command output on the second time it is executed. It means, this task was up-to-date and not executed.
Different from most build-tools dependencies are on tasks, not on targets. So doit can take advantage of the “execute only if not up-to-date” feature even for tasks that not define targets.
Lets say you work with a dynamic language (python in this example). You don’t need to compile anything but you probably wants to apply a lint-like tool (i.e. pyflakes) to your source code files. You can define the source code as a dependency to the task.
def task_checker():
return {'actions': ["pyflakes sample.py"],
'file_dep': ["sample.py"]}
$ doit
. checker
$ doit
-- checker
Note the -- again to indicate the execution was skipped.
Traditional build-tools can only handle files as “dependencies”. doit has several ways to check for dependencies, those will be introduced later.
Targets can be any file path (a file or folder). If a target doesn’t exist the task will be executed. There is no limitation on the number of targets a task may define. Two different tasks can not have the same target.
Lets take the compilation example again.
def task_compile():
return {'actions': ["cc -c main.c"],
'file_dep': ["main.c", "defs.h"],
'targets': ["main.o"]
}
$ doit
. compile
$ doit
-- compile
$ rm main.o
$ doit
. compile
$ echo xxx > main.o
$ doit
-- compile
If your tasks interact in a way where the target (output) of one task is a file_dep (input) of another task, doit will make sure your tasks are executed in the correct order.
def task_modify():
return {'actions': ["echo bar > foo.txt"],
'file_dep': ["foo.txt"],
}
def task_create():
return {'actions': ["touch foo.txt"],
'targets': ["foo.txt"]
}
$ doit
. create
. modify
By default all tasks are executed in the same order as they were defined (the order may change to satisfy dependencies). You can control which tasks will run in 2 ways.
Another example
DOIT_CONFIG = {'default_tasks': ['t3']}
def task_t1():
return {'actions': ["touch task1"],
'targets': ['task1']}
def task_t2():
return {'actions': ["echo task2"]}
def task_t3():
return {'actions': ["echo task3"],
'file_dep': ['task1']}
dodo file defines a dictionary DOIT_CONFIG with default_tasks, a list of strings where each element is a task name.
$ doit
. t1
. t3
Note that only the task t3 was specified to be executed by default. But its dependencies include a target of another task (t1). So that task was automatically executed also.
From the command line you can control which tasks are going to be execute by passing its task name. Any number of tasks can be passed as positional arguments.
$ doit t2
. t2
You can also specify which task to execute by its target:
$ doit task1
. t1
You can select sub-tasks from the command line specifying its full name.
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
yield {'name': filename,
'actions': ["touch %s" % filename]}
$ doit create_file:file2.txt
. create_file:file2.txt
It is possible to pass variable values to be used in dodo.py from the command line.
from doit import get_var
config = {"abc": get_var('abc', 'NO')}
def task_echo():
return {'actions': ['echo hi %s' % config],
'verbosity': 2,
}
$ doit
. echo
hi {abc: NO}
$ doit abc=xyz x=3
. echo
hi {abc: xyz}
Apart from collect functions that start with the name task_. The doit loader will also execute the create_doit_tasks callable from any object that contains this attribute.
def make_task(func):
"""make decorated function a task-creator"""
func.create_doit_tasks = func
return func
@make_task
def sample():
return {
'verbosity': 2,
'actions': ['echo hi'],
}
The project letsdoit has some real-world implementations.
For simple examples to help you create your own check this blog post.