Resources for “Make”¶
Discussions¶
Recursive Make Considered Harmful. This is a well-known paper on why you shouldn’t use nested makefiles. To summarise: if you do this Make can’t see the entire DAG and that leads to problems.
Non-Recursive Make Considered Harmful: This is a research paper describing the failings of Make for large and complex builds.
Blogs¶
Of course we are not the first to suggest the use of Make for reproducibility! The blog posts cited below were found after the above tutorial was written, but can add further information and examples.
Reproducibility is hard. Discusses making a research project reproducible using Make.
GNU Make for Reproducible Data Analysis. Argues for using Make for reproducible analysis in a similar vein as we do above.
Reproducible Bioinformatics Pipelines using Make. A quite extensive tutorial on using Make for data analysis.
Automatic Data-analysis Pipelines. A similar tutorial that uses R for the analysis.
Writing a reproducible Paper. A similar tutorial with Python using variables to populate tables in the manuscript.
Tools¶
Plot the DAG of the Makefile with makefile2graph.
Alternatives to Make¶
There are many alternatives to Make. Below are some that caught our eye and that might be worth a look.
SnakeMake. A Python3-based alternative to Make. Snakemake supports multiple wildcards in filenames, supports Python code in rules, and can run workflows on workstations, clusters, the grid, and in the cloud without modification.
Tup. A fast build system that processes prerequisites bottom-up instead of Make’s top-down. The speed looks impressive and the paper describing it is interesting, but for small projects Make’s speed will not be a bottleneck. The Tupfile syntax is not compatible with that of Makefiles.
Bazel. An open-source version of Google’s Blaze build system.
Buck. Facebook’s build system.
Glossary¶
Makefile: a text file that contains the configuration for the build
Rule: an element of the Makefile that defines something that must be built, usually consists of targets, recipes, and optionally, prerequisites.
Target: the outcome of a rule in a Makefile. It is usually a file. If it is not a file, it’s a phony target.
Recipe: one or more shell commands that are executed by Make. Usually these commands update the target of the rule.
Prerequisite: the prerequisite(s) of a rule correspond to files or other targets in the Makefile that must be up to date before the rule is run.
Phony: a phony target is one that doesn’t correspond to a file on the
filesystem. A target is marked as phony by making it a prerequisite of the
.PHONY target.
Pattern: A pattern rule is a rule that contains exactly one %
character in the target, which can be used to match a part of a filename.