英文:
Parallelize recursive Makefiles for building multiple Latex Documents in CICD Pipeline
问题
You can add the -jN
flag to the make
calls in your Makefiles to parallelize the build process, where N
is the number of parallel jobs you want to run. However, there are a few things to keep in mind:
-
Makefile 1 (
$(MAKE) -C $@
call): Adding the-jN
flag here will parallelize the execution of the subdirectories, which is a good start for parallelization. -
Makefile 2: In Makefile 2, you can also add the
-jN
flag to thelatexmk
command if you want to parallelize the building of individual LaTeX documents within each subfolder. This can further speed up the process, especially if you have many documents.
Here's an example of how you can modify your Makefile 2:
# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard ./*.tex)
DOC_SRCS_NAME := $(basename $(DOC_SRCS))
.PHONY: all clean $(DOC_SRCS)
all: $(DOC_SRCS)
$(DOC_SRCS):
mkdir -p build/img
$(MAKE) -jN -C $(dir $@) latex
cp build/$(basename $@).pdf $(basename $@).pdf
latex:
latexmk -outdir=build -shell-escape -pdf $(notdir $(CURDIR)).tex
clean :
rm -rf build/*
In this modified Makefile, the latex
target is used to build the LaTeX documents in parallel within each subfolder.
Remember to replace N
with the desired number of parallel jobs. Adjusting the number of parallel jobs will depend on your system's resources and the optimal level of parallelization for your specific case.
英文:
I want to speed up the build process on my Latex documents by parallelizing the building of the docs. I know that the latex compiler itself is not able to parallelize but i can at least execute the building of the separate docs in parallel. The whole thing is running on a Gitlab CICD Runner that is just executing the top level Makefile.
My Makefiles currently look like this:
Makefile 1 for top level and mid level folders:
# Makefile to go through all the subdirs and execute the makefile there.
SUBDIRS := $(dir $(wildcard */Makefile))
all: $(SUBDIRS)
$(SUBDIRS):
$(MAKE) -C $@
.PHONY: all $(SUBDIRS)
Makefile 2 for lowest level folders where documents are placed
# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard ./*.tex)
DOC_SRCS_NAME := $(basename $(DOC_SRCS))
.PHONY: all clean $(DOC_SRCS)
all: $(DOC_SRCS)
$(DOC_SRCS):
mkdir -p build/img
latexmk -outdir=build -shell-escape -pdf $@
cp build/$(basename $@).pdf $(basename $@).pdf
clean :
rm -rf build/*
The folder structure looks something like this (I shortened it to make it less cluttered):
project
│ Makefile 1
│
└───folder
│ │ Makefile 1
│ │
│ └───subfolder
│ │ Makefile 2
│ │ file1.tex
│ │ ...
│
└───folder
| │ Makefile 1
│ │
│ └───subfolder
│ │ Makefile 2
│ │ file1.tex
│ │ ...
│ ...
My question now is, is it enough to add the -jN
flag to the make
calls or do i need to take special care of something?
Does the flag only need to be at the Makefile 1 at the $(MAKE) -C $@
call?
I couldn't really get the flag for the jobs to work properly how i intended to.
答案1
得分: 1
你的下面的Makefile有问题。在Make中,配方的目标是你想要构建的文件,而不是构建所需的源文件。
像这样:
# 用于构建当前文件夹中的所有latex文档的Makefile
DOC_SRCS := $(wildcard *.tex)
DOC_PDFS := $(DOC_SRCS:.tex=.pdf)
.PHONY: all clean
all: $(DOC_PDFS)
%.pdf : %.tex
mkdir -p build/img
latexmk -outdir=build -shell-escape -pdf $<
cp build/$@ $@
至于你的问题,我认为这个Makefile可以正常工作,无需额外的内容即可支持并行构建。你应该只在从你的CI/CD系统中调用make时添加-jN
,而不要将它添加到子make的配方中。
英文:
Your lower makefile is wrong. In make the target of a recipe is the file you want to BUILD, not the source file you build FROM.
Like this:
# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard *.tex)
DOC_PDFS := $(DOC_SRCS:.tex=.pdf)
.PHONY: all clean
all: $(DOC_PDFS)
%.pdf : %.tex
mkdir -p build/img
latexmk -outdir=build -shell-escape -pdf $<
cp build/$@ $@
As for your question, it looks to me like this will work fine as-is for parallel builds without needing anything extra. You should add the -jN
only to the invocation of make you do from your CI/CD system: do NOT add it into the recipe for the sub-makes.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论