并行化递归的Makefile以在CICD流水线中构建多个Latex文档

huangapple go评论61阅读模式
英文:

Parallelize recursive Makefiles for building multiple Latex Documents in CICD Pipeline

问题

You can add the -jN flag to the make calls in your Makefiles to parallelize the build process, where N is the number of parallel jobs you want to run. However, there are a few things to keep in mind:

  1. Makefile 1 ($(MAKE) -C $@ call): Adding the -jN flag here will parallelize the execution of the subdirectories, which is a good start for parallelization.

  2. Makefile 2: In Makefile 2, you can also add the -jN flag to the latexmk command if you want to parallelize the building of individual LaTeX documents within each subfolder. This can further speed up the process, especially if you have many documents.

Here's an example of how you can modify your Makefile 2:

# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard ./*.tex)
DOC_SRCS_NAME := $(basename $(DOC_SRCS))

.PHONY: all clean $(DOC_SRCS)

all: $(DOC_SRCS)
$(DOC_SRCS):
	mkdir -p build/img
	$(MAKE) -jN -C $(dir $@) latex
	cp build/$(basename $@).pdf $(basename $@).pdf

latex:
	latexmk -outdir=build -shell-escape -pdf $(notdir $(CURDIR)).tex

clean :
	rm -rf build/*

In this modified Makefile, the latex target is used to build the LaTeX documents in parallel within each subfolder.

Remember to replace N with the desired number of parallel jobs. Adjusting the number of parallel jobs will depend on your system's resources and the optimal level of parallelization for your specific case.

英文:

I want to speed up the build process on my Latex documents by parallelizing the building of the docs. I know that the latex compiler itself is not able to parallelize but i can at least execute the building of the separate docs in parallel. The whole thing is running on a Gitlab CICD Runner that is just executing the top level Makefile.

My Makefiles currently look like this:

Makefile 1 for top level and mid level folders:

# Makefile to go through all the subdirs and execute the makefile there.
SUBDIRS := $(dir $(wildcard */Makefile))

all: $(SUBDIRS)
$(SUBDIRS):
	$(MAKE) -C $@

.PHONY: all $(SUBDIRS)

Makefile 2 for lowest level folders where documents are placed

# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard ./*.tex)
DOC_SRCS_NAME := $(basename $(DOC_SRCS))

.PHONY: all clean $(DOC_SRCS)

all: $(DOC_SRCS)
$(DOC_SRCS):
	mkdir -p build/img
	latexmk -outdir=build -shell-escape -pdf $@
	cp build/$(basename $@).pdf $(basename $@).pdf

clean :
	rm -rf build/*

The folder structure looks something like this (I shortened it to make it less cluttered):

project
│   Makefile 1    
│
└───folder
│   │   Makefile 1
│   │
│   └───subfolder
│       │   Makefile 2
│       │   file1.tex
│       │   ...
│   
└───folder
|   │   Makefile 1
│   │
│   └───subfolder
│       │   Makefile 2
│       │   file1.tex
│       │   ...
│   ...

My question now is, is it enough to add the -jN flag to the make calls or do i need to take special care of something?
Does the flag only need to be at the Makefile 1 at the $(MAKE) -C $@ call?

I couldn't really get the flag for the jobs to work properly how i intended to.

答案1

得分: 1

你的下面的Makefile有问题。在Make中,配方的目标是你想要构建的文件,而不是构建所需的源文件。

像这样:

# 用于构建当前文件夹中的所有latex文档的Makefile
DOC_SRCS := $(wildcard *.tex)

DOC_PDFS := $(DOC_SRCS:.tex=.pdf)

.PHONY: all clean

all: $(DOC_PDFS)

%.pdf : %.tex
        mkdir -p build/img
        latexmk -outdir=build -shell-escape -pdf $<
        cp build/$@ $@

至于你的问题,我认为这个Makefile可以正常工作,无需额外的内容即可支持并行构建。你应该只在从你的CI/CD系统中调用make时添加-jN,而不要将它添加到子make的配方中。

英文:

Your lower makefile is wrong. In make the target of a recipe is the file you want to BUILD, not the source file you build FROM.

Like this:

# Makefile to build all latex docs in current folder
DOC_SRCS := $(wildcard *.tex)

DOC_PDFS := $(DOC_SRCS:.tex=.pdf)

.PHONY: all clean

all: $(DOC_PDFS)

%.pdf : %.tex
        mkdir -p build/img
        latexmk -outdir=build -shell-escape -pdf $&lt;
        cp build/$@ $@

As for your question, it looks to me like this will work fine as-is for parallel builds without needing anything extra. You should add the -jN only to the invocation of make you do from your CI/CD system: do NOT add it into the recipe for the sub-makes.

huangapple
  • 本文由 发表于 2023年6月12日 21:39:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/76457244.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定