Kong自定义的Golang插件在Kubernetes/Helm设置中无法工作。

huangapple go评论117阅读模式
英文:

Kong custom golang plugin not working in kubernetes/helm setup

问题

我已经编写了一个自定义的Golang Kong插件,名为go-wait,按照GitHub仓库https://github.com/redhwannacef/youtube-tutorials/tree/main/kong-gateway-custom-plugin中的示例进行编写。

唯一的区别是我创建了一个自定义的Docker镜像,以便Kong在其/usr/local/bin目录中默认拥有该插件。
以下是Dockerfile的内容:

FROM golang:1.18.3-alpine as pluginbuild
COPY ./charts/custom-plugins/ /app/custom-plugins
RUN cd /app/custom-plugins && \
    for d in ./*/ ; do (cd "$d" && go mod tidy && GOOS=linux GOARCH=amd64 go build .); done

RUN mkdir /app/all-plugin-execs && cd /app/custom-plugins && \
    find . -type f -not -name "*.*" | xargs -i cp {} /app/all-plugin-execs/

FROM kong:2.8

COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/
COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/plugin-ref/

# 循环遍历plugin-ref目录,并在KONG_PLUGINS和KONG_PLUGINSERVER_NAMES环境变量中为每个插件创建一个条目
# 此外,在KONG_PLUGINS列表中附加`bundled`,否则任何未使用的插件都会导致Kong出错

#### 以名为`go-wait`的插件为例的环境变量示例
# ENV KONG_PLUGINS=go-wait
# ENV KONG_PLUGINSERVER_NAMES=go-wait
# ENV KONG_PLUGINSERVER_GO_WAIT_QUERY_CMD="/usr/local/bin/go-wait -dump"
####
RUN cd /usr/local/bin/plugin-ref/ && \
    PLUGINS=$(ls | tr '\n' ',') && PLUGINS=${PLUGINS::-1} && \
    echo -e "KONG_PLUGINS=bundled,$PLUGINS\nKONG_PLUGINSERVER_NAMES=$PLUGINS" >> ~/.bashrc

# 循环遍历plugin-ref目录,并为加载插件所需的QUERY_CMD条目创建一个条目
# 如果插件名称为`eg-plugin`,则格式为KONG_PLUGINSERVER_EG_PLUGIN_QUERY_CMD,并且应指向插件后跟`-dump`参数
RUN cd /usr/local/bin/plugin-ref/ && \
    for f in *; do echo "$f" | tr "[:lower:]" "[:upper:]" | tr '-' '_' | \
    xargs -I {} sh -c "echo 'KONG_PLUGINSERVER_{}_QUERY_CMD='" && echo '\"/usr/local/bin/{} -dump\"' | tr [:upper:] [:lower:] | tr '_' '-' | \
    sed -e '$!N;s/\n//' | xargs -i echo "{}" >> ~/.bashrc; done

这在docker-compose文件和docker容器中运行良好。但是,当我尝试在Kubernetes环境中与kong-ingress-controller一起使用相同的镜像时,我遇到了错误"failed to fill-in defaults for plugin: go-wait",以及日志中的一堆其他错误,包括"plugin 'go-wait' enabled but not installed",导致我无法启用它。

是否有人尝试在他们的Kubernetes/Helm Kong设置中包含Go插件?如果有,请对此进行解释。

英文:

I have written custom golang kong plugin called go-wait following the example from the github repo https://github.com/redhwannacef/youtube-tutorials/tree/main/kong-gateway-custom-plugin

The only difference is I created a custom docker image so kong would have the mentioned plugin by default in it's /usr/local/bin directory
Here's the dockerfile

FROM golang:1.18.3-alpine as pluginbuild
COPY ./charts/custom-plugins/ /app/custom-plugins
RUN cd /app/custom-plugins && \
    for d in ./*/ ; do (cd "$d" && go mod tidy && GOOS=linux GOARCH=amd64 go build .); done

RUN mkdir /app/all-plugin-execs && cd /app/custom-plugins && \
    find . -type f -not -name "*.*" | xargs -i cp {} /app/all-plugin-execs/

FROM kong:2.8

COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/
COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/plugin-ref/

# Loop through the plugin-ref directory and create an entry for all of them in 
# both KONG_PLUGINS and KONG_PLUGINSERVER_NAMES env vars respectively
# Additionally append `bundled` to KONG_PLUGINS list as without it any unused plugin will case Kong to error out

#### Example Env vars for a plugin named `go-wait`
# ENV KONG_PLUGINS=go-wait
# ENV KONG_PLUGINSERVER_NAMES=go-wait
# ENV KONG_PLUGINSERVER_GO_WAIT_QUERY_CMD="/usr/local/bin/go-wait -dump"
####
RUN cd /usr/local/bin/plugin-ref/ && \
    PLUGINS=$(ls | tr '\n' ',') && PLUGINS=${PLUGINS::-1} && \
    echo -e "KONG_PLUGINS=bundled,$PLUGINS\nKONG_PLUGINSERVER_NAMES=$PLUGINS" >> ~/.bashrc

# Loop through the plugin-ref directory and create an entry for QUERY_CMD entries needed to load the plugin
# format KONG_PLUGINSERVER_EG_PLUGIN_QUERY_CMD if the plugin name is `eg-plugin` and it should point to the 
# plugin followed by `-dump` argument
RUN cd /usr/local/bin/plugin-ref/ && \
    for f in *; do echo "$f" | tr "[:lower:]" "[:upper:]" | tr '-' '_' | \
    xargs -I {} sh -c "echo 'KONG_PLUGINSERVER_{}_QUERY_CMD=' && echo '\"/usr/local/bin/{} -dump\"' | tr [:upper:] [:lower:] | tr '_' '-'" | \
    sed -e '$!N;s/\n//' | xargs -i echo "{}" >> ~/.bashrc; done

This works fine in the docker-compose file and docker container. But when I tried to use the same image in the kubernetes environment along with kong-ingress-controller, I started running into errors "failed to fill-in defaults for plugin: go-wait" and/or a bunch of other errors including "plugin 'go-wait' enabled but not installed" in the logs and I ended up not being able to enable it.

Has anyone tried including go plugins in their kubernetes/helm kong setup. If so please shed some light on this

答案1

得分: 1

更新:我找到了我要找的答案,以及设置由镜像生成的环境变量,还有在kong helm chart的_helpers.tpl文件中的修改。
原因是在部署图表中,配置期望在用于覆盖默认设置的values-custom.yml中配置插件。

但是helm chart似乎是针对通过configMaps加载值和插件的特定设置,这导致了一个巨大的瓶颈,因为你为kong生成的任何用于golang的二进制插件都会超过kubernetes中configMaps的最大允许限制。
这就是我开始这个努力将插件作为镜像的一部分的整个原因。

TL;dr
我能够将仓库克隆到我的本地系统,并对以下补丁进行更改,以便从values中加载插件,而无需将它们与lua插件合并。(感谢讨论https://discuss.konghq.com/t/how-to-load-go-plugins-using-kong-helm-chart/5717/10中的thatbenguy的答案)

--- a/charts/kong/templates/_helpers.tpl
+++ b/charts/kong/templates/_helpers.tpl
@@ -530,6 +530,9 @@ The name of the service used for the ingress controller's validation webhook

 {{- define "kong.plugins" -}}
 {{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
 {{- range .Values.plugins.configMaps -}}
 {{- $myList = append $myList .pluginName -}}
 {{- end -}}

将以下代码块添加到我的values-custom.yml中,然后就可以使用了。

希望这对其他试图为helm charts中的kong编写自定义golang插件的人也有所帮助。

env:
  database: "off"
  plugins: bundled,go-wait
  pluginserver_names: go-wait
  pluginserver_go_wait_query_cmd: "/usr/local/bin/go-wait -dump"
  

plugins:
  goPlugins:
    - pluginName: "go-wait"

注意:请记住,所有这些仍然取决于你在镜像中具有预构建的自定义kong插件,在我的情况下,我从上述dockerfile内容(在问题中)构建了一个镜像,并将其推送到我的docker hub仓库,并使用以下代码块替换了values-custom.yml中的镜像。

image:
  repository: chalukyaj/kong-custom-image
  tag: "1.0.1"

PS:正如你们可能已经注意到的,我对此唯一的失望是环境变量不能直接从docker镜像的~/.bashrc中获取,这本来会很棒。但尽管如此,这个方法有效,我找不到任何一篇文章展示如何使用新的go-pdk(而不是旧的go-pluginserver)构建go插件并在helm中使用它们。

英文:

Update: Found the answer I was looking for, along with setting the environment variables generated by the image, there's modifications in the _helpers.tpl file of the kong helm chart itself.
The reason is that in the deployment charts, the configuration expects plugins to be configured in values-custom.yml used to override the default settings.

But the helm chart seems to be specific to values and plugins being loaded via configMaps which turned out to be a huge bottleneck, as any binary plugin you will generate in golang for kong is going to exceed the maximum allowed limit of the configMaps in kubernetes.
That's the whole reason I had set out on this endeavor to make the plugins part of my image.

TL;dr
I was able to clone the repo to my local system, make the changes for the following patch for loading the plugins from values without having to club them with the lua plugins. (Credits : Answer of thatbenguy from the discussion https://discuss.konghq.com/t/how-to-load-go-plugins-using-kong-helm-chart/5717/10)

--- a/charts/kong/templates/_helpers.tpl
+++ b/charts/kong/templates/_helpers.tpl
@@ -530,6 +530,9 @@ The name of the service used for the ingress controller's validation webhook

 {{- define "kong.plugins" -}}
 {{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
 {{- range .Values.plugins.configMaps -}}
 {{- $myList = append $myList .pluginName -}}
 {{- end -}}

Add the following block to my values-custom.yml and I was good to go.

Hopefully this helps anyone else also trying to write custom plugins for kong in golang for use in helm charts.

env:
  database: "off"
  plugins: bundled,go-wait
  pluginserver_names: go-wait
  pluginserver_go_wait_query_cmd: "/usr/local/bin/go-wait -dump"
  

plugins:
  goPlugins:
    - pluginName: "go-wait"

NOTE : Please remember all this still depends on you having the prebuilt custom kong plugins in your image, in my case I had built an image from the above dockerfile contents (in question) and pushed that to my own docker hub repo and replaced the image in the values-custom.yml using the following block

image:
  repository: chalukyaj/kong-custom-image
  tag: "1.0.1"

PS: As you guys might have noticed, the only disappointment I have with this is that the environment variables couldn't just be picked from the docker image's ~/.bashrc, which would have made this awesome. But nonetheless, this works, and I couldn't find a single post which showed how to use the new go-pdk (instead of the older go-pluginserver) to build the go plugins and use them in helm.

huangapple
  • 本文由 发表于 2022年6月10日 18:19:25
  • 转载请务必保留本文链接:https://go.coder-hub.com/72572668.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定