在Google App Engine上将一个Go程序(Web爬虫)作为cron作业执行。

huangapple go评论81阅读模式
英文:

Execute a Go programm (webcrawler) as a cron job on Google appengine

问题

如何在Google App Engine中将名为"gcinfo"的单个Go程序(带有Firebase输出的网络爬虫)作为cron运行?

我已经能够创建一个项目ID,并使用App SDK上传Go程序。cron作业按照cron.yaml中定义的方式每15分钟执行一次。没有错误。但是我在日志中找不到输出,并且Firebase中也没有写入任何内容。在app.yaml、gcinfo.yaml和cron.yaml中进行了很多更改,但没有结果或错误(错误代码204)。我对yaml文件中的设置感到非常困惑。

有人可以给我一个简单的这些设置的示例,或者指导我去找一个吗?我想要在应用引擎中每15分钟运行一个单独的Go程序作为cron。

我的项目结构如下:

  • myproject文件夹,包含app.yaml和cron.yaml
  • myproject子文件夹"hello",包含hello.yaml和简单的hello.go示例,输出为"hello world!"
  • myproject子文件夹"gcinfo",包含gcinfo.yaml和gcinfo.go(带有Firebase输出的Go网络爬虫)

app.yaml

application: myproject
version: 1
runtime: go
api_version: go1

handlers:
- url: /.*
  script: _go_app

cron.yaml

cron:
- description: Ausfuehrung des tasks gcinfo
  url: /gcinfo
  schedule: every 15 minutes from 05:30 to 23:00
  timezone: Europe/Berlin
  target: gcinfo

gcinfo.yaml

application: myproject
module: gcinfo
#version: 1
runtime: go
api_version: go1

handlers:
- url: /gcinfo\.*
  script: gcinfo\gcinfo.go

我的gcinfo.go的结构如下

package gcinfo

import (
    ...
)

....

func gcinfo() {
    ....
}

在"goapp deploy"中没有错误,并且每15分钟appengine会有6ms的反应,但是没有任何输出。我已经尝试将gcinfo更改为main,结果相同。

英文:

How can I run a single Go program namend "gcinfo" (webcrawler with output in firebase) as a cron in Google appengine?

I was able to create a projekt-ID and upload the Go programm with the App SDK. The cron job was executed as defined in the cron.yaml every 15 minutes. There was no error. But i find no output in the log and firebase ist not written. After lots of changes in the app.yaml, gcinfo.yaml and cron.yaml with no result or Errors like (Error code 204). I am now totally confused about the settings in the yaml-files.

Can someone give or point me to a simple example of these settings? I want to run a single Go program as a cron in the app engine every 15 minutes.

My projekt structur is:

  • myproject folder with app.yaml and cron.yaml
  • myproject subfolder "hello" with hello.yaml and the simple hello.go example with "hello world!"
  • myproject subfolder "gcinfo" with gcinfo.yaml and gcinfo.go (go webcrawler with firebase-output)"

app.yaml

application: myproject
version: 1
runtime: go
api_version: go1

handlers:
- url: /.*
  script: _go_app

cron.yaml

cron:
- description: Ausfuehrung des tasks gcinfo
  url: /gcinfo
  schedule: every 15 minutes from 05:30 to 23:00
  timezone: Europe/Berlin
  target: gcinfo

gcinfo.yaml

application: myproject
module: gcinfo
#version: 1
runtime: go
api_version: go1

handlers:
- url: /gcinfo\.*
  script: gcinfo\gcinfo.go

My gcinfo.go has the following structure

package gcinfo

import (
    ...
)

....

func gcinfo() {
    ....
}

There are no errors with this configuration in the "goapp deploy" and the appengine reacts every 15 minutes for 6ms, but there is no output for the go programm gcinfo. I have already tryed to make gcinfo to main with same result.

答案1

得分: 2

我已经找到了一个解决方案,现在cron作业正在运行并在作业控制中写入评论。

在我的项目文件夹中的cron.yaml文件中:

cron:
- description: Ausfuehrung des tasks gcinfo
  url: /gcinfo
  schedule: every 15 minutes from 05:30 to 23:00
  timezone: Europe/Berlin

在子文件夹gcinfo中的app.yaml文件中:

application: myproject
module: gcinfo
version: 1
runtime: go
api_version: go1

handlers:
- url: /gcinfo
  script: _go_app

在gcinfo.go文件中的关键更改(gcinfo子文件夹):

package gcinfo

import (
    "net/http"
    ...
    "appengine"
    "appengine/urlfetch"
)

func init() {
    http.HandleFunc("/gcinfo", gcinfo)
}
...

func gcinfo(w http.ResponseWriter, r *http.Request) {
    c := appengine.NewContext(r)
    ...
}

只写入Firebase引擎无法与appengine一起工作。我需要进行更多的研究。

英文:

I have found a solution and now the cron job runs and writes comments in the Job Control.

cron.yaml in myproject folder

cron:
- description: Ausfuehrung des tasks gcinfo
  url: /gcinfo
  schedule: every 15 minutes from 05:30 to 23:00
  timezone: Europe/Berlin

app.yaml in the subfolder gcinfo

application: myproject
module: gcinfo
version: 1
runtime: go
api_version: go1

handlers:
- url: /gcinfo
  script: _go_app 

and the key changes in gcinfo.go (gcinfo subfolder)

package gcinfo

import (
"net/http"
...
"appengine"
"appengine/urlfetch"
)

func init() {
 http.HandleFunc("/gcinfo", gcinfo)
}
...

func gcinfo(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
...
}

Only writing firebase engine does not work with the appengine. I'll have to do more research.

huangapple
  • 本文由 发表于 2015年1月8日 23:40:40
  • 转载请务必保留本文链接:https://go.coder-hub.com/27843928.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定