Is there a way to disable logs forwarding to GCP when using cloud.google.com/go/logging in development/test environment?

huangapple go评论68阅读模式
英文:

Is there a way to disable logs forwarding to GCP when using cloud.google.com/go/logging in development/test environment?

问题

我考虑过在开发环境中隐藏日志记录器的初始化,并返回一个普通的日志记录器(log),在部署到GCP时返回一个cloud.google.com/go/logging的日志记录器。

但是cloud.google.com/go/logging的日志记录器与log的接口不同,所以需要进行一些额外的包装,我希望能够避免这种情况。

根据文档的描述,我没有找到一种标准的方法来实现这一点。如果没有办法实现,除了使日志变得杂乱无章之外,有没有常见的方法来避免向Google Cloud Logging发送来自非预生产/生产环境的日志?除了使日志变得杂乱无章之外,日志的摄取也会产生一定的成本,我希望在某些情况下能够避免这种成本。

英文:

I though about "hiding" logger initialization and returning a normal logger (log) from it when running the code in development environment and a cloud.google.com/go/logging one when deployed on GCP.

But cloud.google.com/go/logging logger does not have the same interface as log, so it would need some extra wrapping and I hope I can avoid that.

Looking at the documentation, I can't find a standard way to do this. If there is no way to do it, what's the common approach to avoid spamming google cloud logging with logs not coming from preproduction/production environment? Aside from making the logs noisy, logs ingestion have a cost I would like to avoid in some situations.

答案1

得分: 2

除了@Sergiusz的回答之外,您可以将开发日志记录到相同的日志或具有模板名称的日志中,并使用logadmin包按名称删除日志。您可以在执行结束时、定期处理过程中或从命令行中执行此操作。您还可以使用gcloud logging logs delete 命令来执行此操作。如果您有多个日志,您需要对每个日志调用"delete"多次。

然而,我建议您模拟该包,因为使用它可能会耗尽您在开发项目中的API配额。因此,如果将日志写入STDOUT足够,最快的方法就是模拟Logger类型。

P.S. 我建议您在https://github.com/googleapis/google-cloud-go中提出问题,并说明您的用例并要求更改。这可能会导致破坏性变更,但提供反馈仍然很重要。

英文:

In addition to the @Sergiusz answer, you can log your development into the same log or logs with the templated name and use logadmin package to delete the log(s) by the name. You can do it at the end of execution, in scheduled process or from a command line. You can do it also using gcloud logging logs delete command. If you have multiple logs you will need to call "delete" multiple times per log.

However, I would recommend to mock the package instead because, using it may exhaust your API quota on the development project. So, if writing logs to STDOUT is sufficient, the fastest approach would be just to mock the Logger type.

P.S. I would recommend to open an issue in https://github.com/googleapis/google-cloud-go with your use case and ask for the change. It might lead to the breaking change though, but it is still important to provide the feedback.

答案2

得分: 1

总结一下评论:
Google Cloud 允许创建排除过滤器用于 Sinks
根据Google 文档
>你可以选择将日志导出到 BigQuery、Cloud Storage 或 Cloud Pub/Sub,无需支付将其导入 Stackdriver 的费用。你甚至可以使用排除过滤器来收集一定比例的日志,比如成功的 HTTP 响应的 1%。

此外,每个项目每月的前 50 GiB 日志是免费的1
日志存储桶的前 30 天也是免费的。

英文:

To recap the comments:
Google Cloud allows creating exclusion filters for Sinks.
According to Google docs:
>you can choose to export your logs to BigQuery, Cloud Storage or Cloud Pub/Sub without needing to pay to ingest them into Stackdriver. You can even use exclusion filters to collect a percentage of logs, such as 1% of successful HTTP responses.

Additionally, note that the first 50 GiB of logs per project per month are free 1.
First 30 days of storage in the Log buckets are also free.

答案3

得分: 0

最终,根据@leo-y的建议,我模拟了一个日志记录器,并将实现细节隐藏在接口后面。我对此有些犹豫,因为我对golang还不熟悉,但最终代码量并不多:

logging/Logger.go:

package logging

// Logger只是一个接口,用于抽象化初始化时返回的日志记录器的类型
type Logger interface {
	LogInfo(string)
	LogWarning(string)
	LogError(string)
	Close()
}

logging/initialization/logging.go:

package initialization

import (
	"context"
	"log"

	customLogging "organization/module-name/logging"

	"cloud.google.com/go/logging"
	"google.golang.org/api/option"
)

// GetLogger返回一个日志记录器客户端
func GetLogger(onCloud bool) customLogging.Logger {
	if onCloud {
		return getLoggerGCP()
	}
	return loggerConsole{}
}

func getLoggerGCP() loggerGCP {
	ctx := context.Background()
	projectID := "PROJECT-ID"
	client, err := logging.NewClient(ctx, projectID, option.WithCredentialsFile("/path/to/credentials.json"))
	if err != nil {
		log.Fatalf("Failed to create logging client: %v", err)
	}
	return loggerGCP{
		client: client,
		logger: client.Logger("logs-name"),
	}
}

type loggerGCP struct {
	client *logging.Client
	logger *logging.Logger
}

type loggerConsole struct {
}

func (logger loggerGCP) LogInfo(s string) {
	stdlogger := logger.logger.StandardLogger(logging.Info)
	stdlogger.Println(s)
}

func (logger loggerGCP) LogWarning(s string) {
	stdlogger := logger.logger.StandardLogger(logging.Warning)
	stdlogger.Println(s)
}

func (logger loggerGCP) LogError(s string) {
	stdlogger := logger.logger.StandardLogger(logging.Error)
	stdlogger.Println(s)
}

func (logger loggerGCP) Close() {
	logger.client.Close()
}

func (logger loggerConsole) LogInfo(s string) {
	log.Printf("INFO: %s\n", s)
}

func (logger loggerConsole) LogWarning(s string) {
	log.Printf("WARNING: %s\n", s)
}

func (logger loggerConsole) LogError(s string) {
	log.Printf("ERROR: %s\n", s)
}

func (logger loggerConsole) Close() {
}

main.go:

package main

import (
	"organization/module-name/logging/initialization"
)

func main() {
	logger := initialization.GetLogger(false)
	defer logger.Close()
	logger.LogInfo("hello")
}

我还没有对这段代码进行太多挑战,所以可能会有一些小问题。

英文:

In the end as suggested by @leo-y, I kind of mocked a logger and hide implementation details behind an interface. I was reluctant to do this because I'm new to golang, but in the end that's not so much code:

logging/Logger.go:

package logging
// Logger is just an interface to abstract away what
// kind of logger we return on initialization
type Logger interface {
LogInfo(string)
LogWarning(string)
LogError(string)
Close()
}

logging/initialization/logging.go

package initialization
import (
"context"
"log"
customLogging "organization/module-name/logging"
"cloud.google.com/go/logging"
"google.golang.org/api/option"
)
// GetLogger return a logging client
func GetLogger(onCloud bool) customLogging.Logger {
if onCloud {
return getLoggerGCP()
}
return loggerConsole{}
}
func getLoggerGCP() loggerGCP {
ctx := context.Background()
projectID := "PROJECT-ID"
client, err := logging.NewClient(ctx, projectID, option.WithCredentialsFile("/path/to/credentials.json"))
if err != nil {
log.Fatalf("Failed to create logging client: %v", err)
}
return loggerGCP{
client: client,
logger: client.Logger("logs-name"),
}
}
type loggerGCP struct {
client *logging.Client
logger *logging.Logger
}
type loggerConsole struct {
}
func (logger loggerGCP) LogInfo(s string) {
stdlogger := logger.logger.StandardLogger(logging.Info)
stdlogger.Println(s)
}
func (logger loggerGCP) LogWarning(s string) {
stdlogger := logger.logger.StandardLogger(logging.Warning)
stdlogger.Println(s)
}
func (logger loggerGCP) LogError(s string) {
stdlogger := logger.logger.StandardLogger(logging.Error)
stdlogger.Println(s)
}
func (logger loggerGCP) Close() {
logger.client.Close()
}
func (logger loggerConsole) LogInfo(s string) {
log.Printf("INFO: %s\n", s)
}
func (logger loggerConsole) LogWarning(s string) {
log.Printf("WARNING: %s\n", s)
}
func (logger loggerConsole) LogError(s string) {
log.Printf("ERROR: %s\n", s)
}
func (logger loggerConsole) Close() {
}

main.go

package main
import (
"organization/module-name/logging/initialization"
)
func main() {
logger := initialization.GetLogger(false)
defer logger.Close()
logger.LogInfo("hello")
}

I didn't challenge that code much yet, so maybe it will have some quirks

huangapple
  • 本文由 发表于 2021年9月22日 17:50:59
  • 转载请务必保留本文链接:https://go.coder-hub.com/69281902.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定