GCP logging throwing an error when submitting more that 1 log per seco

I am trying to implement logging in Flask. The API run on Google App Engine and I am using GCP Logging. It works, unless I submit more that around 1 log per second. I have the permission to create logs since some of them are recorded without any issues and some of them aren't.

The Python code is:

...
import logging
import google.cloud.logging

...
@app.route("/log", methods=['GET'])
    def log():
        logging.basicConfig(
            level=logging.INFO,
        )
        logging.info('Hello')

        client = google.cloud.logging.Client()
        client.get_default_handler()
        client.setup_logging()

        return 'info log'

The error:

`Failed to submit 1 logs.`

Traceback:

File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 946, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/grpc/_channel.py", line 849, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.PERMISSION_DENIED
details = "Permission 'logging.logEntries.create' denied on resource (or it may not exist)."


The above exception was the direct cause of the following exception:

  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/logging_v2/handlers/transports/background_thread.py", line 114, in _safely_commit_batch
    batch.commit()
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/logging_v2/logger.py", line 453, in commit
    client.logging_api.write_entries(entries, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/logging_v2/_gapic.py", line 160, in write_entries
    self._gapic_api.write_log_entries(request=request)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/cloud/logging_v2/services/logging_service_v2/client.py", line 631, in write_log_entries
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 283, in retry_wrapped_func
    return retry_target(
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/layers/google.python.pip/pip/lib/python3.10/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.PermissionDenied: 403 Permission 'logging.logEntries.create' denied on resource (or it may not exist). [reason: "IAM_PERMISSION_DENIED"
domain: "iam.googleapis.com"

I followed Google tutorial https://cloud.google.com/logging/docs/setup/python and I didn't have any issues locally with Pythons logging library. I tried rewriting code to instantiate the handler only once from a Python module but then I was getting this error every time.

The error IAM_PERMISSION_DENIED, in this case, means that the principal performing the action is lacking the logging.logEntries.create permission. This permission is included in the Logs Writer predefined role.

Reading through the article you shared, looks like if you would like to run your application locally and elsewhere, you need to first:

• Grant the service account the Logs Writer IAM role.
• Set up Application Default Credentials.

If this is not the case, then check that the underlying service account has the Logs Writer role. Hope it helped!