How to logging in Amazon Web Service ( AWS )?

I have a project built in Golang and deployed on a Docker instance in AWS.

Internally I create a log file where the program write several logs.

How can I access that log file?

Is there another correct way to logging?

Thanks

You could mount the log file from your container to your EC2 host. You can do this by using the -v flag when running your container:

docker run -v /var/log/my_host_log_file.log:/var/log/your_container_log_file.log your-image

Alternatively, you can configure your app to log to stdout and use syslog as your log driver (using the --log-driver=syslog switch). Your container logs will then be written to /var/log/messages on your host.

If you use AWS, i would suggest to send Logs direct to AWS CloudWatch.

First create a new Log-Group in AWS Cloudwatch, for example "Production". In your Docker-Compose.yml (or via docker run..) add the AWS Logdriver:

    logging:
    driver: "awslogs"
    options:
      awslogs-region: "eu-central-1"
      awslogs-group: "Production"
      awslogs-stream: "MyApp"

Next creat a IAM user with Access to AWS Cloudwatch and add to the Dockerhost the credentials.

Example IAM Policy:

"Version" "2012-10-17" 
"Statement" 

"Action" "logs:CreateLogStream" "logs:PutLogEvents" "Effect" "Allow" "Resource"

On Ubuntu with systemd:

"Version" "2012-10-17" 
"Statement" 
  "Action" 
    "logs:CreateLogStream" 
    "logs:PutLogEvents" 
  "Effect" 
  "Allow" "Resource"

And add to the File:

[Service] Environment"AWS_ACCESS_KEY_ID=<aws_access_key_id>" 
Environment"AWS_SECRET_ACCESS_KEY=<aws_secret_access_key>"

Run:

systemctl daemon-reload
service docker restart

Now your logs should appear in AWS Cloudwatch.

Thanks for reply.

After a while looking for the solution to the problem, I found it!

Firstly, I needed to mount the file that is inside the instance in the docker-host.

To do this I add a Json file in the root folder of my project called Dockerrun.aws.json
( http://docs.aws.amazon.com/es_es/elasticbeanstalk/latest/dg/create_deploy_docker_image.html#create_deploy_docker_image_dockerrun )

That is the file that declares the shared folder (volumes) (beetwen docker-host and instance) where I save my log file . This line is equivalent to adding -v flag in the docker run command (https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-data-volume). I do this this way because I can not add mount to a running instance and i cant stop it by ssh.

{
  "AWSEBDockerrunVersion": "1",
  "Volumes": [
  {
    "HostDirectory": "/var/log/",
    "ContainerDirectory": "/go/src/app/log"
  }
  ]
}

Then to tell aws that I want to download my log file when I request records. (Tail (last 100 lines), bundle or rotate) I add these files to the .ebextension folder in my project directory. ( http://docs.aws.amazon.com/en_us/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-extend )

Log_bundle.conf

Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_bundle.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Log_rotate.config

Files:
"/opt/elasticbeanstalk/tasks/bundlelogs.d/log_rotate.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Log_tail.config

Files:
"/opt/elasticbeanstalk/tasks/publishlogs.d/log_tail.conf":
    Mode: "000755"
    Owner: root
    Group: root
    Content: |
      /var/log/application.log

Finally, I dont try Amazon Could Watch but is the next step.

Regards

If you use ELK (Elasticsearch, Logstash, Kibana), I would suggest to use "logrus"

Get the library

> go get github.com/sirupsen/logrus

Then in your project

package main

import (

	logrus "github.com/sirupsen/logrus"
)

var log = logrus.New()

func main() {

	conn, _ := net.Dial("tcp", "logstash-address")
	hook := logrustash.New(conn, logrustash.DefaultFormatter(logrus.Fields{"type": "my-app"}))
	log.Hooks.Add(hook)


	log.Info("Hello World!")

}