Linux log redirect and call stack info or core dump

There are some problems I need your help with. Please let me describe it first.

1. program(test) source:

#include <stddef.h>
#include <stdio.h>

int main(int argc, char *argv[])
{
    char * a = NULL;
    printf("print a:\n");
    printf("%c\n", *a);
}

2.excute it in bash

[root@localhost test]# ./test
print a:
Segmentation fault (core dumped)

3.direct to test.log

[root@localhost test]# ./test > test.log 2>&1
Segmentation fault (core dumped)
[root@localhost test]# cat test.log
[root@localhost test]# 

4.use stdbuf

[root@localhost test]# stdbuf -o0 ./test > test.log 2>&1
Segmentation fault (core dumped)
[root@localhost test]# cat test.log
print a:
[root@localhost test]# 

5.use shell script test.sh

[root@localhost test]# cat test.sh 
./test
[root@localhost test]# ./test.sh > test.log 2>&1
[root@localhost test]# cat test.log 
./test.sh: line 1: 63426 Segmentation fault      (core dumped) ./test
[root@localhost test]#

6.use stdbuf in test.sh

[root@localhost test]# cat test.sh
stdbuf -o0 ./test
[root@localhost test]# ./test.sh > test.log 2>&1
[root@localhost test]# more test.log 
print a:
./test.sh: line 1: 16761 Segmentation fault      (core dumped) stdbuf -o0 ./test
[root@localhost test]# 

7. my env

[root@localhost test]# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)
[root@localhost test]# uname -a
Linux localhost.localdomain 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Now I want test.log content is :

print a:
Segmentation fault (core dumped)

and don't use stdbuf(I think it maybe affect performance).

On the other hand, I expect that when encountering segfaults and other similar errors, I can use programmatically generate core or call stack instead of using "ulimit -c unlimited". I understand that it can be solved by catching signals, but I don't know which signals represent errors and exits. thank you all.

The "Segmentation fault (core dumped)" message is generated by the shell, not the program you run.

To get your desired result, you have to run the program in a sub-shell and redirect the output of the sub-shell to the log file. However, sometimes the shell shuts up (doesn't generate the error message) when run as a sub-shell. And experimentation with Bash shows that it doesn't shut up unless you work really hard.

With Korn Shell, you can use this to get the desired result (it worked on RHEL 7.4):

(./test) >test.log 2>&1

Note that redirecting the output to a file (or a pipe) probably means that standard output ceases to be line buffered and becomes fully buffered. And with a core dump, you probably won't get the standard output flushed before the program terminates — so pending I/O may be buffered and not sent to the log file.

Experimenting with Bash shows that the command shown above does not work, in the sense that the shell's error message still appears on the terminal, not in the log file.

Create a script similar to this (I called it ex-crash.sh, for executing crash):

(./crash) >log 2>&1

It can be run in a variety of ways:

$ bash ex-crash.sh
ex-crash.sh: line 1:  2646 Segmentation fault      (core dumped) ( ./crash ) > log 2>&1
$ (bash ex-crash.sh >log2 2>&1)
$ cat log2
ex-crash.sh: line 1:  2675 Segmentation fault      (core dumped) ( ./crash ) > log 2>&1
$ (bash ex-crash.sh) >log2 2>&1
$ cat log2
ex-crash.sh: line 1:  2946 Segmentation fault      (core dumped) ( ./crash ) > log 2>&1
$ bash ex-crash.sh >log2 2>&1
$ cat log2
ex-crash.sh: line 1:  3310 Segmentation fault      (core dumped) ( ./crash ) > log 2>&1
$

This shows that the error output from the script can be redirected to a file — the sub-shells on the command line are not necessary to do that. You could experiment with versions of ex-crash.sh that don't include the sub-shell, too. I got side-tracked into more complex invocations before trying the simpler ones. (The large jumps in the PID numbers occur because someone else is compiling a big project on the machine.)

Generating a stack trace will have to be done by the program. It will have to catch the signal and invoke the stack trace code and exit, possibly by calling abort(). That may not be 100% safe — the stack may be corrupted. You'll need to decide which signals you want to catch. You'l be using sigaction(), of course. You'll need to think about dealing with the SIGABRT, SIGQUIT, SIGFPE, SIGILL, SIGIOT, SIGBUS and SIGSEGV (and maybe SIGSYS, SIGTRAP, SIGXCPU and SIGXFSZ) signals. Your code will need to check if they are defined on your system. You should become familiar with POSIX Signal Concepts and the specification in <signal.h>. The table in the header specification specifies which signals give "abnormal termination" of the process "with additional actions" (which is the core dump part).

You'll need to investigate which stack trace libraries are available for you. The GNU C Library provides a set of backtrace functions. There are probably other libraries too. You won't want to write your own.

> Now I want test.log content is :

> by catching signals, but I don't know which signals represent errors

SIGSEGV signal represents segmentation fault. If you want your program to print something, handle SIGSEGV.