英文:
mmap'ing a huge file as read-only fails with ENOMEM
问题
我正在运行以下（最小重现）代码：
#include <stdio.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
void main() {
int fd = open("file.data", O_RDONLY);
void* ptr = mmap(0, (size_t)240 * 1024 * 1024 * 1024, PROT_READ, MAP_SHARED, fd, 0);
printf("Result = %p\n", ptr);
printf("Errno = %d\n", errno);
}
它输出（使用 `gcc test.c && ./a.out` 编译和运行）：
Result = 0xffffffffffffffff
Errno = 9
`file.data` 是一个 243 GiB 的文件：
```lang-none
$ stat file.data
File: file.data
Size: 260165023654 Blocks: 508135088 IO Block: 4096 regular file
Device: 801h/2049d Inode: 6815790 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ user) Gid: ( 1001/ user)
Access: 2023-05-08 09:22:07.314477587 -0400
Modify: 2023-06-16 07:53:12.275187040 -0400
Change: 2023-06-16 07:53:12.275187040 -0400
Birth: -
其他配置（debian stretch，Linux 5.2.21）：
$ sysctl vm.overcommit_memory
vm.overcommit_memory = 1
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 768178
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 768178
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ free -m
total used free shared buff/cache available
Mem: 192105 671 189213 9 2220 190314
Swap: 0 0 0
我已经遵循的建议：
据我了解，我应该能够mmap这个文件。我将其映射为只读，因此内核可以随时自由地将其全部交换回磁盘。应该有足够的连续内存，因为这是一个64位系统。
如何使mmap调用工作？
程序的/proc/*/maps输出：
2aaaaaa000-2aaaaab000 r-xp 00000000 08:01 6815754 /home/<username>/a.out
2aaacaa000-2aaacab000 r--p 00000000 08:01 6815754 /home/<username>/a.out
2aaacab000-2aaacac000 rw-p 00001000 08:01 6815754 /home/<username>/a.out
3ff7a3a000-3ff7bcf000 r-xp 00000000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7bcf000-3ff7dcf000 ---p 00195000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dcf000-3ff7dd3000 r--p 00195000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dd3000-3ff7dd5000 rw-p 00199000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dd5000-3ff7dd9000 rw-p 00000000 00:00 0
3ff7dd9000-3ff7dfc000 r-xp 00000000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff7fe8000-3ff7fea000 rw-p 00000000 00:00 0
3ff7ff8000-3ff7ffb000 r--p 00000000 00:00 0 [vvar]
3ff7ffb000-3ff7ffc000 r-xp 00000000 00:00 0 [vdso]
3ff7ffc000-3ff7ffd000 r--p 00023000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff7ffd000-3ff7ffe000 rw-p 00024000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff
<details>
<summary>英文:</summary>
I am running the following (minimal reproducing) code:
#include <stdio.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
void main() {
int fd = open("file.data", O_RDONLY);
void* ptr = mmap(0, (size_t)240 * 1024 * 1024 * 1024, PROT_READ, MAP_SHARED, fd, 0);
printf("Result = %p\n", ptr);
printf("Errno = %d\n", errno);
}
It outputs (compiled and run with `gcc test.c && ./a.out`):
Result = 0xffffffffffffffff
Errno = 9
`file.data` is a 243 GiB file:
```lang-none
$ stat file.data
File: file.data
Size: 260165023654 Blocks: 508135088 IO Block: 4096 regular file
Device: 801h/2049d Inode: 6815790 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1001/ user) Gid: ( 1001/ user)
Access: 2023-05-08 09:22:07.314477587 -0400
Modify: 2023-06-16 07:53:12.275187040 -0400
Change: 2023-06-16 07:53:12.275187040 -0400
Birth: -
Other configuration (debian stretch, Linux 5.2.21):
$ sysctl vm.overcommit_memory
vm.overcommit_memory = 1
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 768178
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 768178
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
$ free -m
total used free shared buff/cache available
Mem: 192105 671 189213 9 2220 190314
Swap: 0 0 0
Advice I have already followed:
- The mapping is not MAP_PRIVATE, and not PROT_WRITE
ulimit -vis set to unlimitedvm.overcommit_memory=1
To my understanding, I should be able to mmap this file. I am mapping it as read-only, so the kernel can freely swap it all out back to disk whenever needed. There should be enough continuous memory, as this is a 64-bit system.
How do I make the mmap call work?
The output of /proc/*/maps for the program:
2aaaaaa000-2aaaaab000 r-xp 00000000 08:01 6815754 /home/<username>/a.out
2aaacaa000-2aaacab000 r--p 00000000 08:01 6815754 /home/<username>/a.out
2aaacab000-2aaacac000 rw-p 00001000 08:01 6815754 /home/<username>/a.out
3ff7a3a000-3ff7bcf000 r-xp 00000000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7bcf000-3ff7dcf000 ---p 00195000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dcf000-3ff7dd3000 r--p 00195000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dd3000-3ff7dd5000 rw-p 00199000 08:01 5767499 /lib/x86_64-linux-gnu/libc-2.24.so
3ff7dd5000-3ff7dd9000 rw-p 00000000 00:00 0
3ff7dd9000-3ff7dfc000 r-xp 00000000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff7fe8000-3ff7fea000 rw-p 00000000 00:00 0
3ff7ff8000-3ff7ffb000 r--p 00000000 00:00 0 [vvar]
3ff7ffb000-3ff7ffc000 r-xp 00000000 00:00 0 [vdso]
3ff7ffc000-3ff7ffd000 r--p 00023000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff7ffd000-3ff7ffe000 rw-p 00024000 08:01 5767249 /lib/x86_64-linux-gnu/ld-2.24.so
3ff7ffe000-3ff7fff000 rw-p 00000000 00:00 0
3ffffde000-3ffffff000 rw-p 00000000 00:00 0 [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
答案1
得分: 4
Here is the translated content:
我无法重现您的问题。
我修改了您的程序以添加一个生成示例/测试文件的选项：
-
它可以只是执行
truncate以创建一个大文件。这只需要几分之一秒。 -
然后它可以用真实数据填充它。在我的系统上，这需要大约10分钟来创建一个243 GB的文件。
-
无论在哪种模式下，结果都是相同的。所以，在我看来，快速模式已经足够了（即文件有空洞）。换句话说，任何人都可以在几秒钟内在他们的系统上运行这个程序。
我尝试了我所能想到的各种组合以及其他选项。在任何情况下，我都不能重现。请看下面比较我的系统和您的系统。
在阅读下面的内容后，如果您能想到其他任何想法，我将很高兴在我的系统上尝试以重现您的问题。
以下是修改后的程序：
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#define GBSIZE(_gb) (size_t) _gb * 1024 * 1024 * 1024
#define GBOF(_siz) (double) _siz / (1024 * 1024 * 1024)
int opt_f;
int opt_G;
int opt_v;
const char *file;
char pagebuf[64 * 1024];
#define ONERR(_expr,_reason) \
do { \
if (_expr) { \
printf("ONERR: " #_expr " -- %s\n",strerror(errno)); \
exit(1); \
} \
} while (0)
void genfile(void);
void mapshow(void);
int
main(int argc,char **argv)
{
int fd;
int err;
setlinebuf(stdout);
--argc;
++argv;
for (; argc > 0; --argc, ++argv) {
char *cp = *argv;
if (*cp != '-')
break;
cp += 2;
switch (cp[-1]) {
case 'f':
opt_f = ! opt_f;
break;
case 'G':
opt_G = (*cp != 0) ? strtol(cp,&cp,10) : 243;
break;
case 'v':
opt_v = ! opt_v;
break;
}
}
if (argc == 1)
file = *argv;
else
file = "tmp";
printf("file='%s'\n",file);
if (opt_G) {
genfile();
exit(0);
}
fd = open(file,O_RDONLY);
ONERR(fd < 0,"open/RDONLY");
struct stat st;
err = fstat(fd,&st);
ONERR(err < 0,"fstat");
size_t fsize = st.st_size;
size_t mapsize = fsize - GBSIZE(3);
printf("main: st.st_size=%zu/%.3f mapsize=%zu/%.3F\n",
fsize,GBOF(fsize),mapsize,GBOF(mapsize));
errno = 0;
void *ptr = mmap(0, mapsize, PROT_READ, MAP_SHARED, fd, 0);
printf("Result = %p -- errno=%d %s\n", ptr, errno, strerror(errno));
mapshow();
if (ptr != MAP_FAILED)
munmap(ptr,mapsize);
close(fd);
// remove the temp file
#if 0
unlink(file);
#endif
return 0;
}
void
genfile(void)
{
int fd;
int err;
// get desired file size
size_t mksize = GBSIZE(opt_G);
printf("genfile: unlink ...\n");
unlink(file);
printf("genfile: G=%d mksize=%zu\n",opt_G,mksize);
// create the file
printf("genfile: open ...\n");
fd = open(file,O_WRONLY | O_CREAT,0644);
ONERR(fd < 0,"open/WRONLY");
// truncate
printf("genfile: ftruncate ...\n");
err = ftruncate(fd,mksize);
ONERR(err < 0,"ftruncate");
close(fd);
struct stat st;
err = stat(file,&st);
ONERR(err < 0,"stat");
printf("genfile: st_size=%zu\n",(size_t) st.st_size);
errno = 0;
ONERR(st.st_size != mksize,"st_size");
// fill the file with real data -- not really necessary
if (opt_f) {
printf("genfile: memset ...\n");
fd = open(file, O_RDWR);
ONERR(fd < 0,"open/RDWR");
size_t curlen;
size_t remlen = mksize;
size_t outsize = 0;
int val = 0;
time_t todbeg = time(NULL);
time_t todold = todbeg;
for (; remlen > 0; remlen -= curlen, outsize += curlen, ++val) {
curlen = remlen;
if (curlen > sizeof(pagebuf))
curlen = sizeof(pagebuf);
memset(pagebuf,val,sizeof(pagebuf));
ssize_t xlen = write(fd,pagebuf,curlen);
ONERR(xlen < 0,"write");
time_t todnow = time(NULL);
if ((todnow - todold) >= 1) {
todold = todnow;
double pct = outsize;
pct /= mksize;
pct *= 100;
printf("\rELAPSED: %ld %.3f/%.3f %.3
<details>
<summary>英文:</summary>
**I'm _unable to reproduce_ your issue.**
I modified your program to add an option to generate a sample/test file:
1. It can _just_ do a `truncate` to create a large file. This takes a fraction of a second.
1. It can then fill it in with real data. This takes about 10 minutes to create a 243 GB file on my system.
1. The result is the _same_ in either mode. So, IMO, the quick mode is sufficient (i.e. the file has holes). In other words, anybody can run the program in a matter of seconds on their system.
I tried every combination I could think of this and other options. In no circumstance, could I reproduce. See below for a comparison of my system and yours.
After reading below, if you can think of any other idea, I'd be glad to try it on my system to reproduce your failure.
----------
Here is the modified program:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#define GBSIZE(_gb) (size_t) _gb * 1024 * 1024 * 1024
#define GBOF(_siz) (double) _siz / (1024 * 1024 * 1024)
int opt_f;
int opt_G;
int opt_v;
const char *file;
char pagebuf[64 * 1024];
#define ONERR(_expr,_reason)
do {
if (_expr) {
printf("ONERR: " #_expr " -- %s\n",strerror(errno));
exit(1);
}
} while (0)
void genfile(void);
void mapshow(void);
int
main(int argc,char **argv)
{
int fd;
int err;
setlinebuf(stdout);
--argc;
++argv;
for (; argc > 0; --argc, ++argv) {
char *cp = *argv;
if (*cp != '-')
break;
cp += 2;
switch (cp[-1]) {
case 'f':
opt_f = ! opt_f;
break;
case 'G':
opt_G = (*cp != 0) ? strtol(cp,&cp,10) : 243;
break;
case 'v':
opt_v = ! opt_v;
break;
}
}
if (argc == 1)
file = *argv;
else
file = "tmp";
printf("file='%s'\n",file);
if (opt_G) {
genfile();
exit(0);
}
fd = open(file,O_RDONLY);
ONERR(fd < 0,"open/RDONLY");
struct stat st;
err = fstat(fd,&st);
ONERR(err < 0,"fstat");
size_t fsize = st.st_size;
size_t mapsize = fsize - GBSIZE(3);
printf("main: st.st_size=%zu/%.3f mapsize=%zu/%.3F\n",
fsize,GBOF(fsize),mapsize,GBOF(mapsize));
errno = 0;
void *ptr = mmap(0, mapsize, PROT_READ, MAP_SHARED, fd, 0);
printf("Result = %p -- errno=%d %s\n", ptr, errno, strerror(errno));
mapshow();
if (ptr != MAP_FAILED)
munmap(ptr,mapsize);
close(fd);
// remove the temp file
#if 0
unlink(file);
#endif
return 0;
}
void
genfile(void)
{
int fd;
int err;
// get desired file size
size_t mksize = GBSIZE(opt_G);
printf("genfile: unlink ...\n");
unlink(file);
printf("genfile: G=%d mksize=%zu\n",opt_G,mksize);
// create the file
printf("genfile: open ...\n");
fd = open(file,O_WRONLY | O_CREAT,0644);
ONERR(fd < 0,"open/WRONLY");
// truncate
printf("genfile: ftruncate ...\n");
err = ftruncate(fd,mksize);
ONERR(err < 0,"ftruncate");
close(fd);
struct stat st;
err = stat(file,&st);
ONERR(err < 0,"stat");
printf("genfile: st_size=%zu\n",(size_t) st.st_size);
errno = 0;
ONERR(st.st_size != mksize,"st_size");
// fill the file with real data -- not really necessary
if (opt_f) {
printf("genfile: memset ...\n");
fd = open(file, O_RDWR);
ONERR(fd < 0,"open/RDWR");
size_t curlen;
size_t remlen = mksize;
size_t outsize = 0;
int val = 0;
time_t todbeg = time(NULL);
time_t todold = todbeg;
for (; remlen > 0; remlen -= curlen, outsize += curlen, ++val) {
curlen = remlen;
if (curlen > sizeof(pagebuf))
curlen = sizeof(pagebuf);
memset(pagebuf,val,sizeof(pagebuf));
ssize_t xlen = write(fd,pagebuf,curlen);
ONERR(xlen < 0,"write");
time_t todnow = time(NULL);
if ((todnow - todold) >= 1) {
todold = todnow;
double pct = outsize;
pct /= mksize;
pct *= 100;
printf("\rELAPSED: %ld %.3f/%.3f %.3f%%",
todnow - todbeg,GBOF(outsize),GBOF(mksize),pct);
fflush(stdout);
}
}
printf("\n");
close(fd);
}
}
void
mapshow(void)
{
char file[100];
char buf[1000];
printf("\n");
sprintf(file,"/proc/%d/maps",getpid());
FILE *xfsrc = fopen(file,"r");
ONERR(xfsrc == NULL,"fopen/maps");
while (1) {
if (fgets(buf,sizeof(buf),xfsrc) == NULL)
break;
fputs(buf,stdout);
}
fclose(xfsrc);
}
----------
Here is my configuration:
```lang-none
COMMAND: uname -r
5.3.11-100.fc29.x86_64
COMMAND: sysctl vm.overcommit_memory
vm.overcommit_memory = 0
COMMAND: ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 47763
max locked memory (kbytes, -l) 16384
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 47763
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
COMMAND: free -m
total used free shared buff/cache available
Mem: 11972 3744 750 68 7477 7842
Swap: 122879 1147 121732
Slight differences:
-
You have 192 GB of RAM. But, I only have 12 GB of RAM. This difference should work in your favor. But, it doesn't. The program works on my system that has less than 1/10 of the amount of RAM.
-
I have a 128 GB swap disk. But, I reran the program after doing
swapoff -ato disable all swap disks. There was no difference in program operation. -
vm.overcommit_memoryis 0. But, I set it to 1 and there was no difference in program operation. -
On my
vm.mmap_min_addris 65536 (seeTASK_SIZEbelow) -
My computer system is over ten years old.
-
I'm (probably) running a much older kernel version.
At the time of the test, I had:
- A few
gnome-terminalwindows firefoxwith pages on SOthunderbird- A few background shell programs (of my own design).
Because of my much smaller RAM, I have to dispute neo-jgrec's answer:
On an x86 (64 bit) system, TASK_SIZE can be either:
- Normal system:
1ul << 47131,072 GB (128 TB) - 5 level paging enabled:
1ul << 5667,108,864 GB (65,536 TB)
Even using the smaller address value we are clearly not going beyond TASK_SIZE
I've done mmap on many 100+ GB files, in the past, without issue. For example, see my answer: read line by line in the most efficient way platform specific
Here is the stat of the file:
File: tmp
Size: 260919263232 Blocks: 509608032 IO Block: 4096 regular file
Device: 901h/2305d Inode: 180624922 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ user) Gid: ( 1000/ user)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2023-06-18 15:39:51.253702772 -0400
Modify: 2023-06-18 15:58:43.512226035 -0400
Change: 2023-06-18 15:58:43.512226035 -0400
Birth: -
Here is the program output:
file='tmp'
main: st.st_size=260919263232/243.000 mapsize=257698037760/240.000
Result = 0x7edf00cf9000 -- errno=0 Success
00400000-00401000 r--p 00000000 09:01 180624914 /home/user/bigmmap/orig
00401000-00402000 r-xp 00001000 09:01 180624914 /home/user/bigmmap/orig
00402000-00403000 r--p 00002000 09:01 180624914 /home/user/bigmmap/orig
00403000-00404000 r--p 00002000 09:01 180624914 /home/user/bigmmap/orig
00404000-00405000 rw-p 00003000 09:01 180624914 /home/user/bigmmap/orig
00405000-00415000 rw-p 00000000 00:00 0
013bb000-013dc000 rw-p 00000000 00:00 0 [heap]
7edf00cf9000-7f1b00cf9000 r--s 00000000 09:01 180624922 /home/user/bigmmap/tmp
7f1b00cf9000-7f1b00d1b000 r--p 00000000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00d1b000-7f1b00e68000 r-xp 00022000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00e68000-7f1b00eb4000 r--p 0016f000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00eb4000-7f1b00eb5000 ---p 001bb000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00eb5000-7f1b00eb9000 r--p 001bb000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00eb9000-7f1b00ebb000 rw-p 001bf000 09:00 1202975 /usr/lib64/libc-2.28.so
7f1b00ebb000-7f1b00ec1000 rw-p 00000000 00:00 0
7f1b00f16000-7f1b00f17000 r--p 00000000 09:00 1182318 /usr/lib64/ld-2.28.so
7f1b00f17000-7f1b00f37000 r-xp 00001000 09:00 1182318 /usr/lib64/ld-2.28.so
7f1b00f37000-7f1b00f3f000 r--p 00021000 09:00 1182318 /usr/lib64/ld-2.28.so
7f1b00f3f000-7f1b00f40000 r--p 00028000 09:00 1182318 /usr/lib64/ld-2.28.so
7f1b00f40000-7f1b00f41000 rw-p 00029000 09:00 1182318 /usr/lib64/ld-2.28.so
7f1b00f41000-7f1b00f42000 rw-p 00000000 00:00 0
7fff0d6d7000-7fff0d6f8000 rw-p 00000000 00:00 0 [stack]
7fff0d75a000-7fff0d75d000 r--p 00000000 00:00 0 [vvar]
7fff0d75d000-7fff0d75e000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
答案2
得分: 1
你正在经历的问题源于尽管你的系统是64位系统，但内核仍然有一个取决于你系统架构的寻址限制。
默认情况下，Linux将可寻址内存空间的一半分配给内核，一半分配给用户。所以，对于64位系统，内核和用户空间都将有2^63字节。
然而，内核并未使用整个空间。内核使用可寻址内存的一部分进行内存映射，范围从mmap_min_addr到TASK_SIZE。TASK_SIZE通常在内核中设置为一个特定的值，取决于你系统的架构，可能小于最大可寻址空间。
你的mmap请求可能会失败，因为它试图分配的内存超过了你系统的TASK_SIZE。如果你尝试一次性mmap 240GiB，它可能会超过你系统的TASK_SIZE。
一种解决方法是使用循环以较小的块映射文件，直到映射整个文件。以下是如何执行此操作的示例：
#include <stdio.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#define CHUNK_SIZE ((size_t)64 * 1024 * 1024 * 1024) //64GiB
int main() {
int fd = open("file.data", O_RDONLY);
if (fd < 0) {
perror("open");
return EXIT_FAILURE;
}
struct stat statbuf;
if (fstat(fd, &statbuf) < 0) {
perror("fstat");
return EXIT_FAILURE;
}
size_t file_size = statbuf.st_size;
size_t offset = 0;
while (offset < file_size) {
size_t size = (file_size - offset > CHUNK_SIZE) ? CHUNK_SIZE : file_size - offset;
void* ptr = mmap(0, size, PROT_READ, MAP_SHARED, fd, offset);
if (ptr == MAP_FAILED) {
perror("mmap");
return EXIT_FAILURE;
}
// 在这里处理内存
munmap(ptr, size);
offset += size;
}
close(fd);
return 0;
}
这段代码以64GiB的块读取文件。最好根据你的具体情况调整块大小。请始终检查系统调用的返回值以处理任何错误。错误消息将更具描述性和信息性。
在移动到下一个块之前，记得在处理完文件的某个部分后调用munmap()。
英文:
The problem you are experiencing comes from the fact that even though your system is a 64-bit system, the kernel still has an addressing limit, which depends on your system's architecture.
By default, Linux allocates half the addressable memory space to the kernel and half to the user. So, for a 64-bit system, this would be 2^63 bytes for the kernel and the same amount for user space.
However, the kernel doesn't use this whole space. The kernel uses a range of the addressable memory for memory mapping, which is from mmap_min_addr to TASK_SIZE. TASK_SIZE is typically set in the kernel to a certain value, depending on the architecture of your system, which might be less than the maximum addressable space.
Your mmap request is likely failing because it's trying to allocate more memory than your system's TASK_SIZE. If you try to mmap 240GiB at once, it might exceed the TASK_SIZE on your system.
One solution would be to mmap smaller chunks of the file in a loop until you've mmapped the whole file. Here is an example of how to do that:
#include <stdio.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
#define CHUNK_SIZE ((size_t)64 * 1024 * 1024 * 1024) //64GiB
int main() {
int fd = open("file.data", O_RDONLY);
if (fd < 0) {
perror("open");
return EXIT_FAILURE;
}
struct stat statbuf;
if (fstat(fd, &statbuf) < 0) {
perror("fstat");
return EXIT_FAILURE;
}
size_t file_size = statbuf.st_size;
size_t offset = 0;
while (offset < file_size) {
size_t size = (file_size - offset > CHUNK_SIZE) ? CHUNK_SIZE : file_size - offset;
void* ptr = mmap(0, size, PROT_READ, MAP_SHARED, fd, offset);
if (ptr == MAP_FAILED) {
perror("mmap");
return EXIT_FAILURE;
}
// do something with the memory here
munmap(ptr, size);
offset += size;
}
close(fd);
return 0;
}
This code reads the file in chunks of 64GiB. It would be best to adapt the chunk size to your specific case. You should always check the return values of the system calls to handle any errors. The error messages will be more descriptive and informative.
Remember to call munmap() when you're done with a section of the file before moving onto the next one.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论