英文:
ARMv8A hypervisor - PCI MMU fault
问题
我正在尝试在ARMv8A（QEMU版本6.2.0上的Cortex A53）上实现一个最小的hypervisor。我已经在EL2中编写了一个最小的hypervisor代码，Linux在EL1中成功启动。现在我想启用stage-2 MMU。我已经在stage2中编写了基本的页表（只包括将1GB RAM映射到必要的页表项）。如果我在DTB中禁用PCI，内核会成功启动。以下是QEMU命令行：
qemu-system-aarch64 -machine virt,gic-version=2,virtualization=on -cpu cortex-a53 -nographic -smp 1 -m 4096 -kernel hypervisor/bin/hypervisor.elf -device loader,file=linux-5.10.155/arch/arm64/boot/Image,addr=0x80200000 -device loader,file=1gb_1core.dtb,addr=0x88000000
当在DTB中启用PCI时，我遇到了以下内核恐慌：
[ 0.646801] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[ 0.647909] Unable to handle kernel paging request at virtual address 0000000093810004
[ 0.648109] Mem abort info:
[ 0.648183] ESR = 0x96000004
[ 0.648282] EC = 0x25: DABT (current EL), IL = 32 bits
[ 0.648403] SET = 0, FnV = 0
[ 0.648484] EA = 0, S1PTW = 0
[ 0.648568] Data abort info:
[ 0.648647] ISV = 0, ISS = 0x00000004
[ 0.648743] CM = 0, WnR = 0
[ 0.648885] [0000000093810004] user address but active_mm is swapper
[ 0.653399] Call trace:
[ 0.653598] pci_generic_config_read+0x38/0xe0
[ 0.653729] pci_bus_read_config_dword+0x80/0xe0
[ 0.653845] pci_bus_generic_read_dev_vendor_id+0x34/0x1b0
[ 0.653974] pci_bus_read_dev_vendor_id+0x4c/0x70
[ 0.654090] pci_scan_single_device+0x80/0x100
我在'pci_generic_config_read'中设置了GDB断点，并观察到故障指令是：
>0xffff80001055d5c8 <pci_generic_config_read+56> ldr w1, [x0]
寄存器X0的值如下：
(gdb) p /x $x0
$4 = 0xffff800020000000
主机硬件配置为总共有4GB，Linux（客户机）通过命令行和DTB提供了1GB。这是一个单核系统，'kaslr'已禁用。
DTB中包含PCI部分的摘录如下：
pcie@10000000 {
interrupt-map-mask = <0x1800 0x00 0x00 0x07>;
interrupt-map = <0x00 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x03 0x04 0x00 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x04 0x04 0x00 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x05 0x04 0x00 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x06 0x04 0x800 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x04 0x04 0x800 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x05 0x04 0x800 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x06 0x04 0x800 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x03 0x04 0x1000 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x05 0x04 0x1000 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x06 0x04 0x1000 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x03 0x04 0x1000 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x04 0x04 0x1800 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x06 0x04 0x1800 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x03 0x04 0x1800 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x04 0x04 0x1800 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x05 0x04>;
#interrupt-cells = <0x01>;
ranges = <0x1000000 0x00 0x00
<details>
<summary>英文:</summary>
I am trying to implement a minimal hypervisor on ARMv8A (Cortext A53 on QEMU Version 6.2.0).I have written a minimal hypervisor code in EL2 and the Linux boots successfully in EL1. Now I want to enable stage-2 MMU. I have written basic page tables in stage2 (Only the necessary page table entries to map to 1GB RAM). If I disable PCI in DTB the kernel boots successfully.The QEMU command line is given below.
qemu-system-aarch64 -machine virt,gic-version=2,virtualization=on -cpu cortex-a53 -nographic -smp 1 -m 4096 -kernel hypvisor/bin/hypervisor.elf -device loader,file=linux-5.10.155/arch/arm64/boot/Image,addr=0x80200000 -device loader,file=1gb_1core.dtb,addr=0x88000000
When the PCI is enabled in DTB, I am getting a kernel panic as shown below.
[ 0.646801] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]
[ 0.647909] Unable to handle kernel paging request at virtual address 0000000093810004
[ 0.648109] Mem abort info:
[ 0.648183] ESR = 0x96000004
[ 0.648282] EC = 0x25: DABT (current EL), IL = 32 bits
[ 0.648403] SET = 0, FnV = 0
[ 0.648484] EA = 0, S1PTW = 0
[ 0.648568] Data abort info:
[ 0.648647] ISV = 0, ISS = 0x00000004
[ 0.648743] CM = 0, WnR = 0
[ 0.648885] [0000000093810004] user address but active_mm is swapper
[ 0.653399] Call trace:
[ 0.653598] pci_generic_config_read+0x38/0xe0
[ 0.653729] pci_bus_read_config_dword+0x80/0xe0
[ 0.653845] pci_bus_generic_read_dev_vendor_id+0x34/0x1b0
[ 0.653974] pci_bus_read_dev_vendor_id+0x4c/0x70
[ 0.654090] pci_scan_single_device+0x80/0x100
I set a GDB breakpoint in 'pci_generic_config_read' and observed that the faulting instruction is
>0xffff80001055d5c8 <pci_generic_config_read+56> ldr w1, [x0]
The value of register X0 is given below
(gdb) p /x $x0
$4 = 0xffff800020000000
The hardware (host) is configured to have 4GB in total and the Linux (guest) is supplied 1GB through command line and DTB. This is a single core system with 'kaslr' disabled.
Excerpt from the DTB containing PCI part is given below.
pcie@10000000 {
interrupt-map-mask = <0x1800 0x00 0x00 0x07>;
interrupt-map = <0x00 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x03 0x04 0x00 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x04 0x04 0x00 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x05 0x04 0x00 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x06 0x04 0x800 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x04 0x04 0x800 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x05 0x04 0x800 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x06 0x04 0x800 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x03 0x04 0x1000 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x05 0x04 0x1000 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x06 0x04 0x1000 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x03 0x04 0x1000 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x04 0x04 0x1800 0x00 0x00 0x01 0x8001 0x00 0x00 0x00 0x06 0x04 0x1800 0x00 0x00 0x02 0x8001 0x00 0x00 0x00 0x03 0x04 0x1800 0x00 0x00 0x03 0x8001 0x00 0x00 0x00 0x04 0x04 0x1800 0x00 0x00 0x04 0x8001 0x00 0x00 0x00 0x05 0x04>;
#interrupt-cells = <0x01>;
ranges = <0x1000000 0x00 0x00 0x00 0x3eff0000 0x00 0x10000 0x2000000 0x00 0x10000000 0x00 0x10000000 0x00 0x2eff0000 0x3000000 0x80 0x00 0x80 0x00 0x80 0x00>;
reg = <0x40 0x10000000 0x00 0x10000000>;
msi-parent = <0x8002>;
dma-coherent;
bus-range = <0x00 0xff>;
linux,pci-domain = <0x00>;
#size-cells = <0x02>;
#address-cells = <0x03>;
device_type = "pci";
compatible = "pci-host-ecam-generic";
};
If my interpretation of DTB is right, the PCI device is mapped to the address range '0x40_1000_0000' (offset) '0x1000_0000' (size 256MB). that is, it starts from 100GB in the physical address space.
I have written a page table entry mapping to this physical address as well (as a device memory).
Is it right for the PCI to map to such a higher address in the physical address space? Any hints on debugging this issue is greatly appreciated.
</details>
# 答案1
**得分**: 0
是的，对于64位CPU，这是找到PCI控制器ECAM区域的预期位置。virt板将一些“大型”设备内存区域放在4GB标记之外（具体来说，是PCIE ECAM、第二个PCIE MMIO窗口和用于123号CPU以上的重新分配器）。（如果您愿意，可以使用-machine highmem=off关闭此功能，尽管这会限制您可以分配给虚拟机的RAM量为3GB。）
根据您的虚拟化管理程序的操作，您可能希望或不希望它直接与主机PCI控制器通信。
<details>
<summary>英文:</summary>
Yes, for a 64-bit CPU this is the expected place to find the PCI controller ECAM region. The virt board puts some "large" device memory regions beyond the 4GB mark (specifically, PCIE ECAM, a seconD PCIE MMIO window, and redistributors for CPUs above 123). (You can turn this off with -machine highmem=off if you like, though that will limit the amount of RAM you can give the VM to 3GB.)
Depending on what your hypervisor is doing, you might or might not want it to be talking directly to the host PCI controller anyway.
</details>
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论