ARM:启动/唤醒/唤醒其他 CPU 内核/AP 并传递执行起始地址? [英] ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

问题描述

在过去的 3-4 天里，我一直在思考这个问题，但找不到合适的解释性文档(来自 ARM 或非官方)来帮助我.我有一个 ODROID-XU 板(big.LITTLE 2 x Cortex-A15 + 2 x Cortex-A7) 板，我正在尝试更多地了解 ARM 架构.在我的实验"代码中，我现在已经到了要从它们的 WFI(等待中断)状态唤醒其他内核的阶段.

I've been banging my head with this for the last 3-4 days and I can't find a DECENT explanatory documentation (from ARM or unofficial) to help me. I've got an ODROID-XU board (big.LITTLE 2 x Cortex-A15 + 2 x Cortex-A7) board and I'm trying to understand a bit more about the ARM architecture. In my "experimenting" code I've now arrived at the stage where I want to WAKE UP THE OTHER CORES FROM THEIR WFI (wait-for-interrupt) state.

我仍在努力寻找的缺失信息是:

The missing information I'm still trying to find is:

1. 在获取内存映射GIC的基地址时，我明白我需要阅读CBAR；但是没有任何文档解释 CBAR 中的位(2 个 PERIPHBASE 值)应该如何排列才能到达最终的 GIC 基地址

1. When getting the base address of the memory-mapped GIC I understand that I need to read CBAR; But no piece of documentation explains how the bits in CBAR (the 2 PERIPHBASE values) should be arranged to get to the final GIC base address

2. 当通过 GICD_SGIR 寄存器发送 SGI 时，我应该选择 0 到 15 之间的哪个中断 ID?重要吗?

2. When sending an SGI through the GICD_SGIR register, what interrupt ID between 0 and 15 should I choose? Does it matter?

3. 当通过 GICD_SGIR 寄存器发送 SGI 时，我如何告诉其他内核从哪里开始执行?

3. When sending an SGI through the GICD_SGIR register, how can I tell the other cores WHERE TO START EXECUTION FROM?

4.我的代码由 U-BOOT 引导加载程序加载这一事实如何影响此上下文?

4. How does the fact that my code is loaded by the U-BOOT bootloader affect this context?

Cortex-A 系列程序员指南 v3.0(可在此处找到:链接)在第 22.5.2 节(Linux 中的 SMP 启动，页面 271)中陈述了以下内容:

The Cortex-A Series Programmer's Guide v3.0 (found here: link) states the following in section 22.5.2 (SMP boot in Linux, page 271):

当主核心启动时，辅助核心将保持待机状态，使用WFI 指令.它(主核)将为次核提供启动地址，并使用处理器间中断 (IPI)，意思是通过 GIC 发出信号的 SGI

Linux 如何做到这一点?文档-S 没有提供关于它将为辅助核心提供启动地址"的任何其他详细信息.

How does Linux do that? The documentation-S don't give any other details regarding "It will provide a startup address to the secondary cores".

我的挫败感与日俱增，如果您能得到答案，我将不胜感激.预先非常感谢您！

My frustration is growing and I'd be very grateful for answers. Thank you very much in advance!

额外细节

我使用的文档:

• ARMv7-A&R 架构参考手册
• Cortex-A15 TRM(技术参考手册)
• Cortex-A15 MPCore TRM
• Cortex-A 系列程序员指南 v3.0
• GICv2 架构规范

我现在所做的:

• UBOOT 在 0x40008000 处加载我；我已经设置了转换表 (TTB)，相应地编写了 TTBR0 和 TTBCR，并将 0x40008000 映射到 0x8000_0000 (2GB)，所以我还启用了 MMU
• 设置我自己的异常处理程序
• 我通过串口获得了 Printf 功能(ODROID-XU 上的 UART2)

以上所有似乎都可以正常工作.

All the above seems to work properly.

我现在正在做的事情:

• 获取 GIC 基地址 => 在我读取 CBAR 的那一刻，我简单地将其值与 0xFFFF8000 进行 AND (&) 并将其用作 GIC 基地址，尽管我几乎可以肯定这不是对
• 通过将值 0x1 写入 GICD_CTLR 来启用 GIC 分发器(在 GIC 基地址的偏移量 0x1000 处?)
• 使用以下参数构造一个 SGI:Group = 0, ID = 0, TargetListFilter = "All CPUs except Me" 并通过 GICD_SGIR GIC 寄存器发送(写入)
• 由于我没有为其他内核传递任何执行起始地址，所以这一切之后什么也没有发生

....更新....

我已经开始查看 Linux 内核和 QEMU 源代码以寻找答案.这是我发现的(如果我错了，请纠正我):

I've started looking at the Linux kernel and QEMU source codes in search for an answer. Here's what I found out (please correct me if I'm wrong):

• 当给电路板加电时，所有内核都从复位向量开始执行
• 软件(固件)组件在辅助核心上执行 WFI 和一些其他代码，当后者想要唤醒时，这些代码将充当这些辅助核心和主要核心之间的协议再起来
• 例如，EnergyCore ECX-1000 (Highbank) 板上使用的协议如下:

• When powering up the board ALL THE CORES start executing from the reset vector
• A software (firmware) component executes WFI on the secondary cores and some other code that will act as a protocol between these secondary cores and the primary core, when the latter wants to wake them up again
• For example, the protocol used on the EnergyCore ECX-1000 (Highbank) board is as follows:

**(1)** 辅助核心进入 WFI 以及何时

**(2)** 主核发送一个 SGI 来唤醒它们

**(3)** 他们检查地址 (0x40 + 0x10 * coreid) 处的值是否为非空；

**(4)** 如果它是非空的，他们就用它作为地址跳转到(执行一个BX)

**(5)** 否则，他们通过重新执行 WFI 重新进入待机状态

**(6)** 所以，如果我有一个 EnergyCore ECX-1000 板，我应该写 (0x40 + 0x10 * coreid) 和我希望每个内核跳转到的地址并发送一个SGI

问题:

• 1.执行此操作的软件组件是什么?是我写在 SD 卡上的 BL1 二进制文件，还是 U-BOOT?
• 2.据我了解，该软件协议因板而异.是这样，还是只依赖底层处理器?
• 3.我在哪里可以找到有关此协议的适用于选择一个 ARM 板的信息?- 我可以在 ARM 官方网站或董事会网页上找到它吗?

推荐答案

好的，我回来了，宝贝.结论如下:

Ok, I'm back baby. Here are the conclusions:

• 让 CPU 进入睡眠状态的软件组件是引导加载程序(在我的例子中是 U-Boot)
• Linux 以某种方式知道引导加载程序如何执行此操作(在每个板的 Linux 内核中硬编码)并且知道如何再次唤醒它们

对于我的 ODROID-XU 开发板，描述此过程的来源是 UBOOT ODROID-v2012.07 和在这里找到的 linux 内核:LINUX ODROIDXU-3.4.y(它会有如果我从分支 odroid-3.12.y 查看内核版本会更好，因为前者没有不会启动所有 8 个处理器，只启动其中的 4 个，但后者启动).

For my ODROID-XU board the sources describing this process are UBOOT ODROID-v2012.07 and the linux kernel found here: LINUX ODROIDXU-3.4.y (it would have been better if I looked into kernel version from the branch odroid-3.12.y since the former doesn't start all of the 8 processors, just 4 of them but the latter does).

无论如何，这是我想出的源代码，我会发布上述源代码树中的相关源文件，这些文件有助于我以后编写此代码:

Anyway, here's the source code I've come up with, I'll post the relevant source files from the above source code trees that helped me writing this code afterwards:

typedef unsigned int DWORD;
typedef unsigned char BOOLEAN;
#define FAILURE (0)
#define SUCCESS (1)
#define NR_EXTRA_CPUS (3) // actually 7, but this kernel version can't wake them up all -> check kernel version 3.12 if you need this

// Hardcoded in the kernel and in U-Boot; here I've put the physical addresses for ease
// In my code (and in the linux kernel) these addresses are actually virtual
// (thus the 'VA' part in S5P_VA_...); note: mapped with memory type DEVICE
#define S5P_VA_CHIPID (0x10000000)
#define S5P_VA_SYSRAM_NS (0x02073000)
#define S5P_VA_PMU (0x10040000)
#define EXYNOS_SWRESET ((DWORD) S5P_VA_PMU + 0x0400)
// Other hardcoded values
#define EXYNOS5410_REV_1_0 (0x10)
#define EXYNOS_CORE_LOCAL_PWR_EN (0x3)

BOOLEAN BootAllSecondaryCPUs(void* CPUExecutionAddress){

// 1. Get bootBase (the address where we need to write the address where the woken CPUs will jump to)
//    and powerBase (we also need to power up the cpus before waking them up (?))
DWORD bootBase, powerBase, powerOffset, clusterID;

asm volatile ("mrc p15, 0, %0, c0, c0, 5" : "=r" (clusterID));
clusterID = (clusterID >> 8);
powerOffset = 0;
if( (*(DWORD*)S5P_VA_CHIPID & 0xFF) < EXYNOS5410_REV_1_0 )
{
    if( (clusterID & 0x1) == 0 ) powerOffset = 4;
}
else if( (clusterID & 0x1) != 0 ) powerOffset = 4;

bootBase = S5P_VA_SYSRAM_NS + 0x1C;
powerBase = (S5P_VA_PMU + 0x2000) + (powerOffset * 0x80);

// 2. Power up each CPU, write bootBase and send a SEV (they are in WFE [wait-for-event] standby state)
for (i = 1; i <= NR_EXTRA_CPUS; i++)
{
    // 2.1 Power up this CPU
    powerBase += 0x80;
    DWORD powerStatus = *(DWORD*)( (DWORD) powerBase + 0x4);

    if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == 0)
    {
        *(DWORD*) powerBase = EXYNOS_CORE_LOCAL_PWR_EN;
        for (i = 0; i < 10; i++) // 10 millis timeout
        {
            powerStatus = *(DWORD*)((DWORD) powerBase + 0x4);
            if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == EXYNOS_CORE_LOCAL_PWR_EN)
                break;
            DelayMilliseconds(1); // not implemented here, if you need this, post a comment request 
        }
        if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) != EXYNOS_CORE_LOCAL_PWR_EN)
            return FAILURE;
    }
    if ( (clusterID & 0x0F) != 0 )
    {
        if ( *(DWORD*)(S5P_VA_PMU + 0x0908) == 0 )
        do { DelayMicroseconds(10); } // not implemented here, if you need this, post a comment request
        while (*(DWORD*)(S5P_VA_PMU + 0x0908) == 0);
        *(DWORD*) EXYNOS_SWRESET = (DWORD)(((1 << 20) | (1 << 8)) << i);
    }

    // 2.2 Write bootBase and execute a SEV to finally wake up the CPUs
    asm volatile ("dmb" : : : "memory");
    *(DWORD*) bootBase = (DWORD) CPUExecutionAddress;
    asm volatile ("isb");
    asm volatile ("\n   dsb\n   sev\n   nop\n");
}
return SUCCESS;
}

这成功唤醒了 7 个辅助 CPU 中的 3 个.

现在是 u-boot 和 linux 内核中相关源文件的简短列表:

And now for that short list of relevant source files in u-boot and the linux kernel:

• UBOOT:lowlevel_init.S - 注意 363-369 行，辅助 CPU 如何在 WFE 中等待 _hotplug_addr 处的值不为零，并且跳到它；_hotplug_addr实际上就是上面代码中的bootBase；还有282-285行告诉我们_hotplug_addr要重定位在CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base(_hotplug_addr - nscode_base 是 0x1C 并且 CONFIG_PHY_IRAM_NS_BASE 是 0x02073000，因此上述 linux 内核中的硬编码)

• UBOOT: lowlevel_init.S - notice lines 363-369, how the secondary CPUs wait in a WFE for the value at _hotplug_addr to be non-zeroed and to jump to it; _hotplug_addr is actually bootBase in the above code; also lines 282-285 tell us that _hotplug_addr is to be relocated at CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base (_hotplug_addr - nscode_base is 0x1C and CONFIG_PHY_IRAM_NS_BASE is 0x02073000, thus the above hardcodings in the linux kernel)

LINUX 内核:通用 - smp.c(查看函数__cpu_up)，平台特定(odroid-xu):platsmp.c(函数boot_secondary，由泛型__cpu_up调用; 也看看 platform_smp_prepare_cpus [在底部] => 这是实际设置引导基础和电源基础值的函数)

LINUX KERNEL: generic - smp.c (look at function __cpu_up), platform specific (odroid-xu): platsmp.c (function boot_secondary, called by generic __cpu_up; also look at platform_smp_prepare_cpus [at the bottom] => that's the function that actually sets the boot base and power base values)

这篇关于ARM:启动/唤醒/唤醒其他 CPU 内核/AP 并传递执行起始地址?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！