ARM：启动/唤醒/ Bringup其他CPU核心/接入点，并通过执行起始地址？ [英] ARM: Start/Wakeup/Bringup the other CPU cores/APs and pass execution start address?

问题描述

我一直在敲打我的头这在过去的3-4天，我找不到一个像样的说明性文件（从ARM或非官方的）来帮助我。
我有一个 ODROID-徐板（的big.LITTLE 2×的Cortex-A15 + 2×的Cortex-A7）板，我想了解更多的关于ARM架构。在我的实验code现在我已经到了，我想的唤醒了其他内核从他们的WFI（等待换中断）状态的阶段。

I've been banging my head with this for the last 3-4 days and I can't find a DECENT explanatory documentation (from ARM or unofficial) to help me. I've got an ODROID-XU board (big.LITTLE 2 x Cortex-A15 + 2 x Cortex-A7) board and I'm trying to understand a bit more about the ARM architecture. In my "experimenting" code I've now arrived at the stage where I want to WAKE UP THE OTHER CORES FROM THEIR WFI (wait-for-interrupt) state.

缺少的信息我还在试图寻找的是：

The missing information I'm still trying to find is:

1 ：当越来越内存映射GIC我明白，我需要阅读CBAR的基址。但是，没有一块文档解释了CBAR位（2 PERIPHBASE值）应如何安排才能到最后的GIC基地址

1. When getting the base address of the memory-mapped GIC I understand that I need to read CBAR; But no piece of documentation explains how the bits in CBAR (the 2 PERIPHBASE values) should be arranged to get to the final GIC base address

2 ：当通过GICD_SGIR寄存器发送SGI，什么中断0到15之间的编号，我应该选择？有什么关系？

2. When sending an SGI through the GICD_SGIR register, what interrupt ID between 0 and 15 should I choose? Does it matter?

3 ：当通过GICD_SGIR寄存器发送SGI，我怎么能告诉其他内核的其中，启动执行FROM

3. When sending an SGI through the GICD_SGIR register, how can I tell the other cores WHERE TO START EXECUTION FROM?

4。这是如何我code是由U-BOOT引导程序加载的事实影响此背景下？

4. How does the fact that my code is loaded by the U-BOOT bootloader affect this context?

在的Cortex-A系列程序员指南3.0版（这里找到：的链接）规定在部分22.5.2 （SMP Linux中的引导，页面的 271 ）以下内容：

The Cortex-A Series Programmer's Guide v3.0 (found here: link) states the following in section 22.5.2 (SMP boot in Linux, page 271):

虽然主要核心引导，辅助核心将在待机状态下进行，使用
  WFI指令。它（主芯）将提供一个启动地址给二次铁芯和使用唤醒他们
  处理器间中断（IPI），意味着SGI通过GIC 的

While the primary core is booting, the secondary cores will be held in a standby state, using the WFI instruction. It (the primary core) will provide a startup address to the secondary cores and wake them using an Inter-Processor Interrupt(IPI), meaning an SGI signalled through the GIC

Linux如何做到这一点？该文档 - 取值不给有关任何其他细节它将提供一个启动地址到辅助内核

How does Linux do that? The documentation-S don't give any other details regarding "It will provide a startup address to the secondary cores".

我的挫折成长，我会寻找答案非常感激。
非常感谢你提前！

My frustration is growing and I'd be very grateful for answers. Thank you very much in advance!

额外的细节

我的文档使用：

• 的ARMv7-A和; R体系结构参考手册


• 的Cortex-A15 TRM（技术参考手册）


• 的Cortex-A15 MPCore的TRM


• 的Cortex-A系列程序员指南3.0版


• GICv2体系结构规范

• ARMv7-A&R Architecture Reference Manual
• Cortex-A15 TRM (Technical Reference Manual)
• Cortex-A15 MPCore TRM
• Cortex-A Series Programmer's Guide v3.0
• GICv2 Architecture Specification

什么我现在做的：

• 加载UBOOT在我0x40008000;我已经设置了转换表（TTBS），书面TTBR0和TTBCR因此并映射到0x40008000 0x8000_0000（2GB），所以我也启用了MMU


• 我的
的建立异常处理程序

• 我有printf的功能，通过串行（UART2上ODROID-XU）

• UBOOT loads me at 0x40008000; I've set-up Translation Tables (TTBs), written TTBR0 and TTBCR accordingly and mapped 0x40008000 to 0x8000_0000 (2GB), so I also enabled the MMU
• Set-up exception handlers of my own
• I've got Printf functionality over the serial (UART2 on ODROID-XU)

以上所有似乎正常工作。

All the above seems to work properly.

我试图现在要做的：

• 获取的GIC基地址=>此刻我读CBAR和我简单（安培）其价值与0xFFFF8000并以此作为GIC基址，但我几乎可以肯定这是不是权


• 与价值为0x1书面方式GICD_CTLR启用GIC分配器（从GIC偏移地址为0x1000），


• 构造具有以下PARAMS的SGI：组= 0，ID = 0，TargetListFilter =所有CPU除了我，并通过GICD_SGIR GIC发送（写）寄存器


• 因为我没有为其他内核传递的任何执行的起始地址，没有这一切
后会发生

• Get the GIC base address => at the moment I read CBAR and I simply AND (&) its value with 0xFFFF8000 and use this as the GIC base address, although I'm almost sure this ain't right
• Enable the GIC distributor (at offset 0x1000 from GIC base address?), by writting GICD_CTLR with the value 0x1
• Construct an SGI with the following params: Group = 0, ID = 0, TargetListFilter = "All CPUs Except Me" and send it (write it) through the GICD_SGIR GIC register
• Since I haven't passed any execution start address for the other cores, nothing happens after all this

....更新....

我已经开始寻找Linux内核和QEMU源$ C ​​$ CS里寻找答案。下面是我发现了什么（请纠正我，如果我错了）：

I've started looking at the Linux kernel and QEMU source codes in search for an answer. Here's what I found out (please correct me if I'm wrong):

• 当通电董事会所有内核开始从复位向量执行


• A 软件（固件）的组件执行WFI在辅助内核和其他一些code，将作为这些二级核心和主核心之间的协议，当后者要唤醒他们再次


• 例如，在 EnergyCore ECX-1000（Highbank）所使用的协议的板如下：

• When powering up the board ALL THE CORES start executing from the reset vector
• A software (firmware) component executes WFI on the secondary cores and some other code that will act as a protocol between these secondary cores and the primary core, when the latter wants to wake them up again
• For example, the protocol used on the EnergyCore ECX-1000 (Highbank) board is as follows:

**（1）**二次核进入WFI时

**（2）**主要核心发送SGI唤醒他们

**（3）**，他们是否在地址（0X40 + 0×10 * COREID）的值不为空;

**（4）**如果非空，就用它作为一个地址跳转到（执行BX）

**（5）**否则，他们重新进入待机状态时，通过重新执行WFI

**（6）**所以，如果我有一个EnergyCore ECX-1000主板，我应该写（0X40 + 0×10 * COREID）与地址我希望每个内核跳到并发送SGI

问题：

• 1。什么是做这个软件的组成部分？难道是BL 1二元我写的SD卡，或者是U-BOOT？


• 2。据我了解，这个软件协议从电路板的不同登机。是这样吗，还是只能依靠底层处理器上的？


• 3。我在哪里可以找到有关此协议一挑一ARM板的信息？ - 我能找到它的官方ARM网站上或木板网页
？

• 1. What is the software component that does this? Is it the BL1 binary I've written on the SD Card, or is it U-BOOT?
• 2. From what I understand, this software protocol differs from board to board. Is it so, or does it only depend on the underlying processor?
• 3. Where can I find information about this protocol for a pick-one ARM board? - can I find it on the official ARM website or on the board webpage?

推荐答案

好吧，我回来了宝贝。以下是结论：

Ok, I'm back baby. Here are the conclusions:

• 这使CPU的休眠状态的软件组件是引导程序（在我的情况的U-Boot）


• 的Linux某种程度上知道引导程序如何做这个（硬codeD在Linux内核中每块电路板），并且知道如何唤醒他们再次

• The software component that puts the CPUs to sleep is the bootloader (in my case U-Boot)
• Linux somehow knows how the bootloader does this (hardcoded in the Linux kernel for each board) and knows how to wake them up again

有关我ODROID，徐板描述这个过程中，来源是 UBOOT ODROID，v2012.07 < /在这里找到了>和Linux内核： LINUX ODROIDXU-3.4.y （它会更好，如果我看着内核版本从分支 odroid-3.12.y 因为前者没有按T启动所有8个处理器的，只是其中的4，但后者则）。

For my ODROID-XU board the sources describing this process are UBOOT ODROID-v2012.07 and the linux kernel found here: LINUX ODROIDXU-3.4.y (it would have been better if I looked into kernel version from the branch odroid-3.12.y since the former doesn't start all of the 8 processors, just 4 of them but the latter does).

总之，这里的源$ C ​​$ C，我想出了，我就从上面的源头code树，帮助我后来写这篇code发布相关的源文件：

Anyway, here's the source code I've come up with, I'll post the relevant source files from the above source code trees that helped me writing this code afterwards:

typedef unsigned int DWORD;
typedef unsigned char BOOLEAN;
#define FAILURE (0)
#define SUCCESS (1)
#define NR_EXTRA_CPUS (3) // actually 7, but this kernel version can't wake them up all -> check kernel version 3.12 if you need this

// Hardcoded in the kernel and in U-Boot; here I've put the physical addresses for ease
// In my code (and in the linux kernel) these addresses are actually virtual
// (thus the 'VA' part in S5P_VA_...); note: mapped with memory type DEVICE
#define S5P_VA_CHIPID (0x10000000)
#define S5P_VA_SYSRAM_NS (0x02073000)
#define S5P_VA_PMU (0x10040000)
#define EXYNOS_SWRESET ((DWORD) S5P_VA_PMU + 0x0400)
// Other hardcoded values
#define EXYNOS5410_REV_1_0 (0x10)
#define EXYNOS_CORE_LOCAL_PWR_EN (0x3)

BOOLEAN BootAllSecondaryCPUs(void* CPUExecutionAddress){

// 1. Get bootBase (the address where we need to write the address where the woken CPUs will jump to)
//    and powerBase (we also need to power up the cpus before waking them up (?))
DWORD bootBase, powerBase, powerOffset, clusterID;

asm volatile ("mrc p15, 0, %0, c0, c0, 5" : "=r" (clusterID));
clusterID = (clusterID >> 8);
powerOffset = 0;
if( (*(DWORD*)S5P_VA_CHIPID & 0xFF) < EXYNOS5410_REV_1_0 )
{
    if( (clusterID & 0x1) == 0 ) powerOffset = 4;
}
else if( (clusterID & 0x1) != 0 ) powerOffset = 4;

bootBase = S5P_VA_SYSRAM_NS + 0x1C;
powerBase = (S5P_VA_PMU + 0x2000) + (powerOffset * 0x80);

// 2. Power up each CPU, write bootBase and send a SEV (they are in WFE [wait-for-event] standby state)
for (i = 1; i <= NR_EXTRA_CPUS; i++)
{
    // 2.1 Power up this CPU
    powerBase += 0x80;
    DWORD powerStatus = *(DWORD*)( (DWORD) powerBase + 0x4);

    if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == 0)
    {
        *(DWORD*) powerBase = EXYNOS_CORE_LOCAL_PWR_EN;
        for (i = 0; i < 10; i++) // 10 millis timeout
        {
            powerStatus = *(DWORD*)((DWORD) powerBase + 0x4);
            if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) == EXYNOS_CORE_LOCAL_PWR_EN)
                break;
            DelayMilliseconds(1); // not implemented here, if you need this, post a comment request 
        }
        if ((powerStatus & EXYNOS_CORE_LOCAL_PWR_EN) != EXYNOS_CORE_LOCAL_PWR_EN)
            return FAILURE;
    }
    if ( (clusterID & 0x0F) != 0 )
    {
        if ( *(DWORD*)(S5P_VA_PMU + 0x0908) == 0 )
        do { DelayMicroseconds(10); } // not implemented here, if you need this, post a comment request
        while (*(DWORD*)(S5P_VA_PMU + 0x0908) == 0);
        *(DWORD*) EXYNOS_SWRESET = (DWORD)(((1 << 20) | (1 << 8)) << i);
    }

    // 2.2 Write bootBase and execute a SEV to finally wake up the CPUs
    asm volatile ("dmb" : : : "memory");
    *(DWORD*) bootBase = (DWORD) CPUExecutionAddress;
    asm volatile ("isb");
    asm volatile ("\n   dsb\n   sev\n   nop\n");
}
return SUCCESS;
}

这成功地唤醒的备用CPU

而现在相关的源文件的U-Boot的短名单和Linux内核：

And now for that short list of relevant source files in u-boot and the linux kernel:

• UBOOT：<一href=\"https://github.com/hardkernel/u-boot/blob/odroid-v2012.07/board/samsung/smdk5410/lowlevel_init.S\">lowlevel_init.S - 通知行 363-369 ，次要的CPU在WFE如何等候在值 _hotplug_addr 是非归零，并跳转到它; _hotplug_addr实际上是bootBase在上面code; 也行 282-285 告诉我们，_hotplug_addr在 CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr搬迁 - NS code_base （_hotplug_addr - NS code_base为为0x1C 和 CONFIG_PHY_IRAM_NS_BASE是0x02073000 ，从而在Linux内核上的 hardcodings ）

• UBOOT: lowlevel_init.S - notice lines 363-369, how the secondary CPUs wait in a WFE for the value at _hotplug_addr to be non-zeroed and to jump to it; _hotplug_addr is actually bootBase in the above code; also lines 282-285 tell us that _hotplug_addr is to be relocated at CONFIG_PHY_IRAM_NS_BASE + _hotplug_addr - nscode_base (_hotplug_addr - nscode_base is 0x1C and CONFIG_PHY_IRAM_NS_BASE is 0x02073000, thus the above hardcodings in the linux kernel)

LINUX内核：通用 - <一个href=\"https://github.com/hardkernel/linux/blob/odroidxu-3.4.y/arch/arm/kernel/smp.c\">smp.c （看功能的 __ cpu_up ），平台特定的（odroid旭）：<一href=\"https://github.com/hardkernel/linux/blob/odroidxu-3.4.y/arch/arm/mach-exynos/platsmp.c\">platsmp.c （函数 boot_secondary ，由通用__cpu_up调用;还看 platform_smp_ prepare_cpus [底部] =>这实际上设置引导基地和电力基地的功能值）

LINUX KERNEL: generic - smp.c (look at function __cpu_up), platform specific (odroid-xu): platsmp.c (function boot_secondary, called by generic __cpu_up; also look at platform_smp_prepare_cpus [at the bottom] => that's the function that actually sets the boot base and power base values)

这篇关于ARM：启动/唤醒/ Bringup其他CPU核心/接入点，并通过执行起始地址？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！