Forum Stats

  • 3,875,374 Users
  • 2,266,908 Discussions
  • 7,912,186 Comments

Discussions

OracleLinux9 - UEK7 -- some version can't use kprobe with KPROBE_FTRACE

Levi Yun
Levi Yun Member Posts: 5 Green Ribbon
edited Nov 10, 2022 8:20AM in Oracle Linux

Hi, i'm using the new kernel UEK-9, 5.15.0-0.30.20.1.el9uek.x86_64 and 5.15.0-0.30.20.el9uek.x86_64.

But I met some problem when i try to use kprobe with KPROBE_FTRACE -- it's going to panic.

when I take a look the ftrace trampoline which generated while registering kprobe.

There is one strange at the end of trampoline.

   ...
   0xffffffffc08ed0c3:  call   0xffffffff8188d820 <kprobe_ftrace_handler>
   0xffffffffc08ed0c8:  mov    0x90(%rsp),%rax
   0xffffffffc08ed0d0:  mov    %rax,0xa8(%rsp)
   0xffffffffc08ed0d8:  mov    0x80(%rsp),%rax
   0xffffffffc08ed0e0:  mov    %rax,0xb0(%rsp)
   0xffffffffc08ed0e8:  mov    (%rsp),%r15
   0xffffffffc08ed0ec:  mov    0x8(%rsp),%r14
   0xffffffffc08ed0f1:  mov    0x10(%rsp),%r13
   0xffffffffc08ed0f6:  mov    0x18(%rsp),%r12
   0xffffffffc08ed0fb:  mov    0x38(%rsp),%r10
   0xffffffffc08ed100:  mov    0x28(%rsp),%rbx
   0xffffffffc08ed105:  mov    0x78(%rsp),%rax
   0xffffffffc08ed10a:  mov    %rax,0xa0(%rsp)
   0xffffffffc08ed112:  mov    0x78(%rsp),%rax
   0xffffffffc08ed117:  test   %rax,%rax
   0xffffffffc08ed11a:  xchg   %ax,%ax
   0xffffffffc08ed11c:  mov    0x20(%rsp),%rbp
   0xffffffffc08ed121:  mov    0x40(%rsp),%r9
   0xffffffffc08ed126:  mov    0x48(%rsp),%r8
   0xffffffffc08ed12b:  mov    0x70(%rsp),%rdi
   0xffffffffc08ed130:  mov    0x68(%rsp),%rsi
   0xffffffffc08ed135:  mov    0x60(%rsp),%rdx
   0xffffffffc08ed13a:  mov    0x58(%rsp),%rcx
   0xffffffffc08ed13f:  mov    0x50(%rsp),%rax
   0xffffffffc08ed144:  add    $0xa8,%rsp
   0xffffffffc08ed14b:  popf   
   0xffffffffc08ed14c:  jmp    0xffffffffc18670fc 

See the last operation at the end of trampoline, actually it should be jmp to

__x86_return_thunk

But when I get the address, it is:

crash> p __x86_return_thunk
__x86_return_thunk = $6 = 
 {<text variable, no debug info>} 0xffffffff828023c0 <__x86_return_thunk>

So, ftrace trampoline is to wrong address at the end, I meet some panic whenever i use kprobe with KPROBE_FTRACE.


Is there similar issued reported?

Thanks!

Best Answer

  • Alexandre Chartre-Oracle
    Alexandre Chartre-Oracle Member Posts: 4 Employee
    Answer ✓

    I confirm there is an issue with 5.15.0-0.30.20, and this issue is fixed in 5.15.0-1.43.3.

    The problem is that the kernel is built with a missing configuration option which causes the kernel to always use return thunks. This causes a "jmp __x86_return_thunk" instruction to always be present in ftrace_stub(). At run time, when the kernel doesn't enable the X86_RETURN_THUNK capability, ftrace trampoline will end by copying the content of ftrace_stub() causing the "jmp __x86_return_thunk" instruction to be incorrectly copied (because the jmp distance is not adjusted).

    This issue is referenced as Oracle bug 34395937 (bpf is broken by retbleed), and it is fixed in 5.15.0-1.43.3.

    For this particular ftrace issue, I think a possible workaround is to boot with retbleed=unret. This should force the kernel to enable the X86_RETURN_THUNK capability and the ftrace trampoline should directly add a "jmp __x86_return_thunk" instruction instead of copying it from ftrace_stub().

    Levi Yun

Answers

  • Sergio-Oracle
    Sergio-Oracle Member Posts: 2,668 Employee

    I asked one of our kernel developers, and his response was:

    "In the reported code, I think the last jmp is a jmp to ftrace_epilogue, not to __x86_return_thunk. ftrace_epilogue will do the jump to __x86_return_thunk (or a direct ret).

    So you need to check that 0xffffffffc18670fc is effectively ftrace_epilogue."

  • Levi Yun
    Levi Yun Member Posts: 5 Green Ribbon

    Hi, Sergio.

    I think you should confirm the backported kernel source for 5.15.0-30.20.1

    When you see the "create_trampoline" function.

    the "jmp ftrace_epilogue" is changed to jmp __x86_return_thunk by below code snippet.

    arch/x86/kernel/ftrace.c
    
    static unsigned long create_trampoline(struct ftrace_ops *ops, unsgined int *tramp_size)
    {
        ...
        
        ip = trampoline + size;
        if (cpu_feature_enabled(X86_FEATURE_RETHUNK)) {
            __text_gen_insn(ip, JMP32_INSN_OPCODE, ip, &__x86_return_thunk, JMP32_INSN_SIZE);
        } else {
            ...
        }
    }
    

    Thou, ftrace_64.c make end as ftrace_epilogue, based on cpu feature, and code is changed to jmp to __x86_return_thunk or just "ret" instruction.

    And the address of ftrace_epilogue is below:

     crash p ftrace_epilogue
        0xffffffff81888400 <ftrace_epilogue>
    

    Thx!

  • Sorry, I have provided the answer to Sergio which, as you noticed, is incorrect.

    The retq/jmp __x86_return_thunk is copied after ftrace_regs_caller_end (where "jmp ftrace_epilogue" is), so it should effectively end the function. It looks we have the correct code, but let me have a closer look.

    Thanks.

  • Alexandre Chartre-Oracle
    Alexandre Chartre-Oracle Member Posts: 4 Employee
    Answer ✓

    I confirm there is an issue with 5.15.0-0.30.20, and this issue is fixed in 5.15.0-1.43.3.

    The problem is that the kernel is built with a missing configuration option which causes the kernel to always use return thunks. This causes a "jmp __x86_return_thunk" instruction to always be present in ftrace_stub(). At run time, when the kernel doesn't enable the X86_RETURN_THUNK capability, ftrace trampoline will end by copying the content of ftrace_stub() causing the "jmp __x86_return_thunk" instruction to be incorrectly copied (because the jmp distance is not adjusted).

    This issue is referenced as Oracle bug 34395937 (bpf is broken by retbleed), and it is fixed in 5.15.0-1.43.3.

    For this particular ftrace issue, I think a possible workaround is to boot with retbleed=unret. This should force the kernel to enable the X86_RETURN_THUNK capability and the ftrace trampoline should directly add a "jmp __x86_return_thunk" instruction instead of copying it from ftrace_stub().

    Levi Yun
  • Levi Yun
    Levi Yun Member Posts: 5 Green Ribbon

    Thank for your clear answer. and I'll check the Oracle bug 34395937 (bpf is broken by retbleed).

    Sincerely,

    Levi.