How to let gcc recover register `$rdx` used in function?

Question

Hi I have a program with assembly and c file to compile, which is listed in the end.

The compilation command is

gcc prog.o runtime.c -g -o run.out
/usr/bin/ld: prog.o: warning: relocation in read-only section `.text'
/usr/bin/ld: warning: creating DT_TEXTREL in a PIE

However, the registers in C functions stringEqual are not recovered. The assembly are as follow:

000000000000132e <stringEqual>:
    132e:   55                      push   %rbp
    132f:   48 89 e5                mov    %rsp,%rbp
    1332:   48 83 ec 10             sub    $0x10,%rsp
    1336:   48 89 7d f8             mov    %rdi,-0x8(%rbp)
    133a:   48 89 75 f0             mov    %rsi,-0x10(%rbp)
    133e:   48 8b 55 f0             mov    -0x10(%rbp),%rdx
    1342:   48 8b 45 f8             mov    -0x8(%rbp),%rax
    1346:   48 89 d6                mov    %rdx,%rsi
    1349:   48 89 c7                mov    %rax,%rdi
    134c:   e8 0f fd ff ff          call   1060 <strcmp@plt>
    1351:   85 c0                   test   %eax,%eax
    1353:   74 07                   je     135c <stringEqual+0x2e>
    1355:   b8 00 00 00 00          mov    $0x0,%eax
    135a:   eb 05                   jmp    1361 <stringEqual+0x33>
    135c:   b8 01 00 00 00          mov    $0x1,%eax
    1361:   c9                      leave
    1362:   c3                      ret

It can be seen that register $rdx was saved but not recovered, and it actually messed up in function strcmp@plt. I tried to write the string compare without calling strcmp, the result remains, $rdx was used during the comparing process, but not recovered when quit the function.

Thus I would like to ask:

why is this, the function has only 2 parameters, which tooks $rdi and $rsi. The $rdi should be saved.
how should I tell gcc to recover the used registers inside of the function?

runtime.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

long tigermain(long a);

void* checked_malloc(long len) 
{
    void *p = malloc(len);
    assert(p);
    return p;
}

long stringEqual(char* s, char* t)
{
    if (strcmp(s,t)) {
        return 0;
    } else {
        return 1;
    }
}

void print(char* s)
{
    printf(s);
}

long main()
{
    return tigermain(0 /* static link */);
}

prog.asm

extern print
extern checked_malloc
extern stringEqual

global tigermain
segment .note.GNU-stack
segment .text
; PROCEDURE top_level START

tigermain:
    push rbp
    mov rbp, rsp
    sub rsp, 16
L17:
    mov rdi, 16
    call checked_malloc
    mov rdx, rax
    mov rdi, 0
    mov [rdx + 8], rdi
    mov rdi, L11
    mov [rdx + 0], rdi
    mov rdi, L12
    mov rsi, L13
    mov [rdx + 0], rsi
    mov rsi, 3
    mov [rdx + 8], rsi
    mov rsi, [rdx + 0]
    call stringEqual
    mov rdi, 0
    cmp rax, rdi
    jne L14
L15:
    mov rax, 0
    jmp L16
L14:
    mov rdi, [rdx + 0]
    call print
    jmp L15
L16:
    mov rsp, rbp
    pop rbp
    ret

; top_level END

segment .data
L13: db 97,115,100,0
L12: db 97,115,100,0
L11: db 78,97,109,101,0

Maybe you're getting mixed up with AT&T syntax disassembly; your debug build of `stringEqual` is just spilling the incoming register args, RSI and RDI, [because](https://stackoverflow.com/questions/53366394/why-does-clang-produce-inefficient-asm-with-o0-for-this-simple-floating-point) it's a debug build. [also](//stackoverflow.com/q/51976465). And then reloading them as args for `strcmp`. The compiler-generated code doesn't read its incoming RDX, and overwrites it itself. Since you're using NASM for your asm, probably easier to use `objdump -drwC -Mintel`, and GDB's intel-syntax mode. — Peter Cordes, Aug 30 '22 at 15:22
You can't practically change the calling convention; use call-preserved registers in your own code if you want them to survive across function calls. — Peter Cordes, Aug 30 '22 at 15:23
@petercordes I'm building the program using nasm first, to make an elf object file for prog.s, then using gcc to compile them together....... As far as I know, the calling convention is that first 6 parameters passed using registers, and returning value in RAX..... Which I cannot figure out why is this... — Shore, Aug 30 '22 at 15:27
Yeah, that's correct. Those registers are all call-clobbered, along with r10 and r11. The rest are call-preserved. Use those in your NASM code that expects register values to survive function calls. Or are you asking why the calling convention was designed that way? [Why should certain registers be saved? What could go wrong if not?](https://stackoverflow.com/q/69419435) / [What's the advantage of having nonvolatile registers in a calling convention?](https://stackoverflow.com/q/10392895) — Peter Cordes, Aug 30 '22 at 15:32
@petercordes By using call-preserved registers meaning save and recover all RDI, RSI RDX, RCX, R8, R9 before any call to C functions, and recover them afterward??? — Shore, Aug 30 '22 at 15:32
No, by keeping your loop counter in RBX, R12, R13, etc, which won't get destroyed when you `call` a function that follows the ABI. See the linked duplicates, especially [What are callee and caller saved registers?](https://stackoverflow.com/a/56178078) and [What registers are preserved through a linux x86-64 function call](https://stackoverflow.com/q/18024672). You *could* push/pop some registers around every `call`, but that would be slow and inefficient, which is why calling conventions have call-preserved registers. — Peter Cordes, Aug 30 '22 at 15:33
@petercordes I still do not understand, the function has only 2 parameters, which means it only will use RDI and RSI to pass parameters. The use of RDX, or calling other function which will use RDX, should have recovered at the end of function, or after the call. There is no loop in the function `stringEqual` and `strcmp` is compiled by GCC, also no loop in the assembly..... — Shore, Aug 30 '22 at 15:42
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/247688/discussion-between-shore-and-peter-cordes). — Shore, Aug 30 '22 at 15:42
RDX is call-clobbered. Any call to a compiler-generated function must be assumed to destroy its value. **In your NASM source, use different registers.** You can't change what the compiler will do. Sorry, I saw a label near the top of your asm and assumed there was a loop, but actually nothing branches back to it. — Peter Cordes, Aug 30 '22 at 15:43

How to let gcc recover register `$rdx` used in function?

0 Answers0