Funny calling convention (using esi/edi)

[iago]

So while I was writing that tutorial posted here, I was playing around in Starcraft. I noticed a couple functions that had funny calling conventions. For example, I notice parameters being passed in eax, occasionally, and once (shown below), parameters are passed in esi and edi.

Is using esi and edi for parameters some known calling convention? Or Is that an optimization?

Here's the code I noticed:

Code Select Expand

.text:0041F060 ; vsnprintf_wrapper
.text:0041F060
.text:0041F060 arg_0           = dword ptr  8
.text:0041F060 arg_4           = byte ptr  0Ch
.text:0041F060
.text:0041F060                 push    ebp
.text:0041F061                 mov     ebp, esp
.text:0041F063                 mov     ecx, [ebp+arg_0]
.text:0041F066                 lea     eax, [ebp+arg_4]
.text:0041F069                 push    eax             ; va_list
.text:0041F06A                 push    ecx             ; char *
.text:0041F06B                 push    esi             ; size_t
.text:0041F06C                 push    edi             ; char *
.text:0041F06D                 call    __vsnprintf
.text:0041F072                 add     esp, 10h
.text:0041F075                 mov     byte ptr [edi+esi-1], 0
.text:0041F07A                 pop     ebp
.text:0041F07B                 retn
.text:0041F07B vsnprintf_wrapper endp


[Joe[x86]]

I was under the assumption that as long as their pushed on to the stack it doesn't matter where they came from.

[Hell-Lord]

I forget what it is exactly but it has something to do with constant value(s).

[iago]

x86] link=topic=16602.msg167771#msg167771 date=1176179194]
I was under the assumption that as long as their pushed on to the stack it doesn't matter where they came from.

I'm not talking about the function called there, I'm talking about where esi and edi came from. They aren't assigned anywhere in that function, they're assigned when calling it.

I forget what it is exactly but it has something to do with constant value(s).

Hmm, that's possible. But I've looked at this function before in older versions, and that never used to happen.

[Joe[x86]]

.text:0041F075                 mov     byte ptr [edi+esi-1], 0

Apparently it is constant values then, because that appears to be setting [edi+esi] to (what will look like) a null string.

[MyndFyre]

x86] link=topic=16602.msg167835#msg167835 date=1176319343]
.text:0041F075                 mov     byte ptr [edi+esi-1], 0

Apparently it is constant values then, because that appears to be setting [edi+esi] to (what will look like) a null string.

No.  Look at the arguments to __vsnprintf and the documentation for the equivalent functions (pulled from MSDN documentation):

Code Select Expand


int vswprintf(
   wchar_t *buffer,
   size_t count,
   const wchar_t *format,
   va_list argptr
);

edi is clearly storing the destination string buffer.
esi is the size of the destination buffer.
ecx is the constant input format string.
eax holds the varargs argument list.

The purpose of the line you quoted:

Code Select Expand


mov byte ptr [edi+esi-1], 0
should be obvious to you based on what those arguments are.
edi+esi-1 is the address of very last character in the output string.  This sets it to null, presumably to ensure that if there's a buffer overflow, the result doesn't overflow another buffer.

In any case, that doesn't really have anything to do with iago's question, which is - since when are edi and esi used in a calling convention?

@iago: I would guess that there was probably some kind of modified pseudo-inline fastcall calling convention, or perhaps it's a result of storing register values?

[Hell-Lord]

esi is the position is it not?

[Skywing]

This is due to advanced compiler optimizations on module-internal functions, where the compiler knows of all callers of a particular function and can then apply custom calling conventions to allow for better register use across callers.

Functions that are `externally visible' will not have such optimizations applied to them.

Matt Pietrek authored an article providing a basic overview of these sort of optimizations as implemented in cl 13.  Some of the information in that article is dated and not entirely accurate now, but it conveys the basic idea.

In versions of cl prior to 13, there are a couple of instances where you might see strange calling conventions in compiler generated code.  For instance, for a number of releases, the `_alloca' special function was implemented using a custom calling convention using `eax' as an argument register and altering the stack pointer of the caller.  The various versions of `_SEH_prolog' also use custom calling conventions.

[iago]

Aha, so it is a weird optimization. I'll definitely have a look at that paper.

Thanks!

[Joe[x86]]

Just for the record, is this similar to fastcall using different registers?

[warz]

x86] link=topic=16602.msg169191#msg169191 date=1179471490]
Just for the record, is this similar to fastcall using different registers?

no, not traditionally. fastcall uses ecx and edx to pass the first two arguments, and the rest are passed through the stack. sometimes you'll see fastcall use eax, too, but i think this is a borland optimization. in broodwar's case, youll only see fastcall using ecx, for the most part.

on x64 systems, though, most calling conventions, such as stdcall, fastcall, etc, are ignored by the compiler, and the convention it uses is similar to fastcall. it passes the first four arguments by way of registers. sounds like it'd make debugging much nicer.

edit: see this

[iago]

x86] link=topic=16602.msg169191#msg169191 date=1179471490]
Just for the record, is this similar to fastcall using different registers?

I'm not really sure what warz is talking about, although I think he either didn't read your post, or he's agreeing with you in an odd way. So I'll answer.

Yes, it's absolutely identical to __fastcall except that it uses different registers.

[warz]

If you're talking about the __vsnprintf call, it isn't identical to fastcall at all.

[iago]

Except that the defining characteristic of fastcall is that parameters are passed in registers, and that function passes parameters in registers.

He specifically asked if it's the same as fastcall but with different registers. Are you saying that's not true?

[Skywing]

It is not a fixed calling convention.  The compiler takes a look at the function itself, and all callers of the function, and from there decides the best way to pass parameter values.  It is subject to changing completely at a recompile.