[Python] MPQ Stuff

topaz · October 12, 2006, 01:43 AM

I've been porting over some of the code used to authenticate and extract the versioning libraries from the MPQ. I had no trouble porting over the earlier code, since it was relatively simple to decipher - but now that I recently found newer functions in Storm that are likely to do it better, I've tried to port them over (with some help by rob's vercheck1/2) -

peekaboo.py

Code Select

from ctypes import *

storm_handle = windll.LoadLibrary('./storm.dll')

def destroy():
    """BOOL WINAPI ORDINAL(0x100) SFileDdaDestroy()"""
    return storm_handle[262]()

def open_archive(archive):
    """BOOL WINAPI ORDINAL(0x10A) SFileOpenArchive(LPCSTR lpFileName,
        DWORD nArchivePriority, DWORD grfArchiveFlags, HSARCHIVE *lphMPQ)"""
    mpq_handle = c_void_p()

    result = storm_handle[266](archive, 0, 0, byref(mpq_handle))

    if result == 0:
        return -1
    else:
        return mpq_handle

def authenticate_archive(mpq_handle):
    """BOOL WINAPI ORDINAL(0xFB) SFileAuthenticateArchive(HSARCHIVE hMPQ,
                LPDWORD lpdwAuthenticationStatus)
    Status codes:
        0 = SFILE_AUTH_AUTHENTICATION_UNAVAILABLE,
        1 = SFILE_AUTH_NOT_SIGNED,
        2 = SFILE_AUTH_INVALID_SIGNATURE
        3 = SFILE_AUTH_UNSUPPORTED_SIGNATURE_FORMAT
        5 = SFILE_AUTH_AUTHENTICATED"""
    auth_status = c_long(0)
    result = storm_handle[251](mpq_handle, byref(auth_status))

    if result == 0:
        return -1
    else:
        return auth_status

def load_file(mpq_handle, file_name):
    """BOOL WINAPI ORDINAL(0x119) SFileLoadFileEx(HSARCHIVE hMPQ, LPCSTR
    lpszFileName, LPVOID *lplpFileData, LPDWORD lpnFileSize, DWORD
    nExtraBytesToAllocateAtEnd, DWORD grfSearchScope, LPOVERLAPPED lpOverlapped)"""
    file_data = c_void_p()
    file_size = c_long(2)
    
    result = storm_handle[281](mpq_handle, file_name, byref(file_data),
                            byref(file_size), 0, 0, 0)

    if result == 0:
        return -1
    else:
        return [file_data, file_size]
        
def set_locale(new_locale):
    """void WINAPI ORDINAL(0x110) SFileSetLocale(LANGID lnNewLanguage)"""
    storm_handle[272](new_locale)

destroy()
fefe = open_archive('./ver-IX86-0.mpq')

result = authenticate_archive(fefe)

if result == 5:
    file_junk = load_file(fefe, 'ver-IX86-0.dll')

topaz · October 12, 2006, 01:47 AM

The main reason for me posting this is to ask for help.

In SFileLoadFileEx, what is 'lplpFileData'? It appears to be a long (I'm getting 17301644), but what is it? Is it a filetime?

MyndFyre · October 12, 2006, 03:02 AM

Well, according to the notation it's a long pointer to a long pointer to the file data.

My guess, is that its data type is traditionally LPVOID. It's being passed by-reference and is a return value. It could be decorated LPVOID OUT *lplpFileData. OUT is typically #defined to nothing, but is informational metadata.

You can:

Code Select


LPVOID lpFileData = *lplpFileData;

Then you have a pointer to the beginning of the file data.

topaz · October 14, 2006, 02:30 AM

Quote from: MyndFyre[vL] on October 12, 2006, 03:02 AM
Well, according to the notation it's a long pointer to a long pointer to the file data.

My guess, is that its data type is traditionally LPVOID. It's being passed by-reference and is a return value. It could be decorated LPVOID OUT *lplpFileData. OUT is typically #defined to nothing, but is informational metadata.

You can:

Code Select Expand
LPVOID lpFileData = *lplpFileData;

Then you have a pointer to the beginning of the file data.

Thanks... if only I knew how to do that with ctypes

MyndFyre · October 14, 2006, 05:02 AM

Use byref. You're passing lpFileData by reference and getting a pointer (type LPVOID) to the beginning of the file data.

Look at it this way...

Let's say the file data starts at 0x0401c0c0

lpFileData is equal to 0x0401c0c0.

*lpFileData gets the value AT 0x0401c0c0, which is the first value in the file (depending on how LPVOID is dereferenced, I'd guess the first four bytes).

When you have LPVOID* lplpFileData, that means you're passing the pointer by reference. That's because the function's return value is a status code (error/success), and since you want to get more than one value returned, you pass parameters by reference. What usually happens is when you pass parameters by value, the value is copied onto the stack and then you never get the changed value from the stack. However, when you pass by reference, the address of the value is passed onto the stack, and then the value at the address is updated.

So 0x0401c0c0 is where the file data is, let's say 0x04000da0 is where lpFileData is located. If you look at the value AT 0x04000da0, it's 0x0401c0c0.

Using byref, you actually pass 0x04000da0 onto the stack. The called function then writes the appropriate value in memory at 0x04000da0.

You did something similar with SFileAuthenticateArchive.

topaz · October 14, 2006, 05:09 AM

Quote from: MyndFyre[vL] on October 14, 2006, 05:02 AM
Use byref. You're passing lpFileData by reference and getting a pointer (type LPVOID) to the beginning of the file data.

Look at it this way...

Let's say the file data starts at 0x0401c0c0

lpFileData is equal to 0x0401c0c0.

*lpFileData gets the value AT 0x0401c0c0, which is the first value in the file (depending on how LPVOID is dereferenced, I'd guess the first four bytes).

When you have LPVOID* lplpFileData, that means you're passing the pointer by reference. That's because the function's return value is a status code (error/success), and since you want to get more than one value returned, you pass parameters by reference. What usually happens is when you pass parameters by value, the value is copied onto the stack and then you never get the changed value from the stack. However, when you pass by reference, the address of the value is passed onto the stack, and then the value at the address is updated.

So 0x0401c0c0 is where the file data is, let's say 0x04000da0 is where lpFileData is located. If you look at the value AT 0x04000da0, it's 0x0401c0c0.

Using byref, you actually pass 0x04000da0 onto the stack. The called function then writes the appropriate value in memory at 0x04000da0.

You did something similar with SFileAuthenticateArchive.

Maybe I'm thinking too hard... I'll try that.

topaz · October 14, 2006, 05:45 AM

~~Can someone upload the ver-IX86-#.dlls? I don't know what I'm actually looking for, so I think that would be a good place to start.~~

Edit:

~~http://advancedcontent.net/topaz/etc/ver-IX86-n.rar~~

Files removed, Blizzard sent a DMCA takedown notice to Warrior, who hosts my FTP.

topaz · October 15, 2006, 09:46 PM

Looks like I won't be able to do this without using win32-specific API (like CreateFile and WriteFile). SFileLoadFileEx returns a pointer to the file data, rather than the data itself (and CreateFile and WriteFile are intended to use that pointer). I don't think I can do much with a barebones knowledge of C.

see below

MyndFyre · October 15, 2006, 11:49 PM

If you want to use MPQ functionality, I recommend using ShadowFlare's MPQ API (sfmpq.dll). It implements the Storm functions such as SFileOpenFile and it's fairly documented. Then it has SFileReadFileEx, which reads data.

The Storm functions (and the SFmpqAPI functions) are modeled after the Win32 API pattern of CreateFile and WriteFile.

topaz · October 16, 2006, 09:35 PM

Quote from: MyndFyre[vL] on October 15, 2006, 11:49 PM
If you want to use MPQ functionality, I recommend using ShadowFlare's MPQ API (sfmpq.dll). It implements the Storm functions such as SFileOpenFile and it's fairly documented. Then it has SFileReadFileEx, which reads data.

The Storm functions (and the SFmpqAPI functions) are modeled after the Win32 API pattern of CreateFile and WriteFile.

~~I can still use the Storm functions, just not SFileLoadFileEx and some of the other ones. I was hoping to use the newer ones to do the same task just because I could.~~

see below

topaz · October 19, 2006, 01:20 AM

By the way, here's the module I wrote that uses ~~SFileReadFile~~, SFileReadFile and SFileLoadFileEx:

Code Select

from ctypes import *

storm_handle = windll.LoadLibrary('./storm.dll')

def destroy():
    """BOOL WINAPI ORDINAL(0x100) SFileDdaDestroy()"""
    result = storm_handle[262]()

    if result == 0:
        return -1
    else:
        return result

def open_archive(archive):
    """BOOL WINAPI ORDINAL(0x10A) SFileOpenArchive(LPCSTR lpFileName,
        DWORD nArchivePriority, DWORD grfArchiveFlags, HSARCHIVE *lphMPQ)"""
    mpq_handle = c_void_p()

    result = storm_handle[266](archive, 0, 0, byref(mpq_handle))

    if result == 0:
        return -1
    else:
        return mpq_handle

def authenticate_archive(mpq_handle):
    """BOOL WINAPI ORDINAL(0xFB) SFileAuthenticateArchive(HSARCHIVE hMPQ,
                LPDWORD lpdwAuthenticationStatus)
    Status codes:
        0 = SFILE_AUTH_AUTHENTICATION_UNAVAILABLE,
        1 = SFILE_AUTH_NOT_SIGNED,
        2 = SFILE_AUTH_INVALID_SIGNATURE
        3 = SFILE_AUTH_UNSUPPORTED_SIGNATURE_FORMAT
        5 = SFILE_AUTH_AUTHENTICATED"""
    auth_status = c_long(0)
    result = storm_handle[251](mpq_handle, byref(auth_status))

    if result == 0:
        return -1
    else:
        return auth_status

def load_file(mpq_handle, file_name):
    """BOOL WINAPI ORDINAL(0x119) SFileLoadFileEx(HSARCHIVE hMPQ, LPCSTR
    lpszFileName, LPVOID *lplpFileData, LPDWORD lpnFileSize, DWORD
    nExtraBytesToAllocateAtEnd, DWORD grfSearchScope, LPOVERLAPPED lpOverlapped)"""
    file_data = c_char_p()
    file_size = c_long(0)
    
    result = storm_handle[281](mpq_handle, file_name, byref(file_data),
                            byref(file_size), 0, 0, 0)

    if result == 0:
        return -1
    else:
        return file_data

def unload_file(file_data):
    """BOOL WINAPI ORDINAL(0x118) SFileUnloadFile(LPVOID lpFileData);"""
    result = storm_handle[280](file_data)

    if result == 0:
        return 1
    else:
        return result

def open_file(mpq_handle, file_name):
    file_handle = c_void_p()
    """BOOL WINAPI ORDINAL(0x10C) SFileOpenFileEx(HSARCHIVE hMPQ,
    LPCSTR lpFileName, DWORD grfSearchScope, HSFILE *lphFile);"""
    result = storm_handle[268](mpq_handle, file_name, 0, byref(file_handle))

    if result == 0:
        return -1
    else:
        return file_handle

def close_file(file_handle):
    """BOOL WINAPI ORDINAL(0xFD) SFileCloseFile(HSFILE hFile);"""
    result = storm_handle[253](file_handle)

    if result == 0:
        return -1
    else:
        return result

def write_data(file_data, file_name):
    new_file = open('./' + file_name, 'wb')
    new_file.write(file_data)
    new_file.close()

def read_file(file_handle, file_size):
    """BOOL WINAPI ORDINAL(0x10D) SFileReadFile(HSFILE hFile, LPVOID lpBuffer,
    DWORD nNumberOfBytesToRead, LPDWORD lpnNumberOfBytesRead,
    LPOVERLAPPED lpOverlapped);

    Notes:
        This function doesn't seem to exist in any of the Storm libraries,
        save the one for Warcraft III/Frozen Throne.
        
        Traceback:
            AttributeError: function ordinal 287 not found"""
    file_buffer = create_string_buffer(file_size)
    
    result = storm_handle[269](file_handle, file_buffer, file_size, 0, 0)

    if result == 0:
        return -1
    else:
        return file_buffer
    

def close_archive(mpq_handle):
    """BOOL WINAPI ORDINAL(0xFC) SFileCloseArchive(HSARCHIVE hMPQ);"""
    return storm_handle[252](mpq_handle)
        
def set_locale(new_locale):
    """void WINAPI ORDINAL(0x110) SFileSetLocale(LANGID lnNewLanguage)"""
    return storm_handle[272](new_locale)

def get_file_size(file_handle):
    return storm_handle[265](file_handle, 0)

destroy()
fefe = open_archive('./' + 'ver-IX86-0.mpq')
result = authenticate_archive(fefe)

if result <> -1:
    x = open_file(fefe, 'ver-IX86-0.dll')
    fsize = get_file_size(x)
    s = read_file(x, fsize)
    write_data(s, 'ver-IX86-0.dll')
    close_file(fefe)
    close_archive(fefe)

I'll write a class for it later, if anyone actually cares

Edit: I got SFileLoadFileEx to work, although the workaround I used is extremely dirty:

Code Select

def load_file(mpq_handle, file_name):
    """BOOL WINAPI ORDINAL(0x119) SFileLoadFileEx(HSARCHIVE hMPQ, LPCSTR
    lpszFileName, LPVOID *lplpFileData, LPDWORD lpnFileSize, DWORD
    nExtraBytesToAllocateAtEnd, DWORD grfSearchScope, LPOVERLAPPED lpOverlapped)"""
    file_data = create_string_buffer(0)
    file_size = c_long(0)
    
    result = storm_handle[281](mpq_handle, file_name, byref(file_data),
                            byref(file_size), 0, 0, 0)

    file_data = create_string_buffer(file_size.value)

    result = storm_handle[281](mpq_handle, file_name, byref(file_data),
                            byref(file_size), 0, 0, 0)

    if result == 0:
        return -1
    else:
        return file_data

It seems I was wrong about accessing file pointers, I was fooled into thinking there was only three bytes in the buffer (probably some exotic bug involving reading certain characters with ctypes) when there was actually all the file data I needed!