I need help decompressing D2 packets

[Ringo]

hey, my falt, i should have explained it.

Im not to good at explaining things, but i hope you follow.

Its imprtant to note, that TCP can brake packets up/buffer them together when sending, and the best way to parse compleat messages is to have a header with the lengh of the packet being sent.
This way you can check the lengh in the header and snip out the data and read it, or if the data isnt all there, wait untill the rest is sent, so you can compleat the packet and snip/read it.
D2GS protocol uses a header like this, as i will explain with this packet:

Code Select Expand


07 1f 7f ff ff ff c0

07 is the lengh header in this case, it tells us the lengh of the message is 7 bytes. (including the header)
The Offset for this will be 1, meaning that the header is 1 byte in lengh, so we can strip the data (1f 7f ff ff ff c0) away from the header and decompress it.
The header isnt always 1 byte in lengh, in some cases where the payload is over 0xF0 bytes long, 2 bytes are used ( see GetPacketSize() )

then when the data 1f 7f ff ff ff c0 is decompressed to the following:

Code Select Expand


8F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The same thing happens again, there maybe multiple decompressed packets inside a compressed packet, and they also maybe broken across compressed packets, and almost all of them have a fixed lengh, apart from a few variable sized packets that have a lengh byte or other method to to work out the decompressed packets lengh.

Im pritty tired, so i hope this makes sence

[l2k-Shadow]

Each packet length is stored in one byte prior to the actual data, for some packets, however, the size of the packet is larger than 255, thus having to be stored in 2 bytes, offset simply returns the amount of bytes the length is in, being 1 in most cases.

[Juniper]

Ok, I understand that now.
So, let me ask you this, I've modified my code to take into consideration the length byte. Meaning, I'm starting to decode the REAL payload now (without the length byte).

Here are some examples:

Original packet (with length byte): 0c 58 0c 87 f0 02 7a 42 32 11 d9 5a
The Lengh byte is 0c, which means my real data should be 11 bytes long.
Decoded data: F6 03 00 00 4F 6C C0 13 D5 13 80


Original packet (with length byte): 0f 52 b8 22 01 62 73 85 56 40 8b 2f 4a 2a 58
The Lengh byte is 0f, which means my real data should be 13 bytes long.
Decoded data: 00 01 AB 8B 3D 01 E5 13 94 13 01 00 0D 46 00 05


Original packet (with length byte): 19 58 0c 87 f0 02 7a 42 32 11 d9 5a 5d 03 0c 06 62 70 e8 43 8e a4 40 d8 be
The Lengh byte is 19, which means my real data should be 24 bytes long.
Decoded data: F6 03 00 00 4F 6C C0 13 D5 13 80 95 9F 05 F9 40 7C E0 7D E2 73 02 00 00

Now, these are all packets that weren't splitted between several packets. As you can see they are short enough to do so.
My problem is that I cannot find the message ID of these packets (and many more packets) on bnet docs, or anywhere else.
Are most of the packets still unmapped?
I'm pretty sure (check my packets if you wish) that I'm decompressing the packets correctly, but I can't figure out what any of these packets actually do.

Can someone help me out?
thanks in advance,
J.

[l2k-Shadow]

Unless you're actually attempting to make a d2 game bot and getting past warden, you can safely ignore 99% of the packets. <.<

[Ringo]

Original packet (with length byte): 0f 52 b8 22 01 62 73 85 56 40 8b 2f 4a 2a 58
The Lengh byte is 0f, which means my real data should be 13 bytes long.
Decoded data: 00 01 AB 8B 3D 01 E5 13 94 13 01 00 0D 46 00 05

That looks pritty much perfect, apart from "00 01" should be "67", so i think you maybe decompressing the lengh header with the payload, causeing it to "smudge"

[teK]

Unless you're actually attempting to make a d2 game bot and getting past warden, you can safely ignore 99% of the packets. <.<

You don't need to get passed warden to make a game bot, 45 seconds is enough time to kill pindle

[Ringo]

I just hook a d2 window up as a back ground program in a fake game, to answer warden for multiple bots at a time
That way, i can run d2 remotely, and have warden scan a differnt system to what the bots are running from.

[teK]

I just hook a d2 window up as a back ground program in a fake game, to answer warden for multiple bots at a time
That way, i can run d2 remotely, and have warden scan a differnt system to what the bots are running from.

Interesting. Mind sharing your code Ringo?

[Juniper]

Hi,
I want to clarify a few things that I'm still not sure about.

Again, I want to give an example to best illustrate the situation:

Code Select Expand

Packet #1: 32 bd 04 4e 0d 01 88 1f 5a 1d 01 59 b1 a1 10 1c 1b 09 e6 87 64 04 65 32 08 81 c0 e6 ba 50 1c 85 00 15 9b 1a 03 71 e8 41 1d 0d 0e

Code Select Expand

Packet #2: 1b 56 18 00 78 f2 14 8c 0b c4 60 7c ca 2a b0 00 74 1f 91 66 00 48 8c 10 99 39 80

As we already know, byte "32" in packet #1 is our length byte, which means the real compressed data payload is 49 bytes.
It's obvious that packet #1 holds less data than 49 bytes, it actually holds 42 bytes of payload. This means that the rest of the payload is in another packet.

This is a good place to ask my first question, when the data payload is fragmented, does the rest of the payload ALWAYS come in the following packet?

Let's suppose for a minute that packet #2 is indeed the continuation of packet #1, my logic says that I should treat packet #2 as if the data in it is a continuity of the data in packet #1. Meaning, the 1st byte of packet #2 is NOT a lenght byte, is this assumption correct?
So I decompress the first 7 bytes of packet#2 normally and glue them back to the data I alredy decoded in packet#1.

Now, after I've managed to identify the full payload that started in packet#1, I'm left with a half decoded packet (packet #2). Should I treat the 8th byte of packet#2 (which is for me, really, the 1st byte of new data) as a length byte? the 8th byte is 8C which is 140 bytes of data.

This is my understanding on how this whole mechanism works, can anyone confirm/deny what I just said?
It's really important to me to understand the complete flow of data.

thanks in advance,
J

[l2k-Shadow]

You're on the right track.. but not quite there yet. I think this site may help you understand D2GS a little bit more: http://bnetdocs.valhallalegends.com/content.php?Section=d&id=6 (scroll down to D2GS headers)

[Ringo]

Hi,
I want to clarify a few things that I'm still not sure about.

Again, I want to give an example to best illustrate the situation:

Code Select Expand

Packet #1: 32 bd 04 4e 0d 01 88 1f 5a 1d 01 59 b1 a1 10 1c 1b 09 e6 87 64 04 65 32 08 81 c0 e6 ba 50 1c 85 00 15 9b 1a 03 71 e8 41 1d 0d 0e

Code Select Expand

Packet #2: 1b 56 18 00 78 f2 14 8c 0b c4 60 7c ca 2a b0 00 74 1f 91 66 00 48 8c 10 99 39 80

As we already know, byte "32" in packet #1 is our length byte, which means the real compressed data payload is 49 bytes.
It's obvious that packet #1 holds less data than 49 bytes, it actually holds 42 bytes of payload. This means that the rest of the payload is in another packet.

This is a good place to ask my first question, when the data payload is fragmented, does the rest of the payload ALWAYS come in the following packet?

Let's suppose for a minute that packet #2 is indeed the continuation of packet #1, my logic says that I should treat packet #2 as if the data in it is a continuity of the data in packet #1. Meaning, the 1st byte of packet #2 is NOT a lenght byte, is this assumption correct?
So I decompress the first 7 bytes of packet#2 normally and glue them back to the data I alredy decoded in packet#1.

Now, after I've managed to identify the full payload that started in packet#1, I'm left with a half decoded packet (packet #2). Should I treat the 8th byte of packet#2 (which is for me, really, the 1st byte of new data) as a length byte? the 8th byte is 8C which is 140 bytes of data.

This is my understanding on how this whole mechanism works, can anyone confirm/deny what I just said?
It's really important to me to understand the complete flow of data.

thanks in advance,
J

Yeah, you got it now
apart from:

So I decompress the first 7 bytes of packet#2 normally and glue them back to the data I alredy decoded in packet#1.

Glue the missing data in packet#2 the end of packet #1 so you can re-read its header 0x32, see that you have the whole message, then decompress it.
And yeah, you can safely say that the next packet will be continueing on from the last.

Example:
GetData#1:

Code Select Expand


32 bd 04 4e 0d 01 88 1f 5a 1d 01 59 b1 a1 10 1c 1b 09 e6 87 64 04 65 32 08 81 c0 e6 ba 50 1c 85 00 15 9b 1a 03 71 e8 41 1d 0d 0e

Packet lengh is 50 (payload of 49) but len(data) is 43 (data is missing), so we we store this data and wait.

GetData#2:

Code Select Expand


1b 56 18 00 78 f2 14 8c 0b c4 60 7c ca 2a b0 00 74 1f 91 66 00 48 8c 10 99 39 80

Now we can append this to the end of are stored incompleat data:

Code Select Expand


32 bd 04 4e 0d 01 88 1f 5a 1d 01 59 b1 a1 10 1c 1b 09 e6 87 64 04 65 32 08 81 c0 e6 ba 50 1c 85 00 15 9b 1a 03 71 e8 41 1d 0d 0e 1b 56 18 00 78 f2 14 8c 0b c4 60 7c ca 2a b0 00 74 1f 91 66 00 48 8c 10 99 39 80

Now we re-read the lengh header that says 50, len(data) is => the packet size, so we can snip out the 49 bytes of payload, and skip to the packet past that.
After decompressing the message and snipping it out, go onto the next:

Code Select Expand


14 8c 0b c4 60 7c ca 2a b0 00 74 1f 91 66 00 48 8c 10 99 39 80

The lengh header says its 20 bytes (19 payload size), and all of the data is already there, so its just a case of decompressing this packets payload untill you come to the odd 0x80 byte, which is safe to presume its part of another broken packet, so again, store what you have and wait for the rest.


[EDIT]:

Interesting. Mind sharing your code Ringo?

Theres not much more to it than the method i said above
Set up 3 sockets to listen, one on port 6112 (as bnet) and another on port 6113 (as realm) and another on port 4000 (as d2gs) and have them answer everything as success so d2 can go all the way to ingame, thinking its in a reall realm game.
Also on the same project, have the bot connections so you can make a queue of warden packets and transmite them to d2 as quick as d2 can transmite the result back.
It gets as confusing as you like tho, like when 1 bot does the warden version check on-join, and needs to DL a new warden module.
I spose the best tip i can give, is to make full use of the warden client we already have

[l2k-Shadow]

Interesting. Mind sharing your code Ringo?

I made one of these programs long time ago, basically what you do is make like a gateway between you and battle.net and then when d2 gets the packet with the IP to which to connect for game, edit the IP to yourself and then when your program gets the warden, send it to your open d2 game, d2 game will answer it and you send it back... So basically you just have a d2 dummy running in the background answering warden but everything else you are doing on your own. Does that make sense?

[teK]

Yes I understand it now. I have started work on it but ran into a problem, how do you create packet 0x50 S>C?

Thanks.

[warz]

Has anyone looked into reverse engineering the warden protocol? Is there anything we know about warden as of right now? How does it work? Is there a specific dll file containing the warden related routines? I think I remember there being a warden.dll, or wardenclient.dll or something. I know, after looking at a lot of brood war in a debugger, that warden is present in starcraft and brood war too.

I wouldn't mind doing some reverse engineering in this field.

[Juniper]

Hi!
Me again!

Using the tips you guys gave me here I am now able to glue a lot of S>C traffic back together and then decompress it.

There are still some packets that are confusing me though.

Take this packet for example:

Code Select Expand

01 dc 8a 17 8b 40 18 be 38 87 cc 9e 8a 21 68 92 17 83 92 59 24 ac 4c 0c c1 88 1c 84 01 3c 47 11 42 e8 49 0e c5 e4 20

This is the data portion of a given TCP segment. My previous D2 packet ended in the previous TCP segment (meaning all bytes required to fill whatever was in the D2 size byte were found in the previous TCP segment).
So this means I need to start processing this packet as a "new" one.

This would also mean that the D2 size byte is "01" meaning that the data portion of our D2 packet is zero bytes in length. That's already fairly odd.
If I for a minute assume this to be correct, that would mean that the first D2 packet to be found in this TCP segment is "01" and then the next one starts with "dc". The size byte in the 2nd packet is "dc" and processing could continue afterwards, having whatever is missing here be appended from following TCP segments, and so on.

The problem I am facing though is when I do as I just described then all the output of the packets/segments that follow this one stop making sense. If I ignore this packet (and all other packets where there apears to be more than 1 D2 packet to be found in the same TCP segment), then all my output continues to make sense. I am able to append segments to eachother forming new/real D2 packets as required by the D2 size byte and they decode/decompress to packets that actually seem to be correct for the situation in question.

So anyway, I guess my question is; if anyone were to encounter the packet I pasted above, how would it be processed? Assume that this is the output of ALL D2 bytes, so INCLUDING the header.

thanks,

J