Valhalla Legends Archive

Programming => Battle.net Bot Development => Topic started by: Archangel on March 27, 2005, 06:38 PM

Title: BNCSutil Memory Problems?
Post by: Archangel on March 27, 2005, 06:38 PM
Well, actually, when connecting with a single cdkey product (not war3) memory goes up to 2,000 kb,  when connecting using warcraft3 memory goes up to like 10,000 kb.

Each time i "reconnect" (SEXP) using BNCSUtil memory never goes down, and goes up by 2,000 kb. Its a problem im causing or a BNCSUtil.dll problem?
Title: Re: BNCSutil Memory Problems?
Post by: The-FooL on March 27, 2005, 07:02 PM
I just tested with my bot(newly implemented BNCS util code).

Connecting via WAR3 causes the jump of over 10k mem vs connection with BNLS.  Every connection with sc/d2 causes a jump of 2,000k (these are cumulative.  Connecting a few times doubles the memory usage).

I am using the NLS_Free and kd_free method, and calling them as soon as the decoding is done/login sequence is finished.  I also noticed that I am getting a handle of zero for the decoder(though it still works).  What is the issue here?
Title: Re: BNCSutil Memory Problems?
Post by: Archangel on March 27, 2005, 07:17 PM
Ok, StealthBot Beta got same problem, so i suppose it must be the library.
Title: Re: BNCSutil Memory Problems?
Post by: UserLoser. on March 27, 2005, 07:21 PM
Does BNCSUtil cache all the file data for War3 game files?  Those are quite large, and may be the cause
Title: Re: BNCSutil Memory Problems?
Post by: The-FooL on March 27, 2005, 07:28 PM
It must be some sort of memory leak, because the increase is cumulative.
Title: Re: BNCSutil Memory Problems?
Post by: Networks on March 27, 2005, 08:54 PM
Quote from: The-FooL on March 27, 2005, 07:02 PM
I just tested with my bot(newly implemented BNCS util code).

Connecting via WAR3 causes the jump of over 10k mem vs connection with BNLS.  Every connection with sc/d2 causes a jump of 2,000k (these are cumulative.  Connecting a few times doubles the memory usage).

I am using the NLS_Free and kd_free method, and calling them as soon as the decoding is done/login sequence is finished.  I also noticed that I am getting a handle of zero for the decoder(though it still works).  What is the issue here?

Oh thank god, I thought I was just a poor programmer. whew, yes Archangel found this out in my moderation bot. I posted about this earlier and thought it got fixed but I guess I just connected using a different client than WAR3. I guess my topic is unresolved then. What ever occurs happens to me.
Title: Re: BNCSutil Memory Problems?
Post by: Archangel on March 27, 2005, 09:08 PM
I made test with 3 bots, Vanquish Bot, StealthBot and with one im currently developing. Im waiting for Cloaked answer or fix.
Title: Re: BNCSutil Memory Problems?
Post by: Kp on March 27, 2005, 09:28 PM
Rebuild with debugging symbols (dwarf-2 preferred), and run it in valgrind with --tool=memcheck.  Report what line(s) are leaking the memory.
Title: Re: BNCSutil Memory Problems?
Post by: Networks on March 28, 2005, 07:55 AM
Quote from: Kp on March 27, 2005, 09:28 PM
Rebuild with debugging symbols (dwarf-2 preferred), and run it in valgrind with --tool=memcheck.  Report what line(s) are leaking the memory.

Wow...I am sorry but I didn't really comprehend that but ok.
Title: Re: BNCSutil Memory Problems?
Post by: MyndFyre on March 28, 2005, 11:36 AM
Quote from: Networks on March 28, 2005, 07:55 AM
Quote from: Kp on March 27, 2005, 09:28 PM
Rebuild with debugging symbols (dwarf-2 preferred), and run it in valgrind with --tool=memcheck.  Report what line(s) are leaking the memory.

Wow...I am sorry but I didn't really comprehend that but ok.

What's hard about that statement?

Rebuild the library.  Make sure the compiler/linker emits debugging symbols (a debug build) into the PE file.
Get the utility called "valgrind" and run the program with that utility, using the command line option "--tool=memcheck".

It's English.  Do you speak it?
Title: Re: BNCSutil Memory Problems?
Post by: Arta on March 28, 2005, 01:23 PM
Relax. For someone not familiar with those tools and techniques, that advice would probably be rather difficult to follow.

No point in assuming that everyone's an expert :P
Title: Re: BNCSutil Memory Problems?
Post by: Archangel on March 28, 2005, 01:32 PM
[quote author=MyndFyre link=topic=11069.msg105976#msg105976

What's hard about that statement?

Rebuild the library.  Make sure the compiler/linker emits debugging symbols (a debug build) into the PE file.
Get the utility called "valgrind" and run the program with that utility, using the command line option "--tool=memcheck".

It's English.  Do you speak it?
Quote

Ok calm down, alot of people dont understand alot of stuff in here, so why being so damn agressive?
Title: Re: BNCSutil Memory Problems?
Post by: MyndFyre on March 28, 2005, 02:05 PM
Quote from: Archangel on March 28, 2005, 01:32 PM
Ok calm down, alot of people dont understand alot of stuff in here, so why being so damn agressive?
Quote from: Arta[vL] on March 28, 2005, 01:23 PM
Relax. For someone not familiar with those tools and techniques, that advice would probably be rather difficult to follow.
To clarify my last statement and then question:

I'm not familiar with the tool "valgrind".  I've never heard of it, nor do I know what it does.  My point was that there is a lot of information in the statement Kp made that you can read and understand if you try to think about the statement, and then there is a lot of implied information.

It's like the other day when I came home early to help my mom out with a party she was getting ready to have.  I told her that I came home to help.  Her response, "Help?  What do you mean, 'help'?"  Well wtf do you think I mean?  If she had thought about the context of the situation and my statement, it would have been pretty clear.  It's hardly an exercise in critical thinking.

Similarly, it's reasonable to expect that Networks would analyze the situation in terms of what Kp was suggesting; I understand if someone doesn't know what debugging symbols are.  However, the rest is quite clear: valgrind is a tool to obtain, and it uses the binary and debugging symbols file to figure out what lines in the source code are causing memory leaks.

Appropriate questions would have been "How do I build with debugging symbols?  What are dwarf-2 symbols, and how are they different?  Where do I get valgrind?"

The question was not meant to be aggressive, and for that, I apologize.  It's just frustrating when people don't try to figure something out.
Title: Re: BNCSutil Memory Problems?
Post by: iago on March 28, 2005, 03:05 PM
I grant you what you said, but you have to admit that Kp goes out of his way to make what he says difficult to understand :P

And, to post a bit back on topic, I'll talk to the guy who wrote BNCSutil tonight and point him to this thread.
Title: Re: BNCSutil Memory Problems?
Post by: Networks on March 28, 2005, 03:56 PM
Quote from: MyndFyre on March 28, 2005, 02:05 PM
Quote from: Archangel on March 28, 2005, 01:32 PM
Ok calm down, alot of people dont understand alot of stuff in here, so why being so damn agressive?
Quote from: Arta[vL] on March 28, 2005, 01:23 PM
Relax. For someone not familiar with those tools and techniques, that advice would probably be rather difficult to follow.
To clarify my last statement and then question:

I'm not familiar with the tool "valgrind".  I've never heard of it, nor do I know what it does.  My point was that there is a lot of information in the statement Kp made that you can read and understand if you try to think about the statement, and then there is a lot of implied information.

It's like the other day when I came home early to help my mom out with a party she was getting ready to have.  I told her that I came home to help.  Her response, "Help?  What do you mean, 'help'?"  Well wtf do you think I mean?  If she had thought about the context of the situation and my statement, it would have been pretty clear.  It's hardly an exercise in critical thinking.

Similarly, it's reasonable to expect that Networks would analyze the situation in terms of what Kp was suggesting; I understand if someone doesn't know what debugging symbols are.  However, the rest is quite clear: valgrind is a tool to obtain, and it uses the binary and debugging symbols file to figure out what lines in the source code are causing memory leaks.

Appropriate questions would have been "How do I build with debugging symbols?  What are dwarf-2 symbols, and how are they different?  Where do I get valgrind?"

The question was not meant to be aggressive, and for that, I apologize.  It's just frustrating when people don't try to figure something out.

Rebuild with debugging symbols (dwarf-2 preferred), and run it in valgrind with --tool=memcheck.  Report what line(s) are leaking the memory.

In bold are the only real lines I can fully comprehend. I understood what he was saying. Similarily when your mom says "help!", I understood that this was to help us find where the memory leaks are occuring. The method in which he is explaining is hard to comprehend.

Doing what Kp said isn't go to help us, it's up to Cloaked (ShadyPalm88) to fix it and he probably will be able to understand what Kp stated. =)
Title: Re: BNCSutil Memory Problems?
Post by: Kp on March 28, 2005, 08:16 PM
Wow.  It's been a while since I've caused so much activity with so little effort. :)  Remarks, in no particular order:

Networks: if you don't understand something, a vague "I don't get it", isn't very helpful.  With such a non-specific problem, I doubt I would've been motivated to address your confusion.  Also, although shadypalm is the primary distributor at this point, it's not his obligation to fix it.  I suspect he'd be grateful for a patch instead of an "it doesn't work; fix it!" message.  That is, you should try to fix it yourself and send him the patch if it works.  See below about valgrind.

Arta: sure, not everyone's an expert.  I was a bit time pressed, and felt it better to toss out some concise instructions that an expert could interpret (and hope for the best).  If the recipient wasn't an expert, he would have the side joy of learning about valgrind and its functionality as a consequence of my message. :)

iago: not at all.  I only make it difficult to understand if it's not a lot of effort for me. :)  Jokes just aren't worth it if I have to spend days setting them up. ;)

For those who are not familiar with valgrind: it's a tool to perform dynamic instrumentation of a binary for purposes of profiling, correctness checking, or just about anything else you want to write a tool to do.  Binaries built with debugging symbols produce more informative reports than those which are stripped ( = have no debugging symbols).  The tool memcheck would detect any out of bounds reads/writes, as well as detect memory leaks when the process terminates.  Caveat: valgrind is Linux specific.  There's some rumors that they might try to port it to other Unices, but Microsoft Windows is entirely out of the question for the foreseeable future.  MS-Windows simply has far too many system calls to consider writing wrappers, since valgrind cannot follow execution into kernel mode, and thus must wrap system calls to update its post-call view of process state.  BNCSutil builds on Linux systems, so that's not a problem in this case.

There, that wasn't so unclear, was it? ;)
Title: Re: BNCSutil Memory Problems?
Post by: effect on March 28, 2005, 09:21 PM
or.. you couldve just said that originally instead of being a hardass on a forum!.

Lowest level of life = forum hardasses
Title: Re: BNCSutil Memory Problems?
Post by: BaDDBLooD on March 28, 2005, 09:26 PM
Quote from: effect on March 28, 2005, 09:21 PM
or.. you couldve just said that originally instead of being a hardass on a forum!.

Lowest level of life = forum hardasses

Keep that to Private Messages please.

EDIT: Thanks KP, that will really come in handy in the future.
Title: Re: BNCSutil Memory Problems?
Post by: MyndFyre on March 28, 2005, 09:29 PM
[off-topic]
Quote from: Kp on March 28, 2005, 08:16 PM
Caveat: valgrind is Linux specific.  There's some rumors that they might try to port it to other Unices, but Microsoft Windows is entirely out of the question for the foreseeable future.
Hahaha.  I never thought of the plural of either of those being "Unices" nor "Linuces," although I suppose they are accurate.  Nice. :)
[/off-topic]
Title: Re: BNCSutil Memory Problems?
Post by: warz on March 28, 2005, 11:27 PM
Quote from: MyndFyre on March 28, 2005, 02:05 PM
Quote from: Archangel on March 28, 2005, 01:32 PM
Ok calm down, alot of people dont understand alot of stuff in here, so why being so damn agressive?
Quote from: Arta[vL] on March 28, 2005, 01:23 PM
Relax. For someone not familiar with those tools and techniques, that advice would probably be rather difficult to follow.
To clarify my last statement and then question:

I'm not familiar with the tool "valgrind".  I've never heard of it, nor do I know what it does.  My point was that there is a lot of information in the statement Kp made that you can read and understand if you try to think about the statement, and then there is a lot of implied information.

It's like the other day when I came home early to help my mom out with a party she was getting ready to have.  I told her that I came home to help.  Her response, "Help?  What do you mean, 'help'?"  Well wtf do you think I mean?  If she had thought about the context of the situation and my statement, it would have been pretty clear.  It's hardly an exercise in critical thinking.

Similarly, it's reasonable to expect that Networks would analyze the situation in terms of what Kp was suggesting; I understand if someone doesn't know what debugging symbols are.  However, the rest is quite clear: valgrind is a tool to obtain, and it uses the binary and debugging symbols file to figure out what lines in the source code are causing memory leaks.

Appropriate questions would have been "How do I build with debugging symbols?  What are dwarf-2 symbols, and how are they different?  Where do I get valgrind?"

The question was not meant to be aggressive, and for that, I apologize.  It's just frustrating when people don't try to figure something out.

Dang, your mom is dumb. My mom is definantly better. Sorry.
Title: Re: BNCSutil Memory Problems?
Post by: Kp on March 29, 2005, 12:29 AM
Quote from: effect on March 28, 2005, 09:21 PMor.. you couldve just said that originally instead of being a hardass on a forum!.

Lowest level of life = forum hardasses

Like I said (which you'd know if you'd actually read my post), I was time pressed and felt it was better to give him some hints about where to start, rather than just ignore him.  Besides which, I hardly think that my original message constitutes "being a hardass".  It could've been more helpful if I'd had more time and inclination when crafting it, but it was by no means rude.  That said, don't provoke the moderators.
Title: Re: BNCSutil Memory Problems?
Post by: Networks on March 29, 2005, 11:00 AM
Isn't Cloaked the main distributor? It's his dll, his open-source project. I don't know as much C++ as I'd like to know to probably fix it.
Title: Re: BNCSutil Memory Problems?
Post by: iago on March 29, 2005, 11:19 AM
Quote from: Networks on March 29, 2005, 11:00 AM
Isn't Cloaked the main distributor? It's his dll, his open-source project. I don't know as much C++ as I'd like to know to probably fix it.

Yes, he is.  But it's opensource, so if I find the problem I can send him a .patch file and he can apply it, then everybody is happy.  If he has made other changes since then, a .patch will merge with them.  It's very handy.
Title: Re: BNCSutil Memory Problems?
Post by: shadypalm88 on April 03, 2005, 09:55 PM
Hi everybody.  When I heard about the memory usage I figured there was only one thing that could possibly use that much: CheckRevision.  On Windows systems this uses memory mapping.  Although the memory handle was being closed, the memory wasn't actually being unmapped.  This was fixed and should end the memory troubles.

Edit: I also fixed a (much smaller) leak in WarCraft III decoding (thanks for the ValGrind tip), and it compiles on *NIX "out of the box" now.
Title: Re: BNCSutil Memory Problems?
Post by: UserLoser. on April 03, 2005, 10:05 PM
Quote from: UserLoser on March 27, 2005, 07:21 PM
Does BNCSUtil cache all the file data for War3 game files?  Those are quite large, and may be the cause
Title: Re: BNCSutil Memory Problems?
Post by: iago on April 03, 2005, 10:48 PM
Quote from: UserLoser on April 03, 2005, 10:05 PM
Quote from: UserLoser on March 27, 2005, 07:21 PM
Does BNCSUtil cache all the file data for War3 game files?  Those are quite large, and may be the cause

Wow, you're cool!  If you knew the problem, why didn't you fix it while shadypalm88 was gone?
Title: Re: BNCSutil Memory Problems?
Post by: UserLoser. on April 03, 2005, 11:35 PM
Quote from: iago on April 03, 2005, 10:48 PM
Quote from: UserLoser on April 03, 2005, 10:05 PM
Quote from: UserLoser on March 27, 2005, 07:21 PM
Does BNCSUtil cache all the file data for War3 game files?  Those are quite large, and may be the cause

Wow, you're cool!  If you knew the problem, why didn't you fix it while shadypalm88 was gone?

If you are going to say something that would probably get you punched in the mouth in person, do not say it. (http://forum.valhallalegends.com/phpbbs/index.php?topic=7754.0)
Title: Re: BNCSutil Memory Problems?
Post by: Maddox on April 04, 2005, 04:18 AM
Then shut up.
Title: Re: BNCSutil Memory Problems?
Post by: iago on April 04, 2005, 08:40 AM
Quote from: Maddox on April 04, 2005, 04:18 AM
Then shut up.

Haha, exactly.