[Python] Pybbot

[topaz]

I recently started work on a python bot (title: Python battle.net bot). The code to it is here.

However, I've run into a problem with how I check for data on the socket (select/polling the socket):

Code Select Expand

while True:
    readable, writable, error = select.select([socket], [], [], 0.10)
   
    if readable <> []:
        temp_data = socket.recv(1024)

That was what I currently use, but as far as I can tell that is extremely inefficient and CPU heavy (it checking the socket for data every 100ms isn't a good thing). Is there a better way to do this? I've seen the topic about using WSAAsyncSelect and WSAEventSelect, but I don't know if that can be done in Python.

[FrOzeN]

I don't know Python so I can't show any code. But I'm pretty sure you would hook the socket just before calling WSAStartup, then when something happens windows should fire the events in the call back function which you can parse out to your bot instead of running a check every 100ms. Before closing you would also have to call WSACleanup, and then unhook it to avoid crashing.

[Kp]

The entire point of select is to sleep until the socket becomes ready or the timer expires.  If you are not doing any other processing when select returns with no data ready, raise the timeout.  My programs typically have a 1-5 minute timeout on select when they are not doing any other work.  Battle.net clients need it to be somewhat shorter for the sake of ad banner requests, connection loss discovery, etc.  Still, you could easily sleep for 5-15 second intervals and not miss anything.

[Banana fanna fo fanna]

Have all your I/O done with a blocking select call and placed into a Queue.Queue as a OnReadEvent or something. Have all your GUI actions run in a separate thread which places GUI events into the said queue. Have a reactor thread that waits on the event queue and takes care of running everything. Have a queue system for IO writes and GUI updates (that is, reactor->gui and reactor-> IO communication).

By setting your bot up in a 3-threaded system like this, you will have a fast, responsive GUI, a fast, responsive I/O system, and a core that is easy to write and maintain. Single-threading your entire bot with I/O and GUI is a bitch to write and will eat your CPU cycles. Multithreading with shared data (i.e. NOT using queues) will fuck you over with deadlock and race conditions. Use queues and events to pass eveyrthing around and you should be OK.

Put as much of the work as you can in the reactor thread. The crux of it should look somehting like:

Code Select Expand


while True:
____evt = self.events.get()
____if evt.type == Event.QUIT:
________break
____elif evt.type == Event.DATARECV:
________handle_data_received(...)
____...


And, for example, when you want to send some data,

Code Select Expand


self.iosubsystem.events.put(WriteEvent("...data goes here..."))


...and when the select call (with the infinite timeout of zero) reports the socket in a ready state for writing and you have received this event in the queue, you write the data.

If you need a prototype I can make one for you. It would be fun. Good luck.

[warz]

In response to the 3 threaded setup - I'm not sure if it'd end up being faster than a single, or even double threaded setup. It costs cpu time to switch between threads like that. It'll slow down network operations, gui operations, etc. The difference may be minimal, and unnoticable, but then again, it could be slow and messy for somebody that hasn't written threaded apps before.

[Kp]

Yes, there would be a cost to the context switches.  However, depending on how well Python's Win32 extensions integrate with the Microsoft Windows GUI model, it may be very difficult to effectively multiplex GUI and network in a single thread.  Such multiplexing is much easier for Unix applications, which enjoy the luxury of treating the console (or X11 connection) as another object on which select can be called.

[Banana fanna fo fanna]

You can't, as far as I know, multiplex with the GUI without using win32 native API calls, which would kill portability, or having a miniscule timeout to select() and just running a loop that would hog CPU cycles.

Python's threads are shitty (GIL). If I were actually doing this and had to do it 100% the right way, I'd spawn 3 separate processes and communicate using message passing on stdio. Your bot instantly becomes dual-core compatible

EDIT - forgot to mention that the GIL is the global interpreter lock; Python locks everything in the execution space automatically, so the efficiency of threads is reduced (especially when we are using a share-nothing message-passing model emulated here using queues), though still beneficial in this situation.

[topaz]

Thinking about using Twisted -- there isn't really a *good* way to do it so far. Threads are pretty much overkill, and the infinite select/polling i'm doing isn't good either.

[Banana fanna fo fanna]

If Python had real (non-purely-functional) lambdas, Twisted would be a lot less of a pain in the ass. I'd rather use the 3 thread/process model.

[lordkrandel]

I know really little of Python but I want to learn.
Maybe I could give/have some help. I really like the
idea of an open-source interpreted battlenet bot.

Can this page be of any help?
It is about a core library handling asyncronous sockets.
It is called "asyncore"

http://www.python.org/doc/2.4/lib/module-asyncore.html

I have another link for you...
http://squirl.nightmare.com/medusa/async_sockets.html

[squeegee]

Asyncore is a piece of crap, I suggest you never take a look at it again

[Banana fanna fo fanna]

squeegee is retarded