Efficient Logging

[MyndFyre]

Yo --

Anyway, I'm trying to find the most efficient logging method for my bot.  The catch is, that my bot has three concurrent connections.  The logs will be basically chat logs; it has its own schema and is not related to the users database (thus far I have been unable to make 1 users database for three connections, but I think that the current situation is optimal due to my intended uses of the three connections).

Anyway, what I have set up is basically an XML typed DataSet that I add information to whenever an appropriate chat event happens.  Every thirty seconds, unless the connection is closed, I persist the DataSet instance to the hard drive.

At least I don't keep getting "File in use" errors.  But I'm convinced that there is probably a less memory-intensive and generally faster method of doing something like chat logging.  Any suggestions?  The log doesn't need to be a DataSet, but as long as I can write XML out (I'd probably just use String.Format() for it), I'm happy.

Thanks!

[Zeller]

y dont u just create a quary type thing. When a chat event goes off, add the info to to some sort of stack and then raise an event that checks the stack.

or just never close the connection.

[MyndFyre]

y dont u just create a quary type thing. When a chat event goes off, add the info to to some sort of stack and then raise an event that checks the stack.

or just never close the connection.

What exactly are you trying to say?  I'm trying to create a text log file on my hard drive that records the chat events....

And using a stack would be counterproductive to event parsing anyway.  All the newest events would be parsed first, potentially leaving the old events in the dust.

[Zeller]

so create an array of structures or some sort. I just used a stack as an example becous I thought you would just add the new line of text to it everytime some 1 sais somthing + i forgot it works backwards. I meant to say an array

Anway what im trying to say is why wait 30 seconds? Or im just not understanding what your trying to accomplish

[MyndFyre]

While I sit idle in the channel, or while I talk via the interface, I want the bot to record the events, and then persist them to the hard disk.

This is a sample of what I have so far:

Code Select Expand


<?xml version="1.0" standalone="yes" ?>
<ArmaBotLog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.armabot.net/schemas/ArmaBotLog">
<ChatLog Date="2004-02-06T00:00:00.7355440-07:00">
 <ChatEvent Type="UserJoined" Time="2004-02-06T00:28:55.0293360-07:00" User="LoverOfCat-AoA@USEast" Medium="PXES 0 0 11 0 0 0 0 0 0" xsi:nil="true" />
 <ChatEvent Type="UserLeft" Time="2004-02-06T00:29:52.7923952-07:00" User="LoverOfCat-AoA@USEast" xsi:nil="true" />
 <ChatEvent Type="UserJoined" Time="2004-02-06T00:30:17.4478480-07:00" User="LoverOfCat-AoA@USEast" Medium="PXES 0 0 11 0 0 0 0 0 0" xsi:nil="true" />
 <ChatEvent Type="UserLeft" Time="2004-02-06T00:30:20.7025280-07:00" User="LoverOfCat-AoA@USEast" xsi:nil="true" />
 <ChatEvent Type="UserJoined" Time="2004-02-06T02:57:10.7707840-07:00" User="ultimate_offer@USEast" Medium="RATS 971 0 37 0 0 971 0 0 0" xsi:nil="true" />
 <ChatEvent Type="UserLeft" Time="2004-02-06T02:57:33.1629824-07:00" User="ultimate_offer@USEast" xsi:nil="true" />
 </ChatLog>
</ArmaBotLog>


With .NET, I am simply adding rows to the ChatLog table as each event takes place.  Every 30 seconds, I am persisting the XML DataSet back in this form to the disk.

However, after a busy day in the channel, I have no doubt that this would become nice and large, wasting away in memory.  My question is, would there be an easier, less memory-intensive way to do this?

[Grok]

Can you load MSDE?  It supports exporting to XML documents using FOR XML clause.  If you need to connect to it from other machines, you'll need standard edition of SQL.  MSDE does not, iirc, support remote connections, but you could trick it with your own ADO.NET proxy.

Use the bot connection (unique name, or unique combination of fields) as the key and log all chat events to the same table.  When you're ready to write to a text file (why do you need to do this??), put WHERE key='uniquename' in the select.

HTH.
Grok

[Adron]

Can you load MSDE?  It supports exporting to XML documents using FOR XML clause.  If you need to connect to it from other machines, you'll need standard edition of SQL.  MSDE does not, iirc, support remote connections, but you could trick it with your own ADO.NET proxy.

Are you sure MSDE doesn't support remote connections? I was under the impression that it allowed 5-10 connected users with similar performance as an Access database.

[Grok]

It appears to support "five concurrent batch workloads" but no mention of remote users that I can immediately see.

Overview http://www.microsoft.com/sql/msde/productinfo/overview.asp

Features http://www.microsoft.com/sql/msde/productinfo/features.asp

Appropriate Uses http://www.microsoft.com/sql/msde/howtobuy/msdeuse.asp

[AnonProgrammer]

Is the log for viewing purposes only? Does it ever have to be read back in as a dataset?

[MyndFyre]

Is the log for viewing purposes only? Does it ever have to be read back in as a dataset?

Yes, the log is meant to be read back as a DataSet so that I can parse through each event in a log viewer application, that looks similar to the bot chat window.  (The bot chat window only displays the last 100 or so events; this should display them all from a given day).

[Grok]

Is the log for viewing purposes only? Does it ever have to be read back in as a dataset?

Yes, the log is meant to be read back as a DataSet so that I can parse through each event in a log viewer application, that looks similar to the bot chat window.  (The bot chat window only displays the last 100 or so events; this should display them all from a given day).

Ah very easy to do with SQL.

SELECT TOP 100 * FROM EVENTS ORDER BY [YourAutoNumVarName] DESC

[MyndFyre]

LoL Grok, you're a bit confused about what I'm doing...

The chat events come in.  I parse them, and the last 100 or so events to come in are displayed in the output window; the earliest are taken off of the top.  As chat events come in, I also dump them into a separate DataSet instance for logging purposes.  They are later read back into a DataSet when you want to look at the log, and the entire set is displayed in a window.

[AnonProgrammer]

LoL Grok, you're a bit confused about what I'm doing...

The chat events come in.  I parse them, and the last 100 or so events to come in are displayed in the output window; the earliest are taken off of the top.  As chat events come in, I also dump them into a separate DataSet instance for logging purposes.  They are later read back into a DataSet when you want to look at the log, and the entire set is displayed in a window.

Let me see...You don't want to use a database because it adds distribution complexity, and it looks like the appending requirement is going to exclude the standard DataSet and XML and SOAP formatters, and let me guess, you want to be able to view the info in the file by opening it in a text editor as well, hmm, so lets exclude the binary formatter.  How about classes that can append DataRows to a tab delimited text file and then re-construct the DataRows when called upon?  Maybe with a managing class that handles the details of knowing when to create new files from the system date, and loading into a dataset when called upon, and returning a list of existing file dates.

[MyndFyre]

Well in any case, I've been simply using an output stream to generate the XML file, not a typed dataset.  It's worked fine; this problem is pretty old.

[AnonProgrammer]

Well in any case, I've been simply using an output stream to generate the XML file, not a typed dataset.  It's worked fine; this problem is pretty old.

Just keeping a days worth of data in the dataset?