my goal is to have most of the key game data already parsed and in the database. to an extreme, this is a fantasy, but I think I can get a lot done. this would include all sector, port and fig hit data.

Storing all fighit data in the database would be difficult. The database would have to be massive,
and probably prone to corruption. That would have to be carefully engineered.

I routinely kept full logs from every day of game play for historical data mining. that was essentially a complete database of the game, albeit difficult to manage and completely uncompressed. it was megabytes of data. even a marginally competent database design that contained all useful game data would be at least an order of magnitude smaller. so I don't buy the massive database size argument.

I agree that it should be carefully designed for stability and speed. I'm using sqlite for my database solution and I have a clear design for recovering from accidental "corruption".

Yeh, but some perspective is in order. All of that was stored as text, and not as a blob. Additionally, you don't play a lot of unlims. That limits the amount of data you have to store. If all you do is append each line to a text file, you're not storing a lot of information. There's a complete millisecond timestamp of each hit, there's all the ansi codes, and all the relational information like index number to consider too. A person playing a fast-paced unlim might get over 1 million fighit messages over the course of the game. If each record is 100 bytes, which is not an unreasonable record size, and you have 1 million fighits, that's 100megs of data.

That automatically limits some people on hard drive size, and requires some degree of compression. But additionally, you can't store that kind of data in memory... you'd have to page it out. That's some creative engineering given the timings involved. Read and write operations cannot be slower than a few milliseconds, or otherwise it's useless in competitive play. It can be done, I'm sure, but you might find that it's useful to break up the data and store fighits in a separate DB, with an option to expire stuff beyond a certain age. I've been down that road on a few projects, and SQL is very CPU intensive on write operations as record volumes increase.

You might want an option of turning off hits by aliens for storage in the database. I've had games where thousands of hits per day were caused by them in the unlims I play. As sing pointed out, unlims - esp the truce variety - have an awful lot of fig hits either during truce or the grid war that erupts just before truce ends. The option of when and if to start recording might be good even for player hits.

               / Promethius / Enigma / Wolfen /

"A man who has no skills can be taught, a man who has no honor has nothing."

for most purposes, all you really need to know is who was the last person to hit a sector. so you could practically limit the fig hit list to the number of sectors in the game. so unless you want to do some behavioral analysis of player movement, you can toss a lot of historical fig hit data with little penalty. (and something tellsme unlimited turns games do not lend themselves to tons of analysis.)

Nod, which is pretty much what's being said. In TWXproxy we have sector parms. Last fighit (date and
who) is a parm I use frequently. If all you do is limit each sector to one hit, you'd solve the problem.
Maybe that should be an option, keep X hits per sector?

Unlims can have some pretty intense analysis of fighits tho. Not for gridding list sake, more for fighit
patterns and timing's sake. It would not be bad to pull up all fighits, sort by time, then determine the
average timing between the last 20 fighits as a timed plock technique.

i'm willing to be wrong about unlims requiring technical analysis. hehe. i know there is a lot that can be done with a critical mass of game data. that is mostly where i'm coming from myself - i was a game data monkey. if i can find a way to keep all the data in an accessible format without sacrificing speed and playability, that would be ideal. i can see you have thought about it perhaps more than i have. i'm getting pretty close to having a full vertical slice of features coded - proxy, parsing, database, and scripting being the four horsemen - live data sharing with teammates being the killer addon - and useful plugins (graphical map, quickstats, macrobar) being icing on the cake.

what language are you writing your helper in rev?

i've been playing around with mac programming a bit in obj-c . there are no helpers for mac that are worth crap. well there is one, but i didn't like it. i didn't like the interface. i'm sure it could work fine since it used applescript, but still i didn't like the programs interface at all. and i don't really want to run in wine. i tried with twx but it was flakey. i'd rather use something built for my mac haha. why did i have to try a mac. all because i've never had one. i don't want to dual boot either. or run vmware.

i started researching telnet protocols. and playing with socket streams. but never ventured further then that.

is it written in c? or c++? lol. maybe it could be a easy port to a mac with some modifications. if not, maybe some day i'll get off my butt and make one myself with the features I like best from each helper.