| View unanswered posts | View active topics |
It is currently Wed May 06, 2026 3:32 am |
|
All times are UTC - 5 hours |
|
Page 2 of 3 |
[ 38 posts ] | Go to page Previous 1, 2, 3 Next |
| Print view | Previous topic | First unread post | Next topic |
Accessing data files
| Author | Message |
|---|---|
|
Commander
Joined: Thu Feb 20, 2003 3:00 am Posts: 1529 Location: USA 
|
 From John:
quote: As long as he's not doing anything to undermine the registration system, it's fine. I've even given details about file structure to people on request. This kind of 3rd party add-ons for TW used to be more common, but the v3 engine is more complex than earlier versions, and that makes it more difficult. But it's still allowed. |
| Fri Jun 17, 2005 3:17 am |
 
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Thank you! I get that info at the same time as solving the TWINDEX.NDX file structure on my own. Once I realized every record had fixed length, it all fell into place. Only two records I can't identify, and that's because in all the games I went through their data was all null, so I suspect they're unused or reserved records anyway.
Now I can start the actual programming part, since I know all of the file offsets to use. [:)] Edit: Anyone have a suggestion as to a nice elegant algorithm for converting a 48-bit float value (the type used for credit fields in the TW data files) to decimal in C++, which only natively supports 32-bit and 64-bit floats? I've got a conversion routine that manually calculates out the mantissa and exponent and so forth but it's ugly and I'm sure there's a better way. _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Fri Jun 17, 2005 3:49 am |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Well, I've officially created a little 8k program that simply outputs the # of fighters and their owners in each sector that has fighters. So my calculated offsets and data are confirmed accurate. Now to do a little more with the gathered data. [:)]
Sample output from a defunct game I'm using for dummy data: Game has 20000 sectors. Sector 31 has 1 fighters which belong to Player 3. Sector 64 has 1 fighters which belong to Player 3. Sector 67 has 1 fighters which belong to Corporation 1. Sector 76 has 1 fighters which belong to Corporation 1. Sector 78 has 1 fighters which belong to Player 3. Sector 93 has 1 fighters which belong to Player 3. Sector 138 has 1 fighters which belong to Corporation 1. Sector 168 has 1 fighters which belong to Player 3. Sector 181 has 1 fighters which belong to Player 3. Sector 188 has 1 fighters which belong to Player 3. Sector 254 has 1 fighters which belong to Player 3. Sector 269 has 1 fighters which belong to Player 3. Sector 281 has 1 fighters which belong to Player 3. Sector 282 has 1 fighters which belong to Player 3. Sector 305 has 1 fighters which belong to Player 3. Sector 316 has 1 fighters which belong to Player 3. Sector 360 has 1 fighters which belong to Player 3. Sector 435 has 1 fighters which belong to Player 3. Sector 443 has 1 fighters which belong to Player 3. Sector 459 has 1 fighters which belong to Player 3. Sector 461 has 1 fighters which belong to Player 3. Sector 477 has 1 fighters which belong to Corporation 1. Sector 478 has 73 fighters which belong to an Alien Race. Sector 484 has 1 fighters which belong to Player 3. Sector 516 has 1 fighters which belong to Corporation 2. ...and of course it goes on through sector 20000. I takes about one tenth of one second to run. Just a tad of an improvement over a tedit script. [;)] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Sat Jun 18, 2005 9:46 am |
|
|
|
|
Lieutenant Commander  Joined: Thu Mar 21, 2002 3:00 am Posts: 890 Location: USA
|
Nice! does the server have to be stopped to run it? Mind sharing it?
_________________ http://tw-cabal.navhaz.com - THE TW info site Man, I gotta quit showing up here...next thing you know i'll get dragged back in. |
| Sat Jun 18, 2005 2:21 pm |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Nope, since it opens the files in read-only mode, it runs fine even while the server is in use. It keeps the file open just for a split second to pull the data into memory and then does all the processing in memory, so there's little to no danger of data changing mid-read.
I'm not distributing this program, but only because I'm working on a more robust one that will report more information. This was just a "tech demo" of sorts, not clean enough for wide release, but I wanted to release the *results* of this beta test so people would know I'm making good progress. My current thought is to write a program that will simply convert the TW database to an XML file. From there, a web application can process the data in any way it wants, a CSS could be written to format it in nice ways for a web-served stats page, and so on. This first little iteration was just proving to myself and the community that it can be done, and works very well. [:)] Adding the rest of the information shouldn't take me too long, and when I have something worthy of distribution, I will do so. The whole point of this is to serve the community and possibly start an entire third-party utility revolution for v3. After that, next step will be attempting writing data and seeing if that works reliably, without corruption, with or without the server running. Even if I can only get it to work without the server running, it would allow for some interesting alternatives to TEdit for setting up scenarios that involve the placement of a lot of assets. Wish me luck! [:D] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Sat Jun 18, 2005 3:35 pm |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Well, I've made quite a bit of progress today, including converting Pascal-style strings to C++ ones, and, the trickier one, converting those 48-bit floating point "real48" values to doubles. It seems to be working, as all of the various credit values I tried (500, 5000, 598234789) were interpreted exactly as they were input.
I've also confirmed that a bunch of various data is being interpreted as expected. I've converted all of the access routines to classes and added some info to the previous demo. Sample output: Sector 18433 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18441 in {y-}Ferrengi Space has 50000 offensive fighters which belong to Gold Alien Race #62. Sector 18450 in unexplored space has 5000 defensive fighters which belong to Corporation #1. Sector 18452 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18465 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18509 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18537 in unexplored space has 1 toll (with 5 credits) fighters which belong to Corporation #1. Sector 18607 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18622 in The Slogovian Estates has 1 toll fighters which belong to Corporation #1. Sector 18642 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18738 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18748 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18755 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18756 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18787 in unexplored space has 1 toll fighters which belong to Player #3. Sector 18788 in unexplored space has 2938 defensive fighters which belong to Player #3. A beacon is present in the above sector which says: Bye Sector 18789 in unexplored space has 1 toll fighters which belong to Corporation #1. Sector 18828 in unexplored space has 1 toll fighters which belong to Player #3. Note: The odd characters in front of Ferrengi Space are, I believe, color codes or something, since they're only present for that gold alien nebula, and Ferrengi space is gold/dark yellow rather than green like other nebulae. Oh, and another side note, the program is if anything even faster now. There's no discernable delay between hitting enter to run it and the next command prompt appearing, output.txt file fully created. At any rate, I'm coming close to having something worth distributing. Won't be long. [:)] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Sun Jun 19, 2005 12:23 am |
|
|
|
|
Commander  Joined: Tue Oct 07, 2003 2:00 am Posts: 1134 Location: Augusta, GA
|
This could be a huge time saver for sysops, and you're probably just scratching the surface of how it could be used. I'm interested to hear about the file formats and how to interpret them when you've got it whipped.
+EP+ _________________ Claim to Fame: only guy to ever crack the TW haggle algorithm, and fig/shield/hold price formula, twice. |
| Sun Jun 19, 2005 9:40 pm |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
I'm sure that's true (just scratching the surface). Keep in mind it'll be read-only for the time being, though. I'm 98% certain I can write successfully, but would rather make sure things work and I know what all of the values mean on the read side first.
I plan to do one better than just documenting the formats and release a full abstracted interface class for file access. So far I have per-file classes, and plan to add another couple of layers of abstraction above that one, a master TWDataInterface class containing each of the other file classes (to ensure initialization of the data in the proper order; twcfig first, since # of sectors, max players, max planets, max ports, etc. are important to open the other files), and then perhaps a set of interface objects that make sense from a user standpoint, such as a Sector class (complete with iterator) that you pass a TWDataInterface to and which you can poll information on various sectors with. At the base level of abstraction that the classes I have now work from, you have to know what info is where and poll 3 or 4 different files for information on the same basic "thing" sometimes, and I'd like to hide the implementation as much as possible such that you can get all of the information you think of as sector information from one source, rather than having to know that oh, the number of fighters is in TWSectData, but who OWNS those fighters is in TWIndexData, and so on. I'd rather add a layer of opacity and save myself a lot of documentation. [:)] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Sun Jun 19, 2005 11:21 pm |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Here are the basic record structures, since JP freely gives these out anyway, converted to C++ form. The game is written in Pascal, and the original types are Pascal types; I had to use closest available types in some cases. Note that Pascal strings are different from C++ strings, so I had to write a function to convert them. I also had to write a custom function for conversion of Real48's (an old Delphi/Pascal-only floating-point variant, which I just load as a 6-byte array of unsigned characters) to integers (I decided against conversion to double since the only real48 fields are credit values, which are whole numbers anyway, and 1 billion is the limit on # of creds OH, so simply returning unsigned longs is more than adequate; citadel credits can go well beyond 1 billion, but are stored as a doubles in the data files anyway.)
The TWINDEX.NDX file format is not provided here because it's not quite the same simple fixed-record format as these files. I also didn't include TWCFIG.DAT's structure because all of the configuration data available in that file is easily directly accessible via * on the main menu anyway. Code: struct sectDataStruct { unsigned short ID; char nebulaName[42]; char beaconName[42]; unsigned short adjWarp[6]; unsigned short portNum; unsigned char navHaz; unsigned long numFigs; unsigned char numArmid; unsigned char numLimpet; unsigned char figType; // 0 - defensive, 1 - toll, 2 - offensive unsigned char tollCreds[6]; //real48 unsigned long pWaveStart; }; struct userDataStruct { unsigned short ID; bool initialized; char gameName[42]; char BBSName[42]; char password[9]; unsigned short lastDayOn; unsigned short killedBy; unsigned short turnsLeft; double bountyVal; unsigned short numBounties; double contractVal; unsigned short numContracts; double bankVal; unsigned short bountyClaimant; unsigned short contractClaimant; unsigned short navPoints[4]; unsigned short lastDayTurns; unsigned short lastHourTurns; float turnFraction; unsigned short eventFlags; unsigned short userFlags; unsigned char timesBlownUp; unsigned short onPlanetNum; unsigned short lastSector; unsigned long alignment; unsigned long experience; unsigned short currentShip; unsigned char creditsOH[6]; //real48 unsigned long timeOnline; unsigned short lastRobbed; unsigned short lastBusted; unsigned char numPods; // # used today }; struct corpDataStruct { char name[42]; char CEOName[42]; char creationDate[9]; char password[9]; unsigned char rank[6]; // real48 unsigned short kills; }; struct portDataStruct { unsigned short ID; char name[26]; char lastShipPorted[31]; unsigned long lastVisitedTime; unsigned long lastUpdatedTime; unsigned char type; // class - can't name it that though ;) unsigned short oreInv; unsigned short orgInv; unsigned short equInv; unsigned short oreProd; unsigned short orgProd; unsigned short equProd; signed char oreMCIC; signed char orgMCIC; signed char equMCIC; unsigned char daysUntilComplete; unsigned short lastRobbedBy; unsigned char profit[6]; // real48 }; struct planetDataStruct { unsigned short ID; char name[42]; char creatorName[42]; unsigned short lastDayProd; unsigned short lastMinProd; unsigned long oreColos; unsigned long orgColos; unsigned long equColos; unsigned long oreOH; unsigned long orgOH; unsigned long equOH; unsigned char figReaction; unsigned char atmosQuasar; unsigned char sectQuasar; unsigned char typeID; unsigned long numFigs; unsigned char citLevel; // 1 = none, 0 = uninitialized unsigned short daysUntilComplete; unsigned short numShields; double treasuryCreds; unsigned short xportLvl; unsigned char lastSecProd; unsigned short unknownValue; }; struct shipDataStruct { unsigned short ID; char name[31]; char builtDate[31]; char password[11]; unsigned long numShields; unsigned long numFigs; unsigned char numHolds; unsigned char oreOH; unsigned char orgOH; unsigned char equOH; unsigned char coloOH; unsigned short planetNum; unsigned long numPorts; unsigned char typeID; // ship class unsigned char subtypeID; // manufacturer unsigned char TWarp; // 0 - none, 1 - TWarp, 2 - TWarp tow unsigned char numCloaks; unsigned char LRscan; // 0 - none, 1 - density, 2 - holo unsigned char numGenTorps; unsigned char numArmids; unsigned char numLimpets; unsigned short numKills; unsigned short numBeacons; unsigned char numPhotons; unsigned char numAtomDets; unsigned short numCorbo; unsigned short numProbes; unsigned short numDisruptors; bool psychicProbe; bool planetScanner; unsigned char xferLock; unsigned short limpetOwner; unsigned short towOwner; }; Once again, I plan on having a library written and ready for public consumption within a week or two (depending on how busy my actual paying job keeps me), so I wouldn't go through too much trouble writing your own access routines at this point, honestly. [:)] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Mon Jun 20, 2005 5:37 am |
|
|
|
|
Captain  Joined: Sat Jun 18, 2005 2:00 am Posts: 2214 Location: USA
|
Xentropy, you are a true Golden God, you assume man!! Were/how did you obtian you ability to program? You seem to be pretty CHT hot, you ever think about mentoring? Looking for a pupil, hehee... [;)]
On another note, the next TW version should be coming out soon, your going to continue your project to ensure compatibility with that to right? _________________ Your reliance upon subjective IRM's, subjugates you through utter omission, obfuscation, and distortion of fact! Don't mess with me, I will 26 U.S.C. § 7212(a) your IRS! |
| Mon Jun 20, 2005 6:59 am |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
v3 shouldn't change file formats too much from version to version. An added field here and there or a data type switched from one to another is possible, but rare. I will keep up with such changes as necessary.
v4 will be a completely different engine, so if such an interface library is necessary at all for it (it may be designed up-front to be friendlier to 3rd-party apps for all I know) it would require working from scratch again, so I'll cross that bridge when I come to it. No matter how it goes, I won't bother with interfacing with v4 until it hits a solid beta release and the formats are stable. _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Mon Jun 20, 2005 5:10 pm |
|
|
|
|
Captain Joined: Sat Jun 18, 2005 2:00 am Posts: 2214 Location: USA
|
That makes sense about waiting for the bugs for v4 to be repaired before creating new utilities for it, i didnt think of that. Overall all things considering, thats good new though!
_________________ Your reliance upon subjective IRM's, subjugates you through utter omission, obfuscation, and distortion of fact! Don't mess with me, I will 26 U.S.C. § 7212(a) your IRS! |
| Mon Jun 20, 2005 7:00 pm |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
Well I (finally) completed a version of the barebones interface classes that I (a terrible perfectionist) can be happy with. Now I just need to work on the upper-level interface classes that will actually be transparent to the user and I'll be ready to release a library for everyone.
Since I think it would be less work to use what I have now to release a basic program to demo the functionality than to write the rest of the interface classes, I think I'll work on that next. I was thinking of two possibilities, and I may do both. One would just list the net worth of each player and corporation based on their total assets (multiplied by actual game prices of various assets, current fig/shield prices, and so on), and then list the contents of every sector with something other than just a port or single fighter in the same format as the game does sector displays. Just basically "this is what you can do" stuff. The other thing I'm considering releasing as a demo is much less useful but fun and amusing. As to not spoil the joke, I won't say more at this time. If they work anything like my current driver program to test out the interface classes do, they'll take well under a second to process a 20000-sector universe. (What I have currently takes 0.06 seconds to write a 73k output file of various info on a 20k universe, though I'm on a 3.6Ghz P4 so that 0.06 seconds may actually take half a second on some systems. [;)]) Sorry for all the posts, but I wanted to keep everyone apprised on my progress. I'm not going to just drop this project. [:)] _________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Wed Jun 22, 2005 5:03 am |
|
|
|
|
Gameop
Joined: Mon Dec 16, 2002 3:00 am Posts: 241 Location: USA
|
I've done a search on accessing the CIM from the command prompt and only found this one post. I know you hit ^ to get into the CIM but it just sits there doing nothing. Am I missing something here?
_________________ Ahab ahabstwgs.game-host.org msn orkidz4u@hotmail.com yahoo orkidz4u@yahoo.com msn IM Ahab yahoo IM Orkidz4u |
| Thu Jun 23, 2005 9:54 am |
|
|
|
|
Lieutenant J.G.
Joined: Fri Apr 05, 2002 3:00 am Posts: 332 Location: USA
|
I'm not sure why you got this thread as a hit from that search, but once you're at the : prompt (which means you're in the CIM), hit ? and you'll get a menu of options. It's not really meant for manual use, though I do find it faster than the computer method for clearing avoids.
_________________ Creator of the TWGS Data Access Library http://twgs.xiuhtec.com |
| Thu Jun 23, 2005 5:17 pm |
|
|
|
|
|
Page 2 of 3 |
[ 38 posts ] | Go to page Previous 1, 2, 3 Next |
|
All times are UTC - 5 hours |
Who is online |
Users browsing this forum: No registered users and 9 guests |
| You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot post attachments in this forum |