Ok, so the easiest way is to just for me to record the bug in
action and show you.

This is the beta on twgs.classictw.com, it reports as v2.00.39.

This is a script that builds a large burst macro when I input a
list of sectors. It then mows them out as quickly as it can.

The code in question is:



Now this is not a fancy block of code. It just loops thru all of the
sectors I've given it and sends them out. I don't even do them in a
macro here, altho I could (and do in other scripts), it just sends
directly to the command queue.

Lines with # before them are commented out and not executed.

When it gets to 50, it sends a @/, then waits for that. This clears
the command queue, since it basically waits for all of the queue to
finish before sending the next block. With this in, the queue never
gets too large. Without it, it stops very earily in the burst.

The video shows this in action. There are a few pauses in the video,
those are just artifacts of the screen recorder I used.



I recommend using the 480p mode and letting it download before
watching it. Otherwise it can be a bit hard to watch.

Direct link:
http://www.youtube.com/watch?v=sJvcsSVu9j0

http://www.youtube.com/watch?v=2KWErz2Wg4s

somethin to listen to while you watch.

Since a FedCom test is easier to implement in general, can we use that as a test case? I tested sector move lists before because I thought the bug might be specific to that, but you said in your post that a FedCom burst also failed if it was too large.

I ran a test at the twgs.21-6.com server using three different clients, one the Vandyke CRT app that I typically use for connections, one the standard Windows telnet, and the third a tcp/ip test app that I wrote (not standard telnet). In each case, I pasted in a line of 1000 characters with an obvious "end of line" marker to verify that the entire block of data went. In each case, the entire block went.

If you recreate this test in your setup, does the block of data go through, or is it capped? Test it on twgs.classictw.com as well as twgs.21-6.com if you don't mind.

If this works for you, then we need to find another test case that's easy and quick to set up. If it doesn't work for you, then it suggests that the issue is with your client communicating with TWGS, which will narrow it down considerably.

Have not tested it on fedcom. Have tested it via private hail.

Attached is the video.

I'm using TWXproxy. I've tried this with the old 2.04, and it
happens there so it's not a version change. I've used that for
years, and this problem is new. I've had to modify scripts since
the beta, if I run them on navhaz (non-beta) then the problem
doesn't occur. It also didn't occur on twgs.classictw.com during
a few other games, but it wasn't running beta then either.

I'm also using 2.05, and it's the same way. If I run it on a
non-beta, the problem isn't there.

For the term client, I'm using putty here because it's the
fastest. I could use zoc or swath, or the windows telnet client
or the linux telnet client. I've tried it in the windows telnet
client and the result is the same.

I don't have a way to test it on 21-6 as I don't have a corp
setup there.



Now there has been a slight change between this version of the
beta and the last. The last beta would hang around 300 and not
continue. Now it continues, but as you can see it still drops
quite a bit.

I believe that I am replicating this issue now, but only with the standard Windows telnet client. Using CRT, Putty, and my own raw TCP/IP test app, I have been unable to reproduce it. I'm going to study what's going on with the Windows telnet client and see if I can track down the cause of this. I'm sure it has something to do with handling of buffer overrun within TW itself. If I increase the buffer size, I can effect at what point the problem appears. As soon as I can spend a bit more time on this, I'm sure I'll get it tracked down. It's in my sights.

Ok, I've studied this problem quite a bit and here's what I've found out.

a) The jump in data occurs at the point where TWGS reads data off of the socket into a buffer. This means that the error occurs prior to TWGS even getting the data. The sending application is not pacing its sends to avoid overrunning the buffer. Some applications do this just fine, and I don't see this problem with those applications.

b) I verified that the same problem appears in TWGS v1.03, the last public release from 2004. I can't speak to the particular circumstances of your test, but with my test, pasting 20K warp data lines to FedCom, I see jumps in the data in the older and current version, but only for Windows Telnet and not with CRT or Putty with default settings.

I think this problem is more obvious in the current version because of some changes in load balancing on the server. But the actual problem has not changed. If a sending application is sending data too fast for TWGS to keep up, that application will run ahead and lose chunks of data. I can optionally increase the size of buffers so TWGS/TW is able to clear data from the socket at a faster rate, but the problem will remain even if less likely to occur. At some point, large bursts of data like this need to be paced by the sender.

In the older version, both scripts I've shown here work w/o the modifications
I've demonstrated. The "sector blaster" script did no pacing at all, and I've ran
lists as large as 7000 sectors before in a single setting. Maybe TWXproxy does
some pacing behind the scenes, just only on larger blocks, I don't know.

The warpsend script has always used some form of pacing, simply so that if
the receiver dropped I could stop the script by hand if I had to. Prior to the
beta, it was every 1000 sectors. I adjusted it down to 200, and then to 100.



Have you tried TWXproxy? The actual end client is not the one sending the
burst here. Not sure how TWXproxy handles this, will look at it and see. EP
briefly mentioned working on the socket code, so maybe this presents a good
opportunity there.

In reality, this is a small issue, but it does cause some scripts to have problems.
Scripts like the two I've shown here were easily changed, and it's not a bad idea
to have pacing in them for other reasons. But some scripts, like buydown,
product movers, and avoid managers, will be trickier to fix. If you can increase
the buffer size and reduce this to a problem of extremes, that would probably
be enough.

How big is the buffer? How big can you safely make it? How long does it take to
clear the buffer? If I were to just send a burst, then wait a half-second before
sending the next burst, would that be enough to clear without having to sync
up to the server if the burst wasn't too large?

It is my understanding that a client is responsible for avoiding buffer overruns. A client can easily avoid sending too much data by verifying that the data that was being sent was actually sent. The easiest approach is a blocking socket, and the standard winsock send function won't return until all data has been sent. If a buffer is full, it will wait until space is available. Non-blocking sockets are more tricky, because you actually have to check to see what was actually sent, and handle cases where fewer bytes were sent than intended. My guess is that one of the sending applications here is not testing for actual bytes sent and whenever the buffer is full, chunks of data are going into the bit bucket.

Having a larger buffer will mask the problem, but there will always be cases where the problem shows up. When many players are online at once, especially several running these data bursts, the speed that the buffer is cleared will be slowed and eventually you'll have an overrun. Also, slower systems are more likely to experience this problem, especially when the client machine is significantly faster. Connection speeds are also a factor. And finally, not every sockets implementation even supports changes to the default buffer size, so that increasing the buffer is an unreliable "solution". The point is, this problem is there, and it's better for clients to handle a full buffer condition gracefully than for me to mask it.

If there's something I'm doing wrong that is causing a client to believe that the server is ready for more data, I definitely want to fix that. But I'm not aware of anything I'm supposed to be doing explicitly to manage data flow. It should be managed by the socket layer through the send and recv functions.

With TWXproxy, TWXproxy is the client. You can run scripts completely headless
if you want, no terminal at all. I'm not sure what code it uses, or whether it even
uses winsock functions. For example, in your tests you said you didn't see the
problem with putty. But in the videos above I was using putty as my end client.
If that was the only element, then it wouldn't be an issue... yet it is.



Masking the problem is good tho. There's several times where small bursts are
used, and trying to pace each of them will be a problem. As long as the issue
doesn't show up on regular buydowns and similar bursts, there isn't a problem.
I've never had the problem at all pre-beta, so something between now and
then has made it more pronounced. It's not entirely the client's doing.

Many of the end clients aren't fixable anyway, a lot of telnet projects are no
longer in development. We're an old crowd with old tools, lol. While TWXproxy
can probably be tweaked a bit in 2.05 for this, non-scripters will find it's hit
or miss trying to keep up.

May I ask, what is this script doing?

As I've said, my specific scripts can be fixed (see above if you really want
to know).

My concern is smaller scripts, like buydown and product mover. I've already
said that several times, too.

Please, do us both a favor and read the backthread before asking questions
that have already been answered.

Thanks

TWX uses Delphi's TClientSocket.Socket.Send(), which ultimately calls WinSock.send. TWX does not currently verify that the results from that call is 0 though. I'll have to look at more examples, as I'm not sure it's necessary.

Update: Okay, I've rewritten the send procedure in what I hope will catch an incomplete send. Testing now.

Ok, as long as you know that increasing the buffer isn't a fix, just a patch.

In my tests, I increased the buffer size and was able to paste the entire 20K warp data to FedCom without any dropped data. We'll have to see if this improves the situation in actual use. I'll send an update to SG so he can let you test it.

Elder, TClientSocket has a property called ClientType that can be ctNonBlocking or ctBlocking. On a client, you're usually ok to use ctBlocking and allowing the socket send to wait until all data is sent. ctNonBlocking would cause the code to return from a send call immediately, whether or not the data was successfully sent, and that's where you'd need to check for any partial sends.

i really like what i see in this thread - some real intelligence in the discussion of sockets. it's always a good thing to be reminded to check return values; it's easy to get comfortable with certain functions always succeeding - but we forget that they return information for a reason. we can't rely on the happy path all the time. i'm working on my helper at the moment and i'm taking this as a reminder to make sure i'm checking my socket.send calls.