[crossfire] Protocol & compression.

[tchize]

Mark Wedel a écrit :

>tchize wrote:
>  
>
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>Just a note about the suggested
>>s->c compressed <data>
>>c->s compressed <data> (yeah imho shoud go in both directions)
>>    
>>
>
>  I'm not sure if there is anything to really gain by the client compressing the 
>data.  I can't really think of many times the client is actually sending enough 
>data that compression will do any good what so ever.  The one exception might be 
>very long chat/shout/say messages, but even then, that seems fairly unlikely.
>
>  Granted, probably wouldn't be that hard to add, but does add some at least 
>minimal complication, and if it doesn't gain us anything, I'd rather avoid that.
>
>  
>
>>if we assume data contains a compressed list of 1 or more commands, i
>>think the question on what need and what need not to be compressed os
>>not immediate and we can have various attempt with various server /
>>client version. The fact of assuming any command can be compressed by
>>algorithm X but that not all commands will be compressed, is enough to
>>write this protocol add-on. Then it's a matter of tuning the
>>compression triggers, but this can be done without breaking
>>interaction with previous versions. So imho, the 'when' to compress
>>should not be fixed in protocol, only the how is to be fixed.
>>    
>>
>
>  correct.  That was my assumption - the server would figure out what it thinks 
>is worthy of compression.  A server with gobs of bandwidth but not a lot of cpu 
>time could decide that nothing is worth compressing.
>
>  
>
>>One of the most important question is which algorithm do we use? You
>>said you gave a try with zlib, but which algorithm does it uses? Does
>>it involves a dictionnary? If yes, do we plan to reset the dictionnary
>>content between each compress command or do we plan to keep it from
>>begin to end?
>>    
>>
>
>  I used the 'compress' function of zlib:
>http://www.zlib.net/manual.html#compress
>
>  Since each call to compress is self contained, the compressed data then 
>includes all the dictionary or other info necessary.  Note that given crossfire 
>will be working on multiple sockets, we can't use a library that holds any 
>state.  And if there are new structures needed to hold state, this starts to 
>increase the complication level some.
>  
>
Not really, it's just a matter of storing a pointer to the library
handle. I think if we should use a library that as some sort of
permanent state. This will allows compression of similar messages far
better (i think about all those you hit xxxx very hard' )


>  
>
>>My opinion currently is
>> - assume client can receive any data in compressed mode, but that not
>>all datas are compressed (the sender has choice for every command)
>> - assume the compression stream as something interleaved with not
>>compressed stream, identified by a specific marker (compressed)
>> - assume the compressed stream continuous (same 'zlib/any chosen
>>algorithm' session, usefull considering number of repeated text messages)
>>    
>>
>
>  This point is the real gotcha however - since the server can choose what data 
>to compress, the question then becomes what portion of the commands are being 
>compressed?
><sniped most>
>
I was unclear. I just meaned, the server would do something like
<command x> and his datas
<compressed> <datas>  (--> decompress to <command y> and it's data)
<command y> and his datas

I never suggested to compress partially a command :) It's complicate and
the job of compression libarry to do such decision :)
I just said we should have both compressed and uncompressed command on
the wire,  sender choosing if a specific command is worth compressing.

>
>  As I said before, at current time, I'm much more interested in code 
>cleanliness and simple code than getting things too complicated.  Especially 
>because complicated code tends to take longer to write, and if it never gets 
>done, there isn't much point.
>  
>
I never suggested to complicate the job

>  If we are going to do stream compression, I'd say we just compress everything 
>we send to the client, and don't care about the cpu time and/or data that 
>doesn't compress well.  That is the simpler approach, and could get done in 
>relatively easily.
>  
>
I don't agree, we can just have a flag telling 'can compress' when we
send a command, and the socket writer will decide if it encapsulate it
in a compression. However, once again if client assume data can be
compressed or uncompressed, we can implement selective compression later
and for now compress everything.

>  
>
>> - assume setup negociate the algorithm (client say i support x,y,z
>>then server send ok, let's go for y algorithm, this is how http does it)
>>    
>>
>
>  Yes, but IMO, I don't see the need for more than one algorithm.  I'd say we 
>just standardize on zlib.
>  
>
Don't agree, we might now be ok with zlib, but people might give a try
to a few other algorithms and in 6 month someone comes with an algorithm
that get 20% better compression with 5% less cpu overhead. That day we
don't want to have an awfull hack in client/server protocol to handle
this new algorithm. As you said, it's better to have a clean code, that
also mean a clean protocol :)

>  Sure, there may be other libraries which do marginally better.  But once 
>again, is it worth while to have another compression method that might be 
>marginally better (in terms of code complication, library support, etc).  If 
>there is some library that is clearly better, or one is written, we could easily 
>enough add support for it at that point.  One could very well see things like:
>
>gzstart
>....
>gzend
>
>and
>
>bz2start
>....
>bz2end
>
>  And so on.   In many cases, the support for different methods is more for 
>backwards compatiblity (at the time, A was best, but now B is better).  We're 
>not at that point, since we don't even have A.
>  
>
This is not clean imho. I suggest using what is the current scheme of
every commands sending, that is

S-C -> <headers defining length of command+data><command><space><datas>
which applied to compression gives
S-C -> <headers defining lenght of command+date><compression
command><space><compressed datas>

As currently all commands are words and data are either binary either
text, i would suggest to use a very small command for compress header
(so we don't lose all gain of compression :/) a simple character like #
or @ or & should be enough

The would end, in
S -> C # <compressed datas>

>  I'd also think that we could use the method that is done right now for the map 
>commands.  Basically, it goes like:
>
>C->S: setup map2cmd 1
>S->C: setup map2cmd false
>C->S: setup map1acmd
>S->C: setup map1acmd false
>C->S: setup map1cmd
>S->C: setup map1cmd true
>  
>

I think it's good to have this
C -> S: setup compress zlib,bzip2,rle
S -> C: setup compress zlib

In one exchange, both sides agreed to use zlib.
If server doesn't support compression (like current server) or doesn't
support any requested algorithm, you would obviously have

C -> S: setup compress zlib,bzip2,rle
S -> C: setup compress false

please note there are already setup commands that send various parameter
and let server confirm a part of them

>  (if the client sends true back for an earlier revision, the client stops its 
>fallback method).  That logic would find for compress methods down the road 
>(setup compression gz, if that doesn't work, setup compression bz2, setup 
>compression whatever).
>
>
>
>_______________________________________________
>crossfire mailing list
>crossfire at metalforge.org
>http://mailman.metalforge.org/mailman/listinfo/crossfire
>
>
>  
>


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: OpenPGP digital signature
Url : http://mailman.metalforge.org/pipermail/crossfire/attachments/20060328/a129b71d/attachment.pgp