[crossfire] A funny thing happened on the way to the new maps

[Mark Wedel]

On 04/ 2/16 09:49 PM, Robert Brockway wrote:
> On Sat, 2 Apr 2016, Mark Wedel wrote:
>
>>
>> I presume the 1000x1000 maps are 50 (or some other size) spaces/side?  Or is
>> each map 1000x1000, but you have some smaller set of maps being tiled together?
>
> Yes I'm using 1000x1000 maps, with each map being 50x50.  While I'd been reading
> C and Python code I'd come to suspect there were implicit assumptions that the
> world tile maps would be 50x50.  Yes this could be fixed but I found it easier
> just to stick to the existing standard.

  I would hope that there are not any such assumptions (the relevant sections of 
code could just look at the maps and see how big they are), but that of course 
doesn't mean that such things don't exist.  Certainly the generic tiling logic 
doesn't have any size assumptions, because I believe there are some other maps 
(non world) which are tiled, and those probably are not 50x50.

  I suppose doing a 'grep 50 ...' would see if anything hard codes in the value 
of 50.

  As cpu power and memory increases, some assumptions made when things were done 
may no longer be true.  each map being a 100x100 tile (while having 4 times the 
number of objects) would probably be fine on current hardware, in both cpu and 
memory standpoint (going way back, I remember when 16 MB of memory was a good 
amount of memory0

>
> It's worth noting that the name of the new world is Quadra and I've layed the
> tiles out in a directory tree to avoid the issues of having too many files in a
> single directory.  So the map tile for maps 117,117 on my new world is:
>
> /worlds/quadra/117/117/tile
>
> I've written a script which take the tiles generated by 'land' or 'bigland' and
> sets the tile paths correctly.  I have special plans for the edge maps which are
> worth their own post some other time.

  Yes, 1,000,000 files in one directory (depending on filesystem) may not work well.


>> As far as compression goes, at one time, the server did support map (or
>> really, all file) compression.  However, the typical size of an entire
>
> Any chance it could be put it back in, even as an optional component enabled in
> the config?

  I don't remember the details (I'm not the one that removed it), but it was a 
config file option.

  One of the main issues is that to open such files, a popen (instead of fopen) 
was necessary, so this littered the code with checks based on if the file was 
compressed, it had to record if the file was compressed (so when it saved it, it 
saved it as compressed), and also dealt with the potential of of many different 
compression methods (compress, gzip, bzip, now xzip, etc).  It was removed to 
make the code cleaner, which is a good thing, and at the time, the given size of 
maps (and other data) wasn't large enough to be a concern.

  Also, at the time, it allowed all files to be compressed (archetypes, player 
files, etc).  Certainly allowing it only on map files would be limit the number 
of places that code would be needed.  Other assumptions could be made, like if 
compress is set in configure, then assume all map files to be opened will be 
compressed, and all map files saved will be compressed (thus, do not need to 
record at time of reading if the file was compressed and what it was compressed 
with).

>
>> installation was small enough on current hard drives, there really wasn't much
>> point to it (even 100 GB isn't that big for modern systems).  Also, some
>
> I am looking at hosting my server in Linode or a similar service but I find the
> cost of 100GB hosted is more than I want to pay on going to run a game server.
>
> I could cut the world down to 500x500 maps and add the rest back when the cost
> of storage drops but I'm hoping not to have to do that.

  Fair point, but you are sort of an edge case on this, which is to say, a 
feature really only one person would probably use.

>
>> newer filesystems (ZFS for example) support compression, eg:
>>
>> NAME                    USED  RATIO
>> export/home/crossfire  18.0G  2.09x
>
> FreeBSD?  Few will be using ZFS on Linux I think and I'm not trusting my data to
> Btrfs just yet :)  There seems to be a FUSE option for on-the-fly compression
> but there are concenrs about stability.  Other than that I think Linux users
> have few options for on-the-fly compression but happy to hear about options I
> may have missed.

  ZFS on Solaris.  I've not looked at the state of other filesystems on linux - 
I would have thought that there would be other transparent compression methods 
for linux, but could be wrong.

>
> If we applied the map compression in the application it would work on any OS
> that the game runs on.

  I think there was some added complication to that compression logic when the 
server was ported to windows, but may be misrembering.  But as noted above, this 
wasn't a widely used feature, which was why it was removed (I don't recall 
anyone at the time caring it was removed).

>
> I compressed a few random tiles from my new world and got compression rations
> ranging from 6:1 to 10:1.

  Yes, the map files would be highly compressible - being plain text, and even 
more so, generally using a few words very often.  That 2x compression was for 
all the crossfire data, some of which is not very compressible at all (png 
images) and some that compresses some but not as well as that (eg, binary files)

  Back to the mega map, one thing that was thought of back in the past was to 
unify the scale.

  That is, instead of having buildings/caves/whatever you enter, everything is 
just on the world map (there may be stairs to different levels).  Thus, you 
would just wander from one shop to another without ever transitioning maps, 
enter a cave, etc.

  On the existing bigworld map (30x30 maps, or 1500x1500 spaces), the scale 
isn't there to do it.  Scorn itself might be 500x500 spaces.  But on a megaworld 
map (50,000 x 50,000 spaces), that wouldn't be an issue.

  However, there probably isn't any easy way to make those changes without a 
fair amount of manual effort.  While tools could be written to put the buildings 
into appropriate mega-world maps, spaces and alignment probably don't work out, 
eg, if you had four buildings in scorn right now like:

12
34

Building 1 might be 20x20, building 2 is 40x40.  Building 3 is 30x30, and 
building 4 is 40x40.  So while those could all be put on the megaworld map (say 
use a spacing of 40 for each building), then build 1 may now have odd games with 
it, or alignment to the road, etc.

  Plus, it would make it take longer to move between buildings in the town if 
they are 200 spaces apart (probably want to reduce tick time a bit)