[crossfire] Data File (Maps, Archetypes) Encodings

[Christian Hujer]

Hello dear co-devs.


We have a common problem with the text encodings of data files.

Examples:
* Daimonin used (until a few minutes ago) the ISO paragraph character 0xA7 for 
separating a map's sound spec from its name.
* Daimonin uses the ISO degree character 0xB0 for highlights in messages.
* Crossfire uses the a circumflex character 0xE2 for the name of a wine in 
map /maps/scorn/houses/house3.bas2.

This leads to some problems.
* Crossfire x11 client displays 0xE2 as a circumflex.
* Crossfire gtk client displays 0xE2 as ? (tested by Ragnor).

For both projects, it makes sense to rethink the file formats. I see three 
possible solutions:

1. Use US-ASCII text only.
That means, only data files with bytes 0x13, 0x20-0x7E are valid.
Pro: easy
Pro: stable
Pro: no changes required.
Con: very limited solution

2. Use ISO-8859-15 text.
That means, bytes 0x13, 0x20-0x7E, 0xA0-0xFF are valid.
Pro: easy
Con: clients need special handling for non-ascii chars if they are UTF-8 aware 
and run on UTF-8 systems (e.g. gtk client).
Con: limited solution

3. Use UTF-8 text.
That means, only valid UTF-8 streams with Unicodes u0013, u0020-u007E, 
u00A0-... are valid.
Pro: future-proof
Pro: Allows full unicode (e.g. Chinese chars if somebody likes, or even 
klingon if the underlying system supports it).
Con: clients need special handling.
Con: Windows users or users of other ancient OS editions with no good UTF-8 
support will have more problems than with ISO-8859-15.

I see two places, where the encoding needs to be specified:
* Data files
* Network protocol

My favorite solution would be 3. UTF-8, followed by 1. US-ASCII. I dislike 2. 
ISO-8859-15 very much.

What do you think?
-- 
Christian Hujer
Free software developer
mailto:cher at riedquat.de
http://www.riedquat.de/
PGP Fingerprint: 03DE 4B72 4E57 A98D C79B 24A5 212E 0217 0554 CCAB
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://mailman.metalforge.org/pipermail/crossfire/attachments/20070131/fc022c35/attachment.pgp