Thanks - and yes, please send chkwf8 when you are home (via direct attach). GoldED already has a validator that rejects overlong forms, surrogates and anything above U+10FFFF, and I would like to run it
against yours on the edge cases.
The pieces are all in place, so writing the detection is an afternoon.
What I do not want to get wrong is the policy, because once it ships people will have configurations leaning on it.
The pieces are all in place, so writing the detection is an afternoon. What I do not want to get wrong is the policy, because once it ships people will have configurations leaning on it.
It only becomes a question when the text has bytes above 0x7F - pure
ASCII is valid UTF-8 and valid in every eight-bit set alike. The rule
I would start from is: treat it as UTF-8 only if there is at least one well-formed multi-byte sequence and no ill-formed one. And judge the
whole message rather than line by line, both because FTS-5003 says the identifier covers all parts of a message, and because answering a man
in the charset he wrote in needs the message to have one.
Here is the case I keep coming back to. A man writes in CP866 and
quotes a paragraph of UTF-8 his editor did not recode. The check fails
on his own text, we fall back to the assumed charset, his words read
and the quote does not - which is what happens today, so nothing is
lost.
Now turn it around: he writes in UTF-8 and quotes CP866. The
check fails on the few bad bytes in the quote, we read the lot as the assumed eight-bit set, and now his own words are the rubbish while the quote is no better. That is where detection would earn its keep, and
the strict rule gives nothing. One bad line poisons the message.
One way out is to look only at the lines the author typed and skip the quoted ones - the charset of a message is the charset its author wrote
in, and a quote is already damaged by whoever sent it. That is not
much work,
and only mildly circular, in that we would be deciding what is a quote before deciding how to read the bytes.
So the question for you: have you run into that second case? It bites hardest where the traffic is not ASCII to begin with, so I can go
looking in the Russian echoes - what I do not know is how it looks
from your side.
On the future you are aiming at, the precedents are on your side, with
one caveat about how such a change arrives. Mail has the same
arrangement we do: the charset rides as a parameter on Content-Type,
and when it is missing the reader does not guess - plain text is
US-ASCII, as it has been for thirty years. That is our rule almost
word for word. And it has held even where it is arguably redundant:
HTML mandates UTF-8, and a document is still required to say so, on
the Content-Type or in a meta tag. The one place the declaration was dropped is JSON, and for your exact reason: once UTF-8 was made the
only legal encoding there was nothing left to declare.
But each of those was settled on a date, by a body with the standing
to settle it. Fidonet will not do that.
It will drift, and there will be no day on which a reader may simply
start assuming - which is the situation a detector is for.
Which is also why I would rather put it in now than when it is wanted. This code can go years with nobody in it, and the day UTF-8 without
CHRS becomes ordinary is not necessarily a day when anyone is here to
add the handling. Better it sits ready and switched off, and the
default moves later, when the network has.
And this is about reading only. What GoldED writes in its own mail is
the sysop's business and already his to configure. A keyword, off by default, settable per group.
So go ahead and implement it.
UTFautodetect Yes|no|override (Default: no)
Settable globally or per group.
UTFautodetect Yes|no|override (Default: no)
UTFautodetect Yes|no|override (Default: no)
Or perhaps XKATUTFautodetect ...
| Sysop: | DaiTengu |
|---|---|
| Location: | Appleton, WI |
| Users: | 1,136 |
| Nodes: | 10 (0 / 10) |
| Uptime: | 00:25:03 |
| Calls: | 14,591 |
| Calls today: | 2 |
| Files: | 186,473 |
| D/L today: |
11,275 files (3,471M bytes) |
| Messages: | 2,577,528 |