#foswiki 2015-05-15,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***ChanServ sets mode: +o gac410 [03:05]
.................... (idle for 1h36mn)
gac410 has left [04:41]
ChanServ sets mode: +o CDot [04:54]
................. (idle for 1h24mn)
andrei_t has quit IRC (Remote host closed the connection) [06:18]
...... (idle for 26mn)
MarcKloterGood Morning. Does anyone know if it's possible to define Macros in a ViewTemplate which is included via AutoViewTemplatePlugin? In my tests it doesn't seem to work. If I try to use the macro it isn't defined [06:44]
...... (idle for 28mn)
jastyou mean using ' * Set FOO ='?
IIRC that doesn't really work in skin templates in general
[07:12]
MarcKloter@jast: ok, so I can at least stop trying - thanks :) [07:20]
........... (idle for 53mn)
jomohiall
CDot: Hudston we have a problem ... see Item13378
[08:13]
GuilainChi jomo [08:15]
jomoMarcKloter: you can't use the ' *Set SOME=someval' in the SomeformViewTemplate but you CAN use the %SET{"SOME", value="SomeValue"}% for runtime pref setting... See System.VarSET [08:17]
CDotjomo: why should sites that do *not* use NFC/NFD pay the price of normalisation/ is there a benefit to them? [08:19]
jast%SET is a trunk feature, not available in 1.1.x [08:19]
CDot%SET comes from a plugin, doesn't it? GetSetPlugin, or something like that?
SetGetPlugin
SadGetPlugin
[08:20]
jomoCDot: the uncode is ALWAYS somewhat normalized. usually it is NFC e.g. the Encode::decode_utf8 will return NFC string... so they havent any problem... [08:20]
CDotSadGitPlugin [08:20]
jastno, it's in core/
on master, anyway
[08:20]
CDotalways? So if I cite a codepoint in utf-8, decode_utf8 will *change* the codepoint when it decodes the utf8?
I doubt that
I can accept there may be normalisation during an *encode*
but even then.....?
[08:21]
jomoplease check the attached script ... the $str == $nfc ...
It is unicode feature - -the Á (Aacute) can be in TWO forms NFC or NFD.. default is NFC (usually) :) so sites default uses NFC (for filenames too). E.g. the normalization problem touches them ONLY when getting files uploaded from NFD enforced OSes, like OSX...
[08:22]
CDoti really don't understand why this matters. So long as the encode/decode is symmetric, where's the problem? [08:24]
jomobecause you will get in the NFC topic content NFD filenames... please TRY attach the files, and you will see yourself. You will get TWO attached file with the SAME name... [08:25]
CDotCDot is up to his ears in unit tests at the moment, will try it later [08:25]
jomoone will be attached as NFC name second as NFD name... how you want use the ATTACHURLPATH/filename for the NFD file?
ok :)
[08:26]
CDotis this problem limited to OSX, or a general problem?
CDot isn't aware of any default normalisation performed on any linux filesystems, but has never researched it,s so....
[08:26]
jomolimited to any case when Foswiki installed on LINUX should ACCEPT uploads from OS X... [08:27]
CDotok, that narrows it down a bit [08:28]
jomono, linux nor freebsd havent any enforced normalisation.. [08:28]
CDotwindows? [08:28]
jomo(and maybe from windows too - not tested)... [08:28]
CDotok, that quantifies the problem, thanks [08:29]
jomosimply - the linux clients and the linux servers would NOT see any change - but it solves the problem for the NDF clients... [08:29]
CDotCDot no longer has any OSX machines, so can't test it [08:29]
jomoNFD [08:29]
jastwell at least nobody claimed unicode support would be simple :) [08:29]
jomoyou dont need OS X machine - simply upload NFD encoded file
the attached script will CREATE one for you :)
[08:29]
CDotyes, I understand how forced normalisation on the server will solve this for an OSX client. However I want to know if there are any alternative solutions
that do *not* require encofrced normalistion on the server.
[08:30]
jastwell, *technically* the NFD form will not cause any problems [08:31]
jomoit IS simple... only need understand it a bit - after this, the plugin developers nor the users WILL NOT cope with this problem... only the core should be done right... [08:31]
CDotyes, I know. But I want to know what the alternatives are. [08:31]
jastit's just that users will not expect they have to use NFD in links to files uploaded using OSX [08:31]
jomoomg... will cause, everytime when someone from OS X will upload to Linux Foswiki installatoni.. [08:31]
jastnor would they be aware that this is even a thing, or know how to do it
which is why I personally would never use full unicode in IDs of any kind
[08:32]
jomohm..
in the 21th century?
[08:34]
CDotso, a use case might be: User A uploads a file called 'FILE' (but with uc chars in it) from OSX. The byte string identifying the file is NFD, so the file appears as 'FILE'. A second user on windows sees this file, and types in 'FILE', but the uc codepoint in their typing is *not* the NFD codepoint, and therefore they have typed a *different* filename. [08:34]
jomo:) [08:34]
jastyes, in the 21st century
CDot: precisely
also this makes it possible for users to upload two files with a canonically equivalent name
i.e. a filename that looks like "ä.txt" and a filename that *is* "ä.txt"
[08:34]
jomoalso, when you editing the topic, and want use ATTACHURLPATH/SomeFileNameFromOsX.jpg - will not find the file, because the EDITOR is NFC and the file is NFD... [08:36]
jasteven if you do the normalization there still are graphemes so similar that you can create two names that look the same but do not normalize to the same name
(which is a distinct problem, but that's the reason I oppose unicode in IDs)
[08:36]
CDotindeed. I often get O and 0 confused in some typefaces. [08:37]
jastyeah, but with unicode the problem is much worse [08:37]
CDotand 1 and l [08:37]
jastΡ vs P
Α vs A
[08:37]
jomoyeah - thats true :) [08:37]
CDotnever mind trad chinese versus simplified [08:38]
jastο vs o [08:38]
CDotok, so I can see an argument for normalistion, but my gut tells me it should be the *client's* problem, not the servers
as in, server should be able to say "I speak NFD" and the client should do the rest
[08:38]
jastit becomes the admin's or developer's problem when the bug reports come in... "I can't link to føø file" [08:39]
jomoahh... see the client from OS X uploads a file to Linux Foswiki installation. How the client should add NFC file into the upload form, when the OSX enforces NFD... [08:40]
jast"well obviously you have to know the right composition of the characters... how about you look at a hex dump of the attachment list" [08:40]
CDotjast: excellent idea. I will add a hex dump feature.... hang on..... ;-) [08:40]
jomolol [08:40]
jastwhile you're at it, add a virtual keyboard to use arbitrary unicode characters in input fields
and an UTF-8-to-unicode calculator
[08:41]
CDotok, so I understand the arguments for normalisation. What's the cost? [08:43]
jomoguys - what is wrong on the proposed convert everything to NFC? Simply Foswiki should enforce NFC (inside foswiki). On the OS X installations that would NOT cause any problem (because the NFD is enforced on the system level) in the Linux installations are NFC everything anyway... So, Foswiki should convert every received filename to NFC and problem is solved... [08:43]
jasta call to a normalize function when receiving an upload [08:43]
CDotwhich costs how much.....? [08:44]
jastthe sanest way to normalize would be NFC or NFKC, I guess [08:44]
CDotwhat do web servers (e.g. apache) do? Do they rely on the underlying filesystem? [08:45]
jastdoes the cost matter? it happens once for each file upload [08:45]
jomojast: exactly [08:45]
CDotno, once for each incoming string [08:45]
jastwell, that would be the "optimal" treatment [08:45]
CDotbecause you never know what params refer to filenames [08:45]
jastbut I think treating each attachment once is "sufficient"
going from the assumption that it's sufficient to normalize the name in the one place where the client is at risk of NFD-ing it
[08:46]
CDotand some poor bugger uploads "ä.txt" and then tries to link to "ä.txt" but it doesn't work, because their NFD filename has been magically converted by the server to NFC....
CDot intensely dislikes systems that rename uploads
[08:47]
jomothe link should works - if the user want link with NFD (%encoded) string the server convert the incoming request to NFC string, so the link WILL works... [08:48]
CDotthe only way normalisation can work is if *everything* nromalises *everything* consistently, otherwise someone, somewhere is going to get caught out. [08:48]
jastthat really only leaves the more involved solution [08:48]
CDotjomo: that's why I asked about Apache. [08:49]
jastand in that case, directly passing /pub/Foo/Bar/Baz.png to the web server is no longer an option [08:49]
CDotbecause Foswiki might normalise ä.txt, but if Apache doesn't *also* normalise the same way, a link to the pubdir will be broken
and then there's webdav, and nfs, and......
[08:49]
jomodamm... true... and in the apache config isn't possible enforce the NFC? [08:50]
jastjust tested, apache does not normalize [08:50]
CDotno idea, as i said, I have never researched it
http://wiki.apache.org/subversion/NonNormalizingUnicodeCompositionAwareness has some interesting discussion regarding the pros and cons of nromalisation
[08:50]
jomojomo just reading it ;) [08:54]
CDotFoswiki is *also* a "sort of file system" so it's quite relevant. not quite clear what their conclusion is, though. [08:55]
jomojomo really wondering what filename is attached to the utf8branch wiki when uploading from Windown 8 or such.. ;( [08:57]
jastdoes anyone have an instance of that branch running on a public server? I happen to have a win8 client here
but AFAIK it just passes through whatever it gets
[09:00]
GuilainCGuilainC is still unable to have a master branch dev env... so for a public utf8branch... in some years, perhaps ! [09:04]
jomookay, kicked daughter from gaming - the result trying upload the ČáŘý (Ccaron aacute Rcaron yacute) file from Windows to current utf8 branch FW... result: "Either you did not specify a file name, or the file you are trying to upload ČáŘý.txt has no content. You may not upload an empty file."... [09:10]
CDotright, you can't upload an empty file
try putting a byte in it :-)
GuilainC: why?
[09:15]
jomoyeah - just discovered this :) [09:15]
CDotI have no idea why that limit exists. There's probably some logic behind it, somewhere. [09:16]
GuilainCCDot, lot of dependency trouble, unable (by my skills to solve) and no enough time for learning all this stuff... at this point [09:17]
.... (idle for 17mn)
jomoSo, (if i checked right) the file from windows comes as NFC - so, no problems - seems, the only problem is when the client is on mac AND trying to upload uni-filenames. - Maybe we could ignore this for now... :) [09:34]
CDotjomo: I think we need to raise a Task to say it needs to be addressed, but it's not a release blocker (unless you decide otherwise after more testing) [09:46]
jomookay... going to write a Task for this... [09:48]
GithubBot[distro] cdot pushed 4 new commits to utf8: http://git.io/vUFVM
distro/utf8 d29dd09 Comment: Item13378: converted tests to PlainFile by default, extended URL macros to encapsulate parameters, allowing correct encoding to be applied.
distro/utf8 169fafe Comment: Item13378: languages need utf8
distro/utf8 8a1ba80 Comment: Item13378: discovered that the StoreTests were not being run on all implementations
[09:50]
***GithubBot has left [09:50]
........... (idle for 52mn)
GithubBot[distro] cdot pushed 1 new commit to utf8: http://git.io/vUFbn
distro/utf8 0daec05 Comment: Item13378: fixes for UseLocale being activated, and some missed calls
[10:42]
***GithubBot has left [10:42]
.......... (idle for 47mn)
CDotjomo: ok, all the unit tests are passing now. [11:29]
jomoGREAT! going a pull and check - (but just now - im under wife management) :) [11:30]
CDot:-) [11:31]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)