#foswiki 2015-06-02,Tue

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
gac410hi all, I'm going to stop apache for a bit. fixing up some logfiles [00:34]
............. (idle for 1h1mn)
***SvenDowideit has quit IRC (Read error: Connection reset by peer) [01:35]
................................................. (idle for 4h4mn)
ChanServ sets mode: +o CDot [05:39]
........ (idle for 35mn)
ChanServ sets mode: +o MichaelDaum [06:14]
........................................................ (idle for 4h35mn)
ChanServ sets mode: +o SvenDowideit [10:49]
.................... (idle for 1h39mn)
ChanServ sets mode: +o gac410 [12:28]
gac410He everyone. Don't forget the releases meeting today: 1300Z #foswiki-release
CDot ... I did a bunch of cleanup work on f.o but held off on the utf8 conversion. We have stuff in t.f.o (1.2) that needs conversion but CharsetConverter won't run on 12
[12:35]
CDotno, it won't
might be best to move those files to f.o. run the converter, then move them back
[12:36]
gac410I think it would mainly be the 1.2. Sandbox. yeah. [12:37]
CDotall depends how much "stuff" there is
don't forget to change the {Store}{Encoding} in t.f.o as well
[12:37]
gac410right. i did do a complete backup into my user dir ... That took a long time, bigger than i thought it would be.
I cleaned Trash web, but held off on Sandbox. Hm. WikiAppInstallerPlugin lists you and padraig as the authors
[12:38]
***ChanServ sets mode: +o Lynnwood [12:54]
CDotme? really? I don't remember that [12:58]
gac410I'm working on the agenda for our meeting. Everyone ... RELEASE MEETING NOW in #foswiki-release [12:59]
................. (idle for 1h23mn)
GithubBot[distro] FoswikiBot pushed 3 new commits to master: http://git.io/vkH39
distro/master 6e01c16 Lieven L. Litaer: Item13252: Translations updated using Weblate (Klingon)...
distro/master c6e1179 Lieven L. Litaer: Item13252: Translations updated using Weblate (German)...
distro/master e1534fd Lieven L. Litaer: Item13252: Translations updated using Weblate (Italian)...
[14:22]
***GithubBot has left [14:22]
...... (idle for 29mn)
GithubBot[ModPerlEngineContrib] cdot pushed 2 new commits to utf8: http://git.io/vkHEs
ModPerlEngineContrib/utf8 6f0fe8a Comment: Item13405: prepare for NFC/NFD normalisation
ModPerlEngineContrib/utf8 1318c87 Comment: Merge branch 'master' of github.com:foswiki/ModPerlEngineContrib into utf8
[14:51]
***GithubBot has left [14:51]
GithubBot[ModPerlEngineContrib] cdot pushed 2 new commits to master: http://git.io/vkHuv
ModPerlEngineContrib/master e1a2d6a Comment: Merge branch 'master' of github.com:foswiki/ModPerlEngineContrib
ModPerlEngineContrib/master 93a5542 Comment: Merge branch 'utf8'
[14:53]
***GithubBot has left [14:53]
GithubBot[FastCGIEngineContrib] cdot pushed 1 new commit to utf8: http://git.io/vkHuu
FastCGIEngineContrib/utf8 4879ca5 Comment: Item13405: prepare for NFC/NFD normalisation
[14:54]
***GithubBot has left [14:54]
gac410CDot Will Fowiki::encode_utf8 be backwards compat to 1.1.x [14:54]
CDoterm, no, but you are pulling them into trunk, are you not? [14:55]
gac410Right, but the extensions still need to be releasable on 1.1.x [14:55]
CDotwell, pull them into trunk and make a task to revert later
no way am I wrapping that code in an IF statement, it's too performance sensitive
[14:56]
gac410well we sort of have a task already http://foswiki.org/Tasks/Item13139
Basically *any* default extensions need to be 1.1.x compatible. we can't release them for 1.2 if they won't work on 1.1
[14:57]
jmk0McDonald and McIntosh are in the default "don't space" list for the spreadsheet plugin? Must be some site-specific stuff from the dev :) [14:58]
CDotjmk0: why? They are common surnames [14:58]
gac410Proper names would never be spaced would they? [14:59]
jmk0just seems fairly random
two names in particular
[14:59]
CDotalong with McKay, McWhirter, McIntosh, MacDonald.. in fact Mc* and Mac* [14:59]
jmk0right
why those two?
rhetorical question
[14:59]
CDotno idea; like you said, site-specific stuff. [15:00]
gac410But not macadam [15:00]
CDotwhy not? [15:00]
gac410Because it's asphault it's a word not a name [15:00]
CDotMacAdam is another common name [15:00]
jmk0haha [15:00]
CDotso is McAdam
it is a proper name, that is now an (archaic) noun
[15:00]
gac410Right. But you are refering to the product that is used in roads, not the inventor. [15:01]
CDotlike "Hoover" [15:01]
gac410Right. But according to wikipedia, which is always right, right?
Macadam is a type of road construction pioneered by Scottish engineer John Loudon McAdam around 1820.
[15:01]
CDotah, most of Wikipaedia is written by Americans, who don't have the advantage of a proper education. [15:02]
jomolol [15:02]
CDotfor one this, they have ab-so-lute-ly no idea how to make tea ;-)
^this^thing
[15:02]
jmk0s'alright [15:03]
jomoCDot: and do you know whats mean "one moment"? [15:03]
CDotyes; why? [15:03]
jomook, describe
;)
what it means - by you?
[15:03]
CDotit means "Please wait a short space of time before I address your request" [15:04]
gac410Same here. [15:04]
CDotalso truncated to "mo" [15:04]
jmk0yep [15:05]
jomoi was saw in some old book its origin. It is a time what is needed for extraction of the Early Grey tea... [15:05]
CDotheh [15:05]
jmk0never heard that before
early grey? :D
[15:05]
jomome neither...
or something such - don't rememer right
[15:05]
CDotone fortieth of a solar hour. [15:06]
jmk0smh [15:06]
CDotEarl Grey tea. It's tea flavoured with Oil of Bergamot, invented by the eponymous Earl. [15:06]
jomothen Earl Grey :) [15:07]
CDotso, a moment is roughly 90 seconds, which I would say is insufficient for a good cup of Earl Grey. I'd allow it to suffuse for at least 1.5 moments, myself [15:08]
gac410CDot back to fcgi/modperl .... Maybe monkey patch Foswiki::encode -> Encode::encode ? Or would that slow things down too. [15:08]
CDotno, that would work, and I'm just doing that in the ModPerlContrib. [15:09]
gac410okay great. thanks. [15:09]
CDotI mean the FastCGIContrib [15:09]
jmk0jmk0 turns on spreadsheetplugin_debug, spends the next 5 minutes regretting it [15:11]
gac410jomo, CDot The bug where wysiwyg -> natedit switch corrupts a topic name. Is that Item13338? And is this confirmed? Your comment from June 1st says it's working.
er attachment image name
[15:12]
CDotI tested what jomo described in #foswiki-release and it is failing. [15:13]
gac410gac410 is confused is it a nfc/nfd issue or double encoding ah... okay So ignore your comment that it's working. [15:13]
CDotwait, sorry, 13388? That is fixed, AFAIK [15:15]
gac410Ah. That's why I was confused. I think that there are two issues on the same task ... maybe? [15:16]
CDot13338 is a different issue [15:16]
jmk0one would hope so :) [15:17]
jomo13338 is fixed.. [15:17]
gac410The 'How to reproduce' describes the switch from wysiwyg to nat messing up the attachment name. But then it goes on to other issues. [15:17]
CDotjomo: do you have a task number for the wysiwyg->natedit switch? [15:17]
jomodamm sry
the 13388 - is the natedit bug - and it is confirmed by CDot
[15:17]
CDot13388 says %QUERY [15:18]
jomoso the 13388 isn't fixed [15:18]
gac410Item13338
338, not 388
[15:18]
jomohttp://foswiki.org/Tasks/Item13338#r7
is about the natedit
http://foswiki.org/Tasks/Item13388 - is ok
dammed numbers
[15:18]
CDotok, once again I didn't read your report carefully enough. Yes, it is still open. [15:19]
gac410And just to be annoying. Should http://foswiki.org/Tasks/Item13441 be a blocker ( The corrupted table on TestCasesUtf8Errors ) [15:21]
CDotyes :-(
gac410: ModPerlEngine is fine, all the Foswiki::decode_utf8 calls are protected by if ($Foswiki::UNICODE)
I did that reluctantly, but I did it
hmm, perhaps that should be if (defined &Foswiki::UNICODE) - otherwise the optimiser might not do a very good job
[15:23]
gac410Okay Item13441 bumped to urgent. [15:25]
CDotany opinions from the Perl acolytes on $Foswiki::UNICODE versus &Foswiki::UNICODE? [15:26]
jmk0apparently i'm not a perl acolyte, I thought & was a function reference
not something I've really used though, so...
[15:27]
jomohttp://search.cpan.org/dist/Perl-Critic/lib/Perl/Critic/Policy/Subroutines/ProhibitAmpersandSigils.pm
:)
[15:29]
CDotok, FastCGI and ModPerl engines updated [15:30]
GithubBot[FastCGIEngineContrib] cdot pushed 3 new commits to master: http://git.io/vkHyU
FastCGIEngineContrib/master 28dc463 Comment: Merge from utf8
FastCGIEngineContrib/master 90d0c2f Comment: Merge branch 'utf8'
FastCGIEngineContrib/master ef6f741 Crawford Currie: Item13405: Merged
[15:30]
***GithubBot has left
JulianLevens1 has left
[15:30]
CDotjomo: that's right, you should *call* functions that way. But I'm not - if(defined &Function) is testing whether the function is defined, without calling it.
^should&shouldn't
^&^^
[15:31]
jmk0pretty sure there's a recommended practice for that, but it's another something I've never done :-/ [15:32]
CDotthe point is that if ($Foswiki::UNICODE) can't be statically optimised out, because $Foswiki::UNICODE might be defined at some point.
actually, neither can if (defined &Foswiki::UNICODE), because sub Foswiki::UNICODE might be dynamically declared at any time
[15:32]
jmk0if it's a function, using $ seems wrong [15:33]
jomoperldoc - defined:
You may also use defined(&func) to check whether subroutine &func has ever been defined. The return value is unaffected by any forward declarations of &func . A subroutine that is not defined may still be callable: its package may have an AUTOLOAD method that makes it spring into existence the first time that it is called; see perlsub.
[15:33]
CDotthat's what if (defined &func) is doing :-) [15:34]
jmk0sounds like CDot's right
at least about the "how", not sure about any optimization details :)
[15:34]
CDotthe optimisation is what I'm concerned about
given that Foswiki < 1.2.0 doesn't define any symbols I can use to determine that it doesn't support unicode (at least, not efficiently) it looks like I'm stuck with $Foswiki::UNICODE
[15:34]
gac410what about the API version [15:36]
CDottoo slow
I need something that can be optimised out
e.g. a use constant
[15:37]
gac410Add a constant to Foswiki.pm ? [15:38]
jomojomo once ago saw something about the premature optimisation ;) [15:38]
CDoty, but that's a constant that doesn't exist in 1.1.9
if (Foswiki::UNICODE) will error out
jomo: this isn't premature, this is just-in-time, and all I'm trying to do is to help the perl interpreter a bit.
[15:38]
gac410gac410 enjoys demonstrating his lack of Perl knowledge. Clearly not a candidate for acolyte [15:40]
jomojomo doesn't even know who is an acolyte - but thinks than in the FW are much more places what really need to optimise, and this sure isn't 1000 IF-s what in the worst scenario adds 3 microsends to response... ;) :)
lol - acolyte it is something with church... so, i'm definitely not an acolyte.... :)
[15:41]
GithubBot[distro] gac410 pushed 1 new commit to master: http://git.io/vkH5t
distro/master a7a06e0 George Clark: Item13438: apply patch from MichaelDaum...
[15:44]
***GithubBot has left [15:44]
CDotjomo: an acolyte is a probationary monk. Perl Monks is the common term for Perl experts, so Perl Acolytes is for those who are not quite there yet :-) [15:44]
gac410jomo, references have to do with the "perl monks" I believe, the true believers in perl. [15:44]
jomoah soo - now get it... yes, i know perl-monks [15:45]
CDotjomo: you are quite right, but Foswiki runtime is cumulative, following a process known as "death of a thousand cuts"
so you try to avoid adding any more cuts, unless you have no choice
[15:46]
gac410Okay MichaelDaum Patch applied for Item13438 Marked task close as bug is not in released version of ERP [15:46]
MichaelDaumya
thx
[15:47]
jomoCDot: ok ;) [15:47]
gac410CDot, just need to add cuts at a slower rate than Moore's law halves them. :)
gac410 wants to try foswik on an RPi someday :D
[15:47]
CDotheh; and devil take the hindmost. No one needs hardware older than 2 years ;-) [15:48]
jmk0"I don't need to swim faster than the shark, I just need to swim faster than you" [15:48]
CDotgac410: I have two RPis, and Foswiki is great (single user) on both [15:48]
jomojomo 's macbook is MacBook Pro (13-inch, Mid 2010) [15:49]
gac410cool [15:49]
jomoquite old... [15:49]
jmk0moore's law is reaching the limits of physics in any case [15:50]
CDotCDot 's macbook is a MacBook Pro (13-inch, 2006) and is far from cool. gets so hot I can't use it on my lap :-( [15:50]
jomobecause it is a notebook and not an laptop :) :) [15:52]
jmk0i can probably get these topic dependency violation notices working with the tools I have but right now I'm too tired to make any meaningful progress. :-P [15:54]
jomook, got an strong WiFe signal... brb in few moments ;) [15:54]
gac410jmk0: ??? topic dependency violations?
gac410 missed something
[15:54]
jmk0that thing i've been working on where one topic changes and other related topics may need to be updated [15:56]
gac410CDot. I'm leaving http://foswiki.org/Tasks/Item13424 open, for your review. Is my change https://github.com/foswiki/distro/commit/5cba582483a5 too many additions to the 1000 cuts
iirc without my patch, and without ASSERTs enabled, it still works. The patch only prevents crash on trunk.foswiki.org when ASSERTS are enabled.
It lets testing on trunk.foswiki.org proceed for the translators of french, and a couple other languages, but is probably of no value to a normal production site.
If you are worried about a single "if" ... what does a try:catch clause add in overhead for every translated string.
[15:56]
jomothe whole maketext is dog slow - it makes 40% of time under vanila CGI (fcgi helps, only the 1st request is long) but anyway - the i18n is a part what needs a massive rewrite (maybe in the 2.3) [16:01]
gac410gac410 is going to do an email to Translators, Asking for review of string changes with beta2 planned for this weekend. [16:02]
GuilainCGuilainC is happy to heard that beta2 should be release, He's preparing the server (upgrade, clean-up & co) [16:07]
jomothe beta2 _AND_ the FO switch to utf8 = major breakthru into "21th century type app" category :) [16:08]
gac410CDot, jomo, do I correctly state "The Foswiki core now operates exclusively with the utf-8 and unicode character set. All conversions are done at the 'edge' (Store, file system, and html interface') [16:09]
jomoperl core simply works with unicode characters (and nobody care in what encoding). Every encoding/decoding work is done at the borders (and would be nice to have a DOC for the plugin developers - what shows where are the borders exactly [16:10]
gac410And CDot, do you want to make a release of CharsetConverterContrib so existing sites can install the latest? [16:11]
jomoor better: perl core simply works with characters (and nobody care in what encoding). Every encoding/decoding work is done at the borders (and would be nice to have a DOC for the plugin developers - what shows where are the borders exactly [16:11]
gac410All the locale files seem to "just work" I assume they are native utf-8 encoded But I still find Locale::Maketext a bit of a black box. [16:13]
jomoreaaly really important: http://perldoc.perl.org/perlunifaq.html#I-lost-track%3b-what-encoding-is-the-internal-format-really%3f [16:15]
CDotrewriting that in slightly better English: "The Foswiki core now works with Perl characters rather than encoded byte strings. All encoding/decoding is handled 'under the hood' and problems with using non-English character sets have been mostly eliminated"
""The Foswiki core now works with Perl characters rather than encoded byte strings. All encoding/decoding across external interfaces is handled 'under the hood', and problems with using non-English character sets have all but been eliminated"
[16:15]
gac410Okay thanks CDot. BTW i ran the charset detect script against locale directory. All files detect as utf-8 [16:17]
CDotwell, that's a good sign :-) [16:17]
gac410So no other encodings have slipped in. Looks clean [16:17]
jomoweblate allows something other as utf8? i doubt... [16:18]
CDotwe wanted to check, just to be sure. [16:18]
gac410So for the translators: "This means that you can expect any characters used in your translations to render correctly throughout foswiki. "
This also means that topic names, attachment names etc. including WikiNames of users can use their localized characters
And collation sequences within foswiki should be correct
[16:20]
jomobut warning about the unicode in the passwords... [16:21]
gac410hm. Isn't that fixed? [16:22]
jomoi don't know
fixed?
http://foswiki.org/Tasks/Item13402 - it isn't urgent...
[16:22]
gac410CDot: More checking, There are 3 topics in git checkout that detect as other than utf-8
EUC-JP /var/www/foswiki/distro/core/data/System/FamFamFamSilkCompanion2Icons.txt
EUC-JP /var/www/foswiki/distro/core/data/System/FamFamFamSilkGeoSilkIcons.txt
EUC-JP /var/www/foswiki/distro/core/data/System/ReleaseNotes01x02.txt
[16:28]
GuilainCgac410, advice for translators, be carreful on confusion about the treatment of the ' sign in the translation, need to be pass into html entity &#39;
GuilainC discover that t.f.o works ! great !
[16:30]
gac410hopefully they are up to speed on that one. ;) But I'll mention it.
GuilainC: and t.f.o will be even better once the utf-8 conversion is done ... hopefully this evening.
[16:31]
CDothow do I change from nat to WYSIWYG, if I started in NatEdit? [16:32]
gac410There is a icon Looks like combination fwd & back arrow [16:32]
CDotEUC-JP usually suggests entity 146 [16:32]
gac410yin/yang symbol [16:32]
CDotbacktick, i think [16:33]
GuilainCGuilainC feels happy to not be the only one who have some difficulties to found the switch for the wysiwyg button
and CDot if you have set somewhere nowysiwyg = 1, the button is not drawn anymore, you need to unset/reset this variable (wysiwyg)
gac410, is normal that i've get in the end of the page
[16:33]
jomofoswiki has something such FontAnvesomePlugin or like? (as the fanfanfan for the images, only for font-avesome icons)... [16:36]
GuilainCERROR: Translation failed, see server error log. [16:36]
CDotGuilainC: ok, I understand, thanks. [16:36]
gac410GuilainC: Apparent bug in Perl 5.16.3 Doesn't fail in 5.18+ or < 5.16 [16:36]
GuilainCok, the utf8 translation will change it ? [16:36]
gac410No. Bug in perl [16:37]
GuilainCor it's "more complicated"
ok
so, at this moment, I ignore it
[16:37]
gac410GuilainC: The error should be gone now. I disabled ASSERTS
The error only shows up if foswiki is converting warnings to errors.
[16:38]
GuilainCright, no error now... fine.
FoswikiBot is not here ? still not fixed ? someone have seen RaulFR these last times ?
[16:40]
gac410WikiRingBot: seen RaulFR [16:42]
WikiRingBotWikiRingBot has last seen RaulFR 5 May 2015 - 21:02 GMT on #foswiki ... said, hello :-)
WikiRingBot has last seen RaulFR_ 16 Mar 2015 - 15:49 GMT on #foswiki ... joined channel #foswiki
[16:42]
jomowhat encode version uses 5.16.3? perl -MEncode -E 'say $Encode::VERSION' [16:42]
GuilainCthanks gac410 ! [16:42]
jomoi havet brewed it [16:42]
gac410jomo, I've tried updating to most recent Encode, still fails. [16:42]
jomoah so.. [16:42]
gac410But trunk.foswiki.org is using Encode 2.55 [16:43]
GithubBot[distro] cdot pushed 3 new commits to master: http://git.io/vkQnP
distro/master 275d41e Comment: Item12888: skip RCS tests that check for 1.1.x broken behaviour
distro/master fe2a385 Comment: Item12888: skip RCS tests that check for 1.1.x broken behaviour
distro/master ee6fd5c Crawford Currie: Item13405: remove unconditional decoding of src attribute, so we retain the encoding if the result is still an img tag
[16:44]
***GithubBot has left [16:44]
gac410jomo, password issue is fixed.
I was able to change a password to dòĺórém and then successfully login with it.
[16:45]
foswiki_irc3Hello -- looking for some help with automated e-mails from Foswiki. Currently it never seems to update the date/time of "new or changed topics in Web, since.." [16:46]
jomoyes? so what happens when do you in the latin1 terminal enters "htpasswd -nb admin éíá" and pastes the result as teh SUDO password? [16:46]
gac410why would you do that? [16:47]
jomobecause this the way how to set the sudo password?
as long the configure says it ;)
[16:47]
gac410No, Just enter the password in the field. [16:47]
jomojomo going to check [16:48]
gac410It will accept a password entered that way, but it will properly encode a plain text password [16:48]
CDotfoswiki_irc3: I'd guess you were running the mail notifier from a user that doesn't have write access to the web directories [16:50]
foswiki_irc3It should be running as www-data
Let me double-check
[16:50]
jomowow - it is changed - no htpasswd -nb anymore ;) [16:50]
gac410jomo :D [16:50]
foswiki_irc3@CDot It does update below the email -- it's just the header that doesn't update [16:51]
gac410Kenneth complained ... we fixed. [16:51]
CDotfoswiki_irc3: my observation stands. [16:51]
foswiki_irc3So under topics in web it says when it was changed but the header hasn't updated
kk
that user that is running the job for the notification has access to the /data/Web folder
[16:51]
gac410foswiki_irc3: just checking, are you running Foswiki-1.1.9 [17:01]
foswiki_irc3I think it's older
1.1.4
[17:02]
gac410that is pretty old - December 2011. It has a number of security vulnerabilities as well. But I don't see any particular tasks jump out as related to notification timestamps [17:05]
foswiki_irc3ok - seems wierd because it's just the timestamp in the header. hte "changed" column for the web is updating. [17:07]
gac410By header, you mean the string: New or changed topics in Foswiki.Main, since 02 Jun 2015 - 08:53: [17:08]
foswiki_irc3correct [17:10]
gac410Okay That string is read from the working/work_areas/MailerContrib/<webname> file
if for some reason that file cannot be written, then the time will be "stuck" for the last run, and you probably will also get duplicate notifications.
[17:14]
foswiki_irc3I see.
ls -l
indeed the last time it was modified is the timestamp
[17:15]
gac410So check permissions for that file. It's critical that mailercontrib can read/write the working/work_areas/MailerContrib/* files or notifications will be messed up. [17:17]
GuilainCbe carefull foswiki_irc3, mailnotify is running with cron [17:18]
gac410Your complete foswiki working tree should be fully owned by the same user (www-data www, apache, ...) depending upon your distro [17:18]
GuilainCcheck the user used in the cron job [17:18]
gac410And ALL cron tasks should be run by that user. Never root or a general user [17:18]
GuilainCso for you, it's www-data which needs to make the cron [17:18]
gac410ie crontab -u www-data -l should list your foswiki crontab
a classic issue is someone runs a foswiki script as a quick sudo and "contaminates" ownership it can badly break your system.
[17:18]
***jomo has left [17:20]
foswiki_irc3yes, I am absolutely sure that has happened before
the permissions for the file are www-data:www-data which match as www-data runs the crontab job for mailnotify
[17:21]
gac410and it never gets updated by a mailnotify run? [17:22]
foswiki_irc3correct
It shows last modified of 5-13 which is what the header shows as a timestamp
[17:22]
gac410Is it broken for all your webs, or just one? [17:23]
foswiki_irc3well that's the only web I get. Judging from "last modified" the others are probably not working either [17:24]
gac410I don't see any closed or pending tasks that are related to this, and Foswiki.org mailnotify has been running correctly for forever. [17:25]
CDotfoswiki_irc3: does the cron job generate anything interesting in the log?
mailnotify is used on most foswiki sites, and I've never heard of anything like this either
[17:25]
foswiki_irc3there is na output.log in this same folder but it ends in 2014
CDot - which log?
[17:26]
CDotthe one you redirect the output to in the cron job [17:26]
foswiki_irc3I don't think that output is redirected
unfortunately
[17:27]
CDotcrontab -l -u www-data [17:27]
foswiki_irc3cd /var/www/foswiki && perl -I bin tools/mailnotify [17:27]
gac41015 */4 * * * cd $WEBSITE/bin && perl ../tools/mailnotify > ../mailnotify.log 2>&1 for ex is what we use on foswiki.org [17:28]
foswiki_irc3I'll add a redirect [17:28]
CDotok, well, do you have sudo rights? [17:28]
foswiki_irc3yes [17:28]
CDotok, then cd /var/www/foswiki && perl -I bin tools/mailnotify from the command-line [17:29]
foswiki_irc3Should I redirect it? [17:29]
CDotyou don't need to, the output will come to the console [17:29]
gac410er.... from the command line ... but sudo -u www-data [17:29]
CDotgac410: thank you [17:29]
gac410:) [17:30]
CDotsudo -u www-data perl -I bin tools/mailnotify [17:30]
gac410So CDot ... how confident are you in CharsetConverterContrib. i konw ... you ask TWICE if a backup has been created. I took a full backup yesterday - took an hour or two. Do I need to repeat it tonight after I shutdown apache? [17:31]
CDothow long is a piece of string? How many angels can dance on the head of a pin? [17:32]
gac410:P [17:32]
CDotthe answer, my firend, is blowin' in the wind
(all together now)
the answer, is blowin' in the wind
[17:32]
gac410Well you'll find out tomorrow morning if I email you a "disaster log" and foswiki.org is still down :D [17:33]
foswiki_irc3I'm getting no such file or dir
I am www-data in /foswiki/tools
[17:33]
CDotCDot plans a quick trip to a country with no extradition treaty with the USA [17:33]
foswiki_irc3perl -I bin tools/mailnotify
nv
nvm
I got it
"sent newsletter for EPC to ..." just filled my console
I think I dumped my putty log though
date is still the same
but no other errors that I saw
[17:33]
gac410probably any error will happen at the END. When it tried to write out the last run date.
hm.
[17:35]
foswiki_irc3Last notification was at 2015-05-13T03:48:51-04:00
No errors
[17:36]
gac410That's expected. The question is, did the working/work_areas/MailerContrib/<webname> get updated [17:36]
foswiki_irc3let me check
yes
the modifed date changed on the files for main and my pther web
I guess that's progress
but the timestamp remains
[17:37]
gac410y. so the big question is why isn't your cron job updating the file. btw it's not the modified time that's important it's the content of the file. [17:38]
foswiki_irc3ah
I can look in the file
I don't remember what it was though
[17:38]
gac410It's just the unix epoch working/work_areas/MailerContrib/<webname> [17:38]
foswiki_irc31433258010
that's right isn't it?
I just used a converter
[17:39]
gac410if you do on your system "date +%s" that will show you the current timestamp. Depends on if your system is on gmt, etc. [17:39]
foswiki_irc31433266812
Looks correct.
or is it...
[17:40]
gac4101433258010 is Tue, 02 Jun 2015 15:13:30 GMT [17:42]
foswiki_irc3yes it is right
I am in CEST, this server is in MT
[17:42]
gac410in any event, that's a lot closer than a date in May. So your NEXT mailnotify run should show Tue, 02 Jun 2015 15:13:30 GMT as the changes since ... date
So the $20,000 question, is why your regular cronjob can't update if it ran okay as sudo -u www-data
So best, update your crontab to capture the log and then review the log after your next cronjob executes.
[17:43]
foswiki_irc3ok will do.
Thank you.
[17:45]
gac410you're welcome [17:46]
CDot: I've just trashed Sandbox/WikiGuest plus WikiGuest cannot view WikiApps so hopefully guest updates are squashed. [17:52]
CDot:-) [17:52]
gac410What should we/I do about those 3 EUC-JP topics.
Symlink them so that the CharsetConverter fixes them from a 1.1.x system?
[17:53]
CDotthey are not really EUC-JP, IIRC
that's just the best match the detector can make
as I recall, they are really cp1252
[17:54]
gac410Okay so CharsetConverter just trips over an artifact [17:54]
CDotor maybe cp1251 - I forget
they still need converting, but *not* from EUC-JP :-(
[17:54]
gac410I'll see if I can find the bad character enter it as utf-8 and save with enc=utf-8
In famfam it's probably Peñate
[17:55]
CDot: it's the author's name Rolando Peñate I remove the ñ and charset convert is fine. I enter it as ^vu00f1 which seems to be the correct unicode, and save as enc=utf-8. Still fails.
So it seems to be valid utf-8 that is mis-detected as EUC-JP
But I have no idea why System/ReleaseNotes01x02.txt is detected as EUC-JP. No backticks, and I don't spot anything else. :(
[18:07]
jomo| [[%BUGS%/Item12450][Item12450]] | Remove T®wiki ref in installation instructions | [18:14]
gac410Yeah. I narrowed it down to the task list, but had not spotted that one.
Thank you!
[18:15]
jomo;)
perl -nlE 'say if /\P{Ascii}/' < ReleaseNotes01x02.txt
[18:15]
gac410gac410 has to write that one down!
Okay, and the two FamFamFamContrib topics are due to the author's name
---++ !GeoSilk icon set by Rolando Peñate
And Encode::Detect::Detect is getting it wrong, I'm pretty sure I managed to save it as utf-8
yech.... this code is ANCIENT. May 27, 2008
[18:16]
***ChanServ sets mode: +o Lynnwood__ [18:22]
jomojomo "loves" cpanm : Expiring 130 work directories. This might take a while... [18:24]
gac410gac410 wonders what this means for CharsetConverter ... Should we trust Encode::Detect
There are a bunch of topics in f.o that detect as EUC-JP that probably should not.
[18:34]
jomohm... [18:36]
CDotwhat choice do we have? You could always code in a couple of heuristics temporarily e.g. xlate EUC-JP -> cp1252, that sort of thing
though TBH I don;t remeber what the codepoint was.
[18:37]
gac410we've found them. One of the authors of FamFamFam is Rolando Peñate
And releaseNotes had a registered trademark character ... eliminated.
I just ran the charset checker on foswiki.org, and *none* of the EUC-* topics are actually japanese. It's 5 topics, mainly a couple of german characters, or the (r) trademark symbol
[18:38]
jomothe Rolando Peñate is imho ok utf8 [18:42]
gac410right. I agree. I manually entered the ñ using Ctrl-v u00f1 which is the valid utf8 character, and saved it using :write++enc='utf-8' [18:43]
jomowho will do the conversion? the admin. He probably knows what encoding he used in the 119 ... so ... [18:43]
gac410The problem is only topics that have manual cut/paste data of non-default encoding. like on foswiki.org [18:44]
jomobut those could be inentional too.. [18:45]
gac410Typical site will just use bulk_copy to convert from {Site}{CharSet} to {Store}{Encoding} ... probably iso-8859-1 to utf-8
CharsetConverterContrib is only needed when there is a mix of topics of various encodings. where the {Site}{CharSet} cannot be assumed.
[18:45]
jomoi have a bunch of topics what edited manually in iso terminal - ony because want test something.. [18:46]
gac410Right, but that's pretty unusuall. [18:46]
jomoso, in the relity: take the old SiteCharset and convert to utf8 - everything what throws some error - leav as is... [18:46]
gac410So the "typical site" running on iso-whatever ... anything unusual is currently corrupted manually c/p and will remain so. [18:47]
jomoyou cant make a diference betwwen iso-1 and iso2...
mean automatically
[18:48]
gac410cdot ... what does bulk-copy do if a topic has inconsistent encoding. throw error, or just keep going with possible double-encoding or other issues. [18:48]
jomothey both are 1byte charset - both has valid characters - only their meaning is different...
IMHO the admin MUST say - from what charset going to convert to utf8...
[18:48]
gac410jomo CharsetConverterContrib. Reads each topic text, checks it with Encode::Detect::Detector and overrides the "from" encoding if the -r repair option is specified. [18:49]
jomoand how the Detector makes a difference between iso1 and iso2??? [18:50]
gac410magic. I have no idea. [18:50]
CDotIt reads the topic, decodes using the {Site}{CharSet}, encodes to utf-8, sends it to the other wiki, which decodes the utf8 and encodes in it's own {Site}{CharSet}
unless -r is set, in which case it uses the detected encoding if it isn't {Site}{CharSet}
[18:50]
gac410CDot, so if the topic is already in utf8, (ie like foswiki.org) it will encode the utf-8 a 2nd time and cause corruption? [18:51]
CDotcorrect
that's why we needed to run the CharSetConverter first
to canonicalise the encoding
to an encoding that we know will support all chars i.e. utf8
but of course having done that, we no longer need to bulk_copy
[18:51]
gac410right. So the issue is that encode::Detect::Detector appears to incorrectly identify EUC-JP and EUC-KR [18:52]
CDotaye
unless it is justified, just whack in a heuristic
[18:52]
gac410So at least for foswiki.org, I'm wondering if we should have a one-off if detected = EUC-*, assume it's already utf-8 [18:53]
CDotand ignore EUC-JP and EUC-KR, in favour of the encoding it should "obviously" have detected
can you assume that?
it seemed to do a fairly good job of detecting utf-8 when I ran it
[18:53]
gac410well there are a total of 4 topics on foswiki.org that detect as EUC [18:53]
jomounderstand in the iso1 the F2 mean: ò - valid character - the SAME F2 in iso2 mean ň = no detector has a change guess what is teh right - e.g. F2 will be converted to utf8 as what? ò or ň ? [18:53]
CDotand what is their "real" encoding? [18:53]
gac410| [[%BUGS%/Item12450][Item12450]] | Remove T®wiki ref in installation instructions | [18:54]
CDothmmm. [18:54]
gac4101 criteria must be in accordance with Abgabenordung (AO) "Steuerbegünstigte Zwecke", paragraph 59-61 (from an old Board Meeting)
clearly german.
[18:54]
CDotclearly iso-8859-1 (or-15) I'd say [18:54]
gac410The string in VIM shows "Steuerbeg\xc3\xbcnstigte Zwecke [18:57]
jomoimho = here is no chance for automatic guessing... the the topic will contain only one 0xF2 (as baňa) (iso2) - the guesser will sure guess it as iso1 and will get baòa :(
0xc3 in iso2 is Ă :)
[18:57]
gac410and the correct character is the u with umlat
https://de.wikipedia.org/wiki/Steuerbeg%C3%BCnstigte_Zwecke
[18:59]
jomobtw the \xc3 is strange, because in the iso1 - the ü has a code: 0xfc...
as utf8 encoded..
jomo just realised -- he reads wrong the \xc3\xbcn
[19:00]
gac410That wikipedia page, displays the ü correctly and shows encoding of utf-8 in the metadata [19:01]
jomoyes the 0xc3 0xbc is utf8 encoded u with double dots :) [19:02]
gac410so encode::Detect::Detector is detecting a utf-8 file as EUC-JP [19:02]
jomoalso" http://www.fileformat.info/info/unicode/char/00fc/index.htm
UTF-8 (hex) 0xC3 0xBC
[19:05]
gac410gac410 head huts
hurts
[19:06]
jomoagain - automatic guessing is pointless - it is IMPOSSIBLE to do it correctly - especially from ISO-something to utf8... [19:07]
gac410So, what to do about foswiki.org. we have an impossible situation then.
The vast majority of the topics detect as windows-1252
[19:08]
jomocp1252 vs iso1 is somewhat guessable - because the 0x8? and 0x9? codes in the ISO are empty (e.g. no text should contain it) but the 1252 used the for some characters... [19:11]
gac410My concern is that I'm going to run CharsetConverter tonight and we'll be left with an unusable mess on foswiki.org
backup file is 6.2 gig, took me 1-2 hours to run the backup yesterday.
maybe I'll pull it down here and do a dry run on my laptop.
[19:13]
jomojomo understands - but doesn't see any solution... only do the conversion as iso1 -> utf8 - because the FO is configured as iso1... [19:15]
gac410which will destroy all your 1.2 test cases for eg. [19:15]
jomoomg [19:15]
gac410They are in git, so we can revert. But there are lots of topics here & there where people have cut/pasted in utf8 I'm guessing. [19:16]
jomogac410: your proposal from the years ago - the topic should contain the used charset ... [19:16]
gac410I dont' think that was mine. Someone actually started to implement it. then reverted.
well copying the backup locally to test won't work. scp estimated time to copy 23 hours
[19:17]
jomoso, special foswiki case - if the topic is utf8 - take it without conversion. otherwise it is iso1... ;) so the question is reduced: guess the iso1 from the utf8 - what is guessable... this is acceptable? [19:22]
gac410tbh I'm not really sure. :( [19:22]
jomoFO upload is so slow? or you have slow link at home? cos 6GB coulnd took 23 hours... [19:23]
gac410gac410 is thinking of doing a "cp -a" of foswiki.org foswiki.utf8 and then run the charset converter on the copy.
And see what sort of a mishmash it creates.
[19:23]
jomo:) [19:24]
gac410my download speed is around 64 Mbps [19:24]
jomohm.. so 6Gb around 20 minutes... ;) [19:26]
gac410yeah. so something at the other end. [19:26]
GuilainCgac410, I supposed you haven't forget to enable compression [19:30]
jomotry with simple curl - without the ssh encryption [19:30]
gac410then I have to move the backup to the web server. not worth it. [19:31]
jomo:) [19:32]
gac410Okay so odds are windows-1251, 1252 and 1255 will correctly convert to utf-8 (/me hopes)
That leaves 30 files with detected odball encodings. 12 are in legacy webs that are read restricted. so 18 topics are potentially subject to damage?
[19:32]
jomocould you tell me one topic what is in 1252? [19:33]
gac410/home/foswiki.org/public_html/data/Home/HowWeUseCookies.txt
so http://foswiki.org/home/HowWeUseCookies?raw=all
er. capital H in Home
[19:35]
jomorestricted - ok, just wondered :)
ok - with H works :)
[19:35]
gac410yeah. I picked one that you could access. [19:36]
jomoyes - this is detectd right - it contains 0x92 what is iso1 isnt defined... so, good ;) [19:38]
gac4101365 topics decode as windows-1252 [19:38]
jomoso, if this one is detected correctly as 1252 - we can hope - the other 1300 will be ok too. ;) [19:39]
gac410http://foswiki.org/Support/Faq45 detected as iso-8859-8 [19:39]
CDotfair enough - "How to speak russian" [19:42]
gac410http://pastebin.com/icnEXNPu is the list of odball encodings [19:42]
jomojomo just reading thru https://metacpan.org/source/JGMYERS/Encode-Detect-1.01/src - it uses some interesant probablity based guessing... (c sources) - so XS/fast... [19:43]
gac410excluding the Legacy* webs which are all read restricted
Strangeness ... why on earth does it thing a simple topic like http://foswiki.org/Support/Question824?raw=all is encoded as Shift_JIS
Ah. I�m
a back-tick as the apostrophe
Yeah any topics with I�m detect as Shift_JIS
[19:43]
jomo0x92 - curved quotes as apostrophe... in 1252 [19:51]
gac410So it must weight a single instance as Shift_JIS, but combined with other characters it becomes 1252 [19:52]
jomothe 0x92 is this in unicode http://www.fileformat.info/info/unicode/char/2019/index.htm
e.g. the 0x92 alone - detected as ShiftJS but together with some other characters like áé it detects it as 1252
[19:53]
gac410so for some overrides for f.o, Treat Shift_JIS as windows-1252 Treat ECU-JP as already utf-8 [19:55]
jomoimho no topic in FO using Shift_JIS (me doesn't even know what is the ShiftJIS is) [19:55]
gac410another one-off oddball. http://foswiki.org/Sandbox/TestTopic133321 deteted as KOI8-R [19:55]
jomoÅȚ - no idea
this is like test topic as i said.. :)
what could mean the ÅȚ in koi - haven't idea
[19:56]
gac410So I think if we override Shift-JIS as win 1252 and ECU-anything as utf-8 we only have a handful of topics that will be maybe corrupted,
And those are probably already corrupt so :P
[19:58]
jomo;) [19:58]
gac410http://foswiki.org/Sandbox/ExperimentationWithListFunctions displays correctly, but gets detected as BIG5 whatever that is.
Looks like it's the not-sign
[20:00]
jomo¬ [20:01]
gac410yes. isn't that a not sign [20:01]
jomoyes, used as table corners in good old times :)
ansi graphics
[20:01]
gac410okay so BIG5 encoding is definitely wrong. It's sure not chinese. [20:01]
jomo:) [20:02]
gac410So ignore BIG5, iso-8859-1 should work considering the topic displays correctly. [20:02]
jomohmm = http://foswiki.org/Tasks/Item13372 as IBM866 [20:04]
gac410\xc2\xb0 displays as the "degree symbol" and gets deteced as iso-8859-7 [20:06]
jomoso, what we have ? have some utf8 (sometimes detected as ECU) and some others what we cab treat as 1252 (1252 has more characters as iso1) - so 1252 -> utf8... right?
s/cab/can/
[20:08]
gac410yes. [20:09]
jomoand if we found really one-two korean topic - we could live with this... ;) [20:10]
gac410So we can let it autodetect the windows, and utf8 encoding. EUC* we override to utf-8 Shift_JS we override as windows-1252 BIG5 we ignore, default it 8859-1
and the few others, well we live with possibly a few corrupted topics.
[20:11]
jomoor default 1252... theyre same - only the 1252 has 32 more characters.. [20:12]
gac410okay
what this says is that sites that decide to resort to CharsetConverterContrib are going to need to do hand tuning.
[20:12]
jomohm.. seems yes.. [20:14]
gac410I wonder if sites that use the bulk_copy to copy from iso-8859-1 Site}{CharSet} really ought to consider using windows-1252 as their site charset, considering that most users are entering data from windows platform.
Odds are that's why we have so may topics detecting as windows-1252 and not iso-8859-1
[20:14]
jomoconversion from 1252 seems be "more safe".. [20:15]
gac410Actually we had NO topics detect as Iso-8859-1 [20:16]
jomobecause the apostophes... and other this what are used in the doc format - and the user simply copyed here something from doc or such..
the iso1 is a form of "subset" of 1252...
1252 has everything what iso1 has + 32 chars (what aren't defined in the iso1) - so seems it is more safe convert from 1252
[20:16]
gac410yeah I get that.
okay I think I've got an idea of what to do. Thanks for all the help guys
[20:17]
jomo;) that was chatting only - the hard work only coming... ;) [20:19]
gac410well CDot did most of the hard work. Just need to tune up the conversion a bit. [20:19]
CDotgac410: if you look at my detect_encoding version in tools/develop[ you'll see I used additional heuristics to decide if a cp1252 was *really* ios-8859-1 (or iso-8859-15) [20:21]
gac410ah... okay, yeah I was just using the simple one in my home dir. Do you use the same huresitics in convert_charset
I'm debating going ahead with this tonight. I've been sitting here for 8 hours, and need a break.
[20:23]
jomoofc - go some lawnmowing ;)
no hurry - few days plays no role
[20:27]
gac410hm. I don't think CDot checked in his detect_encoding. No such file that I can find. [20:31]
jomoright here isn't any detect_encoding file..
jomo going to learn http://webix.com - CUL
[20:32]
....... (idle for 33mn)
GithubBot[CharsetConverterContrib] gac410 pushed 1 new commit to master: http://git.io/vk53t
CharsetConverterContrib/master 30521c4 George Clark: Item13442: Tweaks made on foswiki.org
[21:07]
***GithubBot has left [21:07]
............ (idle for 56mn)
GithubBot[distro] gac410 pushed 1 new commit to master: http://git.io/vk5wu
distro/master 87f5bff George Clark: Item9693: Update release notes and upgrade guide
[22:03]
***GithubBot has left [22:03]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)