#foswiki 2012-02-22,Wed

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
gac410cool - the very next line after CGI::a complains. <nop>s in tooltip are encoded as &lt;nop&gt;. ... and reverses it. I think my disable autoEscape will avoid that issue too. [00:00]
pharveythe bug title is correct: it's double encoded rather than once. [00:00]
gac410okay - then the fix is very easy. I'll fix the unit tests and the code. [00:00]
pharveyawesome, thanks :)
no wonder I had trouble fixing this the first time round - doesn't help to write tests that are wrong :)
[00:00]
gac410:-)
I also couldn't run the tests on the release branch. They also depend upon Foswiki::Address
[00:02]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address [00:03]
pharveyd'oh. I had lib/Foswiki/Address.pm in the release branch of UnitTestContrib, but decided I'd better write tests without it.
gac410: my usage of Foswiki::Address was the beginning of my efforts to refactor link handling. As that work has stalled, I think it's bets to replace with Foswiki::Func::normalizeWebTopicName
I can fix this up tonight if you leave the task WaitingFor me
[00:04]
gac410okay - well let me figure out what's going on with the link - 1 of 2 tests is passing.
gac410 begins to pull hair. Test was passing just fine. I removed my print statements - now it fails. :-(
[00:07]
pharveygac410: the tests have a TRACE mode which might useufl [00:13]
gac410okay - I'm poking around at it. Something strange is going on. [00:14]
pharveyyou've probably already seen that the two tests run through a list of test cases themselves [00:14]
gac410FormattingTests::test_ampersand_querystring: seems to be small - self contained. [00:15]
pharveyoh, yeah [00:16]
gac410pharvey - I think there is something wrong with that standalone test_ampersand_querystring. The rendered link looks fine, but it claims frag: undef, query: undef, addr: a> [00:30]
pharveywhat does the TRACE output show? [00:30]
gac410[[TestTopicFormatting?q=r&amp;s=t]]
<a href="/TemporaryFormattingTestWebFormatting/TestTopicFormatting?q=r&amp;s=t" class="foswikiCurrentTopicLink">TestTopicFormatting</a>
frag: undef, query: undef, addr: a>
So the &amp; looks great. no double encoding.
[00:31]
pharveyindeed
can you commit or pastbin your fix (or push it somewhere? :)
[00:31]
gac410I'll commit - it is very small - + my $aFlag = CGI::autoEscape(0);
my $link = CGI::a( \%attrs, $text );
+ CGI::autoEscape($aFlag);
Clear and save the autoEscape flag, generate the link then restore the flag.
[00:32]
pharveyok, I have a few mins before my run finishes
gac410, have you modified the test? query: looks correct (still fails, but I'll fix that now)
[00:33]
gac410Yes - Didn't seem to matter what I did to the test. Tried it with or without &amp; in the 1st or 2nd parameter. [00:35]
pharveyoh, wait. It passes; I had commented something out
gac410, with your changes, the tests all pass :)
[00:36]
SvenDowideitmmm, i _might be getting ~800installations of foswiki in the last 7 day period via the debs?
thats insane!
836 836 0.03% http://fosiki.com/Foswiki_debian/pool/main/f/foswiki-ldapcontrib/
[00:36]
gac410Excellent - I'm about to commit - forgot to turn off TRACE [00:37]
SvenDowideitoh, maybe thats how many times someone's running update to see if there is one
77 10.30% Feb/20/12 8:20 AM /Foswiki_debian/pool/main/f/foswiki/foswiki_1.1.3-2_all.deb
more like
50 0.02% Feb/19/12 5:57 PM /Foswiki_debian/pool/main/f/foswiki-ldapcontrib/foswiki-ldapcontrib_120112-225_all.deb
[00:37]
pharveygac410: so I think the tests are written correctly after all, but they might be a little misleading. I wrote it so that each entry in the test data had to only be written once, so the test escapes & to derive the expected href="..." value automatically [00:39]
gac410I've checked in my changes btw. [00:40]
SvenDowideitso 50-100 per week ~~ 250-500 debian based installs of foswiki per year? that would be well worth it :) [00:40]
gac410but I still have 1 test failing. [00:40]
pharveywhich one? [00:40]
gac410FormattingTests::test_ampersand_querystring: [00:41]
pharveyit fails to fail? or ... what's the error?
pharvey getting passes here, after svn rebase
we need to remove the expect_failure()
FormattingTests::test_ampersand_querystring *** Unexpected pass
[00:41]
gac410It fails. Unit::TestCase::assert_str_equals('FormattingTests=HASH(0x87b5550)', 'TemporaryFormattingTestWebFormatting.TestTopicFormatting', 'TemporaryFormattingTestWebFormatting.a>', 'address mismatch checking [[TestTopicFormatting?q=r&amp;s=t]]') called at /var/www/foswiki/trunk/core/test/unit/FormattingTests.pm line 1786 [00:42]
pharveyhow utterly odd
are you on trunk?
[00:42]
gac410yup - no Foswiki::Address on release [00:43]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address [00:43]
pharveyoh
it's expecting TestTopicFormatting, but getting 'a' as topic name
can you rm -rf /path/to/data/Temporary* ?
[00:43]
gac410hm. strange. Now it just passed. [00:44]
pharveynot that this should affect it [00:44]
gac410I did a clean and re-pseudo-install [00:45]
pharveypharvey doesn't know how to explain it :/ [00:46]
gac410but two other tests fail now - I rebuild LocalSite.cfg and HierarchicalWebs are disabled ... strange. [00:46]
GithubBot[foswiki] foswiki pushed 1 new commit to master: http://git.io/ZpLU6g
[foswiki/master] Item11366: Prevent double encoding. - GeorgeClark
[00:48]
***GithubBot has left [00:48]
FoswikiBothttp://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ] [00:48]
gac410wtf. EnableHierarchicalWebs is removed from Foswiki.spec [00:55]
pharveywtf. AuthScripts too
wait... it's just not in the configure UI
[00:56]
gac410well EnableHierarchicalwebs is not in Foswiki.spec. [00:57]
pharveyah, somebody made AuthScripts conditional on TemplateLogin [00:57]
gac410yes. probably CDot. [00:58]
pharveyright, because we don't do redirects now
.... DISPLAY_IF looks js-tastic
oh, never mind
anyway, the problem is that if you use ApacheLogin, you get a warning on the Login tab on the AuthScripts but can't see it to fix it (or the warning message)
[00:58]
gac410and pharvey wonders why trunk scares gac410
:-D
[01:00]
SvenDowideityay, finally
http://fosiki.com/blog/2012/02/22/centos-yum-install-foswiki-and-debian-apt-get-install-foswiki/
[01:02]
pharveyscary implies in intractable or not worth the effort... I'd say trunk is tedious, not scary :P
SvenDowideit, w00t.
[01:02]
SvenDowideitstore2, _thats_ tedious
trunk is just ready for testing :)
ok, so gac410 you're the most sane person here right now
should i just copy&paste that to foswiki.org/Blog ?
[01:02]
gac410So CDot deleted hierarchical webs in Foswikirev:14005 [01:03]
FoswikiBothttp://trac.foswiki.org/changeset/14005 [ Changeset 14005 – Foswiki ] [01:03]
SvenDowideitor is it too, er, self promoty [01:03]
pharveyI'm profiling some code and trying not to optimize our damn INCLUDE's habits of parseSection'ing each time there is an INCLUDE statement, but I think that wants readonly topicObjects to work properly [01:03]
SvenDowideitpharvey, correct-ish
gac410, so we have heirarchical webs _always on_?
cos really, we should do that
[01:04]
gac410No - always off [01:04]
SvenDowideitoh oops
nice way to hurt pharvey :)
[01:04]
gac410Config option removed, so any tests for it are false. [01:04]
pharveyI remember for sure now: I couldn't figure out a nice way to cache the parsed section data across all instances of a topicObject [01:04]
gac410I only discovered it because the unit tests expect fail because hierarchical webs are disabled. [01:05]
SvenDowideiti voted for Meta == the code api, the actual topic data moves into meta->{data}{text etc..} [01:05]
pharveygac410: that explains why my trin setup is so broken right now (apart from all the other bugs I'm fighting, which are sadly impacting simple views of Main.WebHome [01:06]
SvenDowideitand then meta->{cache} couls contain anything derived
but, argh :)
mmm, think i will just c&p my blog post to f.o/Blog
[01:06]
gac410please no comments regarding my sanity or lack thereof [01:07]
pharveySvenDowideit, I'd suggest you wait just a tiny bit so somebody else can writeitup :P [01:07]
SvenDowideitmmm, but who would do so in the next 4 hours? [01:07]
pharvey... or not, /me reads the post first [01:07]
gac410So do I just put EnableHierarchicalWebs back into Foswiki.spec ... so the unit tests will work? or open a task so CDot can explain wth is going on. [01:08]
pharveySvenDowideit, you need a grammar nazi to fix your post [01:08]
SvenDowideitpharvey, booooooooo [01:08]
pharveyI guess if it's in foswiki.org/Blog, that could be me :P [01:08]
SvenDowideitgood point!
lets see how well wysiwyg de-re-htmls
[01:08]
pharveySvenDowideit: trunk's WYSIWYG is much better
(and I hope to back-port to Release01x01 for 1.1.5)
okay, I just wanted to see if the post could be perceived as you promoting yourself, but it's a publicly available repo and the post very to-the-point, so I guess not
gac410: can you re-open the task CDot mangled it under?
pharvey wonders what the task was
[01:10]
SvenDowideiti'll bet it will be considered self promotion by someone - because it is - to the degree of 'this is wot i done' but thats kinda what im hoping people will use theblog for [01:12]
gac410Item11528 [01:12]
FoswikiBothttp://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ] [01:12]
SvenDowideithttp://foswiki.org/Blog/Centos_yum_install_foswiki_and_Debian_apt-get_install_foswiki
edit-bombs away
now to prod gac410's post like i promised :)
[01:12]
gac410SvenDowideit: Any ideas why ConfigureTests are bombing out for trunk on your nightly build system? [01:15]
SvenDowideitnone at all atm :/
i wonder if i cna just make a 1.1.4 deb without effort :/
[01:15]
gac410Changes in perl modules? It passes here fine. ('cept for the HierarchichalWebs issue now. :-( ) [01:16]
SvenDowideit11am - overcast, need to mow lawn :(
lets see if i can figure out where they're running
i.need.better.sysadmin.
[01:16]
gac410:P AccessControlTests failing due to elimination of redirect in login. [01:19]
pharveySvenDowideit: I did some simple grammar fixes [01:21]
SvenDowideitpharvey, oooo [01:22]
pharveyof course, it's possible I made it worse, but I hope not.
Insecure dependency in open while running with -T switch at /usr/local/src/github.com/foswiki/core/lib/Foswiki/Configure/Checkers/Htpasswd/FileName.pm line 29.
[01:23]
SvenDowideiti hate that people want foswiki spelt with caps :p [01:24]
pharveyoh well, change it back :P [01:25]
gac410not even the leading f making it a proper name? [01:25]
pharveyI did foswiki -> Foswiki [01:25]
SvenDowideitnopw :)
i foswiki regularly :)
i'm too young&hip to be limited in my definitino of 'propername' :P
I can ~~ was able t
[01:25]
pharveygac410: I fear Foswiki::Configure::Load::expandValue is returning tainted values [01:27]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Configure::Load [01:27]
gac410Hm. why. They come from a file, not from the CGI. [01:27]
SvenDowideitSvenDowideit geos off to steal pharvey's changes [01:27]
pharveywell, it's either that or the line above: my $f = $Foswiki::cfg{Htpasswd}{FileName}; [01:28]
gac410Is this new? I don't think that code has changed in quite a while. [01:29]
pharveyI've occasionally got taint bugs from configure with trunk
this is a new location for it to happen in though (the Htpasswd/Filename.pm checker)
previously it's the logdir checker
[01:29]
gac410Is it repeatable? or is this a sporadic thing that comes/goes/moves around [01:30]
pharveyyeah, it doesn't happen every time. I'll try with a fresh config shortly.
subsequent saves always work.
[01:30]
gac410I wonder if in the initial assignment it's pulling someting off of a path variable or directory search. [01:31]
pharveyI wanted to explore the possibility there's something going on with the initial loading from .spec files [01:32]
gac410Some of the path info gets discovered from current directories, etc. and applied to the file locations.
A lot of the file locations are undefined in Foswiki.spec and are guessed.
So Htpasswd Filename is based off of the DataDir which is guessed in first run.
[01:33]
pharveydamn, I can't reproduce this time. [01:38]
SvenDowideitok, so trunk failures are bizzare, svn up did heaps of processing, so somehting was very hung [01:41]
SvenDowideit ponders that :/ [01:46]
pharveypharvey will email SvenDowideit some notes [01:47]
SvenDowideitmany thanks - would make sense to get it back working now
mmm, is that crazy or what:
Analyzed requests from Mon, Sep 25 2006 at 10:51 PM to Tue, Feb 21 2012 at 12:23 AM (1974.06 days).
[01:47]
...... (idle for 26mn)
GithubBot[foswiki] foswiki pushed 3 new commits to Release01x01: http://git.io/Bwp-8A
[foswiki/Release01x01] Item11547: update debian package to 1.1.4 - SvenDowideit
[foswiki/Release01x01] Item10743: add dependancy on HTML::Tree for compareRevisionAddOn - SvenDowideit
[foswiki/Release01x01] Item10964: add /var/lib/foswiki/working/logs softlink to debian logs dir - SvenDowideit
[02:17]
***GithubBot has left [02:17]
FoswikiBothttp://foswiki.org/Tasks/Item11547 [ Item11547: update core debian package to 1.1.4 ]
http://foswiki.org/Tasks/Item10743 [ Item10743: Fail to install foswiki plugin "foswiki-twikicompatibilityplugin" ]
http://foswiki.org/Tasks/Item10964 [ Item10964: Fresh Debian installation fails to create necessary directories ]
[02:18]
gac410pharvey - jeeze - ran the whole suite and the ampersand test fails again. looking for "a" as the topic name. [02:26]
pharvey - shorter URL support. Parses out the entire link. [02:36]
GithubBot[foswiki] foswiki pushed 1 new commit to Release01x01: http://git.io/u9bzpw
[foswiki/Release01x01] Item11547: add CVE-2012-1004 fix - SvenDowideit
[02:45]
***GithubBot has left [02:45]
gac410Sven - did you apply more than just the encode regex in that patch? The try/catch stuff in the mapper is new - I don't think you can take parts of 1.1.5 beyond the field encode. [02:49]
SvenDowideiteh?
i'm trying to follow 'Hotfix for Foswiki Production Releases 1.1.0-1.1.4
'
[02:50]
gac410Should be a one-line change. [02:51]
SvenDowideitbut dpatch is making me bleed
no, i'm using the Register.pm on that topic
and thats more than on line
[02:51]
gac410crap... It's supposed to be a one-line change. wtf.
I restored a clean 1.1.4, applied the one-line patch and uploaded that module ... I thought.
I specifically did not want any of the other 1.1.5 pending changes.
[02:51]
SvenDowideitthe diff in the commit email is longer
and this is against the 1.1.4.tgz that i have in the github/foswiki-releases repo
which is an rsync of whats on sourceforge
[02:53]
gac410The commit email is NOT the hotfix. [02:53]
SvenDowideit(i hope)
then the Register.pm you put there that people are using may not be either
[02:53]
gac410The hotfix is not what will be in 1.1.5. [02:54]
SvenDowideiti'm building a 1.1.4 pkg from tgz plus the Register.pm linked to from the hotfix
http://foswiki.org/Support/SecurityAlert-CVE-2012-1004#Hotfix_for_Foswiki_Production_Releases_1.1.0_451.1.4
[02:54]
gac410The Register.pm I posted should have the one-line fix that is in the diff in the CVE. [02:54]
SvenDowideiti'm actually in a world of debian pain [02:54]
gac410Did someone else update it? [02:55]
SvenDowideitplease, check :/
cos it chould just as easily be something stuffed with what i have
[02:55]
gac410okay - will take me a moment. [02:56]
SvenDowideitmmm, ok, it might be titally my sodup
debian is bizzaroworld sometimes
that might be a diff from 1.1.3->hotfix
[02:58]
gac410Ah ... That sounds better.
Yes, that's what it is.
[02:59]
SvenDowideitcrazy [03:00]
gac410gac410 likes tarballs. gentoo builds from tarballs. [03:01]
SvenDowideitdebian builds from tarballs too
_once_
[03:01]
gac410:-P [03:01]
SvenDowideitwhereas gentoo adds to global warming
so ner :p
[03:01]
gac410well - keeps my lap warm anyway. Good in winter months when my cat is not cooperative. [03:02]
SvenDowideitits much more environmentally friendly to move to where the cimate suits the body
but then the world starts to tilt and might just topple off
ok, that might work a little better
mmm, oh, the jquery checker is in 1.1.5 huh
[03:03]
pharveySvenDowideit: I think Item10308 was released in 1.1.3 [03:12]
FoswikiBothttp://foswiki.org/Tasks/Item10308 [ Item10308: Need to check if JQuery version is sane ] [03:12]
gac410pharvey - I'm going to commit the unit tests and double encoding fix on 1.1.5 -but the tests fail due to Foswiki::Address. [03:14]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address [03:14]
pharveygac410: k, I'll fix it tonight
I guess, expect_failure till then
[03:14]
gac410Yeah - I didn't bother with expectFailkure - didn't want the 1.1. and trunk tests to diverge [03:15]
pharveyok [03:15]
GithubBot[foswiki] foswiki pushed 4 new commits to Release01x01: http://git.io/bX1hLw
[foswiki/Release01x01] Item11547: lets diff from 1.1.4 so the patch works - SvenDowideit
[foswiki/Release01x01] Item11547: go with jquery 1.7.1 and remove some deprecated plugins - SvenDowideit
[foswiki/Release01x01] Item11366: Prevent double encoding. - GeorgeClark
[03:17]
***GithubBot has left [03:17]
FoswikiBothttp://foswiki.org/Tasks/Item11547 [ Item11547: update core debian package to 1.1.4 ]
http://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ]
[03:17]
GithubBot[foswiki] foswiki pushed 3 new commits to master: http://git.io/g3i-2w
[foswiki/master] Item11512: <base tag must be closed for HTML validation - GeorgeClark
[foswiki/master] Item11366: Both tests pass. - GeorgeClark
[foswiki/master] Item11528: Put back EnableHierarchicalWebs - GeorgeClark
[03:18]
***GithubBot has left [03:18]
FoswikiBothttp://foswiki.org/Tasks/Item11512 [ Item11512: Ensure all topics in System pass ]
http://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ]
[03:18]
pharveygac410: in Item11448 you added a warning to say that restauth needs to be protected, but the reason why I think is misleading. It says currently: "This could permit commenting by unauthorized users" - actually, the failure mode is that the browser doesn't receive a WWW-Authenticate challenge (when using ApacheLogin). Or am I wrong? [03:23]
FoswikiBothttp://foswiki.org/Tasks/Item11448 [ Item11448: {GuestCanComment} setting is not implemented ] [03:23]
pharveyin other words, it doesn't allow unauthorized users to do anything (an unprotected rest script would do the same), it just prevents ApacheLogin from requesting authentication, I think.
no, that's wrong. I'm confusing the apache config.
damnit, I hate {AuthScripts} with the power of a thousand suns
can you believe there's only two lines of code in all of lib/* that aren't lines in the manifest or checkers
which reference AuthScripts
[03:24]
gac410Probably because I don't understand the authScripts either. But my understanding was that CommentPlugin would require restauth if authentication was required. [03:40]
SvenDowideit'CommentPlugin would require restauth if authentication was required if you're using apache auth and don't know or can't set up a better apache conf'
SvenDowideit fixed it for you :)
i really should shut up about that and put out :D (as in, add authenticated script re-writes to ACG2)
[03:40]
pharveygac410: I'm doubting whether it's worth having that warning at all. If anything, it's need on the ={AuthScripts= checker, not in CommentPlugin. Because an unprotected =rest= script has the same issue. [03:42]
gac410no CommentPlugin needs the rest script unprotected.
otherwise the GuestsCanComment switch is a nop
[03:43]
pharveyso I'm struggling to understand the context of the error message
I guess I'm trying to understand: is my understanding wrong, or yours :)
the fact I'm having trouble means this is all quite sucky anyway, regardless
[03:44]
gac410tbh I'm relatively clueless. I thought that restauth must be a protected script. and for guests to comment, rest must not be protected. [03:45]
pharveyyou are right - restauth should be protected; but no more so than rest itself
there's actually three different levels of "needs auth", and that sucks
[03:46]
gac410If both are protected, then why do we need restauth. rest would be enough. [03:47]
pharvey1) in registerRESThandler( authenticate => 1
2) in AuthScripts
3) In apache
restauth is purely for ApacheLogin
so that apache can force a 401 Auth required
[03:47]
SvenDowideitand its not necessary :( [03:47]
pharveywell, not any more [03:47]
SvenDowideitever
non of the *auth ones are
[03:47]
pharveyer
that's not what happened to me
if a rest script needed auth, it failed
[03:48]
SvenDowideitthen ou need to know more apache conf :) [03:48]
pharveyunless apache threw a 401 [03:48]
SvenDowideitwhere's the flipping ACG2 topic
yes, and you can get apache to throw a 401 without needing a script to actually exist
[03:48]
pharveyhow the hell can apache know to require auth on a script that may or may not need it? [03:48]
gac410pharvey I'm with ou on that one. [03:48]
SvenDowideitapache is not as dumb as we've always thought
sun came out, so i won't mow the lawn yet
[03:49]
pharvey... Apache has *no way* of known that /bin/rest/MyPlugin/handler needs auth [03:49]
SvenDowideitmaybe i should just hack up ACG2 to show you
apache does not need to knwo
[03:49]
pharveyso how can it throw 401? [03:49]
SvenDowideit<SvenDowideit> where's the flipping ACG2 topic
it seems you're too sure that you know apache
[03:49]
pharveypresently, the only way Foswiki has to get apache to auth someone is to redirect them to a protected script [03:50]
gac410Foswiki:Support.NewApacheConfigGenerator [03:50]
FoswikiBothttp://foswiki.org/Support.NewApacheConfigGenerator [ NewApacheConfigGenerator ] [03:50]
SvenDowideitand i apparetnly can't tell you the right things for the penny to drop
so..
ta
[03:50]
pharveyok [03:50]
SvenDowideiti can't find anything in the support web
ever
ok, so, does anyone use it yet?
or is it just crazy people like me?
[03:50]
pharveybut I have debugged this extensively before, and you even sent me an E-mail implying I should just give up on /bin/scripts and re-write it so we only have two: unauth and auth [03:51]
gac410I've used it for a bunch of configs. [03:51]
pharveypharvey too [03:51]
SvenDowideitthe q being - whats the risk if i botch it up a few times
meh, i'll make a local copy to hack
[03:51]
pharveyAFAIK the only way Foswiki has to force apache to auth someone is to redirect them to a protected script [03:51]
SvenDowideitcorrect [03:51]
gac410Yeah - that's what I do for major changes though I have not done that in a while. [03:52]
SvenDowideitbut there is __zero__ need for that script to actually exist
where zero is small - there is iirc one senario where you can't use htaccess due to conf restrictions
[03:52]
pharveywell, that's a different hack, I guess [03:52]
SvenDowideitif we're giving people a conf file, then, er, we're giving them a conf file [03:53]
gac410well with fcgi, the only script apache will use is foswiki.fcgi 401 is done with LocationMatch not FileMatch [03:53]
SvenDowideitzactly
same hack really
[03:53]
gac410But need to be careful because LocationMatch is not allowed all the time. I've had issues with it on occasion. [03:53]
SvenDowideity, the one i read about used rewrites or redirects
but even then i thought there was a small caveat
but as we use it in the conf, i can't see why not use it
mmm
where did the edit wikitext button go in trunk
trunk tinymce
[03:54]
gac410I was trying to make the nacg pluggable so that I could eventually build .htaccess files, and also build a working config when mod_rewrite was unavailable. [03:55]
SvenDowideitso the q will be, can i make it work
gads tho, i have a really cusomised magical conf
serves me right
[03:57]
gac410I also really wanted to have all LocalSite.cfg matching parameters detailed in comments in the config file - so users could have a clue. Our manual customizing instructions are really confusing
But I've run into if statement syntax limitations
and ran into dead end.
[03:59]
SvenDowideitSvenDowideit has no idea where the code is actually up to, nor how to test it :( [04:00]
gac410what code? NACG? [04:00]
SvenDowideity
i guess it'd need a www::mechanise that pertabates through the options
[04:00]
gac410My testing was creating config after config, and installing each one into an apache vhost and testing. [04:01]
SvenDowideitand then uses that conf in a fresh virtualhost
that then bashes on it using selenium
SvenDowideit tasks pharvey with that - you have 15mins
mmm, so 'not implemented' i can disable
[04:01]
gac410:P [04:03]
pharveydon't laugh, I began to write some code which did stuff along those lines :) [04:03]
gac410I wanted to eventually enable it and hopefully it's structured to let me add it in. [04:03]
SvenDowideitpharvey, i'm not laughing
gac410, its fine - i'm more worried about breaking something that already works
[04:03]
pharveyin the end I gave up because it was really difficult to test a selenium client against a particular customized Foswiki config [04:04]
SvenDowideithow? [04:04]
pharveyinvolves perl INC path magic and/or restarting apache from the tests or just nuking the current LocalSite.cfg in-place [04:04]
SvenDowideitSvenDowideit wants to know :) [04:04]
gac410I was happy to just manually test it. [04:04]
SvenDowideitgac410, more importantly - what limitations n IF did you hit
and can we work on that
pharvey, i was thinking more using one vhost per cfg
[04:04]
pharveygac410, this was part of the unicode effort. I found bugs that seemed to involve the whole FCGI.pm, CGI.pm, apache & Foswiki.cfg stack, so I wanted that automated.
I think I began writing something called Unit::WebServer
[04:05]
gac410I don't remember now. I think the bug is stll there. The comments in the front of the config - one of them gets munged up and I couldn't get past it and ran out of energy. [04:06]
SvenDowideitand you didn't github it for next year [04:06]
pharveySvenDowideit, did too [04:07]
gac410I was running out of quotes and escaped quotes and got totally lost in escaping. [04:07]
SvenDowideitpharvey, aw, blooming brilliant [04:07]
pharveyhttps://github.com/csirac2/UnitTestContrib/tree/Item1326
it uses an older version of NACG
I'm not sure it's at all useful
but might be an inspiration of what not to do
[04:07]
SvenDowideitwho knows, matters not
i hate losing code :)
i wonder how you remove the /bin/view if mod_rewrite is off
[04:08]
gac410I think there are ways with aliases, [04:10]
pharveyScriptAlias, but be careful of ordering [04:10]
SvenDowideitta - /me is just cleaning a little to get more familiar
setting login=tempalte as default too
[04:10]
gac410ah - and mod_actions is another way i found but never set it up to test. [04:11]
SvenDowideitmod_actions is disabled now, as 'not implemented' [04:11]
gac410right - it's there to remind me to work on it someday. [04:11]
SvenDowideit:)
SvenDowideit nukes some inline URLPARAMs that should use the Vars Set at the top
this really is a great eg of complexity that we should get a group of people reviewing for 'style'
[04:12]
pharveydamn - Insecure dependency in open while running with -T switch at /usr/local/src/wiki.trin.org.au/core/lib/Foswiki/Configure/Checkers/Htpasswd/FileName.pm line 29. again [04:14]
gac410my TML skills are not the best. [04:16]
SvenDowideitgac410, no-ones are [04:16]
pharveygac410: I have a reliable recipe to make it fail now :) Just change the htpasswd file location
happens for LockFileName too
[04:16]
SvenDowideitbecause we've never sat down and figured out best practice [04:16]
gac410I'll give it a try pharvey [04:16]
pharveypharvey is reminded on the brief rampage he underwent to at least make System web resemble something that everybody thought was less crap, resulting in TipsContrib being reverted (rightly so, but I was mad at the time) [04:17]
gac410some reverting is not done very diplomatically [04:18]
pharveyand at the same time, we probably don't do enough reverting [04:18]
gac410hm. So I changed password location from {DataDir}/.htpasswd to {DataDir}/sub/.htpasswd ... no taint errors. [04:19]
pharveybugger
I'm on perl v5.14.2 FWIW
[04:19]
gac4105.12.4 [04:19]
pharveyfunny thing is that the save actually does its job (LocalSite.cfg), so that's a bit terrifying from a "taint mode stops bad stuff" perspective [04:20]
SvenDowideitcrud.
turning off mod_rewrite doesn't get me a working apache conf
[04:21]
gac410perlbrew switch 5.14.2 / perl rewriteshebang ... .htpasswd -> data/Main/.htpasswd ... no fail.
no. esp. with NewApacheCG - mod_rewrite is required.
[04:21]
SvenDowideitoh. you have it as a tickbox
ok, i'll disable it bu leave it turned on
[04:22]
gac410Not implemented. [04:22]
SvenDowideitit says ' mod_rewrite - (Recommended if available) '
so i presumed :)
[04:22]
gac410It's a choice of mod_rewite, mod_alias or mod_actions. Should be radio buttons [04:23]
SvenDowideitah, ok, sweet [04:23]
gac410It's labeled experimental for a reason :-D [04:23]
SvenDowideity, and in the process i'm learning aboot it [04:23]
pharveyso, should I debug the taint error while it's still there, or do some $work
pharvey sends emails
[04:23]
SvenDowideit$work-u-do
mmm, nope, still got nasty docroot issues
lets see - i put in /acg2 as the foswiki url
maybe it needs to be /acg2/
nope, its still toast :(
dammit, i _have_ this working in the rpms
i wonder if i'm just killing myself because i have another non vhost setup tot he same files
[04:24]
gac410AH HA..... Got the Taint error! [04:29]
pharveyit's a sneaky bugger 'innit :) [04:29]
gac410I wasn't on trunk. And I was changin th path not the filename.
Nope - path change fails as well. I't s atrunk thing. Works okay on rlease branch
So why is the field tainted on trunk
[04:30]
pharveyI'm trying to get a stacktrace
I also reproduced CDot's Meta.pm ASSERT fail, I think, still working on that one too.
adding ASSERTS makes the taint issue go away?!
oh, never mind. I'm just mentally challenged.
[04:31]
SvenDowideitoh, mmm
i can't even get the acg2 working as is
the only wrinkle i have is that my foswiki is outside the docroot
oooooh
mmm, bizzare.
no, what i'm doing that is broken
is that i want the script url to bear no resemblance to the real on disk paths
[04:39]
gac410pharvey - the reason it fails on trunk is CDot added code to create the .htpasswd file if it doesn't exist. release11 doesn't try to create the file.
According to Scalar::Util::tainted() it's tainted when read from the config.
No idea why though. as checkers only run on load - and the config is loaded from the file, never from the url.
[04:44]
GithubBot[foswiki] foswiki pushed 1 new commit to master: http://git.io/u2T9wQ
[foswiki/master] Item10942: don't concat undef $result string - PaulAlexander
[04:47]
***GithubBot has left [04:47]
FoswikiBothttp://foswiki.org/Tasks/Item10942 [ Item10942: Nested attributes feature: metadata handling ] [04:47]
gac410my $f = $Foswiki::cfg{Htpasswd}{FileName}; print STDERR "1 Tainted" if Scalar::Util::tainted($f); [04:47]
SvenDowideitsoacg2 should work without script alias and alias? [04:47]
gac410yes. the rewrite rules serve that purpose [04:48]
SvenDowideitok, which looks like its not working [04:48]
gac410works fine here. [04:48]
SvenDowideitfor the setup i have
where the foswiki files are not in the docroot
basically, it keeps whining at me that /var/www/foswiki does not exist
and thus doesn't actually get to the rewrites
[04:48]
gac410The key is the H=cgi-script on the rewrite rule is equivalent to the ScriptAlias [04:49]
SvenDowideiti'm off to see what i did to make the rpm one work
cos i was sure it did 'just work' :/
i added re-write debugging
and the rewrite wasn't triggered
so i'm rather puzzled
[04:50]
gac410Why oh why is $Foswiki::cfg{Htpasswd}{FileName} tainted.
SvenDowideit: If you've moved files around, pay careful attention to the -f tests in the RewiriteCond statements.
[04:51]
SvenDowideitmoved?
the files have always been in one place
and i entered that into the generator
[04:55]
gac410okay - just trying to think of things that might cause regexes not to work. [04:57]
SvenDowideitso the core error i get from apache is:
[Wed Feb 22 14:54:44 2012] [error] [client 127.0.0.1] File does not exist: /var/www/foswiki
which is frustrating, considering that the foswiki is at /var/lib/foswiki
[04:57]
gac410and /var/www/foswiki doesn't show in the acg2 file
pharvey - so a blind untaint in FileName fixes the issue. But I have no idea why it's needed ($f) = $f =~ m/^(.*)$/;
[04:58]
SvenDowideitcorrect [04:59]
gac410don't know Sven ,.. Are you sure apache actually read your config. I've spend hours debuging a config file when the vhost was resolving to a difffernt config :-) [05:00]
SvenDowideityup, something like that, lookslikei lost motivation battling something that worked all morning
so the old scriptalias and alias conf is a little less delicate :/
as i just replaced a working conf with a similar looking acg2 one
and get bubkus
[05:01]
gac410I don't think it's more delicate. As with any config, the paths in the config have to match the file system. [05:04]
SvenDowideiti'm pretty sure taht the fs matches
in fact, all those bits _do_ match
but some other conf is getting there first
[05:04]
gac410That you turned on rewrite debug, and get no messages, suggests that apache is not using that config at all. [05:05]
SvenDowideitwhereas the old conf worked
exactly
[05:05]
gac410So that would be the vhost statement everything else depends on it. [05:05]
SvenDowideitwhereas the one that was working was from the old generator
i don't use vhost
i have that off in everything
as its useless to me here
this is going to send me mental, and then i'll find the one line that makes it work :(
[05:05]
gac410okay - so if the config file is a NOP and rewrite rules log nothing ... did you include the right filename? [05:06]
SvenDowideitinclude?
apache does includeDirs
[05:07]
gac410Some distributions do it differeent ways - I don't know. just thinking of things.
add a syntax error to the config.
I can't imagine how mod_rewuite logLevel wouldn't log.
it should log that nothing matches
[05:07]
SvenDowideitbugger. add syntax error, and apache dies :(
that woulda been an excellent stuffup :)
[05:08]
gac410gac410 running out of ideas. [05:09]
SvenDowideitmy thething is that re-write does not get run [05:10]
gac410The new config is not all that comlex. It just replaces ScriptAlias with H= operand. ScriptAlias is not allowed in .htaccess files, and I was trying to make ACG generate both. [05:10]
SvenDowideitbecause apache already says 'file does not exist /var/www/foswiki'
so its got zero file location matches
what i can't work out, is why this exact same (i copied it) conf works on centos
[05:10]
gac410something is ringing a bell that I've actually run into that issue. [05:11]
SvenDowideiti think its insisting that DOCROOT/URL exists
and you probly have to use LocationMatch or something to give it one
[05:12]
gac410I can't imagine why. What is your DocumentRoot set to [05:13]
SvenDowideitwhatever debian says (/var/www) [05:13]
pharveypharvey always sets DocumentRoot to somewhere bogus [05:13]
SvenDowideithttp://foswiki.org/Support/NewApacheConfigGenerator?rewrite=on;actions=;alias=;htaccess=apache;hostname=localhost;althost=;vhost=;port=;dir=/var/lib/foswiki;symlink=;pathurl=/foswiki;shorterurls=enabled;engine=CGI;fastcgimodule=fastcgi;apver=2;allowconf=;requireconf=;reqandor=and;loginmanager=Template;htpath=;errordocument=UserRegistration;errorcustom=;blockmodphp=on;blockpubhtml=on;blocktrashpub=on;controlattach=on;compressjscss=
on;expireshdr=on;etags=on;blockscripts=;blockspiders=on;foswikiversion=1.1;blockgsasess=on
/var/www is pretty bogus :)
[05:14]
pharveydunno why. I think in the 90s, it was a fun way to ensure busted ACLs wouldn't reveal anything
I've had RewriteRule 'scoping' issues before
crazy++
rewriteloglevel 9 :)
[05:14]
SvenDowideityes, i have that
but i get nothign
because apache doesn't get to the rewrite rules
creating /var/www/foswiki just means it whines about the next level
[05:15]
gac410I don't understand where it
gets that from in the first place.
[05:16]
SvenDowideitsimple - DOCROOT+URL [05:16]
gac410makes no sense. [05:16]
SvenDowideitits one of the first tests it does - and i can't recal the magic to tell it to nick off [05:16]
gac410Don't you need a valid DocumentRoot so the pub file serving has a valid starting location? [05:18]
SvenDowideitSvenDowideit hasn't need one before [05:19]
gac410Apache docs says - DocumentRoot used Unless matched by a directive like Alias, [05:19]
SvenDowideitwhich was what i expected
and why the old acg has an alias
[05:20]
gac410I've always set DocumentRoot to the root above pub - which is why I never need an alias. [05:20]
SvenDowideitbut the new one doesn't
so your conf's are very specific :(
SvenDowideit goes mad (as it works on centos, dispite the foswiki being outside the docroot
[05:20]
gac410yeah. That's the point of entering paths into the NACG [05:21]
SvenDowideityou can't set a second docroot [05:21]
gac410If DocumentRoot is not set, then ServerRoot is used. [05:21]
SvenDowideitand its not like NACG ast docroot
so, um
nacg does not set docroot or serverroot
or is non-vhost also not implemented?
[05:21]
gac410It sets DocumentRoot [05:22]
SvenDowideitnope [05:22]
gac410I've never tried it without a vhost. [05:22]
SvenDowideitargh!
when why make non-vhost an option, and have it off by default!
SvenDowideit shoots self for trying to debug it for the last hours :(
Generate vhost: optional
[05:22]
gac410It's cloned from the old ACG - You are the first person that has tried it in a non-vhost environment. I don' t have the systems available to test it.
Every server I've ever touched in the last 5 years has used vhosts.
[05:24]
SvenDowideitplease, disable stuff you've not tested (i'm going through and disabling the ones i find atm [05:24]
gac410NO.... It will never get tested. The damn thing is expermiental and asks for feedback. [05:24]
SvenDowideitcos its really really hard to try using stuff where you don't know what hasn't been cone
it can't be tested if its not been implemented :)
SvenDowideit realises this discussion sounds like the '
[05:24]
gac410It HAS been implemented. [05:25]
SvenDowideit'whats in trunk' discussion but reverse
<gac410> I've never tried it without a vhost.
~~ notimplemented
[05:25]
gac410It does EXACTLY what the ACG does. I didn't know 1) That servers ran with bogus document roots, or 2) that Alias was reqruied [05:25]
SvenDowideitwhich is why i rarely set to merge to core in time
no
ACG does somethign different
as it sets alias
which allows apache to work
[05:26]
gac410Right. So add some feedback and a note - [05:27]
SvenDowideitok, so, i don't know your assumptions, just like the RM can't know the assumptions made by a coder [05:27]
gac410um. Can I be any MORE CLEAR:
DRAFT DOCUMENT This is an early version of a new ApacheConfigGenerator. it's not ready for use yet, but feedback welcomed!
[05:27]
SvenDowideitbut i kept being told that acg wasn't ready, so i didn't try, now that i have, i can't use it becuase i have no idea how much more i have to use it
yes, you can
just like i can
[05:28]
pharveygac410: I have confirmed that $f is tainted here: my $f = $Foswiki::cfg{Htpasswd}{FileName}; - madness [05:28]
gac410yeah same here. [05:28]
SvenDowideitwe're not communicating our assumptions, so we're having this discussion :/ [05:28]
gac410The very first line of the topic says it is not ready for use.
IT IS NOT READY FOR USE
[05:29]
SvenDowideityes, but it does not tell me what i can expect to test its use for
which is why until today, when i wanted to do q __QUICK__ demo of an idea
i did not try to use it
but __QUICK__ has resulted in us shouting at each other because i was trying to use it to try out an idea
SvenDowideit needs to find a quick success to have in the last 30mins of his day, else the rest of my week will be a horrid shit.
[05:29]
gac410I don't know what I don't know. pharvey has tested. I've used it extensively. But not in your particular configuration. I didn't know that Alias was a substitution for DocumentRoot, Sorry I ruined your day. Time to sleep. [05:32]
***gac410 has left [05:32]
..... (idle for 20mn)
pharveyTemiV found a brilliant bug with INCLUDE{warn="custom text"} - it only works for topic-missing warnings. [05:52]
............... (idle for 1h12mn)
***ChanServ sets mode: +o MichaelDaum [07:04]
.......................... (idle for 2h9mn)
CDotFoswikiBot: seen pharvey [09:13]
FoswikiBotCDot: pharvey was last seen in #foswiki 3 hours 21 mins ago saying "TemiV found a brilliant bug with INCLUDE{warn="custom text"} - it only works for topic-missing warnings.". [09:13]
......... (idle for 40mn)
MichaelDaumWikiRingBot, seen paul [09:53]
WikiRingBotWikiRingBot hasn't seen paul yet [09:53]
MichaelDaumWikiRingBot, seen pharvey [09:53]
WikiRingBotWikiRingBot has last seen pharvey 22 Feb 2012 - 05:52 GMT on #foswiki
WikiRingBot has last seen pharvey_ 20 Feb 2012 - 05:04 GMT on #foswiki
[09:53]
MichaelDaumCDot, got a few spare brain cycles?
I am currently rewriting the page cache
now, storing and maintaining the dependency graph works reliably and fast using DBI
this is tested successfully against a mysql and sqlite
all former Foswiki::Cache implementations will go away
[09:54]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Cache [09:56]
MichaelDaumproblem is: can we afford to raise the technical requirement for page caching to having these things installed?
is even sqlite too much?
given that page cache is off by default
CDot, what do you think?
[09:57]
CDotas an unconditional requirement I would say yes, it's too much
but as a conditional requirement - it works even without it - then fine
[10:01]
MichaelDaumthe requirement is: if you want page caching, then you need sqlite or mysql or whatever sql database is available.
the other option is to write a Foswiki::PageCache::BDB ... which certainly won't perform anyway
[10:03]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache::BDB [10:04]
MichaelDaumeven BerkeleyDB isnt installed on lots of systems
which sort of renders the point mood
[10:04]
CDotmoot. Yes, I don't have a problem with requiring "extras" to enable something optional, BUT performance of a non-cached system should not be compromised by the changes [10:10]
MichaelDaumit can't. because, when there _is_ no $session->{cache} subsystem, then there's no cpu spent inside it [10:12]
CDotthen I think you are on to a winner :-) [10:22]
MichaelDaumI will still make Foswiki::PageCache a virtual classs, even though there's only one impl Foswiki::PageCache::DBI ... just in case somebody figures out how to fire dependencies into a valgrind rev proxy [10:23]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache::DBI [10:23]
CDotgood, I was going to ask [10:25]
.......... (idle for 47mn)
MichaelDaumf.o dead? [11:12]
Babarisitdownorjustforme?
works for me :)
[11:16]
MichaelDaumya, the lady needs her time. [11:17]
ModAcOstHi, is there a way to make www.mywiki display www.mywiki/myWeb/WebHome instead of www.mywiki/Main/WebHome, or do I have to mess around with rewrites? [11:18]
.... (idle for 15mn)
SvenDowideitModAcOst HomePagePlugin [11:33]
BabarRewrite / redirects
it's much easier than anything else
[11:33]
SvenDowideitactually, HomePagePlugin is easier still [11:34]
ModAcOstRedirect does not work unfortunately [11:34]
SvenDowideitmod_rewrite is a simple line in your apache conf / .htaccess
HomePagePlugin is one enable and one cfg
[11:34]
ModAcOstOk, thx, I will try the rewrite [11:35]
SvenDowideitand iirc does not cause a second request either - as bin/view is detected and then web&topic are set [11:35]
ModAcOstOk, I'll have a look at HomePagePlugin first^^ [11:35]
SvenDowideithomepageplugin also adds /bin/view can == the users Main.UsernamePage
which is cuter'n heck
actually, the more i think about it, using HomePagePlugin should add no extra workload to the server too, so its a pretty good win - all i need is more docco and unit tests, and er BOOM :)
[11:35]
ModAcOstHomePagePlugin works perfectly, thx [11:43]
SvenDowideitsweet :) nother happy customer [11:43]
pharveyboo!
MichaelDaum: how do you expire objects in the DBI impl? Is there a cron job which runs periodically ?
Hi CDot, on Item11528 I guess it's just annoying to get a warning on a tab but then not be able to see that warning. Not sure what the solution should be... I guess the hiding should also neuter the warnings
[11:46]
FoswikiBothttp://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ] [11:49]
MichaelDaumpharvey, no cronjob. it just rejects to deliver a page that is expired. [11:50]
pharveyI mean, how do we stop endlessly growing DBs [11:50]
MichaelDaumthere isnt endless content to render pages for
anyway, I've added more docu to the PageCache pod to make clear how pages are purged and when
[11:51]
pharveyok [11:52]
MichaelDaumit is more of a problem of too few pages remaining in the cache when some other topic is saved.
so a much more helpful cronjob would be to warm up the cache over night
[11:53]
pharveyok, so expired objects are removed when Foswiki says they're invalid [11:54]
MichaelDaumpharvey, spot on. thats why there is a dependency graph [11:54]
pharveyI thought there might be a mark & sweep impl. But never mind :) [11:55]
MichaelDaumthere are some single topics that _all_ cache entries depend on: for instance SitePreferences.
so editing this one topic will flush all of the page cache
[11:56]
pharveyI have been curious as to whether SvenDowideit's "ACL Profiles" pattern could be used to enable re-use of cache objects between several users [11:56]
MichaelDaumI leave that for others [11:57]
pharveyyeah, it sounds hard :)
I'm sure DBI backend will be good enough
[11:57]
SvenDowideitit won't be hard to implement, but it'll be hard to prove.
and its really painful to use the ACL profile pattern on disk
i tried twice, and it really didn't work out
same reason as real random access is needed for cache
[11:57]
pharveydidn't work performance-wise? [11:58]
SvenDowideitrandom - as in matix wise
its more that a filesystem is crap for a 4 dimentional matrix
[11:58]
pharveyfilesystems are crap for things that aren't trees [11:59]
SvenDowideitand pagecache is ndimensional kinda [11:59]
MichaelDaumy [11:59]
SvenDowideitthough i guess i could use the TagPlugin arch for it
but even so its ouchy
it'd be nice to move the pagecache into a contrib tho
so that the DEPENDENCIES are clearer
[11:59]
pharveymore important is adding QuerySearch expressions as dependencies. Then, on save, new topics which match could fire that dependency. MichaelDaum, does that sound hard in your new PageCache?
(more important than re-using cache objects, I mean)
[12:00]
SvenDowideithehe - what you want, is in DB triggers
and er that means generated queries
and that means.... >:}
[12:01]
pharveyI guess each [12:01]
SvenDowideitMichaelDaum are you doing this in 1.1? [12:02]
pharveyQuerySearch dependency also needs the web/topic scope set up by an SEARCH/etc. [12:02]
MichaelDaumpharvey, that's what db engies do. they cache results for the same query.
however, the result set doesnt only depend on the query. it foremost depends on the data changing.
SvenDowideit, nope trunk only.
[12:02]
SvenDowideitok, just beware that store2 changes everything :/ [12:03]
MichaelDaumMichaelDaum hoping assuming trunk will be the source for 1.2.0 [12:03]
SvenDowideitthe listeners go away [12:03]
pharveyMichaelDaum, right, but a huge blocker to people turning on the old PageCache, apart from stability issues, is that new topics aren't invalidating the old cache, so you get stale pages [12:03]
SvenDowideitthey probly wont be in 1.2.0 for eg [12:03]
MichaelDaumpharvey, correct. and there's nothing you can do about, besides adding an expiry date to %SEARCH pages [12:04]
pharveyanyway, I don't expect you to solve everything, it's just reminding of it all :) [12:04]
SvenDowideityes, there is
its actually an easy solve - but not fast
[12:04]
pharveyIt seems we "just" need to add QuerySearch + scope as a dependency.. [12:04]
SvenDowideituntil you get to something like mongodb - where the re-query can be done in mongodb
problem is pharvey re-running all 10,000 queries
[12:04]
pharveybut the re-query happens on each topic save. The search set size is 1: the topic being saved [12:05]
SvenDowideitis slow as shit
and
is impossible
[12:05]
MichaelDaumpharvey, adding the query string isnt the missing bit to solve the dyn pages outdating [12:05]
SvenDowideitfor the same reason as you can't check for js exploit onsave
as you don't know that user's VARIABLES
you would have to re-eval all queries for all new topics just as a user requests it
actually, no
you would need to do a partial render of the view that is cached
re-running all queiries on the new topics (new since last cache)
thus you have to cache result sets and other things
[12:05]
MichaelDaumbasically, when queries where stored as part of the db, will it be able to maintain a result cache. [12:07]
pharveyMichaelDaum, I didn't envisage saving the query string exactly, but perhaps the parse tree (well, some normalized form anyway). Yes, I agree, I haven't thought through it all... but I recall thinking it wouldn't be *so* hard to make cache expiry at least a little smarter [12:07]
SvenDowideity - expire when a new topic matches the web= and topics= params [12:07]
MichaelDaummost sites do a CACHEEXPIRE = +5mins
and are done
[12:07]
SvenDowideitwhich is no good
for something like
Tasks/MyItems
if i goto a task from it, close it, then re-fresh MyItems
er, create a new one i mean
[12:08]
MichaelDaumsome stuff simply isnt cacheable [12:09]
SvenDowideitbut trivial to add a trigger for force expire if a new topic matches the web= and topic= touple [12:09]
pharveyI am concerned you guys are obsessing over the 1% case. 99% of SEARCHes, if they are %dynamic, uses RLPARAMs which are already dependencies tracked separately [12:09]
MichaelDaumeither you sacrifice cache correctness by working on an expiry base, or you dont cache at all [12:09]
SvenDowideitpharvey i think i've shown 3 different simple things we _can_ do [12:09]
MichaelDaumpharvey, correct. [12:09]
SvenDowideitbut micha is going for the serve old data for 5mins is ok crowd
which is the majority (i think)
one big optimisation that all search algos do
is based on the web= and topic= SEARCH param
[12:10]
MichaelDaumthere are two main features to tackle this problem in page cache: (1) add an expiry date thus sacrificing cache correctness for a specific time slot and (2) mark fragments of the page as uncacheable by surrounding it with <dirtyarea> ... </dirtyarea> tags [12:11]
SvenDowideitwhich is massivly faster than doing that in the search itselt
adding web= topic= matching for new topic creation would be a third
and a rather powerful one
[12:11]
MichaelDaumweb=topic= matching? whats dat. [12:12]
pharvey"all or nothing" assumes we can't track dependencies in fine enough detail. And that might be true for now, but not impossible [12:12]
SvenDowideitSEARCH{web="Something*" topic="Item*" [12:12]
pharveyanyway, I am just really happy to see MichaelDaum working on it :) [12:12]
SvenDowideitthe thing that reduces the set of topics that SEARCH then processes [12:13]
pharveySvenDowideit, something I was trying to refer to loosely as scope (for the QuerySearch). [12:13]
SvenDowideity
the key thing is that in core, I've used that aggressivly to speed up searches before the hard work is done
[12:13]
pharveyexcept in mongo, topic="Foo*" makes it heaps slower [12:14]
SvenDowideitthe cache can take advantage of it too
um, it shouldn't
it did'n't used to
[12:14]
pharveywhy? indexes are faster than regexes [12:14]
SvenDowideitas topic="Foo*" gets hoisted to a mongodb query [12:14]
pharveya mongodb *regex* [12:15]
SvenDowideitare you sure?
cos it shouldn't
[12:15]
pharveywhat should topic="Foo*" get hoisted as?
nothing that can use indexes
[12:15]
SvenDowideitsame as "name=~'Foo*'
actually it does
[12:15]
pharveyyeah, that's slow too
that's what I'm talking about
[12:15]
SvenDowideitspecifically as its a trailing *
i'm pretty sure that there is docco wrt indexes and testing only leading chars
[12:15]
pharveywhen we have a nested SEARCH over 60,000 topics, it's massively quicker to have another FormField rather than rely on the topic naming pattern [12:16]
SvenDowideitSvenDowideit has no idea about it :/
i do know i wrote code that made it a heffalump faster using topic=
[12:16]
pharveyI did one on Monday, in fact - we had 12345r vs 12345 topics. We saved ~2s (nested search) by using a formfield instead of topic="*r" [12:17]
SvenDowideiter, yes
*r != leading chars
mmmm, i hope i'm not confusing mongo with xapian
[12:17]
pharveyI think you might be [12:18]
SvenDowideitf*. [12:18]
pharvey:) [12:18]
SvenDowideitright, everyone drop mongodb and lets move to TSQL [12:18]
pharveyTSQL? [12:19]
SvenDowideitmicrosoft sql :) [12:19]
pharveyhuzzah! [12:19]
SvenDowideitnope
'Actually, it depends. If the query doesn't use an index, and must do a table scan, then it can certainly be expensive. If you're doing a 'starts with' regex query, then that can use an index'
[12:19]
pharveythis must be new [12:20]
SvenDowideitgod no [12:20]
pharveypharvey tries .explain() [12:20]
SvenDowideitFor simple prefix queries (also called rooted regexps) like /^prefix/, the database will use an index when available and appropriate (much like most SQL databases that use indexes for a LIKE 'prefix%' expression). This only works if you don't have i (case-insensitivity) in the flags.
i wonder if we introduced a regression
[12:21]
pharveythe case-insensitivity
by default SEARCH is case-insensitive
[12:23]
SvenDowideittopic= is case insensitive? [12:24]
pharveyso it would be throwing an /^Foo/i by default [12:24]
SvenDowideitno, it should not
as topic= is outside the query
[12:24]
pharveypharvey checks [12:24]
SvenDowideitI _think_ topic= is case sensitive
i sure hope so, but stuffed if i know
[12:25]
pharveyall I know is, using Field='Foo' makes the nested searches faster [12:25]
SvenDowideitwhich is a bug (i think)
mmm, maybe not
eq might, for good reason be faster than loading the regex engine :/
but 2s sounds like crud
you are still tracking the queries that you have to hand optimise right?
(me knows the answer to reality :)
[12:26]
pharveythe general pattern has always been to remove wildcard/regexes
once I do that, problems go away
the other bottleneck is sheer quantity of topics we draw from to render a page
[12:28]
SvenDowideitwhat scares me a little, is that for 'mongodb-scale' you have a mindbogglingly small number of documents
so regexs that do use the index, and topic= aught to be one
[12:31]
pharveyby nested search, I do mean a page which is executing on the order of 100-200 SEARCHes [12:31]
SvenDowideityou shouldn't really see a huge pain, as it should be using the index to avoid loading and considering the document
mmm, there's a fun-one
i never thought of that :/
if you have 100's of queries on a topic
then we shoulda architected this the other way round :(
[12:31]
pharveyhey, I never expected these to perform [12:33]
SvenDowideitmore like pagecache in fact [12:33]
pharveythe only reason we're doing them, is because they're borderline tolerable :) [12:33]
SvenDowideitdammit, its my fault for not having seen into your users future! [12:33]
pharveyand then I tell Paul.A to please use smaller pagesizes [12:33]
CDotpharvey: easier said than done. The warnings are generated - and counted - server side, but the hiding is client
BTW I wanted to ask; what are the HASH and META leaf types in the infix parser?
[12:33]
SvenDowideitCDot when you hide, count the warnings your hiding
and decrement
[12:34]
CDotCDot thinks they are badly out of place [12:34]
SvenDowideit>:} [12:34]
CDotSvenDowideit: you think? woo, too technical for me [12:34]
SvenDowideitCDot they are ways to get info [12:34]
pharveyIIRC that was to do with RestPlugin getting topics in/out? [12:34]
SvenDowideitand they are rather important from my very hazy memory [12:34]
CDotCDot is feeling particularly sarcastic today [12:35]
SvenDowideitoh - pharvey bingo
they were the begingins of showing you that query can be used for both read and write
and it works well
SvenDowideit has had a shite day getting hit by apache rpm and debian crap
given the time, i'll give up on the last things i hoped to acheive today (newsletter and a simple micro-app) and goto bed
niteall :)
[12:35]
pharveypharvey does the same
but I can't!
promised George I'd fix a unit test
[12:37]
CDotpharvey: ? it looks like it was in support of the query parser
it seems they are used as constants
CDot has abandoned changes to the query language to support non-constant indices. Too complicated for today.
[12:44]
GithubBot[foswiki] foswiki pushed 1 new commit to master: http://git.io/MeYi9g
[foswiki/master] Item11366: Avoid using Foswiki::Address in test - PaulHarvey
[12:48]
***GithubBot has left [12:48]
FoswikiBothttp://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ] http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address [12:48]
GithubBot[foswiki] foswiki pushed 1 new commit to Release01x01: http://git.io/bSb_xA
[foswiki/Release01x01] Item11366: Avoid using Foswiki::Address in test - PaulHarvey
[13:02]
***GithubBot has left [13:02]
...................... (idle for 1h45mn)
gac410Howdy CDot - did you catch my commits last night - had to revert your changes to the <base tag - it was breaking html validations. And somehow EnableHierarchalWebs got removed. [14:47]
..... (idle for 23mn)
MichaelDaum: with your new page cache, how do you invalidate when upgrading to a new foswiki or plugin that will change content because of template changes, code changes, etc? [15:10]
MichaelDaumchanges in template/... and lib/... aren't tracked, only content changes. so when upgrading, you will have to invalidate all of your page cache in case you think this is required using a http://...?refresh=all or the equivalent perl api. [15:12]
gac410okay - we need to be sure to get that into upgrade instructions, etc. I'm pretty sure we've had people run into that here with the old cache as well. [15:12]
MichaelDaumya, docu white spots [15:13]
gac410is the new cache coveed on a FeatureRequest for 1.2?
Or is it a 2.0 thing
[15:13]
MichaelDaumnot sure what your timing is for 1.2 ...
I will implement it on trunk only.
[15:14]
gac410gac410 trying to make sense of a gazillion accepted proposals sprinkled across 1.2/2.0 w/o any idea is for what will be in what or what has committed changes.
Problem is trying to figure out how to branch out a 1.2 without bringing along a mess of half-implemented proposals.
[15:15]
MichaelDaumbasically, the page cache rewrite is required part of a thorough bug fix [15:15]
gac410yes that I agree. the old cache has had it's problems. [15:15]
MichaelDaumdo you plan to release 1.2 from trunk?= [15:16]
gac410well - i'm not even sure if I'll be rm - if Lavr comes back. I get words of wisdom suggesting trunk is too "trunkish" so use 1.1. .as base, and others who are vehement that trunk must be base or it throws everything away. [15:17]
MichaelDaumthe major fixes are in the 1.1.x branch as well. so I guess it is safe to release more 1.1.x from there. [15:18]
gac410I'm concerned that we have a LOT of partially implemented stuff in trunk. If we branch from trunk we need to find/revert the incomplete stuff. If we branch from 1.1, we'll start with a more stable base but have to pull in changes and that will be complex. [15:18]
MichaelDaumI don't see much breakage on trunk, running it all day for my comp. [15:18]
gac410right - but it's all the corner cases for stuff that we don't test, but will break sites happy with 1.1
It's not that there is broken stuff - though I know there is. It's the unknown that scares me.
[15:19]
MichaelDaumthe only thing that makes me a bit nervous is "store2" ... with Sven saying "everything will be different"
store2 really sounds more like fowiki2
[15:19]
gac410yeah - thats off in github and will be applied to trunk once 1.2 is branched. It is massively different from what I gather. The API becomes more more strict. [15:20]
MichaelDaumfoswiki2 [15:20]
gac410If Sven got his way, ACLs will be enforced in store and the "ignore permissions" stuff in func will be ... ignored. [15:21]
MichaelDaumah okay so we branch 1.2 from trunk, just before store2 gets merged in? is that the plan? [15:21]
gac410If I listen to Sven, yes. I Blogged that we would target 1.1.5 for March 31 release, and branch 1.2 shortly after that. [15:22]
MichaelDaumblog.f.o? [15:22]
gac410no Sven told me that is not used. f.o/Blog [15:22]
MichaelDaumoutch
f.o/Blog is absolutely not on the radar of other people out there reading rss n stuff
I gave Arthur and Sven a thorough feedback on the foswiki.org/Blog with the baseline: by far not prod ready.
they are maintained manually
can create an account in a few secs ... want one?
[15:23]
gac410I guess so - though I think our left/right hands need to get coordinated or we'll look rather foolish. [15:27]
MichaelDaumyou should've receive an email about the registration by now [15:31]
gac410nothing yet, but it may take a while to make it through my greylist service. Thanks. [15:32]
MichaelDaumwe might need a board decision about the blog :( [15:32]
gac410yeah I was pondering that. And the trunk or 1.1. base for 1.2 branch might need a community vote. There are some very strong opinions. [15:33]
tsnfooHmm. I just copied my data into a fresh 1.1.4 install, and noticed that AutoViewTemplatePlugin doesn't work unless Override is on. [15:37]
gac410Do you have a VIEW_TEMPLATE set in a WebPreferences topic? [15:38]
tsnfooWhich is odd, because Override is off on my 1.1.2 instance, and there are no VIEW_TEMPLATE or EDIT_TEMPLATE settings to override.
gac410: I noticed this a couple weeks ago on my svn/trunk install, but it went way after fiddling with settings. This time I went setting by setting until I noticed this.
[15:38]
gac410I think we are using it on f.o with override off. checking now. [15:39]
tsnfooVery odd. Like 75% of my topics use auto view templates. [15:40]
gac410yup - debug and override disabled, mode "exists" [15:40]
tsnfooSame here. View templates aren't applied unless override = 1. I've restarted apache, diff'd my confs, etc.
I am running mod_fcgid though, which I'm not on my 1.1.2 install.
What does fw.o run?
[15:40]
gac410mod_fcgid We had one issue in the Support web with view templates not applied. but only on some forms. It was a Set VIEW_TEMPLATE in the WebPreferences [15:41]
tsnfooHmm. My googling hasn't brought that up yet. [15:42]
gac410as a bug? I didn't think it was - we just updated WebPreferences and things started working. [15:42]
tsnfooRight.
The only places I can find VIEW_TEMPLATE on my wiki is in the docs. Strange. I'll keep gripping and googling...
[15:43]
gac410I suspect it was a long-standing issue on f.o though because once we fixed the setting, some of the topics needed fxing. [15:44]
.... (idle for 15mn)
MichaelDaum: I'm registered - thanks. [15:59]
MichaelDaumyw [15:59]
.......... (idle for 46mn)
gac410Sure do wish we had a raw diff mode. SvenDowideit wtf did you do to NewApacheConfigGenerator. I can't get it to *not* generate a vhost, so i can't recreate / fix your problem. [16:45]
......... (idle for 42mn)
MichaelDaumwith the new page cache enabled cached pages are delivered in about 400-500ms [17:27]
gac410nice! [17:27]
MichaelDaumno matter how long it took to compute it first time [17:27]
gac410If you plan this for 1.2, then might want to commit to trunk. Hm... With t.f.o and f.o sharing the same file system will the cache be usable? I'm guessing not.
otherwise changes on one won't invalidate the cache on the other.
[17:28]
MichaelDaumnaw. pages look different on both domains.
thats a problem
pages are even cached properly using ajax-loaded topic sections using jquery.loader or InfiniteScrollContrib
that is: each fragment is cached separately
as it should
[17:29]
gac410cool [17:31]
MichaelDaumthere will be a new cache_ignore urlparam that allows the cache algorithm to ignore certain url params from generating separate page variants
for instance JQGrid adds an nd=random to each rest call
while this is fine to trick out browser-side caching, server-side caching can ignore these safely
[17:32]
gac410so jqgrid has to add a cache_ignore=nd or it's a config param to ignore nd? [17:33]
MichaelDaumI've added it to the gridconnector url in GRID.pm
so that it works for perl connectors as well as topic connectors
same trick might be required for some autocompletion thingy. though ui::autocomplete isn't a candidate of that kind
so this does speed up autocompletion like in solr's input box
[17:34]
gac410too bad ui::autocomplete isn't a candidate. The user search / fill-in stuff can be pretty slow. [17:39]
MichaelDaumthats because the Foswiki session object is too fat
takes too long to build it up again and again
yet still everything is low prio compared to RcsLite::readTo() using the bulk of the time
accordign to Devel::NYTProf
[17:39]
gac410Yeah - improving RcsLite needs to be a priority for 1.2 ... (or even 1.1.5 would be nice) [17:41]
.................. (idle for 1h25mn)
***jurov has left "User accidentally the." [19:06]
........... (idle for 54mn)
ArthurClemensunbelievable how much trivia information on tv shows is on wikipedia
you might get the impression that they have information on everything
[20:00]
......... (idle for 40mn)
Lynnwoodanyone around experienced with LdapContrib?
i'm having some trouble getting it running... I think i'm close.
using command line tool ldapsearch, i can get queries to work, but entering the same values in the LdapContrib configure settings don't seem to work.
i keep getting errors in apache log referring to "10: 0000202B: RefErr: DSID-031007EF" which is apparently a very general error
[20:40]
.... (idle for 15mn)
gac410Try http://forums.sdn.sap.com/thread.jspa?threadID=1811516 - suggests a solution [20:57]
.... (idle for 18mn)
SvenDowideit<gac410> Sure do wish we had a raw diff mode - there is something
and yes, we're using f.o/Blog
the wordpress blog is useless for what I'm trying to acheive, as its limited to a small set of users
http://www.modernperlbooks.com/mt/2012/02/upgrading-user-password-hashes-in-place.html
seems like your pwd stuff, but dumber
[21:15]
Lynnwoodgac410: thanks. i didn't see your comment til just now (too many windows...) [21:17]
SvenDowideitnACG2 - mmm, i was using it for non-vhost the entire time
but the last thing i did, was to default to vhost ON
[21:17]
Lynnwoodbut i got back that problem and am on to the next ;-)
it appears that the LDAP cache is being created ok, but now when i go to log in, i get a blank page
[21:17]
SvenDowideitoh, gac410 the other thing - we did talk to cdot wrt 1.2.0 from trunk
and he said it was a feeling - and that he had no facts to back it up
so isn't actually arguing for it
[21:20]
BabarSvenDowideit: it's chromatic. Of course it's dumb. [21:34]
SvenDowideit:p [21:34]
BabarAt the AAAS meeting's discussion, CERN's director of research, Sergio Bertolucci, placed his bet on what the results would be: "I have difficulty to believe it, because nothing in Italy arrives ahead of time." -- Lol [21:37]
SvenDowideithehe - nice [21:37]
f.o/Blog is basically the feed that i use to create the weekly newletter i started last week
my pov is that f.o/Blog has been stymied by 'its not finished & isn't a ''real'' blog for years, its better to use it and to use that fact to push its improvement
we had the same bike shedding years ago for tracking tasks
[21:45]
gac410Re: trunk and CDot - he is not "comfortable". Neither am I. I look at 1000's of commits and figure it's easier to cherry pick the asserted positive, than to ghost-hunt for the maybe negative. 1.2 should still end up with the same content, but one is IMO more assured "pure" [21:48]
SvenDowideitwhereas i recon that cherry-picking will be more detective work, uless you basically make 1.2 so simple that it doesn't gain us much
but thats not an argument we need to have againagina
and when paul and i asked him, he wasn't _that_ uncomfortable
[21:49]
gac410I'm not strongly against trunk I would prefer that we find a solution where the core developers are comfortable that the contents are clean, predictable, If experienced people around well before me are uncomfortable, who am I to dispute that. [21:49]
SvenDowideitjust unsure
where unsure == i didn't actually look into it
nooooooo excersise just uses up more energy!
think of the environment!!!
[21:49]
Babarhey, couldn't we always release from trunk, and just branch off features and merge them back when they're ready? [21:53]
SvenDowideityes, but we don't like 'ready'
as it delays the time that others start to think about them
merge / release orphan
but thats what we're doing for things that break for longer..
i'm pretty sure branch per feature will be used more when we have git as our main repo... :)
[21:54]
Babaroh, who broke git.foswiki.org?
Babar hides
[21:58]
maybe putting everything there is a bit of a waste :)
let's see if it finished if I grant it enough time
[22:07]
it does [22:13]
.... (idle for 17mn)
gac410SvenDowideit: Reading about HyperDex - mongo sounds so old fashioned. http://hyperdex.org/performance/ [22:30]
ArthurClemensbut you need many servers it seems
or are they metaphorical servers?
hyperspace hashing sounds like Self Organizing Maps
[22:34]
pharveyAs somebody who has been running trunk in production for over a year, I find it profoundly disturbing that people have such a negative opinion on the state of it
I am using Release01x01 JSCalendarContrib
I need to go over my git branch to see what else I've hotfixed/reverted, but that's the only I think I've actively "reverted"
[22:37]
gac410pharvey - JSCalendarContrib is fine - unless you us a non-standard date format that foswikiDate doesn't understand. Then all your calenders will magically revert to 1/1/1970 when touched.
And it's not "fixable" because JSCalendarContrib stores the formatted dates in the meta, and then reads them back with foswikiDate.
[22:43]
pharveyI had a different problem (since fixed) resulting in lots of 1970's for a different reason, but haven't up'd [22:44]
gac410The "right fix" is to convert the Meta date to a serialized date like epoch or the like, which any db would do. separating the date from the display.
The bugs only showed up late in 1.1.4 beta/RC, and Arthur and I agreed that reverting was the right solution for now.
If your configured date format is understood by foswikiDate code, you'll never see the issue.
I don't believe I've said trunk was bad or that I had a negative opinion. It's *unknown* to me. if I RM, the question is the best way to make it known and gain confidence.
Cherry-picking individual commits - probably not. Reverting indivitual commits - probably not as well. Both would take forever.
[22:45]
ArthurClemensit looks like the posts in foswiki.org/Blog are picked up well by Google: site:foswiki.org/Blog [22:52]
gac410Can we wrap up groups of modules into their feature-sets. Like the Query and Search changes. Store, etc. verify against the "FeatureProposal"s and investigate differences?
ArthurClemens: I think Micha's concern was news readers and syndication sources. With the amount of crawler activity from googlebot on f.o, I'd be surprised if it's not indexed.
[22:52]
ArthurClemensI've looked at MD's PingBackPlugin, but it is not ported from twiki
otherwise we could use it on f.o.
[22:54]
gac410So pharvey, I'm assuming we'll branch from trunk now, but I *still* want a process to scrub what's there. and not just take it for granted that because it works for you it works for everyone. [22:55]
ArthurClemensgac410: _The "right fix" is to convert the Meta date to a serialized date like epoch or the like_ see my new proposal [22:55]
gac410Or maybe I'm just being too old fashioned
ArthurClemens: recent? There was one to add a new Meta type for dates/timestamps ?
[22:56]
ArthurClemenshttp://foswiki.org/Development/CreateISODateFormat
added today
[22:59]
gac410We've got waaay too many proposals. with a grand total of 1 "UnderConstruction" 5 MergedToCore
ArthurClemens: Would you actually *store* them as ISODate, or store them serialized.
[22:59]
ArthurClemensas isodate
to keep the timezone data
otherwise we need to come up with our own serialized format like epoch+timezone
[23:01]
gac410I'm really uncomfortable with that. What do databases do? mysql, mongo, anything else that stores dates?
I suspect they probably store a serialzed GMT + original TZ
But that's only a guess - having not looked at that at all.
[23:03]
ArthurClemensuncomfortable storing iso dates?
I believe DBs are optimized handling iso dates
[23:05]
gac410If you sort a query on the ISODate, will it display in a normalized-to-GMT order, or will it display with the TZ and need the user to decide that 0800 EDT is before or after 0800 CET [23:08]
Babarpharvey: (Sven too) http://git.foswiki.org/ nice, isn't it? :) [23:10]
gac410If you enter date/time in your timezone with your current view of daylight savings, and I view that topic, I want to see the dates in MY timezone with my current daylight savings offset. [23:10]
pharveyIf the normalization is formally defined as ISO, and it can be known that the datatype of that field is always an ISO date, then you'd think a DB Cache back-end like Mongo or DBIStore or VDBIStore can choose to store an atomised/optimised version of it [23:10]
ArthurClemensit looks like mongodb uses a 64 bit representation of the date [23:11]
gac410Whatever we do, I just hate the thought of having to look at the displayed dates and then running a timezone converter to figure out when it is relative to "now". That should just be magic. [23:11]
ArthurClemensisodate is storage. how it is rendered is another thing [23:12]
pharveygac410, I agree, but I'm wondering what kind of overhaul we'd have to do on the currently typeless QuerySearch + meta code
Babar: cool :)
but isn't gitweb slowish?
well, seems to work anyway
[23:12]
Babarit's slowish because it has shitloads of projects to parse :) [23:16]
pharveyis it running as plain-old-cgi? Or... fcgid won't help
gac410: we'd have to store something like %META:FIELD{name="Date" value="2012-03-01T13:00:00Z" offset="+10:00"}% I guess - which means extending ArthurClemens proposal perhaps more than he wanted (but perhaps I can help)
[23:16]
Babarit's running plain old crappy cgi for now
I'll configure fcgid when I feel like it :)
[23:18]
pharveywhen can I push there? :P [23:18]
Babarwhen we'll ditch Subversion for git? [23:21]
ArthurClemenswell in that case you better store the date in epoch seconds [23:22]
pharveyI'm not sure epoch seconds is all that valuable. I mean, if the underlying store wants to store it that way, then sure, but... it would needlessly make Topic.txt less human-readable, IMHO [23:22]
ArthurClemensbut why store offset in a separate attribute? [23:23]
pharveysorting behaviour
ArthurClemens, I still think your proposal is useful as-is.
I'm just exploring possibilities.
I guess we're trying to decide if we need to add new SEARCH-side magic to get proper sorting behaviour of multi-TZ date data, or alternatively, store these dates in a way that makes it easier for SEARCH.
But just storing isodate strings is a huge improvement already.
[23:23]
gac410yeah the sorting is probably the big thing - I agree that as long as it's stored in ISO format, display can happen later. I can't imagine the A:B sort compare that would have to deal with dates entered in different timezones and DST locations. [23:27]
pharveydoes DST come into it? If we already have the exact GMT offset [23:27]
gac410oh - okay - the +/-hh.mm takes into account local user's offsets ... nevermind. [23:28]
Babarwhy would anyone want to store a date locally in anything else than epoch? [23:28]
pharveybecause users consider that detail very important
they don't like that kind of information being thrown away
[23:29]
Babarwhat information?! [23:30]
gac410human readability of the meta. It has to be epoch and offset entered, not just epoch. [23:30]
Babara date is a date [23:30]
pharveya date is a date if you throw away its context, yes [23:30]
BabarI want to display it in a format of my choosing, using the timezone I want
but it should be stored as a dat
*date
[23:31]
pharveyit *will* be stored as a date
why do you think DBs preserve TZ info?
[23:31]
gac410If we had .. and honored the user's preferred timezone for all display, would we need to store the original offset? Or is saving the offset a requirement because we don't have local interpreted dates. [23:31]
pharveythis has nothing to do with display
it has everything to do with storing the original data that the user entered
[23:32]
Babarwhat?
now you're talking non-sense
[23:32]
pharveyWhat?
You want to throw away data? That's up to you. We shouldn't force that onto everyone!
[23:32]
Babarwhy would anybody care that I enterred 2011-02-22 23:32:21+01:00
or whatever epoch that is?
[23:33]
pharveyBecause it adds context to the information! [23:33]
Babaror maybe it's because I use JSCalendarContrib on a browser which was setup as GMT+1 [23:34]
pharveyand that's important
(potentially)
[23:34]
Babarwhat's important is that the server stores the date properly [23:35]
pharveyWe use TZ offset all the time when cleaning and validating data [23:35]
Babarbut I still don't see why it's important [23:35]
pharveyyou don't have to; the fact is it may *or may not*, but that's not to up to us to decide
when a datestamp is captured by some datalogging device, seeing the TZ offset align with the stated capture locale is a great signal
[23:36]
SvenDowideitfact is, I, and others i've worked with _do_ want to keep what TZ the date was entered as
but it is separable from the date 'fact'
[23:36]
BabarOracle uses its own internal format to store dates. Date data is stored in fixed-length fields of seven bytes each, corresponding to century, year, month, day, hour, minute, and second. [23:37]
SvenDowideitand mixing the 2 into one datafield is shit. [23:37]
Babarfunnily enough, I don't see any mention of timezone in Oracle's documentation [23:37]
SvenDowideitas it makes sorting and querying very hard [23:37]
Babarso where did your claim that DB stores it come from? [23:37]
SvenDowideitwhereas storing it as a separate field makes sense - as its a 'date field enterer's tz' value [23:37]
Babarwhich is exactly my point Sven. [23:38]
SvenDowideitBabar, g.f.o - sweeeet
Babar, ya, oi was argeeing with you
[23:38]
pharveyBabar: Oracle certainly does store TZ if you use a datetime type that has it
Similarly, MySQL, Postgres also have datestame types with and without TZ
[23:38]
Babarthe "timezone" should be an attribute of the date, for review, display, whatever. But the internal storage should be unique, epoch or whatever [23:38]
SvenDowideitbasically, mashing the enterers tz into the datetime column makes all sorts of things shit.
BTW
[23:38]
ArthurClemensthe date can be stored like 20120301130000+01:00, which is easy to parseint [23:39]
SvenDowideitI wrote a show time in my TZ plugin in ... 2006
where it would take the TZ setting from your homepage
[23:39]
ArthurClemensif you want to throw away the tz [23:39]
SvenDowideitand apply it to any %TIME..
ArthurClemens, yes, but it makes querying & sorting on time and date way too hard
[23:39]
ArthurClemensusing parseInt, which you need to do anyway? [23:40]
SvenDowideitcompared to date=20120301130000, entererstz=+01:00 [23:40]
Babarpharvey: once again, that's BS. [23:40]
pharveyWhich part, oh wise Babar? [23:40]
SvenDowideitSvenDowideit pre-de-snides this conversation (oh the irony) [23:40]
Babarthe only time Oracle stores the timezone is when you use TIMESTAMP WITH TIME ZONE. but all the other DATETIME (so TIMESTAMP WITH LOCAL TIMEZONE, DATE and TIMESTAMP) are stored timezone-free [23:41]
pharveyHow does that conflict with anything I've said? [23:41]
ArthurClemensI will read the comments on the proposal in the morning
night
[23:42]
Babaroh, "that has it". Maybe I overlooked that part [23:42]
pharveyI won't start a war over keeping TZ with the isodate, I just want to defend the practice, because we rely on it here for a scientific data extensively
Ability to compensate for operator error is... important
but perhaps that can be isodate+timezone
[23:43]
Babarand again, I totally support the fact that we should store that somewhere. But I'm strongly against storing it *together* with the actual date
(just like Sven explained)
[23:46]
pharveyOk. So I will add a comment that isodate should store Z time by default, and an isodate+foo can do something different. [23:48]
Babarhttp://foswiki.org/Development/InGMTIMESupportTwoDigitWeek shall I say that this has been accepted by 14 days consensus, and finally commit my 14 days old patch? :) [23:48]
pharveyyup [23:50]
Babaroh no, crap. I forgot to code some unit tests... I'll do that tomorrow, before pushing my commit [23:52]
pharveyItem11506 [23:53]
FoswikiBothttp://foswiki.org/Tasks/Item11506 [ Item11506: Value of GTIME{"$week"} is not ISO8601 conform ] [23:53]
pharveyBabar: do you have time to add a time arg for GMTIME? Sucks it can only render time() >:) [23:54]
SvenDowideitso it isn't hard to separate the info [23:54]
pharveypharvey hides [23:54]
SvenDowideitSvenDowideit calls you all geeks. [23:55]
pharveyit isn't hard, but there might be some inertia against adding a new attribute to the META:FIELD schema just for isodate types, unless we call that attribute something lame like metavalue [23:56]
SvenDowideiti was thinking more like
field type date creates an implied enteredtz field
named the same as the date field plus TZ or something
cos it really is a separate datum - the datetime value (in gmtime) simplifies query&sort&display of time in local tz
and entered tz is important for... validation, record keeping and ?
(i add the ? in the hope you can list the others you know of)
SvenDowideit is busy simplifying DataForm creation and maintainence
[23:56]
pharveythe fact that humans don't all live at Greenwich? [23:59]
SvenDowideitthats a display issue [23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)