| ↑back Search ←Prev date (last date) Show only urls | (Click on time to select a line by its url) |
| Who | What | When |
|---|---|---|
| gac410 | cool - the very next line after CGI::a complains. <nop>s in tooltip are encoded as <nop>. ... and reverses it. I think my disable autoEscape will avoid that issue too. | [00:00] |
| pharvey | the bug title is correct: it's double encoded rather than once. | [00:00] |
| gac410 | okay - then the fix is very easy. I'll fix the unit tests and the code. | [00:00] |
| pharvey | awesome, thanks :)
no wonder I had trouble fixing this the first time round - doesn't help to write tests that are wrong :) | [00:00] |
| gac410 | :-)
I also couldn't run the tests on the release branch. They also depend upon Foswiki::Address | [00:02] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address | [00:03] |
| pharvey | d'oh. I had lib/Foswiki/Address.pm in the release branch of UnitTestContrib, but decided I'd better write tests without it.
gac410: my usage of Foswiki::Address was the beginning of my efforts to refactor link handling. As that work has stalled, I think it's bets to replace with Foswiki::Func::normalizeWebTopicName I can fix this up tonight if you leave the task WaitingFor me | [00:04] |
| gac410 | okay - well let me figure out what's going on with the link - 1 of 2 tests is passing.
gac410 begins to pull hair. Test was passing just fine. I removed my print statements - now it fails. :-( | [00:07] |
| pharvey | gac410: the tests have a TRACE mode which might useufl | [00:13] |
| gac410 | okay - I'm poking around at it. Something strange is going on. | [00:14] |
| pharvey | you've probably already seen that the two tests run through a list of test cases themselves | [00:14] |
| gac410 | FormattingTests::test_ampersand_querystring: seems to be small - self contained. | [00:15] |
| pharvey | oh, yeah | [00:16] |
| gac410 | pharvey - I think there is something wrong with that standalone test_ampersand_querystring. The rendered link looks fine, but it claims frag: undef, query: undef, addr: a> | [00:30] |
| pharvey | what does the TRACE output show? | [00:30] |
| gac410 | [[TestTopicFormatting?q=r&s=t]]
<a href="/TemporaryFormattingTestWebFormatting/TestTopicFormatting?q=r&s=t" class="foswikiCurrentTopicLink">TestTopicFormatting</a> frag: undef, query: undef, addr: a> So the & looks great. no double encoding. | [00:31] |
| pharvey | indeed
can you commit or pastbin your fix (or push it somewhere? :) | [00:31] |
| gac410 | I'll commit - it is very small - + my $aFlag = CGI::autoEscape(0);
my $link = CGI::a( %attrs, $text ); + CGI::autoEscape($aFlag); Clear and save the autoEscape flag, generate the link then restore the flag. | [00:32] |
| pharvey | ok, I have a few mins before my run finishes
gac410, have you modified the test? query: looks correct (still fails, but I'll fix that now) | [00:33] |
| gac410 | Yes - Didn't seem to matter what I did to the test. Tried it with or without & in the 1st or 2nd parameter. | [00:35] |
| pharvey | oh, wait. It passes; I had commented something out
gac410, with your changes, the tests all pass :) | [00:36] |
| SvenDowideit | mmm, i _might be getting ~800installations of foswiki in the last 7 day period via the debs?
thats insane! 836 836 0.03% http://fosiki.com/Foswiki_debian/pool/main/f/foswiki-ldapcontrib/ | [00:36] |
| gac410 | Excellent - I'm about to commit - forgot to turn off TRACE | [00:37] |
| SvenDowideit | oh, maybe thats how many times someone's running update to see if there is one
77 10.30% Feb/20/12 8:20 AM /Foswiki_debian/pool/main/f/foswiki/foswiki_1.1.3-2_all.deb more like 50 0.02% Feb/19/12 5:57 PM /Foswiki_debian/pool/main/f/foswiki-ldapcontrib/foswiki-ldapcontrib_120112-225_all.deb | [00:37] |
| pharvey | gac410: so I think the tests are written correctly after all, but they might be a little misleading. I wrote it so that each entry in the test data had to only be written once, so the test escapes & to derive the expected href="..." value automatically | [00:39] |
| gac410 | I've checked in my changes btw. | [00:40] |
| SvenDowideit | so 50-100 per week ~~ 250-500 debian based installs of foswiki per year? that would be well worth it :) | [00:40] |
| gac410 | but I still have 1 test failing. | [00:40] |
| pharvey | which one? | [00:40] |
| gac410 | FormattingTests::test_ampersand_querystring: | [00:41] |
| pharvey | it fails to fail? or ... what's the error?
pharvey getting passes here, after svn rebase we need to remove the expect_failure() FormattingTests::test_ampersand_querystring *** Unexpected pass | [00:41] |
| gac410 | It fails. Unit::TestCase::assert_str_equals('FormattingTests=HASH(0x87b5550)', 'TemporaryFormattingTestWebFormatting.TestTopicFormatting', 'TemporaryFormattingTestWebFormatting.a>', 'address mismatch checking [[TestTopicFormatting?q=r&s=t]]') called at /var/www/foswiki/trunk/core/test/unit/FormattingTests.pm line 1786 | [00:42] |
| pharvey | how utterly odd
are you on trunk? | [00:42] |
| gac410 | yup - no Foswiki::Address on release | [00:43] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address | [00:43] |
| pharvey | oh
it's expecting TestTopicFormatting, but getting 'a' as topic name can you rm -rf /path/to/data/Temporary* ? | [00:43] |
| gac410 | hm. strange. Now it just passed. | [00:44] |
| pharvey | not that this should affect it | [00:44] |
| gac410 | I did a clean and re-pseudo-install | [00:45] |
| pharvey | pharvey doesn't know how to explain it :/ | [00:46] |
| gac410 | but two other tests fail now - I rebuild LocalSite.cfg and HierarchicalWebs are disabled ... strange. | [00:46] |
| GithubBot | [foswiki] foswiki pushed 1 new commit to master: http://git.io/ZpLU6g
[foswiki/master] Item11366: Prevent double encoding. - GeorgeClark | [00:48] |
| *** | GithubBot has left | [00:48] |
| FoswikiBot | http://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ] | [00:48] |
| gac410 | wtf. EnableHierarchicalWebs is removed from Foswiki.spec | [00:55] |
| pharvey | wtf. AuthScripts too
wait... it's just not in the configure UI | [00:56] |
| gac410 | well EnableHierarchicalwebs is not in Foswiki.spec. | [00:57] |
| pharvey | ah, somebody made AuthScripts conditional on TemplateLogin | [00:57] |
| gac410 | yes. probably CDot. | [00:58] |
| pharvey | right, because we don't do redirects now
.... DISPLAY_IF looks js-tastic oh, never mind anyway, the problem is that if you use ApacheLogin, you get a warning on the Login tab on the AuthScripts but can't see it to fix it (or the warning message) | [00:58] |
| gac410 | and pharvey wonders why trunk scares gac410
:-D | [01:00] |
| SvenDowideit | yay, finally
http://fosiki.com/blog/2012/02/22/centos-yum-install-foswiki-and-debian-apt-get-install-foswiki/ | [01:02] |
| pharvey | scary implies in intractable or not worth the effort... I'd say trunk is tedious, not scary :P
SvenDowideit, w00t. | [01:02] |
| SvenDowideit | store2, _thats_ tedious
trunk is just ready for testing :) ok, so gac410 you're the most sane person here right now should i just copy&paste that to foswiki.org/Blog ? | [01:02] |
| gac410 | So CDot deleted hierarchical webs in Foswikirev:14005 | [01:03] |
| FoswikiBot | http://trac.foswiki.org/changeset/14005 [ Changeset 14005 – Foswiki ] | [01:03] |
| SvenDowideit | or is it too, er, self promoty | [01:03] |
| pharvey | I'm profiling some code and trying not to optimize our damn INCLUDE's habits of parseSection'ing each time there is an INCLUDE statement, but I think that wants readonly topicObjects to work properly | [01:03] |
| SvenDowideit | pharvey, correct-ish
gac410, so we have heirarchical webs _always on_? cos really, we should do that | [01:04] |
| gac410 | No - always off | [01:04] |
| SvenDowideit | oh oops
nice way to hurt pharvey :) | [01:04] |
| gac410 | Config option removed, so any tests for it are false. | [01:04] |
| pharvey | I remember for sure now: I couldn't figure out a nice way to cache the parsed section data across all instances of a topicObject | [01:04] |
| gac410 | I only discovered it because the unit tests expect fail because hierarchical webs are disabled. | [01:05] |
| SvenDowideit | i voted for Meta == the code api, the actual topic data moves into meta->{data}{text etc..} | [01:05] |
| pharvey | gac410: that explains why my trin setup is so broken right now (apart from all the other bugs I'm fighting, which are sadly impacting simple views of Main.WebHome | [01:06] |
| SvenDowideit | and then meta->{cache} couls contain anything derived
but, argh :) mmm, think i will just c&p my blog post to f.o/Blog | [01:06] |
| gac410 | please no comments regarding my sanity or lack thereof | [01:07] |
| pharvey | SvenDowideit, I'd suggest you wait just a tiny bit so somebody else can writeitup :P | [01:07] |
| SvenDowideit | mmm, but who would do so in the next 4 hours? | [01:07] |
| pharvey | ... or not, /me reads the post first | [01:07] |
| gac410 | So do I just put EnableHierarchicalWebs back into Foswiki.spec ... so the unit tests will work? or open a task so CDot can explain wth is going on. | [01:08] |
| pharvey | SvenDowideit, you need a grammar nazi to fix your post | [01:08] |
| SvenDowideit | pharvey, booooooooo | [01:08] |
| pharvey | I guess if it's in foswiki.org/Blog, that could be me :P | [01:08] |
| SvenDowideit | good point!
lets see how well wysiwyg de-re-htmls | [01:08] |
| pharvey | SvenDowideit: trunk's WYSIWYG is much better
(and I hope to back-port to Release01x01 for 1.1.5) okay, I just wanted to see if the post could be perceived as you promoting yourself, but it's a publicly available repo and the post very to-the-point, so I guess not gac410: can you re-open the task CDot mangled it under? pharvey wonders what the task was | [01:10] |
| SvenDowideit | i'll bet it will be considered self promotion by someone - because it is - to the degree of 'this is wot i done' but thats kinda what im hoping people will use theblog for | [01:12] |
| gac410 | Item11528 | [01:12] |
| FoswikiBot | http://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ] | [01:12] |
| SvenDowideit | http://foswiki.org/Blog/Centos_yum_install_foswiki_and_Debian_apt-get_install_foswiki
edit-bombs away now to prod gac410's post like i promised :) | [01:12] |
| gac410 | SvenDowideit: Any ideas why ConfigureTests are bombing out for trunk on your nightly build system? | [01:15] |
| SvenDowideit | none at all atm :/
i wonder if i cna just make a 1.1.4 deb without effort :/ | [01:15] |
| gac410 | Changes in perl modules? It passes here fine. ('cept for the HierarchichalWebs issue now. :-( ) | [01:16] |
| SvenDowideit | 11am - overcast, need to mow lawn :(
lets see if i can figure out where they're running i.need.better.sysadmin. | [01:16] |
| gac410 | :P AccessControlTests failing due to elimination of redirect in login. | [01:19] |
| pharvey | SvenDowideit: I did some simple grammar fixes | [01:21] |
| SvenDowideit | pharvey, oooo | [01:22] |
| pharvey | of course, it's possible I made it worse, but I hope not.
Insecure dependency in open while running with -T switch at /usr/local/src/github.com/foswiki/core/lib/Foswiki/Configure/Checkers/Htpasswd/FileName.pm line 29. | [01:23] |
| SvenDowideit | i hate that people want foswiki spelt with caps :p | [01:24] |
| pharvey | oh well, change it back :P | [01:25] |
| gac410 | not even the leading f making it a proper name? | [01:25] |
| pharvey | I did foswiki -> Foswiki | [01:25] |
| SvenDowideit | nopw :)
i foswiki regularly :) i'm too young&hip to be limited in my definitino of 'propername' :P I can ~~ was able t | [01:25] |
| pharvey | gac410: I fear Foswiki::Configure::Load::expandValue is returning tainted values | [01:27] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Configure::Load | [01:27] |
| gac410 | Hm. why. They come from a file, not from the CGI. | [01:27] |
| SvenDowideit | SvenDowideit geos off to steal pharvey's changes | [01:27] |
| pharvey | well, it's either that or the line above: my $f = $Foswiki::cfg{Htpasswd}{FileName}; | [01:28] |
| gac410 | Is this new? I don't think that code has changed in quite a while. | [01:29] |
| pharvey | I've occasionally got taint bugs from configure with trunk
this is a new location for it to happen in though (the Htpasswd/Filename.pm checker) previously it's the logdir checker | [01:29] |
| gac410 | Is it repeatable? or is this a sporadic thing that comes/goes/moves around | [01:30] |
| pharvey | yeah, it doesn't happen every time. I'll try with a fresh config shortly.
subsequent saves always work. | [01:30] |
| gac410 | I wonder if in the initial assignment it's pulling someting off of a path variable or directory search. | [01:31] |
| pharvey | I wanted to explore the possibility there's something going on with the initial loading from .spec files | [01:32] |
| gac410 | Some of the path info gets discovered from current directories, etc. and applied to the file locations.
A lot of the file locations are undefined in Foswiki.spec and are guessed. So Htpasswd Filename is based off of the DataDir which is guessed in first run. | [01:33] |
| pharvey | damn, I can't reproduce this time. | [01:38] |
| SvenDowideit | ok, so trunk failures are bizzare, svn up did heaps of processing, so somehting was very hung | [01:41] |
| SvenDowideit ponders that :/ | [01:46] | |
| pharvey | pharvey will email SvenDowideit some notes | [01:47] |
| SvenDowideit | many thanks - would make sense to get it back working now
mmm, is that crazy or what: Analyzed requests from Mon, Sep 25 2006 at 10:51 PM to Tue, Feb 21 2012 at 12:23 AM (1974.06 days). | [01:47] |
| ...... (idle for 26mn) | ||
| GithubBot | [foswiki] foswiki pushed 3 new commits to Release01x01: http://git.io/Bwp-8A
[foswiki/Release01x01] Item11547: update debian package to 1.1.4 - SvenDowideit [foswiki/Release01x01] Item10743: add dependancy on HTML::Tree for compareRevisionAddOn - SvenDowideit [foswiki/Release01x01] Item10964: add /var/lib/foswiki/working/logs softlink to debian logs dir - SvenDowideit | [02:17] |
| *** | GithubBot has left | [02:17] |
| FoswikiBot | http://foswiki.org/Tasks/Item11547 [ Item11547: update core debian package to 1.1.4 ]
http://foswiki.org/Tasks/Item10743 [ Item10743: Fail to install foswiki plugin "foswiki-twikicompatibilityplugin" ] http://foswiki.org/Tasks/Item10964 [ Item10964: Fresh Debian installation fails to create necessary directories ] | [02:18] |
| gac410 | pharvey - jeeze - ran the whole suite and the ampersand test fails again. looking for "a" as the topic name. | [02:26] |
| pharvey - shorter URL support. Parses out the entire link. | [02:36] | |
| GithubBot | [foswiki] foswiki pushed 1 new commit to Release01x01: http://git.io/u9bzpw
[foswiki/Release01x01] Item11547: add CVE-2012-1004 fix - SvenDowideit | [02:45] |
| *** | GithubBot has left | [02:45] |
| gac410 | Sven - did you apply more than just the encode regex in that patch? The try/catch stuff in the mapper is new - I don't think you can take parts of 1.1.5 beyond the field encode. | [02:49] |
| SvenDowideit | eh?
i'm trying to follow 'Hotfix for Foswiki Production Releases 1.1.0-1.1.4 ' | [02:50] |
| gac410 | Should be a one-line change. | [02:51] |
| SvenDowideit | but dpatch is making me bleed
no, i'm using the Register.pm on that topic and thats more than on line | [02:51] |
| gac410 | crap... It's supposed to be a one-line change. wtf.
I restored a clean 1.1.4, applied the one-line patch and uploaded that module ... I thought. I specifically did not want any of the other 1.1.5 pending changes. | [02:51] |
| SvenDowideit | the diff in the commit email is longer
and this is against the 1.1.4.tgz that i have in the github/foswiki-releases repo which is an rsync of whats on sourceforge | [02:53] |
| gac410 | The commit email is NOT the hotfix. | [02:53] |
| SvenDowideit | (i hope)
then the Register.pm you put there that people are using may not be either | [02:53] |
| gac410 | The hotfix is not what will be in 1.1.5. | [02:54] |
| SvenDowideit | i'm building a 1.1.4 pkg from tgz plus the Register.pm linked to from the hotfix
http://foswiki.org/Support/SecurityAlert-CVE-2012-1004#Hotfix_for_Foswiki_Production_Releases_1.1.0_451.1.4 | [02:54] |
| gac410 | The Register.pm I posted should have the one-line fix that is in the diff in the CVE. | [02:54] |
| SvenDowideit | i'm actually in a world of debian pain | [02:54] |
| gac410 | Did someone else update it? | [02:55] |
| SvenDowideit | please, check :/
cos it chould just as easily be something stuffed with what i have | [02:55] |
| gac410 | okay - will take me a moment. | [02:56] |
| SvenDowideit | mmm, ok, it might be titally my sodup
debian is bizzaroworld sometimes that might be a diff from 1.1.3->hotfix | [02:58] |
| gac410 | Ah ... That sounds better.
Yes, that's what it is. | [02:59] |
| SvenDowideit | crazy | [03:00] |
| gac410 | gac410 likes tarballs. gentoo builds from tarballs. | [03:01] |
| SvenDowideit | debian builds from tarballs too
_once_ | [03:01] |
| gac410 | :-P | [03:01] |
| SvenDowideit | whereas gentoo adds to global warming
so ner :p | [03:01] |
| gac410 | well - keeps my lap warm anyway. Good in winter months when my cat is not cooperative. | [03:02] |
| SvenDowideit | its much more environmentally friendly to move to where the cimate suits the body
but then the world starts to tilt and might just topple off ok, that might work a little better mmm, oh, the jquery checker is in 1.1.5 huh | [03:03] |
| pharvey | SvenDowideit: I think Item10308 was released in 1.1.3 | [03:12] |
| FoswikiBot | http://foswiki.org/Tasks/Item10308 [ Item10308: Need to check if JQuery version is sane ] | [03:12] |
| gac410 | pharvey - I'm going to commit the unit tests and double encoding fix on 1.1.5 -but the tests fail due to Foswiki::Address. | [03:14] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address | [03:14] |
| pharvey | gac410: k, I'll fix it tonight
I guess, expect_failure till then | [03:14] |
| gac410 | Yeah - I didn't bother with expectFailkure - didn't want the 1.1. and trunk tests to diverge | [03:15] |
| pharvey | ok | [03:15] |
| GithubBot | [foswiki] foswiki pushed 4 new commits to Release01x01: http://git.io/bX1hLw
[foswiki/Release01x01] Item11547: lets diff from 1.1.4 so the patch works - SvenDowideit [foswiki/Release01x01] Item11547: go with jquery 1.7.1 and remove some deprecated plugins - SvenDowideit [foswiki/Release01x01] Item11366: Prevent double encoding. - GeorgeClark | [03:17] |
| *** | GithubBot has left | [03:17] |
| FoswikiBot | http://foswiki.org/Tasks/Item11547 [ Item11547: update core debian package to 1.1.4 ]
http://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ] | [03:17] |
| GithubBot | [foswiki] foswiki pushed 3 new commits to master: http://git.io/g3i-2w
[foswiki/master] Item11512: <base tag must be closed for HTML validation - GeorgeClark [foswiki/master] Item11366: Both tests pass. - GeorgeClark [foswiki/master] Item11528: Put back EnableHierarchicalWebs - GeorgeClark | [03:18] |
| *** | GithubBot has left | [03:18] |
| FoswikiBot | http://foswiki.org/Tasks/Item11512 [ Item11512: Ensure all topics in System pass ]
http://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ] | [03:18] |
| pharvey | gac410: in Item11448 you added a warning to say that restauth needs to be protected, but the reason why I think is misleading. It says currently: "This could permit commenting by unauthorized users" - actually, the failure mode is that the browser doesn't receive a WWW-Authenticate challenge (when using ApacheLogin). Or am I wrong? | [03:23] |
| FoswikiBot | http://foswiki.org/Tasks/Item11448 [ Item11448: {GuestCanComment} setting is not implemented ] | [03:23] |
| pharvey | in other words, it doesn't allow unauthorized users to do anything (an unprotected rest script would do the same), it just prevents ApacheLogin from requesting authentication, I think.
no, that's wrong. I'm confusing the apache config. damnit, I hate {AuthScripts} with the power of a thousand suns can you believe there's only two lines of code in all of lib/* that aren't lines in the manifest or checkers which reference AuthScripts | [03:24] |
| gac410 | Probably because I don't understand the authScripts either. But my understanding was that CommentPlugin would require restauth if authentication was required. | [03:40] |
| SvenDowideit | 'CommentPlugin would require restauth if authentication was required if you're using apache auth and don't know or can't set up a better apache conf'
SvenDowideit fixed it for you :) i really should shut up about that and put out :D (as in, add authenticated script re-writes to ACG2) | [03:40] |
| pharvey | gac410: I'm doubting whether it's worth having that warning at all. If anything, it's need on the ={AuthScripts= checker, not in CommentPlugin. Because an unprotected =rest= script has the same issue. | [03:42] |
| gac410 | no CommentPlugin needs the rest script unprotected.
otherwise the GuestsCanComment switch is a nop | [03:43] |
| pharvey | so I'm struggling to understand the context of the error message
I guess I'm trying to understand: is my understanding wrong, or yours :) the fact I'm having trouble means this is all quite sucky anyway, regardless | [03:44] |
| gac410 | tbh I'm relatively clueless. I thought that restauth must be a protected script. and for guests to comment, rest must not be protected. | [03:45] |
| pharvey | you are right - restauth should be protected; but no more so than rest itself
there's actually three different levels of "needs auth", and that sucks | [03:46] |
| gac410 | If both are protected, then why do we need restauth. rest would be enough. | [03:47] |
| pharvey | 1) in registerRESThandler( authenticate => 1
2) in AuthScripts 3) In apache restauth is purely for ApacheLogin so that apache can force a 401 Auth required | [03:47] |
| SvenDowideit | and its not necessary :( | [03:47] |
| pharvey | well, not any more | [03:47] |
| SvenDowideit | ever
non of the *auth ones are | [03:47] |
| pharvey | er
that's not what happened to me if a rest script needed auth, it failed | [03:48] |
| SvenDowideit | then ou need to know more apache conf :) | [03:48] |
| pharvey | unless apache threw a 401 | [03:48] |
| SvenDowideit | where's the flipping ACG2 topic
yes, and you can get apache to throw a 401 without needing a script to actually exist | [03:48] |
| pharvey | how the hell can apache know to require auth on a script that may or may not need it? | [03:48] |
| gac410 | pharvey I'm with ou on that one. | [03:48] |
| SvenDowideit | apache is not as dumb as we've always thought
sun came out, so i won't mow the lawn yet | [03:49] |
| pharvey | ... Apache has *no way* of known that /bin/rest/MyPlugin/handler needs auth | [03:49] |
| SvenDowideit | maybe i should just hack up ACG2 to show you
apache does not need to knwo | [03:49] |
| pharvey | so how can it throw 401? | [03:49] |
| SvenDowideit | <SvenDowideit> where's the flipping ACG2 topic
it seems you're too sure that you know apache | [03:49] |
| pharvey | presently, the only way Foswiki has to get apache to auth someone is to redirect them to a protected script | [03:50] |
| gac410 | Foswiki:Support.NewApacheConfigGenerator | [03:50] |
| FoswikiBot | http://foswiki.org/Support.NewApacheConfigGenerator [ NewApacheConfigGenerator ] | [03:50] |
| SvenDowideit | and i apparetnly can't tell you the right things for the penny to drop
so.. ta | [03:50] |
| pharvey | ok | [03:50] |
| SvenDowideit | i can't find anything in the support web
ever ok, so, does anyone use it yet? or is it just crazy people like me? | [03:50] |
| pharvey | but I have debugged this extensively before, and you even sent me an E-mail implying I should just give up on /bin/scripts and re-write it so we only have two: unauth and auth | [03:51] |
| gac410 | I've used it for a bunch of configs. | [03:51] |
| pharvey | pharvey too | [03:51] |
| SvenDowideit | the q being - whats the risk if i botch it up a few times
meh, i'll make a local copy to hack | [03:51] |
| pharvey | AFAIK the only way Foswiki has to force apache to auth someone is to redirect them to a protected script | [03:51] |
| SvenDowideit | correct | [03:51] |
| gac410 | Yeah - that's what I do for major changes though I have not done that in a while. | [03:52] |
| SvenDowideit | but there is __zero__ need for that script to actually exist
where zero is small - there is iirc one senario where you can't use htaccess due to conf restrictions | [03:52] |
| pharvey | well, that's a different hack, I guess | [03:52] |
| SvenDowideit | if we're giving people a conf file, then, er, we're giving them a conf file | [03:53] |
| gac410 | well with fcgi, the only script apache will use is foswiki.fcgi 401 is done with LocationMatch not FileMatch | [03:53] |
| SvenDowideit | zactly
same hack really | [03:53] |
| gac410 | But need to be careful because LocationMatch is not allowed all the time. I've had issues with it on occasion. | [03:53] |
| SvenDowideit | y, the one i read about used rewrites or redirects
but even then i thought there was a small caveat but as we use it in the conf, i can't see why not use it mmm where did the edit wikitext button go in trunk trunk tinymce | [03:54] |
| gac410 | I was trying to make the nacg pluggable so that I could eventually build .htaccess files, and also build a working config when mod_rewrite was unavailable. | [03:55] |
| SvenDowideit | so the q will be, can i make it work
gads tho, i have a really cusomised magical conf serves me right | [03:57] |
| gac410 | I also really wanted to have all LocalSite.cfg matching parameters detailed in comments in the config file - so users could have a clue. Our manual customizing instructions are really confusing
But I've run into if statement syntax limitations and ran into dead end. | [03:59] |
| SvenDowideit | SvenDowideit has no idea where the code is actually up to, nor how to test it :( | [04:00] |
| gac410 | what code? NACG? | [04:00] |
| SvenDowideit | y
i guess it'd need a www::mechanise that pertabates through the options | [04:00] |
| gac410 | My testing was creating config after config, and installing each one into an apache vhost and testing. | [04:01] |
| SvenDowideit | and then uses that conf in a fresh virtualhost
that then bashes on it using selenium SvenDowideit tasks pharvey with that - you have 15mins mmm, so 'not implemented' i can disable | [04:01] |
| gac410 | :P | [04:03] |
| pharvey | don't laugh, I began to write some code which did stuff along those lines :) | [04:03] |
| gac410 | I wanted to eventually enable it and hopefully it's structured to let me add it in. | [04:03] |
| SvenDowideit | pharvey, i'm not laughing
gac410, its fine - i'm more worried about breaking something that already works | [04:03] |
| pharvey | in the end I gave up because it was really difficult to test a selenium client against a particular customized Foswiki config | [04:04] |
| SvenDowideit | how? | [04:04] |
| pharvey | involves perl INC path magic and/or restarting apache from the tests or just nuking the current LocalSite.cfg in-place | [04:04] |
| SvenDowideit | SvenDowideit wants to know :) | [04:04] |
| gac410 | I was happy to just manually test it. | [04:04] |
| SvenDowideit | gac410, more importantly - what limitations n IF did you hit
and can we work on that pharvey, i was thinking more using one vhost per cfg | [04:04] |
| pharvey | gac410, this was part of the unicode effort. I found bugs that seemed to involve the whole FCGI.pm, CGI.pm, apache & Foswiki.cfg stack, so I wanted that automated.
I think I began writing something called Unit::WebServer | [04:05] |
| gac410 | I don't remember now. I think the bug is stll there. The comments in the front of the config - one of them gets munged up and I couldn't get past it and ran out of energy. | [04:06] |
| SvenDowideit | and you didn't github it for next year | [04:06] |
| pharvey | SvenDowideit, did too | [04:07] |
| gac410 | I was running out of quotes and escaped quotes and got totally lost in escaping. | [04:07] |
| SvenDowideit | pharvey, aw, blooming brilliant | [04:07] |
| pharvey | https://github.com/csirac2/UnitTestContrib/tree/Item1326
it uses an older version of NACG I'm not sure it's at all useful but might be an inspiration of what not to do | [04:07] |
| SvenDowideit | who knows, matters not
i hate losing code :) i wonder how you remove the /bin/view if mod_rewrite is off | [04:08] |
| gac410 | I think there are ways with aliases, | [04:10] |
| pharvey | ScriptAlias, but be careful of ordering | [04:10] |
| SvenDowideit | ta - /me is just cleaning a little to get more familiar
setting login=tempalte as default too | [04:10] |
| gac410 | ah - and mod_actions is another way i found but never set it up to test. | [04:11] |
| SvenDowideit | mod_actions is disabled now, as 'not implemented' | [04:11] |
| gac410 | right - it's there to remind me to work on it someday. | [04:11] |
| SvenDowideit | :)
SvenDowideit nukes some inline URLPARAMs that should use the Vars Set at the top this really is a great eg of complexity that we should get a group of people reviewing for 'style' | [04:12] |
| pharvey | damn - Insecure dependency in open while running with -T switch at /usr/local/src/wiki.trin.org.au/core/lib/Foswiki/Configure/Checkers/Htpasswd/FileName.pm line 29. again | [04:14] |
| gac410 | my TML skills are not the best. | [04:16] |
| SvenDowideit | gac410, no-ones are | [04:16] |
| pharvey | gac410: I have a reliable recipe to make it fail now :) Just change the htpasswd file location
happens for LockFileName too | [04:16] |
| SvenDowideit | because we've never sat down and figured out best practice | [04:16] |
| gac410 | I'll give it a try pharvey | [04:16] |
| pharvey | pharvey is reminded on the brief rampage he underwent to at least make System web resemble something that everybody thought was less crap, resulting in TipsContrib being reverted (rightly so, but I was mad at the time) | [04:17] |
| gac410 | some reverting is not done very diplomatically | [04:18] |
| pharvey | and at the same time, we probably don't do enough reverting | [04:18] |
| gac410 | hm. So I changed password location from {DataDir}/.htpasswd to {DataDir}/sub/.htpasswd ... no taint errors. | [04:19] |
| pharvey | bugger
I'm on perl v5.14.2 FWIW | [04:19] |
| gac410 | 5.12.4 | [04:19] |
| pharvey | funny thing is that the save actually does its job (LocalSite.cfg), so that's a bit terrifying from a "taint mode stops bad stuff" perspective | [04:20] |
| SvenDowideit | crud.
turning off mod_rewrite doesn't get me a working apache conf | [04:21] |
| gac410 | perlbrew switch 5.14.2 / perl rewriteshebang ... .htpasswd -> data/Main/.htpasswd ... no fail.
no. esp. with NewApacheCG - mod_rewrite is required. | [04:21] |
| SvenDowideit | oh. you have it as a tickbox
ok, i'll disable it bu leave it turned on | [04:22] |
| gac410 | Not implemented. | [04:22] |
| SvenDowideit | it says ' mod_rewrite - (Recommended if available) '
so i presumed :) | [04:22] |
| gac410 | It's a choice of mod_rewite, mod_alias or mod_actions. Should be radio buttons | [04:23] |
| SvenDowideit | ah, ok, sweet | [04:23] |
| gac410 | It's labeled experimental for a reason :-D | [04:23] |
| SvenDowideit | y, and in the process i'm learning aboot it | [04:23] |
| pharvey | so, should I debug the taint error while it's still there, or do some $work
pharvey sends emails | [04:23] |
| SvenDowideit | $work-u-do
mmm, nope, still got nasty docroot issues lets see - i put in /acg2 as the foswiki url maybe it needs to be /acg2/ nope, its still toast :( dammit, i _have_ this working in the rpms i wonder if i'm just killing myself because i have another non vhost setup tot he same files | [04:24] |
| gac410 | AH HA..... Got the Taint error! | [04:29] |
| pharvey | it's a sneaky bugger 'innit :) | [04:29] |
| gac410 | I wasn't on trunk. And I was changin th path not the filename.
Nope - path change fails as well. I't s atrunk thing. Works okay on rlease branch So why is the field tainted on trunk | [04:30] |
| pharvey | I'm trying to get a stacktrace
I also reproduced CDot's Meta.pm ASSERT fail, I think, still working on that one too. adding ASSERTS makes the taint issue go away?! oh, never mind. I'm just mentally challenged. | [04:31] |
| SvenDowideit | oh, mmm
i can't even get the acg2 working as is the only wrinkle i have is that my foswiki is outside the docroot oooooh mmm, bizzare. no, what i'm doing that is broken is that i want the script url to bear no resemblance to the real on disk paths | [04:39] |
| gac410 | pharvey - the reason it fails on trunk is CDot added code to create the .htpasswd file if it doesn't exist. release11 doesn't try to create the file.
According to Scalar::Util::tainted() it's tainted when read from the config. No idea why though. as checkers only run on load - and the config is loaded from the file, never from the url. | [04:44] |
| GithubBot | [foswiki] foswiki pushed 1 new commit to master: http://git.io/u2T9wQ
[foswiki/master] Item10942: don't concat undef $result string - PaulAlexander | [04:47] |
| *** | GithubBot has left | [04:47] |
| FoswikiBot | http://foswiki.org/Tasks/Item10942 [ Item10942: Nested attributes feature: metadata handling ] | [04:47] |
| gac410 | my $f = $Foswiki::cfg{Htpasswd}{FileName}; print STDERR "1 Tainted" if Scalar::Util::tainted($f); | [04:47] |
| SvenDowideit | soacg2 should work without script alias and alias? | [04:47] |
| gac410 | yes. the rewrite rules serve that purpose | [04:48] |
| SvenDowideit | ok, which looks like its not working | [04:48] |
| gac410 | works fine here. | [04:48] |
| SvenDowideit | for the setup i have
where the foswiki files are not in the docroot basically, it keeps whining at me that /var/www/foswiki does not exist and thus doesn't actually get to the rewrites | [04:48] |
| gac410 | The key is the H=cgi-script on the rewrite rule is equivalent to the ScriptAlias | [04:49] |
| SvenDowideit | i'm off to see what i did to make the rpm one work
cos i was sure it did 'just work' :/ i added re-write debugging and the rewrite wasn't triggered so i'm rather puzzled | [04:50] |
| gac410 | Why oh why is $Foswiki::cfg{Htpasswd}{FileName} tainted.
SvenDowideit: If you've moved files around, pay careful attention to the -f tests in the RewiriteCond statements. | [04:51] |
| SvenDowideit | moved?
the files have always been in one place and i entered that into the generator | [04:55] |
| gac410 | okay - just trying to think of things that might cause regexes not to work. | [04:57] |
| SvenDowideit | so the core error i get from apache is:
[Wed Feb 22 14:54:44 2012] [error] [client 127.0.0.1] File does not exist: /var/www/foswiki which is frustrating, considering that the foswiki is at /var/lib/foswiki | [04:57] |
| gac410 | and /var/www/foswiki doesn't show in the acg2 file
pharvey - so a blind untaint in FileName fixes the issue. But I have no idea why it's needed ($f) = $f =~ m/^(.*)$/; | [04:58] |
| SvenDowideit | correct | [04:59] |
| gac410 | don't know Sven ,.. Are you sure apache actually read your config. I've spend hours debuging a config file when the vhost was resolving to a difffernt config :-) | [05:00] |
| SvenDowideit | yup, something like that, lookslikei lost motivation battling something that worked all morning
so the old scriptalias and alias conf is a little less delicate :/ as i just replaced a working conf with a similar looking acg2 one and get bubkus | [05:01] |
| gac410 | I don't think it's more delicate. As with any config, the paths in the config have to match the file system. | [05:04] |
| SvenDowideit | i'm pretty sure taht the fs matches
in fact, all those bits _do_ match but some other conf is getting there first | [05:04] |
| gac410 | That you turned on rewrite debug, and get no messages, suggests that apache is not using that config at all. | [05:05] |
| SvenDowideit | whereas the old conf worked
exactly | [05:05] |
| gac410 | So that would be the vhost statement everything else depends on it. | [05:05] |
| SvenDowideit | whereas the one that was working was from the old generator
i don't use vhost i have that off in everything as its useless to me here this is going to send me mental, and then i'll find the one line that makes it work :( | [05:05] |
| gac410 | okay - so if the config file is a NOP and rewrite rules log nothing ... did you include the right filename? | [05:06] |
| SvenDowideit | include?
apache does includeDirs | [05:07] |
| gac410 | Some distributions do it differeent ways - I don't know. just thinking of things.
add a syntax error to the config. I can't imagine how mod_rewuite logLevel wouldn't log. it should log that nothing matches | [05:07] |
| SvenDowideit | bugger. add syntax error, and apache dies :(
that woulda been an excellent stuffup :) | [05:08] |
| gac410 | gac410 running out of ideas. | [05:09] |
| SvenDowideit | my thething is that re-write does not get run | [05:10] |
| gac410 | The new config is not all that comlex. It just replaces ScriptAlias with H= operand. ScriptAlias is not allowed in .htaccess files, and I was trying to make ACG generate both. | [05:10] |
| SvenDowideit | because apache already says 'file does not exist /var/www/foswiki'
so its got zero file location matches what i can't work out, is why this exact same (i copied it) conf works on centos | [05:10] |
| gac410 | something is ringing a bell that I've actually run into that issue. | [05:11] |
| SvenDowideit | i think its insisting that DOCROOT/URL exists
and you probly have to use LocationMatch or something to give it one | [05:12] |
| gac410 | I can't imagine why. What is your DocumentRoot set to | [05:13] |
| SvenDowideit | whatever debian says (/var/www) | [05:13] |
| pharvey | pharvey always sets DocumentRoot to somewhere bogus | [05:13] |
| SvenDowideit | http://foswiki.org/Support/NewApacheConfigGenerator?rewrite=on;actions=;alias=;htaccess=apache;hostname=localhost;althost=;vhost=;port=;dir=/var/lib/foswiki;symlink=;pathurl=/foswiki;shorterurls=enabled;engine=CGI;fastcgimodule=fastcgi;apver=2;allowconf=;requireconf=;reqandor=and;loginmanager=Template;htpath=;errordocument=UserRegistration;errorcustom=;blockmodphp=on;blockpubhtml=on;blocktrashpub=on;controlattach=on;compressjscss=
on;expireshdr=on;etags=on;blockscripts=;blockspiders=on;foswikiversion=1.1;blockgsasess=on /var/www is pretty bogus :) | [05:14] |
| pharvey | dunno why. I think in the 90s, it was a fun way to ensure busted ACLs wouldn't reveal anything
I've had RewriteRule 'scoping' issues before crazy++ rewriteloglevel 9 :) | [05:14] |
| SvenDowideit | yes, i have that
but i get nothign because apache doesn't get to the rewrite rules creating /var/www/foswiki just means it whines about the next level | [05:15] |
| gac410 | I don't understand where it
gets that from in the first place. | [05:16] |
| SvenDowideit | simple - DOCROOT+URL | [05:16] |
| gac410 | makes no sense. | [05:16] |
| SvenDowideit | its one of the first tests it does - and i can't recal the magic to tell it to nick off | [05:16] |
| gac410 | Don't you need a valid DocumentRoot so the pub file serving has a valid starting location? | [05:18] |
| SvenDowideit | SvenDowideit hasn't need one before | [05:19] |
| gac410 | Apache docs says - DocumentRoot used Unless matched by a directive like Alias, | [05:19] |
| SvenDowideit | which was what i expected
and why the old acg has an alias | [05:20] |
| gac410 | I've always set DocumentRoot to the root above pub - which is why I never need an alias. | [05:20] |
| SvenDowideit | but the new one doesn't
so your conf's are very specific :( SvenDowideit goes mad (as it works on centos, dispite the foswiki being outside the docroot | [05:20] |
| gac410 | yeah. That's the point of entering paths into the NACG | [05:21] |
| SvenDowideit | you can't set a second docroot | [05:21] |
| gac410 | If DocumentRoot is not set, then ServerRoot is used. | [05:21] |
| SvenDowideit | and its not like NACG ast docroot
so, um nacg does not set docroot or serverroot or is non-vhost also not implemented? | [05:21] |
| gac410 | It sets DocumentRoot | [05:22] |
| SvenDowideit | nope | [05:22] |
| gac410 | I've never tried it without a vhost. | [05:22] |
| SvenDowideit | argh!
when why make non-vhost an option, and have it off by default! SvenDowideit shoots self for trying to debug it for the last hours :( Generate vhost: optional | [05:22] |
| gac410 | It's cloned from the old ACG - You are the first person that has tried it in a non-vhost environment. I don' t have the systems available to test it.
Every server I've ever touched in the last 5 years has used vhosts. | [05:24] |
| SvenDowideit | please, disable stuff you've not tested (i'm going through and disabling the ones i find atm | [05:24] |
| gac410 | NO.... It will never get tested. The damn thing is expermiental and asks for feedback. | [05:24] |
| SvenDowideit | cos its really really hard to try using stuff where you don't know what hasn't been cone
it can't be tested if its not been implemented :) SvenDowideit realises this discussion sounds like the ' | [05:24] |
| gac410 | It HAS been implemented. | [05:25] |
| SvenDowideit | 'whats in trunk' discussion but reverse
<gac410> I've never tried it without a vhost. ~~ notimplemented | [05:25] |
| gac410 | It does EXACTLY what the ACG does. I didn't know 1) That servers ran with bogus document roots, or 2) that Alias was reqruied | [05:25] |
| SvenDowideit | which is why i rarely set to merge to core in time
no ACG does somethign different as it sets alias which allows apache to work | [05:26] |
| gac410 | Right. So add some feedback and a note - | [05:27] |
| SvenDowideit | ok, so, i don't know your assumptions, just like the RM can't know the assumptions made by a coder | [05:27] |
| gac410 | um. Can I be any MORE CLEAR:
DRAFT DOCUMENT This is an early version of a new ApacheConfigGenerator. it's not ready for use yet, but feedback welcomed! | [05:27] |
| SvenDowideit | but i kept being told that acg wasn't ready, so i didn't try, now that i have, i can't use it becuase i have no idea how much more i have to use it
yes, you can just like i can | [05:28] |
| pharvey | gac410: I have confirmed that $f is tainted here: my $f = $Foswiki::cfg{Htpasswd}{FileName}; - madness | [05:28] |
| gac410 | yeah same here. | [05:28] |
| SvenDowideit | we're not communicating our assumptions, so we're having this discussion :/ | [05:28] |
| gac410 | The very first line of the topic says it is not ready for use.
IT IS NOT READY FOR USE | [05:29] |
| SvenDowideit | yes, but it does not tell me what i can expect to test its use for
which is why until today, when i wanted to do q __QUICK__ demo of an idea i did not try to use it but __QUICK__ has resulted in us shouting at each other because i was trying to use it to try out an idea SvenDowideit needs to find a quick success to have in the last 30mins of his day, else the rest of my week will be a horrid shit. | [05:29] |
| gac410 | I don't know what I don't know. pharvey has tested. I've used it extensively. But not in your particular configuration. I didn't know that Alias was a substitution for DocumentRoot, Sorry I ruined your day. Time to sleep. | [05:32] |
| *** | gac410 has left | [05:32] |
| ..... (idle for 20mn) | ||
| pharvey | TemiV found a brilliant bug with INCLUDE{warn="custom text"} - it only works for topic-missing warnings. | [05:52] |
| ............... (idle for 1h12mn) | ||
| *** | ChanServ sets mode: +o MichaelDaum | [07:04] |
| .......................... (idle for 2h9mn) | ||
| CDot | FoswikiBot: seen pharvey | [09:13] |
| FoswikiBot | CDot: pharvey was last seen in #foswiki 3 hours 21 mins ago saying "TemiV found a brilliant bug with INCLUDE{warn="custom text"} - it only works for topic-missing warnings.". | [09:13] |
| ......... (idle for 40mn) | ||
| MichaelDaum | WikiRingBot, seen paul | [09:53] |
| WikiRingBot | WikiRingBot hasn't seen paul yet | [09:53] |
| MichaelDaum | WikiRingBot, seen pharvey | [09:53] |
| WikiRingBot | WikiRingBot has last seen pharvey 22 Feb 2012 - 05:52 GMT on #foswiki
WikiRingBot has last seen pharvey_ 20 Feb 2012 - 05:04 GMT on #foswiki | [09:53] |
| MichaelDaum | CDot, got a few spare brain cycles?
I am currently rewriting the page cache now, storing and maintaining the dependency graph works reliably and fast using DBI this is tested successfully against a mysql and sqlite all former Foswiki::Cache implementations will go away | [09:54] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Cache | [09:56] |
| MichaelDaum | problem is: can we afford to raise the technical requirement for page caching to having these things installed?
is even sqlite too much? given that page cache is off by default CDot, what do you think? | [09:57] |
| CDot | as an unconditional requirement I would say yes, it's too much
but as a conditional requirement - it works even without it - then fine | [10:01] |
| MichaelDaum | the requirement is: if you want page caching, then you need sqlite or mysql or whatever sql database is available.
the other option is to write a Foswiki::PageCache::BDB ... which certainly won't perform anyway | [10:03] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache::BDB | [10:04] |
| MichaelDaum | even BerkeleyDB isnt installed on lots of systems
which sort of renders the point mood | [10:04] |
| CDot | moot. Yes, I don't have a problem with requiring "extras" to enable something optional, BUT performance of a non-cached system should not be compromised by the changes | [10:10] |
| MichaelDaum | it can't. because, when there _is_ no $session->{cache} subsystem, then there's no cpu spent inside it | [10:12] |
| CDot | then I think you are on to a winner :-) | [10:22] |
| MichaelDaum | I will still make Foswiki::PageCache a virtual classs, even though there's only one impl Foswiki::PageCache::DBI ... just in case somebody figures out how to fire dependencies into a valgrind rev proxy | [10:23] |
| FoswikiBot | http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::PageCache::DBI | [10:23] |
| CDot | good, I was going to ask | [10:25] |
| .......... (idle for 47mn) | ||
| MichaelDaum | f.o dead? | [11:12] |
| Babar | isitdownorjustforme?
works for me :) | [11:16] |
| MichaelDaum | ya, the lady needs her time. | [11:17] |
| ModAcOst | Hi, is there a way to make www.mywiki display www.mywiki/myWeb/WebHome instead of www.mywiki/Main/WebHome, or do I have to mess around with rewrites? | [11:18] |
| .... (idle for 15mn) | ||
| SvenDowideit | ModAcOst HomePagePlugin | [11:33] |
| Babar | Rewrite / redirects
it's much easier than anything else | [11:33] |
| SvenDowideit | actually, HomePagePlugin is easier still | [11:34] |
| ModAcOst | Redirect does not work unfortunately | [11:34] |
| SvenDowideit | mod_rewrite is a simple line in your apache conf / .htaccess
HomePagePlugin is one enable and one cfg | [11:34] |
| ModAcOst | Ok, thx, I will try the rewrite | [11:35] |
| SvenDowideit | and iirc does not cause a second request either - as bin/view is detected and then web&topic are set | [11:35] |
| ModAcOst | Ok, I'll have a look at HomePagePlugin first^^ | [11:35] |
| SvenDowideit | homepageplugin also adds /bin/view can == the users Main.UsernamePage
which is cuter'n heck actually, the more i think about it, using HomePagePlugin should add no extra workload to the server too, so its a pretty good win - all i need is more docco and unit tests, and er BOOM :) | [11:35] |
| ModAcOst | HomePagePlugin works perfectly, thx | [11:43] |
| SvenDowideit | sweet :) nother happy customer | [11:43] |
| pharvey | boo!
MichaelDaum: how do you expire objects in the DBI impl? Is there a cron job which runs periodically ? Hi CDot, on Item11528 I guess it's just annoying to get a warning on a tab but then not be able to see that warning. Not sure what the solution should be... I guess the hiding should also neuter the warnings | [11:46] |
| FoswikiBot | http://foswiki.org/Tasks/Item11528 [ Item11528: Hide irrelevant =configure= options ] | [11:49] |
| MichaelDaum | pharvey, no cronjob. it just rejects to deliver a page that is expired. | [11:50] |
| pharvey | I mean, how do we stop endlessly growing DBs | [11:50] |
| MichaelDaum | there isnt endless content to render pages for
anyway, I've added more docu to the PageCache pod to make clear how pages are purged and when | [11:51] |
| pharvey | ok | [11:52] |
| MichaelDaum | it is more of a problem of too few pages remaining in the cache when some other topic is saved.
so a much more helpful cronjob would be to warm up the cache over night | [11:53] |
| pharvey | ok, so expired objects are removed when Foswiki says they're invalid | [11:54] |
| MichaelDaum | pharvey, spot on. thats why there is a dependency graph | [11:54] |
| pharvey | I thought there might be a mark & sweep impl. But never mind :) | [11:55] |
| MichaelDaum | there are some single topics that _all_ cache entries depend on: for instance SitePreferences.
so editing this one topic will flush all of the page cache | [11:56] |
| pharvey | I have been curious as to whether SvenDowideit's "ACL Profiles" pattern could be used to enable re-use of cache objects between several users | [11:56] |
| MichaelDaum | I leave that for others | [11:57] |
| pharvey | yeah, it sounds hard :)
I'm sure DBI backend will be good enough | [11:57] |
| SvenDowideit | it won't be hard to implement, but it'll be hard to prove.
and its really painful to use the ACL profile pattern on disk i tried twice, and it really didn't work out same reason as real random access is needed for cache | [11:57] |
| pharvey | didn't work performance-wise? | [11:58] |
| SvenDowideit | random - as in matix wise
its more that a filesystem is crap for a 4 dimentional matrix | [11:58] |
| pharvey | filesystems are crap for things that aren't trees | [11:59] |
| SvenDowideit | and pagecache is ndimensional kinda | [11:59] |
| MichaelDaum | y | [11:59] |
| SvenDowideit | though i guess i could use the TagPlugin arch for it
but even so its ouchy it'd be nice to move the pagecache into a contrib tho so that the DEPENDENCIES are clearer | [11:59] |
| pharvey | more important is adding QuerySearch expressions as dependencies. Then, on save, new topics which match could fire that dependency. MichaelDaum, does that sound hard in your new PageCache?
(more important than re-using cache objects, I mean) | [12:00] |
| SvenDowideit | hehe - what you want, is in DB triggers
and er that means generated queries and that means.... >:} | [12:01] |
| pharvey | I guess each | [12:01] |
| SvenDowideit | MichaelDaum are you doing this in 1.1? | [12:02] |
| pharvey | QuerySearch dependency also needs the web/topic scope set up by an SEARCH/etc. | [12:02] |
| MichaelDaum | pharvey, that's what db engies do. they cache results for the same query.
however, the result set doesnt only depend on the query. it foremost depends on the data changing. SvenDowideit, nope trunk only. | [12:02] |
| SvenDowideit | ok, just beware that store2 changes everything :/ | [12:03] |
| MichaelDaum | MichaelDaum hoping assuming trunk will be the source for 1.2.0 | [12:03] |
| SvenDowideit | the listeners go away | [12:03] |
| pharvey | MichaelDaum, right, but a huge blocker to people turning on the old PageCache, apart from stability issues, is that new topics aren't invalidating the old cache, so you get stale pages | [12:03] |
| SvenDowideit | they probly wont be in 1.2.0 for eg | [12:03] |
| MichaelDaum | pharvey, correct. and there's nothing you can do about, besides adding an expiry date to %SEARCH pages | [12:04] |
| pharvey | anyway, I don't expect you to solve everything, it's just reminding of it all :) | [12:04] |
| SvenDowideit | yes, there is
its actually an easy solve - but not fast | [12:04] |
| pharvey | It seems we "just" need to add QuerySearch + scope as a dependency.. | [12:04] |
| SvenDowideit | until you get to something like mongodb - where the re-query can be done in mongodb
problem is pharvey re-running all 10,000 queries | [12:04] |
| pharvey | but the re-query happens on each topic save. The search set size is 1: the topic being saved | [12:05] |
| SvenDowideit | is slow as shit
and is impossible | [12:05] |
| MichaelDaum | pharvey, adding the query string isnt the missing bit to solve the dyn pages outdating | [12:05] |
| SvenDowideit | for the same reason as you can't check for js exploit onsave
as you don't know that user's VARIABLES you would have to re-eval all queries for all new topics just as a user requests it actually, no you would need to do a partial render of the view that is cached re-running all queiries on the new topics (new since last cache) thus you have to cache result sets and other things | [12:05] |
| MichaelDaum | basically, when queries where stored as part of the db, will it be able to maintain a result cache. | [12:07] |
| pharvey | MichaelDaum, I didn't envisage saving the query string exactly, but perhaps the parse tree (well, some normalized form anyway). Yes, I agree, I haven't thought through it all... but I recall thinking it wouldn't be *so* hard to make cache expiry at least a little smarter | [12:07] |
| SvenDowideit | y - expire when a new topic matches the web= and topics= params | [12:07] |
| MichaelDaum | most sites do a CACHEEXPIRE = +5mins
and are done | [12:07] |
| SvenDowideit | which is no good
for something like Tasks/MyItems if i goto a task from it, close it, then re-fresh MyItems er, create a new one i mean | [12:08] |
| MichaelDaum | some stuff simply isnt cacheable | [12:09] |
| SvenDowideit | but trivial to add a trigger for force expire if a new topic matches the web= and topic= touple | [12:09] |
| pharvey | I am concerned you guys are obsessing over the 1% case. 99% of SEARCHes, if they are %dynamic, uses RLPARAMs which are already dependencies tracked separately | [12:09] |
| MichaelDaum | either you sacrifice cache correctness by working on an expiry base, or you dont cache at all | [12:09] |
| SvenDowideit | pharvey i think i've shown 3 different simple things we _can_ do | [12:09] |
| MichaelDaum | pharvey, correct. | [12:09] |
| SvenDowideit | but micha is going for the serve old data for 5mins is ok crowd
which is the majority (i think) one big optimisation that all search algos do is based on the web= and topic= SEARCH param | [12:10] |
| MichaelDaum | there are two main features to tackle this problem in page cache: (1) add an expiry date thus sacrificing cache correctness for a specific time slot and (2) mark fragments of the page as uncacheable by surrounding it with <dirtyarea> ... </dirtyarea> tags | [12:11] |
| SvenDowideit | which is massivly faster than doing that in the search itselt
adding web= topic= matching for new topic creation would be a third and a rather powerful one | [12:11] |
| MichaelDaum | web=topic= matching? whats dat. | [12:12] |
| pharvey | "all or nothing" assumes we can't track dependencies in fine enough detail. And that might be true for now, but not impossible | [12:12] |
| SvenDowideit | SEARCH{web="Something*" topic="Item*" | [12:12] |
| pharvey | anyway, I am just really happy to see MichaelDaum working on it :) | [12:12] |
| SvenDowideit | the thing that reduces the set of topics that SEARCH then processes | [12:13] |
| pharvey | SvenDowideit, something I was trying to refer to loosely as scope (for the QuerySearch). | [12:13] |
| SvenDowideit | y
the key thing is that in core, I've used that aggressivly to speed up searches before the hard work is done | [12:13] |
| pharvey | except in mongo, topic="Foo*" makes it heaps slower | [12:14] |
| SvenDowideit | the cache can take advantage of it too
um, it shouldn't it did'n't used to | [12:14] |
| pharvey | why? indexes are faster than regexes | [12:14] |
| SvenDowideit | as topic="Foo*" gets hoisted to a mongodb query | [12:14] |
| pharvey | a mongodb *regex* | [12:15] |
| SvenDowideit | are you sure?
cos it shouldn't | [12:15] |
| pharvey | what should topic="Foo*" get hoisted as?
nothing that can use indexes | [12:15] |
| SvenDowideit | same as "name=~'Foo*'
actually it does | [12:15] |
| pharvey | yeah, that's slow too
that's what I'm talking about | [12:15] |
| SvenDowideit | specifically as its a trailing *
i'm pretty sure that there is docco wrt indexes and testing only leading chars | [12:15] |
| pharvey | when we have a nested SEARCH over 60,000 topics, it's massively quicker to have another FormField rather than rely on the topic naming pattern | [12:16] |
| SvenDowideit | SvenDowideit has no idea about it :/
i do know i wrote code that made it a heffalump faster using topic= | [12:16] |
| pharvey | I did one on Monday, in fact - we had 12345r vs 12345 topics. We saved ~2s (nested search) by using a formfield instead of topic="*r" | [12:17] |
| SvenDowideit | er, yes
*r != leading chars mmmm, i hope i'm not confusing mongo with xapian | [12:17] |
| pharvey | I think you might be | [12:18] |
| SvenDowideit | f*. | [12:18] |
| pharvey | :) | [12:18] |
| SvenDowideit | right, everyone drop mongodb and lets move to TSQL | [12:18] |
| pharvey | TSQL? | [12:19] |
| SvenDowideit | microsoft sql :) | [12:19] |
| pharvey | huzzah! | [12:19] |
| SvenDowideit | nope
'Actually, it depends. If the query doesn't use an index, and must do a table scan, then it can certainly be expensive. If you're doing a 'starts with' regex query, then that can use an index' | [12:19] |
| pharvey | this must be new | [12:20] |
| SvenDowideit | god no | [12:20] |
| pharvey | pharvey tries .explain() | [12:20] |
| SvenDowideit | For simple prefix queries (also called rooted regexps) like /^prefix/, the database will use an index when available and appropriate (much like most SQL databases that use indexes for a LIKE 'prefix%' expression). This only works if you don't have i (case-insensitivity) in the flags.
i wonder if we introduced a regression | [12:21] |
| pharvey | the case-insensitivity
by default SEARCH is case-insensitive | [12:23] |
| SvenDowideit | topic= is case insensitive? | [12:24] |
| pharvey | so it would be throwing an /^Foo/i by default | [12:24] |
| SvenDowideit | no, it should not
as topic= is outside the query | [12:24] |
| pharvey | pharvey checks | [12:24] |
| SvenDowideit | I _think_ topic= is case sensitive
i sure hope so, but stuffed if i know | [12:25] |
| pharvey | all I know is, using Field='Foo' makes the nested searches faster | [12:25] |
| SvenDowideit | which is a bug (i think)
mmm, maybe not eq might, for good reason be faster than loading the regex engine :/ but 2s sounds like crud you are still tracking the queries that you have to hand optimise right? (me knows the answer to reality :) | [12:26] |
| pharvey | the general pattern has always been to remove wildcard/regexes
once I do that, problems go away the other bottleneck is sheer quantity of topics we draw from to render a page | [12:28] |
| SvenDowideit | what scares me a little, is that for 'mongodb-scale' you have a mindbogglingly small number of documents
so regexs that do use the index, and topic= aught to be one | [12:31] |
| pharvey | by nested search, I do mean a page which is executing on the order of 100-200 SEARCHes | [12:31] |
| SvenDowideit | you shouldn't really see a huge pain, as it should be using the index to avoid loading and considering the document
mmm, there's a fun-one i never thought of that :/ if you have 100's of queries on a topic then we shoulda architected this the other way round :( | [12:31] |
| pharvey | hey, I never expected these to perform | [12:33] |
| SvenDowideit | more like pagecache in fact | [12:33] |
| pharvey | the only reason we're doing them, is because they're borderline tolerable :) | [12:33] |
| SvenDowideit | dammit, its my fault for not having seen into your users future! | [12:33] |
| pharvey | and then I tell Paul.A to please use smaller pagesizes | [12:33] |
| CDot | pharvey: easier said than done. The warnings are generated - and counted - server side, but the hiding is client
BTW I wanted to ask; what are the HASH and META leaf types in the infix parser? | [12:33] |
| SvenDowideit | CDot when you hide, count the warnings your hiding
and decrement | [12:34] |
| CDot | CDot thinks they are badly out of place | [12:34] |
| SvenDowideit | >:} | [12:34] |
| CDot | SvenDowideit: you think? woo, too technical for me | [12:34] |
| SvenDowideit | CDot they are ways to get info | [12:34] |
| pharvey | IIRC that was to do with RestPlugin getting topics in/out? | [12:34] |
| SvenDowideit | and they are rather important from my very hazy memory | [12:34] |
| CDot | CDot is feeling particularly sarcastic today | [12:35] |
| SvenDowideit | oh - pharvey bingo
they were the begingins of showing you that query can be used for both read and write and it works well SvenDowideit has had a shite day getting hit by apache rpm and debian crap given the time, i'll give up on the last things i hoped to acheive today (newsletter and a simple micro-app) and goto bed niteall :) | [12:35] |
| pharvey | pharvey does the same
but I can't! promised George I'd fix a unit test | [12:37] |
| CDot | pharvey: ? it looks like it was in support of the query parser
it seems they are used as constants CDot has abandoned changes to the query language to support non-constant indices. Too complicated for today. | [12:44] |
| GithubBot | [foswiki] foswiki pushed 1 new commit to master: http://git.io/MeYi9g
[foswiki/master] Item11366: Avoid using Foswiki::Address in test - PaulHarvey | [12:48] |
| *** | GithubBot has left | [12:48] |
| FoswikiBot | http://foswiki.org/Tasks/Item11366 [ Item11366: Ampersands in =[[squab?queries]]= are double-encoded ] http://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Address | [12:48] |
| GithubBot | [foswiki] foswiki pushed 1 new commit to Release01x01: http://git.io/bSb_xA
[foswiki/Release01x01] Item11366: Avoid using Foswiki::Address in test - PaulHarvey | [13:02] |
| *** | GithubBot has left | [13:02] |
| ...................... (idle for 1h45mn) | ||
| gac410 | Howdy CDot - did you catch my commits last night - had to revert your changes to the <base tag - it was breaking html validations. And somehow EnableHierarchalWebs got removed. | [14:47] |
| ..... (idle for 23mn) | ||
| MichaelDaum: with your new page cache, how do you invalidate when upgrading to a new foswiki or plugin that will change content because of template changes, code changes, etc? | [15:10] | |
| MichaelDaum | changes in template/... and lib/... aren't tracked, only content changes. so when upgrading, you will have to invalidate all of your page cache in case you think this is required using a http://...?refresh=all or the equivalent perl api. | [15:12] |
| gac410 | okay - we need to be sure to get that into upgrade instructions, etc. I'm pretty sure we've had people run into that here with the old cache as well. | [15:12] |
| MichaelDaum | ya, docu white spots | [15:13] |
| gac410 | is the new cache coveed on a FeatureRequest for 1.2?
Or is it a 2.0 thing | [15:13] |
| MichaelDaum | not sure what your timing is for 1.2 ...
I will implement it on trunk only. | [15:14] |
| gac410 | gac410 trying to make sense of a gazillion accepted proposals sprinkled across 1.2/2.0 w/o any idea is for what will be in what or what has committed changes.
Problem is trying to figure out how to branch out a 1.2 without bringing along a mess of half-implemented proposals. | [15:15] |
| MichaelDaum | basically, the page cache rewrite is required part of a thorough bug fix | [15:15] |
| gac410 | yes that I agree. the old cache has had it's problems. | [15:15] |
| MichaelDaum | do you plan to release 1.2 from trunk?= | [15:16] |
| gac410 | well - i'm not even sure if I'll be rm - if Lavr comes back. I get words of wisdom suggesting trunk is too "trunkish" so use 1.1. .as base, and others who are vehement that trunk must be base or it throws everything away. | [15:17] |
| MichaelDaum | the major fixes are in the 1.1.x branch as well. so I guess it is safe to release more 1.1.x from there. | [15:18] |
| gac410 | I'm concerned that we have a LOT of partially implemented stuff in trunk. If we branch from trunk we need to find/revert the incomplete stuff. If we branch from 1.1, we'll start with a more stable base but have to pull in changes and that will be complex. | [15:18] |
| MichaelDaum | I don't see much breakage on trunk, running it all day for my comp. | [15:18] |
| gac410 | right - but it's all the corner cases for stuff that we don't test, but will break sites happy with 1.1
It's not that there is broken stuff - though I know there is. It's the unknown that scares me. | [15:19] |
| MichaelDaum | the only thing that makes me a bit nervous is "store2" ... with Sven saying "everything will be different"
store2 really sounds more like fowiki2 | [15:19] |
| gac410 | yeah - thats off in github and will be applied to trunk once 1.2 is branched. It is massively different from what I gather. The API becomes more more strict. | [15:20] |
| MichaelDaum | foswiki2 | [15:20] |
| gac410 | If Sven got his way, ACLs will be enforced in store and the "ignore permissions" stuff in func will be ... ignored. | [15:21] |
| MichaelDaum | ah okay so we branch 1.2 from trunk, just before store2 gets merged in? is that the plan? | [15:21] |
| gac410 | If I listen to Sven, yes. I Blogged that we would target 1.1.5 for March 31 release, and branch 1.2 shortly after that. | [15:22] |
| MichaelDaum | blog.f.o? | [15:22] |
| gac410 | no Sven told me that is not used. f.o/Blog | [15:22] |
| MichaelDaum | outch
f.o/Blog is absolutely not on the radar of other people out there reading rss n stuff I gave Arthur and Sven a thorough feedback on the foswiki.org/Blog with the baseline: by far not prod ready. they are maintained manually can create an account in a few secs ... want one? | [15:23] |
| gac410 | I guess so - though I think our left/right hands need to get coordinated or we'll look rather foolish. | [15:27] |
| MichaelDaum | you should've receive an email about the registration by now | [15:31] |
| gac410 | nothing yet, but it may take a while to make it through my greylist service. Thanks. | [15:32] |
| MichaelDaum | we might need a board decision about the blog :( | [15:32] |
| gac410 | yeah I was pondering that. And the trunk or 1.1. base for 1.2 branch might need a community vote. There are some very strong opinions. | [15:33] |
| tsnfoo | Hmm. I just copied my data into a fresh 1.1.4 install, and noticed that AutoViewTemplatePlugin doesn't work unless Override is on. | [15:37] |
| gac410 | Do you have a VIEW_TEMPLATE set in a WebPreferences topic? | [15:38] |
| tsnfoo | Which is odd, because Override is off on my 1.1.2 instance, and there are no VIEW_TEMPLATE or EDIT_TEMPLATE settings to override.
gac410: I noticed this a couple weeks ago on my svn/trunk install, but it went way after fiddling with settings. This time I went setting by setting until I noticed this. | [15:38] |
| gac410 | I think we are using it on f.o with override off. checking now. | [15:39] |
| tsnfoo | Very odd. Like 75% of my topics use auto view templates. | [15:40] |
| gac410 | yup - debug and override disabled, mode "exists" | [15:40] |
| tsnfoo | Same here. View templates aren't applied unless override = 1. I've restarted apache, diff'd my confs, etc.
I am running mod_fcgid though, which I'm not on my 1.1.2 install. What does fw.o run? | [15:40] |
| gac410 | mod_fcgid We had one issue in the Support web with view templates not applied. but only on some forms. It was a Set VIEW_TEMPLATE in the WebPreferences | [15:41] |
| tsnfoo | Hmm. My googling hasn't brought that up yet. | [15:42] |
| gac410 | as a bug? I didn't think it was - we just updated WebPreferences and things started working. | [15:42] |
| tsnfoo | Right.
The only places I can find VIEW_TEMPLATE on my wiki is in the docs. Strange. I'll keep gripping and googling... | [15:43] |
| gac410 | I suspect it was a long-standing issue on f.o though because once we fixed the setting, some of the topics needed fxing. | [15:44] |
| .... (idle for 15mn) | ||
| MichaelDaum: I'm registered - thanks. | [15:59] | |
| MichaelDaum | yw | [15:59] |
| .......... (idle for 46mn) | ||
| gac410 | Sure do wish we had a raw diff mode. SvenDowideit wtf did you do to NewApacheConfigGenerator. I can't get it to *not* generate a vhost, so i can't recreate / fix your problem. | [16:45] |
| ......... (idle for 42mn) | ||
| MichaelDaum | with the new page cache enabled cached pages are delivered in about 400-500ms | [17:27] |
| gac410 | nice! | [17:27] |
| MichaelDaum | no matter how long it took to compute it first time | [17:27] |
| gac410 | If you plan this for 1.2, then might want to commit to trunk. Hm... With t.f.o and f.o sharing the same file system will the cache be usable? I'm guessing not.
otherwise changes on one won't invalidate the cache on the other. | [17:28] |
| MichaelDaum | naw. pages look different on both domains.
thats a problem pages are even cached properly using ajax-loaded topic sections using jquery.loader or InfiniteScrollContrib that is: each fragment is cached separately as it should | [17:29] |
| gac410 | cool | [17:31] |
| MichaelDaum | there will be a new cache_ignore urlparam that allows the cache algorithm to ignore certain url params from generating separate page variants
for instance JQGrid adds an nd=random to each rest call while this is fine to trick out browser-side caching, server-side caching can ignore these safely | [17:32] |
| gac410 | so jqgrid has to add a cache_ignore=nd or it's a config param to ignore nd? | [17:33] |
| MichaelDaum | I've added it to the gridconnector url in GRID.pm
so that it works for perl connectors as well as topic connectors same trick might be required for some autocompletion thingy. though ui::autocomplete isn't a candidate of that kind so this does speed up autocompletion like in solr's input box | [17:34] |
| gac410 | too bad ui::autocomplete isn't a candidate. The user search / fill-in stuff can be pretty slow. | [17:39] |
| MichaelDaum | thats because the Foswiki session object is too fat
takes too long to build it up again and again yet still everything is low prio compared to RcsLite::readTo() using the bulk of the time accordign to Devel::NYTProf | [17:39] |
| gac410 | Yeah - improving RcsLite needs to be a priority for 1.2 ... (or even 1.1.5 would be nice) | [17:41] |
| .................. (idle for 1h25mn) | ||
| *** | jurov has left "User accidentally the." | [19:06] |
| ↑back Search ←Prev date (last date) Show only urls | (Click on time to select a line by its url) |