#foswiki 2014-06-10,Tue

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/YhcvIQ
foswiki/master 6ac907f GeorgeClark: Item12939: Disable TRACE...
[00:14]
***GithubBot has left [00:14]
FoswikiBothttp://foswiki.org/Tasks/Item12939 [ Item12939: PlainFile logger can crash during rotate of log with invalid timestamps. ] [00:14]
................ (idle for 1h18mn)
***gregg4567 has left [01:32]
........................... (idle for 2h11mn)
gac410 has left [03:43]
................. (idle for 1h23mn)
ChanServ sets mode: +o CDot [05:06]
gregg4567 has left [05:15]
.......... (idle for 49mn)
ChanServ sets mode: +o MichaelDaum [06:04]
................................. (idle for 2h40mn)
GithubBot[foswiki] FoswikiBot pushed 2 new commits to master: http://git.io/ShUsoA
foswiki/master 83a0d6c MichaelDaum: Item12481: improved security settings for rest handlers...
foswiki/master da3911f MichaelDaum: Item12481: replace core copy mechanism with CopyContrib...
[08:44]
***GithubBot has left [08:44]
FoswikiBothttp://foswiki.org/Tasks/Item12481 [ Item12481: working towards 4.0 final ] [08:44]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/rLfuZQ
foswiki/master e5e7a0c MichaelDaum: Item12940: improved error handling; make messages translatable...
[08:58]
***GithubBot has left [08:58]
FoswikiBothttp://foswiki.org/Tasks/Item12940 [ Item12940: improve error handling; make messages translatable ] [08:58]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/GMd7dA
foswiki/master 1dab61a MichaelDaum: Item12865: fully specify all security switches for rest handlers ...
[09:13]
***GithubBot has left [09:13]
FoswikiBothttp://foswiki.org/Tasks/Item12865 [ Item12865: improve rendering of revision info of WikiTopics and derived TopicTypes ] [09:13]
............................................ (idle for 3h35mn)
***ChanServ sets mode: +o gac410
dgretch has left
[12:48]
dgretch1 has left [12:56]
............................ (idle for 2h18mn)
CDotgac410: AAAARRRGGGGHHHH! [15:14]
JulianLevensHello CDot, how are you? (AAAARRRGGGGHHHHs notwithstanding) [15:21]
CDotRepairing slowly, thanks. Still oozing, but no longer in pain :-) [15:21]
JulianLevensGood show [15:22]
CDotCDot is fighting with Configure; what was going to be a bit of light refactoring has turned into a major battle
mostly trying to disentangle the spaghetti added to support REST
[15:22]
JulianLevensYou said on f.o somewhere you really needed two extra developers to work with you on configure. Who
Who else in on the team?
[15:25]
CDotJust gac410, and I haven't shared enough for him to be able to help yet
I really need to tap his expert knowledge of what was changed
[15:28]
gac410howdy CDot [15:28]
JulianLevensHello Mr expert [15:28]
CDotaha, George! Good day to you [15:28]
gac410Good luck tapping my (lacking) knowledge
and good day to you too.
[15:28]
CDotCDot is battling with LogViewerContrib [15:29]
gac410LogViewerContrib ... never heard of it. [15:29]
CDotlet me explain. As I said before, configure had got into a bit of a mess, with pollution of concerns all over
so, I have done a major refactoring
[15:29]
gac410Ah... you are moving it out of configure and into a contrib. Good.,
I actually don't have a problem if you just dropped it. It's nice but was not part of a proposal for 1.2
[15:29]
CDotI have split the "provideFeedback" stuff into a separate type of thing (called a "KeyHandler")
and left the Checkers as just that - simple checkers, with no knowledge of the calling environment
this means that simply calling the checkers will do just that; check. it will not guess, will not deduce, will not probe
[15:30]
gac410IMO the most important provide feedback was the email check stuff. And that ought to be a rest handler from a Mail configuration topic, and not part of the monolythic config [15:31]
CDotit just checks Foswiki::cfg seems to have sensible values [15:31]
gac410Yeah... makes sense.
Any guessing should be part of a bootstrap IMO
[15:31]
CDotso, I haven't actually broken anything, just moved a lot around
the "guessing" only happens under user control, now
so, if you have an error, a "try to help" button would be in order
CDot hasn't got that far yet
but the LogViewerContrib - wherever it came from - and the Email stuff are both nightmarish
[15:31]
gac410I really don't know what LogViewerContrib is. [15:33]
CDotvery complex, no clear intent
ok. I didn't look to see where it came from, I just assumed it was you
[15:33]
gac410I merged a log view API (uses eachEventSince) into Configure. It wasn't as separate contrib.
It was based upon an email from Timothe.
[15:34]
CDotok, sorry, I'm confusing it with something else
it's not a contrib, I can see that
anyway, I'm still not 100% sure what "provideFeedback" is supposed to do
can you describe it, like I am a five year old?
[15:35]
gac410Email was very straight forward. Given a server name. Try in order of preference (Submission, SSL, and plain old SMTP ) ... first with validated certs, then without. Finding the most secure connection that works. Then test that with the provided user/pass. And fall back to mailprogram if nothing else works. [15:36]
CDotit seems to have several different functions, selected under the control of an (undocumented) integer parameter [15:36]
gac410provideFeedback? or email check? [15:37]
CDotprovideFeedback
CDot understands email checking, mostly, though the keys.... whoa, complicated
I'm trying to get a 3000 metre view of the role of provideFeedback
[15:37]
gac410okay. I didn't dig into the internals of it. But it's intention was to have a way to ask the server to do some additional checking and give results beyond the simple check. Not sure about the undoc integer. What's its name ... I'll poke at it. [15:38]
CDoterm...
'$button'
at least, that's one of them
[15:38]
gac410Okay let me check. [15:40]
CDotthere's some explanation at the head of Feedback::deliverResponse, but I don't pretend to understand it
there's this thing called a "Feedback cart" that I just can't fathom. Seems to be some sort of shopping cart concept.
[15:41]
gac410Ah... maybe....guessing until I look. I think that some items can have multiple buttons. Like the Configuration Audit. Maybe the $button indiciates which of multiple buttons were requested?
The only cart I'm aware of is the cart of config changes. Everythng is saved in a cart until you explicitly list save. Checking is done against the cart contents without actually saving the configuraiton
That's probably the most important feature. I've knocked down multiple sites because the save to find errors found a critical one. :(
So error checking of the *complete config* without saving it is probably one of the more fundamental changes.
[15:42]
CDotok, that sounds sensible; however, the code doesn't appear to do that
CDot must be missing something
[15:44]
gac410Well maybe the feedback cart doesn.t but the checking of the whole config without save indeeds works. [15:45]
CDotyes, sure, that's OK.
the cart, though.....
[15:45]
gac410The auto-check fields. When you type into them and tab away, it submits the whole config for validation. The changes stay in the cart even if you move away from configure and come back later.
So the cart provides persistence of your changes while you get interrupted, browse around resolving check errors, etc.
[15:46]
CDotgood grief. OK. [15:46]
gac410So the cart itself is probably wayyy overkill. Convenient to not lose changes when you get interrupted, firefox crashes, etc. But not particularly earth shattering. [15:47]
CDotright now, I have a working configure again - at least, working insofar as it runs the simply checkers. I need to decide what to do next, what is worth trying to save and what is - as you put it - waayyyy overkill [15:48]
gac410Timothe was making the changes to create a complicated event scheduler. So configure was becoming a framework for a completely differnet application iirc. [15:50]
CDotwell, that would explain some of the weirder code I'm finding
the JS code is completely opaque. I haven't even looked at a lot of it
[15:50]
gac410Okay for at least one checker. DATE - The $button is a boolean. 0 - called from Check. 1 - called from a button press [15:51]
CDot"called from Check"? What is that? [15:52]
gac410Sorry ... the Checker default routine. DATE makes it easy to see. And EMAILADDRESS documents it further:
# Button 1 = Test syntax (should be check-on-change)
# Button 2 = Test mail to address (should be a button)
[15:52]
CDotdo you mean "called from the same code that calls sub check"? Or maybe "called by sub check"? [15:53]
gac410Called by sub check. [15:53]
CDotok. That's incredibly bad practice, so it's gone. So is button = 1 always "test syntax"? [15:53]
gac410DATE::check() calls DATE::provideFeedback() with button 0. If a button was pressed I assume core of configure calls provideFeedback with some other value. [15:53]
CDotIIRC I got that far as well, but conuldn't find *where* it was called [15:54]
gac410I doubt it (button 1 always == test.) It's probably going to vary by checker [15:54]
CDotthere seems to be a very complex and delicate arrangement of inter- and cross- calls with different controlling integer values [15:54]
gac410Configure::Feedback calls provideFeedback with a button ID.
Yeah. very old school
[15:55]
CDotAnways, I decoupled the providefeedback from the check completely, so they have to be separately called explicitly now, which makes things a bit clearer
but, what do we want "provideFeedback" to *do*?
(1) react to a button press to provide deeper checking
(2) react to a button press to provide guessing
(3) something else (or both the above)?
[15:55]
gac410It really is very task driven. Depends on the config item. ... (3) Both of the above I guess. [15:57]
CDotok. in that case I'm going to try and identify a set of handlers that correspond to the task operations [15:57]
gac410So for the URL parameters, for ex, the check provides deeper validation including probing the server to determine if the path works. [15:57]
CDotyes
my question is more one of "what operations do we want to expose to the user"?
[15:58]
gac410For email it provides an auto-configure "wizard" function on one field. On EMAILADDRESS it sends a test email [15:58]
CDot"deep check" and "guess" are the current candidates [15:59]
gac410I don't think there are any that should just go away. They are all valuable. Maybe "wizard" more than a simple guess [15:59]
CDotthere is no wizardry (as in a Q&A) that I have found [15:59]
gac410Ah... yeah No q&a. But to me a simple guess guesses one config variable. A wizard will configure a range of interrelated settings. No q&a but maybe with muliple fields of input. [16:00]
CDotright, agreed. And it makes sense to separate these 'wizards' and handle them separately, not conflated with Checkers
that makes a lot more sense to me now, thanks
now I just have to dig around in the code and find what's a wizard, what's a deep check, and what should be a simple check
[16:01]
gac410yup. But one thing missing is some way for the config file to be updated outside of configure in a reliable way. That could be a path toward updates from Extension installer (enabling a plugin, populating the spec)
The deep checks are the Path, and {view} path.
The Config audit. Very thorough job of checking all dependencies, There are multiple version.
[16:02]
CDotbottom line (in my mind) is that FW should have a "fallback" mode, so that it *always* runs, even without LSC (albeit hamstrung) [16:04]
gac410That's where I was going with the Bootstrap classes. [16:04]
CDotthe config audit.... misses almost all the important dependencies, AFAICT [16:04]
gac410Nope. There are multiple audits.
Hang on let me look
[16:04]
CDotcan you check the doc, and make sure it corresponds with your understanding? It didn't seem to be consistent with the code, when i looked [16:05]
gac410There are 3 audits. Webserver Foswiki and Extensions. Each do a different set of dependencies. [16:06]
CDotI'm not convinced we need Bootsrap classes; just siensible defaults.
3 audits? Oh, my.
[16:06]
gac410Acutally there are 5. Cursory and Extensive ... Those are the checkers on all the fields. But not including dep checking. [16:07]
jastis anyone actually going to _use_ those? [16:07]
CDotCDot consider which wrist to slash first [16:07]
jastpersonally I don't see myself clicking _any_ 'audit some things' button [16:07]
gac410Timothe's concern was that running all checks on every transaction was slow. That's what 1.1.x does. So he refactored them out into the buttons. [16:08]
jastand it just so happens that trunk is still slow [16:08]
gac410The data dir permission checks can take a LONG time on initial config. [16:08]
CDotconfigure in trunk is staggeringly slow. Not sure why
even with my simplified checkers it's still slow
[16:08]
jastbecause each time you move your mouse it uploads all settings to the CGI? :) [16:09]
CDotCDot hasn't looked at the net traffic [16:09]
jastand also pushes them around with JS [16:09]
CDotyeah, JS was my number one suspect [16:09]
gac410Moved out a bunch of stuff, and then added in more. I don't find it too slow mostly. Some of it is browser / js engine dependent. Extremely slow on one platform, lightning quick on another [16:09]
jastis there anything other than permission checks that takes a long time and seems worth keeping? [16:10]
gac410The speed issue is in js I suspect moreso than on the server side. [16:10]
jastprobably, but the POST requests take quite some time, too [16:10]
CDotthe major slowness is the initial GET [16:10]
jastand there's plenty of horrible horribleness in the whole REST area [16:11]
CDotCDot split the GET in ConfigurePlugin, so it only GETs what it needs [16:11]
jastif we hard-depend on javascript, that makes sense
though you do have to read all the spec files anyway, right? on each GET request?
[16:11]
CDotCDot caches them [16:12]
jastbecause you don't know what crazy schemes people have put in them [16:12]
gac410Yes. I think so. Though that's not a huge slowdown. [16:12]
jastyeah [16:12]
gac410The slowest check on 1.1 was the file system permissions. Takes FOREVER on windows especially [16:12]
CDotso, is all this audit stuff needed? [16:12]
jastregarding REST and JS, I honestly believe the best thing to do would be to rip it out and start over
I agree separating the filesystem check might be a good thing to do
[16:12]
CDotjast: yes, I agree. But I want to understand *what* it needs to do [16:13]
jastI've been thinking about it a little [16:13]
CDotthis audit stuff is mega-complex, and a potential nightmare to maintain. Is it really worth it? [16:13]
jastI don't think so [16:14]
gac410CDot: Many of the checks done in old configure only exist in Audit. So it might be better to have it done in a separate REST maybe [16:14]
jastmake one specific impl for filesystem checks, add a button, rip out all other audit things [16:14]
gac410I guess I don't know what you mean by not worth doing. Not worth checking for missing dependencies? Not worth checking for file permission errors? [16:14]
jastall the quick things can be done when on-demand-loading individual sections of configure
here's what I'd implement:
[16:14]
CDotCDot isn't sure what other things there are. There aren't that many "real" provideFeedback implementations [16:15]
jasta 'tree' operation that returns the tree of items, but without values. hopefully fast.
a 'values' operation that returns all values for a specific node of the tree
[16:15]
CDotok, so: filesystem tree checks, and perl module dependencies. Anything else?
jast: you are describing ConfigurePlugin :-)
[16:15]
jastI thought so :)
an 'update' operation that... updates...
[16:16]
gac410So for audits. Visit the Audit page, and run the 3 Web / Foswiki / Exten audits. The results are reported below the Analysis results heading. On occasion I've missed that there are even result there. [16:16]
jastand then extra operations for special stuff [16:16]
gac410Have to scroll down to see them. [16:16]
CDotCDot preceives a need for the .spec to declare out-of-the-obvious dependencies. For example "my plugin needs working email" [16:17]
jastCDot: 2.0 ;) [16:17]
CDotmaybe leave to 2.0, yes
CDot still wants the decisions made in 1.2 to not block that concept
[16:17]
jastif we do keep an audit option, make it a single button that audits everything
not blocking it seems sensible, yes
[16:18]
gac410Any field flagged as regex gets a special check as well. That's one that I've killed foswiki using. [16:19]
jastwhen I say "audits everything", I mean "everything that takes too long to do automatically in the background"
we can make background checks much more realistic by changing the one big horrible thing Timothe didn't address: the browser always uploads *all* values
[16:19]
CDoty [16:19]
gac410Yes, I much prefer a single audit function, available through REST, maybe a superadmin only topic. [16:19]
CDotok, this is getting a bit clearer now, thank you [16:20]
jastthat's totally unnecessary. we can keep track of what the user changed within the JS (we have to do that anyway to toggle stored/current/default value), and send only changed things [16:20]
gac410Yes jast that is indeed the one HUGE missing issue. Configure doesn't maintain a server side state of pending changes. [16:20]
jastI don't even think it needs to be server-side [16:20]
CDotConfigurePlugin supports *checking* of a set of potential changes, and *saving* of a set of changes [16:21]
jastunless we think it's necessary to handle conflicts due to two admins simultaneously changing the same options [16:21]
gac410As long as configure sends ALL changed fields, yes that's true. The issue is checkers on the server side though that change other fields based upon client side changes. [16:21]
CDotbut does not cache changes server-side [16:21]
jastit has to send ALL changed fields, yes, of course [16:22]
CDothow real is the "two admins" problem? [16:22]
jastunreal IMO
the list of changed fields is MUCH smaller than the list of all fields
[16:22]
gac410As we move parts of configure into accessible via Web/Topic pages, It's also very important to restrict to a subset of the AdminGroup. Wiki admins are different from Server admins IMO [16:22]
jasteven if someone changes 20 options, that's still nothing [16:22]
CDotindeed. However, the set of checks is potentially much bigger than the set of changed fields [16:22]
jastI wouldn't really move much into web/topic
anything that is useful when foswiki isn't working correctly should stay outside that
[16:23]
gac410I thought the vision for ConfigurePlugin was eventually to move a lot of it to topics. [16:23]
CDottopics? no, never [16:23]
jastmaybe that's someone else's vision, but not mine [16:24]
CDotCDot has actively moved stuff *out* of topics, for years now [16:24]
gac410Another feature I think is really important is to be able to eliminate the sudo user. Auditors get really prickly about shared secret passwords (called a back door) [16:24]
jasttimothe has kind of addressed that
if configure uses apache auth, simply don't let the user set a password
[16:25]
gac410yes. Don't want to lose it during the changes. [16:25]
jastand if it doesn't, force the user to set one [16:25]
gac410Ah... Not "don't let" just "don't require" [16:25]
CDotCDot restricts access to the ConfigurePlugin rest handler similarly [16:25]
jastwell, the exact rules don't concern me too much [16:26]
gac410Yeah. Just permit the site to run with the sudo password removed. That's already there and is fine as is [16:26]
CDotanyhoo, back to basics: here's what I think the fundamentals for 1.2 should be: [16:26]
jast- redo JS/REST [16:27]
CDot1) Foswiki always runs, even with no LSC, using sensible defaults [16:27]
gac410+1 [16:27]
CDot2) LSC, if present, can be trivially checked using REST access to checkers
3) =configure= lives, but with a single AUDIT button that does everything
otherwise it reverts to simple checkers
the current configure UI gets heavily simplified, much JS gets dumped
4) we support the wizards, but only if it's possible without a major gut bust
[16:27]
jastI haven't actually seen 'the wizards;
', I think. you mean the mail stuff for example?
[16:30]
gac410After spending another 4 hours helping someone get email running just the other night on IRC, I make the email autoconfig an absolute blocker for 1.2. [16:30]
CDotsingular, I think. Email configuration, with certs etc
can't we abstract the email autoconfig outside of =configure=?
[16:30]
jastall the better. singular doesn't need nearly as much design work :) [16:31]
gac410The only other important one. is the stuff that probes for the valid url config. The same site with email troubles also had their short urls' misconfigured. [16:31]
CDotURLPATH? [16:32]
gac410General path settings page, Verify buttons [16:32]
CDotCDot still can't connect buttons on the configure page to the code they run :-(
ok, anyway, thank you for this discussion. I will bear it in mind when I dive back in to the code tomorrow. I can't promise to save the wizard, but I'll do my best.
Expelliarmus
[16:32]
gac410I just don't understand the email issues. I've tested that autoconfig stuff on Postfix, Sendmail, two differen Exchange versions, and it's "just worked"
User clicks autoconfigure and ends up with a valid email confiuration
The URL path checking is generalized in Foswiki/Configure/Checkers/ScriptUrlPaths.pm
So Foswiki::Configure::Checkers::ScriptUrlPaths::provideFeedback() calls the path testing code for Button 2.
[16:35]
FoswikiBothttp://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Configure::Checkers::ScriptUrlPaths [16:39]
.... (idle for 18mn)
CDotthe email stuff is fine. the problem is the implementation; it's so tangled and twisted up with lots of other stuff that it will take some work to abstract. I'd prefer to pull it out of configure altogether, truth be told, but I will try to save it. [16:57]
.... (idle for 18mn)
gac410CDot If you want, let me take the task of restructuring the email / ssl configuration tool. I agree that it would be best outside of configure, especially if a rest handler can save the results. [17:15]
I'd like it to ask some basic "requirements" questions first, and then do the checking / configuration from there. [17:23]
............... (idle for 1h10mn)
Jast, CDot, Some discussion of and documentation about the Email configuration wizard here: http://foswiki.org/Development/PendingConfigureWork#Email_wizard
This is basically what it does today, although the configure tab interface makes logical organization very difficult.
Do I validate the TO server (SSL Server cert. validation), do I sign the FROM messages (Client cert on fw server). How can I connect (starttls, tls, ssl, smtp) By (IP or IPv6) and how do I authenticate (PLAIN, LOGIN, NTLM)
And what port (587, 465 or 25) Easy-peasy
[18:33]
..................................................... (idle for 4h21mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/hJjN8A
foswiki/master 16449f8 ScottHoge: Item12938: updates to LatexModePlugin, to remove use of 'defined'...
[22:59]
***GithubBot has left [22:59]
FoswikiBothttp://foswiki.org/Tasks/Item12938 [ Item12938: Plugins use deprecated =defined %hash= syntax ] [22:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)