#foswiki 2012-12-06,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***ChanServ sets mode: +o MichaelDaum [07:29]
......... (idle for 42mn)
foswiki_irc3Hi! Yesterday, I asked some questions regarding performance issues in 1.1.6.
Thanks for all the answers!
I'm contemplating MichaelDaum's performance patch but got a bit put off by CDot's comments about the risks of using it on a broken store.
[08:11]
MichaelDaumhi foswiki_irc3 [08:13]
foswiki_irc3Some topics are editied externally but rcs is used to check them in properly. [08:13]
MichaelDaumit _is_ a risk. some unit tests inevitably fail. [08:13]
foswiki_irc3Some small changes are sometimes done manually.
So what exactly is the risk? What happens? And is there anything I can do to check beforehand?
[08:14]
MichaelDaumalas, for a couple of clients it was the only way to get 1.1.5 up and running back on a non-embarrassing scale
the patches will behave just fine when you've been using foswiki regularly with regards to the rcs ,v files
problems might arise if you've got topics with some pathological meta topicinfo
yet still things are quite robust even then
those patches are in use on the 1&1 intranet wiki with about 47G data.
without, foswiki won't receive a ready-to-go-live after migrating from tmwiki
due to performance problems
[08:14]
foswiki_irc3How can I check for bad topicinfo data? [08:19]
MichaelDaummake sure your topics _have_ a META:TOPICINFO line. if not, then they will bring down performance as foswiki will fork rcs for these topics each [08:22]
foswiki_irc3I'm pretty sure they all have this line. The data might not be correct but the line is always there. [08:23]
MichaelDaumokay thats a prerequisite to get back to performance in any case. [08:23]
foswiki_irc3So there is no additional check to be performed before applying your patch?
And how would I notice if something goes wrong afterwards?
[08:34]
jasthow does one enable debugging in foswiki core? I've got a wiki that times out when trying to view SitePreferences
never mind, found it
[08:40]
foswiki_irc3The second point yesterday was the performance of query vs regex. What's the conclusion there? I get very inconsistent results when using forking or NativeSearch.
I'm using fcgi btw.
Forking seems to be faster for regex while NativeSearch is faster for query. Is this possible?
[08:46]
jasthrm, no helpful output :( [08:52]
..... (idle for 22mn)
... and another stupid problem solved [09:14]
............. (idle for 1h4mn)
foswiki_irc3What's the recommended (i.e. fastest) way of search for all topics with a particular form attached?
Btw, I applied MichaelDaum's patch and got the access times back to 1.0.9 level (but not below).$
[10:18]
MichaelDaumcool. yes, I observed the same. [10:19]
foswiki_irc3MichaelDaum: How would I know that something is wrong regarding the patch and a broken store? What would happen? [10:19]
MichaelDaumthough a normal page with those patches and fcgi was well under 1s. so I stopped tuning further. might be 1.2.0 which def has more store fixes, will be on par again. [10:19]
foswiki_irc3My normal pages are now at 0.3s (vs 0.7s before). [10:20]
MichaelDaumfoswiki_irc3, not sure. I did not observe problems in real live. yet still those patches are only the tip of the iceberg of things that got fixed in the 1.2.o branch [10:20]
foswiki_irc3Pages with SEARCH are upt o around 3s - which is very slow. [10:20]
MichaelDaumlike WebIndex [10:21]
foswiki_irc3OK, thanks. [10:21]
MichaelDaummy tip for WebIndex: remove it
same for WEBLIST in the sidebar
[10:21]
foswiki_irc3I guess no-one uses WebIndex anyway. So it doesn't hurt to stay where it is. [10:21]
MichaelDaumWikiGroups might bring down performance on a large user base too
for now I don't have any more perf fixes for SEARCH other than to circumvent its use
[10:22]
foswiki_irc3MichaelDaum: Not using SEARCH is a bit tricky because to me that's one of the major "selling points" of Foswiki. [10:24]
MichaelDaumthere are two possible replacements for it: DBCachePlugin and SolrPlugin
DBCachePlugin comes with a DBQUERY that has got a slightly different syntax compared to SEARCH
that's for wiki apping
SolrPlugin is for fulltext search requirements
[10:24]
foswiki_irc3I'm using SolrPlugin for full-test search already. [10:25]
MichaelDaumSEARCH tries to cover both use cases...and fails on both
ah ok
well then make use of %SOLRSEARCH+%SOLRFORMAT to replace %SEARCH
[10:25]
foswiki_irc3I was the one asking a few weeks back about a new release of SolrPlugin. [10:26]
MichaelDaumMichaelDaum has seen lots of foswiki_irc[1-9]s since then [10:27]
foswiki_irc3I noticed you did some work on it. Is it any closer to being released?
Yes, I know, I guess I should inverst in a real nick... ;-)
[10:27]
.... (idle for 16mn)
MichaelDaum: A question regarding LdapContrib: On my new 1.1.6 setup I get "wrong" WikiNames, i.e. instead of TestUser (as on 1.0.9) I now get TestUser1. Do you know what could causing this? [10:43]
MichaelDaumyou've got two test users in your directory? [10:44]
foswiki_irc3No, the old setup which accesses the same LDAP (AD) it stillw orks OK. [10:50]
MichaelDaumdid the DN of the TestUser change?
that is: did you move the person object to a different tree within your directory?
[10:52]
foswiki_irc3That could have happened over the last year. But why does it still work in the old setup?
Does it cache this sort of information long-term?
[10:53]
CDot(08:58:26) foswiki_irc3: Forking seems to be faster for regex while NativeSearch is faster for query. Is this possible? - anything is possible. Likely? No.
query breaks down to a regular expression search that is done using the same engine and plain text searches
^and^as
[10:58]
foswiki_irc3CDot:: Strange. I seem to get quite consistent results in these two cases.
MichaelDaum: Should I try to delete LdapContrib's cache.db?
[10:59]
CDotit might be something to do with the formulation of the regexes. However since NativeSearch uses the standard regex library, it would seem unlikely
TBH I'd be surprised if forking was faster for *anything*. But if it is, then use it; NativeSearch is designed to avoid the overhead of creating a new process, and if that overhead is negligible on your system there is no strong justification for using NativeSearch.
[11:01]
foswiki_irc3Clarifying my previous statement: It's actually quite hard to get consistent benchmark results on our system. Don't know why but the results vary from run to run. [11:04]
MichaelDaumfoswiki_irc3, you could also try refreshldap=force to renew all WikiNames, not trying to keep wikinames <-> dn stable....
bbiab
[11:05]
foswiki_irc3MichaelDaum_: Thanks, seemed to have fixed it (at least for the moment). [11:07]
CDotfoswiki_irc3: inconsistent benchmarks have always been the bane of foswiki, on all systems. Perl is highly sensitive to the environment, and performance can change enormously based on things like paging demand and cache performance
the best you can do is to do lots of runs and try to average out. Lots of runs.
[11:08]
foswiki_irc3CDot: I seem to remember some problems wrt Native Search but can't rememeber the details (core dumps IIRC). [11:09]
CDotwell, if you ever find them again, post a problem report. Even better if you can post a fix.
CDot hasn't looked at the NativeSearch code in a long time, but it's really very simple.
[11:10]
foswiki_irc3CDot: This was the bug I encountered: http://foswiki.org/Tasks/Item9090
2 years ago...
[11:22]
CDotah yes, I vaguely recall it. I couldn't reproduce it, and couldn't see from the stack trace what was going on - it dies in the depths of perl's complex and convoluted XS code.
an XS expert would probably look at it and say "aha" immediately. No me, however :-(
[11:24]
CDot has gone Christmas shopping. Morituri te salutant! [11:30]
...................... (idle for 1h48mn)
foswiki_irc3PurePerl seems to be faster than NativeSearch on my system (1.1.6 with fcgi). [13:18]
............................. (idle for 2h24mn)
gac410MichaelDaum: I just uploaded Foswiki:Extensions/Testing/PatchItem11983Contrib Please check it out. I've tested it on Foswiki 1.1.5 and 1.1.6. This should hopefully help those with 1.1.5/6 performance issues. [15:42]
FoswikiBothttp://foswiki.org/Tasks/Item11983 [ Item11983: prevent excessive calls into the revision system ] http://foswiki.org/Extensions/Testing/PatchContrib [15:42]
foswiki_irc3I'm trying to replace some SEARCHes with DBQUERY but don't fully understand its syntax. Are there any examples around (beyond the examples in the DbQueryPlugin doc)? [15:49]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/8WRTdg
foswiki/master 4e29611 GeorgeClark: Item11983: Build patch contrib for 1.1.5 / 1.1.6...
[15:51]
***GithubBot has left [15:51]
....... (idle for 32mn)
MichaelDaumfoswiki_irc3, have you got an example SEARCH query that you'd like to use in DBQUERY? [16:23]
...... (idle for 28mn)
foswiki_irc3MichaelDaum: Something like this here: http://pastebin.com/hds8WnAv
Gtg, will read the log.
[16:51]
................. (idle for 1h23mn)
foswiki_irc4I'm trying to set VIEW_TEMPLATE for a subweb in SandBox. But it doesn't work [18:14]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/BdckLA
foswiki/master ff137a2 GeorgeClark: Item12148: Another typo craters configure...
[18:20]
***GithubBot has left [18:20]
FoswikiBothttp://foswiki.org/Tasks/Item12148 [ Item12148: EditRowPlugin improvements ] [18:20]
............. (idle for 1h0mn)
GithubBot[foswiki] FoswikiBot pushed 2 new commits to master: http://git.io/aE5mUg
foswiki/master 57fdb6b TimotheLitt: Item12282: Better continued comments...
foswiki/master 60f5277 TimotheLitt: Item12180: Better continued comments...
[19:20]
***GithubBot has left [19:20]
FoswikiBothttp://foswiki.org/Tasks/Item12282 [ Item12282: Implementation for SpecFileSyntaxEnhancements ]
http://foswiki.org/Tasks/Item12180 [ Item12180: Implementation for AJAXOnDemandCheckersForConfigure ]
[19:20]
...... (idle for 29mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/LKPcdQ
foswiki/master 9283989 TimotheLitt: Item12180: Email upgrade (mostly). This is still WIP, so expect rough edges....
[19:49]
***GithubBot has left [19:49]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/UUrkRg
foswiki/master fff1d5c TimotheLitt: Item12180: MANIFEST; "tis my destiny to do these separately......
[20:04]
***GithubBot has left [20:04]
FoswikiBothttp://foswiki.org/Tasks/Item12180 [ Item12180: Implementation for AJAXOnDemandCheckersForConfigure ] [20:04]
.......................... (idle for 2h8mn)
gac410Hm Historical question... Is there any reason that {RemoveImgInMailnotify} and {NotifyTopicname} should not be moved under the MailerContrib extension configuration tab. They really are not part of the general email configuration.
NotifyTopicName should probably be with most of the other *TopicName settings - under Miscellaneous Expert Settings
RemoveImgInMailnotify is only used in MailerContrib.
[22:12]
....... (idle for 34mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/WQGAPw
foswiki/master 15ad74a GeorgeClark: Item12180: Clarify some configuration...
[22:50]
***GithubBot has left [22:50]
FoswikiBothttp://foswiki.org/Tasks/Item12180 [ Item12180: Implementation for AJAXOnDemandCheckersForConfigure ] [22:50]
..... (idle for 23mn)
wryfii have foswiki running on a mostly read-only system, and i get errors trying to attach a file
Foswiki detected an internal error - please check your Foswiki logs and webserver logs for more information. CGI open of tmpfile: Read-only file system Status: 302
nothing useful is logged
and 'System temporary directory' is set to a writeable location in bin/configure
any thoughts?
[23:13]
gac410Attach uses CGI's tmpfile location. Not sure what it is though.. eating supper. back later [23:14]
wryfiah, thanks ... SetEnv TMPDIR in my apache conf seems to do it [23:20]
hmm, i take that back [23:29]
gac410The ENV won't be used because it's "tainted" Foswiki runs with the -T flag. [23:30]
wryfiargh [23:32]
gac410We've been trying to address this somewhat on the alpha 1.2 release under development. I'm not sure we tackled CGI tempfile. [23:32]
wryfii'm actually using fcgi
looking at foswiki.fcgi now
(but not a perl jockey)
[23:32]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/C91-QA
foswiki/master 7a145b3 TimotheLitt: Item12180: Fix some noise in the logs...
[23:35]
***GithubBot has left [23:35]
FoswikiBothttp://foswiki.org/Tasks/Item12180 [ Item12180: Implementation for AJAXOnDemandCheckersForConfigure ] [23:35]
gac410Taint mode is set by the "shebang" in the first line of the perl script. #!/usr/bin/perl -wT ... (or something like that). The T is what causes perl to not trust the environment, but much more importantly also will complain if any input from the browser / url is not validated. [23:43]
wryfithere's no -wT on foswiki.fcgi
perhaps FcgidInitialEnv ?
[23:47]
gac410Hm... I don't have it installed here on my testing systems. so TMPDIR doesn't do it for you? [23:47]
wryfidoesn't seem to
notwith SetEnv anyway
[23:48]
gac410Looks like FcgidInitialEnv would be worth a try. What version of Foswiki are you running? [23:49]
wryfi1.1.2 [23:50]
gac410okay. I'll peek at some of that code. [23:50]
gigglesworthMy users are authenticated using mod_ldap , with unix-style usernames like "frankm" and "janedoe". When they log in, the first thing they see is 'Main/WebHome". Can I force these newly authenticated users to go to System/UserRegistration , so they are forced to register? [23:53]
wryfino dice with FcgidInitialEnv
hrmph
[23:54]
gac410gigglesworth: There is a NewUserPlugin that can auto-register users based upon info extracted from ldap so they don't have to manually register.
wryfi: one question ... what changed. Was this working at some point?
[23:55]
wryfigac410: don't think it's worked yet in this environment [23:56]
gigglesworthThanks gag410. I see http://foswiki.org/Extensions/NewUserPlugin , which should get me going. [23:57]
wryfigac410: is there a way to dial up the verbosity of foswiki logs? [23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)