#foswiki 2017-10-09,Mon

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
GithubBot[distro] gac410 pushed 1 new commit to Item14506: https://git.io/vdglY
distro/Item14506 e1f2e16 George Clark: Item14506: Add Token Authentication...
[00:38]
***GithubBot has left [00:38]
FoswikiBothttps://foswiki.org/Tasks/Item14506 [ Item14506: Implement ImprovePasswordResetProcess ] [00:38]
GithubBot[distro] gac410 pushed 2 new commits to master: https://git.io/vdgl0
distro/master 312137d George Clark: Item14481: Remove tinymce_dev, prepare to convert to submodule
distro/master 17aaf9f George Clark: Item14481: Add tinymce_dev submodule
[00:44]
***GithubBot has left [00:44]
FoswikiBothttps://foswiki.org/Tasks/Item14481 [ Item14481: Remove tinymce_dev code and replace with git submodule ] [00:44]
......... (idle for 42mn)
GithubBot[distro] gac410 pushed 1 new commit to master: https://git.io/vdg4q
distro/master 010fc15 George Clark: Item14323: Update to TinyMCE 4.7.0...
[01:26]
***GithubBot has left [01:26]
FoswikiBothttps://foswiki.org/Tasks/Item14323 [ Item14323: Update to latest TinyMCE version ] [01:26]
............................................................. (idle for 5h3mn)
GithubBot[distro] MichaelDaum pushed 1 new commit to Item14288: https://git.io/vdgVn
distro/Item14288 b9b8e9a MichaelDaum: Item14288: Merge 'origin/master'
[06:29]
***GithubBot has left [06:29]
FoswikiBothttps://foswiki.org/Tasks/Item14288 [ Item14288: rewrite to support pluggable edit engines ] [06:29]
***ChanServ sets mode: +o MichaelDaum [06:29]
....................... (idle for 1h50mn)
ChanServ sets mode: +o Lynnwood [08:19]
...... (idle for 29mn)
GithubBot[ImagePlugin] MichaelDaum pushed 1 new commit to master: https://git.io/vdgMC
ImagePlugin/master 58bc148 MichaelDaum: Item14509: fixed broken filter parameter...
[08:48]
***GithubBot has left [08:48]
FoswikiBothttps://foswiki.org/Tasks/Item14509 [ Item14509: broken filter parameter ] [08:48]
......... (idle for 42mn)
GithubBot[LdapContrib] MichaelDaum pushed 1 new commit to master: https://git.io/vdg9m
LdapContrib/master ce85cf0 MichaelDaum: Item14511: fixed encodig errors in dn
[09:30]
***GithubBot has left [09:30]
FoswikiBothttps://foswiki.org/Tasks/Item14511 [ Item14511: encodig error in dn results in failing logins and missing group membership ] [09:30]
..................................... (idle for 3h2mn)
***ChanServ sets mode: +o gac410 [12:32]
......... (idle for 43mn)
FoswikiOnSlack<gregor.godler> hey guys, when is enabled ldap server for authentication, this message is displayed: Can't locate DB_Filepath in @INC (you may need to install the DB_File::Lock module) (@INC contains: path path path-linux-gnupath path path-linux-gnupath path path-linux-gnupath path path path-linux-gnupath-base . path)
<gregor.godler> in LoginManager is set LdapApacheLogin
[13:15]
jastmissing Perl dependency, you need the Perl module DB_File::Lock [13:17]
FoswikiOnSlack<gregor.godler> ok, maybe do you know why I get this message when I run cpan install DB_File::Lock
<gregor.godler> Can't use 'defined(%hash)' (Maybe you should just omit the defined()?) at test.pl line 84. not ok 1 Makefile:806: recipe for target 'test_dynamic' failed make: *** [test_dynamic] Error 255 DHARRIS/DB_File-Lock-0.05.tar.gz make test -- NOT OK
[13:20]
jastoh. well, that's a little tricky. the newest versions of Perl became stricter about a specific piece of syntax the tests are using. you can *probably* bypass it: "cpan notest install DB_File::Lock"
there's a chance the module itself uses the same outdated syntax, though, in that case this is not going to work, so then might be a good time to get the maintainer of LdapContrib involved
but can't hurt to try first :)
[13:23]
FoswikiOnSlack<gregor.godler> I was able to install perl module with help from sh script (https://gist.github.com/vifo/2718520) [13:27]
jastnow the real test is: does it work? [13:28]
FoswikiOnSlack<gregor.godler> i'm not that lucky, wiki works, but when I click login, I get this:
<gregor.godler> Action "viewauth": viewauth requires authentication
[13:30]
jastthis is _probably_ a configuration issue
by setting LoginManager to LdapApacheLogin, you're telling Foswiki that you are handling the LDAP authentication yourself in your web server, i.e. you have an external config for authentication in place (e.g. using Apache's mod_authnz_ldap)
I think the message you're getting says that either that config is not there, or it's in the wrong place
[13:39]
FoswikiOnSlack<gregor.godler> yeah, apache has authnz_ldap.load
<gregor.godler> in config file I have
<gregor.godler> AuthName "Sinergise wiki" AuthType Basic AuthBasicProvider ldap AuthLDAPURL "ldap://ldap.xxx.com ldap2.xxx.com/dc=xxx,dc=com?uid?sub?(&(objectClass=xxxAccount)(hasInternalWikiAccess=TRUE)(!(accountDi$ Require valid-user
[13:54]
jastyeah, that looks fine. Can you post the complete config in a pastebin (e.g. gist.github.com)? Feel free to remove incriminating details, I just want to see where the auth settings are, sometimes finding the right place is a little tricky [13:58]
FoswikiOnSlack<gregor.godler> sure, 1 minute
<gregor.godler> here: https://gist.github.com/cowboy6/0d9a301c51593fae6c15d4836fdc4583
<gregor.godler> here is LocalSite.cfg: https://gist.github.com/cowboy6/9cd61d54ef5456a4abe6fc562c36e0f4
[14:04]
.... (idle for 17mn)
vrurgMichaelDaum: ping? [14:26]
jastgregor.godler: both of the config files look fine. have a look at your Apache error logfile and see if there's anything relevant and/or informative in there...
oh, actually I do see a potential issue in your Apache config
you have a <RequireAll> block a little further up, and the "Require valid-user" is separate. I'm not 100% sure how Apache handles that case, but to be safe you may want to move the "Require valid-user" into the <RequireAll> block (you can remove "Require all granted" which is a no-op after you add "Require valid-user"
[14:27]
gac410jast, in <Directory "${foswikiroot}/bin"> ... there is a "require all granted" Shouldn't it be actually matching the scripts *auth ... and require valid user for them. [14:29]
jastit's a matter of taste whether you want to limit it to *auth scripts. I've seen a fair number of implementations that just blanket-require auth on everything which I guess is reasonable for an internal wiki [14:30]
gac410eg;    #<FilesMatch "(attach|edit|manage|rename|save|upload|mail|logon|.*auth).*">  #    require valid-user #</FilesMatch> [14:30]
jastfor a wiki that is public at least in part, of course the narrower match is important [14:31]
gac410Right. If you want guest access, then you need the FilesMatch otherwise just a blanket require
Oh... and if you are using fastcgi, then it's a LocationMatch, not a FilesMatch
[14:31]
jastthis config is for regular CGI, though
SlowCGI, I suppose :)
[14:33]
FoswikiOnSlack<gregor.godler> yeah :slightly_smiling_face:
<gregor.godler> but it works
<gregor.godler> maybe I will go to fastCGI
[14:34]
gac410one thing at a time :D [14:37]
zak256gac410: You told me latest Perl versions have improved unicode performance. Do you have any resources for that I can give to $boss when requesting time for an upgrade from Perl 5.16 ? [14:37]
jast5.16 contains virtually all of the core Unicode fixes [14:37]
zak256fixes? so not performance but errors? [14:38]
jastinitially Unicode support was a little incomplete, but I think 5.16 has that all ironed out [14:38]
vrurgzak256: I guess perl 5.26 delta has the info on unicode too. [14:38]
jastcan't speak about performance [14:38]
gac410hm. We don't have anyting official, but our experiences show that regular expression performance is considerably faster
On newer perl. much faster.
[14:38]
zak256Newer than 5.16 ?
I don't want to say "Perl 5.24 will make everything much faster", then spend hours in an upgrade and afterwards nobody sees a difference...
[14:39]
jastare there specific things that seem "too slow" right now in your wiki?
unicode regex performance isn't going to affect everything to the same degree, even if there are significant boosts in newer perls (I don't know anything about that, so this is purely hypothetical)
[14:41]
zak256Well... I setup DBCache with the help of MichaelD and that did something. But we noticed format expressions within searches/dbquerys are expensive.
Something like: $percntCALC{\"$LISTMAP([[$dollaritem][$NOP($perFORMFIELD{\\"Name\\" topic=\\"$dollaritem\\"}$per)]], $formfield(Contacts))\"}$percnt
we use something like that extensively
[14:43]
.... (idle for 15mn)
gac410zak256: It's really hard to say what the real world results will be. For simple pages, there really is not much difference. But a page that requires large amounts of regular expression processing, newer perl is better. But that's really subjective. [14:59]
vrurgzak256: I tried to find more info but there is no confirmation for higher regex perfomance on 5.26. There is significant improvement on the cost of subroutine signatures but Foswiki doesn't use them extensively. [15:00]
gac410I'm not having any luck finding any topics where we benchmarked perl performance. I know we did some testing, but it's lost in the wiki. [15:00]
zak256Does the expression above use a lot of regexes? I suppose not?
Ok, then nevermind. I will just tell that a newer perl might improve performance and see where this goes.
[15:00]
gac410It's down in the foswiki render pipeline. We use regexes for parsing the pages for macros, tokens, etc. In some cases we've rewritten code to use other techniques to try to avoid the costly ones. [15:01]
zak256Ok, that sounds it might indeed help some. [15:03]
vrurggac410: Somebody told me that grammar-like regexes are now very nice behaving, especially with /n option. Damian Conway even managed to write a Perl parser as a signle regex: https://metacpan.org/pod/PPR
But their effectiveness is best seen on big grammars; i.e. when split in many small regexes it doesn't provide much of a speedup.
[15:05]
foswiki_irc7Hi, I've got problems with the bulk_copy.pl migrating from foswiki 1.1.3 to 2.14 [15:18]
gac410Hi foswiki_irc7 When migrating from old foswiki, you might be better off using the CharsetConverterContrib. bulk_copy is not very forgiving or data/topic issues. [15:19]
foswiki_irc7The script is printing several perl errors regarding the old server.
It doesn't even start ...
It ends with several errors in configure/Load.pm
[15:19]
gac410Are you tryhing to run a copy of Foswiki 1.1.3 on the new server with a new perl? [15:21]
foswiki_irc7Is it a problem, that on the new machine I've got perl 5.24 where on the old server I've got perl 5.10 [15:22]
gac410Unfortunately there will be no way to get Foswiki 1.1.3 to run on perl 5.24. There are a lot of perl issues in the old foswiki. Perl language has changed. [15:23]
foswiki_irc7I'm only trying to run buk_copy.pl with the new perl, yes.
Oh shit...
[15:23]
gac410You'd have to upgrade from foswiki 1.1.3 to 1.1.10 first, But CharsetConverterContrib is a much better solution. [15:23]
foswiki_irc7does CharsetConverter also help me to get rid of RCS?
and to change to plainfile?
[15:24]
gac410With it, you don't have to run Foswiki 1.1.3. You copy your data unmodified into your new Foswiki 2.1.4 installation. And then run CharsetConverter to convert to unicode / utf-8
No. CharsetConverter only converts to unicode/utf-8. But once you get your data converted, you could then use bulk_copy to convert the store.
[15:24]
foswiki_irc7Thanks, I'll try that! [15:25]
gac410Just be warned that bulk_copy depends upon the old RCS history to be pretty much perfect. And things like auto-attachments, and hidden attachments ( .dot files, and _hidden files, and subdirectories ) are skipped. [15:26]
foswiki_irc7at least I have none of theese... ;-)
but I'll try CharsetConverter on the new server
[15:26]
gac410You want to be certain you never convert a web twice. Once it's utf-8, converting it again will corrupt it.
And best to tell charset_converter that your old encoding is cp-1252. That is assuming you have windows users who might have cut/pasted into topics while editing.
[15:27]
foswiki_irc7thx. I've read that
Yes, I have windows users...
[15:28]
gac410There is a hidden twisty section at https://foswiki.org/System/UpgradeGuide#Alternative_2:_Convert_data_using_CharsetConverterContrib that gives step-by-step
CharsetConverter is **much** faster, bulk_copy has to copy each revision, one at a time. On topics like WikiUsers, and WebStatistics, that can be 1000's of revisions.
So when you eventually do run bulk_copy, you might consider killing off the *,v files at least for topics where you don't need/want history. That will speed it up.
[15:30]
zak256I guess RcsLite is preferable over RcsWrap? Installed rcs is no problem but I don't know which is faster.
Right now I selected RcsLite, but sometimes it takes longer to load when clicking on "Edit". Don't know if that is related though.
[15:37]
gac410zak256: as usual, "it depends" :D for edit, rcs lite vs. wrap should not matter. You are not accessing the older version of the topic. [15:38]
zak256ah... of course. [15:38]
gac410rcsLite is preferred for mod_perl and fastcgi. rcsWrap forks, which is expensive. But generally the compiled C code in system rcs, will be faster than the perl code when processing a history file. [15:39]
zak256Hmm... maybe I will just make some tests. [15:40]
gac410rcs in general, and rcsLlite in particular are very bad with attachments, especially binary attachments, where a "diff" doesn't make sense.
PlainFile has some downsides. It's fast, but you must never "touch" the txt files themselves. They use the file system modified date/time as part of history. Modifying the file timestamp can do bad things.
[15:40]
zak256Yes... I skipped migrating to PFS for now. [15:42]
gac410zak256: Here is one example: https://rt.perl.org/Public/Ticket/Attachment/577590/274332
perl regex 100,000 times slower on utf8. I just ran the testcase locally. My system is probably faster. But perl 5.18.4 (3 seconds execution time for utf-8) perl 5.20.2 (0 seconds execution time)
Need to switch the code to Time::HiRes to see the actual differences
[15:54]
zak256Wow... that's a lot. [15:56]
gac410The original reporter was getting 10 seconds vs. 0 seconds.
For this particular testcase, I switched to Time::HiRes, and spot checked 5.20 thru 5.27, and the differences for utf-8 vs non-utf-8 re negligable.
The big jump is 5.18
[15:57]
zak256I get utf8=OFF: 0 seconds vs. utf8=ON: 5 seconds [16:01]
gac410right. if you add use Time::HiRes qw(time); at the top, you'll get lots of decimal places. [16:02]
zak256Altough that sure does not really tell me how big the improvement will be to our wiki [16:02]
gac410hm... well maybe. Regexes can be very tricky. case insensitive vs case sensitive. anchored^ $ vs unanchored, etc. [16:03]
zak2566.69956207275391e-05 seconds vs. 5.01182198524475 seconds [16:03]
gac410Interestlying I just tried perl 5.10.1 and it was 16 seconds vs. 1 second. [16:05]
zak256did perl 5.10 have utf8 support?
does
[16:05]
gac410Yes, but there are things broken in it. Our unit tests will pass on perl 5.8.8 But it's really strongly recommended to not use that old a perl. [16:06]
zak256We will have perl 5.16 at least, that's for sure.
5.16.3
[16:06]
gac410On perl 5.16.3, I get 1.05 seconds vs. 4.57 seconds
Perl 5.20.2, it's 0.00012 seconds
So... will this make a difference in what the user sees for response time. It's really hard to say. It depends on the size of the page, and the number of regexes executed, and their actual structure.
[16:08]
zak256Is there maybe another way to speed up formulas like the one I mentioned above? with CALCULATE, LISTMAP, IF, and so on... [16:13]
gac410I really don't know.
%CALCULATE will have better performance than %CALC ... just because of how it's found in the page. registered macros are handled differently internally than the commonTagsHandler that CALC uses.
But again, I can't say if that's .0001 % better or 1 % better or ...
[16:14]
zak256Well, I will probably have to rewrite a lot of searches anyway for DBQUERY, I can then replace CALC with CALCULATE as well
And thank god for sed of course :-)
[16:18]
gac410hm If you change the txt file external to foswiki, you should also "touch" the txt,v so the ,v is just slightly newer than the txt file. iirc that makes a difference internally when Foswiki decides whether or not to trust the TOPICINFO in the txt file. [16:23]
zak256good to know. But which cases are these where this is relevant? I modified txt-files quite often in the past directly in the filesystem but didn't see any issue so far. [16:25]
gac410At one point it would improve performance a bit. Foswiki has to decide if the .txt file is trusted, or if it should ask RCS what's the current version. I don't really remember the specifics. [16:26]
zak256Ok, I will keep that in mind. [16:26]
gac410But any calls to ask rcs for history is going to be slower than just reading the txt file. [16:26]
zak256Does that mean there always should exist a ,v file as well?
Even if there is only one version of a topic?
Because we have some pages which are completely auto generated and don't even need a history.
[16:28]
gac410For topics foswiki created, yes. You will always have a ,v file. But topics we ship - most of System web, you will not have a ,v file. [16:29]
Anyway, there is no performance penalty for a missing ,v rcs history, but there is a slight penalty if the topic file is detected as out-of-sync with the history [16:35]
zak256I created a skin template for a page to replace some terrible INCLUDE macro and it works perfectly. But is there a possibility to skip using that template somehow by setting a get parameter?
just for testing purposes
[16:35]
gac410I'm not really sure. [16:36]
zak256I just tried skin=text, but then I have no further layout either. [16:37]
gac410Do you mean a skin, or a view template * Set VIEWTEMPLATE = ... [16:38]
zak256It's a view template.
MypageViewTemplate
[16:38]
gac410I don't know how to overrride it... hmmm I wonder. If you set VIEWTEMPLATE as final in your user topic, that might prevent the setting altogether. It would probably break other stuff though [16:39]
zak256I don't really want to edit the page for that...
But it's not important, was just wondering.
[16:40]
......... (idle for 43mn)
***zak256 has left [17:23]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)