#foswiki 2015-05-11,Mon

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***ChanServ sets mode: +o gac410 [00:37]
..................................... (idle for 3h0mn)
GithubBot[distro] gac410 pushed 1 new commit to master: http://git.io/vUWhE
distro/master 823b2d5 George Clark: Item13354: De-clutter some default pages...
[03:37]
***GithubBot has left
gac410 has left
[03:37]
................... (idle for 1h33mn)
ChanServ sets mode: +o CDot [05:14]
................ (idle for 1h15mn)
ChanServ sets mode: +o MichaelDaum [06:29]
.... (idle for 16mn)
andrei_tHello again, since I didn't get a reply last friday I'm going to try again
I've followed the entire solr installation, but when trying to now index it, I get following error
./solrindex topic=Main.WebHome
WARNING: can't ping solr
ERROR: can't conect solr daemon
ERROR: Can't call method "delete_by_id" on an undefined value
ERROR: Can't call method "delete_by_query" on an undefined value
Indexing topic Main.WebHome
but I can access http://localhost:8983/solr fine via curl
also service solr status gives me:
Found 1 Solr nodes:
Solr process 3008 running on port 8983
{
"solr_home":"/appl/solr_conf/data/",
"version":"5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10",
"startTime":"2015-05-08T13:16:23.171Z",
"uptime":"0 days, 0 hours, 19 minutes, 27 seconds",
"memory":"52.2 MB (%10.6) of 490.7 MB"}
using jre1.8.0_45
I'm not sure what else I could try since the service is obviously running, but for some reason it seems like the communication isn't working
[06:45]
***ChanServ sets mode: +o MichaelDaum [06:57]
MichaelDaumMichaelDaum back from a weekend trip
do I read the checkins to master correctly that it is all utf8 now, i.e. that the utf8 branch got merged to master?
CDot, any comments?
[06:59]
CDotno - you probably saw a push that I did to master:HEAD instead of utf8 by accident
I reverted it. master is still using [Site}{CharSet}
[07:00]
MichaelDaumah ok
cus I need a working system and can't afford sidetracks til the end of may
[07:01]
CDotutf8 *is* a working system :-) [07:01]
MichaelDaumI probably wont pull [07:01]
CDotdon't pull to your main repository; clone to another dir and cross check if you are unsure. [07:02]
MichaelDaumCDot, in how far does your work affect extensions? whats the status quo for JsonRpc and NatEdit? [07:02]
CDotJsonRpc and NatEdit - and all other core extensions - are fine, compatible with both utf8 and {Site}{CharSet} [07:02]
MichaelDaumwhats the status ... with regards to backwards compatibility I mean? [07:03]
CDotother extensions will work fine as long as exclusively unicode (i.e. non-iso-8859-1) codepoints are not used
unless the {Site}{CharSet} is utf-8, in which case they *should* work, but I can't guarantee it
in summary, existing extension will only fail in utf8 if unicode chars outside the 0-255 range are used
[07:03]
andrei_t:( [07:05]
MichaelDaumI am particularly concerned about plugins at the fringe to other systems such as ldap, sql, soap an solr. these all are jumping thru burning loops to bring foreign strings into foswiki ... [07:05]
andrei_tam I being ignored? or does nobody know what's wrong? [07:05]
MichaelDaumeven StringifierContrib calling external helper tools [07:06]
CDotMichaelDaum: if you have used the core interfaces, such as Sandbox, that should not be a problem [07:06]
MichaelDaumandrei_t, your $Foswiki::cfg{SolrPlugin}{Url} is probably wrong [07:06]
CDotif you have called `prog`, then you may be in pain [07:06]
MichaelDaumMichaelDaum never uses backticks in perl web apps for security reasons [07:07]
CDotI didn't think you would. I'm just highlighting the issues. [07:07]
andrei_tit's set to http://localhost:8983/solr [07:07]
MichaelDaumy y
andrei_t, thats wrong then
[07:07]
CDotif you open files directly from disk using topic/web names, there may be issues [07:07]
andrei_t? [07:07]
CDotin utf8, topic and web names are unicode [07:08]
MichaelDaumandrei_t, whats the url you access solr from within your browser? [07:08]
andrei_twhen I'm on the machine and use the command: curl http://localhost:8983/solr
I get a proper html response
[07:08]
MichaelDaumwhich version of solr is it?
which version of SolrPlugin
[07:09]
andrei_twe don't have port 8983 open to the rest of the infrastructure, so I can't access it from my computer direclty [07:09]
CDotin utf8 branch, Foswiki::Sandbox handles encoding and decoding params and program results from/to unicode [07:09]
andrei_tsolr version is: "version":"5.0.0 1659987 - anshumgupta - 2015-02-15 12:26:10"
and solrplugin is the latest one off foswiki
[07:09]
MichaelDaumandrei_t, solr is organized in different "cores" one of which is called "foswiki" [07:10]
CDotthe stores (Rcs and Plain) will work with existing non-utf-8 data through setting eg. {Store}{Encoding}='iso-8859-1' [07:10]
MichaelDaumandrei_t, $Foswiki::cfg{SolrPlugin}{Url} = 'http://localhost:8983/solr/foswiki'; is the default [07:10]
CDotcurrently I'm struggling to understand how low-level perl functions, such as open, opendir and friends understand unicode
it *looks* like they convert unicode filenames to byte strings without any additional re-encoding
[07:11]
MichaelDaumCDot, this all sounds good and work on the utf8 branch looks good ... though some cleanup in NatEdit left behind some convolution not required anymore to convert url params from utf8 to site-charset [07:12]
CDotdid you see my question about NatEdit? [07:12]
MichaelDaumno [07:12]
CDoti looked to me like it could not have worked on a FW earlier than 1.2
because you were always decoding params to unicode
I fixed it on utf8, but I think the code is still wrong on master
"could not have worked" means "could not have worked with multibyte data"
with iso-8859-1 chars it would be fine
[07:13]
MichaelDaumall of the loop thru url params to rest-save arent required on your utf8 branch as it is a null-operation happening from what I see code being left over [07:14]
CDotI kept your old code for compatibility
utf8 uses $Foswiki::UNICODE to detect if conversion is a nop
[07:15]
MichaelDaumproblem with current master REST is that plugins have to decode utf8 url params all by themselves ... as NatEdit and Wysiwyg each do on their own [07:16]
CDotcorrect, but if $Foswiki::UNICODE is true, the core does all decoding for them. [07:16]
MichaelDaumdo you plan to merge your stuff into the 1.2.0 release? [07:17]
CDotso long as hey don't use low-level perl file operations, extensions can ignore charset encodings.
we will merge only when the key stakeholders are all happy - you, me, jomo and George
and not before
[07:17]
MichaelDaumthats more work on a final 1.2.0 than expected [07:18]
CDot1.2.0 could be released with the {Site}{CharSet} solution as it is, but there are big caveats
basically jomo and I do not believe we can claim it works with multibyte characters
because it doesn't (except some limited edge-case operations)
[07:19]
MichaelDaumgawd this all feels so last minute hackish [07:20]
jomomany of problems (utf8) on the master are handled perfectly by the utf8 branch. (and those problems should be patched anyway somewhat in the master)... so.. [07:20]
MichaelDaummakes me nervous [07:20]
CDotI say again, you *could* release the master branch as-is, but you must do so with caveats.
it works with utf-8 at least as well as 1.1.9, but is *not* a unicode solution.
I'm surprisingly happy with utf8 at the moment, though you previously raised performance as an issue and I have not looked at it (nor will I)
CDot is currently focused on making sure all aspects of the core work on the utf8 branch
[07:22]
MichaelDaumI just recently found out that the 3.13 kernel shipped with ubuntu 14.0.4.2 by default has got a serious performance degration in the block layer ... for all FSes ... and me running thru code trying to find out why things got so much slower in Foswiki ... ARG [07:25]
CDotnote that I have uncovered a number of minor bugs that still exist in master
at the rsk of being proved wrong, I thing that *if* you are already using utf-8 *and* you are using up-to-date perl and Encode *then* there will be no measurable performance degradation in utf8
[07:25]
MichaelDaumCDot, alright. thanks for the infos so far on your work. I will have to convince myself in the end and fix plugins so that they work on foswiki/utf8 as well as on older engines [07:27]
CDotif you are using e.g. iso-8859-1 store, then there *may* be a small degradation
yes of course - there is a lot still to be done, expecially for your extension suite
ldap, for example, is a complete unknown to me at the moment
but generally, I have found that 99% of the work on the utf8 branch has involved *deleting* code :-)
[07:27]
MichaelDaum... and thats what I have to delay for a couple of weeks being commited to other work for the rest of may [07:28]
CDotas I said, I will personally not merge untill all the named stakeholders are happy.
I personally think it would be a mistake to release 1.2.0 *without* utf8, but that may just be because I have put a lot of work into it over the last week or so.
[07:28]
jomoi will do as many test as my timeframe and my limited knowledge allows ;) [07:30]
CDotjomo: you have been a huge amount of help. I always said I could not do the unicode conversion alone. [07:32]
jomoCDot: what is the best way document the found bugs/problems on the utf8 branch? write a Task for every-one small thing (at the developemnt process isnt the best idea imho - polluting the Tasks web) - make one summary topic somewhere or how? [07:33]
CDothmmm. I think at the moment, given that it's only you, me and gac410 working on the branch, it's enough to put stuff in 13378 [07:33]
jomook [07:34]
CDotat some point we may need to create a new branch identifier in the Tasks web
depends on how long this goes on for.
and how many other people start to contribute
[07:34]
jomook, now 13378 is the magic number ;) [07:35]
***ChanServ sets mode: +o MichaelDaum_ [07:37]
........ (idle for 36mn)
andrei_tMichaelDaum_: sorry, had to do something else with higher priority. I changed the value to the one you mentioned and it still doesn't work
also
curl http://localhost:8983/solr/foswiki
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing /solr/foswiki. Reason:
<pre> Not Found</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>
[08:13]
jastdoes accessing http://localhost:8983/solr/ give you a Solr web interface?
CDot: I think it's too late in the 1.2 cycle to merge utf-8, but definitely a great thing to merge into master directly after the release (caveat: I haven't looked at the branch yet, so I might be dispensing compliments where none are warranted ;))
and hopefully the next release will be a little bit quicker
[08:27]
CDotjast: as I said, nothing is going to happen without stakeholders buy-in. The easy part is pretty much done, in the utf8 branch. The hard part - testing it, and making sure it actually works - is the big unknown. And jomo, George and I can't do that all by ourselves.
as I work through the code, I keep finding bugs and inconsistencies that I fix and move on from. Sadly master won't benefit from those either, but at least they will be queued.
[08:32]
....... (idle for 34mn)
andrei_tjast: yes it does [09:07]
jastin that case I *think* the 404 isn't a problem
in the cores dropdown, does the web interface list 'foswiki'?
(alternatively, check if you get a response at http://localhots:8983/solr/foswiki/select)
note, if that URL works, it doesn't mean you have to change the config! SolrPlugin automatically appends things to the URL internally
[09:07]
andrei_tI just piped the curl output to an html-file and opened that
apparently there is an initialization failure in solrcore
here's the output of the log
http://pastebin.com/sVCV3NH1
interestingly though, I don't recall seeing any such errors in the log from friday
could it be that old db entries are interferring with the new one?
[09:10]
jomojomo never used the solr but alternatively to exclude perl's LWP errors, try
perl -MLWP::UserAgent -E 'say LWP::UserAgent->new->get(q{ http://localhost:8983/solr/})->content'
[09:13]
jastwell, have you managed to index any data in the first place? [09:13]
andrei_tto give a little background, solr used to work a while ago, but somehow it broke and there was no visible error in any of the logs
so I uninstalled the plugin we had and re-installed a newer version
[09:14]
jastwell, you could try deleting the index (data/ subdir of the solr core), restarting solr, reindexing everything [09:15]
andrei_thmm... but those files shouldn't be here anymore to begin with
I was more referring to perl-DBD-SQLite
[09:15]
jastoh, right
the version I use doesn't use SQLite, so I forgot about that
but the error is from Solr, not from SQLite
[09:16]
andrei_twell, the old one used that
hmm.. well then I guess I can rule that out though
[09:16]
..................................... (idle for 3h1mn)
***sneep has left "Sent from my iPhone." [12:17]
........ (idle for 36mn)
GithubBot[distro] FoswikiBot pushed 2 new commits to master: http://git.io/vU4vn
distro/master a398d58 Lieven L. Litaer: Item13252: Translations updated using Weblate (Dutch)...
distro/master 8a69225 Lieven L. Litaer: Item13252: Translations updated using Weblate (Klingon)...
[12:53]
***GithubBot has left [12:53]
..... (idle for 20mn)
ChanServ sets mode: +o Lynnwood [13:13]
TarboxHello! Is there a way to make a %BUTTON% have target='_blank'? [13:24]
jastdoesn't seem like it [13:33]
............................... (idle for 2h31mn)
foswiki_irc4Hi there [16:04]
WinsonHuangHi everyone. I have a question on Foswiki with the LdapContrib and hope to get any response from here. [16:07]
TarboxAs the saying goes, I'm not an expert but I'll take a look. [16:08]
WinsonHuangFirst of all, I am an novice of Foswiki and will have any silly questions. :)
We have a Foswiki site and integrated with JASIG SSO via LdaContrib years ago and everything works fine until recently an issue camp up and I have no idea why it happens.
[16:11]
TarboxListening. [16:11]
WinsonHuangGreat! Thank!!! Some users can't import into Foswiki from ldap and there are lots of unsure errors in the apache logs etc. [16:12]
TarboxCan you paste one for me? [16:13]
WinsonHuangSure!
[Mon May 04 14:14:45 2015] [warn] [client 192.168.2.25] mod_fcgid: stderr: - LdapContrib - found prevWikiName=WikiName for CN=user@domain.com,OU=users,O=Partners, referer: https://wiki.domain.com/Main/EndUserTrainingOnlyGroup
[16:13]
TarboxAre there more? That's sort of advisory. [16:16]
WinsonHuang[Mon May 04 14:14:45 2015] [warn] [client 192.168.2.25] mod_fcgid: stderr: - LdapContrib - adding wikiName='WikiName', loginName='loginname', dn='CN=user@domain.com,OU=users,O=Partners', referer: https://wiki.domain.com/Main/EndUserTrainingOnlyGroup
there are tonsof this two repeated in the apache log file
[16:16]
Tarboxk give me a minute I want to look up those errors. [16:17]
WinsonHuangMore are [Mon May 04 14:14:45 2015] [warn] [client 192.168.2.25] mod_fcgid: stderr: Use of uninitialized value $loginName in substitution (s///) at /var/lib/foswiki/lib/Foswiki/Contrib/LdapContrib.pm line 1189., referer: https://wiki.domain.com/Main/EndUserTrainingOnlyGroup [16:18]
Tarboxokay that strongly suggests that you haven't filled out whatever you're using as loginName for new users.
Can you go to bin/configure and find what {Ldap}{LoginAttribute} is on the tab for the LdapContrib?
[16:18]
WinsonHuangThanks Tarbox. BUt how can I go to the bin/configure? From UI or from OS?
We have this Ldap Group EndUserTrainingOnlyGroup for long time and there are lots of uses in this group and it works before
but something happened a couple of weeks ago and all users in the groups can't imported into Foswiki
[16:21]
TarboxWell then it may not be that exactly, but it's a good first place to look. [16:23]
WinsonHuangthis group can be seen in Foswiki but no users at all [16:23]
Tarboxhttp://yourfoswiki.org/bin/configure gets you into foswiki config. [16:23]
WinsonHuangall the other groups with users are fine [16:23]
TarboxAre there any users in the nonworking group *and* a working group? [16:24]
WinsonHuangGood question, let me check, wait a minute
Hi Tarbox, I am in the configure page now, what info you would like to know for the LdapContrib?
[16:26]
TarboxWell, let me turn that around. Look for GroupAttribute, PrimaryGroupAttribute, MemberAttribute, InnerGroupAttribute, and make sure the named values are properly set on your misbehaving group's Ldap object. [16:32]
WinsonHuangIn the setting, GroupAttribute is displayName which is correct, we use displayName in Ldap as WikiName.
PrimaryGroupAttribute is set to gidNumber, I am not sure what is that.
[16:38]
TarboxIt implies your ldap server is *nix based, and that ldap groups are mapped onto groups provided by the OS, hence each group has a gid. [16:39]
WinsonHuangMemberAttribute is set to member and InnerGroupAttribute is set to member as well. to be honest, i am not sure those either [16:39]
TarboxDo you have an ldap browser?
I'm more concerned with whether those attributes are set correctly on your misbehaving group.
[16:39]
WinsonHuangyes, the ldap is a windows box and I use ADSI Edit to view them [16:40]
TarboxCheck for me please? [16:40]
WinsonHuangYes, Sir! :) [16:41]
TarboxThank you. [16:41]
WinsonHuangIn Ldap, go to EndUserTrainingOnlyGroup, righr click properties, the displayName is EndUserTrainingOnlyGroup
And there are lots of users inside memeber property
I can't find what mapped to gidNumber
[16:45]
Tarboxah I'm re-reading the help file and it seems gidNumber is going ot be on the user records. [16:47]
WinsonHuangthe issue is none of users from EndUserTrainingOnlyGroup get imported into Foswiki
other Ldap Groups seems fine
[16:47]
TarboxOkay guess I have to look at the code after all.
When you posted me those error messages, were they exactly as they appear in the log? Because now that I'm looking "prevWikiName=WikiName for CN=user@domain.com" looks liek a bunch of default data.
I totally understand if you censored them, it's what I would do, but the sake of argument...
[16:48]
WinsonHuangIt seems these messges are for akk users not only the users that belongs to EndUserTrainingOnlyGroup [16:53]
TarboxYeah. Foswiki converts usernames into wiki-friendly names, and it memorizes the conversions it makes.
So that's the contrib going, "Seen this guy before, using the same name as last time" over and over.
The $loginName being null is more exciting as an error, but my ldapcontrib isn't teh same version as yours so I'm having troubles finding the right line.
[16:55]
WinsonHuangHi Tarbox, Sorry I have a meeting for now and will be back in about half an hour. Thanks! [17:02]
Tarboxgood luck. [17:02]
***ChanServ sets mode: +o CDot [17:13]
.......... (idle for 47mn)
WinsonHuangHi Tarbox, are you still there? [18:00]
TarboxSure. [18:00]
WinsonHuangThanks! In our case, not all ldap users need to be a Foswiki user. So some users might not have a displayName in ldap which is fine. but for those users under *notworking* group, they have the displayName filled in. [18:01]
TarboxThat might explain the $loginName error. Are there any other flavors of error message?
You know maybe you should just pull up a working user next to a not working user and compare fields
or a working group next to the not working group
[18:03]
..... (idle for 22mn)
***Tarbox has quit IRC (*.net *.split) [18:26]
WinsonHuangHi Tarbox, I captured all logs related to a working user and a not working user from Apache error.log file. How can I sent it to you?
I found some interesting things there
[18:26]
It looks to me all users have the following two lines in log file even they are not a memeber of *not working* group
[Mon May 04 14:14:46 2015] [warn] [client 192.168.2.25] mod_fcgid: stderr: - LdapContrib - found prevWikiName=XXX for CN=xxx@abc.com,OU=users,O=Partners, referer: https://foswiki.company.com/Main/EndUserTrainingOnlyGroup
[Mon May 04 14:14:46 2015] [warn] [client 192.168.2.25] mod_fcgid: stderr: - LdapContrib - adding wikiName='XXX', loginName='xxx', dn='CN=xxx@abc.com,OU=users,O=Partners', referer: https://foswiki.company.com/Main/EndUserTrainingOnlyGroup
the EndUserTrainingOnlyGroup is not working group
in Ldap, this user is a member of a working group and not a memeber of the not working group
[18:31]
..... (idle for 20mn)
TarboxSo every user gets a debug line that refers to EndUserTrainingOnlyGroup?
That makes it sound like that's the page you did the ?refreshldap=1 on.
Is there any debug that's not of that pattern?
Cuz it's just advisory. It's saying the users are all users it's seen before.
Were it me I'd go into the code and comment those two lines out, then re-run and check for other debug.
[18:56]
WinsonHuangYes, you are correct. I checked the error log after run the refreashldap.
Sorry, my friend, I am a novice for foswiki and perl so how can I do that?
[19:02]
.... (idle for 19mn)
TarboxCan you get to the commandline of your server? Go to /var/www/yourfoswiki/lib/Foswiki/Contrib/ and edit LdapContrib.pm [19:23]
WinsonHuanggot it... thanks!
Do I need to restart the apache after edting?
[19:24]
TarboxThat would be safest. Technically it depends on how your server is set up.
it seems to be fastest to search on "wikiName=" to get to the right spot in the code.
[19:24]
....... (idle for 32mn)
WinsonHuangI commented out the line writeDebug("found prevwikiName=$prevWikiName for $dn")
and line writeDebug("adding wikiName='$wikiName',loginName='$loginName',dn='$dn'")
and run the refresh cache, it still log all debug info with "wikiName="
it seems need a apache reboot but I can't do it now as it is a live server
[19:57]
TarboxIv'e done service apache restart with a live server countless times. [20:00]
WinsonHuangWIll do tonight and post my update here tomorrow. WIll you be here tomorrow, Tarbox? [20:01]
TarboxProbably. [20:01]
WinsonHuangyes, but the site is pretty heavily used so users will get confused if can't access it. [20:02]
Tarboxk [20:03]
WinsonHuangHi Tarbox, BTW, I am not sure if you provide any consulting service for Foswiki customization? [20:15]
TarboxNo, I'm sorry. [20:15]
WinsonHuangk, do you know any person / company providing that? [20:17]
TarboxHonestly almost anyone else in this room. I'm afraid to name names lest I be playing favorites. There's this: http://foswiki.org/Support.WikiConsultants
They're active and most of them check the logs. Leave a way to contact you here in the room and I bet you would be.
[20:18]
***CDot has quit IRC (Quit: Leaving.) [20:20]
WinsonHuangThanks! Please contact me @ winsonweb@gmail.com if anyone is interested in providing some consulting service especially on LdapContrib customization. [20:32]
......... (idle for 41mn)
***GuilainC has quit IRC (Remote host closed the connection) [21:13]
.......................... (idle for 2h5mn)
WinsonHuang has quit IRC (Quit: Page closed) [23:18]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)