#foswiki 2017-02-03,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
GithubBot[distro] FoswikiBot pushed 1 new commit to Release01x01: https://git.io/vDGJy
distro/Release01x01 a6bdded Raul F Rodriguez: Item12813: Translated using Weblate (French)...
[01:56]
***GithubBot has left [01:56]
FoswikiBothttps://foswiki.org/Tasks/Item12813 [ Item12813: Documentation task for Release 1.1.x ] [01:56]
...................................... (idle for 3h9mn)
***gac410 has left [05:05]
........................ (idle for 1h55mn)
ChanServ sets mode: +o MichaelDaum [07:00]
.... (idle for 15mn)
ChanServ sets mode: +o MichaelDaum [07:15]
.... (idle for 17mn)
ChanServ sets mode: +o cdot [07:32]
.............................. (idle for 2h28mn)
rathierSomeone using solr-plugin with foswiki? If I replace wiki-search with solr-search and open a search page, it seems like the whole index has to be loaded and than gets filterd. So the first loading of the page takes its time.
As we filter the results for the current web by default: would it be possible do just load that part of the results, so the first rendering of the page is faster?
Does that even make sense?
[10:00]
............. (idle for 1h4mn)
jastSolrPlugins's search view does not actually fetch all results in advance
unless something is wrong :)
this is the way it's supposed to happen, on an example site: https://demo.michaeldaumconsulting.com/bin/view/Home/WebSearch
[11:06]
rathierjast: And on that site you see it already knows about 416 result (it shows less, a fixed amount per site). And then you get that number down by filtering more. So the search already needs to know, how much data it has indexed. Doesn't it? [11:13]
..... (idle for 22mn)
jastrathier: sure, but that happens not by fetching all results, but by getting faceting data from solr (i.e. just a summary)
if you want to search by keyword only, the normal approach is to simply not go to the search page and use an inline search box instead (as seen on the same demo site in the top bar)
that will then take you to the search page with the keyword already filled in and load the facet data and the search results at the same time
[11:35]
............. (idle for 1h1mn)
rathierjast: Ah, Ok. The solr-server "hands out" the summary, not all the results. I wasn't aware of that. Thank you for clearing that up. [12:38]
....... (idle for 33mn)
***padraig_lennon has left [13:11]
jastrathier: keep in mind Solr is designed to be able to work with huge amounts of data, so usually you'll find that they've thought about these things ;) [13:18]
***ChanServ sets mode: +o gac410 [13:32]
.... (idle for 17mn)
gac410MichaelDaum_: for Item14317, I assume the fix is to dump the _src code, ship an _uncompressed, and let the JQuery core handle that part? [13:49]
FoswikiBothttps://foswiki.org/Tasks/Item14317 [ Item14317: Under some conditions, JEditableContrib attempts to load an =_uncompressed= source, which we don't ship. ] [13:50]
.............. (idle for 1h8mn)
gac410cdot: Do you know how to actually engage the jeditable javascript. I added a %JQREQUIRE{"JEDITABLE"}% on a page how does it get attached to formfields, and how can I observe it actually working
I think I've fixed Item14317, no errors when jquery debug is enabled now, but I can't figure out how to actually test the thing.
[14:58]
FoswikiBothttps://foswiki.org/Tasks/Item14317 [ Item14317: Under some conditions, JEditableContrib attempts to load an =_uncompressed= source, which we don't ship. ] [14:59]
.... (idle for 16mn)
Lynnwoodmorning all - Spent a fair amount of time last night trying to find the special characters that were getting broken when run through PublishPlugin. [15:15]
gac410Hi Lynnwood [15:15]
Lynnwoodhey there gac410 :-)
I officially hate encoding at this point.
[15:16]
gac410:) [15:16]
LynnwoodI tried running convert_charset using encoding=cp-1252 and that had interesting results. If kind of fixed some characters and completely broke other characters. [15:17]
gac410Had you already converted to utf-8? [15:17]
Lynnwoodyes... and I wanted to ask about that. [15:17]
gac410ugh. That's the one thing that's really difficult to fix - if you have a topic with a mix of encodings - utf8 and 1252 for ex. [15:18]
Lynnwoodyea. I'm beginning to think that my best and only option is try and locate some of these characters and do search and replace to replace them.
There appears to be a handfull.
[15:19]
gac410iirc, I had played around with using find and execute isutf8 to detect files with invalid utf8 [15:20]
Lynnwoodyea... i tried that, but it came up false.
so... i'm not sure there but I think some of these _are_ valid utf8 characters but PublishPlugin still messes them up.
So I think there is some follow up to do on encoding output from PublishPlugin... but I don't have time to solve that today.
It's so hard to even find out what some of these characters are...
[15:21]
gac410Is is when it generates a pdf, or html or ?? [15:22]
Lynnwoodhtml [15:22]
gac410So the characters are valid in the pdf but corrupt in html. [15:23]
Lynnwoodi didn't try pdf [15:23]
gac410oh .. never mind then ;) [15:23]
Lynnwoodpublishing from a web to static html [15:23]
gac410But when you view the topic in the wiki, it's valid. hm If you look at https://foswiki.org/Support/WhatIsUtf8AllAbout ... down at the bottom is some utf8 text
If you cut/paste that into a topic on your wiki, it should work fine.
Unlike the emdash and quotes custom in cp-1252, those should be really obvious utf-8. See if they publish. Easier to find than fishing for corrupt characters in your wiki.
[15:25]
LynnwoodSo tell me: what is the effect of running convert_charset over a web that is supposedly already converted (but perhaps originally as straight ascii rather than cp-1252)? [15:27]
gac410If it's straight ascii, it should do nothing. [15:28]
Lynnwoodwould it fix cp-1252?
the emdash and quotes custom are a bit part of the problem.
[15:28]
gac410If it has c/p MS characters - smartquotes, etc, Yes it will fix them *ONLY If the from-encoding was set to 1252 [15:29]
Lynnwoodcp-1252? [15:29]
gac410yes [15:29]
FoswikiBotyes is is [15:29]
Lynnwoodmaybe i'll try that again.... [15:29]
gac410Be really careful about running more than once! [15:29]
LynnwoodThat's what I did, but it seemed like it broke other characters.
what would be result?
[15:29]
gac410The default encoding use to be iso-8859-1, and if that is set, the converter won't understand the ms charcters [15:30]
LynnwoodWhen i say "ill run it again" - what I mean is roll back to the original versions and then try it again. [15:30]
gac410IF is has converted other ISO-8859 high characters to utf-8 ... running again will then corrupt that utf-8
oh okay
You can also run it in an inspect mode anytime to see what it will do,
[15:31]
Lynnwoodwhat would the ISO-8859 high characters be converted to? [15:31]
gac410It will convert them to utf-8 versions per the mapping of iso-8859 to utf-8 [15:32]
Lynnwoodand then if run again, it would break those utf-8 versions? [15:33]
gac410yes. It uses the "from encoding" to examine each byte and attempt to convert that byte to utf-8.
but utf-8 is multi-byte. So a high ascii single-byte u-umlaut will result in multiple bytes in the utf-8 version
[15:34]
Lynnwoody, that's what I saw. [15:35]
gac410There is a mode / way to run the converter to "detect" the source encoding but it's very complex and unreliable, and is still only one encoding per file. [15:35]
LynnwoodSo some topics got fixed (i guess with lingering ISO-8859 high characters) while other topics were broken. [15:36]
gac410It has to look at the bytes and try to guess what encoding was in use when it was written. In many cases nearly impossible. [15:36]
LynnwoodIn those, I found multi-character strings where previously the characters looked ok.
However, those characters still didn't go through publish plugin unscathed.
[15:36]
gac410Did the file pass the isutf8 test? [15:37]
Lynnwoodlet me check that again [15:37]
gac410If the file is okay in "isutf8" and displays correctly in the browser in foswiki, then clearly publish plugin is busted [15:38]
Lynnwoodyes, i'm fairly certain that is the case...
but i believe I can get through today if I can find and replace those characters with simplier equivalents
e.g getting rid of the smart quotes.
the commend find -L . -name '*.txt' -exec isutf8 {} \; doesn't report any problems in the web...
[15:38]
gac410The command perl -nlE 'say if /\P{Ascii}/' < the.file.to.check Will tell you if a file has any non-ascii characters
ie. if it passes isuft8 (which a plain ascii file will pass as well), then the perl command will tell you that there is utf8 present
[15:42]
Lynnwoodyea, the perl command shows me the line where there are odd characters but doesn't help much in finding out what they are.
this page has helped me some: https://r12a.github.io/apps/conversion/
I pasted in a character like "–
never mind... doesnt' show up in irc.
but it came back as 0x96
[15:45]
gac410AśčÁŠŤśěž ... utf8 should show correctly
You can't tell from the context what the odd characters are ... from the perl command?
[15:48]
Lynnwoodthe perl command showed me the general context but i could find that easier just looking at the raw version of the topic.
They showed up as simply square boxes.
In that case, the characters simply where not being displayed in regular view, but they were actually needed.
they were endashes and quotes.
[15:49]
gac410raw version ... you mean viewing the txt file with vi? Or viewing with ?raw=on The latter, they really ought to show correctly [15:51]
LynnwoodI haven't found how common that case is in the web.
raw=all
I'm hoping to use sed or other tools simply to locate topics that have those kinds of characters.
So perhaps then i can search and replace those... without messing with other topics.
[15:51]
gac410even with raw=all they ought to show correctly. Unless they are characters with fonts/glyphs that are not available on your client.
Lynnwood: See https://foswiki.org/Sandbox/TestUTF8 ... That should be perfeclty viewable in raw and raw=all
And you should be able to cut/paste from the raw view into your wiki, and it should still be viewable. I'm going to try to push it through the PublishPlugin
:( Unescaped left brace
[15:52]
Lynnwoodthat was result from PublishPlugin? [16:03]
gac410y. and it's failing on a call to cgi::param() in list context as well. truly busted on recent perl & cpan [16:04]
I don't think I have any idea how to fix this. It's using CGI->Vars function which returns a "tied hash" ... and then references to that hash are failing with:
Foswiki::Request::param called in list context from package CGI, /usr/share/perl5/CGI.pm line 1251, declare as scalar, or call multi_param.
[16:09]
FoswikiBothttps://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Request [16:09]
gac410Ah... never mind. Turned off Foswiki Debug - ASSERTS - and the problem is ignored. But it's not a good thing to have in the code :(
wtf. It worked, but generated an empty file.
[16:10]
Lynnwood:-( [16:12]
gac410Well not really empty. <hr /> <p></p> <hr />
[16:12]
Lynnwoodlet me see if i can show you example of output i got from publishplugin....
unlike other topics, these are unrestricted.
well... they aren't "topics" anymore...
[16:13]
gac410Lynnwood: okay I did get it to publish html, and indeed the utf-8 is all corrupted. So PublishPlugin is broken.
It correctly generated utf-8 filename and topic names, but the contents was snafu
[16:20]
Lynnwoodany idea how hard a fix will be?
This is kind of what i deduced and concluded that there was nothing i could do about that today...
[16:21]
gac410Not sure yet. no idea how it generates html yet.
I'll look a bit at the code
[16:21]
Lynnwoodok
thanks so much... i'll be into helping find/fix/test that later and republish it.
[16:21]
gac410Interestingly, if I tell it to publish a single topic, it generated an empty file. But telling it to publish a range of topics, it included and correctly generated the topic that was previously empty. [16:24]
FoswikiOnSlack<anselm> Hey there, I wondered if this could be used for support, too? I'm just fighting my way through the whole installation process and ended up at a very strange point, after registration foswiki gives an error message: "Foswiki MAIL:ERROR: Cypt::SMIME is not available. Mail will not be sent" and wondered, this should be a typo in the sourcecode, doesnt it? [16:33]
gac410anselm ... It sounds like you enabled email to be signed in your bin/configure setup. There are optional dependencies needed if that's enabled.
your LocalSite.cfg should probably have $Foswiki::cfg{Email}{EnableSMIME} = 0; It's enabled/disaabled in configure under the Mail Signed Email tab
If it's enabled, then configure should also be complaining about the missing dependency ... or it shows the installed version if it's found
Well Lynnwood I thought I saw how to fix it, but no joy here. :(
[16:34]
Lynnwooddrats
I'm exploring for now whether it will be faster for me to try and fix them now in the wiki or clean up the export html. I'm see which is easier to find the problem characters and replace (without messing up others)
being as... at the moment both are "broken" on some level... :-(
[16:39]
gac410This one might need cdot input. Some of the issues are a bit beyond me. Especially the "tied hash" issue with cgi multi_param ... though that only dies if debug is enabled. [16:42]
Lynnwood: I'm wondering if there is also a display / browser encoding issue. With my change to utf8_encode the generated files, they view correctly in "less" but still are wrong in the browser [16:51]
Lynnwoodah geez... [16:51]
gac410I thought there was a way to force the browser encoding when reading a page - but latest firefox I cannot find it any more :(
Y, missing a header. view - Text-encoding .. detected as western. Need to force to utf-8
[16:52]
Lynnwoodi actually put the encoding meta in my publish template. [16:55]
gac410So without my patch, it generates wide-character-in-print warnings when writing the file. With the patch, it writes utf-8 but the browser still sees it as western. [16:55]
Lynnwoodwould that simply be a result of the lack of encoding meta in the header?
e.g. that would be solved by adding the encoding meta to the publish template
[16:55]
gac410In backend file.pm sub addString, change the "print" to be:       print $fh Foswiki::encode_utf8($string); [16:56]
LynnwoodLynnwood heads to look for that... [16:57]
gac410That causes the generated files to be utf8. The generated index.html is still busted for utf-8 topic names, but the content of the files is correct
And you need the encoding in thee pubish template. yes
[16:57]
LynnwoodI have another little mystery about publishing... which I don't know if I can sort out today or not or just work around.... [17:06]
gac410well first question ... did that patch work? [17:07]
Lynnwoodit's running...
i'll check
[17:07]
gac410ah. [17:07]
LynnwoodI have to run it in a couple passes... it times out.
On this server, I'm always balancing between a time out setting that works well for most of the content versus a few specific scripts which require much longer to run.
i.e. ExcelImportExport (when importing thousands or records), PublishPlugin (publishing hundreds of topics with specific versions + view templates)
I've never found a way to tell apache to give longer time out for specific scripts...
[17:08]
gac410not that i'm aware of either. [17:11]
LynnwoodThe site is running fcgi by default and I can bypass that for regular cgi for certain scripts, but don't know if that will really help much. (I was going to give it a try). [17:11]
gac410well fcgi has it's own timeout. So it woudl be possible to have a shorter fcgi timeout, and a longer apache timeout I suppose. [17:12]
LynnwoodIf I set the timeout to accommodate these long-running scripts (up to 15 minutes or so), then the site get's overwhelmed by routine calls running too long.
I tried to find that once but wasn't sucessful.
[17:12]
gac410You could also run some of the really big stuff from the shell. avoiding web server timeouts altogether.
This plugin is a mystery to me. I have no idea why some areas are working. All Foswiki API data will be unicode. Which *must* be converted to utf-8 for file system access for ex. Why it generates correct utf8 filenames and where that happens, I have not a clue.
So index.html appears to be double=decoded from what I can tell
[17:13]
Lynnwoodgood point about shell... however, i'm need these scripts to work for folks who don't have access to the shell
The other thing i'm trying to figure out about how templates work is exactly how it figures what to include when it runs %RENDER{"head
%RENDER{"head"}%
[17:15]
gac410outa my area on that one. [17:17]
Lynnwoodyea... there's a lot of factors there i think. [17:17]
gac410well index.html is not being double-encoded. Bypassed the encode, and get wide character in print errors. :P [17:18]
LynnwoodI have two kinds of topics with different view templates. Both are using the same Publish template. One generates a very complete head (including correct style sheets and js) while the other is incomplete (almost none of the css and js).
But the view templates are really pretty much similar. Both start with including view - which should include the view for the selected skin.
go figure...
[17:19]
gac410If you skip one, does the other work? I think the head stuff is only rendered once. [17:20]
Lynnwoodinteresting question... [17:21]
gac410well hopefully that patch helps you. The rest of this is over my head. Probably needs cdot to tackle it. [17:21]
LynnwoodRendered once per topic (which i know if correct).
What I think i'll do is file a task about encoding output and report the patch you added and ask Cdot to take a look at what other areas need to be addressed.
[17:21]
gac410grrr. No the filenames are not correct. Only *some* seem to be rendered. So the filenames definitely need to be encoded as utf-8
It's reading topics, but is unable to read the attachments of topics with utf-8 filenames.
[17:23]
LynnwoodTasks.Item14319 [17:33]
FoswikiBothttps://foswiki.org/Tasks/Item14319 [ Item14319: PublishPlugin needs to encode output ] [17:33]
FoswikiOnSlack<anselm> @slack-irc thanks :slightly_smiling_face: [17:34]
gac410Lynnwood: It looks like it also might somehow be entity encoding filenames. I have a topic with an attachment test+file And it tries to read test%2bfile [17:35]
well from your task, my patch fixed your topics? [17:40]
Lynnwoodok, let's see...
well... yes... i think. i appears to be consistent with view in foswiki which is all we can ask.
[17:42]
gac410excellent
so anyway, this plugn needs some serious work
[17:44]
Lynnwoodyep
that's an improvement.
it reduces my work today a bit so thanks so much for that!
[17:47]
***SvenDowideit has quit IRC (*.net *.split)
Babar has quit IRC (*.net *.split)
ChanServ sets mode: +o SvenDowideit
[17:56]
......... (idle for 40mn)
gac410Lynnwood: Because this plugin finds "attachments" by scraping the "http:......" references out of a rendered topic, the filenames have been entity encoded. So a "+" which is not valid in a url has been encoded to %2b.
Anyway, I'm stopping looking. This needs more work than I want to do.
[18:39]
..... (idle for 21mn)
Lynnwoodwhat is %2b character? [19:01]
gac410encoded + [19:01]
Lynnwoodencoded as...? [19:02]
gac410Attached filename is blah+file Since + is interpreted as a space in a url, the <a href= link is generated as blah%2bfile
So it's url encoded
[19:02]
Lynnwoodah [19:03]
gac410This is probably why unicode attachments are having trouble as well. Because <a href... links can be generated encoded [19:03]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)