#foswiki 2015-02-12,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
RiskRewardOK, just checked the documentation, and I see that nonoise doesn't set groupby. Thanks. [00:01]
gac410nonoise should also set noheader, so you should not be getting any headers, I don't think. Maybe a bug. I don't do much with search. [00:01]
RiskRewardYes, it suppreses the default header, but I have defined a custom header in the %SEARCH. [00:01]
gac410Ah... okay that makes sense then. [00:02]
RiskRewardAre you any good with regular expressions? [00:02]
gac410well I can try :) [00:02]
RiskRewardThanks! The start of each of my topics begins with "---+!! RG-001 Document Name"...
In my %SEARCH I look for search="[\r\n\-+!]+\sRG\-"
[00:03]
gac410Looking for level 1 headings? [00:05]
RiskRewardYes. I copied the format from the documentation, but I'm really only interested in the first line of a topic. Is there a way to be more explicit? [00:05]
gac410The first line, or the first L1 heading, heading doesn't have to be in the first line. [00:06]
RiskRewardI think it currently looks for the first L1 heading, but given that I have mandated that that is always the first line, I thought I could reduce the search. [00:07]
gac410What does the heading actually look like? ---+!! RG-... [00:08]
RiskRewardHere's an example: ---+!! RG-002 Abbreviations, Definitions + Acronyms [00:09]
gac410so search="^---!!\sRG-" It should stop at the first match in each topic, unless you include multiple="on"
If the !! is optional search="^---(!!)?\sRG-" ... I think
[00:11]
RiskRewardThanks, I've tried it but it doesn't work. Think I need to escape some of the characters? [00:13]
gac410Ah... forgot the + and that is a meta. Sorry about that
search="^---\+!!\sRG-"
[00:15]
RiskRewardThat works! [00:16]
gac410^ matches "start of line" + is a meta so needs to be escaped, otherwise it causes "one or more" of the prior character/group [00:16]
RiskRewardIn the generated table I want to report the "RG-002" in one cell, and the "Abbreviations, Defintions + Acronyms" in another...
I've got two expressions:
$pattern([\r\n\-+!]+([^\r\n].*?)[\s].*)
$pattern(.*?[0-9]{3}\s*([^\n\r]+).*)
...but think they can be improved. Any thoughts?
[00:17]
gac410hm. I gotta go read pattern docs :D [00:18]
RiskRewardI don't want to trouble you. They're working but I don't really get understand them (yikes!)
I can do a little more homework myself.
[00:20]
gac410Generally with regexes ... if they work, I declare success [00:22]
RiskRewardWho said that wine was dulling your wit?!
Went home a little demoralised yesterday that I couldn't understand regexs. The PERL doc linked in the documentation is dozens of pages long.
[00:23]
gac410tbh I'm having a hard time understanding what the patterns are doing. ... ah, you seem to have a typo in the first one. ([^\r\n].*?) So [^\r\n] matches any character that is not a cr or lf. But it would normally be followed by * (0 or more) not .* which is a separate wildcard
The example on the page is ([^\r\n]*?)
regexes are one of those things that can take a long time to understand, don't be demoralized. It is a horribly complex subject
[00:28]
RiskRewardThanks for the encouragement. Many (many) years ago I had a lecturer that told our class "the reason you think this is hard is because, well, it actually is hard. Some things are." Personally, I subscribe to the "Savant Existence Theorem" - someone somewhere finds this easy." [00:32]
gac410with regexes, it's practice practice. [00:33]
RiskRewardbtw I think you need the .*? The expression doesn't seem to work without the dot. [00:35]
gac410It works in the example on http://foswiki.org/System/FormattedSearch I have to write them down over/under and compare them I guess.
The Example: ([\r\n\-+!]+([^\r\n]*?)[\r\n].*)
Your Regex: ([\r\n\-+!]+([^\r\n].*?)[\s].*)
So the example [\r\n\-+!]+ Match one or more of any of the characters within the square brackets
Then ([^\r\n]*?) non-greedy match, of any characters that are NOT a cr or lf It's a capture group ( )
last [\r\n] match cr or lf (end of line) followed by the .* ... wildcard for the rest of the topic.
[00:36]
RiskRewardOK, so I changed [\r\n] to just [\s] so that instead of displaying the whole line, it would just display up to the first space. [00:46]
gac410okay. yes. I forgot your original objective Sorry about that. If you want to match to the first space, you could also do ([^\r\n\s]*) zero or more characters that are not a cr, lf or space
so ([\r\n\-+!]+([^\r\n\s]*?).*) I think that will work.
[00:48]
RiskRewardHmm, not working. [00:51]
gac410Well as I said, if what you had was working fine, declare success :) [00:52]
RiskRewardOK, many thanks. I'll do some reading tonight. I'm a bit wobbly on the fundamentals, and it's pretty poweful, so worth the effort. [00:53]
gac410Yes definitely.
Sometimes it can be quite vexing in what works and doesn't work.
[00:54]
RiskRewardAt least it's easy to test. Just not very informative when it fails. [00:55]
gac410y. If you have perl on your system, sometimes it's easier to write a one-liner to try stuff Break it apart, match by match. [00:56]
RiskRewardThanks, I'll give it a go. [00:57]
................................................ (idle for 3h59mn)
***gac410 has left [04:56]
......................... (idle for 2h1mn)
ChanServ sets mode: +o CDot
ChanServ sets mode: +o MichaelDaum
[06:57]
................... (idle for 1h32mn)
ChanServ sets mode: +o MichaelDaum
ChanServ sets mode: +o MichaelDaum
[08:32]
.......... (idle for 45mn)
foswiki_irc9Hi, a dumb question that is not answered anywhere in the doc: How do I comment out a line of text in the Foswiki Markup TML? Thanks! [09:19]
foswiki_irc3hmm, don't find it in the doc either :-) sorry. [09:23]
jastto my knowledge TML doesn't have a commenting syntax of its own... you could use HTML comments: <!-- ... --> [09:28]
foswiki_irc9Thanks, so that's why it is not in the doc! [09:31]
.......... (idle for 48mn)
CDotYou can use %{...}% in templates, but only in templates. I always intended to put it into the main syntax but somehow never got round to it :-( [10:19]
MichaelDaumcommenting out is best done via <verbatim class="foswikiHidden">...</verbatim>
anything between normal html comments is still processed, i.e. expensive macros
read: expensive macros are executed best within <!-- --> ... and then forgotten ;)
[10:28]
CDotMichaelDaum: would you support %{...}% for comments? [10:34]
MichaelDaumyes
however we need two different behaviors: (1) taking them out of the page completely (2) render them similar to a <verbatim class="tml"> ... </verbatim>
(1) makes them invisible
(2) is nice in ViewTemplates
[10:35]
CDotsure. We already have the <verbatim> solution.... I'm talking about a "filter and forget", which we don't have. [10:36]
MichaelDaumI use %{<verbatim class="tml">}% ... %{</verbatim>}% for the latter often [10:37]
CDotinteresting [10:37]
MichaelDaumso (2) is "filter and prettify" [10:37]
CDotquestion: why do we have different syntax for templates? or more precisely, why don't topics support %TMPL:DEF?
they support %TMPL:P, after all
[10:38]
MichaelDaum%{<verbatim class="tml"}%%TMPL:DEF{"foo"}% .... %TMP:END% %{</verbatim>}% ... is all of the boilerplate ... too much imho [10:38]
jastdo you want to add the complete template logic to normal topics, too?
rendering is tangled enough as it is already, isn't it?
[10:38]
CDotjast: just a thought experiment at this stage
what we have is effecively a three-phase rendering process already
[10:39]
jastthe only thing you could do with that is influence TMPL:P statements further down the same topic, I think [10:39]
CDotit's just that the first phase only applies to content coming from templates
Phase (1) %TMPL:DEF Phase(2) expand macros Phase(3) render TML
[10:39]
MichaelDaumisn't there a proposal for %STATSECTION{type="comment"}% somewhere? [10:40]
CDotoooh, that stinks! [10:40]
MichaelDaumhehe [10:41]
CDotI'm not talking about changing the logic much, only to apply that first phase to topics as well
which would allow us to normalise and simplify the parsers (I think)
possibly at the cost of caching template content, though
(not that we do that)
hmmm, Radical thought: allow *any* macro to be used in a template, by prepending TMPL: to the name
e.g. %TMPL:SEARCH
would force the macro to be executed at template expansion time
thus eliminating expensive static %IF evaluations at runtime
so phase (1) becomes "expand templates and stick them in a cache"
phase (2) becomes "expand macros specific to this view"
and phase (3) is still "render tml"
[10:41]
jasttemplates often have view-specific stuff in them, too, once you expand the TMPL:Ps [10:44]
CDotyes, and that can remain view-specific [10:45]
jastso you can't expand TMPL:Ps before caching [10:45]
CDotjust the static bits would go to %TMPL: [10:45]
jastso in the end, all you cache is the DEFs and the INCLUDEs [10:45]
CDotno, the cache would only be %TMPL:DEF's [10:45]
jastdoesn't seem like that would save _that_ much [10:45]
CDotyes. Isn't that useful?
hey, we are playing a game of marginal gains, here
[10:45]
jastI prefer to target substantial gains first ;) [10:46]
CDotthere are no obvious magic bullets left (that I can see)
unless you have some ideas
[10:46]
jastsure there are, but they involve breaking compatibility [10:46]
CDotaye, there's the rub [10:46]
jastor, put differently, adding a different way of processing
which would probably be the best thing to do moving forward
in the meantime, I think the best gains probably come from finding ways to allow template authors to express their logic in a more concise way that also happens to be faster to process
[10:47]
CDotquestion is, when does foswiki stop being foswiki? Compatibility has always been a touchstone for the project [10:49]
jasti.e. offer specialized constructs for common use cases
that's why I said "add" rather than "replace"
we've made so many things pluggable at this point... why stop :)
[10:49]
CDoty, well, that's the approach Micha has taken with his dozens of plugins
CDot doesn't like that approach, as it effectively layers lard upon lard
[10:49]
jastyeah, we use many of those plugins, but I think a few more centralized improvements could do an even better job [10:50]
CDotso do I [10:50]
jaste.g. a way to nest macros differently that avoids all the escaping goo
I made a prototype for that at some point, though that way is not necessarily the best way to go
[10:50]
CDothmmmm. think hard about that one.
I think that is *very* difficult to do.
many, many years ago I experimented with optional outside-in evaluation, which got rid of the escaping
[10:51]
jastinside-out evaluation causes a different kind of escaping issue, too [10:53]
CDotthe problem is it comes down to "one or the other" - mixing both evaluation orders is hopeless [10:53]
.......... (idle for 46mn)
MichaelDaumwhat about making Render.pm pluggable [11:39]
jastthe way I made it works but doesn't quite lend itself to extending the existing syntax [11:39]
MichaelDaumpossible rendering engines: "html" ... no tml expansion, "xml" ... processed via xslt, "markdown", "tml" .. of course [11:40]
jastquestion is, what to do about macros.. those are not handled by Render.pm [11:40]
CDotMichaelDaum: y, that's a thought. i didn't pursue it because so many key plugins generate TML that has to be rendered. [11:40]
jastand you can't easily isolate different rendering logic to a single topic... you have to make sure the template snippets can be rendered using a different renderer, for instance [11:41]
CDotI find it very difficult to zero in on where the performance problems actually are. [11:41]
jast(generally TML) [11:41]
MichaelDaumMETA:TOPINCINFO{...type="html"...} [11:41]
CDotmainly because performance is so dependent on how topics are written [11:41]
MichaelDaumoh performance problems mainly come from %SEARCH [11:41]
CDotoptimising %INCLUDE, for example, will achieve nothing on a site where %INCLUDE is barely used
well, yes, but my baseline is the DBI store
where SEARCH is blistering
[11:42]
MichaelDaumperformance drastically improves as soon as you quit using %SEARCH ... [11:42]
CDotrendering search results, on the other hands, is slow as [11:42]
MichaelDaumthere are more problems in %SEARCH than rendering results, honestly [11:43]
CDotdefine %SEARCH when you say that
because it might be text search, search using a cache (like Solr) or native DBI search
[11:43]
MichaelDaumin a polite way? [11:44]
CDotthey all behave very differently
I would never try to ask you to be polite, Micha ;-)
[11:44]
***jast has quit IRC (Read error: Connection reset by peer) [11:45]
MichaelDaumthe core flaw of %SEARCH is that it separates the three things that belong into the backend engine: search+sort+limit ... these three belong together to be performed in a performing waxy
(limit includes paginagion)
this ... as a starter
[11:46]
CDotyes, I agree with that. That's something that can be solved *without* throwing %SEARCH out
so, with that as a starter.... what's next, in your mind?
[11:47]
MichaelDaumnext: I can't see how to access some of the advanced features of Solr using %SEARCH as a frontend such as faceting, grouping, spell correction, highlighting, etc [11:48]
CDoty. the issue there is that different search engines support those things in different ways (if at all) [11:49]
MichaelDaum%SEARCH is too much of a bottleneck [11:49]
CDotfinding common ground is harder there [11:49]
MichaelDaumnext: it does too much heavy lifting trying to cache things ... this blows Foswiki's memory footprint [11:49]
CDotone possible strategy would be to freeze SEARCH as it is, and adopt Solr as our core search solution
i.e. don't throw the baby out, just change the bath water
[11:50]
MichaelDaumnext: %SEARCH as a way to implement WebSearch doesn't cut it ... there should be a REST json api for results ... alas there isn't [11:50]
CDotgot to agree with you there. I do like Google Drive's rest api.
another issue I have had with %SEARCH is the inability to search *rendered* results
think of it as searching over the page cache
however that was a passing requirement, and I only mention it in passing.
[11:51]
MichaelDaumnext: %SEARCH backends should check ACLs ... as part of a search+sort+limit+filter ... that is do NOT try to do ACL checking within the perl frontend ... do it on the DB backend [11:53]
CDoty, hard to do at the moment with ACLs implemented the way they are
CDot would like to move ACLs behind the store interface
[11:53]
MichaelDaumonly when ACL checking becomes part of the backend process will you finally get the performance required ... never before [11:54]
jastmy server just did another one of those spontaneous reboots [11:54]
MichaelDaum... as other wise sort + limit (and with it pagination) fail [11:54]
CDottotally
how hard would it be to do ACLs in Solr?
[11:55]
MichaelDaumSolr caches ACLs as well as the rest of a document
in a denormalized way
[11:56]
CDotdenormalized? [11:56]
MichaelDaumso checking ACLs is just another filter [11:56]
jastthe ACL in solr is a list of users that can view the topic/document [11:56]
CDotok, so canonicalised [11:57]
jastthere are certain downsides to that, as evidenced in wikis that I've seen with 20k users [11:57]
CDotthat could be a very, very long list [11:57]
MichaelDaumtheser are not stored in Solr, just indexed. so no problem with that. [11:58]
jastso they're stored in the index. ;) [11:58]
MichaelDaumno. means: you can't retrieve the list of users with view access. you can only filter. [11:59]
jastI know
still means 20k more terms in the index
and all of them used in most of the documents, so not particularly efficient
[11:59]
CDothmmm, I see that's an implementation question http://lucidworks.com/blog/custom-security-filtering-in-solr/
no reason why number of users need be a limiting factpr
[11:59]
jastI can imagine a better way to index ACLs but it requires a saner user mapping API and is impractical when you have many TopicUserMapping groups [12:00]
CDotso, anyone want to suggest why we *shouldn't* implement Solr as the core search solution? [12:00]
jastwell, java isn't exactly the most lightweight dependency to add [12:01]
CDottrue [12:01]
MichaelDaumit raises the entry boundary to Foswiki a lot [12:01]
jastI could see transitioning core search to something based on xapian, but solr is a bit much [12:01]
CDothalf measures are just that - half measures
if the industry standard is solr.....
[12:02]
jastthen we might as well add a dependency on a database and do storage right [12:02]
MichaelDaumanother candidate is Elastic Search ... there is even a perl store interface for it on cpan
here's why Solr (or Elastic) are particularly well suited for Foswiki
as a structured wiki fits excellent on the notion of facetted queries
[12:02]
jastmy main issue with solr is that there is no way to do proper joining
the join feature in solr is laughably incomplete
[12:03]
MichaelDaumformfields map to facets in queries ... i.e. list formfields [12:04]
CDot"we might as well add a dependency on a database" - Solr is actually more functional in search terms than most databases [12:04]
jastbut less functional in terms of structured searching :) [12:04]
CDotyes [12:04]
jastpostgres has extensive support for fulltext search, too, though the structure is too different to easily use it for something similar to SolrPlugin [12:05]
CDotI was sort of envisaging them living together
postgress fulltext search is not great
[12:05]
jastit's pluggable
end of story :}
[12:05]
CDotit's pretty simplistic
yes, true
so, let's envisage this. Currently we have %SEARCH with a query= that provides a minimum of search (and structured search) functionality.
[12:05]
MichaelDaumhave a look at this https://metacpan.org/pod/Elastic::Model [12:07]
CDotBacking that with a SQL DB can accelerate performance up to a point, but isn't the whole solution [12:07]
jastanother downside of solr is that it's, uh, soft realtime, to put it politely [12:07]
MichaelDaumseen a presentation about Elastic::Model ... gawd only if we could [12:07]
jastI agree that an ideal site would use both a DB and a fulltext index [12:07]
CDotCDot shots up while he goes to ready about Elastic [12:07]
jastelastic doesn't seem to have typed fields...? [12:08]
CDotCDot isn't keen on .org websites that appear to be commercial companies :-(
it's also Java, so same constraints on usage as for Solr
CDot isn't sure how big a constraint Java is, these days
don't most servers have a Java install?
sudo apt-get install java isn't exactly hard
[12:09]
jastjava is fairly memory-hungry
combine that with foswiki, and suddenly you need quite a bit more RAM than you did before :}
[12:11]
MichaelDaumfrom the base line of http://lucidworks.com/blog/custom-security-filtering-in-solr/ ... : "Don’t make the solution more complicated than it needs to be. More often than not, even access control filtering can be implemented using plain ol’ search techniques, by indexing allowed users and groups onto documents"
^bingo
[12:11]
jastunfortunately we can't easily use indexing of groups [12:12]
MichaelDaumwell there would be a solution
in not flattening nested groups down to its users
[12:12]
CDotRAM is cheap [12:13]
jasthave you look at "cloud hosting" prices?
RAM isn't cheap :}
[12:13]
CDotfunny how that particular worm has turned [12:13]
MichaelDaumand only gather: (1) user ids as listed directly in an ACL and (2) gather all groups either directly listed or nested ... during index time. then - during search time - provide the user id doing the query as well as all of his groups. [12:14]
jastand that last bit is the problem
with the standard user mapper we can't do that efficiently
[12:14]
CDotthe problem I get into here is that there is so much we *could* do.... but where's the sweet spot? [12:14]
jastwe have to iterate over all groups [12:14]
MichaelDaum^once [12:15]
jastonce per engine run [12:15]
CDotor cache them [12:15]
jastdifficult to figure out when to clear the cache
especially if you're using a third-party user mapper
[12:15]
MichaelDaumnote that we are talking about optimizing indexing performance ... not query performance [12:16]
jastI'd like to have both reasonably efficient [12:16]
MichaelDaumhonestly, the "crawler" in SolrPlugin needs to be revised first. [12:16]
jastif we pay for lower indexing cost with 400% overhead on queries (made up number), that doesn't sound right [12:16]
MichaelDaumtrue [12:16]
jastyeah, there's many ways the indexer could be improved... good thing we all have infinite spare time, right [12:17]
MichaelDaumI'd rather have a crawler sitting in the background happily chewing on content & acls [12:17]
CDotthe tradeoff has to favour interactive performance [12:17]
MichaelDaumso the question why and when to optimize indexing performance is more: when does it suck
in the sense of: when will it fail to keep up with changes
[12:18]
CDotwhen those changes are very rapid and frequent [12:18]
MichaelDaumboth type sof changes: in ACLs and group changes ... as well as content changes [12:18]
CDotMichaelDaum: are you talking about indexing only static content here, or dynamic (rendered) content as well? [12:19]
***ColasHome has quit IRC (Read error: Connection reset by peer)
Colas has quit IRC (Read error: Connection reset by peer)
[12:19]
MichaelDaumTML ... not rendered. Andreli once asked for an option to render HTML generated by the wiki ... not there yet
I wonder when you'd prefer HTML over TML ... macro expansion over not doing so.
[12:20]
CDotwhen content is extracted from another service. the specific example i had was using content extracted from a DB [12:21]
MichaelDaum%SQL for example [12:21]
CDotright [12:21]
MichaelDaumbut then: why not index the DB itself rather than going thru Foswiki rendering HTML from it [12:22]
CDotthe solution I adopted in the end there was a google search engine [12:22]
MichaelDaumall of the DB's structure is lost, no way to use columns serving as facets [12:22]
CDotright [12:23]
MichaelDaumso instead of expanding %SQL ... you'd crawl the DB directly and get richer meta data
not to mention that the DB itself comes with its own ACLs
[12:23]
CDotyeah, but in that case you then index the DB, not the wiki [12:23]
MichaelDaumthats the point
you don't index the content _view_ but the content itself right from the source
[12:24]
CDotno, you don;t *want* to index the DB. You only want to see DB content *through* the wiki
the wiki is providing the window
[12:24]
MichaelDaum... for the view ... [12:25]
CDotright
the same DB data might appear in many wiki pages
under varying ACL conditions
[12:25]
MichaelDaumsimilar other services such as Disqus
or an IMAP account
[12:25]
CDoty
anyway, indexing rendered content is a distinct requirement, though IMHO a low priority one
low priority becuase you can use a search appliance to do it adequatley
[12:25]
MichaelDaumbest would be to crawl IMAP itself ... but then add an url property pointing into the wiki where a found email can be displayed using some %IMAP wiki app [12:27]
CDotrisk of overengineering here [12:27]
MichaelDaumnot really: this is a proper separation of duties
one for search the other for view
[12:28]
CDotok, so far the main objection to adopting Solr as the core search engine is "Java"
any more?
[12:28]
MichaelDaumElastic could be better as a full fledged store
in that it supersedes Solr
[12:28]
CDotyes; but reading about Elastic makes me think that's a step too far for the core
too close to the bleeding edge
[12:29]
MichaelDaumgood point
what clients want is Oracle
and SUSE or Redhat
no matter how old the RPMs are
on the latest "enterprise" distro
[12:30]
CDotand Solr is getting there
Java is pretty much there, which is why I think it's not the barrier it once was
[12:31]
MichaelDaumyea maybe [12:32]
CDotthe alternative - re-engineering %SEARCH - is IMHO a non-starter
Sven did his best, and frankly I'm glad he gave up when he did
[12:32]
MichaelDaumhe was constantly throwing more code at the problem [12:33]
CDot%SEARCH is dead. Long live %FIND! :-) [12:33]
MichaelDauminstead of simplify and delegate to backend
or %QUERY ... ah wait :)
[12:33]
CDoty, but he was trying to achieve compatibility, which %FIND doesn't have to do, so has a much easier time of it [12:34]
MichaelDaumthe approach we have taken before is: find ways to isolate code
so: what is the maximum level of isolation we can achieve that opens up the alley for the real thing
[12:36]
CDotnot sure what you mean [12:37]
MichaelDaumevolution first - then optional revolution [12:37]
CDotyes, i understand that - but what do you mean by "12:24:35) MichaelDaum: so: what is the maximum level of isolation we can achieve that opens up the alley for the real thing" [12:38]
MichaelDaumthink of %SEARCH{ .....}% as an empty shell [12:38]
CDotok, that's not hard to so
do
[12:39]
MichaelDaumone option is to plug in the ol' lady we are used to talk to today [12:39]
CDotright [12:39]
MichaelDaumit only understands a rather limitted api in terms of parameter and what not
then move granny to the side ... softly ... and show her ways to get settled
[12:40]
CDotwell, yes, that *sounds* sensible, but it's an awful lot of work [12:41]
MichaelDaumthe only thing left in %SEARCH is rendering results ... as stupid as can be [12:41]
CDotre-engineering the existing %SEARCH to make way for alternate solutions is quite hard
and that's even before the alternate solutions are ready
[12:41]
MichaelDaumrendering results is done on an abstraction good enough to be able to delegate search+sort+limit+filter completely to the backend [12:43]
CDotCDot resolves to take another look at %SEARCH and see how credible the incremental aproach is [12:43]
.... (idle for 19mn)
jast: are you basing your new skin on PatternSkin? Or starting again from scratch? [13:02]
jastthe new skin is built from scratch [13:07]
CDotgood :-)
are you building new REST services for use with it?
[13:08]
jastyes, but not really anything that is more generally useful [13:09]
CDotsure, understood. One thing I've been thinking about is providing a more generic REST service that allows access to single macros with JSON responses. So, for example, a SEARCH service that responds with JSON (as Michael alluded to above) [13:10]
jastthe only one that would make sense to use independently from that skin is one that allows you to render multiple macros and return them separately
something generic for that could be very interesting
we use SolrPlugin's 'proxy' handler which does that kind of thing
[13:10]
CDotyes, I think so too. Will investigate further. I'm thinking of a more generic mechanism in the core. [13:12]
MichaelDaumCDot, have a look at RenderPlugin ... [13:14]
CDotoooh, another plugin ;-) [13:14]
jastRenderPlugin doesn't do anything specifically for returning 'multiple records'
that would need support from the various macros anyway
[13:14]
MichaelDaumnor does it grant json being well-formed [13:15]
CDotthere has to be three "modes"; render in place, render for REST, and simple data
simple data yields json
[13:15]
jastone of my main issues with the approach most macros take regarding rendering [13:15]
CDotand isn't applicable to all macros, just some
most macros are implemented assuming they are responsible for rendering. There is no separation between "generate data" and "render"
[13:15]
jastexactly [13:16]
MichaelDaumya [13:16]
jastthat's the biggest problem with 'programming foswiki' IMO [13:16]
CDotwell, it's *one* problem :-)
it sure makes it hard to extract generic rendering steps
[13:17]
jastthe other biggest problem is nesting [13:17]
CDot(this is the main reason I hate %TOC) [13:17]
MichaelDaum%TOC isnt a macro [13:17]
CDotCDot doesn't see nesting as such a big problem [13:17]
MichaelDaumMichaelDaum neither [13:17]
jastit becomes a big problem when you do certain things
for instance: %SOLRSEARCH{"...%URLPARAM{"foo" encode="solr"}%"}% doesn't exist
so you'd have to do %SOLRSEARCH{"...%SUBST{text="%URLPARAM{...}%" ...}%"}% and at some point escaping breaks down completely
[13:17]
MichaelDaumtoo much rendering on the serverside: go javascript [13:20]
jastso you're saying we shouldn't improve it because there are workarounds? [13:21]
CDotCDot isn't convinced [13:21]
MichaelDaumjast, patches are always welcome.
improving is good
[13:21]
jastthe problem is this is a systematic problem. patching plugins is a drop in the ocean. [13:21]
MichaelDaumthe other question is _why_ you read URLPARAMs
on the server side
[13:21]
CDotclient side rendering is fine if you have a megafast client, but I'm seeing increasing mobile/tablet use with largely brain-dead clients [13:22]
jastmaybe because a user entered a search term somewhere and clicked "submit"\ [13:22]
MichaelDaumclients are faast. they can even run doom in the browser
so hey.
[13:22]
jastmaybe I have to do several things on the server side because otherwise I'd have to do a ton of round trips using AJAX [13:22]
CDothah! When they can run GTA 5 I'll be impressed [13:23]
jastsimple things can be solved with a single call to, say, /bin/rest/SolrPlugin/proxy. complex things would require many requests (slow) or complex macro stuff on the server (error-prone, unmaintainable) [13:23]
CDotjast: +1 [13:23]
jastactually we're currently working on an internal app that goes the JS route
it's for project management
[13:23]
MichaelDaumfrom what I see wiki apps in a page are interacting with the backend on their own using ajax. no need for other objects in the page to re-render as well just because one of the widgets updated. [13:24]
jastcurrently viewing the overview page for a project takes ~20 seconds
yes, but what is the backend
if the answer is "foswiki", then it makes sense to improve the way we can generate output in foswiki
[13:24]
MichaelDaum100% agreed
thankfully we added jsonrpc
[13:25]
jastwhich provides an interface but doesn't un-screw-up the logic most macros use [13:25]
TarboxI'm still nervous about that. We do a lot of implementation hiding to please our clients. [13:26]
MichaelDaumtrue [13:26]
Tarboxabout jsonrpc* [13:26]
jastwhat we really need for a big step forward is a new interface in core that all the plugins can be written to [13:26]
CDotthe thought of rewriting all the plugins to a new interface makes my heart sink [13:27]
jastwell it would be optional
but how else would you fix rendering in all the important places
[13:27]
CDotI'm with Michael on this one. Adding jsonrpc is a gateway to much better plugins
I have already recoded a number for that model, and it makes life a lot simpler.
the internal (Func) API is pretty much OK for the jobs I do
though access to SEARCH is an ongoing problem, I grant you.
[13:28]
MichaelDaumidea: to reduce the number of jsonrpc roundtrip we could have a kind of multiplexer that calls multiple jsonrpc handlers and gathers all results in one json result object [13:30]
CDotMichaelDaum: I'd really like a client-side module that would do that sort of thing
I'd also like a way to cancel requests, but hey, I know that is unlikely
[13:30]
jastcalling multiple handlers is all fine if you know what to call in advance [13:32]
MichaelDaumthe service object in AnularPlugin does better canceling requests [13:33]
Guest31927hello. my server log tells me "Can't locate Foswiki/Form/Icon.pm in @INC" - can someone tell me which extension contains this file? [13:33]
jastbut maybe the data I can currently grab means that I only know what other data to fetch after the first request has finished, and so on [13:33]
MichaelDaumusing Deferred [13:33]
Tarboxz42k, the error log for your web server will have more detail on that error message. [13:33]
jastz4k: MoreFormfieldsPlugin [13:33]
z4kjast: thanks, will check [13:34]
jastrelevant example: tree structures
the best you can do on the JS side, given a typical data interface, is $max_depth round trips
anything else needs some kind of support for this structure on the server side
[13:35]
CDotno problem with support on the server side; would just like some sore of optimization of the RPC channel
cos at the moment it's deeply stupid
[13:36]
jastsupport on the server side at this point pretty much requires a perl module [13:37]
CDotyes, no problem with that [13:37]
jastand what I'm after is making more things possible without having to fall back on writing perl code
and without ending up with thrice-escaped macro abominations
[13:37]
CDot:-) [13:38]
jastand without issues whenever a piece of data happens to contain the wrong characters [13:38]
z4kjast: MoreFormfieldsPlugin depends on JQSelect2Contrib, and my foswiki apparently cannot download that. "Not Found". :/ [13:41]
jastdang [13:41]
z4kword [13:41]
jasthere's the problem
I created JQSelect2Contrib, and there has been a contribution on a side branch that generally would make sense to release as part of JQSelect2Contrib (so far there hasn't been any release at all)
unfortunately the changes are incompatible with the interface in 'my' version, and I'm already using 'my' version in a number of places
[13:41]
z4koh well [13:44]
jastI could make a release of 'my' version, or of the modified version on the other branch, but I don't know which one would be compatible with your setup [13:44]
z4kthe problem I'm trying to solve is actually not that bad... I wanted to be able to assign icons for blog categories properly. [13:45]
jastmy solution has been to procrastinate on making a decision. :) [13:45]
Tarboxwhere do you want the icons to show up? [13:46]
jastif you're not using the select2-based components, you can *probably* ignore the dependency on JQSelect2Contrib [13:46]
z4kbefore we saw an error message and the icon we set in the input box didn't work. now the error message is gone and the input box too \o/
if none of the other forms we use are broken we'll procrastinate a bit as well
[13:46]
jastbut I just saw the Icon type is select2-based :( [13:47]
z4kah [13:47]
jastwell it actually seems like it should be compatible with either version [13:48]
z4kso... is there a way for me to get the icons working without installing too many new plugins? maybe remove some? [13:49]
TarboxWhere do you want the icons to show up? [13:49]
z4kTarbox: in the blog categories
of the BlogPlugin
the default icons seem to work fine, but for new categories it sometimes works, sometimes not, and the behaviour at the UI is a bit strange
[13:50]
TarboxSo, you want the icon on the categories listing? [13:52]
GithubBot[JQSelect2Contrib] jast pushed 1 new commit to master: http://git.io/NWfx
JQSelect2Contrib/master 5c58582 Jan Krüger: Item12116: Get rid of invalid $VERSION string to make builds work
[13:52]
***GithubBot has left [13:52]
jastdarn, my build environment is all messed up [13:53]
.... (idle for 19mn)
CDotMichaelDaum: are you going to convert the 1.2 core scripts to /usr/bin/env? [14:12]
MichaelDaumhum, I was hesitating to do that as George once had problems with his env and almost got mad ... might be due to a mix of /usr/bin/perl and /usr/bin/env perl ...
other than that I'd really like to see that happen
[14:13]
CDotok. It's not a big deal; if it's not going to happen, it should probably still be mentioned in the release notes [14:14]
MichaelDaumay [14:15]
***ChanServ sets mode: +o Lynnwood [14:18]
ChanServ sets mode: +o gac410 [14:23]
.... (idle for 18mn)
gac410MichaelDaum: CDot: Now that cdot eliminated the need for -T, go ahead and convert to /usr/bin/env if that is known to be very portable [14:41]
MichaelDaumay [14:42]
gac410I was mostly concerned about eliminating -T testing for the default (non locale) test environment. And CDot addressed that quite nicely.
MichaelDaum: Regarding CSS change, My intention is to NOT activate it by default, but to document it for sites that want to hide stuff. It was a support question that started all that.
Forgot and left it active in my stashed changes ...
[14:42]
MichaelDaumoic [14:44]
.................... (idle for 1h36mn)
CDotwhy is the default search mode on f.o to search the entire site? It's a real pain when you want to search just in the subject area web (which should be the default for the search box in the f.o skin, IMHO) [16:20]
gac410CDot: No idea... I've been annoyed by the same thing... I usually now go to the WebSearch topic to deal with it
One heck of a discussion on the IRC logs ... very interesting futures
[16:23]
CDotyeah, it was good. Need to write it up, really. [16:26]
gac410I think that maybe pharvey was working on a pluggable render proposal at one point. Vague memories.
yup.... I knew that part of your discussion sounded familiar: http://foswiki.org/Development/PluggableRenderers
[16:27]
CDoty, it's and idea been batting around for a long time [16:34]
gac410fsfs: Something really strange is going on with the build http://fschlich.userpage.fu-berlin.de/foswiki/trunk/Foswiki-build.log
one thing, is that the required minifiers are not installed. But down further, the ordering of builds seems really strange.
It says building TopicUserMapping, but then zips the TipsContrib, and then does some TopicUserMapping. But never generates the zip files.
[16:34]
fsfsI found that very confusing. Is there concurrency involved? [16:35]
gac410Not that I've ever seen. [16:35]
fsfsit complains about missing .txt files - do they contain a kind of todo list? [16:36]
gac410It usually steps through each extension one at a time.
WARNING: no Foswiki-1.2.0_GIT-auto85d3067.txt was generated ... no that's normal. Those are only built for extensions, not the core.
[16:36]
fsfsok... [16:37]
gac410Build prereqs are http://nodejs.org/ and https://npmjs.org/package/uglify-js
That's the cause of all the squish errors. Not sure if that's related. Maybe build is getting confused somehow.
[16:38]
fsfsum, I was going to ask if I should install CSS::Minifier::XS and JavaScript::Minifier::XS?
doesn't it want perl modules?
[16:39]
gac410No. They are obsolete. Best to use the nodejs versions.
http://foswiki.org/Development/BuildingARelease has some docs.
[16:39]
fsfsok, I'll have a look [16:39]
***JulianLevens has left [16:40]
gac410MichaelDaum suggested we actually remove the "Minifier" and also the yahoo compressors and stick with just the nodejs versions. [16:40]
fsfslooking at BuildingARelease, I don't run the view script and bootstrap process form autoBuildFoswiki.pl... [16:41]
gac410Right. That's more the manual steps. "build.pl release -auto should "just work"
The bootstrap will already have been done as part of setting up the unit tests iirc
./pseudo-install -A -force is the only bootstrap needed to build a release. The steps in that topic are also for the RM to validate that it is all really working,.
[16:42]
***CDot has quit IRC (Ping timeout: 245 seconds) [16:45]
gac410http://foswiki.org/Development/BuildingARelease#Build_a_final_release_package_and_upload_to_Download_web is closer to what the autobuild does, I noticed one thing though... it says to cd to lib and ../tools/build I've never had to do that. I just cd tools && ./build.pl ... [16:46]
fsfschdir('lib'); and ../tools/build.pl is what the autobuild script does, I could change that for a test... [16:47]
gac410I'll try it here ... I know my build is working fine.
fsfs: I just attached a sample build log from a successful build to the BuildingARelease topic
Let me try again now, with a cd to lib
hm strange. CD to lib and it also appears to be working, but it's building Foswiki-UNKNOWN-autoeef2428 Not sure what that's all about, but I'll let it finish then check
Ah... never mind. I forgot to reset HEAD --hard ... UNKNOWN was my problem.
[16:48]
fsfsI think it looks better now with the uglifyjs minifier installed
I'll run a full auto-build
[16:59]
...... (idle for 26mn)
gac410: ok now? [17:26]
gac410hang on , I'll download and try [17:26]
fsfshmm, TopicUserMapping.pm is in the main tarfile, but the build log still seems a bit out of order... [17:29]
gac410Nope, same thing. I have no idea. something really strange is happening. [17:30]
fsfs:-(
what exactly is missing / in the wrong place?
[17:30]
gac410Foswiki::Users::TopicUserMapping module [17:31]
fsfsFoswiki-1.2.0_GIT-autoeef2428/lib/Foswiki/Users/TopicUserMapping.pm is there? [17:31]
gac410no. hang on let me expand again, with -v
really strange. It says it expanded. Maybe I'm screwing up somehow.
I must have moved the wrong dir. Trying onec more.
my fault :(
All set. Sorry about that. I must have un-tar'd the wrong file this last time.
Okay. So the build is good, and -latest link is working. Any problem if I add them somewhere in Foswiki.org web ... still need to look for a good location.,
I don't want the bots to kill your upload bandwidth too badly.
[17:32]
fsfsno worries about the upload bandwith [17:38]
gac410okay great [17:39]
fsfsit's the university's login and userpage server, and it's fairly recent hardware [17:39]
gac410MichaelDaum: Could you take a peek at https://github.com/colmdoyle/gh-activity It uses twitter-bootstrap, and some fontawesome fonts. Demo here: http://foswiki.org/Sandbox/GithubWidget What's the best way to host it for Foswiki.org ... assuming we want to
It's all embedded in inline html, so I assume we'd want to use pub for a lot of that.
fsfs: http://foswiki.org/Download/NightlyBuilds ... I'll re-point that to your -latest files.
[17:43]
fsfsgac410: fine by me :-) [17:46]
***ChanServ sets mode: +o CDot [17:46]
MichaelDaumgac410, well ... meh
doesn't say much
other than a list of checkin notifications
[17:47]
fsfsgac410: in the past, I've deleted the entire webspace every four days, shortly before the trunk build. Is that ok, or should I rather expire files more than 30 days old or the like? [17:48]
MichaelDaumthis widget doesnt look fancy either [17:48]
gac410yeah. I've been hunting for somthing better. [17:48]
MichaelDaumwe need a kind if fiber curve
not text but eye kandy
[17:48]
gac410I've only found a few, nothing that jumped out as really useful, but better than "No updates in the past 30 days" :(
fsfs: I don't think it's worth keeping old builds.
[17:49]
fsfsok [17:51]
MichaelDaumgac410, htis is nice http://tomgi.github.io/git_stats/examples/devise/activity/by_date.html
based on https://github.com/tomgi/git_stats
[17:51]
gac410Ah.. cool. My searches were not all that successful :)
Any other opinions on nightly builds. Any value to keep old builds, or just the latest
[17:51]
MichaelDaum"some" latest [17:53]
fsfsI was going to delete stuff older than 10 days now [17:53]
gac410last 2, last 5? I've never used the autobuild output, so it's hard for me to comment. [17:53]
fsfsit's only the main Foswiki package anyway, the others are overwritten every time there's a successful build
...there's a tape backup...
[17:54]
gac410Really no reason to ever go back. It's simple enough to git checkout <some old sha> and then build. [17:55]
fsfstrue [17:56]
gac410fsfs: http://foswiki.org/Download/NightlyBuilds updated
jast, any progress on translate? :D
[18:05]
fsfsperfect [18:06]
CDotgac410: changed it to search only in the current web (wow, that was a horrible bit of detective work, and I blame Lynnwood!) [18:07]
gac410:) phew, I ducked the "blame george" day [18:08]
CDotthat's tomorrow :-) [18:08]
gac410gac410 won't be here tomorrow. pile it on
MichaelDaum: git_stats ... seems to be server based - uses ruby gems? :(
[18:08]
MichaelDaumyea bit overdesigned too [18:12]
gac410gac410 was hoping for a little javascript widget that would show some activity. Too bad github doesn't offer anything. [18:12]
***MichaelDaum is now known as MichaelDaum_ [18:26]
Lynnwoodit’s wasn’t me! … what did you say?
ah, what the hell. i’ll take the blame.
[18:30]
.... (idle for 15mn)
CDotIt was you! Your name was all over it!
no-one can hide in Foswiki 3:)
[18:46]
....................................... (idle for 3h11mn)
***MichaelDaum_ has quit IRC () [21:57]
...... (idle for 28mn)
RiskRewardHi all, I'm moving our wiki from the windows box to a linux machine running the Foswiki virtual appliance today. Is there a guide, or does anyone have any tips? [22:25]
gac410hm nothing specific I can think of. What are you running? vmware?
Just to be sure I understand, you are running the foswiki vm as a guest on a linux machine? Or is the host windows?
[22:26]
RiskRewardHost is a linux machine. HyperV 2012 R2 [22:28]
***GuilainC has quit IRC (Remote host closed the connection) [22:30]
RiskRewardIs it just a matter of overwriting the 'data' folder on the target machine, with the equivalent folder from the original install? [22:35]
gac410Same version of Foswiki, yes. Well, more than data though. Just need to be concerned about case sensitivity.
data/ pub/ and working/work_areas esp. if you are running mailercontrib to send emails.
[22:36]
RiskRewardOK, yes, I think mailercontrib is sending emails. I know I need to avoid all the .pm files, because I had to rewrite the shebangs to get them working in windows. [22:37]
gac410If you've installed plugins, you'll need to reinstall or copy over lib. I'd be more inclined to reinstall. yeah as you said, shebangs and getting dependencies in the system updated too
Also I'd probably reconfigure rather than copy over LocalSite.cfg ..
[22:38]
RiskRewardYes. I'm thinking reconfigure, install plugins, then copy data/ pub/ and working/work_areas. [22:42]
.......... (idle for 49mn)
gac410One more thing, the virtual machine has not been updated in a while. So best to run apt-get update, and then apt-get upgrade, to pull down latest updates [23:31]
RiskRewardOK thanks, I'll do that now. [23:45]
It's upgrading now. Earlier when I tried to install one of the extensions it told me that JSON >+2.51 was required, and that the CPAN module was not installed. Are these fixed by the upgrade? [23:52]
gac410Not sure. It might be. I wouldn't be too concerned about the JSON version. lemme go look.
Oh... if it's not installed, yes you need to install it apt-get install libjson-perl I think
For cpan deps. lets say Foo::Bar you would apt-get install libfoo-barperl prefix with lib, suffix with perl
See http://trunk.foswiki.org/System/SystemRequirements#Specific_distribution_details I'm pointing you at trunk, because that version of the requirements is a bit more up to date.
though all the basic stuff should already be installed in the vm.
[23:53]
RiskRewardThanks, I'll start reading. [23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)