#foswiki 2017-10-06,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
gac410Hi RiskReward [01:08]
RiskRewardHi [01:08]
gac410If you are using mod_perl, then apache restart after module changes. FCGI, you can just kill the backend handlers
Or tweak your apache config to allow a url to run as cgi instead of fcgi
[01:08]
RiskRewardThanks. [01:09]
gac410Another way of testing is to run the bin/view script directly. ... just make sure you use the web server user, www-data or whatever, or you'll change ownership of critical files. [01:09]
RiskRewardBy the way, I'm trying to find a way to read the text of a topic into a variable that I can then process with sha256. I can convert a string I define, but can't figure out how to read a topic's text into a string. [01:10]
gac410Or look at UnitTestContrib. You could add a test to exercise the macro
the raw un-rendered text?
[01:10]
RiskRewardYes, I'd like to try it with that first. [01:10]
gac410my ( $meta, $text ) = Foswiki::Func::readTopic( $web, $topic, $rev ) That will return a meta object, and the topic text without the meta [01:13]
FoswikiBothttps://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Func [01:14]
gac410or the deprecated Foswiki::Func::readTopicText( $web, $topic, $rev ) which returns the text and embedded meta [01:14]
RiskRewardThanks, I'll give it a quick go. [01:15]
gac410Or ... You can use the my $meta = Foswiki::Meta::load(); my $rawtopic = Foswiki::Serialise::serialise($meta, 'Embedded') to return the full embedded store form of the topic. [01:17]
FoswikiBothttps://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Meta https://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Serialise [01:17]
gac410unfortunately there are multiple ways to solve it. [01:17]
RiskRewardGreat, well, I'll try to get at least one working :)
What does this mean: " Global symbol "$web" requires explicit package" ?
[01:18]
gac410hm... maybe you didn't declare it ... my $web = ... ? [01:21]
RiskRewardYou were right about the declaration.
Strange, I'm comparing checksums from the topic, with ones I create on my PC. They match if there is only one line in the topic. If there's more than one line, they don't match.
[01:35]
...... (idle for 25mn)
I think it has to do with the way newlines are treated [02:02]
***ChanServ sets mode: +o gac410 [02:06]
ChanServ sets mode: +o Lynnwood [02:18]
GithubBot[distro] vrurg pushed 2 new commits to Item14237: https://git.io/vd0mu
distro/Item14237 b91eca8 Vadim Belman: Item14237: More docs on v3 concepts...
distro/Item14237 cfa7907 Vadim Belman: Item14237: Support for extension class mapping...
[02:29]
***GithubBot has left [02:29]
FoswikiBothttps://foswiki.org/Tasks/Item14237 [ Item14237: Implement Development.OOConfigSpecsFormat proposal ] [02:29]
vrurggac410: good evening! I probably have now enough docs to have them inspected. [02:30]
gac410cool [02:30]
vrurgIf you meet Michael tomorrow it'd be nice to have him informed too.
Leaving now. cu!
[02:31]
gac410okay cu! g'night [02:31]
............................................. (idle for 3h43mn)
***ChanServ sets mode: +o MichaelDaum [06:14]
....................................... (idle for 3h10mn)
zak256MichaelDaum: I just tried DBCacheContrib::Archivist::Storable because we are lucky to have a lot of RAM in the machine. Still one single DBQUERY takes a lot of time (about 50 seconds) if there are more than 1000 results. Can you just confirm that this is to be expected for this number of results?
(I deleted and recreated the working dir of course and restarted the webserver.)
[09:24]
........... (idle for 53mn)
MichaelDaumSereal is preferable to Storable in any case, no matter whether you have lots of ram or not.
a slow dbquery can be caused by these factors: (1) is this an initial hit to the search? note you pay for loading the dbcache into memory. means, are subsequent reloads of the same page as slow as the initial one?
(2) are you really using fcgi? note that only then the dbcache will stay in memory across multiple requests ... and thus boost results
(3) is {MemoryCache} enabled
(4) try the same query but reduce the format="..." string to the absolute minimum. a query can be slow because the _formatting_ part takes long
(5) how big in KB is the resulting HTML page? see the browser's dev tools. It simply might be too much for foswiki to render, and the web server to transmit over to the browser and the browser to render on the screen
[10:18]
....... (idle for 31mn)
zak256Hi Michael. I already switched back to Sereal.
(1) I tried several times and reloaded the page but it wasn't considerably faster.
(2) Yes FCGI is activated, I saw error messages from fcgid before in the apache log and had to increase FcgidIOTimeout for example, so I assume it works. References to foswiki.fcgi are set in httpd conf as well.
(3) Yes: cfg{DBCachePlugin}{MemoryCache} = 1;
(5) Page size is 1.49MB, but network shouldn't be that much of an issue as I am requesting the page in a fast local network. But maybe I should increase some fcgi-memory-parameter?
(4) That is the only thing that I think is the issue here: There are several CALCs, LISTMAPs end Links created in the table.
(5) There is only one big table shown, rendering takes maybe 2-3 seconds I guess.
By the way, a similar table with DATATABLE takes about 20 seconds.
[10:53]
MichaelDaumtry replacing %CALC{} with %CALCULATE{}
as possible
or just for testing swap format="..." to DIS_format="..." ... is this faster?
how many rows does it render in this table?
[10:55]
zak256there are about 1100 results, so ~1100 rows [10:56]
MichaelDaum%DATATABLE will be faster, i.e. if you enable pagination. [10:56]
zak256It is an extreme example yes. [10:56]
MichaelDaumwith pagination turned on will DATATABLES only fetch ... and render ... the first few results ... however you won't be able to use %CALC for spreadsheeting of course ... [10:57]
zak256What means DIS_format? Just trying... [10:57]
MichaelDaumDIS_abled [10:57]
zak256ahh..ok [10:57]
MichaelDaumDIS_format isn't red ... and so %DBQUERY will use its default value for format ... which is format="$topic" ... kinda minimal to get rendering results out of the equation temporarily [10:58]
zak256Unfortunately DATATABLEs are no alternative although I really like them. But I cannot modify the contents here and for example create a link to the topic found.
ok... without "format" it takes 926ms to load
[10:58]
MichaelDaumhave a column called "Topic" [10:59]
zak256the corresponding query I mean [10:59]
MichaelDaumright, so now we know it is not the querying ... but the formatting of 1100 rows that takes 49 seconds [11:00]
zak256yes [11:00]
MichaelDaummeans: _you_ are the one to blame :DDD [11:00]
zak256:-( Yes... I take it
So shall I replace CALC with CALCULATE now?
I will try that...
[11:01]
MichaelDaumdoes the %CALC perform table operations, such as $SUM(), $T() and the like, referencing other table cells? [11:02]
zak256No, we use $LISTMAP() mostly [11:02]
MichaelDaumtry switching it off $percntCALC<nop>{ ...} ...
what does the clock say then?
[11:02]
zak256there are sometimes more than one value(=topic) in a formfield and we need to read the formfields of all those topics again. Like we have a Form for people and "Name" and "Surname" in it and then it says maybe "members=Person1,Person2" and we want "John Doe, Foo Bar" as result in the table.
I will switch it of... just a sec
[11:04]
okay... now it took 67seconds. I changed $percntCALC{... to $percntCALCX{...
I will perhaps just remove them
Ok... much better now without the CALC. But still 14.5seconds, even after a second reload.
There are now still 5 IF clauses. I will remove them as well...
10 seconds
With just header="|*Topic*|" format="| $topic |" it takes 1.5 seconds.
ok, now I wrote: format="| [[$web.$topic][$formfield(Name)]] | $percntFORMFIELD{\"Icon\" topic=\"$formfield(Status)\"}$percnt |"
And that takes 3.5 seconds. It seems the cost are all those extra operations.
[11:10]
MichaelDaumxactly. dbquery won't help you there. it really is the amount of workload per row.
which is multiplied by 1100 ... and therefore the problems
[11:24]
zak256I see... [11:25]
MichaelDaumlikewise, you could try to use the limit="100" (and maybe sort="modified" reverse="on") parameters of %DBQUERY but still use expensive formatting ... and this will perform in an acceptable fashion again [11:26]
zak256Yes, maybe I will implement something like that here. But as I said, this is an extreme example. Usually there are not that many results from a DBQuery, so others will hopefully be acceptable. [11:27]
MichaelDaumbtw I think $percntFORMFIELD{\"Icon\" topic=\"$formfield(Status)\"}$percnt can be rewriten to $expand(@Status.Icion)
@Status.Icon ...
[11:28]
zak256Oh? I will try... [11:29]
MichaelDaumgiven the Status formfield is a link to another topic [11:29]
zak256yes it is
there are several such expressions there... I will rewrite them all and see what that does
So in general: What should be preferred? CALC or CALCULATE?
[11:29]
MichaelDaumby default use %CALCULATE [11:33]
daemonCan't locate setlib.cfg in @INC (@INC contains: . /usr/local/lib/perl5/site_perl/mach/5.24 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.24/mach /usr/local/lib/perl5/5.24) at /usr/local/www/foswiki/bin/foswiki.fcgi line 33.
ah ..
[11:33]
MichaelDaumonly use %CALC if it ultimately requires the table context [11:33]
daemonno still a problem hmm [11:33]
MichaelDaumoutside any table clearly does not require %CALC. then %CALCULATE is the one [11:33]
daemonsetlib.cfg ?
root@stan:/usr/local/www/foswiki # find . -name "setlib.cfg"
./bin/setlib.cfg
#
seems to exist
what did I mess up
[11:34]
zak256Ok, thanks Michael. You have helped me out here greatly again. [11:34]
MichaelDaumdaemon, access rights? you are logged it as root ... yet the server needs read AND write access to files as user www-data (or www, or apache depending on your distro) [11:35]
daemonMichaelDaum, logged in as root to restart apache I did an update to this system a week ago [11:35]
MichaelDaumzak256, y' welcome. [11:35]
daemonjust noticed the wiki is no longer working
MichaelDaum, permissions for everything are www:www but permissions should not have changed anyhow
[11:35]
MichaelDaumtry this as user www: cd <foswiki-dir>/bin; ./view >/dev/null [11:36]
daemonMichaelDaum, indeed:
root@stan:/usr/local/www/foswiki # ls -la bin | grep setlib
-r--r--r-- 1 www www 3978 Aug 10 00:27 setlib.cfg
hmm how do I run things as www with su
have to hit the man pages
one second
ah its this crap again
#! /usr/local/bin/perl5.24.2
perl5-5.24.3 Practical Extraction and Report Language
what is that command to get it to update all its interpreter paths again
[11:36]
MichaelDaumtry rewriteshebang.pl in the tools directory
note that if you are using fcgi (and you should) you will only need to fix bin/foswiki.fcgi
[11:40]
daemonthere we go
she is working now
I think I am ...
[11:41]
MichaelDaumyay [11:41]
daemon[Fri Oct 06 14:42:21.484373 2017] [authz_core:error] [pid 616] [client 172.19.18.1:23099] AH01630: client denied by server configuration: /usr/local/www/foswiki/bin/foswiki.fcgi
looks like fcgi
[Fri Oct 06 16:44:39.360270 2017] [fcgid:warn] [pid 615] (54)Connection reset by peer: [client 172.19.18.1:53356] mod_fcgid: error reading data from FastCGI server
^ old errors
none now
got a rush of work on
now time to pick up the gauntlet https://wiki.tmp.group/HWMatrix
-_-" que the headache
thank you for the help
[11:42]
MichaelDaumhave fun with your new foswiki [11:43]
daemonnot new :P
infact I bet I am behind
what version are we on now days
ah not miles behind
Assert.pm:our $VERSION = "1.200";
Foswiki.pm: use version 0.77; $VERSION = version->declare('v2.1.4');
even
[11:45]
.......... (idle for 45mn)
zak256Can I maybe write the following any shorter as well?
$percntCALCULATE{\"$LISTMAP([[$dollaritem][$NOP($expand(@Person.Name) $expand(@Person.Surname))]], $formfield(Person))\"}$percnt
[12:34]
.... (idle for 15mn)
Argh... this doesn't work if the primary formfield can contain topics with different forms (but all the field 'Name') :)
:(
[12:49]
..... (idle for 24mn)
***ChanServ sets mode: +o gac410
ChanServ sets mode: +o Lynnwood
[13:13]
vrurgMichaelDaum: are you here yet? :) [13:23]
FoswikiOnSlack<lynnwood.brown> MichaelDaum -I'd be interested in hearing your take on how difficult it would be to implement a plugin enhancement I idea that I thought of last night. Basically it's transferring a feature from one of your plugins to a similar function in another plugin: transferring the DBSTAT feature to FilterPlugin to allow for generating stats on any list.
<lynnwood.brown> Just throwing it out there... not looking for immediate response. I have been needing to calculate various stats on list recently - and have been doing it rather crudely by running through FILTERLIST repeatedly and include only certain records and getting counts. It then occurred to me that this is exactly what DBSTATS does so much better.
<lynnwood.brown> It's not far from what is already implemented by MAKEINDEX in FilterPlugin..
[13:25]
zak256gac410: I looked at https://foswiki.org/Download/FoswikiReleaseViewTemplate?raw=on for an example on how ViewTemplates work but didn't understand the %TMPL:DEF{"junk"}% at the end. Can you explain what that does?
I thought that maybe it is just used to hide the text from the normal view, but when I set some parameters in my testpage these didn't work.
[13:38]
gac410Ah... It's an unused template that sets "settings" for the template topic when viewing it directly. [13:40]
zak256yes... with "unused" you mean I could just as well write "foobar" instead of "junk"? [13:41]
gac410So if you view/Download/FoswikiReleaseViewTemplate, then the settings get applied. But if you expand the templates when viewing another topic, they are ignored.
Yes.
[13:41]
zak256Ok, that's what I assumed.
But nevertheless, my Set parameters didn't apply.
They didn't even apply when I put them into %TMPL:DEF{"content"}%
I had to write them directly into the page.
[13:41]
gac410I don't know. Now that the templates support comments, that whole idea of a "junk" template is probably not needed. [13:43]
zak256Ah... stop... I mix up somthing there I think...
These settings are for the ViewTemplate itself?
[13:43]
gac410yes [13:43]
zak256Ahh... then it's clear.
I thought it's used for the pages the template is for.
[13:44]
gac410no. I don't think you can assign settings in a template that way. Though you can use %SET [13:45]
zak256Okay. [13:45]
FoswikiOnSlack<lynnwood.brown> What would really be sweet for doing statistics in Foswiki generally would be an R plugin.
<lynnwood.brown> I'm just trying out a basic tutorial in R and amazed how user-friendly it is for creating basic lists and running calculations on them.
[13:51]
....... (idle for 32mn)
<lynnwood.brown> gac410 - speaking of %SET, I was wondering something about it yesterday. Say I use %SET to define a function that returns a set of values. If I then use the macro defined in the %SET macro, does it then expand the function or the set of value? Do you get my meaning and do you have any idea off hand? [14:26]
Lynnwoodslacks ok... but i like simplier irc...
i commonly use and INCLUDE, inside another INCLUDE to assign a generated set of values to a parameter and I have been wondering if I could do something like that with %SET.
[14:27]
gac410I believe %SET is subject to the inside-out l-r rules like any macro. The docs were very wrong though. I updated them here: https://github.com/foswiki/distro/blob/master/core/data/System/VarSET.txt [14:30]
LynnwoodMost excellent! There's my answer right there.
and that makes perfect sense.
That really is extremely useful in a wide variety of settings.
[14:31]
gac410y. iirc though, the only thing %SET cannot influence is topic ACLs. Those are evaluated and applied during toipc load, before the rendering process starts. [14:33]
Lynnwoodthat sounds like a good thing... [14:33]
gac410yes.
t* did add dynamic ACLs. tbh I think that's a mistake.
[14:33]
LynnwoodIt's hard enough to keep track of them on the topic level as it is... [14:34]
gac410%SET eliminates the need to use %CALCULATE{$SET() to create cross-include settings. [14:35]
Lynnwoodexactly... I was doing a lot of that.
I"ve been thinking about a way to generate reports on topic-level permissions. Something like System.SitePermissions, but for the topic level.
I have been meaning to take a look at System.SitePermissions to see if i could apply the same basic techniques.
[14:35]
gac410y that would be nice, A macro that could expand the ACL "trace" capabiility without generating all the verbosity. [14:37]
Lynnwoodyes. This would be very helpful in case where one wants to broadly restrict access to a particular web but need to leave a number of topics available. E.g. limiting access to user info in Main web - similar to what I notice was done with System recently.
At present, i most just apply global permission to a web and then see what breaks...
very scientific method...
[14:38]
gac410y, we've been trying to set all the defaults in the System web so that it can be restricted without any negative consequences. I think it's mostly there. [14:40]
LynnwoodI'm being asked to generate stats about site visitation by processing the events.log files. To make it easier to access this info within Foswiki, I've created shell script that gathers non-sensitive info from events.logs and storing them in Json format in text documents in pub directory, the INCLUDING them into FORMATLIST macros to process.
Although... looking at R this morning, i'm thinking it might just be easier to go ahead and use that in shell script to calculate the stats I need and store that in text files - which I can then feed to charting macros.
(actually, i've been using Highcharts for charting)
But... not sure i have time this round to learn enough R to accomplish that... :-(
So back to doing it by brute force with FilterPlugin.
Lynnwood has been learning to like shell scripts a lot.
[14:43]
gac410I've long wanted to build a LOGVIEW macro that processes the eachEventSince API. Actually an interim version of configure had a logviewer screen hidden off in github that was never ported to CDot's rewrite [14:50]
LynnwoodThe SEARCHLOGS macro in in ContributorsPlugin is pretty rough around the edges. [14:52]
gac410ContributorsPlugin won't work any more. At least parts of it. It reads cgi session files, and their format has changed. [14:53]
LynnwoodI decided to go the route of processing the events.log file directly via shellscript because of the server load involved with that macro.
I haven't run accross the parts that don't work. However, in my experence the SEARCHLOGS macro is pretty iffy, particularly as relates to date-range selection.
[14:53]
gac410Anything that reports users currently online, or the counts of guest visits.
For one thing, we now recommend disabling guest sessions, so that breaks it as well.
[14:55]
Lynnwoodwhy disable guest sessions? just session file proliferation? [15:08]
gac410at least for public sites, the search bots create 1000's of sessions. Especially since they don't return session cookes, so every hit is a new file. [15:08]
Lynnwoodright... absolutely.
Lynnwood notes to go back and check a few hosted public sites.
[15:09]
gac410Probably not an issue for private sites. They can also use the NO_FOSWIKI_SESSION env variable to suppress their local indexing engines.
disabling guest sessions is still marked experimental in configure, but I think we've got it working well now, so I was planinng on removing that warning maybe in 2.2
[15:10]
Lynnwood👍 [15:15]
....................... (idle for 1h54mn)
***zak256 has left [17:09]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)