#foswiki 2017-09-14,Thu

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***ChanServ sets mode: +o MichaelDaum [06:10]
.............................................................. (idle for 5h6mn)
ChanServ sets mode: +o Lynnwood [11:16]
..................... (idle for 1h41mn)
ChanServ sets mode: +o gac410
schuele has left
[12:57]
........ (idle for 36mn)
zak256gac410: I successfully tested Item14463. Is there anything else you need? [13:35]
FoswikiBothttps://foswiki.org/Tasks/Item14463 [ Item14463: Verification for {FeatureAccess}{Configure} in configure fails to handle login names (cont.) ] [13:35]
gac410Hi zak256, i appreciate the testing. Thanks. At this point the fix is checked into git, so it's ready to go, nothing else is needd. [13:37]
zak256So it will be in 2.1.5 ? [13:37]
gac410Yes, 2.1.5. But no planned release yet. Interesting comment about the group listing users that were not known. That's actually a feature - You can add members of a group that don't exist. [13:38]
zak256Yes, thought so as well. Everything works as expected.
Ah... Development/ReleasePlan still mentions 2.1.4 as the next release ;)
[13:38]
gac410But I suppose for AdminGroup there is also a risk there that someone could register and "claim" the admin rights. I don't recall though why we don't enforce that users must exist, but I'm fairly certain it's (non-enforcement) needed in some situations.
The lazy release manager doesn't keep this stuff updated. :P
[13:40]
zak256Well... I think admins should be aware of that risk when adding users who don't exist yet to the AdminGroup. [13:41]
gac410In some cases with external (ldap, etc) mappers, it's normal to have users log in without wiki identities. Groups would contain the login name iirc. I don't use ldap myself so I'm not really qualified to comment. [13:43]
zak256Nevermind. I am sure I can come up with some different problems as well in the future. [13:45]
gac410Opportunities for improvement. [13:45]
zak256Sure.
Ah, yes... while we're at it... for some reason bulk_copy.pl gives me this:
Use of qw(...) as parentheses is deprecated at .../lib/Foswiki/Configure/Load.pm line 99.
And afterwards:
can't lock cache db: Bad file descriptor at .../lib/Foswiki/Cache/DB_File.pm line 197.
Do you know what's wrong here?
[13:46]
gac410That's strange. In 2.1.4 Load.pm Line 99 is a blank line. [13:49]
zak256ok, that would be our old wiki code...
98 foreach my $var qw( DataDir DefaultUrlHost PubUrlPath WorkingDir
99 PubDir TemplateDir ScriptUrlPath LocalesDir ) {
[13:49]
gac410Oh. Yeah qw() deprecations and a bunch of other perl deprecations have been fixed in recent foswiki. [13:50]
zak256Yes, I guess the warning is not important. But what about the cache db error? [13:50]
gac410Right. $var ( qw( .... ) ) Needs an extra parens.
I really don't know. What release are you running.
[13:51]
zak2561.1.2 :-( [13:52]
gac410If you are tying to mgrate from 1.x to 2.x, I'd recommend using charsetConverter and not bulk_copy [13:52]
zak256Is there a HowTo for this? [13:53]
gac410Stay with whatever store you are using - RCS Wrap or Lite. Dont convert to plain file.
Hang on ... it's written up somewhere.
[13:53]
zak256Really? I thought plainfile would be better...? [13:53]
jastit is, particularly if you have many revisions... but if you're not having issues, staying with whatever you're using now makes the transition a fair bit simpler [13:54]
zak256So I can switch the store later? ok, that makes sense I guess.
Right now we use RcsWrap
[13:54]
jastbulk_copy will get the job done, too, but it's somewhat strict about data and likes to abort in the middle if it sees something it can't make sense of
plus it tends to take its time converting data )
[13:55]
gac410bulk_copy is very particular - it depends on the source store being "perfect" It skips over files that the store API cannot see. Like hidden attachments, subdirectories and other cruft that builds over time. [13:55]
zak256okay... I will think about using charsetConverter [13:56]
gac410right. It converts every revision of every topic/attachment, one at a time. 1000 revs on a file, that's 1000 conversions. It can be very slow. [13:56]
zak256It's this one here? https://foswiki.org/Extensions.CharsetConverterContrib [13:57]
gac410Yes . Also Foswiki:Support/Utf8MigrationConsiderations has more information on CharsetConverter [13:57]
FoswikiBothttps://foswiki.org/Support/Utf8MigrationConsiderations [ Utf8MigrationConsiderations ] [13:57]
gac410And it's also covered in the Foswiki:System.UpgradeGuide [13:58]
FoswikiBothttps://foswiki.org/System.UpgradeGuide [ UpgradeGuide ] [13:58]
zak256Okay, I will read that now. [13:58]
gac410CharsetConverter using -web= and -charset= options, can convert "in-place" on a 2.x system. So you can get 2.x running, then copy over one or more webs data/pub into the new 2.x systems, and convert in place by -web=Someweb -charset=cp1252
Geneerally you want to override the foswiki 1.x default of iso-8859-1 to be cp1252 if you have any windows users who cut/paste into topics.
CharsetConverter is **much** faster. In some cases minutes vs. hours
[14:00]
zak256our 1.1.2 uses {Site}{CharSet} = iso-8859-1 [14:01]
gac410right .. .That's the default. But Foswiki 1.x does not "enforce" that charset. So if windows users paste in from MS Word, then you most likely actually have cp-1252 [14:02]
zak256The migration time is not so much of an issue. I just want to have it clean afterwards.
Yes, if some pages still have invalid chars in the text I think it should be corrected manually afterwards then. Shouldn't be that much.
I want to use utf8 everywhere in the end.
[14:02]
gac410It can be very difficult to find. Store can just "crash" with no indication of what topic had the bad data.
The CharsetConverter -i option to inspect is very handy. And yes that's the objective get to utf-8
Also really important be on the newest perl you can get. At least 5.20 or newer.
[14:03]
zak256CharsetConverter is a plugin for Foswiki? Can't I just use it once on the CLI? [14:04]
gac410It's a contrib. It is CLI only. [14:05]
zak256ah, okay
Unfortunately RHEL7 still has Perl 5.16 :(
[14:05]
gac410it's not a plugin, so no active code in foswiki other than the system topic.
Foswiki 2.x will be slower on older perl.
[14:05]
zak256I know...
But the first goal is to get the new version running, hopefully with some caching added to minimize the performance loss.
If migration of data is finished and everything works, I want to continue improve everything and maybe get a newer perl
[14:05]
gac410Perl 5.16 was last updated back in march 2013 .... I just don't understand why rhat insists on bundlig old obsolete non-performing software.
But that's not a problem to solve here :D
[14:07]
zak256But especially with perl it's not just compile-your-new-version-and-be-done-with-it.
It's hard enough to satisfy all the module dependencies.
I get Email::Simple in version 2.203 from epel for example.
[14:07]
gac410The areas where older perl is slow is the regular expressions over unicode strings. which foswiki uses internally. All regexes will be a bit slower. [14:09]
zak256But PerlDependencyReport tells me at least 2.206 is required.
I know, I know... If you can point me to some rpms with newer perl and modules for RHEL7, I will happily consider using them.
[14:09]
gac410Hm... I don't recall why that particular version is needed. some of the out-of-date modules can be ignored in many cases. [14:10]
zak256For now it works, so I just use it and hope nothing breaks. [14:11]
gac410I have no idea on rhat. I don't use it and would never use it or centos. But that doesn't help you much. [14:11]
zak256No, it's the default here.
Anyway... my next step is to get a data migration concept, so I will try CharsetConverter now.
[14:11]
gac4102.204 and 2.206 added raw header support. iirc we use that in some cases. [14:13]
zak256Can you say which? [14:13]
gac410header_raw was added in 2.204 and then fixed to be compatible with Email::MIME in 2.206 https://metacpan.org/changes/distribution/Email-Simple
Its used by Foswiki::Net when sending email using Net::SMTP if you are using mailprogram and letting the OS agent send email, then it doesn't look like it gets called.
[14:14]
FoswikiBothttps://trunk.foswiki.org/System/PerlDoc?module=Foswiki::Net [14:16]
zak256Yes, MailProgram is set and configured to /usr/sbin/sendmail -t -oi -oeq
But when doing a test registration I got the email.
[14:17]
gac410We thought long and hard about forcing foswiki to remain out-of-box compatible with rhat / centos But we really had to give up too much.
Right. MailProgram does not go down the path that uses the header_raw calls. So you should be okay.
[14:18]
zak256I guess with RHEL8 it will be better supported ;)
okay great.
[14:18]
gac410No idea about RHEL8 ... they stay sooo far backlevel it's really pathetic. [14:20]
.... (idle for 19mn)
zak256Hmm... CharsetConverter gives me the same error:
can't lock cache db: Bad file descriptor at /opt/wiki-test/current-wiki-dump/lib/Foswiki/Cache/DB_File.pm line 197.
[14:39]
gac410zak256: I'd recommend running CharsetConverter in the new Foswiki 2.x system.
Using the -charset=cp1252 and -web= option
not sure why the Cache is involved. I've never used the cache on Foswiki 1.x Though it works much better on Foswiki 2.x
[14:40]
zak256ok, in my Foswiki-2.1.4-dir it works.
why should I use -charset=cp1252 ? Do you mean -encoding=cp1252 ?
[14:41]
gac410whoops. Senior moment. Yes indeed -encoding=cp1252 [14:43]
zak256But encoding should be iso-8859-1 ? [14:43]
gac410Again, foswiki 1.x pretty much ignored the encoding. If your users use Windows ... then it's very likely that topics contain smart-quotes and windows specific characters. [14:44]
zak256I understand. But we have non-windows-users, too. [14:45]
gac410iso-8859 doesn't contain the windows specific characters. sites that we've talked to have had best luck with using cp1252. It's generally a superset of 8859 [14:45]
zak256ah, okay
Can I just convert single pages with CharsetConverter, too?
[14:45]
gac410So a strict 8859-1 content will read just fine using cp1252, [14:46]
zak256Otherwise I would just remove the not-to-be-converted pages from my testcopy. [14:46]
gac410Why would you want to not convert any pages? [14:47]
zak256I want to use the latest versions of pages in System and Main and just migrate our additional pages. [14:47]
gac410Oh... NEVER ever migrate anyting in System. You need to install all that content as new.
And in Main, there are only a small number of topics that we ship. You can copy them from the distributed Main web after your conversion.
[14:48]
zak256okay, no problem. I just recreate the pages then and copy the content. But what about all the users pages for example.
Okay
[14:48]
gac410I'd recommend setting the 214Main web aside. Copy in your 121Main into the new 2.x wiki as Main. Run the converter on it. And then copy over any of the new files from 214Main that you want to update.
same thing for Sandbox
[14:50]
zak256I already made a list of pages in Main... I will do something like that.
Sandbox doesn't need to be migrated. I don't think anyone will miss anything.
[14:51]
gac410y that's probably the best.
The thing you don't want to do is to accicentally convert anything twice. 1252 -> utf8 good. utf8 (read as 1252) -> utf8 garbage.
It's highly unlikely, but if you really do have a mixed bag of cp1252 and other oddball encodings, then the conversion gets pretty difficult.
[14:51]
zak256No that won't happen. I just try to create a concept and work with copies only. If everything is set and tested, I will perform the conversion during a downtime.
And if really some pages break, there will still be the old directory.
[14:53]
gac410when we converted foswiki.org, there were only 1-2 pages that had troubles. And our Sandbox is a huge mess of experimentation from around the world. So it was interesting. [14:55]
zak256:-) I can imagine [14:55]
gac410Also we ran Foswiki 1.1.9 and Foswiki 2.0 in parallel with symlinks to the same webs. So it was challenging. [14:56]
zak256It was done without a downtime? [14:56]
gac410No we had downtime, but we actually converted 1.1.9 to utf-8 first, so it could coexist with 2.x And did have quite a bit of downtime caused by crashes with invalid data
but we were still learning, and found bugs long since fixed ... and our data was particularly bad
[14:58]
zak256How do I tell convert_charset.pl which source-data dir to use? [14:59]
gac410It uses that directories for the wiki you are running it in.
Ie it expectes a configured foswiki installation. and converts in-place.
[15:00]
zak256Ah, so I need to copy my data-dump into this directory and everything will be done inplace...
yes
[15:00]
gac410Right. And with the -web option, you can do a web at a time, or whatever is managable for you. [15:01]
zak256convert_charset.pl keeps telling me: $Foswiki::cfg{Site}{CharSet} is set to 'utf-8'. Is that correct for the data in your Foswiki database? [y/n]
but I changed it to iso-8859-1, like in our current running (old) wiki
[15:09]
gac410y. As long as you are overriding it with -encoding. [15:09]
zak256./convert_charset.pl -a -web=Main -encoding=cp1252
so, I can safely answer 'y' here?
[15:09]
gac410Foswiki 2.x cannot use any other charset than utf-8. even if you change it, it gets changed back on initialization. [15:09]
zak256ok [15:10]
gac410Y es' [15:10]
zak256Whoa... that was quick... Converted 1690 [15:10]
gac410y.... much much faster than bulk_copy [15:10]
zak256So I don't need to use the -r option to repair? [15:11]
gac410no. -r is only needed if you find mixed character sets.
and the default conversion is failing. -i though is important. It will inspect without any changes, and tell you if it finds anything bad
Iinitally we thought that -r and some optional perl modules could detect mixed character sets. but discovered that in the vast majority of cases, encoding=cp1252 was the solution, -r is way too complex.
detecting the character set from the raw data is very very unreliable.
[15:11]
zak256Great. So my plan: 1. Setup a recent Foswiki-installation without too much configuration (DataDir and PubDir), 2. Install CharsetConvertContrib, 3. copy over the webs I want to convert to these directories, 4. run convert_charset.pl as above for those webs, 5. Really setup the new Foswiki-installation, 6. Copy over the pages I need resp. overwrite/merge the ones from the new archive [15:15]
gac410Sounds good. If you have Webnames or user WikiNames that contain non-ascii characters, then you may need to deal with .htpasswd or working/work_areas/MailerContrib [15:16]
zak256that's not needed, currently there is an NTLMv2 authentication working [15:17]
gac410Don't forget to migrate those files. Withtout the work_areas then all the changes will be re-notified.
First time mailnotify runs with the missing workarea ... you will send a LOT of emails. ;)
[15:17]
zak256Uh-oh... thanks for the hint.
I just recognized these files.
I can just copy them over, right?
[15:18]
gac410yes. It's pretty unlikely that the topic and web names themselves contain non-ascii characters [15:19]
zak256No, there are just 6 files in there with no unusual characters
And I can leave out the ones for the webs which are not even migrated.
[15:19]
gac410the file just contains the timestamp of the last notification run for that web. [15:20]
zak256Yes, I have seen it. [15:20]
.......... (idle for 48mn)
***ChanServ sets mode: +o cdot [16:08]
....... (idle for 31mn)
cdotLynnwood: u there? [16:39]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)