#foswiki 2017-03-31,Fri

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
***ChanServ sets mode: +o Lynnwood__ [00:42]
.................................................................... (idle for 5h38mn)
ChanServ sets mode: +o MichaelDaum [06:20]
................................................................................ (idle for 6h35mn)
ChanServ sets mode: +o gac410 [12:55]
zak256Is there a possibility to make %MAKROS% expand in a verbatim environment, i.e. %USERNAME% in my special case? [12:59]
.... (idle for 15mn)
gac410zak256: no, the purpose of verbatim is to stop macro expansion. If you want similar formatting but with macro expansion you could <pre> ... </pre> tag [13:14]
zak256gac410: Ah, yes of course... forgot about pre. Thx. [13:15]
***ChanServ sets mode: +o Lynnwood [13:24]
.................... (idle for 1h38mn)
LynnwoodGood morning all - I'd working on updating an older site to latest Foswiki installed on ubuntu 16.04..
I copied the old site over to the new server and am in the process of cleaning up the character encoding, so I installed CharsetConverterContrib on the old site.
When I run it, at the beginning of the log, i see a bunch of errors related to the "Unescaped left brace" a la Tasks.Item14324
[15:02]
FoswikiBothttps://foswiki.org/Tasks/Item14324 [ Item14324: Unescaped left brace - Perl 5.25.10 in 2.1.3 tarball. ] [15:04]
LynnwoodTwo questions related to this:
1) The script seems to run ok after the initial error messages so I'm wondering if i should just ignore those errors.
[15:04]
gac410hi Lynnwood ... note that you can also install it on the new site, copy in a web, and then use options to force the cp-1252, and select a web to convert.
depending on the perl version and foswiki config (ASSERTS disabledb), then these are just warnings and won't cause failures.
[15:05]
Lynnwoodoh... interesting. So run the import script first and then run the character conversion on the new site? [15:06]
gac410import script?? [15:06]
Lynnwoodsorry... wrong name.. [15:06]
gac410We added the web and charset option when we converted foswiki.org ... and forgot to convert some of the webs ;) [15:07]
Lynnwoodbulk_copy -
i think of it as import because if changes the store, etc.
[15:07]
gac410Ohhh no. You don't do both! [15:08]
Lynnwoodok
misunderstood your comment.
[15:08]
gac410bulk_copy converts character set and the store, but needs "perfect" data charset_converter can't change the type of store. it only supports rcs, but it's a lot more forgiving.
And a lot more faster too ... bulk_copy has to check out each rev, one at a time, and import it into the new store. charset_converter just fixes up the rcs file in place.
[15:09]
LynnwoodDidn't i see in the docs though that you run charset_converter first to clean up encoding and then use bulk_copy to copy content into new site and change the store? [15:10]
jastif you're going to convert to PlainFileStore, you'll probably want to run bulk_copy and let it do both [15:10]
Lynnwoodok [15:10]
jast(or any other store other than RCS, I guess) [15:10]
LynnwoodIn this case, I know that the character encoding is not clean... [15:11]
gac410but again, bulk_copy needs really perfect data. It will probably just fail if you have any mix of charsets / inconsistent data. [15:11]
Lynnwoodso i really need to use charset_converter first. I guess the next question is whether I really want/need to convert the store or not.
Anybody have an opinion on the value of converting to PlainFileStore?
I have just kind of assumed that it's worth doing...
Lynnwood reads back over PlainFileStore docs
[15:12]
jastPFS advantages are most apparent if you have lots of revisions on individual topics/attachments, particularly for large ones [15:17]
Lynnwoodyes... . it's interesting reading PlainFileStore doc. It explicitly states the inefficiencies of RCS and notes the space inefficiency of PFS (not an issue), but doesn't really comment on whether it's more efficient running then RCS. [15:20]
jastpretty much, yes. it eliminates a lot of overhead [15:21]
Lynnwoodok. so probably it is worth converting. So that goes back to my earlier proposition: I guess I can run charset_converter to clean up the encoding on old content and then use bulk copy to convert the store and bring content into new installation. [15:22]
jastfor instance, rcs uses diffs internally and if you want to access old revisions, you have to go back from the latest revision and apply all the diffs in reverse until you get to the revision you want. for r1 in a topic with 200 revisions, that means applying 199 diffs until you can deliver the result [15:22]
Lynnwoodright.
vs PFS, if you look at version x in the history, you get to simply see that version in it's entirety
(looking at the files themselves)
[15:23]
jastand RcsLite in particular always reads in all the revisions, meaning if you just want to know who created revision 199, everything is parsed (on an attachment history with a few dozen megabytes you'll really notice that)
exactly
[15:24]
LynnwoodLooking back over UpgradeGuide, it really doesn't cover the case of updating an older installation using CharsetConverterContrib and then using bulk_copy to update the store...
it seems to assume that one would stick with RCS.
maybe i'll look at the bulk_copy script to see about the handling of character encoding. I don't see anything off hand about basically skipping that part.
Another side note here: I ran convert_charset on the old content with the -i attribute enabled and, although it did initially throw up some errors related to the Unescaped left brace, it did run.
However, when I then tried to run it again without the -i attribute, it appears to hang while throwing up those same errors.
Specifically, it appears to hang on Meta.pm
[15:33]
So... thiinking through how I might proceed with path to use bulk_copy to copy content from old installation to new and convert the store after having already run convert_charset on old content to clean it up....
Although I haven't found this in the code yet, I believe that bulk_copy looks at the old installations LocalConfig file to determine the character encoding of the installation. So before running bulk_copy, I should edit the LocalConfig and change {Site}{CharSet} to utf-8.
[15:48]
Maybe I'll wait until gac410 is back around to get his feedback before I proceed.
If this is the case, it would seem this is a good detail to add to the docs.
I would not think that I'm the only one who would take this path.
hmmm. bummer. doesn't look like I can get convert_charset to actually run on the old installation.
[15:55]
well. ok. It looks like the reason why it wasn't running was because i was trying to output to a log file. When I ran it without that, then it ask some confirmation questions and appears to be running ok. [16:04]
***laburu has left [16:13]
....... (idle for 31mn)
gac410Lynnwood: bulk_copy actually uses the old site Store API to read the topic revisions. So it reads whatever charset is in the old sites LocalSite.cfg
Because it uses the API, it will NOT copy anything that the API would naturally skip. Pub topic subdirectories, files beginnning with dot or underscore, ..
I'm not sure about how it handles files that are not properly attached / checked in. If you have autoattach files, or inconsistent topic attachment meta, it may get confused
[16:44]
LynnwoodMy main question is what does bulk_copy use to determine the character encoding of the old site files to determine if it needs to do conversion.
Because i've already used convert_charset on the old site to find and fix character encoding issues.
does it look at the CharSet setting in LocalConfig?
ah... i see you just answered this question a moment ago...
[16:47]
gac410It reads the files using the old Store/Meta API / configuration, and saves them using the new store API/Configuration. So I'm not sure it explicitly uses the CharSet or if it depends upon store getting it right.
tbh I've not used bulk_copy. I've stuck with rcs on rcs sites.
[16:49]
LynnwoodIt seems that there would be benefit of switching PlainFileStore in my case.
I guess i'll give it a try. if it goes ok, i'll see about adding it to the doc because i doubt it is a rare use case.
I want to clean-up the character encoding in ways that are beyond bulk_copy, but I also want to update the store.
[16:50]
gac410One other gotcha on plain file store. It uses the file system timestamps for revision information. If you "touch" any of the old revisions, it will break the history. [16:52]
LynnwoodSo, on the old site, I updated the CharSet to utf-8 after running convert_charset. Now I'm going to try running bulk_copy to copy over the content and convert the strore.
that's interesting...
when you say it "breaks the history", you mean that it basically gets the times of the revisions wrong.
or it blows up when you try to view history?
that certainly could be an issue... in either case.
[16:52]
gac410I think it just gets the timestamps wrong. There is a way to fix it by reading the meta data and updating the timestamps, but I've never tried or looked at it. [16:55]
LynnwoodOne thing i like about foswiki is the ease of backing up and moving or restoring content.
So... if one makes a zip (or tar) archive of topics and history, will that mess up the history when restored? Guess i'll have to check on this.
i can remember of that messes with the time stamps
bbib - being called to lunch
[16:55]
....... (idle for 33mn)
So, gac410 - do you question whether it's really worth switching to PlainFileStore? I do see that I avoid several potential pitfalls by NOT switching stores. [17:30]
gac410I just figure not to fix what's not broken. If I had a site with big performance issues I'd probably make the conversion. But if things are running along smoothly, ... [17:31]
Lynnwoodthat makes some sense to me... [17:31]
it is a big issue if I switch the store on a new installation from PlainFileStore to RCS? I guess the only new content that's been added is my user topic, plus WikiUsers.
Lynnwood goes back to read up on this...
[17:36]
gac410You need to switch the store before you register the first user. [17:37]
Lynnwoodi was afraid of that. [17:37]
gac410Or you can just delete all the ,pfv directories [17:37]
Lynnwoodyea. ok [17:37]
gac410The topics themselves are not different, just the history. For something like the initial users, no big deal. [17:37]
Lynnwoodright [17:37]
gac410find . -name "*,pvf" [17:38]
........ (idle for 39mn)
***gac410 has left
ChanServ sets mode: +o gac410
[18:17]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)