#foswiki 2014-08-03,Sun

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)

WhoWhatWhen
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/x5V1eg
foswiki/master d3f1e38 GeorgeClark: Item11267: Log github messages by repo....
[03:01]
***GithubBot has left [03:01]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/ZtLT3A
foswiki/master 854053c GeorgeClark: Item11267: Typo in regex...
[03:16]
***GithubBot has left [03:16]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/LwiY_w
foswiki/master ef776a0 GeorgeClark: Item11267: Untaint the log filename...
[03:31]
***GithubBot has left [03:31]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/755JfA
foswiki/master 4989765 GeorgeClark: Item11267: Debug logging process...
[03:44]
***GithubBot has left [03:44]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/p-5JaA
foswiki/master 8a1ecc8 GeorgeClark: Item11267: validate name went too far...
[03:59]
***GithubBot has left [03:59]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/YEFxUQ
foswiki/master d9ea56f GeorgeClark: Item11267: No need to untaint twice...
[04:13]
***GithubBot has left [04:13]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/CfBQYw
foswiki/master 6d4cd9f GeorgeClark: Item11267: undefined string in concat...
[04:28]
***GithubBot has left [04:28]
.... (idle for 15mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/i5Vs3A
foswiki/master 661c2bc GeorgeClark: Item11267: wrong format time call...
[04:43]
***GithubBot has left [04:43]
gac410 has left [04:49]
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/qnU3wA
foswiki/master d9f84cf GeorgeClark: Item11267: Need to extract commit id...
[04:58]
***GithubBot has left [04:58]
............................................................... (idle for 5h13mn)
ChanServ sets mode: +o MichaelDaum [10:11]
MichaelDaumMichaelDaum working on gittifying my working env [10:19]
.... (idle for 18mn)
why is _allDeveloper "not particularly useful" according to http://foswiki.org/Development/GitRepository#Repositories
how is _allFoswiki and _allDeveloper linked?
how do I proceed when I start with _allFoswiki and then have a dozen more repos checked out as well with regards to submodules?
those listed in _allFoswiki don't cover them; _allDeveloper covers too many of them.
MichaelDaum will ask the same q's tomorrow on the release meeting
[10:37]
..... (idle for 20mn)
argh one task too much: new configure, move towards git, what broke what ... [11:01]
................... (idle for 1h31mn)
***PsychoTrahe has quit IRC (Ping timeout: 250 seconds) [12:32]
ChanServ sets mode: +o gac410 [12:41]
gac410Hi MichaelDaum - I saw your questions.
I can try to give you stuff to ponder now if you want.
[12:41]
MichaelDaumhey George
yes please
[12:43]
gac410I'm still "feeling my way through" this stuff. ... so much of what I wrote is clearly subject to revision. [12:44]
MichaelDaumexcellent so far
what I am trying to figure out is in how far submodules help _me_ in _my_ development process
[12:44]
gac410okay. _allDeveloper. It was pharvey's first attempt to make something looking like our old trunk. Check out the "universe" in one go, but using submodules. I don't think anyone really wants *everything* except maybe for exploring [12:45]
MichaelDaumwhich covers +100 plugins more than what is in _allFoswiki and probably the same amount less than what is in _allDeveloper
so I am somewhat in the middle
[12:45]
gac410The big advantage of submodules is that you can link to a module at a "point in time" so _allDeveloper really isn't like trunk 1.8.2 added the ability to also point to a specific branch. [12:47]
MichaelDaumI had to add-apt-repository ppa:git-core/ppa as git in ubuntu 12.04.4 is 1.7.x
which gives me git 2.0.4
but that asside ...
I clearly see the advantages of the submodule feature
[12:48]
gac410So advantage of _allFoswiki is that it links the "release01x01 branch extensions" into a release01x01 collection, before 1.8.2, a submodule pointed only to a specific commit, not a branch.
But we only have designated branches on the default stuff. If you wanted to "version" Nat ecosystem. You could easily add branches to your extensions as well.
[12:49]
MichaelDaumit isn't NatSkin only [12:50]
gac410It's so easy and lightweight to branch under git, we really have an opportunity to be much more accommodating
If you did NOT have git 1.8.2, you could still use git, just instead of the submodule, you would clone each extension (pseudo-install does that automagically) and then manually visit each extension and "checkout branch Release01x01" or master or myExperiment
[12:51]
MichaelDaumI have 2.0.4 now
no problem with that
[12:53]
gac410okay. well you are way ahead of me then. But I'm sure others will ask :) [12:53]
MichaelDaumwhat I am trying to figure out is how to deal with the super project.
_allFoswiki does not cover my stuff --- _allDeveloper covers too much resulting in git submodule foreach iterating over to many repos in a daily work
[12:53]
gac410Exactly. It's probably something you'll need to experiment with. I'm only coming up to speed. [12:54]
MichaelDaum:) [12:54]
gac410You could create your own repo. Under foswiki, or under your own account. Look at _allFoswiki/.gitmodules That one file is all that you initially have to create. [12:55]
MichaelDaumok [12:56]
gac410It could be a mixture. point to default ext's by branch, and let your extensions track head of master. [12:56]
MichaelDaumwhich brings me to the question: how are _allFoswiki and _allDeveloper linked to each other (or merely their pointers to changes in submodules)? [12:56]
gac410After you populate .gitmodules, you do a git submodule --init to checkout all the submodules. [12:57]
MichaelDaumI probably will do that
and others too to cover _their_ working set
[12:57]
gac410No linkage at all pharvey created _allDeveloper, and populated .gitmodules. I did _allFoswiki and used the new branch feature. [12:58]
MichaelDaum... the set of repos maintained by a foswiki dev [12:58]
gac410there is also a _allcoredefault or something like that. which was pharveys _allFoswiki without the branches. It was lost in the noise and I accidentally duplicated it. [12:58]
MichaelDaumthere was a recent update to _allFoswiki wrt EditTablePlugin or so: Item000: Update refs to latest commits, Aug 2
how are refs updated in _allDeveloper to be in sync with this one?
[12:59]
gac410They are not. [13:00]
MichaelDaumso checking out _allDevelioer/master won't give you the same as _allFoswiki/master [13:00]
gac410for one thing allDeveloper doesn't track branches. so correct. _allDeveloper is much closer to checkout of svn trunk. No branches available. [13:01]
MichaelDaumlikewise a developer's own _myFoswikiWorkingSet repo. [13:01]
gac410But all the submodules are still stuck at their last point in time. So still need to do git submodule update --remote to move them forward. [13:02]
MichaelDaumwhat's the difference to git submodule foreach --recursive "git pull" ? [13:02]
gac410Okay. --recursive goes into the submodule and looks deeper for more submodules. So if our TinyMCEPlugin had a submodule pointing to the tinymce project, that would be possibly updated as well.
For our current use, I don't think we have anything that would need the --recursive
[13:03]
MichaelDaumwould you say that (1) git submodule update --remote and (2) git submodule foreach "git pull" do the same? [13:04]
gac410git submodule foreach "git pull" would be I suspect similar to the update --remote There may be subtle differences in how the internal module structure is updated. [13:05]
MichaelDaumMichaelDaum just wants to understand git submodules better [13:05]
gac410Ie. if you cd into a submodule and do stuff (like pull) that doesn't update the sha pointer stored in the umbrella project. git submodule update does do that. [13:05]
MichaelDaumah ok. that's an important difference [13:06]
gac410You can think of the "foreach" as just automating a "cd " into each submodule. [13:06]
MichaelDaumright [13:06]
gac410You can see the effects by looking at the "git status" at the superproject. It just lists the submodules and summerizes their state compared to the current stored hash. [13:07]
MichaelDaumjust want to make sure that (1) forwards the checkout areas like (2) does [13:07]
gac410I'm reasonably certain they do. [13:08]
MichaelDaumthis all to make sure developers work on the same code [13:08]
gac410Note also pseudo-install automatically does the git clone for a extension repository. But it's not covered by the "foreach" processing since what pseudo-install adds isn't a submodule. [13:09]
MichaelDaumyes, have seen that (untracked dirs) [13:09]
gac410Again look at git status in supermodule. pseudo-install added stuff that we don't track at the Release level. BuildContrib for ex. [13:09]
MichaelDaumyes yes got that
so my take-away is that I best create my own super-project and add submodules as required.
[13:09]
gac410Actually for now UnitTestContrib as well. Unless we branch that, it will look like any non-release-managed extension. [13:10]
MichaelDaumis it actually required to git push my superproject or is it sufficient to just use it locally to track submodules? [13:11]
gac410I think that's true. :) I think we'll have a lot to learn for sure. Just remember, creating branches is trivially easy, and cheap.
No, you never have to push.
Heck I've got my /etc directory as a local git repo. So after system updates I can review all that changed.
[13:11]
MichaelDaumonly reason for me to check out _allFoswiki is to update hashes in there for those repos that overlap with my own superproject
MichaelDaum uses etckeeper + git to track changes in /etc ... i.e. when supporting a client's machine
[13:12]
gac410right I was thinking that maybe that should be an automatic update on foswiki.org maybe once/day [13:13]
MichaelDaumoh that would be good [13:13]
gac410and for foswik.org trunk. every 15 minutes we check out _allFoswiki master branch, git submodule update --remote and if any mods have commits pending, do a commit -a / push [13:14]
MichaelDaum... to re-attach submods again [13:15]
gac410right. well not re-attach, just move the sha forward to latest commit in the branch.
BTW You can rollback any repo to an older version by just "git checkout <sha1hash>" You can locally undo CDot's mega configure checkin It helped mtempest yesterday
[13:16]
MichaelDaumthanks. I'll go and create my own super project now...
I've got the new ConfigureContrib running now ... in my old svn checkout
[13:17]
gac410cd core git log find the point in time that's good c/p hash git checkout <hash>
Ohh... it's mo difficult there :P
Be warned. It can't bootstrap currently. LSC must exist.
[13:17]
MichaelDaumI'd rather prefer helping to move forward instead of riding a dead horse. [13:18]
gac410yeah. I agree. But when trying to debug git configuration as you said too many things changing at onece. and really easy to put changes aside. [13:19]
MichaelDaumthere's an undefined $Foswiki{DefaultUrlHost} after logging in to configure. reloading the same page makes it go away ... and stuff like that ... the errors you already mentioned on one of the related bug items ... [13:20]
gac410yeah. I was trying to document for cdot, didn't want to dig in too deeply just yet. too many cooks :) [13:21]
MichaelDaummy main goal right now is to (1) set up my own super project covering all extensions that I maintain (2) merge in pending changes from svn (3) get virtualhosting running pointing to the git checkout area again
I've got quite some changes to FastCGIEngineContrib
[13:22]
gac410Once we are fully under the git model major changes like configure would be done in a separate branch. devs can commit without polluting master. It would make life easier for all of us. [13:22]
MichaelDaumit wasn't stable enough under a ProcManager regime [13:23]
gac410ProcManager ? [13:23]
MichaelDaumthe life time of a FastCGI process is either controlled by the http server itself - as apache and lighttpd does - or
as a backend service started by the system init process
and then contacted by nginx as a proxy
[13:23]
gac410Ah. I've not experimented much with ngnix [13:24]
MichaelDaumthe latter requires FCGI::ProcManager to take over maintenance of the lifetime of forked child processes
... and this wasn't stable enough for production yet.
over time the system blocked itself
this happens when a child process was still running (still having open the fcgi socket) ... while the parent process, the proc manager, was reExec-ing itself due to an external even such as MaxRequests exceeded or LocalSite.cfg being changed.
what happens then is that the running child process fails to complete the request and hangs forever; only a kill -9 could finish it off again; due to it not being able to talk to the upstream http server anymore
reExec fails too as the fcgi socket was already in use
[13:25]
gac410Sound like you had your work cut out for you. [13:28]
MichaelDaumthis was a pita to figure out :(
these errors only occurred sporadically under heavy load ... on a production system.
[13:29]
gac410Those are really the difficult ones.
btw. be sure to add your thoughts to the agenda for monday so we don't overlook them. I'm sure we will have a lot to discuss.
[13:29]
MichaelDaumalright [13:31]
gac410gradually fighing my way throug the f.o githook rest handler. Reminds me of old style mainframe debugging. Waiting for 15 minute "turnaround" for each checkin to the rest handler to be applied. Not as bad as the 24 hour turnaround in the old days.
Lots of noisy commits. You get to see all my stupid mistakes in gory detail Drives our commit stats up though :D
[13:32]
MichaelDaumone line checkins, heh? [13:34]
gac410If I was smart I'd write some unit tests. Yes indeed one painful line at a time :D [13:34]
MichaelDaumhappy you work on this. very valuable to the project. [13:35]
gac410thanks [13:35]
MichaelDaumwhat's the difference of git://github.com/foswiki/foswiki.git and git@github.com/foswiki/foswiki.git ? [13:40]
gac410hm. you have to stump me on the simple ones. the git@github.com is an ssh connection. I think that's the recommended / fastest connection.
git:// is the git protocol over it's own port. ... i think. Don't quote me on it though.
[13:41]
MichaelDaumoh I thought git:// was fastest
git:// > http:// > ssh:// ???
[13:41]
gac410hm. seems like it should be as well. [13:42]
Tarbox1IIRC git:// does not do authentication as well as ssh:// [13:42]
gac410https://help.github.com/articles/which-remote-url-should-i-use [13:43]
MichaelDaumokay git@ then [13:44]
gac410Also it doesn't look like github offers git:// [13:44]
MichaelDaumoic [13:44]
Tarbox1irony [13:44]
gac410On the right side of each repo home page they have a little clicker to cut/paste the desired url. Only offer the 3. ssh, https, and subversion
Eventually we need to refactor GitRepository, HowToUseGit, GitAndGithub, GitAndPseudoinstall, GitCookbook, not to mention all the Subversion related documents in Development web
We are as usual moving into the realm of much too much information spread around, inconsistent and confusing.
[13:45]
MichaelDaumnormal for wikis [13:50]
gac410yes indeed. But still, .. we need a volunteer :)
gac410 is looking at rewriting the install trunk code now. That needs to be ready.
[13:50]
gac410 wonders. Would it make better sense to do "git clean -fdx" remove everything not tracked. And then have a script recreate all the data/pub symlinks into our live foswiki.org [13:57]
MichaelDaumumpf
that clearly goes over my head ...
[13:57]
gac410Cleaning on subversion was difficult. (removing all old cruft from a checkout). git clean is just magic. everything goes away. -f force -d directories -x even the files in .gitignore
For my checkouts, I've been using LocalSite.cfg (in .gitignore !) settings Sandbox => Litterbox Main => Usersweb so I can clean, and not discard any of my local work.
ah. make sure .htpasswd is in .gitignore too :D
[13:59]
MichaelDaumimagine configure would be able to deal with a pseudo-install env
these are loose ends atm
[14:11]
gac410It sort of does. It detects pseudo-installed extensions by detecting if they are a symlink
we also discovered newest version disables the install/remove button if pseudo-install detected. So you can't have a mixture. :P
[14:12]
MichaelDaumyet it does no git pull + cleanup + pseudo-install again
does it?
[14:13]
gac410Ah. configure does not run pseudo-install. That might be difficult since cleanup would remove configure in a normal pi environment.
somewhat a catch-22 especially now that configure is an extension.
[14:13]
MichaelDaumyea [14:15]
gac410Even without being an extension, it needs to remove and rebuild all the .gz / compressed stuff. [14:15]
MichaelDaumsome of the stuff in the auto-generated .gitignores is wrong
that is better not ignored
e.g. locales of extensions
[14:28]
gac410yeah That part needs some work. The .gitignore files also grow and grow. Not very smart adding to them. [14:29]
MichaelDaumthere's one big .gitignore in core ... covering extension stuff as well being ignored there locally as well [14:30]
gac410I'm really hoping pharvey frees up some time. pseudo-install & git integration is his masterpiece
I was talking to paul yesterday. His new adsl2 is only giving him 150 kb or something like that . So he is really hobbled online.
He ran off to buy a new modem - don't know how he made out
[14:30]
MichaelDaumthe svn:propset svn:ignore make up good .gitignores [14:42]
gac410Need to think carefully about runtime files. git clean will remove the logs & working files.
also I need to move the FoswikiOrgPlugin onto f.o from t.f.o but it needs the new rest security model.
You mean as a "seed" for new .gitignore files? Unfortunately git has no propset I don't think I've never touched the svn propset stuff except when I really had to back when building a release on subversion.
[14:43]
MichaelDaumsvn:ignore = .gitignore
however svn:ignore is per directory
whereas .gitignore is better located in the repo root
anyway. not too hard to create new .gitignores from scratch based on git status -s |awk '{print $2}'
[14:44]
gac410One thing though, you don't want everything. You probably want git clean to clean up files generated by pseudo install. .gz, etc. so they get rebuild
Especially when you flip between branches. I very regularly switch between master & release. So stale compressed files are a real problem if I forget to clean.
[14:47]
MichaelDaumfatal: clean.requireForce defaults to true and neither -i, -n, nor -f given; refusing to clean [14:48]
gac410Yeah. You need to use -f to force the clean. Much safer that way Avoids accidental cleanings
-n dry run -i interactive -f force
You can change the defaults with git config I never remeber the syntax though
I use a shellscript to git clean just selected directories ... so all my webs don't disappear. I probably ought to make them symlinks That would be cleaner.
[14:49]
MichaelDaumor use virtual hosting contrib which clearly separates net data from core
makes sense even for a single vhost
[14:53]
gac410I try to stay as true to a basic install as possible. VHC pokes around in the internals too much for me. old dog .. new trick ... Though I suppose with that rationale I should be campaigning for rcs or cvs, never mind the new fangled svn or git. [14:55]
MichaelDaumin real life VHC is almost invisible when debugging the core [14:56]
gac410Thats nice to know. when the work was first started on vhc, i was answering questions iirc, and some of the hoop jumping scared me. So tbh I never really explored it. Plenty of other stuff to work on. [14:57]
MichaelDaumyea indeed.
same for me on PlainFileStore
this is on my todo for a long time. yet I push it backwards for too long already.
[14:57]
gac410yeah. I have not touched that either. We really ought to consider it for foswiki.org. It would **greatly** improve the extension upload process if nothing else. [14:58]
MichaelDaumonce we all feel cozy on a PFS there's no real reason to stay on rcs afaics [14:59]
gac410agreed. It probably ought to be the default store [14:59]
MichaelDaumin 2.0 ... with a store migration well tested [14:59]
gac410Avoids all the rcsLite / RcsWrap headaches [15:00]
MichaelDaumoh yea
though it might even be too fast
[15:00]
gac410too fast? And that's a problem? [15:01]
MichaelDaumsome design problems are in the layers above, that only show up when the store engine underneath is slow enuf [15:01]
gac410yeah. longer running, better chance for race / deadlocks if nothing else I suppose. [15:01]
MichaelDaumthere are areas in %QUERY that forces the store to iterate over all revs down to rev1 [15:01]
gac410Well as long as our unit tests are thorough. We probably ought to add store verification in tests sensitive to the store. QUERY, SEARCH, ... [15:02]
MichaelDaumunit tests isn't the problem. we miss benchmarks. [15:03]
gac410well it looks like the trunk install code does re-link everything that points to foswiki.org so at least for symlinks, a gitclean is safe.
I suppose what I ought to do is create a temporary git install for trunk on f.o and test all this stuff - :D
no guts no glory ... vs Old or Bold, very few are Old & Bold
And ... since we don't have a "svn rev" How should we report current revision when viewing trunk.f.o/Main/WebHome Use timestamp of the last successful update run?
[15:04]
MichaelDaumthe git hash?
no beauty but fits bragging well enuf
[15:12]
gac410Hm. Each submodule has it's own hash. So if SomeExtension moves forward, do we want it's has, or just the last hash update for the "core" respository [15:13]
MichaelDaumthe superproject's hash reflects that well enuf doesnt it
even better than our old svn rev
[15:13]
gac410Reason for a date. If for some reason the cron job fails, a human can tell that it has stalled. hash is obscure. Maybe we need date + superproject hash (or abbreviated hash)
Generally git powers recommend a 12-hex-digit abbreviated hash. 7 is the default. Github will successfully show a 12-digit hash. So I fighured that is what I would use for the Foswikiref: tracking in the Tasks web.
I suppose maybe sometime in future InstalledPlugins could grab the log of each extension and insert a Foswikiref:234a345b345c pointer to the last commit for each extension. But that's a pipe dream for now :D
[15:14]
MichaelDaumas long as we have visions ... [15:20]
gac410I need to go get some stuff done around the house. back in a few hours. if not later, see you tomorrow morning. [15:32]
one little hint .gitignore can be checked in, and distributed to others. .git/info/exclude same format. But is not checked in, so it can be a local ignore list that survives clean, updates, etc.
So there will be stuff we want to ignore on trunk.foswiki.org that we probably don't need to distribute. working directory for one. most test environments are probably okay with cleaning working, but definitely not trunk.foswiki.org
gac410 wonders if rather than fixing pseudo-install, we should just create and commit the recommended .gitignore for each extension. Certainly pseudo-install is broken with the growing-forever ignore files.
Though it would fall on developers. When adding an auto-built file to the MANIFEST, to also add it to .gitignore.
[15:42]
MichaelDaumyes that's what I just started doing: create .gitignores manually per extension [15:48]
gac410pharvey's autobuild was good when we were mainly svn. but on a real git repo, might as well just check them in.
pseudo-install with an empty .gitignore though will create a good starting point for review.
[15:49]
Anyway. good docs here: http://git-scm.com/docs/gitignore Too many options :)
now I really need to do stuff away from my laptop.
[15:59]
................................................................... (idle for 5h32mn)
GithubBot[foswiki] FoswikiBot pushed 1 new commit to master: http://git.io/qTU7Vw
foswiki/master 3384b81 GeorgeClark: Item12980: Fix shebang in git hooks....
[21:32]
***GithubBot has left [21:32]
................... (idle for 1h34mn)
ChanServ sets mode: +o pharvey [23:06]
........... (idle for 53mn)
ChanServ sets mode: +o pharvey
ChanServ sets mode: +o pharvey
[23:59]

↑back Search ←Prev date Next date→ Show only urls(Click on time to select a line by its url)