From demitar at worldforge.org Sun Feb 17 12:08:18 2008 From: demitar at worldforge.org (Anders Petersson) Date: Sun, 17 Feb 2008 21:08:18 +0100 Subject: [WF-Infra] www.worldforge.org down Message-ID: <1203278898.26900.46.camel@rose> When trying to reach http://www.worldforge.org/ I get a 403 Forbidden access to /. Same without the www. Since it's not a proxy error it doesn't seem to be the usual with zope being down. seul.org and any of the other moria hosted sites I checked seem to be humming along just fine. /Anders From erik.hjortsberg at iteam.se Mon Feb 18 12:14:00 2008 From: erik.hjortsberg at iteam.se (Erik Hjortsberg) Date: Mon, 18 Feb 2008 21:14:00 +0100 Subject: [WF-Infra] www.worldforge.org down In-Reply-To: <1203278898.26900.46.camel@rose> References: <1203278898.26900.46.camel@rose> Message-ID: <93BE8A5F0814E64D8CEDC34CA57B764A012E006499@brun.iteam.local> It's up again, what was the problem? /erik ________________________________________ Fr?n: infra-bounces at mail.worldforge.org [infra-bounces at mail.worldforge.org] för Anders Petersson [demitar at worldforge.org] Skickat: den 17 februari 2008 21:08 Till: infra at worldforge.org ?mne: [WF-Infra] www.worldforge.org down When trying to reach http://www.worldforge.org/ I get a 403 Forbidden access to /. Same without the www. Since it's not a proxy error it doesn't seem to be the usual with zope being down. seul.org and any of the other moria hosted sites I checked seem to be humming along just fine. /Anders _______________________________________________ Infra mailing list Infra at mail.worldforge.org http://mail.worldforge.org/lists/listinfo/infra From balinor at pnxs.de Tue Feb 19 15:00:14 2008 From: balinor at pnxs.de (Philipp) Date: Wed, 20 Feb 2008 00:00:14 +0100 Subject: [WF-Infra] svn and mailing lists down In-Reply-To: <4b05edd10802041010w32e15358v285c894164fd0370@mail.gmail.com> References: <93BE8A5F0814E64D8CEDC34CA57B764A012DFE292D@brun.iteam.local> <4b05edd10802041010w32e15358v285c894164fd0370@mail.gmail.com> Message-ID: <47BB5F7E.7060803@pnxs.de> Hello my dear WorldForgers, Erik Hjortsberg wrote (by PM): | Hi, I?ve been trying to get ahold of you regarding some problems with | the worldforge subversion. [...] Hopefully this address is more valid. Thank you Erik, good job in tracking me down. First of all, everybody, I want to sincerely apologize for dropping off the face of the earth like this. A few months ago, my old balinor at ... account started to be so completely flooded with spam, that I just completely gave up on it, including any messages arriving on the WF MLs. Combined with lots of crazyness in RL (hint: fork(2)) this means I haven't even managed to create a new account and resubscribe. Obviously I didn't think that you guys had no other way to contact me in emergencies. Again, I'm really sorry. Starting from now, please direct mail to me to balinor that?tsign yomu d?t de and CC me personally on anything that's urgent. Eric, about the other mail you sent a few months ago - I just found that, let me get back to you on that. Alistair Riddoch wrote: > On 04/02/2008, Erik Hjortsberg wrote: > >> It seems like a couple of different infra components have collapsed recently. >> >> 1) The svn repository has run out of disc space, resulting in error messages when >> trying to access it. I've tried contacting Balinor through his email to no avail. >> this was related to a _different_ (my) mail server _dropping_ so much spam that the log files alone grew to 850 MB, draining apache's root partition (although the worldforge subversion partition was fine). crazy times... I fixed this after a few days. > I have found Balinor sometimes hard to get hold of, though on at least > one occasion I discovered that he was connected to irc, but not in any > channels. I was, sorry. I'm afraid I'm still not much of an IRC person, but will make an effort to show up more regularly. I'll set my xchat to autoconnect to irc.wf again, so i'm around when that's running, atleast. >> Altogether I'm a bit concerned about how the media svn repository is handled. Where is >> it hosted? Are there regular backups? Who's to contact if something breaks etc.. > > I have no idea, except that as far as I am aware no one is involved in > the hosting except Balinor. > Yes, I am the only admin of that box, with two trusted friends roughly knowing their way around and having physical access to reduce the bus factor. There are regular incremental on-machine backups (different filesystem, mirrored to two discs) and irregular off-site backups done manually by me. Back when I set up the box, bear and AFAIK anubis (?) requested a "wfbackup" account to regularly rsync the backup dir, but so much has changed since then in the setup that this probably is not working anymore. >> On a good note, I've done some nice development recently with a new imposter >> system. I've written about it on my development blog and added some screen shots to >> amber. >> let me just take a second here to say: once again it's just great to come back here and see you guys rocking so hard. Finally, Jack Cummings wrote: > This is a good reason to switch to a DVCS... well sync'ed repositories are > idempotent, so backup becomes moot. > I'm open to that idea, altough with a repos the size of wf-media, this might start sucking pretty quickly (diskspace- and performance-wise). Fell free to start playing with different DVCSes against my server, and let's discuss the results. However, before really implementing something like this, I'd really like to get a clean migration of coders@ away from CVS underway. Good night all, keep forging, Balinor From erik.hjortsberg at iteam.se Thu Feb 21 07:35:06 2008 From: erik.hjortsberg at iteam.se (Erik Hjortsberg) Date: Thu, 21 Feb 2008 16:35:06 +0100 Subject: [WF-Infra] svn and mailing lists down In-Reply-To: <47BB5F7E.7060803@pnxs.de> References: <93BE8A5F0814E64D8CEDC34CA57B764A012DFE292D@brun.iteam.local> <4b05edd10802041010w32e15358v285c894164fd0370@mail.gmail.com> <47BB5F7E.7060803@pnxs.de> Message-ID: <93BE8A5F0814E64D8CEDC34CA57B764A012FBEEE06@brun.iteam.local> Hi, glad to hear from you! Sorry to hear about your problems. In general I think we need better information on how the infrastructure is arranged. Things like what servers there are, who's responsible for them, how to contact the person responsible, what's hosted on them, how they are backed up, how they are hosted and so on. I've added information for amber over at http://wiki.worldforge.org/wiki/Coordinators#Hosts, I think it would be great if the people responsible for the other servers would do the same. I would feel much better knowing that crucial parts of the infrastructure such as the code and media repository are properly backed up, and that there's a clear way of getting access to it should anything happen to the person responsible for hosting it. Jayr have been restructuring the svn repository so that all original assets are stored in /art, while end user assets are stored in 3d_objects and 3d_skeletons. The idea behind this is that developers that want to get the media needed for client development only should need to get the two latter directories, while artists interested in working with the media can get the art directory too. One advantage of this is that we can have heavier art files in the repository without burdening client developers with these files. The size of media source files these days can be quite large. For example, we have some MudBox source files which can be > 100Mb. So before we commit such large files to the repository I wanted to check with you how it would handle it. Is there enough space? Enough bandwidth? Will subversion store updates as delta or as complete copies? Regarding the switch from cvs to svn for the code repository: that would be very much welcome. What happened with the previous effort? Development of Ember continues and I regularly update the screenshot section over at http://amber.worldforge.org/ember/screenshots/ I've just implemented a new foliage and undervegetation system which makes the world look a little more alive. I have to fiddle a little bit with the colouring though. Sincerely Erik -----Original Message----- From: Philipp [mailto:balinor at pnxs.de] Sent: den 20 februari 2008 00:00 To: Alistair Riddoch; Erik Hjortsberg Cc: Kai Blin; Simon Goodall; jack at mudshark.org; Worldforge Infra team Subject: Re: svn and mailing lists down Hello my dear WorldForgers, Erik Hjortsberg wrote (by PM): | Hi, I've been trying to get ahold of you regarding some problems with | the worldforge subversion. [...] Hopefully this address is more valid. Thank you Erik, good job in tracking me down. First of all, everybody, I want to sincerely apologize for dropping off the face of the earth like this. A few months ago, my old balinor at ... account started to be so completely flooded with spam, that I just completely gave up on it, including any messages arriving on the WF MLs. Combined with lots of crazyness in RL (hint: fork(2)) this means I haven't even managed to create a new account and resubscribe. Obviously I didn't think that you guys had no other way to contact me in emergencies. Again, I'm really sorry. Starting from now, please direct mail to me to balinor that?tsign yomu d?t de and CC me personally on anything that's urgent. Eric, about the other mail you sent a few months ago - I just found that, let me get back to you on that. Alistair Riddoch wrote: > On 04/02/2008, Erik Hjortsberg wrote: > >> It seems like a couple of different infra components have collapsed recently. >> >> 1) The svn repository has run out of disc space, resulting in error messages when >> trying to access it. I've tried contacting Balinor through his email to no avail. >> this was related to a _different_ (my) mail server _dropping_ so much spam that the log files alone grew to 850 MB, draining apache's root partition (although the worldforge subversion partition was fine). crazy times... I fixed this after a few days. > I have found Balinor sometimes hard to get hold of, though on at least > one occasion I discovered that he was connected to irc, but not in any > channels. I was, sorry. I'm afraid I'm still not much of an IRC person, but will make an effort to show up more regularly. I'll set my xchat to autoconnect to irc.wf again, so i'm around when that's running, atleast. >> Altogether I'm a bit concerned about how the media svn repository is handled. Where is >> it hosted? Are there regular backups? Who's to contact if something breaks etc.. > > I have no idea, except that as far as I am aware no one is involved in > the hosting except Balinor. > Yes, I am the only admin of that box, with two trusted friends roughly knowing their way around and having physical access to reduce the bus factor. There are regular incremental on-machine backups (different filesystem, mirrored to two discs) and irregular off-site backups done manually by me. Back when I set up the box, bear and AFAIK anubis (?) requested a "wfbackup" account to regularly rsync the backup dir, but so much has changed since then in the setup that this probably is not working anymore. >> On a good note, I've done some nice development recently with a new imposter >> system. I've written about it on my development blog and added some screen shots to >> amber. >> let me just take a second here to say: once again it's just great to come back here and see you guys rocking so hard. Finally, Jack Cummings wrote: > This is a good reason to switch to a DVCS... well sync'ed repositories are > idempotent, so backup becomes moot. > I'm open to that idea, altough with a repos the size of wf-media, this might start sucking pretty quickly (diskspace- and performance-wise). Fell free to start playing with different DVCSes against my server, and let's discuss the results. However, before really implementing something like this, I'd really like to get a clean migration of coders@ away from CVS underway. Good night all, keep forging, Balinor From erik.hjortsberg at iteam.se Tue Feb 26 02:19:03 2008 From: erik.hjortsberg at iteam.se (Erik Hjortsberg) Date: Tue, 26 Feb 2008 11:19:03 +0100 Subject: [WF-Infra] infrastructure information Message-ID: <93BE8A5F0814E64D8CEDC34CA57B764A012FBEEEF0@brun.iteam.local> Hi, Worldforge has quite a large and spread out infrastructure with a many different servers on different locations. Some of these servers are critical for the project, such as the code and media repository, while other aren't quite as critical. But all of them needed in one way or another. We recently had some problems with the media svn server, which made me worry a bit about what backups are in place, where the servers are hosted, who to contact and so on. I've therefore added some extra information to the wiki for the server that I host (amber). I've added information on what services are hosted at that specific server, who's hosting it, what the specs of the machine are and whom to contact about it. Could those that are responsible for the other servers also add similar information? The page is over at http://wiki.worldforge.org/wiki/Coordinators Sincerely Erik Hjortsberg -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.worldforge.org/pipermail/infra/attachments/20080226/1f4d7701/attachment-0001.html From demitar at worldforge.org Tue Feb 26 10:25:53 2008 From: demitar at worldforge.org (Anders Petersson) Date: Tue, 26 Feb 2008 19:25:53 +0100 Subject: [WF-Infra] www.worldforge.org down In-Reply-To: <93BE8A5F0814E64D8CEDC34CA57B764A012E006499@brun.iteam.local> References: <1203278898.26900.46.camel@rose> <93BE8A5F0814E64D8CEDC34CA57B764A012E006499@brun.iteam.local> Message-ID: <1204050353.14506.39.camel@rose> m?n 2008-02-18 klockan 21:14 +0100 skrev Erik Hjortsberg: > It's up again, what was the problem? (Took a while to reply since I apparently didn't filter email correctly. *slaps forehead*) It seems the seul people upgraded the apache config and accidentally broke the proxy config (zope was humming along merrily). /Anders