Taking Forums Down Sometime 10/11-10/12

[arclight]

The short story: I have 5GB of new memory to add to the three servers that share the load for austinimprov.com. I have to take the machines down briefly (<1hr each) to add the memory so the forums and such will be down. I don't know when I'll get the time to do it, but the plan is Saturday or Sunday, preferably when I'm awake, well-fed, alert, and calm.

Hopefully the added memory will speed up the forums (probably not a lot.) I'm also adding a second drive to the webserver so I can do better backups and take some load off the existing disk.

Gory details: The big reason for the upgrade is to allow me to 'virtualize' the mail and web servers, which will make them a lot more portable for when I move to Chicago. The physical hardware will be coming with me, but my plan is to migrate the forums and web sites to a 'virtual server' which will run on a server hosted locally in Austin. Or wherever. Once the existing system has been virtualized, it can be run damn near anywhere. Don't worry if you don't understand what that means; the only thing that's important is that you should not panic if the forums go down this weekend. I'm working on them and if I do things properly, you'll never notice.

Let me know if you have any questions...

[KathyRose]

I wonder ... do you ever prune the tree?
There seems to be a lot of really old threads, most of which are of little use or consequence. Why not clean house, rather than add memory?

[Pdyx]

As a relatively newcomer to this particular forum, I can't say, but as a long time BBS/forum user. Posterity! You can't delete old threads! That sets a bad precedent, plus you never know when someone will resurrect them with an update or related story.

[arclight]

That's an interesting question which deserves a three part answer.

The forum database only consumes about 80MB, so for as many posts as there are the forum doesn't eat a lot of resources. As I've found with the Amazon Kindle, text is small (I have over 200 books on mine now and they only take up 60MB, just as a point of reference.)

The second part of the answer gets a bit technical but I'll try to keep it lucid. Back in the days of yore, memory (RAM) was way more expensive than disk, really expensive in general, and tended to limit the size of programs people could run. Some bright guy figured out that you could copy the contents of memory to disk and swap stuff back into memory as it was needed, thus making the computer seem like it had more memory than it actually did. Thus "virtual memory" was born.

There are some downsides, primarily that it takes time to swap data from RAM to disk and back again. This leads to some interesting discussions about trading time for space, a key concept in optimizing systems. There's also a limit to how much virtual memory can be used, and people with really big brains spend a lot of time figuring out the best way to tell which data should be swapped out to disk when - stuff that's not referenced very often (LRU or "least-recently used"), etc. It's a pretty fascinating subject if you're into that applied mathy stuff.

Fast-forward to today. Disk is disgustingly cheap, RAM is almost as disgustingly cheap. Processors are even more blindingly fast. The time lag in swapping data from disk to RAM is really noticeable, so the goal is as it has always been - stuff as much RAM in the box as you can afford. The goal is to have enough RAM so that you rarely use the virtual memory (disk.) For example, running backups at 4am. It doesn't matter if the system is a little slow since most sane people are asleep at that hour.

To get back to reality, the server running the forums has 1GB of RAM and is using 200MB of swap. I've had another 1GB of memory sitting on my desk for a few months because I haven't felt like taking the machine down to install it, notifying people so they don't freak out, etc. I checked the graphs and it looks like I've rebooted the machine three times this year, the last time was 11 weeks ago (the previous uptime record for the machine was ~250 days; I've run some servers for over two years straight, which is pretty good considering they live in my back bedroom and not an actual datacenter.)

In summary, the forum database is small enough that it doesn't desperately need pruning, and the system is slow overall is due to the use of virtual memory (aka swap space.)

But all that leads to the most important question: How do people feel about pruning old posts from the forums?

It's just the flip of a switch in the software and it'd take less than five minutes to do, but I'm wary of doing it because I haven't talked to anyone about it. There are some things I'll do out of sysadmin fiat, for example, taking the system down to put in memory, adding anti-spam features, deleting abusive-looking accounts, etc., but messing with forum content is something I don't do lightly. I've considered culling old posts (older than 12-18 months) in the time-sensitive parts of the forums - Casting, Corkboard, Events, etc., but only because I don't think anyone would miss three year old casting calls. Still, I need to know what the consensus is before I start making changes.

I should post a poll in the General Discussion topic; I don't know how many people watch the Tech Talk topic and I want some feedback and proposals before I do anything I'll regret.

Thoughts?

[Roy Janik]

But all that leads to the most important question: How do people feel about pruning old posts from the forums?

I'd definitely be strongly opposed to removing old content. Even three year old casting calls remind of 1) what shows were around, and 2) what people were around. Since it's not a drain on the system, leave it up! It's the closest thing to a written history the Austin improv community has.

[Aden]

I second what Roy said. for sure.

[KathyRose]

arclight - thank you for the technical explanation. I used to design data systems and software applications for IBM (internal use), so I appreciate the information.

Roy ... Not every word ever written has value. "Posterity" is not a good reason to keep ALL of the forum data. Have you ever known someone who can't throw anything away? Have you seen their house? Giving them a bigger house does not improve the quality of his/her life.

Improv is an ephemeral art. It would be the height of irony (and hubris) to think that every word ever typed about topics relating to (and many not relating to) improv should be preserved for posterity. And having the capacity and ability to do something, doesn't mean you should.

When I first discovered the forum, I spent hours looking through old posts to see what I could learn about improvisation and the Austin improv community; but after a while, I gave it up. Too much chafe. Not enough wheat.

Many of the posts and threads (Caa-Kah!) are blather which was fun at the time, but has no enduring value - historical or conceptual. Forums are just a way of carrying on conversations and sharing information. For the long term - keep the information. Let the idle chat go the way of spoken conversations.

Please understand - I'd classify the "best of" discussions, for example, as information, not idle chat. In general, it wouldn't take the wisdom of Solomon to know which posts could be discarded without detriment to the improv community. It's just an exercise in responsible data management.

I'm not advocating automatic disposal of data - even the time-sensitive data like Events. There IS historical value in keeping a record of shows that were performed, so someone could ask "Just when did pgraph first do it's Villany show ...?" You could even keep the posts that list the cast for each Maestro, but you wouldn't need all of the individual posts that led up to the casting decision.

I would entrust someone with deep AIC roots, who would know what might be of both historical and conceptual value, with the administrative authority to do an annual cleanup. The result would be a valuable repository of useful (and entertaining) information; not a bloated word-by-word "historical record" of every post ever made to the forum.

It's just data - not a religious relic!

[Jessica]

It seems that since we have the space we might as well keep it all up. The time it would take someone to sort out old and useless from old and still important would be a lot. Also, as a historian (or at least a history major) I know that we often throw out the stuff that would have been really valuable. For example we know the names of almost all the Egyptian kings but not how they got the pyramids build. Back then they probably thought it was obvious and silly to mention, but now it is at least an interesting mystery, and at most may be an opening to a different engineering paradigm. So, if it isn't a problem, I say keep it and let others decide what is important to them and what isn't.

[Roy Janik]

When I first discovered the forum, I spent hours looking through old posts to see what I could learn about improvisation and the Austin improv community; but after a while, I gave it up. Too much chafe. Not enough wheat.

That's easily solved by creating a series of posts with links to particularly useful or favorite posts, and then adding those as "sticky" posts to the tops of forums. Often, these evolve into FAQs and whatnot.

[arclight]

That's easily solved by creating a series of posts with links to particularly useful or favorite posts, and then adding those as "sticky" posts to the tops of forums. Often, these evolve into FAQs and whatnot.

We should find a few people to delegate that task to...

Seriously though, I think the more people who have small focused jobs they enjoy really helps the community out. Nobody gets too burdened, they do what they enjoy, and if they fall off the radar for a bit, it's not as catastrophic as if the one person who does everything gets hit by a car.

I'm happy installing and upgrading and setting things up for people, but I don't have much time for editorial and content issues. My near-term goal is to separate the austinimprov.com sites from the hardware that runs them, so I can keep them running regardless of where the physical servers live. I should start a separate thread on what jobs austinimprov.com needs (moderators, ad manager, mail manager, etc.) and see who's interested in taking them on.

[ChrisTrew.Com]

This thread is filled with a bunch of really shitty information.

[DollarBill]

I say NO to pruning. That kind of information loss is what leads to wars and genocide and the show "Will and Grace".

[mcnichol]

I like having all of the history here -- I'd rather not get rid of anything.

Why get rid of it if it's already here?

[arclight]

Sounds like the consensus has spoken.

Btw, I am really enjoying my new Nokia E71.

[Jessica]

By the way, Three Cheers for our amazing administrator! Bob you rock, we will totally miss you.