I’ve gone through this process quite often, and most times its pretty fun to see new things become available that solve particular eccentricities or minor flaws in the tools I use already.

Recently, I’ve been on the “Am I documenting things as efficiently as I can?” kick.

First of all…  I’m not a fan or wikis.  Actually, I dislike wikis.  Most of them are atrocious, with very few exceptions.  I find directly editing HTML easier than editing wikis, with their ridiculous markup language — come on,  two single-quotes for italics; i.e., ''italics'', three single-quotes for bold; i.e, '''bold''', and FIVE single-quotes for bold+italics; i.e., '''''bold+italics'''''?  Really?  That’s 10 mofreakin SINGLE quotes to make something italicized and bold!  HTML?  Much easier.  <i>italics</i>. <b>bold</b> and <b><i>bold+italics</i></b> Ok, maybe not so much easier, but definitely less confusing.  So, when creating documentation that typically lives on a webpage, are wikis actually easier?  I’m inclined to think they’re not.  But editing isn’t the only reason.

I have a philosophy on systems administration documentation, which I’ll get into later, but in essence it overcomes the chicken-and-egg problem.  Basically, if there is documentation for system administration about how to properly set up a wiki, where should this information be stored?  In a wiki?  Most wikis, like Mediawiki, store the information in a database.  Systems administrators need documentation about how to properly set up databases.  We just multiplied our chicken-and-egg problem by 2.

1. Set up a database as a prerequisite to have a wiki
2. Document how to set up a database properly
3. Set up a wiki to hold the documentation
4. Document proper set up of a wiki

Where does the information from #2 and #4 live?

HINT:  “Inside the wiki” is the wrong answer, and if this isn’t obvious to you, you should consider another career field because you are bound to have other more pressing problems than where to store your documentation.

Back to my periodic re-evaluation of tools…

In scouring the Interwebs for new and better tools for documentation, I found this abstract of a talk from a Linux conference in 2008.  It speaks directly to the chicken-and-egg problem I’ve been pondering:

Most professional teams will have some form of team documentation used as a reference by existing team members but also as a means of getting new staff up-to-speed. System administrators are no exception, but have their own set of additional requirements: lots of diverse, complex systems mean lots of procedures to recall. It’s also no good having procedures for repairing a database server stored in a database table.

I recently undertook a project to migrate Newcastle University UNIX team’s internal documentation from a simple filesystem-based scheme to a web-based wiki system using “mediawiki”, the software that powers Wikipedia. Once I had achieved this, it became apparent procedures and tools were necessary to help manage the continued review and updating of the documentation. I also became aware of several issues with the approach that impacted us specifically as systems administrators.

The talk itself was probably much more detailed, but the issue is relatively simple.  You need to be aware of any circular dependencies you’re creating, and try like the dickens to avoid them.  Also, everyone relying on the documentation needs to be well aware of the limitations of the documentation system.  Sure, you can devise a documentation system that doesn’t store its information in a database, but there are other dependencies as well — documentation about how to fix a broken web server might not be best suited inside a wiki, since all wikis rely on web servers to operate.  Granted, you may be able to quickly erect a web server, but this itself creates an unnecessary complexity that needs to be overcome in order to access procedures detailing how to fix a different problem.

Most things in systems administration deal with these types of issues, which I’m going to try not to get into except to say that if you can deal with these types of problems you will probably be a very good systems administrator and not paint yourself into a corner of dependencies.

Back to documentation.  In the abstract of the talk, it mentions a “wiki compiler” as a possible solution to the documentation chicken-and-egg problem.  The concept is that it isn’t strictly a wiki itself, but a method to construct a wiki dynamically based on files on disk.  If the web server explodes, you still have the files.  The files aren’t stored in a database, so that dependency is removed as well.  Also, the files can be under a version control system, so if an author makes a mistake somewhere this can be easily and quickly reverted.  One such “wiki compiler” is ikiwiki, apparently.  The more I read about this concept, the more I like it.  I’m still researching documentation tools, but this is definitely one that I will be exploring more, if only because it seems to make the documentation itself less complex.

My systems administration documentation philosophy is also considerably simple.  Systems documentation basically consists of only 3 types:  online, offline, and hybrid.

• Online systems documentation should only consist of procedures, concepts, ideas, and notes that have zero dependency on web servers, and not depend on any other part of an infrastructure needed to access the documentation itself.  A good example would be procedures about how to create a user account — creating a new user isn’t dependent on the wiki, the web server, or being able to access the documentation within.
• Offline documentation is system details and procedures needed that deal directly with portions of the infrastructure that may be offline or broken; e.g., disaster recovery procedures.  This type of documentation is offline primarily to simplify the review/update process needed to keep the documentation current.  If procedures to restore data from tape are online, of course copies can be printed to use when/if a critical file server needs to be restored.  However, if the data is moved to a different location or the restoration procedure is modified in some way, the documentation needs to be updated.  If this information is kept online, the new information should be added, and then printed again, making sure to destroy the previously printed copies to avoid confusion.  Keeping critical information online quickly becomes a complex procedure in itself because the information needs to be kept offline on paper in the event of an emergency, and updating the information in multiple locations is prone to error and confusion.
• Hybrid systems documentation is a mixture of offline and online.  Hybrid docs are normally created offline, converted or translated into online format, then added somewhere to be available online as well.  They are never edited online, to avoid having multiple conflicting documents that would cause confusion in an emergency.  DocBook is a good example of hybrid systems documentation:  a single source document is created from which other versions can be generated in HTML, PDF, PostScript, plain text, etc.

Recently, a Slashdot submission in this context was:

“Lately I’ve been rethinking my personal security practices. Should my laptop be stolen, having Firefox ‘fill in’ passwords automatically for me when I go to my bank’s site seems sub-optimal. Keeping passwords for all the varied sites on the computer in a plain-text file seems unwise as well. Keeping them in my brain is a prescription for disaster, as my brain is increasingly leaky. A paper notepad likewise has its disadvantages. I have looked at a number of password managers, password ‘vaults’ and so on. The number of tools out there is a bit overwhelming. Magic Password Generator add-in for Firefox seems competent, but it’s tied to Firefox, and I have other places and applications where I want passwords. And I might be accessing my sites from other computers that don’t have it installed. The ideal tool in my mind should be something that is independent of any application, browser, or computer; something that is easily carried, but which if lost poses no risk of compromise. What does the Slashdot crowd like in password tools?”

The response is filled with witty replies and interesting views and suggestions as per usual.  Nothing really new usually surfaces when someone asks this on Slashdot, since it seems the capacity to have passwords for online banking, social networking, work computers, home computers, blogs and whatever else grows and evolves faster than the ability to keep track of them all efficiently (and securely).

A while back, I wrote about my solution to this problem after having tried to solve it different ways.  In that post, I detailed my evolution from a Java application on a USB keychain to a website called Clipperz.

Well, I have been using Clipperz for almost 2 years now.  It is immensely useful and efficient.  I have had ZERO problems.  Yes, none, nada. NO problems whatsoever.  How many things can you say that about?

Clipperz does seem to be growing in popularity, since the last time I remembered the question asked on Slashdot, hardly anyone recommended Clipperz.  This time, a few people mentioned Clipperz on Slashdot.  However, its been 2 years and Clipperz still has the “beta” status.  Granted, Google Mail was in beta for years until they became “production”, but still…

Anyway…

I’m actually still in the initial stages, in the overall context of the project.  Basically, to create a Linux distribution (from scratch, as I am doing), it begins similar to a chicken-and-egg problem.  You have to use an existing system to create building blocks, and use the building blocks to create a complete toolset.  Then, you use the toolset in a limited environment, which makes zero assumptions (a “chroot” environment, for geeks out there), to begin construction of the target system.

While doing this is actually pretty straightforward, usually, the difficulty and complexity enters the stage when thoughts about the future begin to crop up.  Issues such as future maintenance and automation complicate matters because while there might be a working toolset to construct a target system, no one wants to have to manually create the target system each time something changes.  Also, if something changes within the toolset itself, the component will need to be rebuilt, or worse, the entire toolset will require rebuilding such that the whole thing remains cohesive.

This is similar to around the time when man first thought about hunting and killing for food.  He realized he could fashion a tool to make a weapon.  Next, he used the tool to create the weapon, which was then used to hunt and kill for food.  As evolution continued, he created more efficient tools, to in turn create better weapons that would enable him to hunt and kill more exotic food.  The same general principle exists when creating a Linux distribution — though, we’re not depending on the need to eat in our software toolchain, we are depending on the ability to make more software available, which will comprise a Linux distro.  The tricky part is to know how to manufacture tools in such a way that when a component in the process breaks, it is trivial to fix it so much to the point that if a tool very deep in the toolset breaks there is less need to re-manufacture everything that was built with it.  Henry Ford had a very good idea, with small replaceable parts.

At this point, I’m at the point that I have the toolchain built.  Before I go much further, I need to figure out a sensible way to maintain this toolchain so that if/when something becomes stale or breaks, I will spend less time repairing the toolchain and more time tending to the actual Linux software in the distribution.  Also, since the toolchain itself was build manually and painstakingly, I also need a system to churn out software using the toolset I have.  Perhaps a framework/system could be used for both maintaining the toolset, and maintaining the distro?

I’m quite sure this is not entirely a brand new problem — major Linux distros would have had to flesh this out very early on.  The trouble is, though, whatever they’re using to handle this type of framework, they’re not very up front in saying exactly what they use, what it does, or where it came from.

I know what I want my system to do.  I have a general idea how it should fit all together.  My dilemma is that I’m splitting my time between searching the Internet for any existing software to satisfy the need, and planning how I might create my own toolchain/distro maintenance framework/system.  There are, actually, existing systems that do something along these lines, but they were designed for very specific purposes, and implemented with specific needs in mind, which resulted in assumptions that require specific platforms.  Not very useful in my case, because I’m creating from scratch with no assumptions.

Eventually, I will have an automated system that will:

• institute modularity wherever practical
• automatically rebuild (with dependent software if necessary) anything that needs updating/fixing
• handle both the toolchain itself, and the target Linux software

The site, now released to the public, is Suse Studio.  The concept behind the service is that you can remotely create, per your preferences, a Linux distribution, based on a recent Suse flavor — basically, either OpenSUSE or SUSE Enterprise Linux.

How does it work?  You login (with OpenID, or after creating an account), select a base to begin with;  GNOME, KDE, generic X, text only, etc.  You can select 32-bit or 64-bit.  Then, you can select software to be installed down to the specific package.  You can even upload your own packages — rpm, .tar, .tar.gz, .tgz, .tar.bz2, .tbz, or .zip, specify particular software repositories, or choose from over 300 pre-existing software repositories (some with rarely found packages, such as ATI or nVIDIA drivers).  After selecting software to be installed, you are brought to a configuration webpage that allows you to completely configure the system prior to building it, with the added option of uploading your own customized files.  After building the system, it can be tested via “test drive” — completely able to log into the newly built system via the website.

Once the system has been built (and optionally tested), it can be downloaded and installed as either:  Disk image, LiveCD (ISO image), Xen virtual machine, or VMware appliance.

The service is intelligently laid out, and quite intuitive, enabling the user to create systems within minutes.  Suse Studio routinely clears out built systems, but keeps the meta-info such that systems specified previously can be re-built easily and quickly.

It really seems to be a well thought out service.  I have experienced other web services that allow the user to build a Linux distribution, but never anything this detailed.

But, Dave…  There are well over 800 Linux distributions out there already.  What makes you think you can compete?

Competing is not the point.  The point in mountain climbing is not to see who can reach the top first, but to learn something about yourself, and generally “because its there”.  If my Linux distro gains momentum and ends up being something more than just a hobby/pasttime, great!  If it doesn’t, that’s fine too.  I’m not going to make any promises to anyone, at first, because this will be to used fulfill my own needs which don’t necessarily apply to anyone else.  If I later discover that other people have needs similar to mine, we’ll talk about where to go from there.

Basically, there are two approaches in creating your own Linux distribution:  building from scratch, and basing it on existing.  Some Linux distros are based on other Linux distros.  For example, Ubuntu was created out of re-building software packages from the Debian/GNU Linux distribution, but Debian was created from scratch by hundreds of volunteers.  Ubuntu itself has spawned derivative Linux distributions; Kubuntu, Xubuntu, Ubuntu Studio, and Mythbuntu to name a few.  Both methods have their advantages and disadvantages, and both are equally valid (and probably equally popular).

A few existing Linux distros come with their own “roll your own” application that will automagically create a customized ISO image that you can use to modify/install/spin/fold/mutilate to your heart’s content.  Yes, the method bases it on a specific Linux distro, but it will be relatively different depending on the customizations you’ve set in place.  For example, Fedora has Revisor.  There are also non-distro-specific utilities on the web such as Instalinux, as well as complete Linux distros based on customization and optimization in which everything is built from source — Gentoo.

The other method is to construct a Linux distribution completely from scratch, appropriately named Linux From Scratch (LFS).  While this approach isn’t entirely difficult initially — its just building binary files from source files — it is time-consuming.  It is also recursive, meaning that software you compile in the beginning stages are depended upon by other software you build later, and you’ll have to start over from the point of contention if you discover something broken.  This is termed “building the toolchain”.  Building software so that you can build other software with it, that software becomes dependent on the initial software build, which is then used to build even more software — creating a chain of software used to create a system, which will be bundled together to form an installation, and a Linux distribution.  That’s the easy part.  The hard part isn’t in building the software; its in maintaining the whole thing such that when a bug is discovered it is easily and quickly patched/fixed and the remaining components of the toolchain remain relatively unaffected or are automatically rebuilt using the new link in the toolchain.

Whatever method used to build the Linux distribution, there still needs to be some point to the exercise, else time invested is wasted in making an exact copy of something that could have been simply downloaded to begin with.

My goals:

• Hybrid approach, mixing Linux From Scratch with useful tools from established distributions.
• A practical level of security in the system, without sacrificing usability.
• An agnostic mindset for software packages, trying not to rely on one set of software built for one distribution.  A best of breed path would be ideal, taking the best software from well known distributions, and possibly introducing software that is not available in any Linux distro.
• Initially, aim at the desktop and see how that goes before configuring things for a server platform.
• Simple for the user, but without the cost of being difficult or complicated for the system administrator (who is often the user also).

I’ve had quite a few discussions with friends about what I should name it. After all, that’s the most important aspect of a Linux distribution, right?  Amazingly, in the discussions about naming my distro, no one ever asked me what I wanted it to do — but everyone has suggestions for a name.  I guess in the age of Google, and Yahoo, the name doesn’t have to be related to what it does.  I have a pretty good idea what I want to name it, but that may change later.  Considering there are over 600 distributions of Linux in circulation, a few hundred discontinued, and a few hundred more about to be announced (some of which might be discontinued in a few months), the name doesn’t matter that much to me as long as it isn’t completely ridiculous.  Douchebag Linux doesn’t smack of “Download me!  Use me!  I’m useful!”  On the other hand, Master of All Linux sounds good, but is probably just a tad too ambitious.

I wish there was a quick way to check if a name is already taken.  Like a global registry, or something that I could search for to determine if a name is used.  Distrowatch.com is good, but there is a 90 day waiting period for a Linux distro to be added to the list because “this is to discourage submission of new projects that start with great enthusiasm only to vanish in a few short months”.  Like I said…  Easy to start, difficult to maintain.  Incidentally, ParanoidLinux is on the waiting list to be added to Distrowatch.  Its not mine, but I get quite a few people that read this blog after searching for that particular distro.  I wonder why…

LaTeX has been around for quite a while, and is widely known in the scientific and academic community as being the best in desktop publishing and typesetting.  The principle is that the writer should be able to easily focus on the content/topic rather than being distracted by the way the resulting document appears.  Logical structure in articles, chapters, and sections are created by the author with visual presentation happening only after the content is complete.  LaTeX also makes it rather easy to create complicated mathematical formulae in a document otherwise not possible (or very difficult) with word processors — mainly the difference between word processing software and typesetting software, or What You See Is What You Get (WYSIWYG) versus What You See Is What You Mean (WYSIWYM).

Scalable Vector Graphics is a specification for creating 2-dimensional graphics from XML text files.  The resulting graphic images can be either static or dynamic, animated, or even interactive.  SVG is a relative newcomer in the field of graphic image formats, so it is not as widespread as say GIF or JPG.  SVG is also still in the early stages on the web, though there is some support in existing web browsers.  However, to compare SVG (vector images) to GIF or JPG (bitmap images) is tantamount to comparing apples to oranges.  Bitmapped images rely on dots, using positions and color for the dots.  Vector-based images rely on predefined shapes.  This means that vector-based images appear much sharper at very low resolutions where bitmap images begin to appear blocky and coarse.

The cool part:

Recently, a professor I work with came to me with a very specific need.  He wanted a particular piece of software that has been written as a plugin for Inkscape, an SVG editor, that allows LaTeX code to be input into an object and rendered inside an SVG image.  Not only that, but once the LaTeX object is embedded into the image, you can re-edit the rendered LaTeX object by changing the LaTeX code.

Why is this cool?  Simple.  Before, if you wanted to create a mathematical formula or complicated diagram for use in an image say on a webpage, you either had to use LaTex to create your formula inside a document, then convert the document to an image, then edit the image to be used for presentation (i.e., on a webpage), or you had to manually draw the formula by hand and save that as an image. This was a tedious and often error-prone process, with the only payoff in the end being a static image of a formula.

Hand-drawn equation

LaTeX rendered equation

Now, you can create mathematical formulae inside SVG images with simple LaTeX code inside an SVG editor.  The plugin for Inkscape is called Textext, and will allow you to re-edit the rendered LaTeX formula after it has been rendered inside the SVG image.

Recently, I’ve been playing around with other desktop virtual machine managers, trying to install as many operating systems as I can onto my trusty laptop (a Dell Inspiron 1525, with 3GB RAM, 100G hard drive, Intel Core2 Duo 1.83GHz, running Windows Vista), to see what could be accomplished in the way of running something on my machine without having to re-configure it into a multi-boot mess.  The point of this exercise is to gain the benefits of another operating system without having to reboot, wasting some time while I wait for booting to complete.

I’m using Sun’s VirtualBox, which is actually a pretty darn good desktop virtual machine manager.  Setup is easy, configuration of a virtual machine is fairly straightforward, it doesn’t take up much space, and best of all it doesn’t cost anything. VirtualBox can take advantage of VT-x/AMD-V if your chipset supports that function.  My Dell laptop does not, so I’m relying only on software virtualization.  A cool thing about VirtualBox is that not only will the software allow me to easily switch between virtual machine and regular host desktop (Windows Vista in my case), without ‘escaping’ the virtual machine, but it has a function called “seamless mode”.  Seamless mode incorporates windows from a virtual machine desktop into the hosted desktop.  Seamless mode is only possible if the guest OS (the operating system in the virtual machine) is supported by VirtualBox Guest Additions.

Figure 1. Non-seamless VirtualBox

In Figure 1, you can see that the entire Xubuntu Linux desktop is a window on the Windows Vista desktop.  I am able to switch between the two via mouse, though the Alt+Tab trick doesn’t work as well because Xubuntu Linux captures my Alt+Tab keypresses and switches windows within the virtual machine desktop.  After pressing the right Ctrl key, Alt+Tab works, treating the Xubuntu virtual desktop as an application window on the Windows Vista desktop.

Figure 2. Seamless VirtualBox

Figure 2 displays that there is no Xubuntu Linux desktop.  It has been incorporated into the Windows Vista desktop.  You can also see near the bottom of the screen that the Linux taskbar that was at the bottom of the virtual desktop in Figure 1 is now at the bottom of my Windows Vista desktop in Figure 2.

Disclaimer:  No, I do not normally run Internet Explorer.  I only had it open during the screenshots to showcase that it is actually WIndows Vista, since I have a non-standard Vista desktop theme.

Another cool feature of VirtualBox Guest Additions is the ability to have a shared folder between the virtual machine guest OS (Xubuntu Linux, for example) and the host (Windows Vista in my case).  This particular feature, I *love*.

I’m often “printing” certain things into a Postscript file, converting it into a PDF, and archiving the PDF for historical filekeeping purposes (think: receipts you get from websites).  In Linux, I “print to file”, convert the Postscript file into PDF, and then archive the PDF file.  In Windows, this is not as cut and dry.  I can certainly “print to file” as Postscript, but converting to PDF is a bit more involved.  To overcome this, I have a Linux virtual machine running with a guest-to-host share in VirtualBox; Windows Vista sees this as a folder, and the Linux virtual machine sees this as a mounted directory.  I can “print to file” from Windows directly into the VirtualBox share, go into Linux, and “ps2pdf” the file, and it gets written as a PDF in the same place next to the original Postscript file.  From that, I can either view the PDF in Linux, or view it in Vista with Adobe Acrobat Reader.  There might be easier ways to accomplish this only in Windows, but I’m more familiar with the Linux tools, and in Linux I don’t have to launch a separate application.

So far, in VirtualBox, I have created virtual machines for Xubuntu, openSUSE v10.3, openSolaris v2008.11, Haiku, and even WIndows 7.  Haiku runs a bit slowly, though.  Each of these virtual machines have about 512MB RAM and no more than 6GB of hard drive space.  Keep in mind that the amount of memory you allocate for a virtual machine gets used by that virtual machine when it is running.  Similarly, hard drive space you declare for a virtual machine becomes occupied by that virtual machine’s hard drive on the physical machine in the form of a file.  My meager Dell Inspiron with 3GB RAM and 100GB hard drive, can handle the hard drive space and a few virtual machines running at the same time with a few signs of stress, but nothing major.

My next stop on this Virtual Desktop adventure is with some of the older operating systems I’ve used in the past, working on this laptop.

This is one of the main reasons Linux people don’t like Windows.  The computer-human interface is limited to a mouse, and the user is forced into a world that is based on finding the correct icon and double-clicking, as well as “Click Next to Continue” and “Are you sure you want to do this? [Ok] [Cancel]“.  Linux users normally subconsciously ask themselves each time “Of course I’m sure, if I wasn’t sure why would I have clicked on it?” There is rarely an instance on Linux where the user types something onto a command-line and is asked “Are you sure?”

For most Windows users, typing on the command-line is not how things are done.  The command-line itself is hidden from view.  Even when it is launched, only the more savvy Windows users know what to type to accomplish anything, because it is remnant of the old DOS era where most things where typed at a C:> prompt.

What if Windows users could type something to launch applications, or find existing opened windows?

Enso is probably the coolest thing I’ve seen, on a Windows platform, in a really long time.  Humanized has taken the concept of a command-line interface and set it free from the window, which makes it a bit better than a terminal window on a Linux platform.  Want to open Mozilla Firefox? Just hold down the Caps Lock key and type OPEN FIREFOX.  A translucent window with no frames pops up informing you “Opening Firefox…” and then Firefox launches.  Want to make a quick calculation in Calculator?   Hold down Caps Lock, and type OPEN CALCULATOR. Up comes the calculator.  Want to go back to Firefox and not have to Alt+Tab your way through all the opened windows?  Hold down Caps Lock, and type GO FIREFOX. Firefox will be brought to the front.

ALL FOR FREE.

Click here to view the embedded video.

Now what would happen if you brought the diversity of vast number of resources on the Internet to bear on this concept?

Mozilla has taken the concept of a command-line and integrated it into the web browser.  Its still in the very early stages of development, but is already VERY useful, especially to a command-line junky like myself.

Ubiquity lets you do tedious tasks within seconds inside the Firefox browser that you would normally spend minutes doing.  Consult your calendar. Check your inbox. Quickly email a friend. Pull up a map of a list of addresses.  Search Youtube.  Translate highlighted text. The list goes on and on because there are new commands being created all the time — and this has only come out a mere week ago!  The actual cool part is that Ubiquity lets you mash and mix things up — insert a map into your email, find lyrics for a song and post them to a webpage, invite someone for a meeting at a restaurant complete with a restaurant review…

Click here to view the embedded video.

A likely solution comes in the form of virtual machine hosting. Similar to the idea of Apache’s <VirtualHost>, but at a lower level so that <VirtualHost> isn’t really needed — the entire machine, albeit virtual, is dedicated to the web server and subordinate application. This seems to make it easier to migrate a web server around if needed, rather than provisioning a system, installing software and duplicating configuration such that an application can continue to function properly. Instead of starting from scratch, the entire set of operating system, configuration and specific local data migrates with the system, and launches in seconds rather than the testing associated with installing a new system. Granted, I’ve only just started with this concept, it seems quite sound.

I’ve created a few images to be used as virtual machine “templates”, which I use as a headstart when creating a virtual host. Sure, my images are pre-installed with web server functionality, but they could be just about anything really — any type of server, just about. A while back, I played around with VMware and the idea of using whimsically launched virtual machines to test things I wanted to try. Then, I got into downloading pre-installed images of software I wanted to try out, which by the way is a very handy way to test software without all the overhead of installing/configuring the software correctly just for a trial run. Now, I’m strung out on virtual machines running as part of providing a service.

The ability to manufacture machines at will is a very handy utility. However, I can see it easily getting out of control. Left unchecked, I can visualize a machine room with racks of physical machines with no real idea of what is actually running on any of them without digging into a bunch consoles. Documenting a dynamic environment like that seems insurmountable. The ability to easily locate a physical machine to diagnose a malfunctioning service seems to be replaced by easy replacement and/or migration. I’m still not sure which is better; knowing which physical hardware is running what specific software, or not caring which physical hardware is running which software such that if a service goes down a replacement set of software can be launched somewhere else.

I guess I’m just old school in that I like to know which is doing what, and where I can find it, so I can give it a technical tap with my 5 pound hammer. I have seen that there are automated utilities and tools that handle the installation, configuration and even migration if necessary, all through a handy web interface. Someone termed this concept Virtualization Management. (I’m glad its not called Virtual Management, ’cause that could get confusing really quickly.) While I’ve yet to actually try out Virtual Management, it still seems to not solve the problem of knowing which does what and where it is. It actually seems to make a sysadmin care less about location, because it relieves the admin of the task in having an intimate relationship with each system… It becomes a group of physical machines, with invisible virtual machines inside them, running all the services you need.

So, in the new virtualized world… we have new terms. Virtualized Management, which handles virtual machines such that the physical hardware doesn’t succumb to virtual server sprawl.

Server sprawl itself isn’t a new term. It has been around since headless PCs have been installed into racks, more or less.

Picture a room of computers in racks, running the same operating system, and each of them have a random set of software serving a specific purpose. A person would have some sort of inventory just to keep their sanity in making sense of it all, at the very least just to know where stuff is and what its connected to! If you discover a bug in the software on one of them, or someone reports a problem, you consult your inventory, locate the machine and apply the appropriate fix or update. Problem solved.

Now picture a room of computers in racks, running the same operating system, and each of them have multiple virtual machines running inside with specific software installed with the added ability to easily move a web server from one machine in a rack to a different machine on the other side of the room. If you discover a bug in the software or someone reports a problem, how do you attack the problem? Unless you have an ever changing list of virtual machines and a list of which physical machines they’re located, how is the problem service/system to be found? The answer seems to be “just use Virtualized Management”, since this is how virtualized infrastructure is constructed to begin with. I’m not sure that’s the right answer, since that seems to add complexity into the overall system — I used a relatively static list of systems/services/locations in a non-virtualized environment.

Virtualized Management seems to take my static list of systems/services/locations and turn it into a grey hazy cloud of services with no real notation about systems or locations.

I plan to test run a few Virtualized Management systems to determine if it solves my concerns. Some of them are:

• Enomalism
• BixData
• ganeti
  

In the 200MHz and 400MHz Intel XScale® PXA255 processor varieties with 64MB RAM and 16MB Flash memory (with optional Bluetooth) you won’t be the next Quake master or able to level up in World of Warcraft, but its smaller than any regular PC and able to run quite a few applications.

There is even a gumstix Software Development Kit, if you can’t find the applications you need from the pre-built binaries.