I’ll say that again… Email is not a storage mechanism.

By this, I mean that the purpose of electronic mail is not to store important files, information, or future reference material. It was never intended for that purpose, and even in today’s standards it still falls short of that use.  Of course, there is GMail today.  Of course, there are GMail extensions (like gDisk and GMail Drive Shell Extension) that allow you to store your MP3 collection, photos, etc.  That is a good example of what I’m referring to.  I’ll explain…

GMail, as most of us are aware, is not a typical electronic mail system.  It does not operate within the paradigm of traditional email systems.  Google Mail’s primary interface is via the web page in which email messages are only sorted by “threads” (“conversations” in GMail-speak).  But, more on GMail later.  Back to the point…

Email began as a way for users of a time-share system to communicate with one another, coordinating within the same closed system.  Soon thereafter, it became a method of communicating with users of other time-share systems, yet with serious limitations — namely, the sender of a message was required to know the path in which the message took to get to the intended recipient.  Instead of having addresses such as

poe@deadpoets.org

there were “addresses” such as

localhost!nextdoor!nextnextdoor!poe@deadpoets.org

which meant that the message had to travel from localhost, to nextdoor, to nextnextdoor, to finally deadpoets.org in order to reach user ‘poe’.  The machines did this in an automated way, as long as the route specified was correct.  If one of the machines along the message route was offline, or otherwise not accepting incoming mail, the sending machine held the message for a certain period of time until either the message was accepted on the receiving host or the sending machine effectively gave up — at which point, the message was lost forever.

Eventually, the machines connected to the network grew in number and a machine’s knowledge of other networked machines needed to scale as well.  Email needed to change with the new networking methodology, which is why we have user@somesystem.com today.  The sender of a message needs to know only the address(es) of the recipient(s), the subject of the message, and the message itself.

Notice in all the above explaination it reads “the message”, and not “the file”?  There is a reason for that.

Consider this example:

Alice wants to send Bob some files.  The total size of the files is 9.5MB. The contents don’t matter for the purpose of this example, so lets just say the email contains a few large photo images, and a large document.  In order for Alice to send these files, in an email to Bob, she would need to first specify Bob’s email address as the intended recipient.  Next, she will likely describe the contents in a few words in the Subject: field of the message, “The stuff I wanted to give you”.  Then, she sets about attaching each file she wants to send to Bob.  Each of these files becomes encoded in a very long set of letters and numbers, completely unreadable by any human, and inserted into the email message “envelope” so that each email system that handles the message will be aware that it is a message with a Subject: and multiple files attached of differing size and type, such not to get the files intermingled among each other, nor this specific message’s contents intermingled with any other message that might be handled.  Next, Alice presses “Send”.  It takes a moment for her computer to actually send it because most email systems aren’t expecting (or designed) to handle messages of that size… but it gets sent.  The message is then copied into Alice’s “Sent Mail” mail store (sometimes called “outbox”).  Bob does not see this message right away — this is not file sharing, nor is it Instant Messaging (IM).  Alice’s message is received from Alice’s computer, is copied onto Alice’s email server, which then needs to determine which machine handles Bob’s email.  Once that is determined, the message is sent again — to the machine listed after the ‘@’ in Bob’s email address.  That receiving machine typically makes attempts to verify that it is a message coming from an actual person (like Alice, and not a Spam robot), is destined for a person that it handles mail for (like Bob, and not Boob), and that the size of the message is within the system’s constraints for reasonable handling (typically 10MB).  After the message is accepted, it is written to Bob’s email server (this is the 3rd copy of the message) for delivery handling.  Assuming that Bob has not forwarded his mail elsewhere (which would further the process of sending/copying the message again), the message is then stored in a holding area on the server’s hard drive, to await Bob’s email client.  Once Bob’s email client connects to the email server, the message is copied yet another (4th) time to Bob’s computer.  The message will reside on both Bob’s email server, Bob’s computer, and Alice’s email server, and Alice’s computer (in her “Sent Mail”, remember?) until either Alice or Bob delete their respective copies of the message.  For a single 10MB message, it has taken multiple computers copying, and costing a total of at least 40MB of storage space.  This is not taking into account various spam/anti-virus systems, which also typically store each message for a short time.  More importantly, this is also not taking into account that had the message been addressed to more than one person (say Bob and Charles), the message would be stored 6 times — server and user’s computer, for each user — which would amount to a total of 60MB for the sender and two recipients.

Email systems treat messages as such. Sure, each message is a file, but a message to be delivered to user1 cannot/should not/will not be considered the same message as to be delivered to user2, even though it has the same file attached to the message and may contain the exact same contents.  Electronic mail is designed this way for privacy; not file-sharing.

“But, Mr. Linux Ninja Geek…  storage is cheap!”

Yes.  Storage is cheap.  However, transmission is not.  It takes relatively a small amount of effort for your computer to generate data, or even say copy data from your camera, and store it onto your computer.  It is much more effort to transmit that same data across the Internet to another computer, and have it stored there indefinitely.  Enter email into that equation and the effort is mulitplied by each computer the message travels through to get to the final destination.

“Ok.  So, I shouldn’t send files attachments in email at all??”

That is not what we’re talking about.  We’re talking about storing email messages indefinitely.  Consider that information in a typical message has a given lifetime.  Normally, this lifetime is very short, on the order of days or even weeks, possibly even months.  After this time passes, is the information in the message of the same importance, or has it become much less important?

To demonstrate this, let us employ an analogy…

In the old days, before email, people wrote correspondance — stone tablets, papyrus, handwritten, typewritten.  The message itself was carried, by another human, to its intended recipient, and either read aloud or delivered into their hands.  Once the information within the message was received, what happened to the message itself?  In the case of stone tablets, it was likely destroyed — or made into some type of monolith, depending on what the actual message was.  In the case of papyrus, the message was read aloud, retrieved, and kept for futher use — this is why the message was stored on a scroll, because it contained more than a single message for more than a single recipient.  In the case of handwritten or typewritten correspondance, either the letter was kept in a folder in a file, or it was discarded sometime after the message was understood.

That last part, concerning handwritten/typewritten letters, is probably the closest analogy to email.  After the letter was filed away, what was its disposition?  More often than not, the letter sat in the file for a long time, until someone either tossed it out with the trash, or it was framed for historical purposes.  Point:  a letter was hardly ever kept “in case I need it again”.  The physical letter’s disposition was certain, upon the moment of receipt, similarly to stone tablets and papyrus scrolls.  Why?  Because physical objects need space to be stored indefinitely.  The more physical objects that need to be stored, the more space required, of course.

Hard drive space is required to store electronic mail messages as well.  In all cases of message storage, the information contained within does not change after delivery.

Enter GMail.  GMail’s claim to fame was that, initially, the storage amount was enormous, compared to other offerings like Hotmail.  Leveraging Google’s search abilities, supposedly you could instantaneously find any email you ever received in the GMail system.  This goal is in line with having conversational correspondance with other people connected to the Internet, only in a different way.  GMail does not sort messages by date, subject, or even sender of the message like typical email client software.  The only sorting mechanism available is by “thread”, which makes GMail seem more like Usenet, or an online discussion forum.  This design structure does not seem to lend itself for file storage at all — much less attachment storage.  Sure, you can save a message (or entire conversations) indefinitely, for later review.  How easy and practical is it to do that?  How important is that email from years ago?  More importantly, how many other email systems are similar to GMail?  It doesn’t seem that GMail is a good gauge as to what an email system can or cannot do, since it seems to be a consensus that GMail is different from the rest, and since GMail is due to fail without warning among other technical limitations.

Given that a message’s information/content/meaning does not change after it is delivered, why is email kept for so long?

Not just that, but if a message is noticed to be lost (presumably a while after it was actually lost), why is it so important to have the message restored?  What could possibly be contained in a message, that wasn’t noticed to be missing, that has become critical this very moment?  Could the information not be resent from the sender?

More to the point:  Why are people storing information in email?

This term was coined in the mid-’90s when two guys at Yale wrote that lifestreaming is:

…a time-ordered stream of documents that functions as a diary of your electronic life; every document you create and every document other people send you is stored in your lifestream. The tail of your stream contains documents from the past (starting with your electronic birth certificate). Moving away from the tail and toward the present, your stream contains more recent documents — papers in progress or new electronic mail; other documents (pictures, correspondence, bills, movies, voice mail, software) are stored in between. Moving beyond the present and into the future, the stream contains documents you will need: reminders, calendar items, to-do lists.[1]

The Internet is attempting to capture lifestreams of its citizens, but doesn’t appear as inclusive as most companies want it to be — mostly due to the nature of the events, or documents, or because no one web company owns the rites to all of one person’s information.  …This seems to be a good thing.

There are, however, quite a few aggregators on the web, like Facebook, Tumblr, Posterous, Collectedin, and Flavors.me.  These do a fairly good job of aggregating in a social network context, meaning that they typically lifestream content from social networking websites (silos) that people post specific content to.  For example, if a person posts a photo to Flickr (a silo), the photo is on that website of course, but can also be retrieved from sites like Facebook (an aggregator, and a silo).  I say that Facebook is an aggregator and a silo because Facebook started as a silo (only allowing posts from its subscribers), then branched into reading feeds from other sites like Flickr and Twitter, but Facebook itself is difficult to integrate into a secondary aggregator (like an RSS reader) as it has changed its layout and hidden its RSS feeds numerous times.

I particularly like the look and layout of Flavors.me, as it attempts to present information in a person’s lifestream, yet also segregates the data from different sources, which doesn’t intuitively give the audience a flowing context of the lifestream itself.  In other words, personal Flavors.me sites look very nice, but are still somewhat disjointed.

I decided to task myself in creating my own lifestream on a website, but it had to meet my own specifications while maintaining a particular look and feel.  I began the process by researching all the different types of information on the web that one could aggregate — from Facebook posts, to Flickr photos; from blog entries to music tracks played recently.  I came upon a very nice and specific piece of software named Sweetcron.

Sweetcron, created by Jon “Yongfook” Cockle, was designed to be blog software that could capture and display a person’s lifestream.  Similar to Tumblr and Friendfeed, but could be installed and run on a personal server, eliminating the need to subscribe to yet another commercial service.  Sadly, I also found that Sweetcron, as wonderful as it might be, is no longer maintained by the original developer.  This is where I discovered a Sweetcron fork — derivative software, named Lifepress.

Lifepress seems wonderful, as it has the functionality to aggregate feeds from different sources, and also comes with a bunch of plugins to handle the sources that aren’t as intuitive.  It is also quite them-able, though there aren’t many Lifepress themes to be found.  Luckily, Sweetcron themes can be easily adapted to Lifepress, though there aren’t as many Sweetcron themes, either, compared to regular blogging software like WordPress.

As with most lifestream sites, they contain things like Flickr photos, blog posts, etc. that you can read/view in the lifestream itself, I decided that mine would only be an aggregator rather than a full-blown lifestream blog that contained comments, etc.  With mine, only linked posts from other sites can be read at the source if the link is followed.  This allows me ease of maintenance in that I don’t have to worry about comments on a blog post and comments on a aggregated blog post at the same time.  Obviously, this also prevents me from posting once and having software disperse the data — such as when people post to Twitter, and it magically appears on Facebook also.  If I were to do that with a lifestream, it seems it would likely be caught in an endless loop…  blog -> lifestream -> blog -> lifestream, ad infinitum.

So, after a few hours of installing and modifying PHP scripts and CSS stylesheets…even had to edit an image or two…

I have my own lifestream website.  I only have a few feeds added in, but I’m sure there will probably be more as I toy around with things.

[1] http://cs-www.cs.yale.edu/homes/freeman/lifestreams.html

I’ve gone through this process quite often, and most times its pretty fun to see new things become available that solve particular eccentricities or minor flaws in the tools I use already.

Recently, I’ve been on the “Am I documenting things as efficiently as I can?” kick.

First of all…  I’m not a fan or wikis.  Actually, I dislike wikis.  Most of them are atrocious, with very few exceptions.  I find directly editing HTML easier than editing wikis, with their ridiculous markup language — come on,  two single-quotes for italics; i.e., ''italics'', three single-quotes for bold; i.e, '''bold''', and FIVE single-quotes for bold+italics; i.e., '''''bold+italics'''''?  Really?  That’s 10 mofreakin SINGLE quotes to make something italicized and bold!  HTML?  Much easier.  <i>italics</i>. <b>bold</b> and <b><i>bold+italics</i></b> Ok, maybe not so much easier, but definitely less confusing.  So, when creating documentation that typically lives on a webpage, are wikis actually easier?  I’m inclined to think they’re not.  But editing isn’t the only reason.

I have a philosophy on systems administration documentation, which I’ll get into later, but in essence it overcomes the chicken-and-egg problem.  Basically, if there is documentation for system administration about how to properly set up a wiki, where should this information be stored?  In a wiki?  Most wikis, like Mediawiki, store the information in a database.  Systems administrators need documentation about how to properly set up databases.  We just multiplied our chicken-and-egg problem by 2.

1. Set up a database as a prerequisite to have a wiki
2. Document how to set up a database properly
3. Set up a wiki to hold the documentation
4. Document proper set up of a wiki

Where does the information from #2 and #4 live?

HINT:  “Inside the wiki” is the wrong answer, and if this isn’t obvious to you, you should consider another career field because you are bound to have other more pressing problems than where to store your documentation.

Back to my periodic re-evaluation of tools…

In scouring the Interwebs for new and better tools for documentation, I found this abstract of a talk from a Linux conference in 2008.  It speaks directly to the chicken-and-egg problem I’ve been pondering:

Most professional teams will have some form of team documentation used as a reference by existing team members but also as a means of getting new staff up-to-speed. System administrators are no exception, but have their own set of additional requirements: lots of diverse, complex systems mean lots of procedures to recall. It’s also no good having procedures for repairing a database server stored in a database table.

I recently undertook a project to migrate Newcastle University UNIX team’s internal documentation from a simple filesystem-based scheme to a web-based wiki system using “mediawiki”, the software that powers Wikipedia. Once I had achieved this, it became apparent procedures and tools were necessary to help manage the continued review and updating of the documentation. I also became aware of several issues with the approach that impacted us specifically as systems administrators.

The talk itself was probably much more detailed, but the issue is relatively simple.  You need to be aware of any circular dependencies you’re creating, and try like the dickens to avoid them.  Also, everyone relying on the documentation needs to be well aware of the limitations of the documentation system.  Sure, you can devise a documentation system that doesn’t store its information in a database, but there are other dependencies as well — documentation about how to fix a broken web server might not be best suited inside a wiki, since all wikis rely on web servers to operate.  Granted, you may be able to quickly erect a web server, but this itself creates an unnecessary complexity that needs to be overcome in order to access procedures detailing how to fix a different problem.

Most things in systems administration deal with these types of issues, which I’m going to try not to get into except to say that if you can deal with these types of problems you will probably be a very good systems administrator and not paint yourself into a corner of dependencies.

Back to documentation.  In the abstract of the talk, it mentions a “wiki compiler” as a possible solution to the documentation chicken-and-egg problem.  The concept is that it isn’t strictly a wiki itself, but a method to construct a wiki dynamically based on files on disk.  If the web server explodes, you still have the files.  The files aren’t stored in a database, so that dependency is removed as well.  Also, the files can be under a version control system, so if an author makes a mistake somewhere this can be easily and quickly reverted.  One such “wiki compiler” is ikiwiki, apparently.  The more I read about this concept, the more I like it.  I’m still researching documentation tools, but this is definitely one that I will be exploring more, if only because it seems to make the documentation itself less complex.

My systems administration documentation philosophy is also considerably simple.  Systems documentation basically consists of only 3 types:  online, offline, and hybrid.

• Online systems documentation should only consist of procedures, concepts, ideas, and notes that have zero dependency on web servers, and not depend on any other part of an infrastructure needed to access the documentation itself.  A good example would be procedures about how to create a user account — creating a new user isn’t dependent on the wiki, the web server, or being able to access the documentation within.
• Offline documentation is system details and procedures needed that deal directly with portions of the infrastructure that may be offline or broken; e.g., disaster recovery procedures.  This type of documentation is offline primarily to simplify the review/update process needed to keep the documentation current.  If procedures to restore data from tape are online, of course copies can be printed to use when/if a critical file server needs to be restored.  However, if the data is moved to a different location or the restoration procedure is modified in some way, the documentation needs to be updated.  If this information is kept online, the new information should be added, and then printed again, making sure to destroy the previously printed copies to avoid confusion.  Keeping critical information online quickly becomes a complex procedure in itself because the information needs to be kept offline on paper in the event of an emergency, and updating the information in multiple locations is prone to error and confusion.
• Hybrid systems documentation is a mixture of offline and online.  Hybrid docs are normally created offline, converted or translated into online format, then added somewhere to be available online as well.  They are never edited online, to avoid having multiple conflicting documents that would cause confusion in an emergency.  DocBook is a good example of hybrid systems documentation:  a single source document is created from which other versions can be generated in HTML, PDF, PostScript, plain text, etc.

Anyone that puts an important file with an encrypted password on more than a dozen computers, with permissions of the file being world-readable, doesn’t really understand the ramifications involved.

Since then, I have been trying to “crack” the password through regular means.  Given their usage pattern, I’m fairly certain the password is 8 characters long, at least one capital letter, one number, and one special character.  Sadly, only that much information probably won’t be enough for me to crack it without throwing more compute power at it.  The tough part is that its a salted hash, so I can’t really use Rainbow Tables, and I’ve already tried dictionary attacks with Webster’s Dictionary.  So, I’ve resorted to brute-force automated guessing.  JTR seems pretty good at this, but even so it will probably take months to obtain the plaintext password.  It has been running for 11+ days so far.

The encrypted password in question is: {SSHA}KZhA0wzX4AThn9CkxBgYDmmy42pNY9ME

Salted SHA-1, of course.  If you know encryption algorithms, you already know what this is likely used for, but I won’t give that away.  I won’t tell you what its used for, or who it belongs to, or what you might do with it once you’ve cracked it, suffice to say the plaintext password in the wrong hands could cause some damage.

If you know of a quick way to crack such a password, other than what I’ve tried so far, drop me a line.

Recently, a Slashdot submission in this context was:

“Lately I’ve been rethinking my personal security practices. Should my laptop be stolen, having Firefox ‘fill in’ passwords automatically for me when I go to my bank’s site seems sub-optimal. Keeping passwords for all the varied sites on the computer in a plain-text file seems unwise as well. Keeping them in my brain is a prescription for disaster, as my brain is increasingly leaky. A paper notepad likewise has its disadvantages. I have looked at a number of password managers, password ‘vaults’ and so on. The number of tools out there is a bit overwhelming. Magic Password Generator add-in for Firefox seems competent, but it’s tied to Firefox, and I have other places and applications where I want passwords. And I might be accessing my sites from other computers that don’t have it installed. The ideal tool in my mind should be something that is independent of any application, browser, or computer; something that is easily carried, but which if lost poses no risk of compromise. What does the Slashdot crowd like in password tools?”

The response is filled with witty replies and interesting views and suggestions as per usual.  Nothing really new usually surfaces when someone asks this on Slashdot, since it seems the capacity to have passwords for online banking, social networking, work computers, home computers, blogs and whatever else grows and evolves faster than the ability to keep track of them all efficiently (and securely).

A while back, I wrote about my solution to this problem after having tried to solve it different ways.  In that post, I detailed my evolution from a Java application on a USB keychain to a website called Clipperz.

Well, I have been using Clipperz for almost 2 years now.  It is immensely useful and efficient.  I have had ZERO problems.  Yes, none, nada. NO problems whatsoever.  How many things can you say that about?

Clipperz does seem to be growing in popularity, since the last time I remembered the question asked on Slashdot, hardly anyone recommended Clipperz.  This time, a few people mentioned Clipperz on Slashdot.  However, its been 2 years and Clipperz still has the “beta” status.  Granted, Google Mail was in beta for years until they became “production”, but still…

Not just oh-we’ll-find-a-way-to-fix-or-deal-with-it bad, we’re talking a downward spiral that slopes deeper the further we decline until we have reached “terminal velocity”.  The bottom is not yet in sight.

Why?  Well, I’ll tell ya why…  in a minute.  First, I’ll put things into perspective by shedding a bright historical light on the subject.  This is not to mean that the history of email is dark or bad — but the present state of email certainly is, compared to its early days.

Email (not E-mail, since words that are introduced into the English language are often comprised of multiple words that stand on their own, separated by hyphens, normally lose their hyphens as the new words gain wider acceptance) as we know it today, was originally created in the early 1970′s, purely as an experiment, though in a slightly different form.  To put this into proper context, we’ll go back just a bit further. Email (at this time E-mail, or “electronic mail”) only existed in self-contained systems.  People would log into one specific machine (a time-sharing device, which was basically a big expensive computer that a group of people shared at different times) to perform their work, and would occasionally leave messages for one another to read whenever the next person would log in again.  This concept of “self contained” email would eventually evolve into other implementations of the same use — such as Microsoft Mail, which was designed as a central system, namely in an office building, that people would use to talk to only other people in the same office.  I digress…  but, even in its first use case, Email (and E-mail) was used as a convenience.  Some would say, a luxury tool — to save people from leaving yellow sticky notes somewhere, or picking up a phone to talk to someone that may not have time to talk to you.  In tech-geek-speak, email is asynchronous communication:  I can talk to you as much as I like, and you can reply back to me, but it is pure coincidence if we happen to talk to each other at the same time  (there is a variable delay between one person talking and the other person replying).

From being an easy way to leave messages for other people sharing the computer, it turned into a way of leaving messages for people using other computers — no longer “self contained” email, but networked email.  At this point, email diverged into two uses:  local “self contained” messaging on one computer, and “networked” messaging.  The two remained distinct for quite a while, as there were people sharing central computers that had very little need to communicate with people sharing other computers, yet there were people that had a valid need for such distant communication even if “distant” meant “the computer right next to mine in the same room”.  Still, it was viewed as leaving an electronic sticky note on the screen for whenever that person logged in again.  As such with StickyNotes, eventually the glue on the paper dries and at that point it no longer sticks to anything, falling off the surface to become lost when the cleaning lady vacuums the floor.  This was the expectation for early email — “Joe, I left you a quick message about the widget, if you have concerns just give me a call.”  If the email message was lost, deleted accidentally, or was never delivered, it was no big deal because the communication was eventually going to take place in person anyway, and there was no guarantee the intended person would ever read the message in the first place.

As the novelty of communicating with other people on other computers evolved, so did the implementation of email.  To send an email message to someone outside the shared computer, a person needed to know *which* other computer the recipient used.  The @ was born, since that seemed like the most logical delimeter to distinguish “user” from “computer”, and since neither could contain the @ symbol.  For similar computers, the method was “user@computer”, to properly address an email message.  For different systems, it wasn’t so clear.  In fact, it became downright complicated and confusing.  If a person needed to send a message to a distant computer, but the distant system could not accept “user@computer” (possibly it used the @ for something other than a delimeter), the sender of the message needed to know not only who to address the message to, and *which* computer that user used, the sender also needed to know the path the message would take when it was sent from computer to computer to computer.  UUCP (Unix-to-Unix-Copy) was born.  Imagine instead of smith@accounting it was  !cenvax!westnode!accounting!smith.  Gateways from one type of email system to another type had to be erected, to handle the messages and translate one address into another.  Yet, even then, email was still viewed as “fire and forget” in the sense that whenever the recipient got the message, IF they got the message, they will eventually acknowledge by replying in some fashion as courtesy.

Back in my early days of email, I worked in the military in the computer support office.  Then, email was more a novelty than a necessity.  I vividly remember a sergeant I worked with would get daily phone calls after creating a new email account for someone.  Someone would normally call him up to complain “its been 3 days since you created my email account, and I haven’t got any email yet.  I think its broken.”  He would always reply with the same thing:  “you have to send email to get email”, which basically was his smartypants way of saying “it isn’t broken because you didn’t get anything.  You probably didn’t get any email because no one knows you have an email address, or they have nothing to say to you, or all the people you want to talk to don’t have email themselves.”   He would hang up the phone and we would have a chuckle, then I would joke about how the first person in the world with a fax machine probably wondered why he invested so much money in a device that strangely never prints out any faxes.

Slowly, email became the “killer app”.

[For the uninitiated, a "killer app" is an application (a program or function) that is just so utterly cool and awesome it is NEEDED so much that the purchase of an expensive device is justified, simply to use the application.  The other programs and software are bonus, and not needed as much, compared to THE reason the computer was purchased.]

Everyone seemed fascinated with the ability to talk to ANYONE (as long as they were “on email too”) for FREE.  Its better than long distance calling!!  No more busy signals or answering machines!  And its FREE!!

“Move over word processor, I’m going to communicate with the world!!  Shrink yourself into a microscopic icon, Mr. Spreadsheet, EMAIL is the real reason I have a computer!  Now, if only I knew what to say, and who to talk to.  Maybe someone will figure out how to contact me, so we can send messages back and forth.”

Today, no one really needs to know the path a message takes to reach its intended recipient (in some instances, even the recipient need not be known) because we address email to “user@something.somethingelse.com” and we trust the system to do the Right Thing to deliver the message.  To the right person.  At the right time.  “When it absolutely positively needs to be there…”  within the next 15 seconds else I’m going to wonder what the HELL is taking so long, and why haven’t they replied yet because I just got a message that says they’ve read it and it better not have been marked as spam because it wasn’t spam!!

Email has become the primary method of daily communication.  No longer do you “need to send mail to get mail”.  If your email address is on a web page, business card, or if you have ever used your email address to log into a website, YOU’VE GOT MAIL.   Whether you want it or not.  We email each other about meetings, to talk about email.  We email appointments, contact information, political opinions, love letters, chain-messages, advertisements.  The type of content goes on and on.  The problem is no longer about how we communicate with the right person on the right computer, but how to silence the noise to get to the legitimate messages that we need to read.

In the past, whether it was “self contained” or sent from the other side of the continent, each message was read and discarded soon thereafter.  Lately, email is received and almost immediately archived for “safe-keeping”, sometimes without it even being read.   It seems the focus now is not the immediate meaning of each message, but that a potential need might arise in the future where we might need to re-read the message.  Email used to consist of one file, appended to whenever new messages arrived — older mail was at the top of the file and newer mail was at the bottom/end.  Email now has folders, sorting, searching, tagging, categorizing, filtering, and archiving of all types.  We rarely, if ever, delete email that we’ve read.  Sure, it was really nifty when Google unleashed GMail to the world with its “2GB and growing” size limit on the amount of email one person could have, but if we’re only talking about purely text-based messages it amounts to billions of messages. (By the way, it is no longer only 2GB — its more like 7 or 8GB now.)

Email is no longer just the “killer app” in the sense of being able to communicate with anyone.  It is a presentation moniker; an address with @gmail.com is not as prestigious as it once was, but an address with @yourreallastname.com is.  It is a storage mechanism; people have figured out a way to use free online web email accounts to store documents, MP3s, and photos.  It is a calendar; if you’re using a particular email system that is tied into a shared calendar, you can send/receive appointments, and reminders of upcoming events.  It is a ToDo list; some people have an email folder with messages they have sent to themselves containing the errands they need to perform in the course of a day.  It is a webpage; modern email software will accept HTML in the body of an email message and interpret the language of webpages, even in the sense that images need not be attachments to the email but can be referenced to elsewhere on the Internet.  It is submissible legal evidence; there is legal precedence where email messages are a form of evidence, able to be subpoenaed by a court of law.

How did we get this way?  What changed so radically that “e-mail” could come from an experiment on the ARPANET (a solution looking for a problem), to “email”, a common term of the layman’s vernacular so much that it is no longer a privilege but a rite?   How could a function of computer networking change the way we communicate, yet itself change so little?

How is it that email is no longer a novelty method of asynchronous communication, but is now a basic human necessity in the modern world, measured not in its content of communication, but in cosmetic appeal of its address and in its storage size limit?

I haven’t even got to the bad part yet.

SMTP, or Simple Mail Transfer Protocol, was basically an afterthought in the broad historical map of the creation of the experimental networks that were the grandfathers of the Internet we know today.  SMTP is the most widely accepted and “standardized” method of sending and receiving email.  It was essentially created to bridge the gap between unlike electronic messaging systems, back when “e-mail” was growing in popularity and usefulness.  The unfortunate part of the story, though, is that SMTP was created back when there was no real malicious threat or intent proliferating through the networks.  Users basically trusted other users in the sense that everyone followed the same rules because that was what it meant to “be connected”.  After all, if you behaved badly on the network, people would want to network with you less, until eventually you would be partitioned from everyone else in such a way that you gain a decreasing benefit from being part of the network.  It was a self-governing system, yet relatively unofficial.  “Netiquette” dictated good form and respectable practices toward other network-citizens, which mainly consisted of college students and faculty among connected higher education organizations.  SMTP was very trusting back then, and still is.

To this day, anyone can still send email as anyone else — so easily that specialized software is of little concern.  Simply connecting to a mail server with a bare terminal (Telnet), typing the correct sequence of commands and syntax, and voila!  You just forged an email message.  If you’re lucky, someone will believe they’re talking to whom you pretended to be.

What does all this mean, then?  Put together all what I’ve said so far, and it paints a rather dark and confusing portrait.  Email is *everything*, yet flimsy in it being unreliably verifiable.  Email messages pass from machine to machine across the room, or across the hemisphere, and yet they are “essential communications”.  They are submissible in a court of law, yet easily forged.  Messages are quickly and easily created and more easily deleted, yet we archive them for years or even decades with the possibility that we might need them later even though we already know what each message means, resulting in a liability if they are ever subpoenaed, and requiring constantly increasing storage.

How do we end this accelerating downward spiral, or at least slow it down so we might recognize and begin to approach the problem?

When will added functionality, storage space, and guarantees of quality be enough for this old and simple luxury of slow and insecure communication?  When will we finally realize that we have already outlived email’s usefulness and begin using the next electronic communication “killer app”?

If anyone reading this knows the answer to any of the above, drop me an email. 

[3 Oct 2009 Edit: I JUST found out about Google Wave!!  Go here, here, or here to learn more about it.  It is in closed invitation beta right now, but I hear its going to be released this year.]

But, Dave…  There are well over 800 Linux distributions out there already.  What makes you think you can compete?

Competing is not the point.  The point in mountain climbing is not to see who can reach the top first, but to learn something about yourself, and generally “because its there”.  If my Linux distro gains momentum and ends up being something more than just a hobby/pasttime, great!  If it doesn’t, that’s fine too.  I’m not going to make any promises to anyone, at first, because this will be to used fulfill my own needs which don’t necessarily apply to anyone else.  If I later discover that other people have needs similar to mine, we’ll talk about where to go from there.

Basically, there are two approaches in creating your own Linux distribution:  building from scratch, and basing it on existing.  Some Linux distros are based on other Linux distros.  For example, Ubuntu was created out of re-building software packages from the Debian/GNU Linux distribution, but Debian was created from scratch by hundreds of volunteers.  Ubuntu itself has spawned derivative Linux distributions; Kubuntu, Xubuntu, Ubuntu Studio, and Mythbuntu to name a few.  Both methods have their advantages and disadvantages, and both are equally valid (and probably equally popular).

A few existing Linux distros come with their own “roll your own” application that will automagically create a customized ISO image that you can use to modify/install/spin/fold/mutilate to your heart’s content.  Yes, the method bases it on a specific Linux distro, but it will be relatively different depending on the customizations you’ve set in place.  For example, Fedora has Revisor.  There are also non-distro-specific utilities on the web such as Instalinux, as well as complete Linux distros based on customization and optimization in which everything is built from source — Gentoo.

The other method is to construct a Linux distribution completely from scratch, appropriately named Linux From Scratch (LFS).  While this approach isn’t entirely difficult initially — its just building binary files from source files — it is time-consuming.  It is also recursive, meaning that software you compile in the beginning stages are depended upon by other software you build later, and you’ll have to start over from the point of contention if you discover something broken.  This is termed “building the toolchain”.  Building software so that you can build other software with it, that software becomes dependent on the initial software build, which is then used to build even more software — creating a chain of software used to create a system, which will be bundled together to form an installation, and a Linux distribution.  That’s the easy part.  The hard part isn’t in building the software; its in maintaining the whole thing such that when a bug is discovered it is easily and quickly patched/fixed and the remaining components of the toolchain remain relatively unaffected or are automatically rebuilt using the new link in the toolchain.

Whatever method used to build the Linux distribution, there still needs to be some point to the exercise, else time invested is wasted in making an exact copy of something that could have been simply downloaded to begin with.

My goals:

• Hybrid approach, mixing Linux From Scratch with useful tools from established distributions.
• A practical level of security in the system, without sacrificing usability.
• An agnostic mindset for software packages, trying not to rely on one set of software built for one distribution.  A best of breed path would be ideal, taking the best software from well known distributions, and possibly introducing software that is not available in any Linux distro.
• Initially, aim at the desktop and see how that goes before configuring things for a server platform.
• Simple for the user, but without the cost of being difficult or complicated for the system administrator (who is often the user also).

I’ve had quite a few discussions with friends about what I should name it. After all, that’s the most important aspect of a Linux distribution, right?  Amazingly, in the discussions about naming my distro, no one ever asked me what I wanted it to do — but everyone has suggestions for a name.  I guess in the age of Google, and Yahoo, the name doesn’t have to be related to what it does.  I have a pretty good idea what I want to name it, but that may change later.  Considering there are over 600 distributions of Linux in circulation, a few hundred discontinued, and a few hundred more about to be announced (some of which might be discontinued in a few months), the name doesn’t matter that much to me as long as it isn’t completely ridiculous.  Douchebag Linux doesn’t smack of “Download me!  Use me!  I’m useful!”  On the other hand, Master of All Linux sounds good, but is probably just a tad too ambitious.

I wish there was a quick way to check if a name is already taken.  Like a global registry, or something that I could search for to determine if a name is used.  Distrowatch.com is good, but there is a 90 day waiting period for a Linux distro to be added to the list because “this is to discourage submission of new projects that start with great enthusiasm only to vanish in a few short months”.  Like I said…  Easy to start, difficult to maintain.  Incidentally, ParanoidLinux is on the waiting list to be added to Distrowatch.  Its not mine, but I get quite a few people that read this blog after searching for that particular distro.  I wonder why…

I won’t explain the entire following video, as much of it is obvious, except to say that Isao Machii has been referred to as “Modern Samurai”, and he demonstrates superior swordsman skill and flexibility in precision.  An egg, a mushroom, a green bean, and a BB fired from a gun are all sliced by the sword.  Near the end, he is challenged to slice an iron rod, and then challenged again to slice an iron plate.

I will not comment on the sensationalism or grandiose publicity on the clip below.  However, I marvel at his ability. His speed is quicker than the eye, as even the technology to capture his skill is stressed.

[youtube]b_gcPGI-ZMI[/youtube]

In my original post, I noted that it would take some effort to change the way I use my desktop.  BumpTop, while very well thought out, and a genuinely original idea both in solving a problem and implementation of the solution, doesn’t fit well with how I personally use my computer.  This is not BumpTop’s fault.

For decades I have been using computers with different OSes, desktop orientations, window managers, looks and feels, constantly searching for “Desktop Nirvana”, which is to say the desktop state that mixes function with style and elegance while taking into account an optimal amount of practicality.  In other words, I’m all for eye-candy, as long as the eye-candy serves a purpose, even if that purpose is simply a “cool” factor.  As with all “cool” factors, it comes at a price, whether it be slower screen refresh rates like a 3D desktop [see Compiz], or more CPU resources consumed [see Google Desktop], or possibly even reservation of a portion of desktop real estate [see Rainmeter & Enigma].  BumpTop, on the other hand, seemed to not be so much eye-candy, but actually be more functional than cosmetic/stylish.

The problem BumpTop attempts to solve is that of desktop organization.  The analogy is a physical desk with papers and magazines sitting on a desk, in piles, with each item spacially relative to other items on the desk.  This is a brilliant way to see the problem, actually — arrange things spacially however the user of the items uses them, rather than “in a grid”, or “in a folder” as modern desktops coerce users to do.  For example, if I place things into a folder, and keep the folder on the desktop, yes I can organize the items, but I can’t SEE the items until I open the folder. However, if I organize things into a “pile” such as a physical pile on a shelf, I can SEE what items are in the pile without actually doing much at all, and I know which items are in which piles, whereas if they were in folders I would have to open each folder until I found what I was looking for.

This is all great in concept, and in implementation, and is precisely where BumpTop shines.

But, what if you’re like me, and have been organizing your desktop the same way for years?  Ever since Windows 3.0, I have been organizing my desktop to reduce (or eliminate) clutter, more or less in the same manner.  Along comes BumpTop, blazing the way in desktop organization, and I have almost nothing on my desktop to organize.

So what do I do?  I start putting things on the desktop!

I quickly found that while BumpTop solves the problem, I didn’t have the problem it solves.  This became quite annoying for me, partially because I had to adjust my way of doing things on the computer, but also partially because BumpTop is not a desktop — its an application.  BumpTop does not *replace* the desktop, but sits on top of your normal desktop, and hides the desktop that the operating system (Windows) provides.  The desktop that comes with the OS is optimized and made to mesh with the rest of the operating system.  BumpTop is a 3rd party application that acts as a go-between, wedged in the middle of the original desktop and YOU.

While the concept of organizing things on the desktop into piles, groups, and the like, is a sound concept that is proved through its use, the fact is that it is still “just another application” that has to interact with the desktop/operating system.  This makes it slower than your original desktop, and vulnerable to crashes, bugs, and other nuiances like computer resource reservation, installed applications are vulnerable to.  In other words, BumpTop solves a particular problem, but the solution comes at a cost.  The cost is running another application all the time that hides your original desktop — It does not make your original desktop go away; it only hides it and helps you organize things in a more “human” way.  This makes the application slower than if it was the desktop itself, which would elminate the orginal icon-laden desktop altogether.

Interaction with BumpTop was much less responsive than with the original desktop.  Keeping in mind that I only tested the beta, and things are bound to get better overall, BumpTop will still remain an application and not a full desktop replacement that completely eliminates the original unorganized workspace.

…an application that does not solve a problem for someone like me.

Since I have mentioned it, I’ve been using Enso, which almost completely eliminates the need to have icons on the desktop at all.  After a few keystrokes, I’m using whatever software installed on my computer, whether I have a desktop icon for it or not.  It has become second nature in the way I interact with my computer, so much that I have found (don’t laugh) that when Enso isn’t running, for a few seconds I enter a state of panic because I have forgotten how to launch the application software I wanted to use.  Caps Lock, O, “Open:  “  Fire — “Firefox? Mozilla Firefox?  Windows Firewall?  OpenOffice.org Impress?”  Amazingly, “Firefox” is usually at the top of the list, because Enso has remembered the last time I said “Fire”, Firefox is what I wanted.   If I want to run Photoshop, I just Caps Lock, type O, then type “photo”, and I see a list containing: “Adobe Photoshop CS4?, Adobe Photoshop Lightroom 2.2?, Photosynth?, Windows Photo Gallery?, etc”.

When Enso is not running (because it crashed, or hasn’t fully launched yet), I do…
Caps Lock…   Caps Lock light is now on.
Uh oh.
Ok.  Firefox is on the “Quick Launch” bar, so I’ll just click that.
I need to run Photoshop, now.  Caps Lock…  Caps Lock light is now off.
Uh oh.  Umm….   ok, I think its in the “Start Menu”…   Start ->  Adobe….  something…  oh, there it is.  Adobe Photoshop CS4.  Got it.  Luckily.

For people like me, who have pretty much always organized their desktop, BumpTop, albeit a wonderfully thought out piece of software, is probably not for you.

For people like me, and even people not like me, I highly recommend Enso.  BumpTop and Enso are not competing products, and solve completely different problems, but they do both help with the human-to-computer interaction immensely.

I have it installed, and will be trying it out as much as I can.  Ironically, I don’t normally have very much on my computer desktop, because of the problem BumpTop attempts to solve, but I’ll try to get out of the habit of keeping the desktop clean so that I can think of things in a new BumpTop way.

…my actual physical desktop is usually pretty cluttered, though.
I wish there was something that would let me keep papers, magazines, and bills as organized as in BumpTop.

Things don’t work in the beta exactly like the video below, but the concepts are the same.  In addition to what you see in the video, there are “photo frames” on the walls that can be slideshows of photos on my computer, or elsewhere on the web.

[flash http://www.youtube.com/v/M0ODskdEPnQ]