I was one of THE first people to ever have a public GMail account, in 2004 (before the “GMail invitation market” was created in which people would buy and sell invitations to the service).  Since then, whenever there is a new Google service, I typically get invites.  I was among the first in my group of friends to get a Google Wave account, and I got a Google Music Beta invite before many of my friends.  Somewhat related, Google has approached me with employment more than once, over the years.  Somewhat even less related, I own 2 Android phones, and a Barnes & Noble Nook Color that I’ve rooted with Android 2.3.  (Have I made my Google resume’ clear yet?)

What I don’t understand is why I keep hearing about people using Google+, but I have not yet got an invite.  It isn’t like Google doesn’t know about me — granted, they know about everyone, but I would think they know about me more than the average Joe.

I signed up for an invitation on the very first day I heard about Google+ on their website, yet the status on that page remains the same:

Google+ is in limited Field Trial
Right now, we’re testing with a small number of people, but it won’t be long before the Google+ project is ready for everyone. Leave us your email address and we’ll make sure you’re the first to know when we’re ready to invite more people.

I’m not buying it. GMail was in “beta” for years. Five, to be precise.  Google Wave was in beta the entire time it was online (and still is, even though the entire Wave project has been scrapped).  Google Music is still in beta — officially named Music Beta by Google.

Google+, from my understanding, is that it is a social network service similar to Facebook.  Some have even called it “the Facebook Killer”.  As with any social service, success of the service is measured in how many people have subscribed to the service.  If I were Google, I would want as much exposure as possible…  by as many people that enjoy Google services as possible.  Why have a limited rollout of something so central to Google?  Why have a limited rollout of something, and then not invite your most loyal fans?  Is this their way of saying they don’t love me any more?

O, Google, why have you forsaken me?  I feel gypped.  I see you pushing Google+ heavily, redesigning GMail, Google Calendar to have a similar look and feel to Google+, and I can only feel like the 3rd cousin that has been forced to eat dinner in the garage because there are no more seats at the kids’ table.  While reading daily news (on Google Reader), I see tours, tutorials and helpful tips of how to use Google+, and can only think “wow, that seems really nice, and seems to solve what bothers me about Facebook… too bad I can’t try it out.”

What gives, Google?

I’ll say that again… Email is not a storage mechanism.

By this, I mean that the purpose of electronic mail is not to store important files, information, or future reference material. It was never intended for that purpose, and even in today’s standards it still falls short of that use.  Of course, there is GMail today.  Of course, there are GMail extensions (like gDisk and GMail Drive Shell Extension) that allow you to store your MP3 collection, photos, etc.  That is a good example of what I’m referring to.  I’ll explain…

GMail, as most of us are aware, is not a typical electronic mail system.  It does not operate within the paradigm of traditional email systems.  Google Mail’s primary interface is via the web page in which email messages are only sorted by “threads” (“conversations” in GMail-speak).  But, more on GMail later.  Back to the point…

Email began as a way for users of a time-share system to communicate with one another, coordinating within the same closed system.  Soon thereafter, it became a method of communicating with users of other time-share systems, yet with serious limitations — namely, the sender of a message was required to know the path in which the message took to get to the intended recipient.  Instead of having addresses such as

poe@deadpoets.org

there were “addresses” such as

localhost!nextdoor!nextnextdoor!poe@deadpoets.org

which meant that the message had to travel from localhost, to nextdoor, to nextnextdoor, to finally deadpoets.org in order to reach user ‘poe’.  The machines did this in an automated way, as long as the route specified was correct.  If one of the machines along the message route was offline, or otherwise not accepting incoming mail, the sending machine held the message for a certain period of time until either the message was accepted on the receiving host or the sending machine effectively gave up — at which point, the message was lost forever.

Eventually, the machines connected to the network grew in number and a machine’s knowledge of other networked machines needed to scale as well.  Email needed to change with the new networking methodology, which is why we have user@somesystem.com today.  The sender of a message needs to know only the address(es) of the recipient(s), the subject of the message, and the message itself.

Notice in all the above explaination it reads “the message”, and not “the file”?  There is a reason for that.

Consider this example:

Alice wants to send Bob some files.  The total size of the files is 9.5MB. The contents don’t matter for the purpose of this example, so lets just say the email contains a few large photo images, and a large document.  In order for Alice to send these files, in an email to Bob, she would need to first specify Bob’s email address as the intended recipient.  Next, she will likely describe the contents in a few words in the Subject: field of the message, “The stuff I wanted to give you”.  Then, she sets about attaching each file she wants to send to Bob.  Each of these files becomes encoded in a very long set of letters and numbers, completely unreadable by any human, and inserted into the email message “envelope” so that each email system that handles the message will be aware that it is a message with a Subject: and multiple files attached of differing size and type, such not to get the files intermingled among each other, nor this specific message’s contents intermingled with any other message that might be handled.  Next, Alice presses “Send”.  It takes a moment for her computer to actually send it because most email systems aren’t expecting (or designed) to handle messages of that size… but it gets sent.  The message is then copied into Alice’s “Sent Mail” mail store (sometimes called “outbox”).  Bob does not see this message right away — this is not file sharing, nor is it Instant Messaging (IM).  Alice’s message is received from Alice’s computer, is copied onto Alice’s email server, which then needs to determine which machine handles Bob’s email.  Once that is determined, the message is sent again — to the machine listed after the ‘@’ in Bob’s email address.  That receiving machine typically makes attempts to verify that it is a message coming from an actual person (like Alice, and not a Spam robot), is destined for a person that it handles mail for (like Bob, and not Boob), and that the size of the message is within the system’s constraints for reasonable handling (typically 10MB).  After the message is accepted, it is written to Bob’s email server (this is the 3rd copy of the message) for delivery handling.  Assuming that Bob has not forwarded his mail elsewhere (which would further the process of sending/copying the message again), the message is then stored in a holding area on the server’s hard drive, to await Bob’s email client.  Once Bob’s email client connects to the email server, the message is copied yet another (4th) time to Bob’s computer.  The message will reside on both Bob’s email server, Bob’s computer, and Alice’s email server, and Alice’s computer (in her “Sent Mail”, remember?) until either Alice or Bob delete their respective copies of the message.  For a single 10MB message, it has taken multiple computers copying, and costing a total of at least 40MB of storage space.  This is not taking into account various spam/anti-virus systems, which also typically store each message for a short time.  More importantly, this is also not taking into account that had the message been addressed to more than one person (say Bob and Charles), the message would be stored 6 times — server and user’s computer, for each user — which would amount to a total of 60MB for the sender and two recipients.

Email systems treat messages as such. Sure, each message is a file, but a message to be delivered to user1 cannot/should not/will not be considered the same message as to be delivered to user2, even though it has the same file attached to the message and may contain the exact same contents.  Electronic mail is designed this way for privacy; not file-sharing.

“But, Mr. Linux Ninja Geek…  storage is cheap!”

Yes.  Storage is cheap.  However, transmission is not.  It takes relatively a small amount of effort for your computer to generate data, or even say copy data from your camera, and store it onto your computer.  It is much more effort to transmit that same data across the Internet to another computer, and have it stored there indefinitely.  Enter email into that equation and the effort is mulitplied by each computer the message travels through to get to the final destination.

“Ok.  So, I shouldn’t send files attachments in email at all??”

That is not what we’re talking about.  We’re talking about storing email messages indefinitely.  Consider that information in a typical message has a given lifetime.  Normally, this lifetime is very short, on the order of days or even weeks, possibly even months.  After this time passes, is the information in the message of the same importance, or has it become much less important?

To demonstrate this, let us employ an analogy…

In the old days, before email, people wrote correspondance — stone tablets, papyrus, handwritten, typewritten.  The message itself was carried, by another human, to its intended recipient, and either read aloud or delivered into their hands.  Once the information within the message was received, what happened to the message itself?  In the case of stone tablets, it was likely destroyed — or made into some type of monolith, depending on what the actual message was.  In the case of papyrus, the message was read aloud, retrieved, and kept for futher use — this is why the message was stored on a scroll, because it contained more than a single message for more than a single recipient.  In the case of handwritten or typewritten correspondance, either the letter was kept in a folder in a file, or it was discarded sometime after the message was understood.

That last part, concerning handwritten/typewritten letters, is probably the closest analogy to email.  After the letter was filed away, what was its disposition?  More often than not, the letter sat in the file for a long time, until someone either tossed it out with the trash, or it was framed for historical purposes.  Point:  a letter was hardly ever kept “in case I need it again”.  The physical letter’s disposition was certain, upon the moment of receipt, similarly to stone tablets and papyrus scrolls.  Why?  Because physical objects need space to be stored indefinitely.  The more physical objects that need to be stored, the more space required, of course.

Hard drive space is required to store electronic mail messages as well.  In all cases of message storage, the information contained within does not change after delivery.

Enter GMail.  GMail’s claim to fame was that, initially, the storage amount was enormous, compared to other offerings like Hotmail.  Leveraging Google’s search abilities, supposedly you could instantaneously find any email you ever received in the GMail system.  This goal is in line with having conversational correspondance with other people connected to the Internet, only in a different way.  GMail does not sort messages by date, subject, or even sender of the message like typical email client software.  The only sorting mechanism available is by “thread”, which makes GMail seem more like Usenet, or an online discussion forum.  This design structure does not seem to lend itself for file storage at all — much less attachment storage.  Sure, you can save a message (or entire conversations) indefinitely, for later review.  How easy and practical is it to do that?  How important is that email from years ago?  More importantly, how many other email systems are similar to GMail?  It doesn’t seem that GMail is a good gauge as to what an email system can or cannot do, since it seems to be a consensus that GMail is different from the rest, and since GMail is due to fail without warning among other technical limitations.

Given that a message’s information/content/meaning does not change after it is delivered, why is email kept for so long?

Not just that, but if a message is noticed to be lost (presumably a while after it was actually lost), why is it so important to have the message restored?  What could possibly be contained in a message, that wasn’t noticed to be missing, that has become critical this very moment?  Could the information not be resent from the sender?

More to the point:  Why are people storing information in email?

This term was coined in the mid-’90s when two guys at Yale wrote that lifestreaming is:

…a time-ordered stream of documents that functions as a diary of your electronic life; every document you create and every document other people send you is stored in your lifestream. The tail of your stream contains documents from the past (starting with your electronic birth certificate). Moving away from the tail and toward the present, your stream contains more recent documents — papers in progress or new electronic mail; other documents (pictures, correspondence, bills, movies, voice mail, software) are stored in between. Moving beyond the present and into the future, the stream contains documents you will need: reminders, calendar items, to-do lists.[1]

The Internet is attempting to capture lifestreams of its citizens, but doesn’t appear as inclusive as most companies want it to be — mostly due to the nature of the events, or documents, or because no one web company owns the rites to all of one person’s information.  …This seems to be a good thing.

There are, however, quite a few aggregators on the web, like Facebook, Tumblr, Posterous, Collectedin, and Flavors.me.  These do a fairly good job of aggregating in a social network context, meaning that they typically lifestream content from social networking websites (silos) that people post specific content to.  For example, if a person posts a photo to Flickr (a silo), the photo is on that website of course, but can also be retrieved from sites like Facebook (an aggregator, and a silo).  I say that Facebook is an aggregator and a silo because Facebook started as a silo (only allowing posts from its subscribers), then branched into reading feeds from other sites like Flickr and Twitter, but Facebook itself is difficult to integrate into a secondary aggregator (like an RSS reader) as it has changed its layout and hidden its RSS feeds numerous times.

I particularly like the look and layout of Flavors.me, as it attempts to present information in a person’s lifestream, yet also segregates the data from different sources, which doesn’t intuitively give the audience a flowing context of the lifestream itself.  In other words, personal Flavors.me sites look very nice, but are still somewhat disjointed.

I decided to task myself in creating my own lifestream on a website, but it had to meet my own specifications while maintaining a particular look and feel.  I began the process by researching all the different types of information on the web that one could aggregate — from Facebook posts, to Flickr photos; from blog entries to music tracks played recently.  I came upon a very nice and specific piece of software named Sweetcron.

Sweetcron, created by Jon “Yongfook” Cockle, was designed to be blog software that could capture and display a person’s lifestream.  Similar to Tumblr and Friendfeed, but could be installed and run on a personal server, eliminating the need to subscribe to yet another commercial service.  Sadly, I also found that Sweetcron, as wonderful as it might be, is no longer maintained by the original developer.  This is where I discovered a Sweetcron fork — derivative software, named Lifepress.

Lifepress seems wonderful, as it has the functionality to aggregate feeds from different sources, and also comes with a bunch of plugins to handle the sources that aren’t as intuitive.  It is also quite them-able, though there aren’t many Lifepress themes to be found.  Luckily, Sweetcron themes can be easily adapted to Lifepress, though there aren’t as many Sweetcron themes, either, compared to regular blogging software like WordPress.

As with most lifestream sites, they contain things like Flickr photos, blog posts, etc. that you can read/view in the lifestream itself, I decided that mine would only be an aggregator rather than a full-blown lifestream blog that contained comments, etc.  With mine, only linked posts from other sites can be read at the source if the link is followed.  This allows me ease of maintenance in that I don’t have to worry about comments on a blog post and comments on a aggregated blog post at the same time.  Obviously, this also prevents me from posting once and having software disperse the data — such as when people post to Twitter, and it magically appears on Facebook also.  If I were to do that with a lifestream, it seems it would likely be caught in an endless loop…  blog -> lifestream -> blog -> lifestream, ad infinitum.

So, after a few hours of installing and modifying PHP scripts and CSS stylesheets…even had to edit an image or two…

I have my own lifestream website.  I only have a few feeds added in, but I’m sure there will probably be more as I toy around with things.

[1] http://cs-www.cs.yale.edu/homes/freeman/lifestreams.html

For those not in the know, cracking passwords has been going on for decades.  Common rationale is that if the “good guys” can figure out your password before the “bad guys” do, then the “good guys” can inform you so that you can change your password before the “bad guys” do something horrible in your name.  That’s the thinking, anyway, though many systems administrators and security professionals don’t regularly attempt to crack passwords of their users.  Why?  Too much effort for little reward.  They figure “if I have to go through all this trouble to get a password, then there aren’t many people out there that will do the same”.

That is when the level of effort involved outweighs the risk involved.

Good guys’ view:  If the amount of effort to protect something outweighs the risk, it slowly becomes an acceptable risk.  In other words, if you have to put up barriers, maintain the barriers, and remind people to not tear down the barriers, all to protect one person’s Social Security Number, you are less apt to do so because that SSN is probably discoverable by other means.  If the level of risk is greater than the amount of effort to protect it, then new security measures are put into place.  The trick, here, is determining the level of effort.

Bad guy’s view:  If the target is greater than the level of effort to obtain the target, attempt to obtain the target as long as this holds true.  As time passes, the level of effort increases, or the value of the target decreases.  Specifically, if the target is a password of an account at a bank, the amount of effort to obtain the password is about the same as obtaining a password from just about anywhere else.  The value of the bank password is great, however, as time passes the value goes down because the owner of the password might change the password, and the game begins again from the beginning.

Game changer:  GPU.  Graphics Processing Units.  The difference between a CPU and a GPU is enormous.  We’ve talked about this before.  If you missed it, a very good visual conception of the difference between CPU and GPU is here: [flash http://www.youtube.com/v/fKK933KK6Gg]

Enter password cracking.  Historically, it was done on CPUs.

Password guessing applications attempt guess after guess.  1AAAAA, 2AAAAA, 3AAAAA  in what we call “brute force” attempts.  Another approach is to attempt guesses with the use of a dictionary.  Apple, Aardvark, AppleArdvark, ArdvarkApple, and so on until all the words in the dictionary were spent.  Keep in mind, here, that each change in letter represents a completely new attempt at the password.  A group of fast CPUs can crack a “weak” password in just under a few hours.  One CPU can obtain an obvious password in just a few minutes — but then, if it were obvious, you wouldn’t need a CPU to obtain it.  A good, or strong, password can take weeks or even months — if it is able at all.  The historical thinking was that the more CPUs you dedicated to the task, the faster you could crack the password, depending on how strong of a password and the type of algorithm was used to protect it.

It has recently become much easier.  For both the good guys and the bad guys.  Amazon EC2 is able to leverage GPUs very cheaply.  The level of effort to obtain passwords has hit rock bottom.  Your passwords need to be much stronger now.

Follow the link to learn how it was possible to crack multiple (weak) SHA1 passwords in under an hour using Amazon EC2 with CUDA. (For the cost of a few bucks and a few hours preparation.)  As a comparison, it would typically take an average desktop at least a day to accomplish similar results.

Anyone that puts an important file with an encrypted password on more than a dozen computers, with permissions of the file being world-readable, doesn’t really understand the ramifications involved.

Since then, I have been trying to “crack” the password through regular means.  Given their usage pattern, I’m fairly certain the password is 8 characters long, at least one capital letter, one number, and one special character.  Sadly, only that much information probably won’t be enough for me to crack it without throwing more compute power at it.  The tough part is that its a salted hash, so I can’t really use Rainbow Tables, and I’ve already tried dictionary attacks with Webster’s Dictionary.  So, I’ve resorted to brute-force automated guessing.  JTR seems pretty good at this, but even so it will probably take months to obtain the plaintext password.  It has been running for 11+ days so far.

The encrypted password in question is: {SSHA}KZhA0wzX4AThn9CkxBgYDmmy42pNY9ME

Salted SHA-1, of course.  If you know encryption algorithms, you already know what this is likely used for, but I won’t give that away.  I won’t tell you what its used for, or who it belongs to, or what you might do with it once you’ve cracked it, suffice to say the plaintext password in the wrong hands could cause some damage.

If you know of a quick way to crack such a password, other than what I’ve tried so far, drop me a line.

Not just oh-we’ll-find-a-way-to-fix-or-deal-with-it bad, we’re talking a downward spiral that slopes deeper the further we decline until we have reached “terminal velocity”.  The bottom is not yet in sight.

Why?  Well, I’ll tell ya why…  in a minute.  First, I’ll put things into perspective by shedding a bright historical light on the subject.  This is not to mean that the history of email is dark or bad — but the present state of email certainly is, compared to its early days.

Email (not E-mail, since words that are introduced into the English language are often comprised of multiple words that stand on their own, separated by hyphens, normally lose their hyphens as the new words gain wider acceptance) as we know it today, was originally created in the early 1970′s, purely as an experiment, though in a slightly different form.  To put this into proper context, we’ll go back just a bit further. Email (at this time E-mail, or “electronic mail”) only existed in self-contained systems.  People would log into one specific machine (a time-sharing device, which was basically a big expensive computer that a group of people shared at different times) to perform their work, and would occasionally leave messages for one another to read whenever the next person would log in again.  This concept of “self contained” email would eventually evolve into other implementations of the same use — such as Microsoft Mail, which was designed as a central system, namely in an office building, that people would use to talk to only other people in the same office.  I digress…  but, even in its first use case, Email (and E-mail) was used as a convenience.  Some would say, a luxury tool — to save people from leaving yellow sticky notes somewhere, or picking up a phone to talk to someone that may not have time to talk to you.  In tech-geek-speak, email is asynchronous communication:  I can talk to you as much as I like, and you can reply back to me, but it is pure coincidence if we happen to talk to each other at the same time  (there is a variable delay between one person talking and the other person replying).

From being an easy way to leave messages for other people sharing the computer, it turned into a way of leaving messages for people using other computers — no longer “self contained” email, but networked email.  At this point, email diverged into two uses:  local “self contained” messaging on one computer, and “networked” messaging.  The two remained distinct for quite a while, as there were people sharing central computers that had very little need to communicate with people sharing other computers, yet there were people that had a valid need for such distant communication even if “distant” meant “the computer right next to mine in the same room”.  Still, it was viewed as leaving an electronic sticky note on the screen for whenever that person logged in again.  As such with StickyNotes, eventually the glue on the paper dries and at that point it no longer sticks to anything, falling off the surface to become lost when the cleaning lady vacuums the floor.  This was the expectation for early email — “Joe, I left you a quick message about the widget, if you have concerns just give me a call.”  If the email message was lost, deleted accidentally, or was never delivered, it was no big deal because the communication was eventually going to take place in person anyway, and there was no guarantee the intended person would ever read the message in the first place.

As the novelty of communicating with other people on other computers evolved, so did the implementation of email.  To send an email message to someone outside the shared computer, a person needed to know *which* other computer the recipient used.  The @ was born, since that seemed like the most logical delimeter to distinguish “user” from “computer”, and since neither could contain the @ symbol.  For similar computers, the method was “user@computer”, to properly address an email message.  For different systems, it wasn’t so clear.  In fact, it became downright complicated and confusing.  If a person needed to send a message to a distant computer, but the distant system could not accept “user@computer” (possibly it used the @ for something other than a delimeter), the sender of the message needed to know not only who to address the message to, and *which* computer that user used, the sender also needed to know the path the message would take when it was sent from computer to computer to computer.  UUCP (Unix-to-Unix-Copy) was born.  Imagine instead of smith@accounting it was  !cenvax!westnode!accounting!smith.  Gateways from one type of email system to another type had to be erected, to handle the messages and translate one address into another.  Yet, even then, email was still viewed as “fire and forget” in the sense that whenever the recipient got the message, IF they got the message, they will eventually acknowledge by replying in some fashion as courtesy.

Back in my early days of email, I worked in the military in the computer support office.  Then, email was more a novelty than a necessity.  I vividly remember a sergeant I worked with would get daily phone calls after creating a new email account for someone.  Someone would normally call him up to complain “its been 3 days since you created my email account, and I haven’t got any email yet.  I think its broken.”  He would always reply with the same thing:  “you have to send email to get email”, which basically was his smartypants way of saying “it isn’t broken because you didn’t get anything.  You probably didn’t get any email because no one knows you have an email address, or they have nothing to say to you, or all the people you want to talk to don’t have email themselves.”   He would hang up the phone and we would have a chuckle, then I would joke about how the first person in the world with a fax machine probably wondered why he invested so much money in a device that strangely never prints out any faxes.

Slowly, email became the “killer app”.

[For the uninitiated, a "killer app" is an application (a program or function) that is just so utterly cool and awesome it is NEEDED so much that the purchase of an expensive device is justified, simply to use the application.  The other programs and software are bonus, and not needed as much, compared to THE reason the computer was purchased.]

Everyone seemed fascinated with the ability to talk to ANYONE (as long as they were “on email too”) for FREE.  Its better than long distance calling!!  No more busy signals or answering machines!  And its FREE!!

“Move over word processor, I’m going to communicate with the world!!  Shrink yourself into a microscopic icon, Mr. Spreadsheet, EMAIL is the real reason I have a computer!  Now, if only I knew what to say, and who to talk to.  Maybe someone will figure out how to contact me, so we can send messages back and forth.”

Today, no one really needs to know the path a message takes to reach its intended recipient (in some instances, even the recipient need not be known) because we address email to “user@something.somethingelse.com” and we trust the system to do the Right Thing to deliver the message.  To the right person.  At the right time.  “When it absolutely positively needs to be there…”  within the next 15 seconds else I’m going to wonder what the HELL is taking so long, and why haven’t they replied yet because I just got a message that says they’ve read it and it better not have been marked as spam because it wasn’t spam!!

Email has become the primary method of daily communication.  No longer do you “need to send mail to get mail”.  If your email address is on a web page, business card, or if you have ever used your email address to log into a website, YOU’VE GOT MAIL.   Whether you want it or not.  We email each other about meetings, to talk about email.  We email appointments, contact information, political opinions, love letters, chain-messages, advertisements.  The type of content goes on and on.  The problem is no longer about how we communicate with the right person on the right computer, but how to silence the noise to get to the legitimate messages that we need to read.

In the past, whether it was “self contained” or sent from the other side of the continent, each message was read and discarded soon thereafter.  Lately, email is received and almost immediately archived for “safe-keeping”, sometimes without it even being read.   It seems the focus now is not the immediate meaning of each message, but that a potential need might arise in the future where we might need to re-read the message.  Email used to consist of one file, appended to whenever new messages arrived — older mail was at the top of the file and newer mail was at the bottom/end.  Email now has folders, sorting, searching, tagging, categorizing, filtering, and archiving of all types.  We rarely, if ever, delete email that we’ve read.  Sure, it was really nifty when Google unleashed GMail to the world with its “2GB and growing” size limit on the amount of email one person could have, but if we’re only talking about purely text-based messages it amounts to billions of messages. (By the way, it is no longer only 2GB — its more like 7 or 8GB now.)

Email is no longer just the “killer app” in the sense of being able to communicate with anyone.  It is a presentation moniker; an address with @gmail.com is not as prestigious as it once was, but an address with @yourreallastname.com is.  It is a storage mechanism; people have figured out a way to use free online web email accounts to store documents, MP3s, and photos.  It is a calendar; if you’re using a particular email system that is tied into a shared calendar, you can send/receive appointments, and reminders of upcoming events.  It is a ToDo list; some people have an email folder with messages they have sent to themselves containing the errands they need to perform in the course of a day.  It is a webpage; modern email software will accept HTML in the body of an email message and interpret the language of webpages, even in the sense that images need not be attachments to the email but can be referenced to elsewhere on the Internet.  It is submissible legal evidence; there is legal precedence where email messages are a form of evidence, able to be subpoenaed by a court of law.

How did we get this way?  What changed so radically that “e-mail” could come from an experiment on the ARPANET (a solution looking for a problem), to “email”, a common term of the layman’s vernacular so much that it is no longer a privilege but a rite?   How could a function of computer networking change the way we communicate, yet itself change so little?

How is it that email is no longer a novelty method of asynchronous communication, but is now a basic human necessity in the modern world, measured not in its content of communication, but in cosmetic appeal of its address and in its storage size limit?

I haven’t even got to the bad part yet.

SMTP, or Simple Mail Transfer Protocol, was basically an afterthought in the broad historical map of the creation of the experimental networks that were the grandfathers of the Internet we know today.  SMTP is the most widely accepted and “standardized” method of sending and receiving email.  It was essentially created to bridge the gap between unlike electronic messaging systems, back when “e-mail” was growing in popularity and usefulness.  The unfortunate part of the story, though, is that SMTP was created back when there was no real malicious threat or intent proliferating through the networks.  Users basically trusted other users in the sense that everyone followed the same rules because that was what it meant to “be connected”.  After all, if you behaved badly on the network, people would want to network with you less, until eventually you would be partitioned from everyone else in such a way that you gain a decreasing benefit from being part of the network.  It was a self-governing system, yet relatively unofficial.  “Netiquette” dictated good form and respectable practices toward other network-citizens, which mainly consisted of college students and faculty among connected higher education organizations.  SMTP was very trusting back then, and still is.

To this day, anyone can still send email as anyone else — so easily that specialized software is of little concern.  Simply connecting to a mail server with a bare terminal (Telnet), typing the correct sequence of commands and syntax, and voila!  You just forged an email message.  If you’re lucky, someone will believe they’re talking to whom you pretended to be.

What does all this mean, then?  Put together all what I’ve said so far, and it paints a rather dark and confusing portrait.  Email is *everything*, yet flimsy in it being unreliably verifiable.  Email messages pass from machine to machine across the room, or across the hemisphere, and yet they are “essential communications”.  They are submissible in a court of law, yet easily forged.  Messages are quickly and easily created and more easily deleted, yet we archive them for years or even decades with the possibility that we might need them later even though we already know what each message means, resulting in a liability if they are ever subpoenaed, and requiring constantly increasing storage.

How do we end this accelerating downward spiral, or at least slow it down so we might recognize and begin to approach the problem?

When will added functionality, storage space, and guarantees of quality be enough for this old and simple luxury of slow and insecure communication?  When will we finally realize that we have already outlived email’s usefulness and begin using the next electronic communication “killer app”?

If anyone reading this knows the answer to any of the above, drop me an email. 

[3 Oct 2009 Edit: I JUST found out about Google Wave!!  Go here, here, or here to learn more about it.  It is in closed invitation beta right now, but I hear its going to be released this year.]

Anyway…

I’m actually still in the initial stages, in the overall context of the project.  Basically, to create a Linux distribution (from scratch, as I am doing), it begins similar to a chicken-and-egg problem.  You have to use an existing system to create building blocks, and use the building blocks to create a complete toolset.  Then, you use the toolset in a limited environment, which makes zero assumptions (a “chroot” environment, for geeks out there), to begin construction of the target system.

While doing this is actually pretty straightforward, usually, the difficulty and complexity enters the stage when thoughts about the future begin to crop up.  Issues such as future maintenance and automation complicate matters because while there might be a working toolset to construct a target system, no one wants to have to manually create the target system each time something changes.  Also, if something changes within the toolset itself, the component will need to be rebuilt, or worse, the entire toolset will require rebuilding such that the whole thing remains cohesive.

This is similar to around the time when man first thought about hunting and killing for food.  He realized he could fashion a tool to make a weapon.  Next, he used the tool to create the weapon, which was then used to hunt and kill for food.  As evolution continued, he created more efficient tools, to in turn create better weapons that would enable him to hunt and kill more exotic food.  The same general principle exists when creating a Linux distribution — though, we’re not depending on the need to eat in our software toolchain, we are depending on the ability to make more software available, which will comprise a Linux distro.  The tricky part is to know how to manufacture tools in such a way that when a component in the process breaks, it is trivial to fix it so much to the point that if a tool very deep in the toolset breaks there is less need to re-manufacture everything that was built with it.  Henry Ford had a very good idea, with small replaceable parts.

At this point, I’m at the point that I have the toolchain built.  Before I go much further, I need to figure out a sensible way to maintain this toolchain so that if/when something becomes stale or breaks, I will spend less time repairing the toolchain and more time tending to the actual Linux software in the distribution.  Also, since the toolchain itself was build manually and painstakingly, I also need a system to churn out software using the toolset I have.  Perhaps a framework/system could be used for both maintaining the toolset, and maintaining the distro?

I’m quite sure this is not entirely a brand new problem — major Linux distros would have had to flesh this out very early on.  The trouble is, though, whatever they’re using to handle this type of framework, they’re not very up front in saying exactly what they use, what it does, or where it came from.

I know what I want my system to do.  I have a general idea how it should fit all together.  My dilemma is that I’m splitting my time between searching the Internet for any existing software to satisfy the need, and planning how I might create my own toolchain/distro maintenance framework/system.  There are, actually, existing systems that do something along these lines, but they were designed for very specific purposes, and implemented with specific needs in mind, which resulted in assumptions that require specific platforms.  Not very useful in my case, because I’m creating from scratch with no assumptions.

Eventually, I will have an automated system that will:

• institute modularity wherever practical
• automatically rebuild (with dependent software if necessary) anything that needs updating/fixing
• handle both the toolchain itself, and the target Linux software

The site, now released to the public, is Suse Studio.  The concept behind the service is that you can remotely create, per your preferences, a Linux distribution, based on a recent Suse flavor — basically, either OpenSUSE or SUSE Enterprise Linux.

How does it work?  You login (with OpenID, or after creating an account), select a base to begin with;  GNOME, KDE, generic X, text only, etc.  You can select 32-bit or 64-bit.  Then, you can select software to be installed down to the specific package.  You can even upload your own packages — rpm, .tar, .tar.gz, .tgz, .tar.bz2, .tbz, or .zip, specify particular software repositories, or choose from over 300 pre-existing software repositories (some with rarely found packages, such as ATI or nVIDIA drivers).  After selecting software to be installed, you are brought to a configuration webpage that allows you to completely configure the system prior to building it, with the added option of uploading your own customized files.  After building the system, it can be tested via “test drive” — completely able to log into the newly built system via the website.

Once the system has been built (and optionally tested), it can be downloaded and installed as either:  Disk image, LiveCD (ISO image), Xen virtual machine, or VMware appliance.

The service is intelligently laid out, and quite intuitive, enabling the user to create systems within minutes.  Suse Studio routinely clears out built systems, but keeps the meta-info such that systems specified previously can be re-built easily and quickly.

It really seems to be a well thought out service.  I have experienced other web services that allow the user to build a Linux distribution, but never anything this detailed.

But, Dave…  There are well over 800 Linux distributions out there already.  What makes you think you can compete?

Competing is not the point.  The point in mountain climbing is not to see who can reach the top first, but to learn something about yourself, and generally “because its there”.  If my Linux distro gains momentum and ends up being something more than just a hobby/pasttime, great!  If it doesn’t, that’s fine too.  I’m not going to make any promises to anyone, at first, because this will be to used fulfill my own needs which don’t necessarily apply to anyone else.  If I later discover that other people have needs similar to mine, we’ll talk about where to go from there.

Basically, there are two approaches in creating your own Linux distribution:  building from scratch, and basing it on existing.  Some Linux distros are based on other Linux distros.  For example, Ubuntu was created out of re-building software packages from the Debian/GNU Linux distribution, but Debian was created from scratch by hundreds of volunteers.  Ubuntu itself has spawned derivative Linux distributions; Kubuntu, Xubuntu, Ubuntu Studio, and Mythbuntu to name a few.  Both methods have their advantages and disadvantages, and both are equally valid (and probably equally popular).

A few existing Linux distros come with their own “roll your own” application that will automagically create a customized ISO image that you can use to modify/install/spin/fold/mutilate to your heart’s content.  Yes, the method bases it on a specific Linux distro, but it will be relatively different depending on the customizations you’ve set in place.  For example, Fedora has Revisor.  There are also non-distro-specific utilities on the web such as Instalinux, as well as complete Linux distros based on customization and optimization in which everything is built from source — Gentoo.

The other method is to construct a Linux distribution completely from scratch, appropriately named Linux From Scratch (LFS).  While this approach isn’t entirely difficult initially — its just building binary files from source files — it is time-consuming.  It is also recursive, meaning that software you compile in the beginning stages are depended upon by other software you build later, and you’ll have to start over from the point of contention if you discover something broken.  This is termed “building the toolchain”.  Building software so that you can build other software with it, that software becomes dependent on the initial software build, which is then used to build even more software — creating a chain of software used to create a system, which will be bundled together to form an installation, and a Linux distribution.  That’s the easy part.  The hard part isn’t in building the software; its in maintaining the whole thing such that when a bug is discovered it is easily and quickly patched/fixed and the remaining components of the toolchain remain relatively unaffected or are automatically rebuilt using the new link in the toolchain.

Whatever method used to build the Linux distribution, there still needs to be some point to the exercise, else time invested is wasted in making an exact copy of something that could have been simply downloaded to begin with.

My goals:

• Hybrid approach, mixing Linux From Scratch with useful tools from established distributions.
• A practical level of security in the system, without sacrificing usability.
• An agnostic mindset for software packages, trying not to rely on one set of software built for one distribution.  A best of breed path would be ideal, taking the best software from well known distributions, and possibly introducing software that is not available in any Linux distro.
• Initially, aim at the desktop and see how that goes before configuring things for a server platform.
• Simple for the user, but without the cost of being difficult or complicated for the system administrator (who is often the user also).

I’ve had quite a few discussions with friends about what I should name it. After all, that’s the most important aspect of a Linux distribution, right?  Amazingly, in the discussions about naming my distro, no one ever asked me what I wanted it to do — but everyone has suggestions for a name.  I guess in the age of Google, and Yahoo, the name doesn’t have to be related to what it does.  I have a pretty good idea what I want to name it, but that may change later.  Considering there are over 600 distributions of Linux in circulation, a few hundred discontinued, and a few hundred more about to be announced (some of which might be discontinued in a few months), the name doesn’t matter that much to me as long as it isn’t completely ridiculous.  Douchebag Linux doesn’t smack of “Download me!  Use me!  I’m useful!”  On the other hand, Master of All Linux sounds good, but is probably just a tad too ambitious.

I wish there was a quick way to check if a name is already taken.  Like a global registry, or something that I could search for to determine if a name is used.  Distrowatch.com is good, but there is a 90 day waiting period for a Linux distro to be added to the list because “this is to discourage submission of new projects that start with great enthusiasm only to vanish in a few short months”.  Like I said…  Easy to start, difficult to maintain.  Incidentally, ParanoidLinux is on the waiting list to be added to Distrowatch.  Its not mine, but I get quite a few people that read this blog after searching for that particular distro.  I wonder why…

I’ve been putting my photos on Flickr for a while now, but Flickr never seems to do the photos any justice (not just mine, but in general).  Every now and then I post them here on this blog, but they usually contradict the “Linux Ninja Geek” theme, since the photos aren’t actually related to Linux, or ninjas or even technology.

So, I have created an entirely new blog.  A photoblog.

Just in time, too, since my new Strobist gear was delivered today, which more than likely means I’ll be taking MANY more photos that have very little to do with the theme of this blog, and more closely related to photography.

So, if you’re interested in the photos I take, head over to http://photo.mysticmorph.net.  I hope to see you over there.