Sunday, February 10, 2013

A WEEK WITHOUT E-MAIL!



 

It wasn’t a planned vacation.  On Tuesday evening, about 5:40 p.m., we lost our staff’s e-mail connection to our Outlook Exchange server.  Ten minutes later we lost the attorneys’.  We were victims of buzzard’s luck: Two RAID discs and the controller in our Exchange server —the replacement of which had been postponed to avoid a weekend disruption during preparation for a trial — all gave up the ghost in rapid succession. 

As our firm’s resident geek-attorney our IT Director, Sean, sought my advice.  Parts were on order, but would not arrive until the following afternoon.  The Internet, our network, and every other system were purring like a kitten, however there was a high probability we going to be without e-mail for at least most of Wednesday.

Wednesday being “Donut Day” at our firm, I suggested hanging a sign on the door: ‘It’s e-mail-free Wednesday!  Enjoy the donuts!’” 

Sean, wisely, did not take this counsel.

“Okay, let’s pull out ‘Plan B,’” I suggested a little more helpfully.

“Plan B” is our firm’s version of “Reverse 911.”  Years earlier I had suggested that we harvest our employee’s cell phone numbers so we could text-message everyone in case of an emergency.  The exigency we had in mind at the time was a snow day, but to some in our office the absence of e-mail would constitute an emergency and was certainly sufficient cause to deploy Plan B. 

"Plan B"
So while Sean tried to diagnose the failure – the cause of which was still unknown – I composed a text message to our attorneys.  Succinctness not being my strongest suit, this was challenging because of the character limitations imposed on text messages sent between different cellular providers.  Twitter is more generous with its character allowance.  

I finally settled on this:
MW E-mail Services Are Down.  

MW e-mail services crashed shortly after 5:40p this evening.  IT is working heroically tonight to restore service, but restoration time is uncertain.  You may need to rely on personal web-based e-mail on Wednesday.  Incoming e-mail is being held at Postini.  Take whatever measures you need to protect/alert of [sic] your clients.

Even this pithy prose had to be sent in three parts.  I sent it twice because my Comcast webmail kept falsely reporting that my efforts had been unsuccessful, notwithstanding that they were appearing, almost instantly, on my iPhone.

That done, it being after-hours, and with no e-mail alerts to intrude on my thoughts — only the soft glow of the yellow triangle with the black exclamation mark emanating from my message bar declaring Outlook was dead — I had time for a little personal techno-reverie:

I thought back to the earliest days of my practice.   In 1981 written communications were primarily made using an IBM Selectric II typewriter and the U.S. Postal Service.  If something had to get somewhere in a hurry, you dashed to FedEx.  Head, Moye, Carver & Ray owned a Qwip –an early fax machine that was advertised as being able to send one page every two minutes (it was more like six) on a spinning cylinder, but only to another Qwip machine.  The Qwip —manufactured by Exxon Office Systems, which also produced the cabinet-sized Vydec word processor — was spectacularly unreliable, and was thus employed with the same frequency, trepidation, and success as Get Smart’s “Cone of Silence.”
 
Fast-forward about a decade.  When we first put IBM PS/2s on attorneys’ desks, we couldn’t get them to turn them on.  DOS was an operating system only a geek could warm to, and few attorneys had the keyboard skills, or patience, to learn WordPerfect.  There was no e-mail because there was no commercial Internet.  When Banyan “Blue Mail” finally arrived it worked only on our internal network, and no one read it.

My how things change.  Cobwebs grow on our in-boxes, which nowadays deliver mainly magazines and junk mail.  It’s been years since a runner appeared in my doorway breathlessly announcing, “Mr. Luce, it’s a fax for you! It’s a fax!”  Most attorneys don’t know how to use a Dictaphone, and virtually all do their own typing.  We have become addicted to Internet e-mail which, for attorneys, turned out to be the “Killer App.”  We get it on our PCs.  We get it on our smartphones.  We get it 24/7/365.  It is addicting, demanding, disruptive and essential.  And now ours was kaput.

Our IT crew, of which I proudly claim to be an “associate member,” worked valiantly, and largely without sleep, through the night and the next several days to restore modern order.  By dawn Sean had moved us from Postini to McAfee MX-Logic, so that we could actually access new incoming mail.  We backed-up everyone’s local Outlook in-box to supplement our regular full mail-server backups.  When, after the controller replacement arrived, it was discovered the two RAID discs on which our e-mail was stored had bad sectors, we began a restore from our backup tapes to good discs.  Concurrently we prepared our new Exchange server and brought it online by Friday.

The restoration of other, less critical services, lagged.  Outlook “Contacts” and “Autocomplete” had to be manually restored on each PC.  Our voicemail-to-e-mail forwarding service did not get back online until late Friday, and our conference room scheduling was still down as the weekend arrived.  It took about 36 hours for the Internet root servers to recognize our redirect from Postini to MX-Logic, so the release of e-mail spooled at Postini was delayed about two days and arrived in a swarm.  There were sporadic “bounce-back” issues within our network and some Citrix spottiness as the “Nerd Herd” worked to get all the intricate, interlocking electronic gears to mesh again.  But we survived.

Lessons Learned

1.  Shit Happens.  Systems fail – frequently at the most inconvenient of times.  Although the raison d'être of a RAID is its redundancy, a cascading failure such as we experienced not only brought down our e‑mail, but also the flash-frying of our controller did not enable us to confirm there were bad sectors on the affected discs until it was replaced.  

We were snake-bitten, for sure, but could we have avoided this headache?  Few law firms have, or are willing to invest the money required to build completely redundant systems.  Nor are they required to by any reasonable standard of professional care.  Comcast, Amazon and Facebook each experience service interruptions even with formidable redundant systems.  The expense must be balanced against the likelihood of catastrophic failure.  What is required is that attorneys recognize and plan for the arrival of Murphy and the Gremlins before they appear at our doorstep.

2.  Have a “Plan B.”  9-11 finally drove home to most firms the wisdom of providing for disaster recovery.  From our “week without e-mail” experience we learned that firms also need a recovery plan for those disasters falling short of their building being brought down by terrorists.  For example, our Postini system has, at least until recently, been a fabulous SPAM-catcher, protecting our network and increasing attorney efficiency.  However, Postini’s current iteration has no provision for e‑mail continuity.  As a result, last week we could log on to Postini and view umpteen Viagra ads and Nigerian 419 advance-fee scams, but could not access incoming e-mail that we actually wanted.  McAfee MX-Logic has a poorer interface, but provides for e-mail continuity.

Although our IT director acted swiftly to move us to MX-Logic, any e-mail arriving in the gap between our mail-server failure the switch-over to MX-Logic was caught in Postini’s inaccessible spool.  Moreover, our clients had no way of knowing our inbound e‑mail was disrupted since it was spooling securely, albeit inaccessibly, at Postini.  In an age where court orders and pleadings arrive by e-mail, the importance of maintaining e-mail continuity may have been the most valuable lesson learned.

3.  Have a Personal “Plan B.”  Attorneys have become utterly dependent on systems few of them completely understand.  Nevertheless,

A lawyer has a general duty of competence under RPC 1.1, which includes the duty “to keep abreast of changes in the law and its practice.”  RPC 1.1 Comment 6.  To the extent that a lawyer uses technology in his or her practice, the lawyer has a duty to keep informed about the risks associated with that technology and to take reasonable precautions.

WashingtonState Bar Advisory Opinion No. 2215 (2012) (emphasis added).  Similarly, last year ABACommission on Ethics 20/20 Resolution 105(A) recommended this addition to Comment [6] to Model Rule of Professional Conduct 1.1:

To maintain the requisite knowledge and skill, a lawyer should keep abreast of changes in the law and its practice, including the benefits and risks associated with relevant technology, engage in continuing study and education and comply with all continuing legal education requirements to which the lawyer is subject.

(Emphasis in original.)

The Yellow Triangle of Death
 I have been preaching the “competency in computing” sermon at technology-ethics CLEs for years.  Our “week without email” demonstrated the necessity of having a personal Plan B when the yellow triangle tells you “you don’t have mail.”  On earlier occasions when our phone system crashed, our lawyers didn’t miss a beat – they pulled out their cell phones and carried on.  Our network and Internet connections remained rock-solid.  Had they needed to, those attorneys who knew how to use their personal Internet provider’s web-mail service to send mail could have functioned quite well.  Many knew how to do that, but everyone needs this skill.  If we don’t develop such rudimentary competencies now, how are we ever going to cope when an electromagnetic pulse or a hacker attack that brings down the entire World Wide Web?

4.  Don’t Let the Lawyers Dictate Equipment Replacement Schedules.  As noted, our Exchange server had been previously scheduled for replacement, but our IT crew postponed this task to accommodate some lawyers preparing for a Monday trail.  Had IT not been so accommodating would our “week without e-mail” have occurred?  We can never know.  

This is tricky office diplomacy as the practice and our clients come first.  However, in a firm of any size, there will always be some trial or closing, and there will never be an optimally convenient time to take down a major network component.  Accordingly, administrators must be ready to push-back and support their IT’s replacement schedule.  If attorneys were given absolute power to veto an upgrade schedule, we’d all still be running DOS 3.1. 

5.  It’s Still a First World Problem.  E-mail is a “disruptive technology,” in more ways than one.  Psychology Today tells us it’s addictive.  Efficiency experts advise us to turn off e‑mail alerts and our cell phones if we don’t want to experience technology-inducedADD.  (BTW, here’s how to turn off the pop-ups and sounds in Outlook.)  

From my own experience both are right.  That pop-up window and e-mail alert elicit a Pavlovian response demanding attention which disrupts my concentration and momentum like a bumper-car collision.  When the e‑mail is really flowing, the default Outlook configuration makes an average computer light-up and sound like a pinball machine.  Lawyers, competitive by nature, find themselves cast in the roll of the “Bally Table King,” playing electronic Whack-A-Mole, a/k/a “I can answer that complex legal issue in 1 minute,” to the detriment of their clients, their professional wellbeing, and possibly their health.  There’s a reason Intel and other companies pilot-tested “e­‑mail‑freeFridays” in 2007, and those overcome by an avalanche of electronic communications declare “e‑mail bankruptcy” – and option attorneys do not enjoy.

Electronic court orders actually requiring immediate response excluded, losing e-mail for a day or even several days, is still a First World Problem.  It can even be a welcome relief.
While most lawyers have willing embraced electronic tethers out of necessity or convenience, the practice of law is not dependent on any machine.  Though we must be prepared to deploy our personal “Plan B,” we are not, and must not, allow ourselves to be undone by something as mundane as a computer failure.

I keep thinking of Samuel T. Cogley, the attorney hired to defend Captain Kirk in the original classic Star Trek episode, Court Martial.  Kirk is ordered to stand trial for allegedly prematurely jettisoning a research pod, thereby killing Lt. Commander Ben Finney.  Kirk arrives to discover Cogley has encamped in his quarters and bestrewn it with books.

Books!
COGLEY: What's the matter? Don't you like books?
KIRK: Oh, I like them fine, but a computer takes less space.
COGLEY: A computer, huh? I got one of these in my office.  Contains all the precedents.  The synthesis of all the great legal decisions written throughout time.  I never use it.
. . . .
COGLEY: This is where the law is. Not in that homogenized, pasteurized, synthesizer. Do you want to know the law?  The ancient concepts in their own language?  Learn the intent of the men who wrote them, from Moses to the tribunal of Alpha 3?  Books.
(You can view this marvelous scene here.)  Later, Cogley forcefully argues Kirk should be entitled to confront his accuser:

COGLEY: The most devastating witness against my client is not a human being. It's a machine, an information system. The computer log of the Enterprise.  . . . I speak of rights. A machine has none.  A man must.  My client has the right to face his accuser, and if you do not grant him that right, you have brought us down to the level of the machine. Indeed, you have elevated that machine above us.  I ask that my motion be granted, and more than that, gentlemen. In the name of humanity, fading in the shadow of the machine, I demand it. I demand it!

While we have not yet fallen under the yoke of our own electronic creations, there is wisdom in the words of Samuel T. Cogley.  The law is not to be found in that “homogenized, pasteurized, synthesizer,” and we cannot permit our practices to be ground to a halt when technology fails.  From the perspective of professional risk‑management, the essential counterpart to techno-dependency is competence and self-reliance.  Cultivate your inner-geek.  Develop your personal “Plan B.”  Do these things now, and you will easily survive your own “week without e­-mail.”