Disasters do happen
Danny Killen – SproutIT
April 1st 2015. 12:35. Several client sites start reporting Internet connections running slow. Multiple clients’ Uninterruptable Power
Supplies (UPS) start reporting loss of power. Nothing registering on the Internet. Another (New Square) client reports smelling
smoke, ironically, whilst on the phone discussing Disaster Recovery. Our Client Internet Partner updates client sites that there is
an Exchange issue with physical Fibre, BT engineers sent to investigate. First Twitter feeds reporting fire on Kingsway. Power
interruption to more sites, though some come back online briefly, only to go down again.
The above was the start of many hours, if not days, with no power, and days or weeks without their primary Internet connection
(most hopefully running on backup connections). Anyone who walked along Kingsway or around Lincoln’s Inn Fields in the
aftermath would have realised the extent of the damage caused. Entire backstreets have been dug up to replace power and other
cables, whilst huge generators littered the streets that were still paved. This was a very significant event and I have been wholly
impressed by the response of the various services. I had expected the disruption to last much longer, but fortunately for most, this
was not to be the case.
Now that we enjoy reliable Fibre based Internet connectivity, I am of the opinion that the most likely business affecting outage
will be due to an extended power cut, beyond the runtime of the UPS (Uninterrupted Power Supplies). Losing both power and
Internet is less likely to happen, but as we have seen, it does happen.
So, how does this affect chambers and what can we do to mitigate these risks?
1. Stay calm and hope the power and/or Internet turns on again soon
2. Have multiple Internet connections, each running to a different exchange, or a wireless backup technology
3. Have UPS supplies for onsite servers and all phone/networking equipment
4. Buy a generator?
5. Or implement a Disaster Recovery Plan that takes account of the risks to business continuity and plan to address each with
an appropriate action, or indeed acknowledge and accept the particular risk.
We all need an Internet connection. Even if you argue you don’t, your mail server certainly does. Sitting in the clerks’ room
without Internet is not the end of the world; assuming onsite servers, the Diary is still available and calls are still coming in.
However, as more communication is driven over email, what are the clerks missing? A Fibre Internet connection, from a reputable
provider, is very, very reliable. Always make sure that you have two connections out of chambers however, even if the backup is
The key services that we are trying to protect, from a clerks’ perspective, are access to Diary and phones. I would argue that
email now factors into those critical systems, which was not the case just a few short years ago when you asked clerks what was
critical to their operation. Members need access to files and email predominantly; albeit they need their clerks to tell them what
their schedules are in the coming days and weeks. Other barrister resources are now primarily available online though they
require you to remember website addresses, usernames and passwords, if you do not access them from your usual computer. This
can prove a challenge for the member of staff, or the barrister who manages these resources. Add these to your DR pack.
In order to remove a power cut from the list of risks to key services, you need to consider moving your services offsite to a
Datacentre. For even if you have a generator onsite in chambers, this is only likely to keep a few key servers and Internet running
along with (hopefully) your phone system. Those of you using IP phones will also need a generator supply to all the switches that
power the handsets sat on your desk. With a generator running you can pack up and head home, or to another location with an
Internet connection, to access chambers’ services. Without a generator you are merely counting down the few minutes, or at
best, few hours, until the UPS batteries deplete and shut down. I recently visited a site with 3 UPS power supplies, all of a
reasonable capacity to run the attached equipment for around an hour. That is, of course, if all 3 UPS supplies did not have alerts
showing that their batteries had failed – such alerts can, and should be monitored. A surprising but common occurrence.
Compare this with hosted or cloud services in a Tier 3 data centre: There are physically diverse power and Fibre feeds into the
Datacentre. These feeds are backed by rooms of batteries to allow multiple generators time to take up the load (pre-heated and
ready to automatically switch on). The diesel used to run them is monitored and the data centre will have at least one contract in
place for emergency fuel delivery on an SLA. Cooling systems are redundant (and keep running when mains power is off), and the
data halls are protected by advanced fire suppression systems. So, this addresses our power issue. Or, does it?
Barrister Article – Disasters do happen Page | 2
Where is your phone system? Where are your staff? How are the phones connected? Are you able to reroute your main number (or
fax) to an alternative during a power failure, or only if your ISDN 30 line fails (an ISDN 30 often carries the calls to/from
chambers)? What use are the dual screens sat in front of staff when the power goes off (this applies regardless of where your
services are located)? Do you have a charged laptop that is able to connect to the network or Internet to get to services? Are you
even permitted to occupy the building if the mains power is off?
For some of the above reasons, a number of chambers have reciprocal agreements whereby, if they must vacate their chambers, a
number of staff may relocate to the conference room of the other to continue working. I expect most will however arrive, ready
to work and find that many things will not work, as they are trying to access their phone system and servers that are located in
their chambers which now has no power. It is entirely possible to have a phone ready to plug in and work under these
circumstances. It just requires the phone system to be hosted (either your physical VoIP appliance, or use a hosted service) and
for both chambers to have the right Internet connectivity in place. This is where it gets interesting.
In order to have reliable access to hosted services on a daily basis you MUST have the right connectivity. The fact that you have
50Mb or 100Mb Internet connection matters little – not all Internet connections were created equal. What you do require is a
private and low latency connection direct into your core services that does not traverse the public Internet, and therefore does not
compete with other businesses or users, to get data to and from where it needs to go. With such a connection in place, you can
relocate your phone system without loss of function or reliability. It is also possible to have that same connection presented to the
chambers with which you have a DR relocation agreement. If your production or DR servers are located in the same location, you
have a fully functioning clerks’ room ready to switch on.
Is this going too far? I will let you answer that. Only to say that if you add together the hourly rate of your members, multiply
that by the 50% who are at their desks on any given day, and again by the number of hours it takes them to relocate (assuming the
servers are still powered on) then one usually comes to an answer fairly promptly.
You will have noticed the absence of terrorists, a chambers fire or flood, access exclusion or pandemic. These will all be on your
list of business risks, but hopefully in discussing power and connectivity I have covered the most probable causes of disruption to
chambers; to your business. And, don’t despair. If you have a fresh set of servers sitting in the basement, it is possible to have
your entire onsite server estate backed up to the Datacentre, ready for activation within 30 minutes – that includes all email
routing, mobile phone sync, remote access and any other required services.
To summarise. Dust off that DR plan and seriously consider what, if anything, it does for you. Consider the risks to your business,
the time that you can afford for various services to be down, and take steps to address those risks given your respective budgets.
It need not cost the earth, but it can cost you in terms of reputation and business if you fail to act.