@@ -0,0 +1,120 @@
+++
layout = "post"
title = "Keep Yourself Drain Ready"
email = "drainready"
date = 2023-08-08
draft = true
tags = ["off-topic", "balance", "human"]
summary = """You have an, ethical, moral, and professional responsibility to
keep yourself drain ready at all times."""
+++
I used to work for a big data company. One of the responsibilities I had, was to
help ensure the availability of all of our services. My individual slice was a
very narrow subset of what could be considered the prime directive for my org as
a whole.
> Keep the site up.
So critical was this one directive, in what seemed to be daily drills, someone
would turn off one of the many data centers, and route all that traffic to the
other data centers. One of the critical prerequisites of these tests, was the
constant maintenance of the excess buffer of global capacity and availability.
Any violation or loss of this buffer immediately became an internal incident,
one that would get managed as an emergency, and then would complete the incident
review process. There didn't need to be any noticeable symptoms, or degradation
of the services. Merely the loss of the buffer became its own incident.
The buffer size for us was `$CAPACITY_LARGEST_DC + $TOTAL_GLOBAL_CAPACITY * 7%`.
That was the line in the sand we drew to keep the site "drain ready". But,
that's *our* bar that we used for *our* site/servers. I assume you have
something that's more important to you than our site, right? Where's your bar?
How do you know that **you** are drain ready?
### Load Shedding
Please forgive this tangent, but would you like to see something cool? You can
make any civil engineer turn white, no really, it'll work on just about anyone
who works energy generation or power delivery, you can get any of them to turn
dead white. Just find one, and repeat to them: *"mandatory load shedding"*. (or
if you're in the EU it's *compulsory load shedding*). Cool right? You can also
try *black start* if you want to try to make them break out into a cold sweat
too.
For the uninitiated, load shedding is the process (generally) for disconnecting
power consumers from the energy grid. Usually the goal here is to protect the
energy grid from total colapse, and avoid that same *black start*. Keeping an
energy system running is so much easier than starting, or restarting one that
"avoid a black start" is the equivalent to "keep the site up".
If you're ~~an engineer~~ a human, you should hold yourself to the same
standards. You should do *everything* in your power to avoid having to black
start yourself. Including making those decisions, as early as you're able to so
you don't have to make them when the pressure is on. There's far to many
variables for me to be able to give you a distinct number that will be useful
for you. But mine is 4. Studies have shown that once humans go over 8 disparate
responsibilities, it becomes exponentially easier to make a mistake, miss
something, lose track of the current state, or simply see degraded performance.
4 is just where I start because I want to have made the decision, well in
advance of the case where I suddenly need to double my workload with more urgent
items.
That decision, is what specifically you're going to **intentionally** drop on
the floor. Ask yourself. If I picked up X number more urgent things that I
couldn't ignore, or deal with later, which of the tasks am I currently dealing
with, will I drop, ignore, passoff, set aside, call good enough, which ever name
you decide to give it; so long as the result is, it's no longer something you
allow yourself to spend any time, or give ***any*** attention. And then, make it
a rule! The decision can *not* be, "what will I ignore unless it's really
important." It **must** be "What will I ignore so I can save my attention for
the more important stuff".
Avoiding a black start is an emergency situation, something has already gone
wrong, and now you're in triage mode. There are now more problems than there are
people who are able to work on resolving said problems. Which means, priority 0
is now not allowing yourself to become an additional problem. If you lose the
ability to help, you're only going to make everything worse, for everyone. Don't
allow yourself to get there! Start **your** load shedding procedures, and start
ignoring everything you need to make sure you're doing a good job at what does
have your attention!
### Your Drain Readiness
I can still clearly recall the sudden feeling of unease, back at
`$BIGDATACOMPANY` I was sitting in a meeting with a number of engineers, when a
manager asked, what's your capacity. Nearly everyone at the table, across
multiple teams, said they were at 100% capacity. Everyone gave an answer over
90%.
This was shortly after I had more than a few engineers, all of whom I deeply
respect tell me either, not to burn myself out trying to do my job, or ask me if
I was sure I was doing ok, and I wasn't getting over loaded. After taking a
minute to make sure I really was doing ok. I said yes, I wasn't at risk of going
over my total capacity, but I was out of buffer, and starting to enter red line.
That's when I decided that it was well past time for me to shed some load
myself.
My intent was to hand off one of the incidents I was working on, but everyone I
wanted to trust was already at, or over the capacity they should have. My first
reaction was much closer to disappointment, that there wasn't anyone I could
trust to help. But that was short lived, and replaced by that feeling of dread
when I realized, that I wouldn't have been able to volunteer either. I wasn't
drain ready myself.
No matter how perfect the metrics you use to predict how capable you are as an
individual, or how excellent you are at triaging under stress, unless you're
omniscient, there still might be something much more important around the corner
that you didn't predict. And if it's something important to you, that's the only
thing that should get your attention first. And don't allow yourself to make the
mistake that I made, just because you have some personal buffer you can tap
into, that doesn't mean you're drain ready. If you spend your personal buffer,
you've just lost the ability to protect yourself, which also means you've lost
the ability to prevent yourself, and the problems you were fixing from dropping
on top of someone else.