Campfire 365 Podcast – Episode 4: Creating a Disaster Proof Organization

Carolyn Norton 00:11
Hello, happy campers. This is your host, Carolyn Norton. And in our last episode, we cover how to get back on track after you survive a big disaster. A lucky few can come out ahead after such an event. But how can organizations become proactive instead of reactive? Joining us today on our season finale of the big disaster.

Rich Fowler 00:34
I am rich Fowler, and I’ve spent about 25 years working with partners in the partner channel.

Daryl Moll 00:41
And this is Darrell mom, principal Cloud Architect with over 20 years of IT experience and 15 years of cloud experience.

Eric Robertson 00:49
Hi, this is Eric Robertson, Director of BI T. Eat sleep breathing disaster recovery.

Amy McKie 00:55
Hi, this is Amy Mackay, Director of Client sales. I’ve spent the last 25 years working with clients in the IT industry.

Sam Miller 01:04
Hi, this is Sam Miller. I’ve been in the IT space for enterprise systems for about 30 years, having been in sales, marketing, development, consulting, and implemented systems as well.

Carolyn Norton 01:18
Welcome, everyone. So we’ve had an incredible journey, we’ve taken steps to understand are you really are you ready for the big disaster? What makes up that big disaster and what happens after the big disaster happens? So we’ve had the opportunity to look at all those aspects. And hopefully, everyone has had the opportunity to gain some insights and really think through disaster and disaster recovery and protecting organizations from that. So in our, in our final episode here, let’s talk about shifting from reactive and being more proactive. We talked a lot about planning. So what are those different steps that organizations can start to venture into creating, say, a disaster recovery process, and getting to be more proactive so that if a disaster happens, there are a lot more prepared.

Daryl Moll 02:24
I mean, I’ll start off, I mean, I think the key to, you know, really transforming from that reactive to proactive thought approach is, again, starting that process of what we mentioned before is coming up with that plan. So really, even to the point of, you know, starting those meetings, and starting those first meetings, if you’ve never done this before, if you’re starting from scratch, really just get it kicked off in the aspect of starting to talk about the low hanging fruit, or what the what what the biggest threats to your organization are, and meet and then commit to having a regular occurrence. And depending on if you’re again, if you’ve never done it before, and you’re starting from scratch, or if you’ve done it before, and you’re just kind of going through the tweaking process, and trying to make it better, it’s all going to, you know, vary on what that frequency is initially, but I would say, you know, switching from proactive or switching to proactive from a reactive, it’s just a matter of getting started, and then committing to, you know, some frequency.

Sam Miller 03:26
I think another piece that

Rich Fowler 03:27
we a lot of times we forget about is that we live in an ecosystem. And we cannot live long enough to learn everything the hard way, we’ve got to learn from other people’s mistakes. So not only do we need to get that plan going, but we formulate that plan by talking to other people by talking to people in our business network by reading by listening by learning from the folks who have already followed it up before us. And I’d much rather learn from their mistakes than I would to make myself. You know,

Sam Miller 03:57
I think a lot of it is not just just talking about it, and maybe sending out some of the emails that I’ve seen in organizations where they’re, you know, testing to make sure people aren’t going to let a virus in or lead something destructive into the company. But it takes investment. It takes educating people just like was said earlier, learning from other people’s mistakes. Just a couple of case studies send out even if they’re short and humorous enough that people will read them, but educating people as to what can happen to a company and what can happen to your job. And your ability to just get the day done. When disaster happens is important, as well as investing in certain technologies that are going to help protect the company and the assets of the company. A lot of people haven’t really invested in data protection Data Governance solutions. And some of those solutions don’t really have to cost a ton. But at least you know where your your best data is, by using solutions like that, and you know where to start protecting that data first. Because you have to prioritize being

Rich Fowler 05:17
proactive.

Eric Robertson 05:19
Yeah, and I think one of the big points you hit on there, Sam is, is getting that buy in. And it’s not just from the people who are creating the plan, but from everybody in the organization, you are going to be more successful if you actually have that buy in from everybody, especially the stakeholders at the top level who are helping drive this so that everybody else has that purpose that they feel, hey, I have a part in this plan, I need to make sure I’m ready because it’s going to affect my job overall. And what am I going to do in case this happens? What are the parts that I’m in? So I know what I’m supposed to be doing? Where I’m supposed to get my communication from? If communication is disturbed, it’s having everybody know what’s going to take place and be prepared for it and tested so that it’s not just, Hey, disaster happened? What do we do now?

Amy McKie 06:04
And that’s what I was gonna say, Eric, and Sam is kind of, you know, who’s who’s taking ownership and having the companies, you know, the leaders, the employees, everybody kind of needs to take ownership of that business continuity plan, you know, and what’s going to happen if something happens, not just maybe if but when something happens?

Sam Miller 06:23
Yeah, it’s like the fire drills that a lot of people did when when they were kids and certain schools, you know, what’s what’s, what are you going to do?

Carolyn Norton 06:37
Then maybe that’s the first step, right, what what for, you know, a handful of these items, probably the most common scenarios that would apply to organizations? What do you do?

Daryl Moll 06:50
Well, I think a key component is, you know, initially deciding what, what you’re actually protecting against, I mean, you know, it, you know, take a look at what those threats are, like we’ve mentioned in the past, you know, in past episodes, or previous episodes, what are the different obstacles that you can run into? What are the different threats that you have that, that could be presented with, pick one and start going through there, and then you take another one and build off of the other one, and what you know, what are these have in common that can can apply to everything, what is going to be unique to this particular disaster or threat, and go from there. But as you pick out which one is the most likely to be your problem, you know, you have that focus, and you have that strategic, you know, buy in from your stakeholders, you have to, you know, go again, figure out what the scope of what you’re what you’re trying to protect against at first. And, and again, I don’t think you go with an all encompassing one right out of the gate, I think you pick one piece and move on. And that allows you to be a little bit modular in fate and in a phased approach. But then you, you go through, and you figure out what those roles and responsibilities are of all the different people, whether it be the people with the boots on the ground, whether it be all the way at the top at the stakeholders at your C suite that are you know, you know, funding and trying to drive some of this from that level, figuring out what your critical business functions are, and what your tolerance for downtime is, when you go through that, you know, you’ll always hear RPO and RTO, recovery point objectives and recovery time objectives. Those are critical pieces that you’re gonna have to decide, as an organization, what you need to have defined so that you can say, here’s what I need to work with, because the budget between working with a, you know, 15 minute recovery, recovery point objective with a one hour recovery time objective is very different than saying I need to have a one a 24 hour recovery point objective with a three day recovery time objective. There’s lots of different solutions out there to cover, you know, all of these scenarios. They all come at different costs. And it all kind of varies by what those RPO and RTO are.

Amy McKie 08:52
Well, but Darryl, there’s also I mean, all of this varies by industry. So each each company is different. Each business is different, you know, you’ve got clients that are maybe online retailers, their their, you know, DR plan, business continuity plans a lot different if they’re relying on, you know, PCI compliance, somebody’s not hacking into credit card data, or making sure their online stores open all the time making sure you know, there’s enough bandwidth, even for Black Friday, you know, let’s throw that out there. So those are different business scenarios that a manufacturer that you know, they can’t lose that downtime if something happens to their system, or they lose electricity or something along those lines. So I think everybody’s a little bit different. And it’s defining what their certain plan is and identifying that.

Eric Robertson 09:40
So I think, I think if we, if we take it holistically, and we just break it down to three simple questions, everybody could get started. And I think that’s how we get on this path of being prepared is what needs to be protected. What might disrupt it in how and what is the impact if it gets disrupted? And if you start with those three questions, and just start to branch off and figure out what it is. That’s when that plan starts to come together no matter what industry you’re in, no matter what your vertical or what your specialty is, if you answer those three questions, you’re already on that right path to get what you need to make sure that you’re going to protect yourself. There’s no way to get risk to zero. But we can at least minimize the impact when when a disaster does strike.

Rich Fowler 10:24
So what are sorry, Carolyn, I’m gonna lead Eric down a path because I have a question that I he knows the answer to. But I want to see if we get to the same answer. So when in those three questions, Eric, whose responsibility is it to answer those three questions?

Eric Robertson 10:37
Oh, great question. Great question. So when it comes to defining a business continuity plan, it’s not just one team, it’s not just it is not just finance. It’s not just one or the other. Everybody, it’s, it’s having stakeholders from all the different areas to identify these because not one area is going to know all those those those responses. We might know some, like coming from an IT perspective, I might be able to give an IT perspective. But I don’t know what the impacted finance was not able to do something, I might know it, generally, but they’re going to be able to actually say, hey, if this function breaks, that’s a big problem. And I might not know that so. So when coming up with these plans, having more people involved is always the better strategy, not everybody, but having some representative from each of the groups that can put their insights to make sure that plan is not just pigeon holed down one focal point such as it,

Rich Fowler 11:30
but not only so not not everybody has to be involved in the meetings, but everybody has to be communicated to with the purpose. And I always talk about it from the janitor to the CEO, because any weak link breaks the chain, of course. Thank you, I was leading you down to pass on your answer, as I hoped you would. Sorry, Carolyn.

Carolyn Norton 11:52
No, no. And I’m going to piggyback off of that and say, Okay, great. We’ve identified these key business critical functions. But how do you break it down even further? What are the two or three, maybe five different aspects of that critical of that function that needs to be addressed? And I’m thinking back to an organization that primarily deals with shipping, and they had a power outage which, which also led to an internet outage. Okay, great. Now they can’t ship and that’s their primary function. Now they know if we can’t shift, that’s going to be a problem. All right. What do you drill into? After that? Does there need to be equipment conversation? Is there need to be other provider? Like, what are those? What’s that next level down that we’d need to start talking about?

Sam Miller 12:46
How that’s so interesting. I never thought about, because I’ve actually looked at disaster recovery plans before and it was always around what happens if x happens? And I’ll list of people and oldest of contacts, you know, things that, you know, I’m imagining that all systems are down, but I really haven’t seen plans by process. Like, here’s a specific process at the company that’s completely broken down. What are we going to do? What’s the way around it to keep the lights on and keep the business operating? Office?

Carolyn Norton 13:22
Right. Right. You know, is it a power generator? Yeah, I don’t know. Is it backup internet connection? Is it a phone with a hotspot? Like? What’s that? What’s that? What are the components that make up that I, you know, we’ve identified it now, we know this is important, we have a list of things, breaking it down further, and what are what are the different pieces that if they were not operational, it’s going to affect this key service? And what do we do to plan for that? Because that’s part of the disaster, right?

Daryl Moll 13:55
And I think part of the you know, what you’re what you’re getting to Carolyn, is, is that prioritization that we’ve mentioned, you know, previously in previous sessions, is that prioritization of, you know, hey, we do a through z, but I really need you know, core functionality is, you know, A to F and etc. But, you know, prioritizing that list of business functions and whether they be, you know, you know, how, you know, higher level, you know, it, you know, hey, I need to be able to ship product out the door, okay, well, let’s, you know, drive the next level deeper with those people who are responsible, like Eric mentioned, the people with boots on the ground and getting more information. What do we need to do to ship product out the door? Well, let me tell you, you know, you tell me the, you know, top five top 10 processes that we need to be able to do because something may, you know, slip your mind or you may not know of something that’s, you know, a critical cog in that wheel. But, you know, going through and defining the priority and prioritizing those processes. And then, you know, saying here’s the critical pieces, here’s the pieces that we need to take care of first, and then here’s the pieces that we need to do as much as poss well to make sure they don’t go down or if you know, power is lost or internet is lost, what can we do? defining those lists?

Eric Robertson 15:08
I think I think when when we come all together and we create these processes, nothing’s going to be the catch all, be all end all. And I think that’s why testing, the business continuity plan is so important. Because that’s when we’re going to see if our plan is actually going to work, it’s going to help us identify the gaps, and it’s going to also prepare our team for what’s going to happen when that emergency occurs. So testing is that big piece, and you know, at least annual testing is required. And once that plan is in place, you know, figuring out, did we actually identify all those gaps? Are there pieces that we’ve missed? Is there a process that like nobody thought of, and it’s like, hey, this, this was missed. Now we can be addressed, added to the plan and built upon it, I don’t think any first initial pass is going to capture everything. And it’s that continual testing that’s going to get us to that, that goal of hey, how can we be best prepared by by finding out all these different scenarios?

Amy McKie 16:04
Well, Eric is also revisiting that every year or every six months, you know, as businesses changes, technology changes, you know, not just, you know, hey, we came up with it once and just setting it aside and moving on.

Eric Robertson 16:17
Totally, I mean, I think we saw that with, you know, the pandemic, you know, a lot of a lot of business continuity plans were around offices, and offices still play a big part, but not as big a part or role as they once did. And I think a lot of business continuity plans had to be adjusted, because it wasn’t about the infrastructure of the offices as much as it was making sure employees could work remotely, and how they can work. And what happens if now we have an outage in a certain area, or whatever else, you know, it adjusted, and if you didn’t take those changes in the industry from occurring, they would have never kind of made that shift. To redo their plan, they would have just still been like, well, we got to make sure the power is on at the office, even though we only have three people there right now. So you know, it does it has to be reviewed and has to be updated, or you’re gonna miss stuff.

Rich Fowler 17:06
I think that’s the point that I was going to jump in on is we’re talking about the the big disaster, and we’re kind of, we’re kind of approaching it like it’s always going to be a power outage or a thunderstorm like Amy is about to get but aren’t there multiple types of disasters, Amy mentioned that you know, somebody that can’t process orders, a credit card hack, an email, fish that gets into your system, there’s more than one type of disasters, they’re not shouldn’t be planned for all of them.

Sam Miller 17:36
You know? I’d always say the answer is yes. But there’s a, you know, a cost benefit to trying to break down every single thing that could go wrong. And it’s almost an intersection grid of I don’t want to make it 3d. But it’s systems that are supporting the business and the people that are supporting the business mixed with the process. And so I just want to tell a quick story about a customer that I worked with several years ago. And they were on a relatively new system. And they seriously couldn’t ship one day suddenly, like shipping was running so slow, that they would hit the enter key, and they couldn’t ship out. And this was happening across Europe. And it ended up being a bug in a system that had been updated, however, that nobody had any idea how to fix this problem. And then people started testing, like it’s adult, the whole group just started testing, how do we how do we get things out the door, because it seems like if I stand here and stare at the screen for 30 minutes, it’ll finally give me the shipping ticket I need, or the pole for the for inventory. And nobody knew who to call. So they finally ended up waking someone up in the US, they figured out who to call because somebody had their number on a cell phone. So they woke them up. Everybody got online and started talking. And then they all figured out alright, so every time someone hits enter, it’s having collisions. So we can only ship one item at a time. And it’s I they just had to put heads together and figure out how do we solve this problem. So it was people it was a rope broken business process. And if If only they had simulated like what could go wrong, they probably could have saved millions of dollars in lots of hours. But to get through this process for another 24 hours, they literally put all of the offices in Europe that were sharing the same system on a speakerphone. And before anyone hit the shipping key. They would yell out shipping. I kid you not. And they’d hit the they’d hit the key and everything was working. So they got everything. They got everything up and running. But from the beginning, all of the things that cost money and slow down the business was Who do we call what He said, What piece of the process is broken? Why is this broken? Is it the credit card? Is it inventory? Is it you know, whoever’s running the ERP, all of that stuff. So because all the pieces that are so important, and it all tied to revenue, so it’s almost that the priority of everything that you’re going to check from a business process is, what does it take to get money in the door and product out the door? If that’s the kind of company you are or services complete, so you can bill, so you follow the revenue chain for priority.

Rich Fowler 20:35
So I can tell my it nerd that I don’t care about his phishing scams, because it’s not revenue related.

Daryl Moll 20:40
Whoa, oh, only if you’re prepared to be hit by what kind of revenue it’s going to generate or impact when you when you when you click on that link

Carolyn Norton 20:51
rich? It’s true.

Daryl Moll 20:56
I mean, I think the I think, again, I get you asked the question earlier, which and you know, it’s like, Oh, do we plan for everything, ultimately, eventually, yes, you get there, or you think you that you think you got there, but you should never assume that you’ve gotten to everything, you should always be looking for the, you know, the one piece that you didn’t, but I think, again, taking a phased approach, and whether you’re weighing it on, you know, hey, what’s most likely what disaster or issues are most likely to happen to my organization, or looking at it from which ones are going to be the most financially impactful to my organization, and then doing kind of a balanced approach on that, but I think you’ll pick, you know, one or two and start working towards it, when you and then as Eric mentioned, as you’ve tested it, and, and tweaked it, and gone through that, you add a little bit, you had a next one, you know, each iteration going through there, so that, you know, as the as a phased approach comes along, you ultimately get to, you know, more protected every time you’re doing it.

Amy McKie 21:49
And then Darryl, do you just think about these weird wild scenarios that we think never gonna happen? Like, you know, the pandemic, because I don’t think anybody ever thought that we were all going to be working from home. You know, what weird, wild thing could come at us next?

Daryl Moll 22:03
No, but I mean, I think if you you know, you know, judging or, you know, taking it back to, you know, previous experiences, I worked at a manufacturing company, that their industrial park, we had one way into the industrial Parkway, which when you drove in was surrounded on the left and the right, by a floodplain, you know, or overflow place for floodwaters and stuff. And, you know, you know, in looking at that, you know, what could go wrong? Well, you start having serious rains and serious flooding issues, guess what happens to telephone poles, they become weak, and they tip over and you lose your telephone lines, you lose your internet connectivity, you, you know, get in the road get flooded to the point where nobody can even get into the office, you know, start looking at it from the aspect of which ones are most likely to happen first, or you flip flip side and look at it, which ones are going to have the most financial impact? Well, if I, you know, get hacked, and, you know, get a ransomware and lose all of my customer data, and all their personal and private information. And that gets out, you know, and gets exposed, my reputation is gone. Not to mention, I’m down for business for however long, you know, that’s going to have a huge financial impact on my organization, if we can even recover. So you know, you got to kind of weigh again, and as each iteration goes through your, your check one off the list, and then you add another one onto the list, and you kind of keep, but as you’re going you keep testing and moving forward. And that’s the difference between proactive and reactive is you’re not just waiting for the next one to happen. You’re being proactively and trying to think about things and weighing which ones are most likely or which ones are going to have the biggest impact.

Eric Robertson 23:39
You’re never going to catch them all. And there’s no way we’ll get rid of all the risks. So it’s literally just trying to be as prepared as you can be. And having something in place is better than having nothing in place. Because something eventually will happen.

Carolyn Norton 23:55
Yeah, and once those once those items are identified the top two or three, right? And then you understand what the possibilities are around that. And I have to assume no organization has never dealt with some sort of disaster, right? There’s always there’s there’s some historical reference and an organization can draw upon, oh, I remember the time. And I remember when this happened, or like Amy mentioned earlier, the pandemic really left everyone with a, a point of reference to really gauge well, when that thing happened. This is where we were at and we didn’t really know what to do. And well, let’s start from there and talk through that experience and determine you know, what, what do we need to focus on and how do we better prepare for the next or if it I hope doesn’t, but I’ll say it if it happens again. At least we’re ready for it the next time around because it can happen right disasters, and I’ll use weather as one that’s not predictable. But what but is predictable. can come back around And and we’ll constantly come back and test you to see how ready you are the next time around. So that might be a way organizations can take a step back and really think through what would they want to focus on beyond just honing in on the business processes, right? What experiences? Are they aware of? Mitt, whether it’s directly or indirectly because again, we we all have experiences, and we talked to other people. So I’m sure there are stories that can be shared. Absolutely, Carolyn,

Daryl Moll 25:33
I agree. And I think that, you know, a lot of it comes down to, you know, I think it was Sam that mentioned, you know, the cost benefit of doing it, because, again, you have to define those processes, you have to define what, what you need to do and put to put it in place. And you have to weigh, you know, the benefits of, you know, hey, is this worth what I’m doing? Or can I take, you know, a step down. And if I say, again, the difference between, you know, being back up and running in an hour and back up and running in, you know, a day, you know, where’s that threshold of, you know, loss and indoor indoor downtime that you can can handle. But you have to weigh those benefits as far as that goes. And you know, to determine if that investment is worth the risk, because there’s always, you know, like Eric said, do something, it’s something is better than nothing, there’s things that you can do that will get you a lot of, you know, a good amount of protection for at a very, extremely low budget. And then you work your way up from there. And it all depends on what level you need to get to.

Carolyn Norton 26:35
So let’s pretend our listeners listen to our podcast, they sat down, they identified something and their relative components. And now it’s time to put that paper plan, whatever that plan is, and start rolling it out. So how does how does an organization do that? Hey, we sat down, we talked through everyone, we identified these things. How do we roll it out? And how do we get everyone to support the initiative? How do we explain what’s been designed and why it’s been designed?

Rich Fowler 27:08
I’ll even ask another question on top of that, do we have to do it ourselves? Or can we? Can we enlist the help of a partner are there? Are there teams that can help us do those kinds of things? If I don’t, I’m thinking of my own business? If I’m, if I don’t have that experience, if I don’t have the barrels? And the Eric’s on my team, what do I call?

Eric Robertson 27:29
Ghostbusters? Sorry, I’ve been wanting to say that, but no, there. Yeah, there’s of course, and I think, you know, everybody’s kind of talked about it. But the internet is such a great resource for all the stuff that we’re talking about, you know, all the things that we talked in the beginning about what has come before the experiences others have had people have documented, like, how do you create a business continuity plan? How do you roll it out at what are examples of some of those disasters? There are lists and lists. I know FEMA even has a business called New impact analysis, where they have a worksheet that just goes against the operational and the financial impacts. And with that, you’re kind of able to figure out like, Hey, what are those operational impacts, and what’s going to be that financial impact from it by filling out this worksheet and identifying what those processes are. But to go back to the other point, you know, once you once you’ve designed it, and you’ve thought about, you know, there’s there’s a lot of different pieces to a business continuity plan, because it’s human resources. So all the people, it’s the hardware, the software, technology, all the assets, a company has our utilities, and then there’s all those third party services. So it’s a big scope. And I think that’s why when you’re having those discussions, and we’re talking about designing it, it is a it is a full organizational view, because it’s not just one area, and it’s going to touch upon a lot. So you want to make sure that you’re you’re addressing all these points as you’re working through it, and when you’re implementing and rolling it out. It’s it’s communicating. And I think we’ve talked about in the other episodes, but communication, honestly, is one of those pieces people forget about. And it’s also communicating the plan. And before you can test it, you need to communicate like here’s what’s going to be happening, here’s why we’re doing this. Here’s the purpose, people are more willing to do something when they understand why they’re doing it and understand the you know, the impact it’s going to have on us as an organization. So communicating out that plan and having them test and know what their role is in it. Well, that’s

Rich Fowler 29:27
Sam’s point to the fire drill. I mean, they always told us what the fire drill was for and why we do it and why we do the tornado if you’re in the Midwest, why we do the tornado drill and those kinds of things. Absolutely. Tell me why I get it. Don’t tell me why I thought you

Amy McKie 29:42
were like me right before we joined this call going, Hey, we’ve got a massive storm rolling through guys. You know, if I lose internet, you know, I’ve got my backup plan. I’m just going to join via the hotspot, but it was communicating at least you know, I’ve got a little bit of a plan if something happens.

Sam Miller 29:56
So does anybody on here remember y2k? A you know, some of these kids were boring. But what was so interesting about that, you know, it was it was a different world. But you know, we’re all people. So everybody kind of realized, okay, there’s a risk. Yeah, oh, look, here’s here’s a field on the screen, that’s only two digits. I wonder what’s behind that. And I remembered, every company looked into it differently. And in the end, it didn’t end up being a disaster, but for a few companies, but a lot of the companies actually bought brand new multimillion dollar systems, because they figured out via committee, that it was going to be a big issue. And that’s why it didn’t end up being a big issue. We knew it was coming. People looked at it going, I wonder what the risk is they formed committees, they actually had a group go through major organizations figuring out where all the risk, what was they documented, and they started addressing it, but it was they knew what was coming. And in this case, we don’t know, you don’t know what’s coming. And for many companies, it’s overwhelming. So I can absolutely see hiring experts, any stage in any step of this process, to either do an ROI of how deep do we go for for risk assessment, as well as advice, you know, or even guiding you through all the way through disaster recovery planning. But companies can do it for themselves too. Just just by doing research and studying the past. Because organizational behavior, when a company recognizes Oh, each of us as employees were at risk, because if the company doesn’t exist, we don’t have jobs. Like if it’s shut down, it’s shut down. You know, so everybody kind of got on board with y2k, because we knew a disaster was coming. So back to the fire drill and simulations. I mean, imagine picking out certain users and having monstrous things on their, on their laptops, and then having them testified to the company what that was like, never heard of a company doing that. But if you were to do that to some key key employees, and then get feedback, and do an educational session for a company, these are ideas I haven’t heard of companies doing but it’s worth it’s worth the cost.

Eric Robertson 32:32
And I think that’s where testing really comes because it also shows the team members that might need additional help it you know, from a fishing standpoint, I know somebody shots fired at me before, but a phishing test kind of identifies who who needs that extra help. And in that plan, when you’re doing when you’re, when you’re doing the testing of your business continuity plan, it could also identify the people who are either aren’t bought into the why this is important. Or maybe they’re not understanding, like, why this is important. And making sure they can do it when the time comes. So having that buy in and testing is, you know, we’re keep going back to it if you if you don’t test it, you don’t really know if it’s going to work until the problem happens. And that’s really when you don’t want to be testing something that you’re not confident about. So be proactive test.

Daryl Moll 33:23
Yeah. And also, just to add on to that, Eric, I think, yep, be proactive test. But a critical piece of that testing is an immediate, like, kind of follow up meeting post mortem discussion on what did we learn because you don’t want to just test and be like, Oh, we have all this stuff? I mean, I know it kind of, I think it probably was an assumption of yours to go without saying, but make sure that you have those post mortem discussions and come out with lessons learned so that you are doing that next and have that information for the next time you do that tweak.

Eric Robertson 33:53
Make the adjustments right away. Lessons Learned. Super important. I couldn’t emphasize that more Darrow Great point.

Carolyn Norton 34:03
And also, you know, everyone is learning but they’re not going to get penalized for you know, maybe not paying attention to first time out when the plan was rolled out. Right. They can feel free to voice Oh, yeah, I made a mistake here or I didn’t realize or understood or maybe I just need more help understanding

Daryl Moll 34:27
that lesson lesson learned more training like Eric like Eric said you know, that absolutely is a you know, a valid component and yeah, I mean, people need to understand that needs to be properly communicated right Carolina there isn’t a you know, you’re you’re going to be in massive trouble for you know, not doing this or doing that, you know, it’s Hey, okay, here’s what you did. Here’s what you know, you could have done a little better. Here’s, here’s what we’re looking for. And you know, here’s what to look for next time or here’s what to do next time.

Carolyn Norton 34:57
I have heard of companies implement Then policies if they don’t follow the processes that there is going to be some sort of repercussion. But I have to imagine there are many studies out there that show that was going more positive reinforcement yields better responses from you know,

Rich Fowler 35:20
your Oh, there’s nothing like, there’s nothing like the reinforcement of having your ad I capitalize the end and it nerd just for the record because I couldn’t do their job. But there’s nothing like that positive reinforcer. Hey, dummy. Do you realize you clicked on this? Why next time call me first. That’s really all he had to say. But I suddenly I haven’t forgotten that, you know?

Eric Robertson 35:44
Yeah. You want to

Daryl Moll 35:45
talk about a disaster, Carolyn, and a new and a new disaster that, you know, you’re talking about the people don’t necessarily think of? What’s that disaster when, when your person that you didn’t know, and your shipping department was a tick tock star, and they go viral on Tiktok. With your policy of the beatings will begin when you didn’t do our disaster recovery plan testing correctly?

Carolyn Norton 36:05
Yeah, that’s true. That’s true. That’s true. Well, there’s,

Sam Miller 36:10
you know, we’re, we’re talking tech a lot. But you know, boards have meetings, like, what happens if this person evaporates? Overnight? What what are we going to? What are we going to do? I mean, we’re not just talking passwords into systems, it’s knowing, knowing the business process, knowing everything that was going on, everything that they were working on, you know, how do we deal with this?

Carolyn Norton 36:37
Absolutely. You know, there was, and it’s still probably prevalent right now. But if key people in the organization leave, it’s, that’s that’s a possible disaster to your point, you know, what knowledge are they carrying off with them? What information, you know, that’s not tangible, right, is possibly leaving? And what can an organization do to, you know, make sure that, you know, that proprietary piece of data stays around for others to enjoy and continue on as that person moves on to their next? That’s absolutely a factor services, utility people people’s a big part of all this, right? How does that how does that get incorporated? And I asked earlier, okay, we got a plan, we’re rolling it out. Is that also when we would get people and I mean, people that weren’t involved in the active conversations, their inputs to the plan, like I’m just, I’m just I’ll use was it a janitor? I’m just a janitor. And now I’m, I’m being told about a plan is that one, I can also give my input is that where is there an opportunity to also call that out?

Daryl Moll 37:49
I think that’s part of the communication. And an important part of the communication is, hey, you know, while you may have not been an initial respondent in developing of these, you know, first version of this plan, every single person who, you know, is a part of this organization or, you know, has any vested interest in this organization has a, you know, vested interest in this plan. So, please, you know, bring all ideas to, you know, to, you know, X, Y or Z, you know, and have that that identified list of who they’re contacting. I think that’s a very good call out point.

Carolyn Norton 38:21
So we got the plan. Again, we’re, we’re feeling like a very prepared organization. Now I have my plan, I’ve communicated, I’m getting support from others that may not have been involved initially. Now, the next step is, and I heard many allude to this, but we’re gonna test out the plan. Am I envisioning some binder, and I’m thinking of TV world where you pull out a binder. And you know, we randomly pick an exercise, a hurricane comes into, you know, the area, and we’ve been ordered to evacuate. Go. Is that how you test a plan? How do you test a plan? What what does that look like? How does how does an organization envision it? Is it the binder or is it something else?

Sam Miller 39:08
I go to a binder but my my generation likes paper.

Rich Fowler 39:12
I’m thinking of the Apollo 13 movie and the mission. And yeah, it was it was tables full of binders. And if this happened, this is what we do. And there’s there’s a procedure and there’s a process and there’s people to talk to. And I’m betting that Darryl and Eric would say there’s a better electronic way to do it.

Carolyn Norton 39:29
Sure, sure. I like binders. I like binders.

Eric Robertson 39:34
Typically, it’s already been communicated to everybody. There’s going to be a place where it’s stored within some sort of intranet, let’s say SharePoint, where somebody can go and see what’s going on. Where it kind of outlines exactly what’s your business continuity plan, a lot of other organizations that you might be doing business with, they might not want to do business with you unless you have a business continuity plan already established anyway. So you’re probably also providing that to external party so that they see that you’re prepared. So a business continuity plan isn’t meant to be held close to the chest, it’s meant to be shared with everybody. It’s meant to have that feedback that you mentioned, Carolyn, you want to have everybody have that buy in so that people own it. And it’s not just one party. Again, it’s all the stakeholders, all the employees, and you’re going to be using it with your customers as well, because they want to make sure you’re doing the right things. But yeah, let’s put it into a binder because some people like paper, like Sam enrich,

Carolyn Norton 40:30
or let’s pretend the power goes out, and the Internet goes out. And we need a paper reference of some sort. That’s also possible. That’s part of disaster planning, right? That

Eric Robertson 40:41
is part of disaster planning, you are correct. There usually is a physical hard copy of it somewhere, especially with contact information as well, just in case your communication methods do go down. So yes, it is recommended to have a hard copy stored in multiple places in case of a large system or power outage, because even if you stored it somewhere in that system went offline. Now you don’t even have that plan. So yes, having a plan for your backup plan is also very important.

Daryl Moll 41:09
And to go back to your question, Carolyn, yes, testing is typically, you know, you are trying to in some way, shape or form safely recreate some of those situations that will be encountered for something like that, and having people run through the situation. I even had a Operations Officer walk in and pull the cable to the T one line to simply simulate an internet outage at that manufacturing company. And, you know, what do you do? And he was standing there, first thing I walked in, was walked over to go to start checking the equipment. He’s like, I did that. But let’s assume I didn’t, and it’s out what are we doing? And, you know, get everything going. And, you know, I mean, it’s, you know, that was a bit of an extreme, you know, as far as a test goes, but it was, you know, it really put that stress on, you know, the organization as far as, okay, what are we doing? How are we doing it? And how are we accomplishing what those, what we’ve designed, decided were priorities, and are in our business critical processes.

Carolyn Norton 42:15
And I’m assuming with the plan and testing the plan, we also have some metrics that we’re trying to achieve T right, we talked about RPO and RTO. But what else are we measuring? When we’re testing? We’re obviously we’re testing the plan. But what else should we be thinking about when we’re testing the plan?

Rich Fowler 42:32
We should be testing the blood pressure, the people that are trying to figure it out? I think that would be hilarious.

Carolyn Norton 42:39
We should everyone should have some sort of monitoring device to see how they do under pressure. Absolutely Good.

Daryl Moll 42:46
Well, not necessarily. I mean, I kind of took it a little bit different the way he what he just said, but I mean, you are managing that stress levels. And, you know, do we need to reallocate people that absolutely can’t do anything or don’t need to do anything, because of what they were doing is a lower priority thing. We need to shift that manpower over to the, you know, the people power over to the, you know, these other critical functions that absolutely get stressed out and slammed because they’re having to, you know, they’re critical, you know, cog in the wheel. And they have, you know, a lot of stuff to do when whatever happens, but shifting that man, or people power around, really, you know, comes in handy. As far as that, you know, those are all things that you need to look

Amy McKie 43:27
on. The one thing that comes to my mind, I don’t know, just thinking about that is I think people often leave out customer service. And those are, you know, that’s the frontline of the company, a lot of you know, a lot a lot of lines of communication, especially if, you know, the company is not able to, you know, make production or the internet’s down or something. So maybe taking a look at that area as well.

Rich Fowler 43:50
I was in the military, the Marine Corps specifically it is every man or rifleman. So when it hits the fan, everybody grabs a gun and goes to the front line, your customer service example is exactly correct.

Sam Miller 44:04
You know, there’s actually a flip side that I experienced to an outage, I was recently renting something, not a car, but just in a store renting something. And their systems were down. And the two people standing at the desks were panicked, because the whole business, it’s like a franchise, franchise franchise model. And the entire business was going down. And they were completely panicked. And because they had an app that was available for consumers to download to the phone, anybody who had a phone and was able to run their app was able to keep their business running. So going through a disaster planning like this really can give birth to new ideas for the company. So that you literally create A better customer service. I mean, because they had that app, they actually have new ideas about what to do with that app now because of the outage. And it’ll result in better customer service.

Carolyn Norton 45:14
That’s a crazy, amazing story. And you’re right. Sometimes when we’re put against the wall, it really does drive really cool innovations. And it can help an organization be that much more efficient or better.

Eric Robertson 45:26
Were you at Blockbuster.

Sam Miller 45:32
And it was on the palm pilot that his Nino, remember the little antenna?

Eric Robertson 45:40
Yeah, just checking out your stylus,

Sam Miller 45:43
we could board an American Airlines flight and see the news.

Carolyn Norton 45:50
So taking everything we’ve learned, I think I can summarize it in my mind as what do you do if? What do you do? I mean, really, that’s what we’re trying to really investigate and figure out how does an organization survive anything that can be thrown at them, whether it’s small, medium, or large? What do you do, and then I would take a step further and say, if you you’ve taken the opportunity to listen along with us and learn what you can do as an organization to better position yourself, whether you’ve already had a plan, whether you’ve thought about a plan, whether you have a plan, I challenge you to challenge the plan, think differently. I hate to lean on this a pandemic really shaked up a lot of gaps that organizations didn’t have to face. And I’m sure there’s other challenges that come will come. And some of the, you know, known suspects will come back around. So I challenge you to challenge the plan. When you have a plan, challenge it think differently. Think of other scenarios that can happen. That way you really can get through all the gaps that may come from the plan.

Rich Fowler 47:07
I guess if it’s if it’s my turn, I I come from the self defense world. And we always talk about a plus one. So if you’re, if you’re if you’re out there, and you’re dealing with a you’re dealing with a bad thing, which a disaster would qualify as a bad thing. What’s the plus one? What’s the one thing you didn’t count on? Are you ready for the thing you weren’t expecting? The answer is always no. So you’re always thinking about what’s my plus one? In the self defense world as an additional attacker? Is it a weapon? Is it something else in the disaster recovery world? What is it? Is it power? Is it internet? Is it fishing? Is it credit card? Is it something else? I don’t know, we have to determine what’s our plus one, figure it out and then plan for the next one.

Daryl Moll 47:51
Rich? I mean, I agree. I mean, everybody, you know, it comes down to starting the process. And again, a phased approach, have a plan. And then, you know, defining those, what are those defining and prioritizing those business critical processes, defining what your threshold for downtime is, and what the budget for, you know, having a budget for what you can, you know, use to overcome that downtime, defining what those actions are needed to continue your business, testing that plan, and then learning and adjusting from that plan. I mean, the those are the high level key areas that are critically important to, you know, a disaster recovery plan.

Eric Robertson 48:33
Yeah, and it’s just to remember, there’s no, there’s no way to reduce your risk to zero. You can prepare for disasters to limit their impact. And the risk that drives your business continuity plan, it starts with analysis, and that analysis never ends. So just test, test and test some more, because you’re going to find where your where your plan has issues. And you’ll be able to identify those gaps. So ask yourselves those three questions to start, what needs to be protected, what might disrupt it and how and what is the impact if it gets disruptive? And just remember, it’s not if but when a disaster will strike. Make sure to be prepared as best you can.

Rich Fowler 49:09
So Eric, is your role kind of like a weatherman because he keeps saying there’s no way to get to zero so as long as you’re right 30% of the time you’re in the Hall of Famers that what I’m hearing you go,

Eric Robertson 49:17
it’s gonna happen eventually. Time. Time never stops.

Sam Miller 49:25
Yeah, so at one point, I brought up y2k, and we’ve all brought up the pandemic. And I’m thinking about the difference between the two. And the the main difference was with the y2k thing, we all knew what was coming. And we could respond to that one thing that was coming. And with the, with the pandemic, if you were a researcher, if you saw stuff online, in 2018, you kind of saw that there was a risk of things and that kind of stuff, but in many cases, we had no idea it was Coming. And it came. And I think people need to approach disaster planning. And putting together a group internally to put together that book we’ve been describing with an urgency that we, they have to expect that it’s coming, something is coming. So there needs to be an urgency attached to this, as opposed to not just, oh, you clicked on that email, there are so many things that can happen in an organization that could have been preventable, if there was planning around the most important processes at the company. So those are my final thoughts. I really look past,

Amy McKie 50:45
you know, the last several, you know, episodes that we’ve done, kind of the thing I’ve always taken away with clients and working with clients over the years is, you know, again, having that business continuity plan, but really taking a look at as a business owner, as a leader as your company. What, what’s your what’s your risk that you’re, you know, willing to put up? How much and whether that’s, you know, money, is it people that, you know, how much are you willing to lose, by not having some kind of DR plan, business continuity, you know, whatever we want to call it in place. And that’s, again, on several different levels, whether it’s technology, whether it’s people, you know, whether it’s machines so on, it’s just how much are you willing to lose? So that’s, that’s kind of what I’ve always taken away with clients. And when working with them, it’s, you know, okay, you’re a small company, maybe you can afford to lose this much more, rather than, hey, a larger company. So but But everybody’s different. So when everybody’s business is different. So that’s been kind of my takeaway, and kind of closing thoughts over all this. So thanks for joining us.

Carolyn Norton 52:08
Creating a disaster proof organization is no easy task. It’s a continuous process with multiple challenges. But it’s the difference between living to fight another day, or becoming a headline in the news. when a disaster strikes. It’s not just the organization that’s affected, but it’s the people within it too. That’s why people, your people are the single greatest asset to fight against a big disaster from the crew at campfire 365. We sincerely thank you for listening to our first season, and we hope you’ve gained greater insights into disaster recovery and business preparedness. Until next time, this is your host, Carolyn Norton. Wait, wait, wait, wait. The campfire isn’t over yet. We’ve got more stories. So go ahead and grab another bag of marshmallows because season two is just around the corner. And we’re going to be talking about digital transformation. What is it, the business benefits around it? How to get started and so much more. There’s so much to talk about when it comes to digital transformation. start prepping your s’mores and get ready to start a brand new season off campfire 365