Tales of Crisis IT: Work from Home, and FAST! 

By Rhonda Studnick Kaiser, Director of Customer Experience, HCL BigFix

You face many crises and challenges in 8 years being a part of, and then leading, the compliance management team responsible for more than 200,000 endpoints at a Fortune 50 corporation. With 100,000 users in 90 countries, I learned a lot about day to day patching as well as the craziness of crisis management when faced such things as security threats, data center outages, floods and hurricanes, all at massive scale.  

Leading the Compliance and Patching Engineering team as the Product Owner for BigFix, our team covered automated patching, distribution and deployment of new software on servers and end user machines of all types.  We also ensured that the business and security policies our organization relied on were in place and working 24×7 through continuous patch deployment and configuration management.  

Fast forward to today: Working from home has come to many companies as an urgent crisis, forcing what were always-in-the-office workers into their homes. Some businesses have it planned and are resilient, others not so much. No matter the size of your business, you want to ensure that you are delivering ready-for-business, compliant endpoints into the hands of your employees and that you are keeping them secure over time.  More than that, you want to ensure that your most important assets – customer information, product design information, financial information – is kept secure.

We all know that businesses were in different states of readiness for remote work pre-crisis versus what they have now in place or are starting to put in place, and we’ve all heard the difficulties some of those businesses are facing. 

As the Product Owner for BigFix, our team had already built our capability with the acknowledgment that our users needed to work anywhere they needed or wanted to be – VPN, disk encryption, and security and compliance capabilities were built into the basic stack delivered on day 1, and nearly everyone already had a laptop, though the availability of internet service varied by geography.   

From the start, our team was charged to ensure that our systems were appropriately managed, patched and rebooted, but we couldn’t do that without taking into account both the user experience while their laptop was being patched, the mission-critical nature of the server that was being patched and rebooted, and doing that while always ensuring that the network bandwidth that effort took did not otherwise impact productivity.  Our biggest mandate was to ensure that our users at smaller sites, and more importantly, our manufacturing and delivery capabilities were never degraded in the course of business. 

We also recognized that those users might be on the corporate network, even over VPN, but that they might also just be online taking college classes or helping their kids with homework while disconnected.  We had to build our infrastructure, processes and security safeguards with those challenges in mind. 

So, what can others learn from my experience? What can BigFix offer you if you’ve already recognized the reality of remote work? 

First, you’re still going to have to keep those endpoints up to date and you’re going to have to force patch and maybe even reboot those systems.  We all know that patching is not something that the typical user enjoys.  We used BigFix to quickly and automatically patch endpoints in the background, typically getting to 95% deployment in just a couple of days, and only rebooted systems when other user actions didn’t already take care of it.  Enabling our patching teams to “set it and forget it”, meaning that they could define downtimes and patching window durations was another key to our success.  We built tools on top of BigFix which guaranteed that the basic patching for more than 30,000 servers and multi-user devices was as simple as we could make it, but also ensuring that it was consistently occurring and within the timelines the organization defined.  A lot of what we custom built is now part of the basic features BigFix delivers today, in part inspired by the work we did.

BigFix also allowed us to operate with great flexibility, targeting specific sets of users/endpoints based on the specific threat, requirement, or characteristic of those users.  We were able to target those workers connecting over the Internet instead of on the corporate network via VPN when necessary, and we were also able to bypass those same VPN customers when the size of the updates coming out would degrade their performance or the performance of the VPN gateways.   

We used BigFix to stage large patches on the endpoints via the pre-caching mechanism on more than one occasion – think Windows 10 upgrades or the migration from Win 8.1 to Win10 – and then triggered the installation of the patch when the optimum conditions for that patching was in place.  We often excluded users from patching actions whose battery was low or who weren’t on AC power where the result of installing a patch and losing power mid-upgrade would cause their laptop to fail the upgrade…or become a really expensive paperweight. 

We also used BigFix to notify our customers when an incoming patch would require a reboot or required a specific program to be closed so they wouldn’t lose their work and cause IT’s image to be tarnished.  I really can’t tell you the infinite number of ways we took advantage of custom targeting in BigFix – quite frankly, we were able to come up with targeting choices on almost every occasion to meet the unique requirements of the software, the customer, or the global weather/network/crisis situation we needed to address.  Beyond that, knowing that we could rely on targeting, we were able to save both resource time and rework on many fronts because we did it right the first time.  BigFix reporting was always right there too, to confirm the work completed. 

How many of you walked into the office one morning in the middle of May 2017 to be confronted with an email in your inbox, or maybe you got a text while pulling into the parking lot, with something to the effect of, “You urgently need to tell me how many machines need MS patch MS17-0101  I need to update the CISO at 9 AM with that information!”  Yes, that was the famous WannaCry exploit.  We were quickly able to provide many dimensions of data to our business within just a few minutes using out of the box BigFix content, as well as quickly developed and tested BigFix analyses.  And with the BigFix community behind us using the BigFix Forums and our support within BigFix itself, we were able to share solutions and drive to deeper levels of information as the situation evolved.  Honestly, the only limiting factor was how long it took for the endpoints to be on the network to give us the feedback. 

With respect to work from home, most of the time our in–house client engineering team was able to provide our end users, no matter where they were located, the ability to quickly receive and set up a laptop.  We didn’t often get the opportunity to use BigFix to support the simple “phone home” enrollment which BigFix has recently introduced.  That said, it was so nice to know that we could quickly and easily get to those devices that were delivered to ensure they were compliant and keeping our data safe.   

The times aren’t easy, and we’re all facing challenges we may not have run into before.  I wish you the best on your journey to productivity and would love to connect if you would like some insight into these or other work from home use cases.   

For more information please visit BigFix.com 

Download the new BigFix Work From Home Solution Guide.