I used the search to try to find an answer to my question, but only found this post from 2003 in which someone brings up the same issues I have, but there doesn't appear to be any response, so I figured I'd ask again.

We're a small bank, about $350 million in assets, one branch, about 35 employees total. I currently deploy patches using PatchLink, which I really like, but I do not really have any method to "test" patches before deploying them.

My current process is to deploy a patch to one PC in every department (operations, loan operations, tellers, etc), since they may have different software packages installed, then if everything looks ok, I deploy the patch enterprise wide.

This is great for patching desktop PCs, but my problem is patching servers. We don't have the luxury of a "test lab", so patching critical system servers is a harrowing experience to say the least.

How do other small banks handle patches? Do you have some way to test patches first, or do you just deploy the patches, then hope for the best, relying on backups in a worst case scenario (blue screen)?

Short of a test system, and the time to do it all, you may revert to making a mirror image, employ the patch, if there are consequences seen in the real world, uninstall the patch or write over the drive with the good image you'd just made. The risk you have here is potential saved data in the interim.

I'm not an IT expert here, but as a person with deep pockets and short arms, this is a community bank alternative. Hopefully others will offer solutions as well. Although, I think you're handling it well as it is.

Another thing is to determine the level of risk the patch repairs and hold off applying that until you hear about results elsewhere. That isn't a sure thing either because, as you noted, you'll have different apps, configurations, etc.

We use St. Bernard, which is a software application that allows you to remotely push patches out to workstations and monitor what patches are installed on the workstation, and which ones have not been installed. Personally, I think it's a great software solution that handles the patch management nightmare quite nicely.

Unfortunately, there are never any guarantees that the patches are not going to crash your workstations / servers. Microsoft indicates that they test everything before they release it; however, that doesn't mean it is not going to affect your specific workstation / server configurations.

A test bed environment is not all that expensive to put together. Get two workstations, one that has your workstation apps installed and one that has your system critical server application(s) installed. Push patches out to those computers, see how they're affected, if things look alright, start pushing out in small groups.

We have our 165 servers / workstations tiered in groups so we push out in small amounts over a week period. That way, if a group of workstations go down, it never cripples an entire branch or department.

I use the Microsoft WSUS to push out patches for Microsoft. Otherwise I do upgrades/patches per station. I'm in the same boat. We have enough computers for business use and no extra's for me. So WSUS allow you to assign groups. I have 3 groups. Branch 1, Branch 2, and my Test Group. I pulled a sampling of stations into the test group and I send it to those first. If all goes well then I send it to everyone and hope for the best.

I usually try to hit the forums like tek-tips.com and expertsexchange.com to see what others are saying.

First off, I am in no way affiliated with VMware, I worked for a bank in Massachusetts for 6 years doing information security and system/network administration.

With that said, if your main concern on the server end is how the patch interacts with your mission critical software (and it should be a high concern), then look into VMware http://www.vmware.com VMware workstation makes a nice, inexpensive, flexible test environment. Install your server OS in the virtual environment, install your applications, databases, etc. when you have a "baseline" of your "real" server you can take a snapshot to save the state. Now if you test a patch and it blows up, you can simply revert the test environment back to the known good state with the click of a button. This is also great for testing new applications for both the client and server and you can also generate movie clips of the VM session you are running, which is nice for presentations, documenting tricky installs, malicious code analysis, etc.

Now the one sticking point you could have, is you need to make sure your licensing of the operating system and software you install in the VM will cover you because the auditors will look for that.

Nick

I'd have to agree on the VMware. Having a test environment for patches is very important, and VMware is a very workable solution. It takes a little expertise to get it going properly, but it can be a powerful tool.

Ideally, you'd have an extra server and a few workstations to work on instead of doing it virtually, but you do what you can.

Virtual Machines can also be used in production environments as quarantine areas where you don't want a lot of people going. There are numerous other uses as well.