Operating System Migration and Windows updates

Considerations for testing Microsoft Windows updates in organisations to reduce the associated risks.

By Steve Mellor, Test Architect - Roq

When deploying Windows updates, organisations can be reasonably confident application and device compatibility will be high. By the time an update is available for general use it will have gone through rigorous internal and external testing, culminating with the Insider Preview Program. Many applications will have gained application assurance from Microsoft and other major software vendors. In addition, device compatibility checks will further support this assurance process.

However, each organisation will be unique in terms of their own application and device portfolio. This means it is important that an organisation assess the risks that a Windows release may have on their own user community. This will allow them to fully understand any issues that may result from deployment. The primary areas of consideration when deploying new updates are:

• The impacts on applications, with specific focus on business-critical applications and those deemed high risk from a Windows update

• Device assurance, targeting device compatibility and potential performance degradation

• Windows build assurance, any Group Policy Organisation (GPO) rules and specific Windows customisation that an organisation employs

• The user experience of the deployment process and the impact on working time

Assessing Application Risk

The most “at risk” area and the primary focus of attention for new Windows updates will be application compatibility and compliance.

When taking into consideration the wider application portfolio, an example of risk categories may include:

Low risk

• Standard Microsoft applications with minimal or no configuration (Microsoft will certify version compatibility with Windows updates – known issues will be reported for older software versions)

• Widely deployed Third Party applications (Vendors will certify against Windows updates – compatibility and know issues for older versions will be reported)

Medium risk

• Third party applications that are heavily customised for an organisation

• Microsoft applications with bespoke templates and unique interfaces

• Internally developed windows applications still under active maintenance

High risk

• Legacy Vendor Windows applications that are out of support

• Legacy internally developed applications

• Problematic applications and OS features that have had known issues during previous Windows updates

Assessing Device Risk

The majority of organisations will have a range of Windows devices within their infrastructure estate.

During standard Windows updates, the risk of pure device compatibility issues is low. However, with the consideration of Windows build customisation or major updates, it is best practice to ensure that testing is conducted across a range of devices to provide as much coverage as possible. The selection considerations for devices to include in this testing should include:

• Most used devices, to provide the greatest user coverage

• VIP devices

• Business-critical devices

• End-of-life devices and peripherals that may have compatibility issues

A crucial consideration for devices is application performance between updates. Automated performance testing can compare the user experience of key application user journeys between updates to check for performance degradation. In addition, monitoring of key device metrics such as memory and CPU usage will provide early diagnosis of any potential device issues prior to user rollout.

Test Approach

Planning

• Create a catalogue of applications and features that require testing based on assessment of business criticality and application risk

• Identify any specific hardware tests that may be required after a Windows Update (Fingerprint readers, USB functionality, etc.)

• Highlight the key GPO customisations that require validating for each build release

• Identify test devices requirements – both physical and virtual devices.

• Plan test coverage vs. device matrix

• Plan time-scales and resources required for compatibility testing

• Identify and recruit volunteers for UAT

Release testing

Once the build has been prepared and packaged it can be deployed onto the allocated test devices to validate:

• The in-place upgrade deployment process - assessing user experience and time scales, via the in-place upgrade process

• The fresh build deployment process - assessing user experience and time scales when building a device with the latest Windows version from scratch

• Functional testing of the applications and features across the selected devices - a combination of automated regression testing, scripted manual testing and exploratory testing

• Application performance testing - baselined against the previous release

User Acceptance Testing

The tested release is now ready to be deployed to the early adopters within the business. Clear communication into this group will ensure they understand how to provide feedback and the time scales for responding. This testing isn’t part of a formal process but an opportunity to give the user group time to trial the release within the context of normal everyday use. This provides wider exposure to the application portfolio and real end user experience and feedback prior to organisational wide roll out. Upon successful trial user testing the release is available for large scale deployment via the selected deployment process

A Windows update test strategy should put be in place to address the associated risks and provide an efficient repeatable process to provide user confidence and ensure minimal disruption to the business community.

Steve Mellor, Test Architect – Roq

Business as Usual

The structure and deliverables from the release cycles provide significant benefit to the Windows BAU support in areas such as:

• Monthly security patches – pre-built automation for key business applications, Performance benchmarking, regression testing of key application and known production issues / hotspots

• Introduction of new device models – baseline performance benchmarking, key application testing

• Emergency hot fixes – automated regression cycles to speed up testing

For organisations to minimise disruption to their business and maximise performance, security and governance, it is critical to have robust processes in place to ensure updates are thoroughly tested and implemented across your IT infrastructure. The effort to ensure this as best practice is far outweighed by the benefits to your operations and user experience across your whole IT estate.

If you would like to discuss this further topic, or find out more about our Windows device services please reach out to us at ask@roq.co.uk.