Automotive Giant Chooses Roq for Platform Testing

The Client 

The client is an institution of the British motor industry, with a global presence and outstanding reputation. Renowned for its innovation and elegant designs, it has maintained its status for over a hundred years.

The client had recently moved to an enterprise-wide cloud-based single sign-on (SSO) solution and had serious performance issues across its global Google Mail estate.

One of the longer term strategic options to resolve the issue was to move to Microsoft’s ADFS 2 solution, but initially the client needed to improve the performance of the existing PING Federate solution (v6.5).

Roq was engaged to understand the limitations of the existing solution in order to set a baseline for its new solutions. Roq provided its Hosted Performance Testing Services to deliver the solution.

Roq’s Solution

The initial scope of the engagement was to baseline the performance of the Ping Federate solution, so that they could demonstrate improvements and justify the move to ADFS 2 . This set out the fundamental strategy of the solution and Roq developed a robust test approach to do the following:

• Understand the current performance issues of the PING solution through a defined set of performance tests covering 7 different scenarios (single servers, multiple servers, firewalls, loadbalancers etc.) and from multiple injection points

• Build and execute test scripts using Visual Studio (the clients own performance testing tool) – all this was delivered remotely from the Roq Test Lab

• Execute several cycles of tests to enable application and infrastructure tuning before baselining the revised configuration.

This approach proved very successful and there were some significant improvements in the PING solution (highlighted below) that provided the client time to plan the ADFS 2 implementation properly:

• Peak load experienced before testing (2 servers) showed only 1700 logins per 15 minutes. The changes recommended through the performance testing allowed a single PING Federate server to handle up to 7200 logins per 15 minutes (over 4 times as fast on one server)

• Memory leaks identified and resolved in ”Duration Tests” are now unlikely to occur even at sustained “2x peak load” usage on a single server, whereas before they were severely impacting performance.

Once this first set of performance tests were com-pleted, we were in position to start mapping out the performance testing approach for the ADFS 2 implementation. This was new technology and meant a complete change of scripts, although the same 7 scenarios were used, to ensure a fair comparison.

The performance testing team faced some different technical challenges due to the levels of authentication from Kerberos tokens and NTLM tokens for the Single Sign-On Process. Both had to be simulated through data driven tests with user names and passwords that could be deleted upon acceptance so as not to influ-ence results through caching.

The ADFS 2 performance test used the same ap-proach, with Visual Studio run remotely to prove the performance levels. The key results were:

• The ADFS solution could comfortably handle 16,000 logins per 15 minutes (which is more than 10 times the clients anticipated usage pattern)

• The ADFS solution was demonstrably 50% faster than the PING Federate solution.

The Outcome  

From a service point of view, Roq was able to influence and challenge the client early in the planning process – ensuring that the approach to performance testing was robust.

The solution was flexible enough to be done incrementally against configuration and technology changes and it provided the relevant information to make business decisions rather than purely IT decisions.

The evidence presented to the senior management team was used to support the reasons for its current position (in terms of poor service), the rationale for change, together with the evidence to justify the budget spend for implementing a new solution – which would provide them with longer-term strategic benefits.

If you’d like to find out more about how Roq can support your business, please email us at ask@roq.co.uk.