Collective Intelligence Benchmarking: What It Can Mean For Your Next IT Project

Congratulations, you have set up a whole virtualized infrastructure for a batch of your users and the completion of that project adds it to your list of successful projects that have improved the environment. But success in anything, including IT, is relative. So how do we prove the project has been successful?

The right way of knowing and measuring the benefits IT projects bring organizations is a disputed concept within enterprise IT. However, despite how it’s done, it’s a valuable concept to understand what was or is currently being experienced within your own IT environment, as well as what other companies within your industry are also experiencing. Through these comparisons, you can ensure that your environment is continuously improving and that IT isn’t placing your organization at a competitive disadvantage.

Some monitoring tools offer built-in components for baselining key IT metrics to help monitor IT projects. Gartner categories this functionality as collective intelligence benchmarking (CIB).

Free Download: Collective Intelligence Benchmarking with SysTrack

What Is Collective Intelligence Benchmarking?

The term “collective intelligence” is used across disciplines. Wikipedia defines it as “shared or group intelligence that emerges from the collaboration, collective efforts, and competition of many individuals and appears in consensus decision making.” It encompasses shared intelligence between humans and computers as well.

CIB applies collective intelligence principles to IT monitoring to aid decision-making. CIB tools provide IT teams with larger and more diverse data sets to help teams understand how their environment is performing in a wider context.

According to Gartner,

“Collective intelligence benchmarking (CIB) is an emerging and increasingly popular approach being adopted by monitoring tool vendors to ascertain a baseline of performance for a given service or application, based upon aggregated (often network) data from hundreds/thousands of end users. The aggregated data can be internal or external, although shared datasets from third-party end users is anonymized.”

Essentially, CIB allows you to easily compare performance baselines against your own historical data (internal benchmarking) or against anonymized data from other companies (external benchmarking). Items that are useful to compare include metrics like CPU usage and user experience as well as operating system composition, physical/virtual desktop composition, and device age.

While CIB sounds interesting enough, why bother benchmarking?

Why Use Collective Intelligence Benchmarking?

CIB can be used in many ways including:

• Project Planning: Perhaps you’re interested in a migration project, but you’re not sure if it’s worth the effort. CIB technologies can show you how other organizations who have migrated are performing so that you can baseline what success should look like for you, beyond a simple “launched” or “in-progress” status.
• Staying competitive: Have insight into how other IT teams are performing both within your organization and outside your organization. With that visibility you can understand how your peers’ environments are performing, being used, and built (OS, key metrics such as CPU, etc.) and in turn, know where you need to make improvements.
• Budget Rationalization:  Use internal and external peer data on their environment performance and usage to make intelligent budgeting decisions. For instance, if you see a large percentage of your industry peers with virtualized systems and your team has been considering moving to VDI recently, this piece of information can be a helpful factor in making that decision.
• Asset Management: See the age of other enterprises’ technologies, like hardware and operating systems, so that next time you are asked about a laptop (or other) refresh, you can make a decision that considers where you stand relative to your peers.

But maybe you’re just curious to see where you personally stand as far as resource consumption compared to your coworkers in your company. As stated by Gartner, CIB can really be practiced in two main ways, an external comparison against other companies in your industry and an internal comparison, comparing yourself against others within your enterprise.

[Want to hear more about the benefits of CIB? Listen to the latest episode of the Lifeguard IT Podcast, “Using Benchmarking to Meet IT Goals + A Wild Use Case for IoT,” on iTunes, Google Play, Podbean, Overcast or Stitcher!]

Collective Intelligence Benchmarking Examples

Internal Benchmarking

Let’s say you’re an IT admin for the US region and you want to know at an enterprise level how your group is doing compared to other regions within your company. You notice that while your region has a pretty good end-user experience score, the EMEA region has an end-user experience score that is noticeably better. Since all the same metrics, thresholds, and tools are used to compare one internal group to another, you decide to investigate further. By practicing internal CIB, you are able to intelligently benchmark and compare yourself against others and adjust to improve your environment. You also can continuously compare region’s service quality against other regions and even against your own past performance since you can baseline and compare it against previous times.

External Benchmarking

Consider a scenario where you are debating migrating all users from Windows 7 to Windows 10. Not being sure whether it is a move that a) will negatively impact key resources in the environment, b) improve or worsen end-user experience, and/or c) make sense for all users, you might want to turn to your peers to see if they are on Windows 10 and how their environments are performing.

CIB is a useful method to compare yourself as a company and see how other companies within the industry are running. By using a CIB tool, you may notice that most people in your industry are already on Windows 10. You take note of other benchmark metrics such as boot and login time, CPU usage, system age and notice that they are even performing slightly better in those categories while having around the same system age. Coupling those findings with a comprehensive assessment of the technology and its fit with your current environment, you can make a more confident and well-informed decision about the migration. Using this knowledge as a general benchmark, you can intelligently begin to plan a migration to a Windows 10 operating system.

Read the White Paper on SysTrack Community

What Software Programs Facilitate Collective Intelligence Benchmarking?

Vendors in the CIB space include Lakeside (digital experience monitoring), Indeni (configuration management), Logz.io (log monitoring), Nyansa (network analytics), ThousandEyes (cloud/network monitoring), and Unify Square (unified communications monitoring). When evaluating CIB tools, consider whether they offer internal benchmarking, external benchmarking, or both; the breadth and depth of data provided; and the area of specialization.

SysTrack is Lakeside’s leading digital experience monitoring platform with native CIB features that enable benchmarking using a wide set of real endpoint datapoints. It allows you to reliably practice internal comparison because all the systems have the same in-depth metrics being collected and analyzed, removing any vagaries between comparisons. For internal and external comparison, the end-user experience score is standardized across SysTrack, using the same 13 KPI’s to calculate the end-user experience score. Between an enterprise comparison using Enterprise Visualizer, down to a group comparison using Desktop Visualizer, SysTrack provides a clear comparison across all variations. With the additional Community Visualizer feature, you can compare your company against other companies. Our Community Visualizer automatically includes valuable metrics that help you benchmark against others minus the guesswork.

Want to learn more about CIB with SysTrack? Request a demo here!