Skip to main content
Microsoft Security

Microsoft Threat Protection leads in real-world detection in MITRE ATT&CK evaluation

The latest round of MITRE ATT&CK evaluations proved yet again that Microsoft customers can trust they are fully protected even in the face of such an advanced attack as APT29. When looking at protection results out of the box, without configuration changes, Microsoft Threat Protection (MTP):

Beyond just detection and visibility, automation, prioritization, and prevention are key to stopping this level of advanced attack. During testing, Microsoft:

Microsoft Threat Experts provided further in-depth context and recommendations for further investigation through our comprehensive in-portal forensics. The evaluation also proved how Microsoft Threat Protection goes beyond just simple visibility into attacks, but also records all stages of the attack in which MTP would have stepped in to block the attack and automatically remediate any affected assets.

While the test focused on endpoint detection and response, MITRE’s simulated APT29 attack spans multiple attack domains, creating opportunities to empower defenders beyond just endpoint protection. Microsoft expanded defenders’ visibility beyond the endpoint with Microsoft Threat Protection (MTP). MTP has been recognized by both Gartner and Forrester as having extended detection and response capabilities. MTP takes protection to the next level by combining endpoint protection from Microsoft Defender ATP (EDR) with protection for email and productivity tools (Office 365 ATP), identity (Azure ATP), and cloud applications (Microsoft Cloud App Security [MCAS]). Below, we will share a deep-dive analysis and explanation of how MTP successfully demonstrated novel optic and detection advantages throughout the MITRE evaluation that only our solution can provide.

Incident-based approach enables real-time threat prioritization and remediation

Analyzing the MITRE evaluation results from the lens of breadth and coverage, as the diagrams below show, MTP provided exceptional coverage for all but one of the 19 tested attack stages. This means that in real life, the SOC would have received alerts and given full visibility into each of the stages of the two simulated attack scenarios across initial access, deployment of tools, discovery, persistence, credential access, lateral movement, and exfiltration. In Microsoft Threat Protection, alerts carry with them rich context—including a detailed process tree showing the recorded activities (telemetry) that led to the detection, the assets involved, all supporting evidence, as well as a description of what the alert means and recommendations for SOC action. Note that true alerts are attributed in the MITRE evaluation with the “Alert” modifier, and not all items marked as “Tactic” or “Technique” are actual alerts.

MTP detection coverage across the attack kill-chain stages, with block opportunities.

Figure 1: MTP detection coverage across the attack kill-chain stages, with block opportunities.

Figure 1: MTP detection coverage across the attack kill-chain stages, with block opportunities.

Note: Step 10, persistence execution, is registered as a miss due to a software bug, discovered during the test, that restricted visibility on Step 10—“Persistence Execution.” These evaluations are a valuable opportunity to continually improve our product, and this bug was fixed shortly after testing completed.

The MITRE APT29 evaluation focused solely on detection of an advanced attack; it did not measure whether or not participants were able to also prevent an attack. However, we believe that real-world protection is more than just knowing that an attack occurred—prevention of the attack is a critical element. While protections were intentionally turned off to allow the complete simulation to run, using the audit-only prevention configuration, MTP also captured and documented where the attack would have been completely prevented, including—as shown in the diagram above – the very start of the breach, if protections had been left on.

Microsoft Threat Protection also demonstrated how it promotes SOC efficiency and reduces attacker dwell time and sprawl. SOC alert fatigue is a serious problem; raising a large volume of alerts to investigate does not help SOC analysts understand where to devote their limited time and resources. Detection and response products must prioritize the most important attacker actions with the right context in near real time.

In contrast to alert-only approaches, MTP’s incident-based approach automatically identifies complex links between attacker activities in different domains including endpoint, identity, and cloud applications at an altitude that only Microsoft can provide because we have optics into each of these areas. In this scenario, MTP connected seemingly unrelated alerts using supporting telemetry across domains into just two end-to-end incidents, dramatically simplifying prioritization, triage, and investigation. In real life, this also simplifies automated response and enables SOC teams to scale capacity and capabilities. MITRE addresses a similar problem with the “correlated” modifier on telemetry and alerts but does not reference incidents (just yet).

Figure 2: MTP portal showing 2nd day attack incident including correlated alerts and affected assets.

Figure 2: MTP portal showing 2nd day attack incident including correlated alerts and affected assets.

Figure 3: 2nd day incident with all correlated alerts for SOC efficiency, and the attack incident graph.

Figure 3: 2nd day incident with all correlated alerts for SOC efficiency, and the attack incident graph.

Microsoft is the leader in out-of-the-box performance

Simply looking at the number of simulation steps covered—or, alternatively, at the number of steps with no coverage, where less is more—the MITRE evaluation showed MTP provided the best protection with zero delays or configuration changes.

Microsoft believes protection must be durable without requiring a lot of SOC configuration changes (especially during an ongoing attack), and it should not create friction by delivering false positives.

The chart below shows Microsoft as the vendor with the least number of steps categorized as “None” (also referred to as “misses”) out of the box. The chart also shows the number of detections marked with “Configuration Change” modifier, which was done quite considerably, as well as delayed detections (“Delayed” modifier), which indicate in-flight modifications and latency in detections.

Microsoft is one of only three vendors that made no modifications or had any delays during the test.

Microsoft is one of only three vendors that made no modifications or had any delays during the test.

Similarly, when looking at visibility and coverage for the 57 MITRE ATT&CK techniques replicated during this APT29 simulation, Microsoft’s coverage shows top performance at 95 percent of the techniques covered, as shown in the chart below.

A product’s coverage of techniques is an important consideration for customers when evaluating security solutions, often with specific attacker(s) in mind, which in turn determines the attacker techniques they are most concerned with and, consequently, the coverage they most care about.
Figure 5: Coverage across all attack techniques in the evaluation.

Figure 5: Coverage across all attack techniques in the evaluation.

MTP provided unique detection and visibility across identity, cloud, and endpoints

The powerful capabilities of Microsoft Threat Protection originate from unique signals not just from endpoints but also from identity and cloud apps. This combination of capabilities provides coverage where other solutions may lack visibility. Below are three examples of sophisticated attacks simulated during the evaluation that span across domains (i.e., identity, cloud, endpoint) and showcase the unique visibility and unmatched detections provided by MTP:

Figure 6: Golden Ticket alert based on optics on Domain Controller activity.

Figure 6: Golden Ticket alert based on optics on Domain Controller activity.

Figure 7: Suspicious LDAP activity detected using deep native OS sensor.

Figure 7: Suspicious LDAP activity detected using deep native OS sensor.

Microsoft Threat Experts: Threat context and hunting skills when and where needed

In this edition of MITRE ATT&CK evaluation, for the first time, Microsoft products were configured to take advantage of the managed threat hunting service Microsoft Threat Experts. Microsoft Threat Experts provides proactive hunting for the most important threats in the network, including human adversary intrusions, hands-on-keyboard attacks, or advanced attacks like cyberespionage. During the evaluation, the service operated with the same strategy normally used in real customer incidents: the goal is to send targeted attack notifications that provide real value to analysts with contextual analysis of the activities. Microsoft Threat Experts enriches security signals and raises the risk level appropriately so that the SOC can focus on what’s important, and breaches don’t go unnoticed.

Microsoft Threat Experts notifications stand out among other participating vendors as these notifications are fully integrated into the experience, incorporated into relevant incidents and connected to relevant events, alerts, and other evidence. Microsoft Threat Experts is enabling SOC teams to effortlessly and seamlessly receive and merge additional data and recommendations in the context of the incident investigation.

Figure 8: Microsoft Threat Experts alert integrates into the portal and provides hyperlinked rich context.

Figure 8: Microsoft Threat Experts alert integrates into the portal and provides hyperlinked rich context.

Transparency in testing is key to threat detection, prevention

Microsoft Threat Protection delivers real-world detection, response, and, ultimately, protection from advanced attacks, as demonstrated in the latest MITRE evaluation. Core to MITRE’s testing approach is emulating real-world attacks to understand whether solutions are able to adequately detect and respond to them. We saw that Microsoft Threat Protection provided clear detection across all categories and delivered additional context that shows the full scope of impact across an entire environment. MTP empowers customers not only to detect attacks, offering human experts as needed, and easily return to a secured state with automated remediation. As is true in the real world, our human Threat Experts were available on demand to provide even more context and help with.

We thank MITRE for the opportunity to contribute to the test with unique threat intelligence that only three participants stepped forward to share. Our unique intelligence and breadth of signal and visibility across the entire environment is what enables us to continuously score top marks. We look forward to participating in the next evaluation, and we welcome your feedback and partnership throughout our journey.

Thanks,

Moti and the entire Microsoft Threat Protection team

Related Links: