Differential Analysis of X86-64 Instruction Decoders

EasyChair Preprint 5095

Abstract

Differential fuzzing replaces traditional fuzzer oracles like crashes, hangs, unsound memory accesses with a difference oracle, where an implementation of a specification is said to be potentially erroneous if its behavior differs from another implementation's on the same input. Differential fuzzing has been applied successfully to cryptography and complex application format parsers like PDF and ELF.

This paper describes the application of differential fuzzing to x86-64 instruction decoders for bug discovery. It introduces MISHEGOS, a novel differential fuzzer that discovers decoding discrepancies between instruction decoders. We describe MISHEGOS's architecture and approach to error discovery, as well as the security implications of decoding errors and discrepancies. We also describe a novel fuzzing strategy for instruction decoders on variable-length architectures based on an over-approximated model of machine instructions.

MISHEGOS produces hundreds of millions of decoder tests per hour on modest hardware. We have used MISHEGOS to discover hundreds of errors in popular x86-64 instruction decoders without relying on a hardware decoder for ground truth. MISHEGOS includes an extensible framework for analyzing the results of a fuzzing campaign, allowing users to discover errors in a single decoder or a variety of discrepancies between multiple decoders. We provide access to MISHEGOS's source code under a permissive license.

Keyphrases: automatic test generation, differential fuzzer, differential testing, instruction decoder fuzzing, software testing

@booklet{EasyChair:5095,
  author    = {William Woodruff and Niki Carroll and Sebastiaan Peters},
  title     = {Differential Analysis of X86-64 Instruction Decoders},
  howpublished = {EasyChair Preprint 5095},
  year      = {EasyChair, 2021}}