View on GitHub


A compiler and mutation testing framework for Pascal-86

Download this project as a .zip file Download this project as a tar.gz file

LLVM-P86 is a compiler and mutation testing framework intended for the programming language Pascal-86. Compared to a regular compiler, LLVM-P86 is able to create mutated versions of its input and encode several mutants into a single program. The compiler front-end is written entirely in Python, while the code generation provided by LLVM is written in C++.

Some of the most noteworthy features found in LLVM-P86 include:

About Mutation Testing

Mutation testing is an automatic fault-based testing technique that makes syntactic changes to a program under test in order to simulate real faults otherwise caused by a programmer. Similar to structural coverage criteria such as statement coverage, mutation testing is used to assess the quality of a test suite. After a syntactic change has been made, the program is referred to as a mutant that can either survive a test suite, or be killed by one. If a mutant is killed, it means that the test suite has detected the syntactic change and reported it as an error, resulting in an increased mutation score. If a mutant survives, it means that the test suite failed to detect the fault and the mutation score is decreased.

More information is available in my master thesis.

Getting Started

LLVM-P86 depend on the following software packages:

For people running Ubuntu 13.10, all packages are available via apt-get.

$ sudo apt-get install python2.7 python-ply python-llvm

Once the dependencies have been met, the source code needs to be prepared by issuing the following set of commands.

$ git clone
$ cd llvm-p86
$ python prepare

There are a few small sample applications available in the samples folder. To execute one of them directly using the LLVM JIT compiler, just type:

$ ./llvm-p86 -e samples/snippets/if.p

LLVM-P86 also ships with two Pascal-86 modules used to demonstrates how mutation testing can be put into practice. Unfortunately, LLVM-P86 is not able to link object files into a single binary, and thus gcc is required. From the project root folder, execute the following set of commands:

$ cd samples/triangle
$ ./ operator

where operator is one of the following:

To view each individual mutant, launch the small python webserver located in the root folder of LLVM-86 (preferably from a second terminal window)

./llvm-p86-webserver -r wwwroot/

Now, point your browser to localhost:8000.

Missing Language Features

Since LLVM-P86 was designed with a specific code base in mind, some language features are missing.