Financial losses in Medicaid, from Fraud, Waste and Abuse (FWA), in the United States are estimated to be in the tens of billions of dollars each year. This results in escalating costs as well as limiting the funding available to worthy recipients of healthcare. The Centers for Medicare & Medicaid Services mandate thorough auditing, in which policy investigators manually research and interpret the policy to validate the integrity of claims submitted by providers for reimbursement, a very time-consuming process. We propose a system that aims to interpret unstructured policy text to semi-automatically audit provider claims. Guided by a domain ontology, our system extracts entities and relations to build benefit rules that can be executed on top of claims to identify improper payments, and often in turn payment policy or claims adjudication system vulnerabilities. We validate the automatic knowledge extraction from policies based on ground truth created by domain experts. Lastly, we discuss how the system can co-reason with human investigators in order to increase thoroughness and consistency in the review of claims and policy, to identify providers that systematically violate policies and to help in prioritising investigations.