We present a search for periodic gravitational waves from the neutron star in the supernova remnant Cassiopeia A. The search coherently analyzes data in a 12 day interval taken from the fifth science run of the Laser Interferometer Gravitational-Wave Observatory. It searches gravitational-wave frequencies from 100 to 300 Hz and covers a wide range of first and second frequency derivatives appropriate for the age of the remnant and for different spin-down mechanisms. No gravitational-wave signal was detected. Within the range of search frequencies, we set 95% confidence upper limits of (0.7–1.2)× 10− 24 on the intrinsic gravitational-wave strain,(0.4–4)× 10− 4 on the equatorial ellipticity of the neutron star, and 0.005–0.14 on the amplitude of r-mode oscillations of the neutron star. These direct upper limits beat indirect limits derived from energy conservation and enter the range of theoretical predictions involving crystalline exotic matter or runaway r-modes. This paper is also the first gravitational-wave search to present upper limits on the r-mode amplitude.