The Physics of the Accelerating Universe (PAU) camera is an optical narrow band and broad band imaging instrument mounted at the prime focus of the William Herschel Telescope. We describe the image calibration procedure of the PAU Survey data. We rely on an external photometric catalogue to calibrate our narrow band data using stars that have been observed by both datasets. We fit stellar templates to the stellar broad band photometry of the Sloan Digital Sky Survey and synthesise narrow band photometry that we compare to the PAUS narrow band data to determine their calibration. Consequently, the PAUS data are in the AB system as inherited from its reference calibrator. We do several tests to check the performance of the calibration. We find it self-consistent when comparing repeated observations of the same objects, with a good overall accuracy to the AB system which we estimate to be at the 2% precision level and no significant trends as a function of narrow band filter or wavelength. Repeated observations allow us to build a spatial map of the illumination pattern of the system. We also check the wavelength dependence of the calibration comparing to stellar spectra. We find that using only blue stars reduces the effects of variations in the stellar template fitting to broad-band colours, improving the overall precision of the calibration to around 1% and its wavelength uniformity. The photometric redshift performance obtained with the PAUS data attests to the validity of our calibration to reach the PAUS science goals.