Channel estimation plays an important role in underwater acoustic communications, with applications in phase-coherent modulation schemes and preparation of replay channels. Quantifying the estimation error is a challenge, however, as the true channel remains unknown. This paper introduces the channel replay error (CRE) as the difference between a test signal passed through an at-sea or synthetic ocean channel, and the same signal passed through the replay channel. The CRE is investigated for popular probing waveforms and estimation methods, including correlative channel sounders and (sparse) adaptive algorithms. By necessity, the test signal for at-sea data is the probing waveform. This potentially reproduces any type of in-band acoustic power, including noise, and can yield an overoptimistic CRE. Synthetic channels allow CRE computation for arbitrary reference waveforms, which yields valuable insights.