Michael Allen in his book Designing Successful e-Learning: Forget What You Know About Instructional Design and Do Something Interesting refers to a phenomenon he calls the “posttest paradox.” It appears based on a Google search of the term that he must have coined it, as no one else has used it (until now!). The phenomenon, however, has certainly been described. It’s the idea that while high stakes testing is generally intended to raise accountability and therefore motivation and therefore learning, it is possible it may actually decrease useful learning.
There are a number of aspects to this. One is that test preparation changes how participants engage with a course. Attending a course in order to get better at some real world task encourages learners to think about the concepts and principles and strategies taught in the class through the lens of how they will apply in the real world, which helps them encode information in a usable, retrievable way. The presence of a high stakes test on the other hand refocuses learners so they think about the content in terms of how it might be tested, which is a less useful way to encode the information in the long term, making it less accessible after the test is complete.
Tests also create anxiety, which can mask performance and inhibit learning.
In terms of adult learners, high stakes testing does not honor their ability or responsibility to engage with the material in the course that is most meaningful to them most heavily and with irrelevant material less. Anything could appear on the test.
It would also be fair to say that as tests become more high stakes, the administrative burden climbs. High stakes incentivises cheating, for instance, plus it becomes critical that records be flawless. The resources spent policing the test and keeping records are resources not devoted toward creating the best possible learning experience.
In short, post-testing as a means of creating motivation carries costs, many of them hidden.