If we replaced synchronized with atomic in this example, we would run into a deadlock of sort, where one of the two threads would get stuck in the while loop infinitely. Since the atomic blocks must either execute completely or not at all, once one of the two threads reaches the while loop, they will never be ale to observe an update of the necessary flag that would cause loop termination. Thus, we have neither thread able to make progress past the while loop.

I wholeheartedly agree with @nickbowman's analysis of the situation. I just wanted to add some terminology from slide 8 where we talked about the semantics of transaction memory. The reason why there is deadlock is because transactions are isolated, which means that no other processors can see intermediate results (i.e. the writing to flagA and flagB). This is one of many differences between transactional memory and lock-based programs; with lock-based programs, processors are not isolated and can observe intermediate results.

I agree with @kevtan. I think precisely, the problem is not atomicity but isolation.