In comparison to lock/unlock mechanism, it takes extra resources like cache to keep records of the transactions to resolve conflicts.

Are we guaranteed an order to whether transaction A or transaction B comes first since they conflict? I'm thinking that if the transactions are atomic, then the ordering could be in any order.

Yes. I would say generally the order of the transaction is not determined when there is a conflict, but it can be different depends on the actual implementation.