Aamdahl's law is relevant here; our payoff decreases from the increased parallelization as other parts of the computation start to take up a bigger proportion of runtime.