Software development is a hard problem.
Books like The Mythical Man Month, Set Phasers on Stun and The Inmates are running the Asylum have all pointed out in their own way that creating software is hard. Fred Brooks focused on the problem of large complex projects, and the problems that face project managers, the other two remind us that even small projects can fail because we still are not able to create software that is both easy to use and powerful enough to do that tasks that we want to do with software.
Until we are able to understand why software development is such a hard problem, we are not going to make much beyond incremental improvement. There will always be a few projects that through the operation of blind luck across millions of projects that results in seemingly reproducible improvement, but the normal regression to the mean will correct that eventually.