In a way, i suspect that Prey was the nail in the coffin for DNF, it was clear after 5 years that both Prey (which remember was a quake engine game and was announced in 95) and DNF weren't going to be anything like the games originally promised, but while DNF kept giving us sneak peeks every 2 years or so, which frankly wiped out any remaining interest by converting it into a long running joke (see serious sam's comment about blondie) prey at least dissapeared until the release date.
Then prey pretty much failed (I liked it, but critically and commercially it was very poorly recieved) and 3drealms took a major beating from their stock holders over it. Following another poor release, it was only sensible to ditch the white elephant that was DNF, even after 12 years and hundreds of millions of dollars. (especially since they will have been given the option when prey failed to excite customers, of getting DNF out of the door soon or losing it, when 3drealms failed to get it finished, the shareholders killed it.)