I haven't ever read a book with that as the premise. Would be a great read though.
One of the Isaac Asimov short stories (I forget which one) dealt with the beginning of a robot ecosystem, with a bird robot designed to only kill a specific kind of pest.
Your idea sounds a lot like the player race "Minions" from the old computer game "Ascendancy", if anyone remembers it.
They were a race of robots comprised of nanites that evolved from servents sent to colonise a world by their extragalactic 'masters', who never ended up following them.
The Minion society evolved and developed space travel, and encountered the other player races, etc.
Minions were my favourite race to play, as they always were successful in invasions of enemy worlds (something to do with the nanites able to self-replicate by converting any matter into more nanites, thus entirely overrunning a world in a few days).
Later on in SG1 the Replicators reminded me of the Minions in some ways.