Consider Phlebas is my least favorite Culture novel. Which is to say I liked it, just not as much as the others. Probably makes sense to read it before Excession and Look to Windward, which make reference to the Idiran War, but definitely think The Player of Games is the best place to start for most readers.
Funnily enough I didn't read them that way and in fact Consider Phlebas was one of the last Culture books I read but I don't recall it being a problem when reading E and LtW.