Travel? or at least Location, Location, Location.
For me an Adventure story has to have some element diverse locations.
An Action story can take place in one building, and maybe only a few floors of a building.
There are crossovers, of course.
James Bond films are action stories with a lot of adventure thrown in.
And Indiana Jones has a lot of adventures while dealing with some pretty mean action.