As you said, it's true that Gandalf can't be directly linked to Gondor as Boromir or Faramir could; his nature is in fact boundless and he doesn't have a stable residence. Nevertheless, for the reasons mentioned above, he had better remain in Gondor, even if this doesn't respect perfectly the true characterisation of Gandalf. Other options would create for sure more issues than benefits.
Using your own words, Arnor will nonetheless never be able to acquire Gandalf the White. This is the real difference