The issue is that it makes the tower either too strong when it is garrisoned by archers or too weak when it isn't. I'll take your example, Seleukos:
Very low health would make it (probably) balanced when it is garrisoned by archers. It would deal fairly high damage (much higher than now because of the archers) but be quite vulnerable, very much a glasscannon that would still need further support from melee units to protect it. But without archers, it would be very underwhelming for its cost: The damage would be ok, but not great; and it would still go down in seconds. That's not worth the spellpoints.
On the other hand, if the durability of the tower stayed the way it currently is, it would be perfectly fine without any archers. If you did garrison archers in it though, it would suddenly become much stronger than it used to be, which wouldn't be warranted at all.
In short: From a balance perspective, this change would bring only more problems. Gondor is, after all, not an archer faction like Lorien; and this suggestion would make archers pretty much mandatory for Gondor players who choose the left path in the spellbook. That doesn't mean the idea of making the tower garrisonable is dismissed entirely. Instead, removing the damage entirely and giving the tower other perks could be one way to go. Feel free to come up with other ideas, you've proven to be a bunch of very creative people after all^^