No, you don't.
You don't use the "Set height", but the "add height". If you make the brush 1000000 feet huge and click once, the whole map will get heigher. All of your cliffs, ramps or other things remain exactly the same - just higher or lower.
You can use that whenever you want - even at the very end after you've done everything else.