First when transaction costs are involved the trader has to make a tradeoff between return and risk. Continuous rebalancing/hedging could lead to infinite transaction costs but provides (in theory) a perfect hedge. Discrete hedging enables to minimize transaction cost but leads to hedging errors and more risk.
To find a price one must introduce an optimization criteria (e.g. a utility function)
As already mentioned it is hard to name "the one approach" that is done in practice. I will thus refer you to some texts to get you started on the topic.
Banks etc. will develop their own methods using results from research/academia.
Dynamic option delta hedge (FRM T4-14)
Good methods equal good money - so people will not readily part with their secrets. Knowing the theory and the reasoning behing some of the approaches will, however, enable you to make your own condlusions.
Delta Hedging Model using Monte Carlo Simulations – Assumptions
Some interesting papers/results on the topic:
For a good overview I also suggest the following Phd-Thesis - e.g. check the Bibliography.
Some general remarks: A method must not be sophisticated to work well.
There are different types of hedging approaches. Some traders might use algorithms other might trust their gut.
You should also be aware that the hedge itself strongly depends on the type of product and on the assumptions you made on the model/market.
A hedge of a plain vanilla call will differ depending on the model (e.g. B&S vs. Heston) An American option is hedged differently than a European one.
The model assumptions on the other hand depend on the type of product you want to price/hedge.
Also there is often a big gap between what is written in books and done in practice.
The productive solutions are frequently a mixture of different approaches presented in different sources.
Some quant stack exchange topics that you might find interesting/relevant: