Short selling question.

Can one or more of you experienced hands explain a nuts-and-bolts issue with short selling?

As I understand someone that wants to short sell a stock “borrows” the shares from the owner in an attempt to push the share value down? If this is true who would want to loan out their stock just for the opportunity to watch it get decimated?

Do the brokerage houses (E Trade, Fidelity etc.) that hold your shares in trust have the ability to loan out your shares for a fee to your detriment? How then can the custodian insist they have a fiduciary duty to the owners of said securities by lending them out without specific consent to be shorted, potentially depriving the owner of value?