You'll have no problem towing with the Lexus. I have a BMW 328is with a manual transmission that I use as a tow vehicle. The owner's manual says not to tow anything, but it has plenty of power to tow a couple jetskis and more than enough braking power as well. I could never bare the thought of getting a truck cuz I can't stand driving them. For the amount of time you likely spend with a trailer behind your car you're better to stick with what you enjoy driving for the remainder of the 97% of time you drive without a trailer.
Like others have said, just be kind to the clutch on the loading ramps and you'll be fine. No need to rev it to the moon and ride the clutch up the ramp. The e-brake is definitely your friend on the ramps too to keep you from rolling back.
This past summer we had our car loaded to the max and it never seemed over-burdened at any time. A trailer with a SJ and an Octane as well as all our camping gear for a week.