I love my armrest - although my 45 mile round trip to work every day does not tend to need a lot of handbrake useage!
However, I can still operate it ok with it down! Although if doing a hill-start, I tend to throw it back to the upright position, especially with the selespeed box
I would take a test drive in a Stilo with an armrest first, before you fork out the cash and decide you don't like it. I find it OK on the motorway, but otherwise it just gets in the way.
To start off before i wasnt use to the armrest; it was abit of a pain around town, but after a while you just get use to it! The only time i may fling it up is on a hill start! V.handy for long journey; just adds to the comfort of ya motor!