I've built from git master, and it's the first time I've used the recording functionality. I was lucky I remembered that pressing Enter takes a screenshot for an area, but if we could perhaps add a simple way of letting the user know to press Enter to begin recording (or take a screenshot) that would help.
Yeah, the dimmed overlay could have some kind of message in it, like the one for screenshots does.
In the new UI, there's now a visible "Record" button after you drag a box to make a selection.