Totally different idea: You already have a method to draw the bright rectangle, right? Then you also have all four corner coordinates of that rectangle. How about adding 8 more entirely black layers with 50% opacity, so that they cover everything that isn't inside the selection?
For the frame... well, just draw it in yet another additional layer
On a side note: I like your sample image
"When one person suffers from a delusion, it is called insanity. When a million people suffer from a delusion, it is called religion." (Richard Dawkins)