No, it's easy as cake.
There's a screw inside the handle. It's probably covered up with a little tab. Remove that screw.
The whole rest of the panel is in one piece and held on by snap tabs. Gently wedged your hand under where the window goes, and pull straight out. Or you can do it from the sides, it really doesn't matter. The hardest part is getting that first clip. After that it goes quicker the more tabs you pop out.
Oh yeah, this should help too.
http://www.nicoclub.com/articles.php?id=248028