Here is a link to the parts diagram:
http://www.courtesyparts.com/pathfinder ... _4723.html
Scroll down the page untill you see 2003-2004 3.5L for your vehicle.
I would guess ether the pedal bracket is loose in which case simply retightening it should help, , or something is cracked / fatigued / broken, in which case you will will probably need to replace the bracket.
Check the throttle cable while you are under there and make sure it pulls smoothly, and returns to the resting position smoothly.
I don't think there there are any adjustments under there. The throttle cable adjustment is at the throttle body on the engine.