Quest windows

[Scotty law]

I have a 2006 quest. My windows are going crazy. When I turn the key off and open the doors both driver and passenger window will roll down. If I turn the key off and leave it in the ignition without opening the doors... Windows will roll down in about 30 secs. I have removed battery's from the key fobs to eliminate that as a problem. If I disconnect the driver window switch panel they will not roll down when the key is turned off. Also now the passenger window won't roll back up with the panel connected. The driver window will work properly when connected but both the driver, and passenger panel will not make the passenger window go back up. Any help appreciated. Thanks.