If you download something, they can track you. Remember that your connection to a p2p client, etc goes through many intermediaries on the way. Try tracing the route from yourself to any IP. You have to go through your provider, their provider, etc, etc. If anyone along this path (or who manages to work their way into this path) wants to check out what you're doing, who's going to stop them?
Then again, they don't even need to do that. They could be scanning you right now and you wouldn't even know it!
It may seem like you are far from prying eyes on your computer, but it isn't that hard to get into someone's computer if you are determined to.