I suspect the transmission. Specifically the bearings for the counter gear set. The counter gear set is used in all gears except direct drive (fourth). When in direct drive (fourth), then the counter gear is more or less on slack (no strain) while all the other gears are spinning around the main shaft on their needle bearings.
My bearing failed in the center of the counter gear set, popped out, then the counter gear shaft broke in half. I drove all the way home in the only remaining gear, fourth, running stop signs, bogging and working the clutch.
Luckily it was very late at night. The cause was poor transmission lubrication of course, but at some point, it's too late for those bearings to just add more oil.
I suggest putting the car all the way up on jack stands and listening to the noise location from underneath the car and suspect you'll find it coming from the 'belly' of the transmission. If it's what I think, then you must replace the bearing or the transmission to fix the problem before it breaks all the way. You can also drain and check the plug for bearings or shrapnel.