I have to agree with chad on this one.I know you aren't getting much help with this. I wish I could help more, but I am at a loss for specific solutions. It seems like decompression back through the intake. I'm not sure why that would happen. Could be a leaking intake valve allowing compression back through the intake. I assume you don't have the EVAP connected in anyway. That is the only other form of pressure/vacuum that I can think of that would be plumbed in to the intake.
I have to agree with chad on this one.
Possible low compression when you shut it off that it is rolling the engine backwards. It almost sounds like it sneezed? How many miles on the engine?
How did the car run before you took it apart?
Air should rush in since the inside of the manifold is under vacuum when you shut the engine off. You could test that theory by putting a piece of tape or a piece of string that dangles over the the TB and see which direction it goes when the whoosh happens.
Doesn't really matter though. If it's dirty, then it's probably a good idea to clean it regardless.