I went passive using a Athlon XP mobile 2500+ and a Thermalright SP-97 heatpipe cooler for testing purposes...let me tell ya, it works..but if you don't have some kind of fans in the case...it gets pretty damn hot!
Pasive cooling is tough on hardware...unless you are set on COMPLETE silence..using low RPM large diameter fans at least on the case is gonna keep your hardware much happier..with or without a
CPU fan.
Most 120MM fans at low rpm are inaudable unless you stick your ear next to it...
Most coolers you find with a 80MM or larger fan at low rpm will do the job and
won't be noticable once in a case..I assume you had a loud crappy cooler on there?