The balloons almost certainly run on hot air, not helium.
I thought they were hydrogen balloons? - certainly not helium to my way of thinking, because you couldn't generate helium on the fly to re-inflate a damaged balloon whilst you can generate hydrogen, hot air or steam. Whatever the lift gas is, there's a hefty dose of Handwavium mixed into to get the physics to work.
But the question posed by the OP remains valid. Whatever the balloon is filled with, it's lighter than air - be that hydrogen, helium, hot air or steam.
If the balloon is not highly pressurised, there's likely to be local deformation around the point of damage, and a generalised deflation from the bottom up. Unless there's a rigid skeleton, in which case the deflation will be inwards, between the ribs/spars/struts. The way the Junker deflates suggests a basic spar and rib skeleton, with inflated ballonets pushing against an elastic outer membrane. The way the Spire/Galleon deflates is just odd, now you come to mention it.