I do.
Built my bench & set up my gear in house's small bedroom.
Keep the powder in an old foot locker, figure if I'm here and a fire starts I'm going to be fighting it, if it's brush fire coming I put the locker in whichever truck I'm going to drive off in if we have to.
If fire starts while we're not here, it doesn't matter where powder, primers, ammo is stored, it's burnt up.
We live rural, nearest fire station is over 8 miles away, if any building catches fire around in this area it most likely is pretty burnt up before first fire truck arrives.