A decade and change ago, in a past life, I was tasked with switching SELinux to permissive mode on the majority of systems on our network (multiple hundreds, or we might have gotten above one thousand at that point, I don’t recall exactly). This was to be done using Puppet. A large number of the systems, including most of our servers, had already been manually switched to permissive but it wasn’t being enforced globally.

Unfortunately, at that point I was pretty familiar with Puppet but had only worked with SELinux a very few times. I did not correctly understand the syntax of the config file or setenforce and set the mode to … Something incorrect. SELinux interpreted whatever that was as enforcing mode. I didn’t realize what I had done wrong until we started getting alerts from throughout the network. Then I just about had a panic attack when I couldn’t login to the systems and suddenly understood the problem.

Fortunately, it’s necessary to reboot a system to switch SELinux from disabled to any other mode, so most customer facing systems were not impacted. Even more fortunately, this was done on a holiday, so very few customers were there to be inconvenienced by the servers becoming inaccessible. Even more fortunately, while I was unable to access the systems that were now in enforcing mode, the Puppet agent was apparently still running … So I reversed my change in the manifest and, within half an hour, things were back to normal (after some service restarts and such).

When I finally did correctly make the change, I made sure to quintuple check the syntax and not rush through the testing process.

edit: While I could have done without the assault on my blood pressure at the time, it was an effective demonstration of our lack of readiness for enforcing mode.