Interrupt storm

In operating systems, an interrupt storm is an event during which a processor receives an inordinate number of interrupts that consume the majority of the processor's time. Interrupt storms are typically caused by hardware devices that do not support interrupt rate limiting.

Background

Because interrupt processing is typically a non-preemptible task in time-sharing operating systems, an interrupt storm will cause low perceived system responsiveness, or even appear to freeze the system completely. This state is commonly known as live lock. In such a state, the system is spending so much time processing interrupts that it is not completing any other work. Therefore, it does not appear to be processing anything at all, because of a lack of output to the user, the network, or otherwise. An interrupt storm is sometimes mistaken for thrashing, since they both have similar symptoms, but different causes.

An interrupt storm can have many different causes, including misconfigured or faulty hardware devices, faulty device drivers, or flaws in the operating system. Most modern hardware implements methods for reducing or eliminating the possibility of an interrupt storm. For example, many Ethernet controllers implement interrupt "rate limiting", which causes the controller to wait a programmable minimum amount of time between each interrupt it generates.

The most common interrupt storm is a faulty driver under an APIC (Advanced Programmable Interrupt Controller) where a device "behind" another signals an interrupt to the APIC. The OS then asks each driver on that interrupt if it was from its hardware. Faulty drivers may always claim "yes", but then proceed no further as the hardware attached actually did not interrupt. The device which originally interrupted did not get its interrupt serviced, so interrupts again and the cycle begins anew. Many operating systems, e.g. Linux, lock dead under an interrupt storm; others have mechanisms to avoid it. Only a kernel debugger can break the storm by unloading the faulty driver.

Many OSes implement a polling mode that disables interrupts for devices which generate too many interrupts. In this mode, the OS periodically queries the hardware for pending tasks. As the number of interrupts increase and the efficiency of an interrupt mode diminishes, an OS may change the interrupting device from an interrupt mode to a polling mode. Likewise, as the polling mode becomes less efficient than the interrupt mode, the OS will switch the device back to the interrupt mode. The implementation of interrupt rate limiting in hardware almost negates the need for such polling modes.

History

Perhaps the first interrupt storm occurred during the Apollo 11's lunar descent in 1969.

Considerations

Interrupt rate limiting must be carefully configured for optimum results. For example, an Ethernet controller with interrupt rate limiting will buffer the packets it receives from the network in between each interrupt. If the rate is set too low, the controller's buffer will overflow, and packets will be dropped. The rate must take into account how fast the buffer may fill between interrupts, and the interrupt latency between the interrupt and the transfer of the buffer to the system.

Interrupt mitigating

There are hardware-based and software-based approaches to the problem. For example, FreeBSD detects interrupt storms and masks problematic interrupts for some time in response.

The system used by NAPI is an example of the hardware-based approach: the system (driver) starts in interrupt enabled state, and the Interrupt handler then disables the interrupt and lets a thread/task handle the event(s) and then task polls the device, processing some number of events and enabling the interrupt.

Another interesting approach using hardware support is one where the device generates interrupt when the event queue state changes from "empty" to "not empty". Then, if there are no free DMA descriptors at the RX FIFO tail, the device drops the event. The event is then added to the tail and the FIFO entry is marked as occupied. If at that point entry (tail−1) is free (cleared), an interrupt will be generated (level interrupt) and the tail pointer will be incremented. If the hardware requires the interrupt be acknowledged, the CPU (interrupt handler) will do that, handle the valid DMA descriptors at the head, and return from the interrupt.

See also

• Broadcast radiation
• Inter-processor interrupt (IPI)
• Non-maskable interrupt (NMI)
• Programmable Interrupt Controller (PIC)