![ubuntu xenial rssowl ubuntu xenial rssowl](https://proyectosbeta.net/wp-content/uploads/2017/03/Gimp-2.8.21-en-Ubuntu-Xenial-16.04-LTS-imagen-destacada.png)
![ubuntu xenial rssowl ubuntu xenial rssowl](https://2.bp.blogspot.com/-NBpXW8dWOCg/VxlVhlRigqI/AAAAAAAATcA/h3R6GyStj3oO0BISd1ffsFmmbNS7kc4FQCKgB/s1600/drivemeca-instalando-ubuntu-xenial-desktop%25287%2529.png)
Maybe we just needed to run fewer apps on each machine, even though the rest of our monitoring showed plenty of room? Maybe some app was doing something very strange to fill up lots of kernel caches with unfreeable data? With some tips from my friend Nelson, I figured out exactly what kernel allocation was failing, and eventually found the right search terms to find an Ubuntu bug report. I wrote a custom check for the Datadog monitoring service so I could see how often the OOM killer was striking and so I could tell if my attempted fixes were working.Īt first, I operated under the assumption that we were doing something wrong. Fortunately, our overall system functioned and restarted the apps automatically, but our users weren’t happy to see unexpected and unexplained process deaths. We started noticing large hosted apps being killed even though the machines seemed to have plenty of memory. Most allocations are for a single page at a time, so the bug manifests rarely, but kernel stacks require a contiguous 16k allocation, so the bug occurs most frequently when fork()ing a new process. In particular, the bug occurs when the kernel attempts a so-called “higher order” allocation, meaning an allocation of more than one contiguous page (a “page” of memory on x86 is 4kb). In practice, this means that at random times, the OOM killer would strike at big processes when the kernel tries to allocate, say, 16 kilobytes of memory for a new process’s thread stack - even when there are many gigabytes of memory in reclaimable kernel caches!
UBUNTU XENIAL RSSOWL FREE
Unfortunately, a bug was recently introduced into the allocator which made it sometimes not try hard enough to free kernel cache memory before giving up and invoking the OOM killer. When you run low on memory, the kernel first frees memory in its caches the OOM killer only kicks in if this isn’t good enough. A healthy Linux system tends to have very little completely free memory: leaving RAM chips completely unused is wasteful, and the kernel will happily fill it with caches, especially of filesystem data. When a Linux system runs out of usable memory, a system called the “OOM killer” (for “Out Of Memory”) kills big processes until there’s enough memory to keep going. Right now, the default Linux kernel on a Xenial system contains a nasty bug. One of these rare problems happened in early January. The kernel team at Ubuntu does a great job of keeping the kernel versions they release stable by only backporting important bug fixes, but just as with any release engineering process, once in a blue moon a new bug shows up with a bug fix. With any software, updating frequently runs the risk of bringing in new bugs along with the desired bug fixes.
![ubuntu xenial rssowl ubuntu xenial rssowl](https://i.imgur.com/GtsVfDD.jpg)
And we keep up to date with security patches, especially for exploitable bugs in the Linux kernel. We use the most recent “Long Term Support” distribution: 16.04 LTS, also known as Xenial Xerus.
![ubuntu xenial rssowl ubuntu xenial rssowl](https://proyectosbeta.net/wp-content/uploads/2018/01/Terminal-imagen-destacada.png)
Galaxy, our hosting service for Meteor apps, runs user apps on Ubuntu Linux. Here at Meteor Development Group, we do our best to keep our systems up to date with the latest security patches. You know it’s a good day when you’re staring at kernel stack traces.