Red Hat and the Clone Wars VI: Obfuscating Kernel Code for Fun and Profit: Dissociated Press

In our last episode we talked about the origins of Oracle Linux. This time around, we’ll look at one of Red Hat’s responses to the threat posed by Oracle Linux. Specifically, Red Hat’s decision to “obfuscate” the kernel source delivered in Red Hat Enterprise Linux (RHEL) 6, and how it communicated (or didn’t) those decisions.

The kernel looms large

Worth a quick note, the Linux kernel used to be in the spotlight much more than it is today. Open source was making great strides, but it would’ve been premature to say that it had “won” or was truly mainstream.  But when “open source” was discussed, the kernel was at the forefront of the conversation. The Linux kernel has taken a backseat in the past decade or more in terms of attention and importance in the world of Linux distributions.

What I mean by that is that Linux, the kernel, was still maturing rapidly in the mid-aughts. New features had a clear impact on user features that was much more obvious, and kernel development and developments were followed much more closely. Hardware vendors and companies like Google were struggling mightily with how to contribute to the kernel, and in fact the kernel development process was getting bogged down.

It still makes news today when Linus Torvalds does something like release a kernel from a machine with Apple Silicon, but the kernel has largely become “boring” in the past 10 years. That’s a good thing! But it might not be clear today just how much focus and attention was on Linux kernel development in the RHEL 5 and 6 time-frame.

That brings us to the 2.6.32 Linux kernel and 2011, when people took note of how Red Hat was shipping its kernel source code.

What do you mean, obfuscate?

Before we dig in, let’s look at what Red Hat did… or in this case, didn’t do. What Red Hat didn’t do, or rather stopped doing, was breaking out its patches against the Linux kernel that it shipped. Instead, Red Hat shipped the source code as a tarball with all the patches applied already.

That’s totally compliant with the kernel’s license (the GPLv2), though some folks complained that it was “clashing with the spirit of the GPL.” Maximilian Attems, a member of Debian’s kernel team at the time, had some strong words for Red Hat.

The only real big bastard on the cool 2.6.32 “sync” is Red Hat. Red Hat Enterprise 6.0 is shipping the linux-2.6 2.6.32 in obfuscated form. They released their linux-2.6 as one big tarball clashing with the spirit of the GPL. One can only mildly guess from the changelog which patches get applied. This is in sharp contrast to any previous Red Hat release and has not yet generated the sharp and snide comments in press it deserves. Red Hat should really step back and not make such stupid management moves. Next to them even the semi-maintained Oracle “Unbreakable” 2.6.32 branch looks better: It is git fetchable.

There’s a little more backstory there, and I’m realizing that that’s probably another topic that deserves its own coverage.

Long story short: The 2.6.32 Linux kernel was a long term support (LTS) release. This was a kernel that got long-term maintenance, with Greg Kroah-Hartman (best known as “Greg KH”), as the maintainer of the -stable branch of the Linux kernel. This was an offshoot of his work with SUSE (owned by Novell at the time) and work on the SUSE Linux Enterprise Server (SLES) 10 kernels.

That kernel was chosen by most of the major distributions at the time for their next and/or long-term releases. As a side note that kernel was originally released in December of 2009 (2.6.32) and its final release (2.6.32.71) was EOL in March 2016.

Red Hat had also chosen the 2.6.32 kernel but wasn’t working straight off the upstream LTS kernel the way other distributions were. And it was shipping the source code, but was no longer breaking out the patches that would make it obvious what Red Hat had changed.

In a related move, Red Hat also started gating some of its knowledge base. This also irked people and led to grumbling about Red Hat’s practices. For the record I think I heard the first “Red Hat is the next Microsoft” complaint somewhere in the 2006 timeframe. It made as much sense then as it does now.

Speed bump

The obvious target for the move was Oracle and other enterprise Linux vendors that were offering to support RHEL. That’s not just speculation, Red Hat’s CTO (at the time) Brian Stevens said so in a blog post in March 2011:

Why did we make this change? To speak bluntly, the competitive landscape has changed. Our competitors in the Enterprise Linux market have changed their commercial approach from building and competing on their own customized Linux distributions, to one where they directly approach our customers offering to support RHEL.

Frankly, our response is to compete. Essential knowledge that our customers have relied on to support their RHEL environments will increasingly only be available under subscription. The itemization of kernel patches that correlate with articles in our knowledge base is no longer available to our competitors, but rather only to our customers who have recognized the value of RHEL and have thus indirectly funded Red Hat’s contributions to open source that will advance their business now and in the future.

As an open source company, Red Hat is held to high standards. We embrace this. In 2011 you can expect us to increase our investment in open source contributions, while simultaneously competing with companies who are threatened by the continuing disruptive advancement of open source in the enterprise.

You see, Oracle wasn’t the only corporate player to try to undercut Red Hat by supporting RHEL. Novell, then the owner of SUSE, also offered support for RHEL. The idea was that Novell would support RHEL and eventually coax customers over to SLES.

KSplice and the kernel

One of the reasons Red Hat might’ve wanted to obfuscate its kernel was to make it harder to sell a service it didn’t have at the time. Namely, the ability to patch and update the kernel without a reboot.

Oracle snapped up Ksplice in July 2011, a company that had tech to apply updates to a running kernel without requiring a reboot. After Oracle snapped up Ksplice, the company dropped RHEL support and said it planned to make Ksplice “a standard feature of Oracle Linux Premier Support,” though it did say it would honor existing customer contracts. Oracle also said it would continue the free service for users of Fedora and Ubuntu.

In fact, it appears Oracle still provides the Ksplice updates for Ubuntu. It has also extended its support to glibc and openssl “for all Oracle Linux versions that are covered for Oracle Linux Premier Support.”

Oracle’s announcements at the time said that the Ksplice git repo would be moved to oss.oracle.com, but I’ve been unable to find a git repo for KSplice. There is a tarball but it doesn’t seem to be current. If you’re following along, Oracle complained (and complains) about Red Hat’s not breaking out patches or publishing source in a way that is most open. Then Oracle acquired Ksplice and … stopped publishing patches. For a while, Jiri Slaby provided a repo on GitHub with more recent patches but that seems to have lost interest.

For a time, Oracle was providing a service called “RedPatch” that broke Red Hat’s source into patches again. The links to that project have been redirected to Oracle’s top-level “Open Source Projects at Oracle” page.

Seems familiar, no?

The RedPatch discussion on LWN in 2012 walked through a lot of the same things people are talking about now with Red Hat and its policy around RHEL and CentOS source releases. That is:

• Whether the RHEL subscription policies violate the GPL’s “no restrictions” clause (they don’t).
• What the GPL means by “preferred form of the work” (no, you don’t have to provide source in whatever version control system that upstream is using).
• Also, reminders that the FSF’s stewardship of the GPL as a license doesn’t give it standing to sue / step in when that license is used by other parties.

Note that LWN also covered Red Hat and the GPL pretty well shortly after the “obfuscated code” thing. I do want to call attention to one thing in the LWN coverage that amuses me, largely because (as noted) there’s a clause in the GPL that everybody pretty much ignores:

There is an oft-ignored GPL clause about modifications to consider here as well: “You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change.” Red Hat may well be technically violating that clause, but it is hardly alone in that as it is pretty much completely ignored by everyone, without any real consequences. But this GPL requirement is of little interest to most. What everyone really wants is the reason that the change was made (along with which changes to other files go along with it), and the GPL is silent on any requirement to provide that level of detail.

It’s easy to see how kernel developers would be upset by Red Hat’s actions to “obfuscate” its kernel patches. While Red Hat was never obligated to break out patches for inspection by external developers, its business decision impacted the development community that Red Hat was and is a participant in.

Chloe don’t know better

Worse still, Red Hat didn’t announce its intentions and had to be prodded into a statement that its changes were intentional. A couple of years’ difference now, and those lessons never learned^[1]. Here we are in 2023, and Red Hat has to be prodded into an acknowledgement about changes to its source code policy. While it’s well within its rights, Red Hat had to know that the change would be taken poorly and that delivering bad news by surprise is one of the least good ways to deliver bad news.

Then again, here’s the conundrum for Red Hat trying to span the gulf between open source communities and the cut-throat tech business. Red Hat saying “hey, we’re going to stop doing this” gives its competitors time to adjust strategy. In an industry where you’re only as good as last quarter’s numbers, those days and weeks matter.

As easy as it is to understand how the kernel community might feel like Red Hat had done them wrong, it’s also readily apparent that neither the upstream developers nor the larger community were the targets of Red Hat’s actions. And if the response is “we understand the business reasons, we just don’t care” then it’s easier to understand why Red Hat would choose not to go through the motions of a public notice beforehand.

It’s equally apparent that Oracle felt no need to live by the standards it would like Red Hat to live by with regards to source code, at least not across all of its products. The same community that’s expressing anger at Red Hat for failing to live up to their expectations and standards doesn’t seem inclined to demand those standards of Oracle or anyone but Red Hat.

Back to today

In the decade and change that has passed since the “obfuscated” kernel kerfuffle, Red Hat implemented similar technology called kpatch for RHEL 7, 8, and 9. That project does seem to be alive and well on GitHub.

However, in the intervening years a lot has happened to make the live patching technology less interesting. The rise of public cloud, Infrastructure-as-a-Service (IaaS), automation, Infrastructure as Code (IaC), DevOps, Linux containers and Kubernetes have all helped move us away from the big iron days of systems that must have eternal uptime.

That brings us to Red Hat bringing CentOS in-house. But first, we’re going to look at another competitor to RHEL that wasn’t just a Xerox of RHEL source code: Ubuntu. The next “Clone Wars” post will look at the early history of Ubuntu and how Canonical built it a massive, if not immediately profitable, following.

Previous entries

[1] While writing this I kept hearing the lyrics “those lessons never learned,” in Andrew Wood’s voice, so I took some creative liberties with the sentence and lyrics. No, I’m not sorry. If they didn’t sound familiar to you, then I would love to be the first to introduce you to Mother Love Bone‘s “Chloe Dancer / Crown of Thorns.” You’re welcome.