Re: An Open Letter to Red Hat: Guidelines for Fedora Core Releases

[Dean Henrichsmeyer <dean@beret.net>]

Bryan J. Smith wrote:
> Dear Red Hat:  
> 
> - For many know who I am ...
> 
> I have commonly been labeled as a "Red Hat Apologist" for the last 4 or
> so years.  As much as I hate the title, referring to the definition of
> "apologist," I have to admit it is myself in the strict interpretation.
> I regularly take the time to explain when, how and why Red Hat is the
> greatest influence in the GPL world, the crucial GPL work Red Hat
> provides and how every product-based decision by Red Hat has been
> prompted by another company, but still in the community interest.  I've
> integrated many, many other distros (and still deploy Debian and Gentoo,
> as well as SuSE Enterprise Linus Server), so I'm not just one of those
> "I only use Red Hat" types -- just one of those "I only trust Red Hat
> for many applications" types based on almost a decade of Red Hat's
> "under-promise and over-deliver."

OK.

> I was one of many who appreciated the change to GLibC 2.0 in Red Hat
> Linux 5.0, recognizing that LibC 5 and Red Hat Linux 4.2 support was not
> dropped until Red Hat Linux 6.0 so everyone was taken care of.  

I did too.

I
> appreciated the move to GCC 2.96/3, forcing C++ code to be ANSI
> compliant, and many other changes in Red Hat Linux 7[.0] onward.  I am
> also one of the very few that recognize that Red Hat tried to offer
> Enterprise SLAs on its single Red Hat Linux product line with Red Hat
> Linux 6.2"E", only to see SuSE Linux Enterprise Server (SLES) be
> released and more accepted by companies requiring Red Hat to follow
> suit.  And I understand all the issues and that lead to and can even
> appreciate the advantages for the community in the creation of the
> Fedora Project -- out of those with Sun and other commercial non-
> licensees who made the "Red Hat(R)" trademark an issue to the fact that
> Red Hat Linux as a product was not serving our interests anymore after
> Red Hat Enterprise Linux was introduced, and Fedora Core was a better
> move for everyone.

I agree it was, but they made the transition wrong.

> No Linux distribution has a history as proven, trusted and understood by
> its integrators.

I know where you're coming from but that's really an impossible 
statement to qualify. I would disagree with you, but my statement about 
another distribution wouldn't be anymore qualifiable.

> Fedora Core has seemingly continued this route in releases to date.  Red
> Hat Linux 10 became Fedora Core 1, and other than some very much needed
> changes that added 2 months to its release date, it is largely what I
> call "Community Linux 3.2" (CL3.2), the ".2" revision of CL3, which
> matches "Enterprise Linux 3" (EL3).  The ".0" and ".1" revisions would
> be Red Hat Linux (RHL) 8 and 9, respectively.  It also explains why Red
> Hat Linux 7.3 (CL2.3) is still supported by Legacy, alongside Red Hat
> Enterprise Linux 2.1 (EL2) commercially, while Red Hat Linux 8 (CL3.0)
> has been dropped.  Because we all know .2+ revision are the most stable
> and there is no sense in supporting earlier .0 or .1 releases.
> 
> - Is It Now Broken for Fedora Core 4+?
> 
> The same started to happen again with Fedora Core 2, which I refer to as
> CL4.0.  In typical Red Hat fashion, a new, major version starts with an
> Application Binary Interface (ABI) change of GLibC, GCC and, sometimes,
> the kernel.  Fedora Core 3 (CL4.1) then followed suit, 6 months into
> this new CL4 series, more programs had adapted to the new ABI, and more
> maturity found its way into the ".1" revision itself.  Nothing
> unexpected, as most Red Hat integrators had grown to love since Red Hat
> Linux 4.0 long-ago before the Enterprise line.
> 
> As a "Red Hat Apologist," I had found the CL labeling has calmed most of
> the critics, and several at Fedora Legacy had rather liked my unofficial
> labeling.  There was always no guarantee that a CLx.0 revision would be
> completely ABI compatible with a CLx.1 release, especially given that
> some major changes or "lessons learned" in a CLx.0 release could result
> in a CLx.1 change that broke some ABI compatibility (e.g., introducing
> the Native Processor Thread Library after CL3.0/RHL8 in CL3.1/RHL9, it
> was a very good, future-planning move).  But in general, the core GLibC
> and GCC were always maintained to help this, and it was trusted by many.
> 
> Now I see Fedora Core 4 on the horizon, and GCC 4.0 is being introduced
> in Test 1 (version 3.90).  At first glance, I have to assume CL4.1/FC3
> is now the "end-of-the-line" for CL4, but EL4 just came out.  At the
> same time, there are not so many radical changes to suggest it is a new
> CL5 series.  Again, I understand Red Hat makes no promises on Fedora
> Core, same as it did on Red Hat Linux before it, but at this point, I
> have to start scratching my head.
> 
> Is there going to be a 0-1-2 "Community Linux" relationship to
> "Enterprise Linux" or not?  Even if unofficial, it's never been about
> marketing, but the trust of long-time, staunch Red Hat supporters like
> myself to believe that Red Hat has a plan, even if it has to change at
> times to accommodate needs, concerns and to just "shouve the community"
> in the right direction.
> 
> - What I Would Have Liked to Have Seen ...
> 
> I'm here to be constructive, not merely to complain as so many Anything
> But Red Hat (ABRH) individuals seem to be doing without any solid
> statements (e.g., especially so-called "bugs" that plague other kernel
> 2.6 distros as well, not just Fedora Core 2/3).  For those of us who
> have been deploying Red Hat Linux on corporate networks for almost a
> decade, we have come to trust this model.  So as something who is seeing
> this trust become an unknown quantity, I beg Red Hat to at least clarify
> some details, even if unofficially.
> 
> I had hoped that Fedora Core 4 would be the CL4.2 I envisioned, the ".2"
> of the CL4 series which would go on to be the Legacy supported version
> alongside the EL4 commercial product.  That would set up for the new,
> major version in Fedora Core 5 to be CL5, which would match the next EL5
> in early-to-mid 2006.  It was my sincerest hope that this next ".0"
> revision would be labeled Fedora Core 5.0, and the following one Fedora
> Core 5.1, etc..., finally matching the product lines again between CL
> and EL.  Now I can understand if there is going to be no CL4.2 (see
> farther below), but I would like to see that reflected in the versions
> (again, see farther below).
> 
> Now I know every single Red Hat developer going to balk at that because
> they will correctly say they have never promised this.  They cannot do
> such, and have never done such, because there is no guarantee that
> Fedora Core, like Red Hat Linux before it, will not have minor ABI
> changes, etc...  This was the case between Red Hat Linux 8 (CL3.0) and
> Red Hat Linux 9 (CL3.1) and, even to a lesser extent, even in Fedora
> Core 1 (CL3.2) after Red Hat Linux 9.  But the reality is that there has
> to be a general guideline, there has been such in Red Hat Linux before,
> and right now the understood and trusted guidelines is starting to hold
> very poorly in Fedora Core.
> 
> This is especially sad, because a guideline for Fedora Core can not only
> be devised, but it would still easily explain the changes in Red Hat
> Linux 8 to Red Hat Linux 9 too!
> 
> - The Release Model It Has Been and Should Continue To Be ...
> 
> First off, go back to the dot (.) revisions on the Community Releases.
> You don't have to guarantee that the next ".1" will be 100% ABI
> compatible with the preceding ".0", because some things can and do
> change that early in a new major version series.  In fact, in addition
> to wanting Red Hat to call Red Hat Linux 9 as what it really was, 8.1, I
> also wish it would have just adopted GCC 3.0 in Red Hat Linux 7.1 after
> GCC 2.96 was included in Red Hat Linux 7[.0].  Because anyone who has
> been around Red Hat knows that while a ".0", while stable, it is new-
> technology adoption that may have API and ABI incompatibilities until so
> accommodated.  Even FreeBSD releases it's ".0" releases in the same way,
> because they are lesser used and lesser accommodated new changes in a
> ".0" forces people to start doing such en-masse.
> 
> Secondly, dot revisions should continue to be release for the same major
> version _until_ a Red Hat Enterprise Linux (EL) is released on the same
> version.  Once that is done, once the EL produced is released, that is
> the _defining_ ABI for the entire Community Linux (CL) itself.  Even if
> the ABI is not complete compatible in earlier revisions of the same CL
> version, from that point forward, it is the official ABI of the CL
> versions.  That solves the "we don't guarantee" attitude, that Fedora
> Core versions may not be compatible between revisions, but at least
> there is that guideline _once_ the EL based on it is released.
> 
> A guideline that is well enforced with the re-introduction of dot (.)
> revisions.  Because we know the higher the revision, the more finalized
> the ABI is for the major version -- and the lower the revision
> (especially .0), the less finalized the ABI is for that major version.
> It's the simple logic and unchanged we've known since Red Hat Linux 4.0
> forward.
> 
> The "litmus test" of when to change the major version then becomes when
> the ABI is broken by a new release.  As far as I'm concerned, if GCC 4.0
> as implemented in Fedora Core 3.90 (Fedora Core 4 Test 1), then it
> should be labeled Fedora Core 5.0 (CL5.0).  If Red Hat decides to go
> back to the same GCC (and GLibC) in Red Hat Enterprise Linux 4 (EL4),
> then it should be labeled Fedora Core 4 (CL4.2).  Some may snicker this
> seems complicated, but that is only because of the current versions
> without dot (.) revisions -- and will finally be solved from this point
> forward.
> 
> Because, in a nutshell, I'm arguing the next Fedora Core release that
> greatly alters ABI compatibility from Red Hat Enterprise Linux 4 (EL4)
> should be called Fedora Core 5.0 -- regardless where or not there is
> Fedora Core 4, which should only be so (CL4.2) if it ships with the same
> GCC, GLibC as EL4.  If not, forget linear, solve this version non-sense.
> 
> So from there, we go back to dot (.) versions.  Fedora Core 5.0 (CL5.0)
> and Fedora Core 5.1 (CL5.1) could differ in ABI compatibility, but once
> Red Hat Enterprise Linux 5 (EL5) is released, that "marks" the ABI
> "litmus test" of whether or not the next community revision is going to
> be Fedora Core 5.2 or Fedora Core 6.0.  That's all I'm asking for here.
> 
> Red Hat doesn't have to guarantee a minimum of three 0-1-2 "Community
> Linux" (CL) versions for every "Enterprise Linux" (EL) like in the past.
> It would be nice to just know _when_ we've reached the "last" and most
> finalized/stable revision of the CL series that still matches an
> equivalent EL.  Again, there is no guarantees with Fedora Legacy for
> long-term support, but it would be nice to have that guidelines so the
> community can do so if it wishes.  Without such a guideline, my trust of
> Fedora Core will slip.
> 
> Which means my complementary purchasing of RHEL will slip as well
> (especially with what Novell is up to -- see farther below).
> 
> - The Court of Trusting Red Hat Integrators ...
> 
> Now I'm sure many will scoff at this suggestion, and that's fine.  Red
> Hat has every right to believe it is impossible to follow these
> guidelines and state there has never been an official release model
> anyway, even in the Red Hat Linux 5, 6 and 7 days.  That might be the
> marketing line, but the reality is that Red Hat is starting to lose in
> the Court of Trusting Red Hat Integrators who have always been able to
> balance Red Hat Linux, now Fedora Core, alongside Red Hat Enterprise
> Linux.  The visibility of Red Hat Enterprise Linux as a complementary
> solution to Fedora Core is like Red Hat Linux before it, it only lasts
> as long as we trust it.
> 
> This means I don't see Red Hat as, the same as most other Trusting Red
> Hat Integrators, a so-called "greedy" company.  Especially since the
> "Enterprise" product was SuSE's idea in the first place and "won out"
> over ideas like Red Hat Linux 6.2"E" because the market just prefers it.
> This is despite my professional preference of approaches like old Red
> Hat, now heralded by companies like Progeny -- "Enterprise Releases" is
> merely an extension of internal configuration management of a single
> development line (long story and tangent).
> 
> Furthermore, I don't consider CentOS, White Box Enterprise Linux or any
> other source RPM (SRPM) rebuild solution to be an "alternative" either.
> I agree 100% with Red Hat CTO Michael Tiemann's varying comments on RHEL
> as a SLA guaranteed product, and how Fedora fits in as its complement,
> which these community rebuilds do not address at all.  Even though CAos
> maintains their own Fedora Core equivalent too -- it's still reliant on
> how Fedora Core is developed, just like RHEL too.  So it that project
> doesn't solve the problem either.

I agree with a lot of this above, however I do think something 
fundamental has been overlooked. The wrong people at Red Hat are making 
the decisions. Talk with some of the many folks who've left Red Hat. I 
appreciate all that Red Hat has done for the open source community and 
what they continue to do. While it's not my distribution of choice, I'll 
hardly run around bashing it. I used it for many years and there's still 
good stuff coming out of Red Hat. The problems you're talking about 
right now stem from the same thing that made the transition from Red Hat 
Linux to Fedora a PR nightmare. That is that Red Hat is a publicly 
traded company who in the last few years ended up in a corner and the 
wrong people were able to influence decisions too much.

> Which brings me to Novell.
> Novell is quickly turning SuSE into the Fedora-like release and
> trademark of its portfolio.  Novell, unlike Red Hat, _does_ maintain a
> versioning match between it's continuing SuSE Linux releases and its
> newfound Novell Linux product line.  Novell has begun the first fully
> redistributable, GPL-centric releases of SuSE Linux 9.x in full CD/DVD
> set form, no restrictions (unlike SuSE before).  And from the looks of
> it, it looks like SuSE Linux Enterprise Server (SLES) will be sporting a
> Novell(R) trademark instead for version 10.  All while Novell mirrors
> the, now lesser, SuSE trademark after the Fedora approach -- which will
> be required as more and more distributions become SuSE-based forcing the
> issue (which those of us who understand US trademark law actually
> followed and understood Red Hat's moves of the same).
> 

> Now SuSE does not have history of Red Hat, both technically and
> community-wise, which makes me still hope.  And I've come to trust Red
> Hat's technical considerations over SuSE's, especially for UNIX file
> servers (which the majority of Linux integrators who are focused on web
> and application servers don't seem to understand why).  But Novell is
> re-writing that history, and I like what I see in the Novell Linux
> Desktop and the plans for the split version 10 line.  And Novell**
> _does_ seem to understand the distribution channel better than Red Hat**
> too.

I'd be careful here. SUSE (all caps since the acquisition) does not have 
the extent of history that Red Hat has in the US. Globally however, SUSE 
has a very significant community and technical history. While trusting 
technical considerations is going to depend largely on experience, I 
would certainly trust SUSE's long term technical considerations more 
than Red Hat's. Let me reiterate, that is not at all a comment about Red 
Hat's technical staff. I know left to themselves things would be much 
better for all of us. Many of them are top notch.

> [ ** SIDE NOTE:  I see Sun's attitude towards Red Hat to be similar to
> Apple's towards IBM 20 years ago -- Novell is Sun's _real_ competitor,
> just like Microsoft was Apple's.  And like Apple, Sun is going to be
> blindsided by its agreements with Novell, just like Apple was with
> Microsoft, because they think Red Hat is their enemy, much like Apple
> thought IBM was. ]

Sun's problem is that they don't know who their real competitor is. The 
Linux kernel is more of a threat to Sun than all of these companies put 
together. They know Linux is their competitor, but they honestly have no 
idea what that means. Right now their disposition is to copy what they 
believe Linux's success centers around: open source (which is not 
entirely accurate). Unless they realize what this is all about, they 
will flounder indefinitely.

> Which means if Red Hat doesn't give us long-time, Red Hat trusting
> integrators the same, warm, fuzzy feeling of how Fedora core's release
> and versioning/revisioning will be handled, Novell is seemingly
> dedicated to doing much better with its new split SuSE-Novell release-
> product offerings.  I'm already benchmarking SuSE-Novell against Red Hat
> installations, and if I can verify the technical preferences I have, the
> release-product alignments are definitely looking better than what Red
> Hat is presenting.
> 
> Which is why I'm writing this letter.  Because I'm not being critical of
> Red Hat as a company.  Everyone who knows Red Hat knows that Red Hat has
> donated far more to the Linux community, and continues to do so, more
> than even companies like IBM.  Red Hat's GPL focus and the software-side
> half of Bob Young's vision (even if you disagree with the other half of
> "cut-throat business" in his vision, the GPL-analness is to our benefit
> as a community regardless) will always win praise.  

> Because Red Hat, the
> majority of the company, is one big collection of massive GPL projects,
> and not this "greedy" image the ignorant ABRH crowd proliferates.

I agree. It's too bad that small segment of Red Hat that fits the 
proliferation has as much influence within Red Hat as they do.

> But the reality is that there is no excuse why Fedora Core can't be
> "unofficially" trusted like Red Hat Linux before it.  Especially with
> what Novell has been putting forth in SuSE, despite SuSE having no
> history that can touch Red Hat's prior to Novell's purchase.  And that's
> the opinion that even I, one considered to be one of the most and
> staunchest of "Red Hat Apologists," are starting to think.
> And that's not a good sign for Red Hat if Novell delivers.

I agree again. Competition is good. The Linux world can benefit from the 
same thing every other free market benefits from. If Novell keeps going, 
they will incite Red Hat to change. Also, vice versa. Innovation will 
only get better and filter right back into Linux in general. *That* is 
what will frustrate Sun, and Microsoft for that matter, to no end.

Thanks for the read, it was good stuff. I'd also be happy to get it 
published for you if you're interested. I guarantee you your voice will 
be heard.

Dean

-
To unsubscribe, send email to majordomo@luci.org with
"unsubscribe luci-discuss" in the body.