Re: This was posted on Digg.com -- Linux Distribution blog/article

["Bryan J. Smith" <b.j.smith@ieee.org>]

Jeff Licquia <jeff@licquia.org> wrote:
> Sure.  But when you list a lot of criteria, and every one
> says "Advantage: Ports", you tend to get the idea that the
> person in question is a ports advocate.

On most of the things people look at first, of course!  Then
I went into the seriousness of regression and integration
testing, configuration management, etc...  That's where
Gentoo really hurts, and puts the buck on you.

Understand this is a _blog_ entry, and the flow is _not_
finalized yet.  I keep promising a major publication I'll
deliver a 3,000 word version, but I've been working 80
hours/week at my current job (I wrote that blog entry when I
was working at a job that rarely pushed me more than 50
hours/week).

> Plus, your point seemed to be that integration testing is
> a) spotty even in the package distros

For most package distros, hell yes.  Outside of Debian
(official), Fedora Core and RHEL, I'm not impressed with the
testing -- although SuSE/Novell is getting better.

But when it comes to Debian (official), Fedora Core and RHEL,
I've always been most impressed -- that's why I deploy them
*FAR*MORE* than BSD or Gentoo.

> and b) redundant since you have to do your own integration
> testing anyway.

No, the usefulness of packages distros is inversely
proportional to the amount of customization you have to do on
them -- which is proportional to the amount of new
integration testing that must be introduced.

> Perhaps I was misinterpreting.

As I said, it's easy to read what you want out of something.

I have this problem when I speak on politics from my clearly
Libertarian viewpoint.  Not only do nany Democrats just look
at the things I agree with on with Republicans and
vice-versa, when I say _both_ parties are _dead_wrong_ on
things (e.g., health care, immigration, taxes, etc... --
those are big ones), they interpret as I'm agreeing with the
other party.

It's not because I'm necessarily explaining things poorly,
but it's because most Gentoo v. packages, or Debian v.
Fedora, etc... "versus" debates start focusing on
differences, instead of the greater issue.  This is classic
on health care, immigration, taxes, etc... -- you no longer
hear about them, but what the Democrat v. Republican
viewpoint is.

E.g., nothing makes me laugh more than to hear both sides say
we have privatized healthcare (when it is the system we've
built around it), that immigration is a problem (which it is
the system we've built around it), etc....  People might be
balking about my using politics as an analogy, but the
"versus" brand name X distribution debates are a _perfect_
example.

That's why you _never_ hear me talk about "Gentoo" but
"ports" distros, much less "Debian v. Fedora/Red Hat" but
talk about the specific release models of "packages" distros.
 There's so much overlap in distros, including many distros
using the best features from others, that there's little
point.

Heck, even Gentoo's emerge is starting to be used for select
portions of packages distros -- e.g., to maintain web or
scripting subsystems.  Heck, even Perl-CPAN could be
considered such a similar approach.

> You don't have to convince me that Progeny is on the right
> track, especially since I'm responsible for that in a
> not-insignificant way. :-)

As I said, I think we agree more than we disagree.

In all honesty, I think about 75-80% of Gentoo users give it
a bad name.  They promote it for things Daniel _never_ said. 
As you'll note in my blog entry, the "optimizations" is a
rather over-stated bunch of non-sense.  It really goes to the
heart that most Gentoo users don't know how the heck GCC
works and can optimize for platforms -- especially the fact
that extension support is modular in the majority of
codebases (or they wouldn't work on various systems). 

> Maybe I'm just ignorant on this point.  On Gentoo, how do
> you maintain that fixed configuration, or reproduce it on a
> second system?  It was my understanding that emerge just
sucks
> down the latest.

First off, there is still a "portage maintainer" that
packages the Makefiles, etc...  Some maintainers are better
than others, so it all depends.  You're at some of the mercy
of some of the assumptions of the "portage maintainer" as
much as any "package maintainer" -- although with some less
binding influences.

Secondly, APT and YUM still "pull the latest" too, from
whatever the "package maintainer" says is the latest. 
Granted, the packages are typically not only regression
tested, but integration tested against other packages before
they make it to the "release" repository -- whereas ports
aren't typically integration tested (or only have limited to
what the ports maintainer did).  But you _still_ need to
maintain your _own_ internal repository, and roll out
configurations _after_ you have tested them for your needs. 
With Gentoo, this is just an increased amount that falls on
you.

So, lastly, anyone who runs "emerge" directly on an
_end-system_ is _stupid_ -- at least from the standpoint of
"proper configuration management" IMPO (IM Professional
Opinion).  They should be building on one system, testing
them, then integration test them against existing
configuration, and then roll out.  But then again, I'd say
the same with regards to anyone who does an APT or YUM from
the Internet too!  Even though there might be more
integration testing done on the DPKG or RPM files, there is
still a chance of an issue.

Ports just have additional testing considerations than
packages distros.  I personally _avoid_ running a program to
fetch new packages from the Internet on my end-systems --
because there's a good chance they can have undesired
effects.  Especially since on any given day, the package set
could have _changed_!  I want to maintain consistency, and
that means *I* mirror repositories and dictate what is
updated for consistency.  With Gentoo, that means I have 1-2
non-production systems that I only run emerge directly on.

> Right.  But a large part of that work is done for you when
> you use a stable packaged distro.

*IF* you don't need to make heavy modifications, I agree
entirely!

> In fact, so much of that work is done for you that I fail
to
> see how using ports could possibly be a savings for the
> vast majority of needs.

*UNLESS* you are modifying the distro to the point it is A) a
lot of work and, more importantly, B) to the point that
really renders the integration testing and mindshare
irrelevant

If I plop down a distro as a file server, I'm _not_ going to
change the kernel, authentication, directory and file
services much.  I explicitly trust that knowledge from others
(especially Red Hat when it comes to NFS).

Same deal for a database server, let alone when I'm going to
run binary-only solutions with SLAs, etc...

For a stock web server where I'm modifying little, you
betcha, I'm going to stick with the proven, tested stock
distro.

But once you really start modifying your Internet services
heavily, or you need a development system with lots of
packages (sometimes conflicting) that aren't included, etc...
 a pre-made packages distro may not only become a dependency
nightmare, but all the added software really makes the
existing integration testing moot -- because I'm doing my own
with the additions/removals.

> Well, sorta.  We basically gave up on the end-user model
> years ago.  Our customers are savvy enough to understand
their
> needs better than the average end user.

And people complain about Red Hat dropping Red Hat Linux as a
product with all the identity crisis and trademark issues in
addition.  ;->

> Plus, for binary-only software, the LSB is key to our
> efforts.  Binary-only software is OK for us as long as the
> standard they adhere to isn't "RHEL 4".

But does LSB define the kernel, the GCC and the GLibC
versions?
That's binary compatibility right there!  Until the LSB
starts putting down _hard_ version requirements on that
trifecta, even considering binary compatibility means
nothing.

> But the biggest place we play is in the middle.  Most
> people who customize care about very specific pieces of the
> puzzle.  They need to diverge, and thus lose their vendor
> support, but they don't need to diverge so much that they
must
> do all their own QA on the whole distro, or totally lose
> compatibility with the wider world. 

But if it works better for them, there is a reason.  I agree
that 75-80% of people that deploy Gentoo don't know the first
thing about proper configuration management (and damn if
their repeat issues don't show that ;-), but for those 20-25%
of us that understand how to maintain software from source
and see it through the entire software engineering model,
sometimes it is less work than "fighting the assumptions."

> Before, these people were pretty much forced into the
latter
> approach.  Now, they can call us.
> I believe you about your use of Debian/RHEL/FC, and did
> before.  But I still think this is an overstatement.

BSD still has a huge following in major enterprises because
the enterprises put forth the testing and management
required.  To them, it's worth it versus a packages Linux
distro.

> Ports start to become a good approach when the need
> diverges from standard practice by more than 50% or so.

Yeah, something we agree on!  You see, you don't think so
differently than I.

> But I believe that the set of organizations with those
needs
> has a membership approaching zero.  Most of the
organizations
> in the distro business today would like to get out of it,
and
> stay in only because of the lack of alternatives.

Most business organizations are looking for authentication,
directory, file, desktop, etc...  Definitely no reason to go
Gentoo.  You get to Internet services, it gets more
interesting.

In the end, I see a near-future where more and more systems
have a source-level management interface for specific
subsystems.  Kinda like Perl has for CPAN, we're starting to
see the same of Apache.

> That's not the question.  The question is: why did they use
> either RH or SuSE, when they could have built their own? 
> Certainly no one thinks they weren't competent enough to do
> so.

They didn't want to get into the business of maintaining what
already works for others.  Furthermore, I'm also sure they
wanted to avoid the conflicts that plagued Caldera, IBM and
others in maintaining more than 1 UNIX flavor.

> The question here is not "who is X's customer?" but "who
> should X be going after as a customer?"

No, "who is X's customer?"  Sun is just trying to keep
customers.  In reality, distros don't drive new customers --
only maybe enthusiast interest (at least the ones obsessed
with brand name marketing, not actual technical differences).

> If you ask me, Sun isn't quite sure yet in the Linux space,
> and Red Hat has the wrong answer.

Ironically enough, I'm starting to go back to Solaris (on
Opteron) rather than deal with some seriously lacking
features in Linux.  Now that Solaris/x86-64 is here, I get
performance and enterprise features, for a few bucks more. 
Well worth it.

If Red Hat ever gets the kinks worked out on LVM2+DM2, let
alone adopts a real enterprise filesystem like XFS, then I'll
reconsider.

> But customization is not either/or.  Certainly JDS isn't
> just rebranded SuSE, any more than Ubuntu is just rebranded
> Debian.

But Sun can afford to put forth all that generic development
for customers, because it is used by thousands upon
thousands.

When you're talking a single organization, they do it when
they need to.  Some just need to.

> However, in both cases, they used a starting point, and
> keep divergence as low as possible, rather than do all the
> work themselves.  Why?

Because Sun is developing a desktop system that is very
common is assumptions.  They are going after a _mass_ market
use.

For those that are not, it's less work in the end, even after
all the testing and management.  That's the key.  It's not
common, but it does exist.

> Does Sun fear that they can't hack maintenance of a Unix
> derivative?  Does Canonical (who employs a boatload of 
> top-notch Debian developers, and is headed by a former
Debian
> developer) fear maintaining a Debian derivative?

If they are going after a common application, of course, it
makes sense!  The same is happening with Fedora too.  No one
wants to fork anymore, they want to leverage the greater
mindshare.

The problem is when the customizations are heavy enough that
the assumptions and fixed configurations in those packages
distros are very binding, and have to be removed anyway
(removing the benefit of the testing by others).  Linux From
Scratch (LFS) was really "from scratch."  Gentoo allows you
to get past that, and focus on the testing and management
issues of rolling your own.

Again, there _is_ a point it is more viable.  It's not often
touched, but there _is_.

For desktop systems?  Hell no.  For LAN servers?  Hell no. 
For general web servers?  Hell no.

But with development systems?  Hmmm, now we're getting
closer.  What about heavily customized Internet platforms? 
Definitely.

What about application-specific black boxes?  Oh, definitely
now.

> In both cases, the reason they start with a known quantity
> is that it's silly to throw away the integration work
that's
> already happened without a good reason.

What is a "known quantity" after it's hacked to death?!  How
much good does an Apache implementation do me if I have to
rebuild the entire set from scratch?

And what do you mean by "silly to throw aawy ... without a
good reason"?  I think I've _painstakenly_ taken the time to
say exactly when there is such, and that it's not common.

So give me some credit.

> Let me give you an example.  The functionality of hotplug
> has been rolled into udev recently, but getting the right
> combination of hotplug (or not-hotplug), udev, and kernel
> together isn't as easy as it might seem, especially when
> designing an upgrade process.

What if you don't want to use udev and hot-plug?
Really?!  That's one scenario where I really like Gentoo.
Especially for application-specific black boxes.

> How many organizations really need to hash all this out for
> themselves, as opposed to just taking whatever the vendor
was
> able to figure out and assume the vendor got it right?

What if udev and hot-plug are unwarranted?

> Certainly, there are some who care about this.  But do they
> also want to hash out the interaction between CUPS and
> GhostScript?

What if they are using an alternative print engine and
service?

> Or the right way to compile the Java modules in
OpenOffice.org
> 2?

Now you're talking more desktop.  Desktop is definitely
_better_ with a packages distro.  That's one area where I
think 75-80% of Gentoo advocates are nuts.

> Or whether to use acx100 or ndiswrapper to support certain
> wireless cards? 

What about the application-specific black box that already
uses one fixed hardware device?

> I'd like to see the organization that really has enough of
an
> interest in all four issues that they really need to
customize
> it all, outside of OS vendors.

I'm not shouving Debian into an application-specific black
box.  It's overkill and bloated for what I need.

> And that's just four out of thousands.

Of which, I only need 100, and they are often _not_ what
Debian says is "the standard."


> I don't see that ports are actually being used in that way.
> Most people with radically different needs still start with
a
> packaged distro and customize that.

When they largely add to it, yes.

But when they are radically changing the base functionality
and assumptions of the distro, definitely not.

> Trust me on this: the demand for support for a
> regression-tested Gentoo in the market is so low as to be
> undetectable.

Since when is a "ports" distro able to deliver such?  A
"ports" distro, by its very nature, puts the testing on the
_end-builder_!  So your statement is rather
conflicting/non-real-world.

But you don't see that, because you're focused on delivering
services around packages distros.  That's why you don't see a
need for it, because you don't see a service model for it as
an organization that provides such.

That's a little nearsighted IMHO.  ;->

> What would be the difference between that and a package
distro
> offering, besides the fact that the ports system is a lot
more
> difficult to reproduce reliably?

Who says a ports distro is more difficult to reproduce
reliably?

If I'm only pushing out *MY* builds to systems, and
_removing_ any trace of "emerge" on the run-time/production
systems, what are you talking about?

Again, I think you think I'm one of the 75-80% of Gentoo
pukes that talks about "oh, it's so easy to emerge my
updates!"  That just goes against all configuration
management principles!

> (This doesn't mean that demand for Gentoo itself is
> nonexistent, just that Gentoo's penetration into the market
> for commercial customized Linux is.)

Well duh!  You're just realizing that's _exactly_ what I've
been trying to get you to understand?  Again, you and I do
_not_ think differently -- you're just trying to see
differences.

People who deploy Gentoo are either doing it for their
specific needs, or they are an integrator/provider that is
providing an _application_ -- _not_ a distro or services
around that distro.

Gentoo, for all intents and purposes, is merely just a more
integrated way to do "Linux From Scratch" (LFS).  That's all
Daniel Robbins really intended it to be -- take some of the
concepts of the BSD ports approach and make it easier.  Take
away all the hunting and finding and other things, giving
more time to focus on the most important aspects of source
management -- the testing, the management, etc...

75-80% of the Gentoo users out there sell it on things it is
_not_.  They sell "emerge" like you can run it on
end-production systems and have no worries.

> Ports-based distros with real support interest are distros
> with such high value that they overcome the ports weakness
> (OpenBSD), or distros that mitigate the support burden with

> stable releases (again, OpenBSD) or distros where the
support
> interest comes from an OS vendor (FreeBSD).

Or perhaps they are the countless BSD-based commercial
variants that are _not_ supported as a BSD OS, but as an
application-specific UNIX implementation.

No one will _ever_ make money on supporting Gentoo.  But they
will on supporting an application-specific creation from
Gentoo.




-- 
Bryan J. Smith                | Sent from Yahoo Mail
mailto:b.j.smith@ieee.org     |  (please excuse any
http://thebs413.blogspot.com/ |   missing headers)

-
To unsubscribe, send email to majordomo@luci.org with
"unsubscribe luci-discuss" in the body.