Re: This was posted on Digg.com -- Linux Distribution blog/article

[Jeff Licquia <jeff@licquia.org>]

On Fri, 2005-12-09 at 15:09 -0800, Bryan J. Smith wrote:
> Jeff Licquia <jeff@licquia.org> wrote:
> > Right.  But Debian is a package-based distro, with its own
> > QA processes that Munich is leveraging.
> 
> Yes, I know.  At first, I didn't know where you are coming
> from.  But not it seems you only have the viewpoint that "you
> don't want to go with anything less tested than Debian."  You
> seem to have a very narrow-minded focus, because I can see a
> variety of reasons why someone might go less, or even more
> (e.g., enterprise).

If it makes you feel better to believe that this is my viewpoint, be my
guest.  I suspect I may not be able to convince you otherwise.

Perhaps it would help if we compared starting points.  I start with
something that's well-tested.  You start with something that's much more
poorly tested (often; when using Gentoo, I mean).

Now, perhaps you can tell me about something you get by starting with a
poorly-tested system that's so great that you're willing to give up all
the testing my well-tested system has.  All I've heard, though, is that
it's a little bit easier to customize your Apache stack.  I can do your
custom Apache stack *and* give the customer a well-tested foundation for
that stack to run on, and I can probably do it with the same effort.
Why should a customer go with you instead of with me?

> Then you asserted that a service model on Gentoo can't work. 
> Where am I disagreeing with you?!?!?!  Gentoo -- from a
> commercial standpoint -- is about delivering
> application-specific implementations and services built
> around them, not delivering Gentoo consulting services.

And Debian isn't?

> > See the examples I gave in my other mail.  Do they test
> > those things, especially if those things don't directly
> > impact their current use case?
> 
> Here's the return thought -- do they *INCLUDE* those things
> at all when they don't use them?!?!?!

Perhaps, perhaps not.  But you do agree that there are lots of essential
pieces in the system that people don't customize, right?

> > My guess is that they don't, and that they only worry about
> > those things when problems actually show up.
> 
> What problems?  If they don't use them, they don't include
> them!
> 
> We're not building a desktop or general LAN server here. 
> Anyone who uses Gentoo to do so is just missing the point.

I'm not talking a desktop or general LAN server either.

> > This is a risky strategy to take unless you can rely on
> > someone else's testing, because fixing issues that come
> > up could require non-trivial changes to the base.
> 
> But you'd be stuck doing that if you were using something
> _different_ than what the distro includes anyway!  Do you see
> my point now?!  ;->

No.

If you were using something different than what the distro includes
anyway, then you'd be doing so because it directly affected your use
case, right?  In which case, the risk is unavoidable either way.

But I was talking about stuff that didn't affect your use case.

Sometimes you can strip that stuff out, but sometimes you can't.  When
you can't, you had better test it, even if it seems irrelevant, or it
could bite you in the butt later if your requirements change.

> > Right.  My point is that no one, in fact, makes such
> > radical changes as to totally negate the previous testing.
> 
> Could you drop the absolutes?
> 
> Especially since you've basically said that you are focused
> on providing services around Debian, as Debian, for Debian.

When did I say this?

> People who provide services using Gentoo don't provide them
> as Gentoo, they provide them as an application-specific
> Internet server, a black box solution, etc...
> 
> Totally different market and focus entirely!

How?

I mean, no one provides a RH or Debian-based vertical as RH or Debian,
either.  (They can't, as you pointed out with Sun JDS and as we've
learned with DCC.)

> It's like someone delivering a vertical app versus someone
> who provides Delphi programming.  The end-user doesn't pay
> for someone to write a program in Delphi, they pay for
> someone for a niche vertical application.
> 
> I have several colleagues that provide a variety of
> non-standard, custom, but exacting Internet services for
> specific industries.  Because they build, assemble and
> maintain a number of these Internet servers over time, they
> maintain their own, custom Gentoo builds, and _only_ emerge
> on those internal systems.  They then test and roll out
> releases based on those custom implementations.  Using a
> distribution like even Debian is a major issue, because many
> of their components are not in Debian, or are built very
> differently -- and they litterally have to build so much
> custom already.

Right.  They're in the distro maintenance business, only they don't have
the user base of Debian or Fedora to fall back on for all the corner
cases they don't have the resources to test.

Of course, no one has those resources for the custom parts.  But why
throw away those resources for the parts you don't customize?

> > I am solely focused on the ports-vs-packages thing.  If
> > Gentoo's goal, as you describe it, includes adoption by 
> > commercial customizers, then they have largely failed at
> > that goal.
> 
> Not commercial customizers or people offering commercial
> Gentoo services.  You keep assuming that's what I mean.
> 
> I'm talking about people offering custom application
> services, especially ones with configurations well outside
> what you traditionally get with package distros.

Why do you think I'm not talking about the same thing?

> > What do you call a "radical mod"?  And how radical does it
> > have to be to negate the distro's advantages in integration
> > testing?
> 
> When you're completely chucking the entire Apache tree.
> Or maybe you're putting in another SMTP services stack.
> Or how about maintaing your own cross-compilers and
> libraries.
> Etc...

Done, done, and done.  I'm sorry, but those are not radical changes, and
I would not presume to throw away perfectly good QA on kernels,
libraries, low-level systems, etc. for something as trivial as these.

> > I must respectfully disagree here.  Maintaining a distro
> > properly is hard work for anyone, for any purpose.
> 
> But what if a packages distro is not shipping what you need?

Then you do what you have to do.  How does that make distro maintenance
any easier?

> But for those who are maintaining their own release of
> software, for their own application-specific products, they
> are already doing this.  My first question whenever I tackle
> an embedded project is, "Do I have to use MontaVista or can I
> get away with Gentoo for source control?"

If there is one place where I might consider going all custom, it's the
tiny-footprint embedded space.  And yet, even here, there is a strong
custom base you can start from that has better QA than you can give your
own product.  Again: why abandon the QA MontaVista has put into their
product?  Are you really better than they are at providing an embedded
Linux?

> And when it comes to Internet servers, once I start looking
> at having to rebuild significant portions of various Internet
> services, it starts to make far more sense to look at Gentoo.
>  Not when I'm selling someone a "Linux Internet server," but
> when I'm providing them them an application.

Right.  Now you've sold them a black box, that they expect to work
forever.  But what happens when there's a security bug, and they have to
update, but the fixes that are out there are only available for the
latest versions, far newer than what you deployed?

You do the work of backporting the fix, building it, testing it, etc.
That's what we do, too.

If the problem touches the custom part of the solution, then yes,
there's no getting around it.  But if not, you can avoid a lot of work
by using upstream's fix to the problem as a starting point.

> > Replacing the entire Apache stack is trivial, as distro
> > customization goes.  In fact, Progeny is building a
> > do-it-yourself toolkit for just such a thing.  Nothing
> > about replacement of the Apache stack has an impact on
> > bootup, service management, utility availability,
> > kernel-libc compatibility, hardware detection, driver
> > support, library ABI compatibility, etc.  Heck, you can
> > even keep LSB certification across the change.
> 
> If Progeny has such a do-it-yourself toolkit in development,
> then I'd be very interested in exploring it.

http://componentizedlinux.org/index.php/PDK

> As far as hardware detection, I'm *NOT* building an
> installer.  As I said before, people don't build custom
> Gentoo releases for _others_ to install -- they build it for
> their _specific_ uses.  Same deal with drivers.
> 
> The whole kernel-libc compatibility is something that I've
> only trusted Red Hat too -- they've done a damn fine job of
> maintaining it, and providing compat-* packages for older
> compatibility.

Beautiful.  Can we agree, though, that there's a ton of stuff you
deliver on a regular basis that you don't customize, and that much of it
you've probably never customized?  Can you tell me what benefit you get
from building that from source, and how that benefit is more important
that the QA you get from a package distro?

> > And if you're not testing all those other things (or
> > relying on someone else's testing of all those other
> things),
> > you're not doing a good job of maintaining a distro.
> 
> And I disagreed with this how?  How many times do I have to
> say it, people who do _not_ test ports proper deserve what
> they get!
> 
> Did you ever think that maybe, just maybe, my article was
> partially written for Gentoo users who are ignorant of why
> they must maintain proper configuration management?  Many
> talk how Debian or Fedora are useless, yet they don't realize
> it's the integration testing.
> 
> That's _exactly_ why one Gentoo user commented "why are you
> so anti-Gentoo?"!
> 
> And yet again, you want to assume what I'm saying based on
> your viewpoints, not what I'm saying are actually in-line
> with your viewpoints.  ;->

Well, let me be blunt, then, since it doesn't seem to be getting
through.

When I talk about people who don't test their systems well, I'm not
talking just about those nasty Gentoo users who emerge on production
systems.  I'm talking about people who build anything at all from source
that they don't test thoroughly.  That includes you.

You do not test your deployed Gentoo-based custom distros enough.  You
can't.  It's too big of a job.  Maybe you're up to the task of testing
simple customizations to Red Hat or Debian, but if so, you're a rare
breed.

I'm not saying this because I have some aversion to source code, or
Gentoo, or you.  I'm saying this because distro maintenance is hard.
I've been burned by poorly tested rebuilds on Debian before, where I'm
only rebuilding a single package.

What kind of assurance can you give your customers when you rebuild
things like libc, coreutils, or the shell just so you can customize
Apache a little easier?  Can you tell them that your
libc/coreutils/shell combination has been run on many thousands of
computers throughout the world without any trouble?  Of course you
can't--not with Gentoo, anyway.

I'm also saying this because that's what our customers are telling us.
These are large customers, with millions of dollars in resources, and
they are realizing that they are unable to do better than a crap job of
maintaining their little stripped-down-and-customized Fedora.

Do you think that Gentoo would be better for them?  Don't say it to
their faces if you do; they would laugh at you.  They are not Linux
experts, and often don't understand what's going on in the updates
coming from Fedora.  If they had to do those updates themselves, the
result would be far worse--probably a lot like those people who run
emerge on production systems.

And they know that Gentoo is a worse base platform for customization
than Fedora or Debian.  How do I know that?  Because we aggressively go
after business with people using Linux for vertical custom apps, and we
don't see a Gentoo base in those apps.  It's just not there.  We see
lots of old Red Hat, lots of Fedora and CentOS, lots of Debian, and the
occasional SuSE or Mandriva.  But no Gentoo.

I don't doubt that consultants and IT employees are deploying Gentoo.
But, as you yourself admitted, most of them don't even do the minimal QA
you do.  Given that they won't improve their QA, are they better off
with a completely untested binary platform, or a binary platform that's
reasonably well-deployed elsewhere?  The answer to that should be
obvious.


-
To unsubscribe, send email to majordomo@luci.org with
"unsubscribe luci-discuss" in the body.