Re: This was posted on Digg.com -- Linux Distribution blog/article

["Bryan J. Smith" <b.j.smith@ieee.org>]

On Fri, 2005-12-09 at 21:34 -0500, Jeff Licquia wrote:
> Perhaps it would help if we compared starting points.  I start with
> something that's well-tested.  You start with something that's much more
> poorly tested (often; when using Gentoo, I mean).

Remember, I start with Debian, Fedora and RHEL far more than Gentoo --
like 50:1.  But there are a few applications where I build from source.
That's all Gentoo is, little more than a framework for building from
source.  It's like a management system for "Linux From Scratch."

> Now, perhaps you can tell me about something you get by starting with a
> poorly-tested system that's so great that you're willing to give up all
> the testing my well-tested system has.

When I'm modifying the well integration tested system enough that it is
no longer well integration tested.  And fighting the distro on the
modifications is a major factor.

> All I've heard, though, is that it's a little bit easier to customize your
> Apache stack.  I can do your custom Apache stack *and* give the
> customer a well-tested foundation for that stack to run on, and I can
> probably do it with the same effort.  Why should a customer go with
> you instead of with me?

If Debian has a way to rebuild the entire Apache stack including mix'ing
and matching what I need, I'd be _very_ interested.  I *WOULD* much
rather use Debian as a base than Gentoo.

You see, that's the point you keep missing.  I _avoid_ source _until_
the distribution presents enough of a limitation that I'm forced to
rebuild major portions from source.

I would much rather see packages distros with subsystems that can be
fully rebuilt from source.  It would reduce my tendency to use BSD or
Gentoo at all!

> And Debian isn't?

Debian can be something integrators use into their own products, I
agree.  So can Fedora.

I'm just saying Gentoo can't be something you support as a distro.  It's
not designed for that -- it's really nothing more than a source code
management system.

> Perhaps, perhaps not.  But you do agree that there are lots of essential
> pieces in the system that people don't customize, right?

There are a lot of essential pieces that can be _left_out_ because they
are not needed.  Now the stock kernels suck at many things.  That's my
#1 beef with a source-based distro.  But in embedded or application-
specific builds, I'm typically using non-standard kernel options.

In fact, that's a good sign that I'm going to use a source-based distro.
If I'm rebuilding the kernel into modes that are well off the path for a
tested packages distro -- e.g., optimizing as a bridge/router.

> If you were using something different than what the distro includes
> anyway, then you'd be doing so because it directly affected your use
> case, right?  In which case, the risk is unavoidable either way.

Exactly!  *I* now have to take responsibility for testing.  At some
point, it's just better to rebuild from source -- because I'm modifying
so much that I'm only going to 'fight' with the 'distro's assumptions.'

> But I was talking about stuff that didn't affect your use case.
> Sometimes you can strip that stuff out, but sometimes you can't.  When
> you can't, you had better test it, even if it seems irrelevant, or it
> could bite you in the butt later if your requirements change.

Did I disagree with this statement?  How many times do I have to say
that you have the burden of testing -- both regression of the package
itself as well as integration testing -- when you rebuild from source?

> How?

As you pointed out, Debian has a major commercial service model where
consultants help customers leverage their distribution models in their
organization.  They are helping them with customization, configuration
management, etc... -- leveraging all the existing testing of packages,
etc...

Gentoo does not.  Gentoo never will.  It's a ports distro, not a pre-
tested solution.  It's basically a source distribution system.  So by
its very nature, you are customizing everything.  There is little
commercial model to offer services around Gentoo as a fixed distro --
because it's not.

So don't make the comparison.

Now source code builds _are_ typically used when you build a very
application-specific solution, especially where packages distros have
very conflicting defaults, assumptions, etc..., or make it very
difficult to change.  Fixed, system-specific builds for a specific
application.

No installer.  No hardware detection.  Not anything that would be "re-
installed" or "repackaged" on arbitrary hardware.  You have repeatedly
made comparisons in such regards and I don't know how many times I have
to say that Gentoo is _not_ for that before you'll give up those
comparisons.

Now you _can_ use package distros to do application-specific solutions
too.  I never said you could not.  But at some point with some
solutions, you're maintaining so much of the solution outside the distro
that the distro doesn't offer anything but headaches in preventing
things from being updated, etc... from the stock repositories.

> Right.  They're in the distro maintenance business, only they don't have
> the user base of Debian or Fedora to fall back on for all the corner
> cases they don't have the resources to test.

No, they are _not_ in the distro maintenance business.  They are _not_
selling/providing a distro that end-users install, use and add to.  The
end users *NEVER* touch the software itself.  To them, it's a "black
box" solution.

They are maintaining a fixed, _pre-installed_ source code release that
they maintain for a number of systems.  These systems have been built
completely custom, with all the crap that is not needed (and will
_never_ be needed) left out.

> Of course, no one has those resources for the custom parts.  But why
> throw away those resources for the parts you don't customize?

Because they are _not_ needed!  How many times do I have to say this.
No installer.  No build tools on the end-system.  No fancy, flexible
subsystems.

Think of such systems as a fixed configuration run-time.

They are *NOT* "installed" distros.  They are built "run-times" that do
1 thing, are _never_ modified (except when updated with a newly tested
build, etc...).

> Why do you think I'm not talking about the same thing?

No, you're talking about 2 things and merging them.

Debian can do either well, but to a point on 1.
Source builds only do 1 well.

> Done, done, and done.  I'm sorry, but those are not radical changes, and
> I would not presume to throw away perfectly good QA on kernels,
> libraries, low-level systems, etc. for something as trivial as these.

But what if you're customizing the kernel to the point that you're
throwing major switches on how the kernel acts?

> Then you do what you have to do.  How does that make distro maintenance
> any easier?

When you build a "run-time" that fits in 50MB.  How about that?
I'm serious, I'm talking about "run-times" that fit in under 50MB.
Thinks that have so much crap cut out that the package distro will force
upon you over and over and over.

I do not need several hundred MBs that Debian forces me to install for a
web server.  Fedora Core is even worse (and don't get me started on
RHEL).  That's yet more packages with more dependencies (yes, I know,
Debian isn't as bad as Fedora), and sometimes they have things that
conflict with the portions of my source builds.

Now I could choose to give myself a lot of headache by attempting to
build DPKG or RPMs of these components, and try to merge them with the
stock packages.  But then assumptions of the stock packages by the stock
package maintainers change, and then I've gotta change too.

If I was maintained a distro that _others_ installed and maintained,
then yes, I'd build DPKG and RPMs.

But I'm maintaining a "run-time" that *I* maintain, for fixed
configuration systems that *I* update -- "black box" solutions.

> If there is one place where I might consider going all custom, it's the
> tiny-footprint embedded space.

Duh!  That's one of the major apps I've repeatedly covered!  Everytime I
do, you ignore it.

> And yet, even here, there is a strong
> custom base you can start from that has better QA than you can give your
> own product.  Again: why abandon the QA MontaVista has put into their
> product?  Are you really better than they are at providing an embedded
> Linux?

The question is the flexibility of the MontaVista solution.  If it
provides what I need, great!  But if it doesn't, then what's the
purpose?

Here's a nugget for thought:

Gentoo is little more than a source-build control system, so has far
more familiar with an embedded development platform than a packages
distro.

In fact, that's the most effective approach with Gentoo -- treating it
as such.  I have my host Gentoo build system, and then I build "run-
times" from it that are deployed _fixed_ to end-systems/solutions.
That's really the _only_ way to make it feasible and viable.

People who run emerge on the end/production systems are fooling
themselves.  They are completely ignoring proper configuration
management.

So at what point are you going to recognize that I have been
saying ...  

A)  Gentoo is _not_ a distro in the traditional sense, but a source code
build and release system, and should be used as such with all the
assumptions and requirements when doing so

B)  Gentoo is _only_ good when the builder/integrator accepts the
testing load that would traditionally be provided in a packages distro

C)  Because it's a source code build system, Gentoo is _only_ viable
when enough of the distro is changing to the point where a packages
distro's tools and stock repositories are going to cause continual
conflicts and differing assumptions -- especially when it requires
features that are not needed at all, and cause additional headaches

D)  And finally, Emerge is _not_ a tool to be run on end-systems like
APT or YUM, even though 75-80% of the Gentoo users are ignorantly saying
it is and that it's "better" -- it's _not_, it's "different"

Now, do you see my points?

> Right.  Now you've sold them a black box, that they expect to work
> forever.  But what happens when there's a security bug, and they have to
> update, 

I'm providing the service level agreement (SLA) to them for the box!

That's where you aren't following how much I'm talking about the
commercial model "differing" from what you're used to.  Yes, I could use
Debian and then just provide them those updates -- or the end-customer
could do it themselves.  That _would_ be easiest.

But what about the case where the distro is so customized that an APT
would _break_ the system?  That there are stock components and
requirements that conflict with what I use?  Or assumptions are made in
the distro that cause my software to act incorrectly?

That's what I'm talking about.  And that's where anytime a critical
issue is released, I'm at my "host" system rebuilding a new "run-time"
to put on their system.

> but the fixes that are out there are only available for the
> latest versions, far newer than what you deployed?
> You do the work of backporting the fix, building it, testing it, etc.
> That's what we do, too.

You don't have to sell me on the value of backports.  My not some
ignorant user who complains about Debian and Fedora/RHEL doing that.

But in these solutions, I do _not_ need backports.  These are custom
solutions from source.  That's really a major advantage of ports -- when
you don't need backports for binary compatibility or to deal with the
_assumptions_ of _other_ packages in the system.

When you are rebuilding the system _whole_ and testing it as such, as a
very integrated solution that only does 1 think and does it well.
Again, don't think of it as a distribution, think of it as a run-time.

> If the problem touches the custom part of the solution, then yes,
> there's no getting around it.  But if not, you can avoid a lot of work
> by using upstream's fix to the problem as a starting point.

The thing is that the only time I'd care is _when_ its touching the
custom solution.  Otherwise, why wouldn't I use Debian, Fedora/RHEL,
etc...?

Seriously, that's the point you keep driving home that is wholly
_inapplicable_ to what I'm saying.  You're arguing against someone who
is saying Gentoo should be used for situations where there are a lot of
stock components, unmodified.

And I'm almost screaming that if that was the case, I would _not_ be
using Gentoo!  I'd be using Debian, Fedora/RHEL.

That's why I'm saying that our viewpoints are very much alike.  The only
difference is that you can't see any situation where someone would use a
source-level build.  And I'm saying, just as you said, when the system
is over 50% customized, I'm using a Gentoo host and building run-times
for deployment on systems.

That means I'm not maintaining other components I don't need.  I'm not
putting in udev, an installer, a hardware change detection program, no
hot-plug scripts, etc...  An end-user damn better well not touch that
box!  At most, they are interacting with it from a user/network
perspective and _nothing_ else.

> http://componentizedlinux.org/index.php/PDK

I look forward to seeing it, seriously.
Until then, I'm having to build from source.

And when that reaches a threshold where it's less work for me to test
the entire "run-time" than to "fight" the distro by adding my own, then
that's where I go Gentoo.

> Beautiful.  Can we agree, though, that there's a ton of stuff you
> deliver on a regular basis that you don't customize, and that much of it
> you've probably never customized?

I said 50:1, that is the case.  ;->

> Can you tell me what benefit you get from building that from source,
> and how that benefit is more important that the QA you get from a
> package distro?

When I'm literally _obliterating_ some many packages with all that QA,
and the package distro fights me all-the-way -- puts a great burden on
myself because I'm not doing things they way it wants.

That's when I run into the threshold where it's less effort for me to do
my own QA, on the _small_ set of packages I need.

> Well, let me be blunt, then, since it doesn't seem to be getting
> through.

You're actually making statements I agree with for the most part.
We've even hit on the customization thresholds, etc...
We're actually very eye-to-eye on that.

Now you can talk about PDK solutions all you want -- something I'm
_very_much_ interested in when it becomes available.  But until then, if
I'm maintaining source for the _majority_ of the platform, I'd rather
have a build system that is catered to that.

> When I talk about people who don't test their systems well, I'm not
> talking just about those nasty Gentoo users who emerge on production
> systems.  I'm talking about people who build anything at all from source
> that they don't test thoroughly.

Exactly!  I 100% agree.  The majority do not.  They have that (joking)
attitude of Linus that "if it builds, ship it."  They ignore the
software engineering model, and all the requirements that come with
that.

> That includes you.

WTF?!

> You do not test your deployed Gentoo-based custom distros enough.

WTF?!

> You can't.  It's too big of a job.  Maybe you're up to the task of testing
> simple customizations to Red Hat or Debian, but if so, you're a rare
> breed.

WTF?

Who the freak gives you the right to assert how well I do or don't test!
Really?  That is the most arrogant, insulting and assuming statement
I've ever been presented with!

> I'm not saying this because I have some aversion to source code, or
> Gentoo, or you.  I'm saying this because distro maintenance is hard.
> I've been burned by poorly tested rebuilds on Debian before, where I'm
> only rebuilding a single package.

You should consider that maybe, just maybe, someone like myself has been
working with GNU/Linux for 12 years in industries like ... say,
aerospace and semiconductor design?

I'm sorry, at this point, I will not respect your opinion any further.

This thread is ended.



-- 
Bryan J. Smith   mailto:b.j.smith@ieee.org
http://thebs413.blogspot.com
------------------------------------------
Some things (or athletes) money can't buy.
For everything else there's "ManningCard."



-
To unsubscribe, send email to majordomo@luci.org with
"unsubscribe luci-discuss" in the body.