Re: This was posted on Digg.com -- Linux Distribution blog/article

[Jeff Licquia <jeff@licquia.org>]

On Sat, 2005-12-10 at 22:20 -0500, Bryan J. Smith wrote:
> On Sat, 2005-12-10 at 20:34 -0500, Jeff Licquia wrote:
> > You seem to think that this job is somehow split into two groups, as if
> > you're exempt from the kinds of things Red Hat has to do.
> 
> Stop!  Please point out where I said I was an exception to regression
> and integration testing?!  Really!  I am an engineer who is called
> "anal" to my face when I push the hard lines on the software engineering
> cycle.  When I rip people a new one when they don't commit their changes
> to /etc to revision control.  When people roll things out without
> testing.

Great.  Glad to hear it.  But it's still not as good as the testing a
mass distro gets.  Or is it?

> I merely said that I do not have to test for installer issues, hardware
> issues, running desktop applications, other applications, etc... that
> Red Hat does, and spends a lot of time doing!  Certifying!  Etc...  You
> have brought up this point over and over and I'm trying to tell you I'm
> _not_ doing such.

So, when I say that "you don't have to worry about an installer", what
exactly do you think I'm saying?  That you do have to worry about an
installer?

> > And what I'm trying to tell you is that you're throwing away a lot of
> > good work when you insist that you have to build everything,
> > work Red Hat has done and that you should do too,
> 
> For the last freak'n time, I _rarely_ use Gentoo.  I use Debian, Fedora
> or RHEL 50:1 over Gentoo or BSD.  I understand that value, so stop
> preaching it.

Sure.  That's why I made the exceptions.  I've always been saying that
your use of Gentoo is what I'm focused on, not your use of Red Hat or
Debian.  Heck, I'm advocating that you use it even more.

> But I am *NOT* putting a Debian distro into a 64MB flash!

I never said you should.  (MontaVista, maybe...)

> And I am *NOT* going to load up all sorts of scripts and support crap
> when I'm putting up a heavily customized Internet server that is bare-
> boned, lean and specifically configured.  Something that goes on a self-
> certified piece of hardware with those specific, self "tested and
> approved" roll-outs of software.
> 
> Yes, 75-80% of people providing such solutions from source code don't
> know the first thing about software engineering.  And I'd even argue 99%
> don't "get it right."
> 
> I'm sorry, I'm not one of them!  Writing mission critical financial and
> defense applications will tend to make you anal on that!

Right.  So you've earned your chops, and now you're all bent out of
shape because you think someone is questioning your bona fides.

Read closely, because I've written it several times, and you don't seem
to be getting it amidst the rage:

 --> I AM NOT SAYING YOU'RE A BAD ENGINEER. <--

Got it?  Calm now?

Every engineering solution has tradeoffs, right?  Every decision has
good and bad points, and you do your best to maximize the good and
minimize the bad, right?

So perhaps there's a benefit to throwing away a lot of integration
testing that I don't see.  On the one hand, you've got all this good
work by Red Hat and Debian, and on the other you've got this must-have
feature that you just can't do without rebuilding the whole shebang.  So
you build from source.  And it's still the best decision, even though
there is a cost and a risk because your custom build has potential
integration issues you're unlikely to discover.

But what you seem to be saying is that no, there is no cost or risk to
throwing away the RH/Debian testing efforts, and I'm an asshole for even
daring to suggest that Bryan J. Smith ever makes a decision that has a
cost.

I get that this impression is nearly certainly wrong, but I'm clawing
for evidence of that, and you're not helping.  All I see is a lot of
bluster and a specific task, replacing the Apache stack, that doesn't
look that hard to me.  (OK, you mentioned a few others, but they looked
even easier to me.)

Maybe if you would describe an example of the kind of thing you just
can't do without starting from source, I'd be a little more appreciative
of your position.

> > but can't because you don't have Intel and HP and Dell sending you free
> > computers to test your little distro on.
> 
> This is _exactly_ where you are so _not_ listening to me!
> 
> I do *NOT* need to test on Intel and HP and Dell.  They will *NEVER* go
> on them!  You continually refuse to accept that I am _not_ providing the
> same services as you.  You keep thinking I'm "redistributing" some sort
> of software that users see.
> 
> I have to test *ONLY* on the system I've rolled out, as they are the
> *ONLY* systems I will be putting *AND* supporting them on!  Be it a Tyan
> S2885/2895 (dual-Opteron) or a Boser HS-2606 (ViA CLE266/1.2GHz C3 SBC).

I'm sure that's all true.  But the value of testing on a diverse set of
hardware isn't all in the hardware and support itself.  It's in the
brittleness of the software you're delivering.

When Red Hat ships something, they have the benefit of finding all those
parts where things fit together pretty well, but not quite perfectly,
because someone, somewhere, has the weird setup that stresses the
combination just right to show the flaw.  And when they find these
little problems, they apply the little fixes that have as little impact
on the whole system as possible.  99.9% of the system doesn't change,
and the little fix gets run through the same wringer to ensure it
doesn't have its own little ripple effect.  All this adds up to a system
that generally tolerates any kind of change a little better.

Now, you may disagree, or you may think the effect here is minor.  But
I've got a set of rocket scientists on my side here, in the form of
Oracle support.  I think they know a thing or two about good software
engineering, and when they give you their software, they give you a list
of OSes you can run it on, and you can't deviate one byte from the
packages in those distros without losing all support.

Now, as far as I can tell, Gentoo doesn't have that kind of assurance.
Every build of Gentoo is a crap shoot, because you're building
everything, and the combination of parts getting built together hasn't
been seen together in quite that way before.  So, little mismatches
don't have as much of a chance to get seen before they blow up in the
field.

Or, at least, that's the way I thought things were.  I seem to be
hearing from you that this isn't the way it is.  Perhaps you could
enlighten me.

> Now stop!  I am _not_ giving customers a distro, I am installing an
> image on a "black box" or "serviced" system.  That system *I* provide
> the hardware and service of that hardware for!
> 
> Specific hardware configuration with a specific bill-of-materials I'm
> ensuring I can get the part for 2-3 years, and I am buying that _exact_
> -- over and over.  And if *1* part changes, I make a "Materials
> Review" (MR) to ensure that the entire test process just for the
> hardware is addressed!  I am _not_ installing to more than *1* hardware
> product.  Because I am certifying my hardware configuration with the
> firmware/software I create!

Right.  And hardware *never* degrades over time.  And manufacturers
*never* ship radically different hardware with the same part number.
And manufacturer QA is *always* top-notch.

> Now do you understand where I'm coming from?
> 
> Or do you continue to label what I'm doing as a "distribution"?!?!?!

OK, I get it.  You're hung up on the word "distro" because you think
distros are things that get put on CDs and have installers and lots of
users and GUIs and such.

So what do you call the bits you put on the systems you're building?
And how do you describe the process you go through to change those bits,
for whatever reason?

> > [shrug]  Maybe you should consider that you're talking to someone who
> > may be just as experienced, or even more so.
> 
> But I'm not assuming what you are and aren't capable of.
> *YOU*ARE*!
> 
> You have continually asserted many things with regards to *ME*.
> 
> I have _avoided_ doing such with you, and the only time I point things
> out about yourself is _after_ you make a viewpoint and assert things,
> and I'm trying to say I'm not talking about that!

Well, I assume that you can't fly by flapping your arms.  Is that an
unwarranted assumption?

I mean, I tell you that proper distro testing
(sorry--"shipping-Linux-bits" testing) is the kind of thing you need
piles of employees and labs for, just as a starting point, and you get
all mad at me, that I'm assuming that you aren't better than a pile of
employees and labs.  I tend to take claims of immortality with a grain
of salt.  What can I say?  I'm weird that way. :-)

> > More to the point, I make my living cleaning up messes created by people
> > who think they can fully test a distro all by themselves.
> 
> Then *STOP* taking that attitude out on me!  Damn that's the problem
> right there!  Dude, I'm in the same boat with regards to _others_.  Just
> know, I'm not one of them!

So you don't think you can test a distro effectively?  Then what are you
so mad about?

> I have installed too many financial and defense systems at Fortune 100
> companies to really tolerate this type of "you don't know" crap.  I'm
> not talking web servers -- I'm talking production desktop and servers
> back in the mid-'90s!  Back when people thought Linux was a web or print
> server and little else!

You know, I've done my share of all that.  But, for some reason, I still
think it's a good idea not to take criticism personally, or assume the
person delivering it is just a moron.  Heck, I even try to learn from
it.  You know: if what the other guy is saying sounds like an insult,
assume you misunderstood, and all that.

I know: silly me.  I'm never going to get ahead in consulting without a
major ego enhancement.  Such is life, I guess.

> > But really, all this is just posturing.  What matters is what we know,
> > and how well we know it.
> 
> Then *STOP* telling me what I am and am not capable of based on
> _others_.
> 
> > I think I know the distro market, but who knows?
> 
> I will _not_ assert what you do or don't know.
> So please do _not_ do the same of me!

Why do I get the vague premonition that you're going to break your
promise in just a few minutes?

Most of the engineers I know are humbled by the magnitude of possible
screwups they can't cover, and I've known some world-famous engineers.
Evidently, you've got something on them.

> > But if you're not interested in having your knowledge questioned, then
> > oh well.  I suppose I wasted my time.
> 
> No, I have a real problem with someone asserting what I know and what
> I'm capable of -- from a standpoint you have continually proven to be
> _different_ than what I'm talking about.

Proven?  Now there's an interesting statement.

So far, all you've brought up to me are some hand-waving about
"mission-critical" this and "production" that, and a specific example
about Apache.  And I've told you, several times, that you need to give
me some better examples, because the Apache thing is lame.

So no, you have proven no such thing, at least not to me.

> Again, please refer back to the need for Intel, HP, Dell, etc... to send
> me systems!  Why the heck do I need to test on them if I will _never_
> put my run-time images on them?!?!?!

I suppose you test for other unlikely situations before shipping, right?

> That's why you don't have the faintest idea where I'm coming from.  

Ah, here's why I had that premonition above.  I must be psychic.

> To
> you, there is no market for such things.  If anything, I don't think you
> have ever worked in an environment where you develop software for a
> _specific_ piece of hardware, and there is an entire BOM relationship to
> software.

Well, you'd be wrong if you thought that.  But if you prefer to think of
me as just some rube... well, whatever gets you through the day.  No
sweat off my back.

> I mean, at NASA, we didn't just say ... "oh, well, part X isn't
> available, but we've built our software to run on any type like part X."
> The _entire_ certification of the software release is built on the
> _exact_ BOM!  Right down to the part number, firmware and the tiniest of
> revision.

Unless they're O-rings, in which case the published temperature
tolerances are just suggestions. :-)

But I'm not trying to slam on NASA, or on you.  I'm trying to get you
off your high horse, so you can tell me something interesting.  You're
so intent on *winning* this debate that you have completely ignored
several point-blank questions I've asked you that get to the heart of
why you make the decisions you do.  Had you answered them, you might
have heard me say "Oh, well, yeah, I see it now" and we'd all be happy.

Namely:

 - What's so hard about Apache stack customization that makes it so much
easier to do on Gentoo than Debian?

 - If Apache isn't that hard, what else might be?

 - What advantage is so gosh-darned important that throwing away all
that good QA in the popular package distros can be justified?

I'm even willing to grant that these might be the wrong questions, and
let you tell me what questions I would be asking if I had a brain cell
in my head.  Go on, humor me.  I promise to be a lot less irascible. :-)


-
To unsubscribe, send email to majordomo@luci.org with
"unsubscribe luci-discuss" in the body.