In high school, my dream job was to be a Network Engineer or System
Administrator for an Internet Service Provider. As a junior in college
I got that job. I learned that network operations is really hard:
everything breaks, and automation can only fix part of it. For
example, if the line hunting on BellSouth's 1960s era 1aESS telephone
switch broke, there wasn't anything I could do to fix it except call
it in to Larry Cook down at the CO.

I decided I wanted to do software. In particular I wanted to work at
an Application Service Provider, and run hosted applications — fancy
web sites. Amazon.com was everybody's inspiration in 1999. I loved the
way they tracked everything you looked at or clicked on.

As I was finishing college, I approached my boss, Lee McArthur with my
idea. I wanted to build a gateway between retail stores' existing
inventory management system and the web. The idea is to allow a
physical store to suddenly have an online catalog. And it'd help
people find products in local stores instead of having everything
shipped in.

They decided to start the company. Jimmy Whatley, who also owned the
ISP, backed the project. They brought in the smartest Computer
Scientist I knew personally, Chris Vaughan.

We got started working in July 2009. We worked really hard and wrote a
lot of good code. Our idea evolved into a supply chain management
(SCM) tool for chemical manufacturers, and we had a trial customer.

Along the way, we hired a smart lady, Lin, to work with our users and
write documentation. We hired someone else to develop the web
interface — to make it look pretty. Chris and I could then focus on
the core code. And we hired a programmer to byle our system operations
guy keep the servers healthy and running. We even had an intern CS
major, Justin, who was doing software development for us.

What great debates and design discussions we had!

How to implement fine-grained access controls in Oracle based on roles
stored in the database? (I saw an article in the IBM Systems Journal
about this very problem years after our work.)

How to give web developers freedom without risk they'd accsently
disclose secrets? (we settled on a strict Enterprise Javabean
implementation.)

How to manage configurations on our servers and keep everything
documented, secure, and above all consistent? (cfengine plus
cryptographic signatures on config files, plus CVS revision control.)

How to use the Oracle database but retain the freedom to change to
another DBMS, like Postgres or MySQL? (Don't use features in Oracle
that aren't present in other systems.)

How to access the database from Java code without embedding SQL
statements as string constants inside the source code? (EJB's with the
Jonas EJB container “engine”)

How to run a 24×7 non-stop failure-free system with perfect data
consistency?

How to make backups in an era of 10gb hard drives at a location we
can't physically visit? (a tape library with a robotic tape loader.)

What format to use for Documentation that will let us generate book
and web-viewable pages? (DocBook)

We worked through these and numerous other problems. We didn't just
make plans: we built it and made it work.

Things seemed to be going well, and I wanted to get a graduate degree.
So I went back to school for an MS in Computer Science, and continued
to help a little on our company part time. Chris, Chad, Justin, and
Lin kept things going. I expected they'd do great.

Then our trial customer was purchased by Dupont. And they installed a
mature SCM system called SAP. And September 11 2001 happened, and we
couldn't get another customer to sign up.

Chris et al did some other projects, but eventually they had to move
on to other jobs. It was really rough on them.

—

Looking back, I learned a ton. We did an astounding amount of
innovation in that time. We just didn't get the satisfaction of
bringing our development to a larger market.

In the following years, Dr Fred Brooks told Chris and me that we
probably learned more from that failure than we would have from a
success.

In another way, you could say it was a huge

What'd I learn?

1. Prototypes are the product. Build the simplest thing that can
work, and then evolve it according to actual feedback from users.
Don't try to build the whole thing in the first pass.

2. Get some good sales support. Don't build speculatively: build to
solve real problems. To do that, work with real live customers. If
youre going more than a few weeks without some feedback from users on
the recent changes, you're waiting too long.

3. At our startup, we had policies for everything; e.g. There was a
policy on where an when to buy printer supplies. It may be possible to
program small company policies to ensure good outcomes. But it's often
more fun to just hire people with good judgment and keep aware of
what's going on, then clean up things that were done poorly.

4. Avoid buying anything until you absolutely must. The task of
system engineering is to make the system work at the lowest cost.

5. Don't dream up requirements just for fun or principle. For example,
out fault tolerance goals were noble, but we had no money coming in to
support such requirements. For another example: why did we need to
retain database agnosticism? It's lofty, but expensive, and ultimately
added a lot of cost to the development.

6. Use “tracer bullets” that test the full path through the system.
One way of doing this is to Keep the whole application running, all
the time, so you know what performance is like. On our system, we
built the layers separately, then hooked it all up. When we tested it,
the processing time was really too slow, so major optimization
research (profiling) was required.

7. Engineers love to design and build. It's what we do. So it's
natural that we'll dream up problems, then find ways to solve them.
It's what we do. But without definite spending budgets (which implies
time deadlines, when you consider payroll) Engineers will go on
creating these problems and their solutions indefinitely. I saw this
again on a much larger scale at the Be11South VoIP development project
in 2005: Engineers with no limits could design new problems and solve
them forever.

(I may add to this list later. )