Sunday, November 30, 2008

Large vs. Small Businesses

When I first left graduate school I worked for about a decade for a very large company with close to one hundred thousand employees. After that I started working with and for smaller companies, often with fewer than ten employees. The company for which I currently work is less than ten years old. In that time it has grown from about five people to around two hundred. During my three years at this firm the feel has moved from that of a small company to that of a large company. This has prompted me to think about both the forces that favor either large or small businesses and the human consequences of each.

I believe that organizations are becoming larger, that this is inevitable, but it is a dangerous trend for a couple of reasons. One is increased risk to the society as a whole. A second is depersonalization that treats humans as disposable commodities.

THE LARGE BUSINESS ADVANTAGE

In the United States, the Census Bureau keeps statistics on industries, employees, payroll and revenue. See http://www.census.gov/csd/susb/susb.htm and http://www.census.gov/epcd/www/smallbus.html About seventy five percent of businesses have no employees but these account for less than four percent of all business receipts and will be ignored here.

My data analysis here is simple minded, but I think the trend is obvious enough to eliminate subtlety.

In 2002 businesses of 500 or more employees accounted for about 59% of all receipts, 50% of all employees and about 55% of all payroll.

Very large businesses, those with 10,000 or more employees, accounted for about 35% of all receipts, 27% of all employees and about 30% of all payroll.

In 2002 there were about 5.7 million businesses with employees. Of these, 891 (.015%) have sales or receipts of 2.5 billion or more. That small fraction of businesses accounts for 42% of all sales or receipts. They account for 23% of all employees and about 29% of all payroll.

100% of employees generate 100% of receipts, but these are not evenly distributed for firms of different sizes. The 891 businesses with receipts of 2.5 billion or more got 42% of receipts with only 23% of the employees. That means the rest of the businesses require 77% of all employees to get 58% of receipts. Larger businesses pay their employees more than average. For those 891 firms, their 23% of employees get 29% of all payroll, but that 6% difference in payroll/employment is dwarfed by the 19% point difference between receipts/employee.







Firm Sizereceipt%employee%payroll%receipt% - employee%payroll% - employee%
500+ employees59505595
10,000+ employees35273083
2.5 billion+ rev.422329196


Looking at different size businesses, the disproportionate revenue and employees hold, but it is not as simple as more employees implies more efficient. Other issues, like the type of industry, play an important role. I do not have the data or the time for more than this cursory analysis. Larger businesses seem to have an advantage.

Business failures have been studied. Here too, larger businesses seem to have an advantage. Larger business go bankrupt less often and failures take more time. Smaller businesses are more inclined to quick death rather than a long lingering death. The reason for these failures have been studied, but the results tend to boil down to "bad management", by which is meant lack of planning, lack of proper controls, or even bad salesmanship. Because everything is more ad hoc in smaller companies, the opportunity for both mistakes and fraud increase.

I have not found good data on long term trends in business size, but I think it is safe to say that the largest organizations have been getting larger and that large organizations take a larger portion of the economic pie. Much of this is due to the computer and telecommunications. Computers have automated processes and eliminated the need for human involvement. Telecommunications allow those automated processes to be accessed from anywhere and at any time. I can go an a trip with no human contact with either the airline or my bank. Reservations and accounting are completely automated. The volume of transactions handled by the average airline would have been impossible a century ago. In the same way, markets can be searched and supplies bought with little human intervention. This allows for more efficient business, but only if the business venture is large enough to buy the expensive systems needed.

LARGE BUSINESS CHARACTERISTICS

I think the root of the advantage for larger businesses is a larger financial cushion. Small business tend to operate closer to the edges. One edge is the number of customers. It is not uncommon for a small company to rely on a small number of clients for the lion's share of income. If something happens to the relationship with one of those major customers, the small business may have no way to recover. On top of that, small companies often borrow money for initial company set up and for expansion. This debt is a fixed operational cost while revenue is dependent on highly variable sales. Large businesses typically have some diversification so that an unsuccessful segment can be subsidized by more successful segments. This makes the total revenue less susceptible to radical swings.

Larger sales volume also means larger absolute profits that can be reinvested. A small company selling a million dollars a year with a 10% profit margin gets one hundred thousand dollars a year that can be invested in product improvements and company efficiency. A large company selling one hundred million dollars a year with a 5% profit margin has five million dollars to invest.

Companies with more customers can also amortize investment over the larger set of customers. I suspect the most comfortable chair each of us owns is the driver's seat of our car. The driver's seat has been undergoing research and development for close to one hundred years. Sales volume at a large car manufacturer gives it the funds to invest in design to keep up with the latest materials and technology. For a company like Ford, GM, or Toyota, I suspect there is at least a man-century of engineering design invested in their automobile seats.

When a company decides to reinvest profit there are two major areas where the money can be spent. One is business process and the second is product development. Over time, the processes used to decide where to reinvest money tend to be formalized. Money is tracked more carefully and each person must be prepared to explain where and why money is being spent. This leads to careful evaluation of prospective markets and the best way to meet those needs. Tight accounting processes reduce the risk of small scale fraud. On the product development side, manufacturing processes are streamlined and the methodology for creating and refining products becomes more formalized.

Conceptually, this formalization is a strength because the business can constantly improve processes. As the process becomes more fixed, standardized documentation of work and results become more important. The person who performs the work has fewer choices in a standardized process. Jobs are categorized and it becomes less important which individual from the category does the work. This depersonalization allows the business to continue smoothly in the face of personnel changes and reduces the skill level needed for any particular job. At the same time, the business can afford to pay somewhat higher wages and, theoretically, attract more competent employees.

More formal processes take a great deal of time and energy to create and just as much to modify. That ossifies the business processes. In addition, the emphasis on written, formal communication as the basis of decision making slows the business.

I do not know of any evidence that shows that more formal planning processes are more effective the the ad-hoc planning that tends to be used in small businesses. In both cases, the basic decision making is actually made at a high level of the organization, usually by a single person. The formal process simply reinforces and refines the decision. In fact, the basic direction is generally decided before the machinery is started. More formal development practices are also largely unproven. Formal processes tend to quantify and avoid yesterday's errors, but they do not prevent tomorrow's.

SMALL BUSINESS CHARACTERISTICS

By contrast, small businesses tend to have fewer established procedures. Workers are not interchangeable and the enterprise tends to be more personality driven. In small enterprises jobs are less categorized and people have both a wider range of tasks and more discretion in how they are performed. Small businesses are less stable year to year than a large business, but day to day they are generally more humane and pleasant to work in.

The lack of completely established procedures allows small business to evaluate the business situation day by day and to respond quickly to changes. Most of the formal planning processes used by larger businesses do not work any better than the ad-hoc planning of smaller businesses, but large businesses have guaranteed overhead that makes them slower. Large business review processes also make it less likely that any given idea will make it to the marketplace. Lots of people can say no. Few people can say yes.

There are some business niches that require a large capital investment to exploit (e.g. large LCD screen manufacturing). Aside from these, small organizations are much more likely than large firms to find and exploit new opportunities. Large organizations simply cannot perceive and act quickly enough.

There is a symbiotic relationship between smaller and larger businesses. Larger businesses often view small businesses as a zero cost research and development arm. Most small businesses fail. The successful small business identifies a market opportunity. If the opportunity is one that can be scaled, the small business becomes a target for acquisition by a larger one. One of the most common strategies for high-tech entrepreneurs is to identify a niche that new technology makes available, create a small company that exploits the niche, then cash out by selling the small company to a larger one when the market is confirmed. This works well for the small business owners, but less well for the employees.

WHY VERY LARGE BUSINESSES MAY BE BAD FOR US

Network theory has shown us the value of self organizing systems of small components. This is true of both biological and technological systems. Systems created in this way can be self healing and require less intelligence at each node in the system. That is because failures tend to be local and are locally repaired. For example, internet routing is remarkably resilient in the face of many single failures because each routing node can detect and work around failures of its neighbors.

An economy based on fewer, larger players loses these advantages. Simply because of their size, large enterprises tend to be tightly coupled. The meltdown in 2008 showed this quite starkly. All of the large financial players were tightly coupled in risk and the risk was contagious. The mechanisms used to hedge the risks in fact increased the coupling and magnified the risk. Home loan risk reduced the capacity of the system for all loans and that infected all businesses that required credit.

Having fewer, and larger players is inherently riskier for the economy as a whole. This is not a problem of monopolies. It occurs when power in any segment of the economy is concentrated among very large interconnected players. A second problem is in the relationship between the company and its employees.

As a manager in a large business, my concern is business efficiency and profit. The natural tendency of the business to formalize processes and standardize jobs aids this because it makes labor more interchangeable. From an economic point of view the ideal labor market is one where both workers and employers are completely unbound. That is, if a better opportunity comes along for me as a worker, I should quit and take it. As an employer, if a more cost effective worker is available, I should fire a current worker and hire the new one.

As a high tech worker, this open marketplace has been the reality for me. Over the past fifteen years I have changed jobs about once every three years. About half of those changes were because I was offered a better position somewhere else. About half of the changes were because the business folded or was bought.

This open market of employees is very good for employers and would be good for employees if opportunities for work existed everywhere, and the workers were perpetually twenty five years old and in good health. However, that is not the real world. Jobs are often moved wholesale from one location to another leaving behind high unemployment that cannot be absorbed by the local market. For individuals, a perfectly open market is simply a way for employers to dump workers with temporary or permanent problems and to shift work location for even minor economic gains.

This type of labor fluidity is only reasonable if we as a society commit to those left behind. So far, I think the United States has done a very poor job of helping unemployed workers create and take advantage of new opportunities.

It is my experience that small enterprises, by their very nature, care more about individual employees. Small groups of people form loyalties. Any business faced with survival, will fire employees. In a small business, these decisions are personal. In a larger business, the decisions are impersonal. I have worked with a number of large employers where the decision to fire large numbers of employees and move the work to a different country is strictly a spreadsheet decision. If we are solely concerned with short term business efficiency, that is how the decision should be made. If we care about the communities in which we do business, if we care about the people with whom we work, this is not how the decisions should be made.

Sunday, November 9, 2008

Software and Chaos

As with many of my blog posts, this is a little long. To make things slightly easier, here is a map

The Problem
My Experience
Business Expectations and Results
Software and the Elevator Speech
All is Not Lost

THE PROBLEM

Roughly seventy five percent of all software projects fail in the sense that they cost more than expected, deliver less than expected, or are delivered later than expected. This has been true for decades. Behind the scenes at these projects is untold stress and pain as development teams and management try to cope. This coping typically starts when it becomes apparent that the carefully constructed project plan does not describe what is happening and "success" is unlikely. I would like to believe that my projects perform better than average, but many of them go through this readjustment to reality. It starts with a meeting where the current situation is explained and options are explored. At some point, someone asks what went wrong, and all eyes turn toward me as system architect and technical manager.

I pride myself on being honest and not ducking blame. I go through a careful analysis of projections and actualities. I point out places where we made mistakes and where we may have lost time, but also explain that this is not where most of the problems lay. I point out the areas where there was unanticipated work or where an underlying system did not perform as expected.

All of this is true, but it is not the actual problem. The actual problems are structural, and most organizations with whom I work will never make the changes necessary to rationally develop software. They will not change because they cannot accept the truth about the type of software development I work on. No team, regardless how smart or how much time is spent in planning, can accurately predict the effort necessary to produce the software. The predictions of effort are based on what is known and are always optimistic.

MY EXPERIENCE

Let me preface this with some characteristics of the projects I work on. My generalization may not apply to other types of work. I work on new development. Typically it is putting a system in place to replace some existing process. Often there is no existing software in place. In the case where there is existing software, the organization plans to completely scrap that system. Along with new software, the business processes will be substantially changed. The projects are usually medium sized, say three to fifteen developers working for between six months and a year and a half.

The parent organization is usually quite large, but not a software development house. They usually have an IT department that maintains and expands existing software. They may have a reasonably large body of homegrown software, but developing new products is not their expertise. The existing IT staff is often expected to do much of the development work. It is common for the organization to have outsourced their data center operations.

These are generally business and process problems, not the development of shrink wrapped products. I have worked on products as well. The details of the problems change in product development, but the basic outline remains.

BUSINESS EXPECTATIONS AND RESULTS

Modern business is based on rational sounding self deception. As a rational business person faced with investing in the future I want to get answers to a set of questions.

What business are we in?
What and when are the opportunities?
How much will it cost to address the opportunity?
How much will we earn if we are successful?
What are the risks of action and inaction?

There are existing tools to address each of these questions. Upper management usually spends a lot of time and money to define the business. This includes vision statements as well as long and short term goals. Market research is used to answer the what, when, and size of opportunities as well as the cost of inaction. Technology research and initial product research are used to estimate the cost of action.

Given this information, I can compare projects to project the highest return on the investment. As a rational person I want to guard against incomplete or incorrect information, so I will want information at every step of the way to make sure assumptions were valid and to readjust the course. Recognizing that it is cheaper and easier to correct errors early in the process, I will want to make sure that we understand the problem and the solution as early as possible. This leads to a typical waterfall process.

Gather Requirements – understand the problem
Detailed Design – understand the solution
Implementation – fill in the details
Test – make sure users see a reliable product
Release – cha ching - here is where we get the return on investment
Support – if we have done things well, we can minimize support

After every step of this process there are checkpoints to assess progress and readjust based on our best information.

This is called a waterfall process because it moves in one direction, at the end of any stage, it is difficult to move back upstream.

The end result of this rational process is generally failure. Not just failure to deliver the software as envisaged, but failure at every step. Management changes in the middle of a project are common. With every change in management there is a shift in the way the business is understood. What was critical yesterday is marginal today. Market research is almost invariably wrong. You can confirm this yourself. Take a look at your marketing reports from two years ago. Project requirements change in the middle of the project. This is one of the most commonly listed critical problems in software post-mortems. The implementation takes longer than projected. As the pressure mounts because the project is late, developers hack functionality together to meet deadlines. Adequate testing is often one of the first casualties. As the bugs mount and the project slips, functionality is cut. Test exposes defects that require rework and sometimes the rework is not simple bug fixing. Sometimes the fix requires major architectural work. When release finally occurs, users find that things they consider essential have been chopped out, or the carefully crafted requirements do not correspond to the way users work.

The sad reality is that this process will fail for all but the most constrained projects. The large uncertainties inherent in the world combined with our limited ability to understand and communicate our understanding doom the entire methodology.

In the software community, almost no one believes that waterfall processes are either effective or desirable. The first descriptions of the process describe its failures as well. The failures have been spectacular. The US Federal Government has had numerous multi-billion dollar projects that have been complete losses (FAA - 2.6 billion IRS - 4 billion http://spectrum.ieee.org/sep05/1685/failt1). The money would have been better spent if it had been bundled into logs and burned to provide heat for the poor. Despite this, many large organizations mandate a waterfall approach. In large organizations around the world, untold amounts of money are spent to describe, mandate, and adjust a process that will never work.

SOFTWARE AND THE ELEVATOR SPEECH

"I'm a busy man, tell me what you want." This is the origin of the elevator speech. As an entrepreneur you find yourself in an elevator with the very person who can fund your project. You walk in and by the time the elevator gets to your floor you want to hook the prospective funder. To do so you answer the implicit questions, What am I going to get (and why should I care), when am I going to get it, and what will it cost? You have perhaps a minute to hook the person.

The basic problem is that the "what", "when", "how much" questions are essentially impossible to answer. The world is chaotic in the mathematical sense. That is, the circumstances that lead to wild success are often indistinguishable from the circumstances that lead to complete disaster.

Even in constrained fields like books and movies where the processes to develop the end product are well known and understood, no once can predict the success of the final product.

For new software, the prediction problems are magnified. The problems I address typically span multiple groups in an organization and involve manual processes that currently require human judgement. It is common that no single person in the organization understands the entire problem. Like the blind men and the elephant, each individual understands the the part they are touching, but no one sees the whole.

I always say that there is only one thing worse than fully specifying a computer system before code is written. It is getting the system that was specified. No matter how carefully we work, the requirements specified at the start of the project often prove untenable in real life.

At best, the process itself encourages wishful thinking. At worst, outright lying. Software is expensive. In a world where the low bid sets a general bar, there is every reason to underestimate the uncertainty and to assume that everything will work as predicted. As a result, I get to sit in meetings where someone asks the "what went wrong" and all eyes turn to me.


ALL IS NOT LOST

Although this picture is bleak, the software development community has developed techniques that actually work. Most of these fall under the rubric "agile development".

Agile development is the antithesis of the elevator speech. The answer to the what, when, how expensive questions are: I don't know, I don't know and I don't know. Agile development says that if you want to get effective systems in place, you get a bunch of smart people in a room and let them loose. The pact with the business is:

Choose the project as best you can.
Define a level of investment.
Involve customers from the first day to the last.
Address the most important customer problems first.
Produce working software early and often.
Insist on complete transparency.
If cost is too high or benefit too low, pull the plug.

That is, as a software provider I will provide actual value as quickly as possible and if I don't earn my keep, fire me. Because large systems generally have a roll-out to a large number of end users, it may be a long time before the system is actually deployed, but every three weeks or a month there is a new, actual working system that can be evaluated, tested and re-directed. When that system provides enough value to warrant its deployment, roll it out. After roll out, continue investment so long as the implemented improvements justify their own cost.