Why Does Government Interfere With Markets?

Since I wrote my blog post on big vs. small business I have been thinking a fair amount about the role of commerce in society as a whole and our view of commerce in the United States.

This story has to do with the cultural narrative about economics and choice. In some of the circles in which I run it seems that people have a simplistic notion of economics. The basic story is that the free market, unencumbered by government intervention, is an ideal condition and that deviation from this ideal is a distortion that is bound to backfire. I do not put this argument forward as a straw man. I hear the simplistic point of view expressed and I think it influences public debate and policy in the United States. Since the financial meltdowns and obvious abuses of 2008, the argument has lost some power, but it is still seems to be a bedrock view for many people.

The argument starts off with free market economics. In a free market buyers and sellers negotiate with each other to establish prices. Goods that better satisfy a need are more popular in the marketplace and command a higher price and perhaps higher profit. Higher profit draws competition which tends to reduce price and profit. This in turn spurs innovation in both production (lower production costs at a fixed price increases profit) and the good itself (better satisfaction of the consumer need leads to higher price and profit).

The second step is laissez-faire economics. If a free market leads to better satisfaction of self stated needs (ever cheaper and better products), then any interference with the operation of a free market is a distortion that will lead to less than optimal results. In this view, government intervention in markets should be restricted to ensuring that they are free. Any discussion of government intervention centers around whether the market has ceased to be free and what the corrective mechanism should be. For example, intervention to break up monopolies may be considered to be a restoration of a free market. If there is a single supplier of a good and that supplier can prevent the emergence of competitors the free market ceases to operate and government may intervene to restore the free market.

In outline this seems sensible and, I believe, is the basic narrative for economic discussion in the United States. There are some hidden simplifications in this basic narrative. Serious discussions of economics and society should at least acknowledge some of the additional complexities. I will go through a some of these in what I consider to be reverse order of importance. As I move toward the more important considerations the discussion moves farther from strictly economic concerns.

EXTERNALITIES

Hidden in the free market explanation is the assumption that the important parties in an economic transaction are the buyer and seller. This is often untrue. The classic case is pollution. A mine or factory may produce a good but as a side effect pollute air, water or land. The people downwind or downriver may pay a price for additional health care or reduced utility of their property, but this cost is not reflected in the transaction between buyer and seller. These are called externalities because they are costs or benefits that are external to the buyer and seller in the transaction. When the externality is a cost, the price paid for a good is artificially low, so the good tends to be overproduced relative to the overall benefit. While we normally discuss externalities in terms of costs there can be external benefits as well.

Externalities are sometimes time-based. This brings up a second shortcoming of the simple free market story. Transactions are based on immediate needs and products. It is true that each actor may be planning for the future, but the transaction is today and the evaluation of price takes only the immediate situation into account. This is easy to see in the exploitation of natural resources. The cost of tropical hardwood is based on the immediate supply of tropical trees. As long as there are large stands of forest, the cost of the hardwood basically reflects the cost of cutting and transport. As the forests decline, the wood will become more scarce and the price will increase. But that will only occur after the native utility of tropical forest is lost, essentially forever. The immediate cost and profit available from the trees is taken into consideration, but future utility of the rainforest is not.

For future externalities some of the problems are often simplified as the "tragedy of the unregulated commons". For situations like logging and fishing, the resource is not controlled, so there is no immediate incentive for anyone to restrict their activity. Each individual has an incentive to exploit the resource until it is gone.

In the United States the predominant reaction to the tragedy of the unregulated commons is to privatize the resource. For example, pollution can be addressed by capping and trading emissions. Resources like fisheries can be sold to individuals. This gives the owner a greater incentive to consider the long term in managing the resource. There is another solution. The basic problem is the tragedy of the unregulated commons. Resources can be regulated instead of owned. For example, Balinese rice farmers traditionally handled allocation of water by setting irrigation schedules. The water was controlled by the community instead of individuals and a water board arranged irrigation schedules to grant use to the farmers in the community. Once we agree on a problem, the solution may depend on both the physical and social situation. For example, for preserving fisheries a combination of protected reserves around the most productive areas and owned fishing rights in other areas seems to be most effective.

EXTERNALIZING COSTS

To the extent they can, businesses will always externalize costs. A cost not paid is direct profit to the business. A simple example can be seen in the retail market. In many stores there is no one who actually understands the products being sold. The job of understanding the products and their fitness for the task is left completely to the consumer. This allows the store to hire fewer, less skilled, and cheaper workers. On the whole, consumers seem to prefer this model. There has always been a conflict of interest between the salesman as a purveyor of information and the salesman as someone trying to sell the products on the floor. Another example, is "big box" stores like Sam's Club or Home Depot. These are essentially warehouses open to the public. The selection and price make it likely that the consumer will find something close to what they need at a relatively low price. The externalized costs include transportation. The larger store draws customers from a larger area, so rather than have to reship to more, smaller, local stores the company can stock a single warehouse and let the consumer transport goods the final miles.

For many businesses the largest cost is labor. Lowering labor costs can be done in a number of ways. Typical techniques include automation, de-skilling work, and externalizing labor costs. Automation typically reduces the amount of human input needed in a process. On the whole, this tends to increase the quality and reduce the cost of goods and services. Skill implies experience and training. Skilled workers get higher pay because it is cheaper for the business to pay a higher wage to a known skilled worker than to risk the time, energy and lost productivity of training an unskilled worker. Externalizing labor costs can be done in a couple of ways. Pushing needed work outside the boundaries of the business decreases labor costs. So does forcing labor costs outside the business.

The contemporary supermarket shows all these trends. The supermarket itself was an innovation that reduced labor costs by having the consumer choose products directly off the shelf (externalize work). Two generations ago, supermarket prices were marked on each object. Stockers had to both put items on the shelf and mark the prices. The advent of the bar code and scanner eliminated the need for marking prices on individual items (automation). It also eliminated the need for checkers to know produce prices or how to work a cash register (de-skilling). The checker's job today is sufficiently deskilled that customers now check themselves out (externalize work). This does two things, it reduces the number of employees needed and it reduces the wages of those who are left because they can be replaced easily.

Forcing labor costs outside the business is an interesting development. Businesses do this on both a short term and a long term scale. When there are jobs, highly skilled workers are largely exempt from these processes because they can demand and receive higher wages. In the short term, unskilled labor always commands a lower price. The floor of this price is somewhere near the minimum amount of pay it takes for a single person to live. The existence of a public safety net reduces this number. For example, if a low income person can have food subsidized through government food stamps, and medical care through Medicaid, then the business hiring low skill workers can effectively subtract that amount from the wages paid to low skilled workers. In one idealized labor market, workers move completely freely between employers and employers pay for exactly the work done while the person is working. Unfortunately, people sometimes get sick and always, if they are lucky, get old. The most fortunate among us spend about a quarter of our lives under the care of others and unable to gainfully work. The society as a whole must pay for this downtime. For a brief period of time in the United States, many employers took long term responsibility for workers in the form of pensions. These defined benefit plans have essentially disappeared and it is the responsibility of employees to make provisions for old age in the form of defined contribution plans (e.g. 401K plans). Employers may provide such retirement plans, but are not compelled to. Employers may contribute to these plans, but are not compelled to. The net effect is to push the long term care of those who cannot work, particularly low wage earners, onto the government.


ASYMMETRY OF POWER

In a true free market, both buyer and seller freely enter into an agreement and choose to make the trade. In fact, this ideal rarely holds. When we add to that the almost limitless capacity for cruelty in our species, abuse and exploitation become almost inevitable.

Asymmetry in trading power only matters if one party or the other is effectively forced to trade. It is unimportant for non-essential goods. For example, manufacturers of television sets can conspire on price. This will raise their relative prices, but if the price becomes too high, buyers will simply withdraw from the market. Essential goods include employment, food, and shelter. Employment is probably the most important of these and the root cause of much abuse.

First let me say that employment is a squishy concept because individuals do not have to trade their labor for money. Anyone who has skills or goods can become self employed. That said, the ability of people to become self employed also depends on macro economic conditions. The level of self-employment has varied tremendously over time. In the United States, self employment probably reached its zenith in the early Eighteenth Century when vast tracts of land were opened to ordinary people and the ideal of a country of yeoman farmers seemed within reach. Right now, no more than 10% of the population is self employed http://www.smallbusinessnotes.com/aboutsb/rs243.html.

When there are more people than jobs, pay decreases and conditions become more miserable. To understand the necessity of government intervention, the words "triangle shirtwaist factory" should be sufficient. The height of abuse can be seen in company towns, where the employer controls not just employment, but food and shelter as well. Prices for necessary commodities can be set slightly higher than wages resulting in virtual slavery. As the Merle Travis song says:

You load sixteen tons and what do you get?
Another day older and deeper in debt.
Saint Peter, don't you call me 'cause I can't go,
I owe my soul to the company store.

LOCAL DECISIONS, GLOBAL OPTIMUMS

I certainly feel best when I can freely make my own choices. I like to think I make good decisions and I suspect others feel the same way. It seems to be an article of faith in the U.S. that these sorts of personal and local decisions lead to an optimum (whatever that means) for the society as a whole. Or if individual decisions do not lead to a global optimum, they are still better than any other decision making regime. This fits well into laissez-faire, free market economics. If each of us is left alone in the economic sphere things will work out as well as they can.

We can take this a step further and say that the definition of global optimum is the result of completely free economic choice. Let us avoid that tautology. In social situations, where each person has his or her own views, it is hard to make any decisions about "best" or "optimal". However, there are situations that we can agree are "not best". As a society we have some agreements about unacceptable situations. Check the papers after someone has been denied medical care and died as a result (http://www.msnbc.msn.com/id/25475759/, http://www.bookrags.com/news/911-dispatchers-denied-dying-woman-moc/).

The notion that laissez-faire economics leads to any kind of global optimum in the real world is vacuous. As any physicist will tell you, there are local optimums that are difficult to escape, but are far from a global optimum. In the economic sphere the parties to a transaction often cannot make free and rational choices; there are almost always inherent asymmetries in power and information. Given this, it takes blind faith, unsupported by fact, to assume that a reasonable, sustainable society can be built strictly on economic choices.

Some government interventions are aimed at creating a better global situation than simple economic transactions can provide. Public funding for education is one example. Everyone, particularly property owners, is charged for public education whether or not they have children. The same is true of child nutrition programs. The benefits of these programs cannot be seriously questioned. The quality of childhood education and nutrition drastically affect the quality of life and the productivity of the adults these children become. By making education and nutrition artificially cheap, we encourage "overuse" of these "products". The benefits may be clear, but the distance in time and space between investment in the child and the payoff in the improved quality of life of the adult is too large to be handled by ordinary transactions.

NOT EVERYONE PARTICIPATES ECONOMICALLY

In most societies, only about half the population directly get income. The other half are too young, too old, too infirm, or participate in direct unpaid work caring for those people. That means that on average, each wage earner supports a non-wage earner. The largest part of this income transfer is within families as family members take care of each other.

It often happens that those who cannot work do not have families to take care of them. Sometimes the most economically ruthless among us seem to take the attitude Oscar Wilde parodied. "To lose one parent Mr. Worthing may be considered misfortunate, to lose both looks simply careless". In all industrialized nations there are welfare programs that take the place of family income transfer where that is not available. We have these programs because, as a society, we have decided we do not want people starving and dying on the streets.

Even when people are gainfully employed, it often happens that there are catastrophic events that can bankrupt them. These are typically medical emergencies, but there are other emergencies like fire, tornadoes, hurricanes and global economic collapses. All industrial countries have a mix of private insurance and government aid for these events. In the United States, there is a large reliance on private insurance and personal bankruptcy. The government does step in for national disasters with a combination of direct aid, grants, and loans. For medical expenses, the government provides a floor on care with Medicaid and Medicare. To smooth temporary job dislocations, unemployment insurance is mandated.

Where there are welfare (including Social Security in the United States) and emergency aid programs, they must be paid for. In the United States these programs consume roughly half of the federal budget. That means overall taxes must be set high enough to cover these costs. These taxes are straight, necessary, income redistribution.


ECONOMIC VALUE VERSUS HUMAN VALUE

My basic objection to the simple laissez-faire free market story is much more basic. In the simple story, monetary value is essentially equated with human value. This view cheapens us all.

We live in a money economy. The economy puts a value on goods and services. The tendency to confuse monetary value with personal value is natural. As I look at my life, the falseness of this becomes clear. Many of the things that make my life worth living have no monetary value and I would be insulted if anyone tried to put one on them. For example, the chance glance of a beautiful woman, a sunset, a musical passage that I happen to play well. I could pay a beautiful woman to smile, but the mere existence of payment cheapens the experience. Things I purchase often have a personal value far in excess of the monetary value. My filing cabinet and my cordless power drill are two inexpensive items. They are inexpensive because production and intellectual property costs are low. They have improved my life immeasurably and I value them greatly. Although their manufacturers would be delighted if they could charge me in proportion to the personal value I place on them, this would be unfair and artificial.

Human value is distinct from monetary value. Sometimes it is practical to pretend that human and monetary value are interchangeable (creating markets for clean air) but let us acknowledge that this is a pretense. We are economic animals, but that does not mean we can be explained, or our actions should be based, strictly on economics. This simplistic view makes no more sense than insisting that because we are sexual animals, all our behavior and decisions are based on sex.

Because I have gotten to sex, I feel I can end this little essay.

Large vs. Small Businesses

When I first left graduate school I worked for about a decade for a very large company with close to one hundred thousand employees. After that I started working with and for smaller companies, often with fewer than ten employees. The company for which I currently work is less than ten years old. In that time it has grown from about five people to around two hundred. During my three years at this firm the feel has moved from that of a small company to that of a large company. This has prompted me to think about both the forces that favor either large or small businesses and the human consequences of each.

I believe that organizations are becoming larger, that this is inevitable, but it is a dangerous trend for a couple of reasons. One is increased risk to the society as a whole. A second is depersonalization that treats humans as disposable commodities.

THE LARGE BUSINESS ADVANTAGE

In the United States, the Census Bureau keeps statistics on industries, employees, payroll and revenue. See http://www.census.gov/csd/susb/susb.htm and http://www.census.gov/epcd/www/smallbus.html About seventy five percent of businesses have no employees but these account for less than four percent of all business receipts and will be ignored here.

My data analysis here is simple minded, but I think the trend is obvious enough to eliminate subtlety.

In 2002 businesses of 500 or more employees accounted for about 59% of all receipts, 50% of all employees and about 55% of all payroll.

Very large businesses, those with 10,000 or more employees, accounted for about 35% of all receipts, 27% of all employees and about 30% of all payroll.

In 2002 there were about 5.7 million businesses with employees. Of these, 891 (.015%) have sales or receipts of 2.5 billion or more. That small fraction of businesses accounts for 42% of all sales or receipts. They account for 23% of all employees and about 29% of all payroll.

100% of employees generate 100% of receipts, but these are not evenly distributed for firms of different sizes. The 891 businesses with receipts of 2.5 billion or more got 42% of receipts with only 23% of the employees. That means the rest of the businesses require 77% of all employees to get 58% of receipts. Larger businesses pay their employees more than average. For those 891 firms, their 23% of employees get 29% of all payroll, but that 6% difference in payroll/employment is dwarfed by the 19% point difference between receipts/employee.

Firm Size	receipt%	employee%	payroll%	receipt% - employee%	payroll% - employee%
500+ employees	59	50	55	9	5
10,000+ employees	35	27	30	8	3
2.5 billion+ rev.	42	23	29	19	6

Looking at different size businesses, the disproportionate revenue and employees hold, but it is not as simple as more employees implies more efficient. Other issues, like the type of industry, play an important role. I do not have the data or the time for more than this cursory analysis. Larger businesses seem to have an advantage.

Business failures have been studied. Here too, larger businesses seem to have an advantage. Larger business go bankrupt less often and failures take more time. Smaller businesses are more inclined to quick death rather than a long lingering death. The reason for these failures have been studied, but the results tend to boil down to "bad management", by which is meant lack of planning, lack of proper controls, or even bad salesmanship. Because everything is more ad hoc in smaller companies, the opportunity for both mistakes and fraud increase.

I have not found good data on long term trends in business size, but I think it is safe to say that the largest organizations have been getting larger and that large organizations take a larger portion of the economic pie. Much of this is due to the computer and telecommunications. Computers have automated processes and eliminated the need for human involvement. Telecommunications allow those automated processes to be accessed from anywhere and at any time. I can go an a trip with no human contact with either the airline or my bank. Reservations and accounting are completely automated. The volume of transactions handled by the average airline would have been impossible a century ago. In the same way, markets can be searched and supplies bought with little human intervention. This allows for more efficient business, but only if the business venture is large enough to buy the expensive systems needed.

LARGE BUSINESS CHARACTERISTICS

I think the root of the advantage for larger businesses is a larger financial cushion. Small business tend to operate closer to the edges. One edge is the number of customers. It is not uncommon for a small company to rely on a small number of clients for the lion's share of income. If something happens to the relationship with one of those major customers, the small business may have no way to recover. On top of that, small companies often borrow money for initial company set up and for expansion. This debt is a fixed operational cost while revenue is dependent on highly variable sales. Large businesses typically have some diversification so that an unsuccessful segment can be subsidized by more successful segments. This makes the total revenue less susceptible to radical swings.

Larger sales volume also means larger absolute profits that can be reinvested. A small company selling a million dollars a year with a 10% profit margin gets one hundred thousand dollars a year that can be invested in product improvements and company efficiency. A large company selling one hundred million dollars a year with a 5% profit margin has five million dollars to invest.

Companies with more customers can also amortize investment over the larger set of customers. I suspect the most comfortable chair each of us owns is the driver's seat of our car. The driver's seat has been undergoing research and development for close to one hundred years. Sales volume at a large car manufacturer gives it the funds to invest in design to keep up with the latest materials and technology. For a company like Ford, GM, or Toyota, I suspect there is at least a man-century of engineering design invested in their automobile seats.

When a company decides to reinvest profit there are two major areas where the money can be spent. One is business process and the second is product development. Over time, the processes used to decide where to reinvest money tend to be formalized. Money is tracked more carefully and each person must be prepared to explain where and why money is being spent. This leads to careful evaluation of prospective markets and the best way to meet those needs. Tight accounting processes reduce the risk of small scale fraud. On the product development side, manufacturing processes are streamlined and the methodology for creating and refining products becomes more formalized.

Conceptually, this formalization is a strength because the business can constantly improve processes. As the process becomes more fixed, standardized documentation of work and results become more important. The person who performs the work has fewer choices in a standardized process. Jobs are categorized and it becomes less important which individual from the category does the work. This depersonalization allows the business to continue smoothly in the face of personnel changes and reduces the skill level needed for any particular job. At the same time, the business can afford to pay somewhat higher wages and, theoretically, attract more competent employees.

More formal processes take a great deal of time and energy to create and just as much to modify. That ossifies the business processes. In addition, the emphasis on written, formal communication as the basis of decision making slows the business.

I do not know of any evidence that shows that more formal planning processes are more effective the the ad-hoc planning that tends to be used in small businesses. In both cases, the basic decision making is actually made at a high level of the organization, usually by a single person. The formal process simply reinforces and refines the decision. In fact, the basic direction is generally decided before the machinery is started. More formal development practices are also largely unproven. Formal processes tend to quantify and avoid yesterday's errors, but they do not prevent tomorrow's.

SMALL BUSINESS CHARACTERISTICS

By contrast, small businesses tend to have fewer established procedures. Workers are not interchangeable and the enterprise tends to be more personality driven. In small enterprises jobs are less categorized and people have both a wider range of tasks and more discretion in how they are performed. Small businesses are less stable year to year than a large business, but day to day they are generally more humane and pleasant to work in.

The lack of completely established procedures allows small business to evaluate the business situation day by day and to respond quickly to changes. Most of the formal planning processes used by larger businesses do not work any better than the ad-hoc planning of smaller businesses, but large businesses have guaranteed overhead that makes them slower. Large business review processes also make it less likely that any given idea will make it to the marketplace. Lots of people can say no. Few people can say yes.

There are some business niches that require a large capital investment to exploit (e.g. large LCD screen manufacturing). Aside from these, small organizations are much more likely than large firms to find and exploit new opportunities. Large organizations simply cannot perceive and act quickly enough.

There is a symbiotic relationship between smaller and larger businesses. Larger businesses often view small businesses as a zero cost research and development arm. Most small businesses fail. The successful small business identifies a market opportunity. If the opportunity is one that can be scaled, the small business becomes a target for acquisition by a larger one. One of the most common strategies for high-tech entrepreneurs is to identify a niche that new technology makes available, create a small company that exploits the niche, then cash out by selling the small company to a larger one when the market is confirmed. This works well for the small business owners, but less well for the employees.

WHY VERY LARGE BUSINESSES MAY BE BAD FOR US

Network theory has shown us the value of self organizing systems of small components. This is true of both biological and technological systems. Systems created in this way can be self healing and require less intelligence at each node in the system. That is because failures tend to be local and are locally repaired. For example, internet routing is remarkably resilient in the face of many single failures because each routing node can detect and work around failures of its neighbors.

An economy based on fewer, larger players loses these advantages. Simply because of their size, large enterprises tend to be tightly coupled. The meltdown in 2008 showed this quite starkly. All of the large financial players were tightly coupled in risk and the risk was contagious. The mechanisms used to hedge the risks in fact increased the coupling and magnified the risk. Home loan risk reduced the capacity of the system for all loans and that infected all businesses that required credit.

Having fewer, and larger players is inherently riskier for the economy as a whole. This is not a problem of monopolies. It occurs when power in any segment of the economy is concentrated among very large interconnected players. A second problem is in the relationship between the company and its employees.

As a manager in a large business, my concern is business efficiency and profit. The natural tendency of the business to formalize processes and standardize jobs aids this because it makes labor more interchangeable. From an economic point of view the ideal labor market is one where both workers and employers are completely unbound. That is, if a better opportunity comes along for me as a worker, I should quit and take it. As an employer, if a more cost effective worker is available, I should fire a current worker and hire the new one.

As a high tech worker, this open marketplace has been the reality for me. Over the past fifteen years I have changed jobs about once every three years. About half of those changes were because I was offered a better position somewhere else. About half of the changes were because the business folded or was bought.

This open market of employees is very good for employers and would be good for employees if opportunities for work existed everywhere, and the workers were perpetually twenty five years old and in good health. However, that is not the real world. Jobs are often moved wholesale from one location to another leaving behind high unemployment that cannot be absorbed by the local market. For individuals, a perfectly open market is simply a way for employers to dump workers with temporary or permanent problems and to shift work location for even minor economic gains.

This type of labor fluidity is only reasonable if we as a society commit to those left behind. So far, I think the United States has done a very poor job of helping unemployed workers create and take advantage of new opportunities.

It is my experience that small enterprises, by their very nature, care more about individual employees. Small groups of people form loyalties. Any business faced with survival, will fire employees. In a small business, these decisions are personal. In a larger business, the decisions are impersonal. I have worked with a number of large employers where the decision to fire large numbers of employees and move the work to a different country is strictly a spreadsheet decision. If we are solely concerned with short term business efficiency, that is how the decision should be made. If we care about the communities in which we do business, if we care about the people with whom we work, this is not how the decisions should be made.

Software and Chaos

As with many of my blog posts, this is a little long. To make things slightly easier, here is a map

The Problem
My Experience
Business Expectations and Results
Software and the Elevator Speech
All is Not Lost

THE PROBLEM

Roughly seventy five percent of all software projects fail in the sense that they cost more than expected, deliver less than expected, or are delivered later than expected. This has been true for decades. Behind the scenes at these projects is untold stress and pain as development teams and management try to cope. This coping typically starts when it becomes apparent that the carefully constructed project plan does not describe what is happening and "success" is unlikely. I would like to believe that my projects perform better than average, but many of them go through this readjustment to reality. It starts with a meeting where the current situation is explained and options are explored. At some point, someone asks what went wrong, and all eyes turn toward me as system architect and technical manager.

I pride myself on being honest and not ducking blame. I go through a careful analysis of projections and actualities. I point out places where we made mistakes and where we may have lost time, but also explain that this is not where most of the problems lay. I point out the areas where there was unanticipated work or where an underlying system did not perform as expected.

All of this is true, but it is not the actual problem. The actual problems are structural, and most organizations with whom I work will never make the changes necessary to rationally develop software. They will not change because they cannot accept the truth about the type of software development I work on. No team, regardless how smart or how much time is spent in planning, can accurately predict the effort necessary to produce the software. The predictions of effort are based on what is known and are always optimistic.

MY EXPERIENCE

Let me preface this with some characteristics of the projects I work on. My generalization may not apply to other types of work. I work on new development. Typically it is putting a system in place to replace some existing process. Often there is no existing software in place. In the case where there is existing software, the organization plans to completely scrap that system. Along with new software, the business processes will be substantially changed. The projects are usually medium sized, say three to fifteen developers working for between six months and a year and a half.

The parent organization is usually quite large, but not a software development house. They usually have an IT department that maintains and expands existing software. They may have a reasonably large body of homegrown software, but developing new products is not their expertise. The existing IT staff is often expected to do much of the development work. It is common for the organization to have outsourced their data center operations.

These are generally business and process problems, not the development of shrink wrapped products. I have worked on products as well. The details of the problems change in product development, but the basic outline remains.

BUSINESS EXPECTATIONS AND RESULTS

Modern business is based on rational sounding self deception. As a rational business person faced with investing in the future I want to get answers to a set of questions.

What business are we in?
What and when are the opportunities?
How much will it cost to address the opportunity?
How much will we earn if we are successful?
What are the risks of action and inaction?

There are existing tools to address each of these questions. Upper management usually spends a lot of time and money to define the business. This includes vision statements as well as long and short term goals. Market research is used to answer the what, when, and size of opportunities as well as the cost of inaction. Technology research and initial product research are used to estimate the cost of action.

Given this information, I can compare projects to project the highest return on the investment. As a rational person I want to guard against incomplete or incorrect information, so I will want information at every step of the way to make sure assumptions were valid and to readjust the course. Recognizing that it is cheaper and easier to correct errors early in the process, I will want to make sure that we understand the problem and the solution as early as possible. This leads to a typical waterfall process.

Gather Requirements – understand the problem
Detailed Design – understand the solution
Implementation – fill in the details
Test – make sure users see a reliable product
Release – cha ching - here is where we get the return on investment
Support – if we have done things well, we can minimize support

After every step of this process there are checkpoints to assess progress and readjust based on our best information.

This is called a waterfall process because it moves in one direction, at the end of any stage, it is difficult to move back upstream.

The end result of this rational process is generally failure. Not just failure to deliver the software as envisaged, but failure at every step. Management changes in the middle of a project are common. With every change in management there is a shift in the way the business is understood. What was critical yesterday is marginal today. Market research is almost invariably wrong. You can confirm this yourself. Take a look at your marketing reports from two years ago. Project requirements change in the middle of the project. This is one of the most commonly listed critical problems in software post-mortems. The implementation takes longer than projected. As the pressure mounts because the project is late, developers hack functionality together to meet deadlines. Adequate testing is often one of the first casualties. As the bugs mount and the project slips, functionality is cut. Test exposes defects that require rework and sometimes the rework is not simple bug fixing. Sometimes the fix requires major architectural work. When release finally occurs, users find that things they consider essential have been chopped out, or the carefully crafted requirements do not correspond to the way users work.

The sad reality is that this process will fail for all but the most constrained projects. The large uncertainties inherent in the world combined with our limited ability to understand and communicate our understanding doom the entire methodology.

In the software community, almost no one believes that waterfall processes are either effective or desirable. The first descriptions of the process describe its failures as well. The failures have been spectacular. The US Federal Government has had numerous multi-billion dollar projects that have been complete losses (FAA - 2.6 billion IRS - 4 billion http://spectrum.ieee.org/sep05/1685/failt1). The money would have been better spent if it had been bundled into logs and burned to provide heat for the poor. Despite this, many large organizations mandate a waterfall approach. In large organizations around the world, untold amounts of money are spent to describe, mandate, and adjust a process that will never work.

SOFTWARE AND THE ELEVATOR SPEECH

"I'm a busy man, tell me what you want." This is the origin of the elevator speech. As an entrepreneur you find yourself in an elevator with the very person who can fund your project. You walk in and by the time the elevator gets to your floor you want to hook the prospective funder. To do so you answer the implicit questions, What am I going to get (and why should I care), when am I going to get it, and what will it cost? You have perhaps a minute to hook the person.

The basic problem is that the "what", "when", "how much" questions are essentially impossible to answer. The world is chaotic in the mathematical sense. That is, the circumstances that lead to wild success are often indistinguishable from the circumstances that lead to complete disaster.

Even in constrained fields like books and movies where the processes to develop the end product are well known and understood, no once can predict the success of the final product.

For new software, the prediction problems are magnified. The problems I address typically span multiple groups in an organization and involve manual processes that currently require human judgement. It is common that no single person in the organization understands the entire problem. Like the blind men and the elephant, each individual understands the the part they are touching, but no one sees the whole.

I always say that there is only one thing worse than fully specifying a computer system before code is written. It is getting the system that was specified. No matter how carefully we work, the requirements specified at the start of the project often prove untenable in real life.

At best, the process itself encourages wishful thinking. At worst, outright lying. Software is expensive. In a world where the low bid sets a general bar, there is every reason to underestimate the uncertainty and to assume that everything will work as predicted. As a result, I get to sit in meetings where someone asks the "what went wrong" and all eyes turn to me.


ALL IS NOT LOST

Although this picture is bleak, the software development community has developed techniques that actually work. Most of these fall under the rubric "agile development".

Agile development is the antithesis of the elevator speech. The answer to the what, when, how expensive questions are: I don't know, I don't know and I don't know. Agile development says that if you want to get effective systems in place, you get a bunch of smart people in a room and let them loose. The pact with the business is:

Choose the project as best you can.
Define a level of investment.
Involve customers from the first day to the last.
Address the most important customer problems first.
Produce working software early and often.
Insist on complete transparency.
If cost is too high or benefit too low, pull the plug.

That is, as a software provider I will provide actual value as quickly as possible and if I don't earn my keep, fire me. Because large systems generally have a roll-out to a large number of end users, it may be a long time before the system is actually deployed, but every three weeks or a month there is a new, actual working system that can be evaluated, tested and re-directed. When that system provides enough value to warrant its deployment, roll it out. After roll out, continue investment so long as the implemented improvements justify their own cost.

Rules Schmules

There is a joke among business consultants that when you go into an organization, figure out if they are a top down organization where decisions flow from above, or a bottoms up organization where individual units have a great deal of autonomy. If a business is top down you recommend it become more bottoms up so that individual units can understand and respond to the marketplace more quickly. If it is bottoms up you recommend that it become more top down to avoid chaotic redundancy and lack of focus.

There is a lot of truth in this. There is no ideal organization. Every organizational structure has problems. Over time, organizations tend to oscillate between structures based on the current set of problems. The same is true in other aspects of life. Should dictionaries be descriptive or prescriptive? How much freedom should parents allow their children?

This note is about the balance between individual freedom and enforced order. Based on what I see in the United States, I think the pendulum has swung too far toward order and it is time to move toward more freedom of action. I think the current situation is largely based on unreasonable fears combined with a desire to make the world safer for ourselves and our children. The end result is not a safer society, but a society with less freedom and more rigidity of thought.

I think the current situation is based on widespread misunderstanding of humans and their capabilities. As a species we are remarkable in our ability to exist in social groups and to flexibly react to each other and the environment. Despite the fact that we are capable of unspeakable cruelty, on the whole we are extremely kind and cooperative with each other. We are also protective, particularly of the most vulnerable among us. I do not deny the atrocities and cruelty, the scams, the sociopathic behavior of both individuals and groups. I merely say that these are anomalies. Bad behavior makes the news and becomes a focus of our thoughts precisely because it is uncommon in a stable society like the US.

As individuals we also misunderstand risk and emotionally overreact to perceived danger. Lately Hans Monderman has been in the news. For example, Wired Magazine. Monderman was a Dutch traffic engineer who died in January 2008. Monderman improved safety by removing traffic controls. That's right, he removed speed limit and other traffic signs and signals to make the roads safer. At a busy intersection there is nothing to segregate pedestrian, bicycle, auto and truck traffic. He has a stunt to prove his point to reporters. I have seen several separate reports on this, but from the wired article:

We drive on to another project Monderman designed, this one in the nearby village of Oosterwolde. What was once a conventional road junction with traffic lights has been turned into something resembling a public square that mixes cars, pedestrians, and cyclists. About 5,000 cars pass through the square each day, with no serious accidents since the redesign in 1999. "To my mind, there is one crucial test of a design such as this," Monderman says. "Here, I will show you." With that, Monderman tucks his hands behind his back and begins to walk into the square - backward - straight into traffic, without being able to see oncoming vehicles. A stream of motorists, bicyclists, and pedestrians ease around him, instinctively yielding to a man with the courage of his convictions.

From the International Tribune.

"Who has the right of way?" he asked rhetorically. "I don't care. People here have to find their own way, negotiate for themselves, use their own brains."

When asked, people who use these uncontrolled intersections say they are not safe, despite the statistics to the contrary. The fact that people do not feel as safe, as controlled, leads them to be more careful. Take off the explicit controls and people use their own judgment to behave in ways that increase safety rather than the perception of safety.

We have allowed our fears to distort our behavior and our society in ways that I find unacceptable. One example of this is the way we treat our children. As a seven or eight year old child I had large amounts of time that were unsupervised. My mother had a bell to call us in for dinner. As long as I was within earshot of the bell at that time, I could do what I wanted. When I got my first bicycle at about the age of seven, my world expanded and I was allowed to roam at will as long as I was home at designated times. Because our family was a large tribe I had more freedom than most, but I was not alone. At Halloween groups of children, without parental accompaniment, roamed miles from their homes to scour the best neighborhoods for candy. Neighborhood children organized their own games without supervision.

The United States has changed, but it has not become less safe. I do not believe that the rate of child kidnapping or abuse by strangers has increased. There is more traffic, but I do not believe that our neighborhoods present more dangers. What has changed is the level of fear. We have institutionalized the belief that the world is unsafe and children cannot be trusted to operate without constant supervision. As a result, elementary school children are not allowed to walk a quarter mile to school by themselves. Because parents believe that children can never be left alone, activities outside of school are supervised and, as a result, expensive. The constant supervision also diminishes the development of conflict resolution skills. I am not saying that adults should not look out for children. To the contrary, each of us has a responsibility to keep everyone around us safe, especially the young and defenseless. The world is unsafe. Children do not understand much of what is going on around them. But it is safe enough and we are here to help them.

The same attitudes and fears have helped turn children into incipient economic units. The current school debates focus on more time in school and more controlled school environments. This regimentation devalues the children and, in the long run, will hurt us economically. The genius of the American economy is our flexibility and inventiveness. As we become more regimented, as children's education become more standardized, I believe we lose that flexibility and inventiveness.

I recognize that I am neither typical nor a role model. In a sense that is my point. Few of us are typical. Only a few people think I am stupid or incompetent. I do well in a field that requires both constant education and inventiveness. I consider myself to be reasonably well educated. I also didn't do homework until I entered the university. Homework was assigned, I just never did it. My grades through high school were spotty, but performance on tests seemed to overrule the bad homework grades. As far as I know, no one ever considered holding me back. Based on college entrance tests, I was admitted to the university after my junior year in high school.

In junior high school instead doing my homework, I was reading a book a day. I read all the science fiction, "boy in the country", and biographies in the school library. In high school I was reading the daily newspaper, two to three weekly news magazines, and several monthly magazines including Scientific American and the Atlantic Monthly (it used to be a monthly). I also studied mathematics on my own. Had I spent more supervised time in school or had more structured after school activities, I would not have had time to get my real education. Looking back, I think I could have benefited from better schools. By better schools I mean those populated by more educated, and flexible teachers. I do not believe I would have benefited by more structure or more hours in school.

As a society we seem to have grown more and more rule bound. By rule bound I mean respecting and enforcing rules without looking at the sense of the situation. I have always followed Bob Dylan's dictum "To live outside the law you must be honest". A simple example from my masters swim group. We rent lanes at one of the pools in town. In addition to the pool life guards, we provide our own coach on the deck. The lifeguards and pool administration have recently gotten upset because some of our swimmers did not enter the pool feet first as the rules require. This is an experienced group of swimmers, many of them competitive. In my several years of swimming with the group, I have not seen a dangerous entry into the pool. The lifeguards either have no idea what is dangerous, have no idea why the rule exists, or are simply complaining about infractions for the sake of the rule itself.

As Monderman's practical experiments with traffic show us, posting and enforcing rules does not always make us safer. We are at our best when we take the time and effort to understand and respond to the particular circumstances in which we find ourselves.

Rules are a means to an end, not the end itself. The valid reasons behind the rules should be obeyed and enforced. If a rule is about safety and you are in a position to enforce, enforce safety not the rule. This is not a slide down a slippery slope toward anarchy. It is a slide down the slope toward respect and humanity. As with all things, you must do this intelligently and with awareness of your own limitations. Don't let your like or dislike of a person cause you to abuse your power and selectively enforce the rules.

If you are in a position to make rules, make sure that you are addressing a real problem, not the perception of a problem. Do not make rules in the mistaken belief that you can enforce conflict out of existence (the homeowners association fallacy). In work environments, try not to rigidly enforce standard practice at the expense of better solutions.

In our day to day lives each of us should remember the old saying, "People who like sausage and respect the law should never watch either of them being made". Rigid order is both impossible and overrated. Do the right thing. Help and protect those around you. This has nothing to do with rules, it has to do with humanity.

Following Politics Too Closely Makes You Stupid

I'm reading (and recommend) "The Black Swan" by Nassim Nicholas Taleb. As with his preceding book, "Fooled by Randomness" it deals in part with human limitations in predicting the future and, in particular, our inability to judge risk.

One of his themes is what others call the pull of narrative. As a species we like a nice story and we also wonder what happens next. It doesn't seem to matter how banal the story is (gossip or business plans) or how outlandish (soap operas or business plans). We are drawn in. Something is being shown or explained. We want to understand how the pieces fit together and to imagine what happens next. One of Taleb's points is that the narratives we see and invent for the world around us are not only false, they also lead us astray and cause us to miss what is actually occurring. A black swan is something that we do not see coming, but seems obvious once it has occurred and we create an appropriate narrative. The current financial meltdowns are a case in point. We will be paying for the debacle for generations. While the seeds were being sown many smart, and sometimes well intentioned, people gave us a story about how the financial instruments at the base of the crisis reduced risk. Now that credit markets are freezing, we have new stories that tell how greedy people acted blindly and exposed everyone to grave risks.

In political discourse the same human foibles apply and are even magnified. Every side is selling a story. The vehicle for the story is mostly TV and radio through ads and news stories. Each installment of the story must be communicated in about thirty seconds. Very occasionally a full minute can be used. Because the audience only observes intermittently, each installment must be repeated many times to make sure everyone keeps up. This is one of the origins of "talking points". Consistent messages are easier to remember. Constant repetition makes it more likely the message will be heard, and more likely the ideas will be accepted at face value (everyone is saying it). This leads to a slow accretion of fragments that fit into a consistent theme or story that the proponent hopes is persuasive.

In an odd way, the fragmentary delivery of political messages makes them more effective. Like the next installment of a soap opera, new talking points fill in a missing piece and lead us to wonder what is next. The abbreviated format force reduction of the message to slogans and codewords that mean more than the actual statement. These not only reinforce the story, they act as cheers to emotionally charge supporters who have accepted the truth of the underlying message. All sides engage in this but, in the United States, conservatives have been much more effective at this style of communication. They just have better slogans. There are keywords like "liberal" and phrases like "tax and spend". They are even better at adjectives. One recent ad talked about an opponent supporting "massive government", which has succeeded the epithet "big government". Each of these keys on a part of a larger narrative. Political narratives have existed for as long as the human race, but the modern form is based on commercial advertising and uses all the tools and techniques of that craft including testing and focus groups.

Political communication has been simplified to mere advertising, but the problems we face are not simple. There is essentially nothing that can be learned about a complex situation in thirty seconds. Regardless of the party or position, every ad you listen too is misleading. The same is true of the news stories. Simplified communication can communicate at most first order forces. Complex situations are often the result of many different forces, some of them quite removed from the obvious.

I take one ad, http://www.youtube.com/watch?v=9pRHliGZl-o, as an example, not because there is anything special about it, but because there isn't anything special. It typifies the level of discourse. It happens to come out of the Republican camp. The ad shows gas prices going up, "the US has more oil resources than Saudi Arabia" but "85% of it is off limits" and the opposing candidate "voted against development of American energy".

The first level of skepticism should be the facts presented. Anyone who works with numbers about the real world knows how difficult it is to get valid numbers and that the numbers are usually based on simplifying assumptions. The statement that the US has more oil resources than Saudi Arabia presumes that we have a good idea about what those reserves are. For both the US and Saudi Arabia, we do not know about the reserves. Some of this is genuine ignorance (we haven't looked), some of it is global politics (the Saudi's have every reason to lie). It also depends on what you mean by "reserve". Does oil shale in Colorado count the same way as a typical Saudi oil field?

Moving past the "facts" we get to the implied world view. Implied is the notion that gasoline and oil are synonymous with energy and that lower gas prices are good public policy. Beyond that is the notion that opening the 85% of reserves that are off limits will reduce the price of Gasoline. None of these are particularly good assumptions for reasons that would take more than thirty seconds to explain.

After hearing a typical political ad or even candidate speech, you are actually more ignorant than you were before. You have been deliberately deceived and your emotions played upon to accept a world view that is questionable and policies that, if you investigated further, you might very well disagree with.

What to do then? Try to get real information. On the whole, books are much more valuable than any kind of daily/weekly/monthly news. But recognize that books can be commissioned as propaganda as easily as many other things. Trust authors who examine the situation dispassionately and who genuinely research all sides. If you are interested in oil, do not start with anything political. Instead, read science articles on exploration and estimation. On most subjects it is best to get a good grounding in the state of the information and its limitations, then move to issues of policy. At least you will recognize the blatant distortions.

Policy has to do with where we are and what we want for ourselves and the world. In terms of policy, it is not enough to have an intent, we must also examine the risks, the likelihood of failure, and strategies that have worked and failed in the past. Because each of us is in a different situation, wants different things, and weighs choices differently, your judgements on policy and tactics will differ from your neighbors.

If you do all this, you will be as well informed as you can. You will also be wrong on basic issues. The way you think the world works will be incorrect. There is no avoiding this. The world is a complicated place. We just do the best we can.

Machine Politics

Political machines are self perpetuating systems generally based on personal loyalty and patronage. In the United States people generally think of political machines in terms of large cities. For example: the Richard Daley machine in Chicago, Tammany Hall in New York. But a political machine can exist at any level of government. At the state level we have had Huey Long in Louisiana and Thomas Catron in New Mexico. At the national level, Tom Delay ran a comprehensive and relatively effective national political machine.

The nature of a machine depends on the power structure of the society in which it is embedded. In societies where political power is influenced by election, the basic mechanics of the machine center on getting votes. In hierarchical societies (feudal, tribal...) machines center on fealty often coerced by violence.

Political machines are not an aberration. They are based on an essential part of being human. As a social animal, we live in groups. We band together for mutual support and protection. We form friendships and animosities. We try to help our friends and expect them to do the same. When someone thwarts our plans, we will undermine him in order to accomplish our aims. Political machines are not unalloyed evil. In order to maintain a power base, a machine may be extremely response to their supporters. In many cases this improves government services. In supporter's areas, the trash is picked up and the roads are good.

Political machines are about power. Its members may espouse, and believe in, a particular ideology, but the heart of a machine is not ideological, it is personal. A machine must have a way to reward supporters and punish enemies. In fact, one of the hallmarks of the machine mentality is an enemies list.

For political machines the source of power is the machinery of government. Government taxation provides a base of money. This money provides direct means to reward in the form of jobs and contracts. Withholding jobs and money provides a means of punishment. Government regulation provides another carrot and another stick. Businesses can be shut down through regulation or over-vigorous enforcement. Conversely, the skids can be greased by removing regulations or declining to enforce them. Taxation itself is a great lever. Taxes can be reduced on friends and increased on enemies. All of this requires that the machine controls the levers of government.

City machines are perhaps the easiest to understand and examine. The Daley organization controlled on the order of 30,000 patronage jobs. There was a strict hierarchy based on electoral politics. The city was divided into wards and precincts. Political appointees were responsible for estimating and delivering the votes in their area. At every level of the machine, there were rewards and punishments. The plus side is government responsiveness. An influential person gets his requests answered. A important precinct might get better city services. On the other hand, precincts where the machine lost might find themselves without city services and city jobs.

Electoral politics are used to ensure control over government, but machines are personal. By cooperating with the machine, you should personally benefit. By defying, you are personally punished. Reward can be through direct payment, perhaps by being given a job that may or may not require work. More profitable is the ability to steer government money. A lucrative government contract can act as payment. If you are the person deciding where the contract should go, that power is worth something. Typical forms of personal payment can be bribes, kickbacks or jobs for friends. Political machines thrive on graft. When personal profit takes precedence over public good, everyone suffers.

In national politics the same basic mechanics exist, but because of the scale the details change. On a national level, campaigns are won with money. Loyalty can be bought with campaign money.

Direct contributions to particular candidates and parties are restricted. Campaigns that address large numbers of individual voters can raise tremendous amounts of money. However, each individual contributor is not that important to the campaign. If you want to gain attention, you can become a "bundler". A bundler gathers together the campaign money from a set of individuals. The bundler may not put up all the money himself but becomes personally powerful by being able to control a large donation. There have, or course, been scandals where bundlers paid the money to the individuals who turned around and contributed it. The bigger the bundle, the more powerful the bundler becomes.

At the national level, lobbyists for industries and particular points of view are very important. Lobbyists must be able to argue persuasively, but another important function is to provide monetary support for those who support them. This support provides access to the political process. Tom Delay took this to a new level. First, he steered contributions to his political action organizations. That made candidates beholden directly to him. He also inserted himself into the lobbying firms themselves. Access required political approval in hiring. This made lobbying itself a patronage job.

At the national level, the personal loyalty and reward structure of machines remains the same, but the amount of money available is much larger.

Rules and Reality

For the past few years I have been consulting with large businesses on software projects. As a result, I see a lot of different companies and cultures. Recently I have been working with a very profitable media company with a more extreme schizophrenia than most companies.

Part of the business involves coordinated release of merchandise from many different manufacturers. The list of manufacturers is constantly shifting. The product standards are high and manufacturers must work off the latest artwork. The artwork is constantly being refined and the time-lines are short and getting shorter. This creates a business need for rapid processes. In addition to a shifting cast of manufacturers, the company relies on contractors to do much of its essential work. Many contractors have worked in the same capacity for years, but there is also a churn of contractors and contractor employees.

Another part of the business is concerned with protection of intellectual property. Protection often requires tight control on access. That is, no one should be allowed access to company property until all the correct agreements have been drafted, reviewed, revised, and finally approved. This desire for protection has bred a culture of slow deliberation. Every formal decision involves slow and ponderous study and documentation (with periodic formal reviews).

These two strains, the need for rapid response and the slow processes that protect, exist in many organizations. In the case of this media company the culture has fractured. The protective processes are so slow and insular that they must be subverted. As an example, access to buildings is controlled by key cards. Getting permission to issue a key card is difficult and slow. The need for access may evaporate before a card can be approved. As a matter of common practice, those without key cards simply follow a carded employee into the building. Carded employees understand the situation and do not question when someone enters behind them. One morning I followed someone to the door only to find that neither of us had cards, so we both waited by the door for a carded employee. That employee let us in without question or hesitation.

For the software system I am examining, the development process involves asking people who use the system for their requirements. A fair number of those requirements are essentially "We have a way to subvert company policy because we cannot possibly follow it, but it is difficult and cumbersome. Could you make the subversion easier?"

For both individuals and organizations there is often a disconnect between our rules and what we actually do. Because people and the world are more complicated than our notions of them, some disconnect is inevitable. The healthy side is when we recognize the need to be flexible to accommodate novel situations. The unhealthy side is hypocrisy and self deception. If you cannot follow the rules the fault may be yours, but do not discount the possibility that the rules or their implementation is broken. Besides, strict adherence to any set of rules is not much of a virtue.

It is Your Civic Duty - Lie to Political Pollsters

It is your civic duty to lie to political pollsters to make polls as unreliable as possible. It is not enough to refuse to answer the questions, you have to lie.

The rest of this article deals with why, but I want to take a moment on the practical - how to lie. There are several amusing techniques for this. Here are two. The most reliable one is to pretend you are someone else, but completely stereotype their views. Choose your crazy Aunt Myrtle or your grandmother who votes for the "most handsome" candidate. The important thing is that their views do not agree with yours. This technique has the advantage of a coherent, if mildly insane, point of view. That makes it hard for pollsters to simply discard your answers as nonsensical. I suspect that most polling techniques do not allow for throwing out responses. That allows for more extreme techniques. I choose random numbers, the digits of pi for example, and use those to determine my answers. If the next digit is 3 and there are three choices, I answer with the third one. If there are only two answers I choose the first because 3 and 1 are both odd numbers. The other day I owned a company with over 100 employees, but made less than 20 thousand a year and had no health insurance.

At first you may find it hard to give an answer that you absolutely and positively disagree with, but I assure you, you get over it and it becomes fun after a while.


Here is what I want from political candidates and the media that covers elections. I want to know how the candidates think about the world and our place in it, and how they are likely to vote on issues I care about. Political polling makes it less likely that I will get what I want from the media, from political parties, and from the candidates themselves. Each of us should do what we can to remove polling from the political landscape. Lying is the easiest and most effective way I can think of to make the polls disappear.


The Media:

Media coverage of elections is bad and not getting better. From "A First Look at Coverage of the 2008 Presidential Campaign".


Just 12% of stories examined were presented in a way that explained how citizens might be affected by the election, while nearly nine-out-of-ten stories (86%) focused on matters that largely impacted only the parties and the candidates. Those numbers, incidentally, match almost exactly the campaign-centric orientation of coverage found on the eve of the primaries eight years ago.

Here is what we get from the media

• Horserace coverage. Candidate Smith is 3% up in the polls, candidate Jones is gaining in a final stretch surge in the polls. Candidate McMurry wins by a nose. Candidate Rostnikov outdid candidate McMurry in fund-raising in the month of June.



• Political strategy coverage. Polling shows 27% of the electorate is "very concerned" about issue "Horses that have left the barn". The Animal party intends to take advantage of this with a direct mail campaign to people who do not own horses emphasizing their position that barn doors should be self closing.



• Discussion of the polls. After a poll proves unreliable, the media spend a lot of time talking about why. This is particularly corrosive because it uses up time that could be used discussing something of importance. It is also completely vacuous because if the pollsters knew why they were wrong they would have corrected for it before the poll was published. My goal is to stop this discussion by making ALL political polls unreliable.



• Reporting of partisan polls. Many polls are commissioned by people with an axe to grind. Questions are created that invite a particular answer, then it is reported that "By a vast majority, the electorate oppose beating dead horses."

Nonsensical explanations. Poll results are often analyzed as though the polls were answered by a single person and that person had a coherent set of beliefs. This invented person if often called "the electorate". For example, "The electorate seem to feels that taxes are too high." In fact, poll results do not reflect anyone's views. They are an aggregate measure.

Parties and Special Interest Groups

Here is what we get from the parties and special interest groups:

• Attack ads on targeted, hot-button topics. "Candidate Rostnikov voted to allow horses to escape, often to be killed". These ads invariably inflame passion, distort the candidates views, and cheapen the debate. Here is a simple rule. There is nothing meaningful that can be learned about a candidate in 30 seconds.



• Feel good ads. These are essentially selling the candidate with the same techniques as Coca Cola. There are some general techniques: attractive smiling people doing good things, flags waving, children and a better future. Polling and focus groups allow the political machines to tune these ads like a drug formula.



• Pandering. Here is a definition of pandering: "To cater to the lower tastes and desires of others or exploit their weaknesses." Pandering is often subtle, but sometimes not. If someone offers you money "A $1000 tax rebate" or "repeal/suspend a tax", they are pandering. "Lower tastes and desires" are easy to figure out. In politics, money usually does the trick. But, effective pandering requires some excuse so you feel justified in receiving the the largess. That is where polling comes in.



• Very specific targeting of particular people. Based on your answers to political polls, you will categorized. This allows you to be targeted very specifically. If a candidate Hernandez knows that it is very important to you that koala bears remain a symbol of affection and comfort, you may be targeted with a koala bear ad. It will paint the opposing candidate, Walenski, with some plausible evidence, as someone who just doesn't care about koalas. Hernandez is counting on your affection for koalas to motivate you to support him and oppose Walenski. A closer examimination of the candidates voting records may show that Hernandez is likely to vote for a measure that will cost you your job, but he does like those koalas and Walenski seems to hate them.

A few interesting links to follow if you are interested in the topic:

• Polling Politics, Media, and Election Campaigns


• Poll-obsessed media focus on strategy over substance


• Horse race reporting - 92 election

Nanny nanny boo boo

Some time ago when I both had and worked with young children I noticed an interesting thing. Kids of particular ages have their own expressions that are not used by either adults or children of other ages. My favorite example is from kids under about six years old who chant "Nanny nanny boo boo." This is a shortening of the full expression "Nanny nanny boo boo. Stick your head in doo doo." This taunt is typically used in chasing games and means "You can't catch me you ...". I have heard this chant in multiple locations thousands of miles apart, and I have heard it over a period of years. As far as I can tell it is: persistent, widespread, and completely restricted to young children. I don't think I have ever heard an adult use the full expression.

Other age groups seem to have similar cultural artifacts. Junior high kids who would never dream of being so immature as to say "Nanny nanny boo boo" may tell you the joke about the pygmy tribe called the fakawi. In tall grasslands, people of this tribe can be found jumping up and yelling "We're the fakawi". Say it with a Brooklyn accent - its an aural joke. I had an adult neighbor repeat this one to me the other day, but I believe it is largely confined to junior high aged kids.

This is anecdotal and I have no real conclusions about the phenomena, I just find it fascinating that culture can be transmitted horizontally through an age group without the mediation of adults or even family.

Mangled Alphabet

There are times when it is important to have the correct spelling of a spoken word. Because many English letters are difficult to distinguish when they are spoken (c, g, z), it is common to distinguish them by using a word. There is a standard word alphabet for this: Alpha, Bravo, Charlie, Delta, Echo...

But suppose your desire is not to be understood, but to be obscure. For example, the Internal Revenue Service calls to track down your offshore bank account. You don't want to be convicted of perjury, but you don't really want to be helpful either.

Here is a mangled alphabet for your pleasure. Just imagine being on the phone and saying "My name is Jim, J as in jalapeno, i as in ingenue, m as in mnemonic. What do you mean repeat myself, don't you speak English?".

The alphabet is a combination of spellings that do not match sounds, sounds that say a letter that is not the one of interest, accents on a misleading syllable, and obscure words that distract from the task at hand. With some letters I found it hard to be truly misleading. Suggestions are welcome.

A - Aoife (EE-feh) Irish female name
B - byssus (BIS-uhs)
C - cent (sent)
D - Django (JANG-oh) Django Reinhardt jazz great
E - elephant (EL-uh-fuhnt)
F - feign (feyn)
G - gnostic (NOS-tik)
H - honor (ON-er)
I - ingenue (AN-zhuh-noo)
J - jalapeno (hah-luh-PEYN-yoh)
K - knife (nahyf)
L - llano (YAH-naw)
M - mnemonic (ni-MON-ik)
N - nigeria (nahy-JEER-ee-uh)
O - opossum (POS-uhm)
P - phrenology (fri-NOL-uh-jee)
Q - quran (koo-RAHN)
R - Rhone (rohn)
S - sent
T - tsar (zahr)
U - umbilical (uhm-BIL-i-kuhl)
V - verisimilitude (ver-uh-si-MIL-i-tood)
W - whole (hohl)
X - Xerox (ZEE-rox)
Y - Ysolde (ee-ZAWL-duh)
Z - zakat (zuh-KHAT)

Running the dog

In looking through old writing, I ran across this letter I wrote to the city after receiving the second in a series of tickets. The topic of the letter is of less than earth shaking importance, but there are serious issues on the function of government and freedom versus order.

After writing the letter I did some research. I discovered that the level of injury and death caused by dogs is almost exactly the same as that resulting from baseball, and the most common victim in both cases is children. It also turns out that dogs are most likely to cause trouble in or near their homes and least likely to cause trouble in public places. In addition to normal animal control my city (through its Natural Resources department) spends on the order of a quarter million dollars a year enforcing leash laws. The leash law applies to all animals including cats, but there is a specific exemption for birds under voice control.

In the end, the city had its way. The fines double on every ticket and I could no longer shoulder the burden. I made some attempts to change the law, but could find no leverage point. I no longer spend much time at the park. Other dog owners have also been driven away. When I do go by the park the dogs are gone, and so are many of the people. I was genuinely surprised at how important the simple pleasure of running the dog at "my" park was to me, and how betrayed I felt at having local government stop me. When my dog running disappeared, so did my pride and sense of responsibility toward the city.

The Letter

This note is about a bad law that has been badly applied. On September 29, 2004 I received a ticket for having my dog off leash in Spring Creek Park (animal at large). This is my second ticket. Each ticket was issued by park rangers I had never seen before and have not seen since. Each of them said they had no discretion in issuing the ticket. The second ticket was issued in what looked like a concerted effort by the city to crack down on unleashed dogs in the park. There were at least two rangers, one of them driving a pickup truck through the park to make sure lawbreakers did not escape.

Over the past nine and a half years I have come to Spring Creek Park at least twice a day virtually every day: rain, snow or shine. I spend between ten minutes and half an hour at each visit. That makes about 7000 visits to the park or about 1400 hours. I think it is accurate to say that over the past ten years I have spent more time in Spring Creek Park than anyone else. Until receiving this latest ticket, it was my intention to create and publish a photo essay book showing daily views of the park over an entire year.

The purpose of those 7000 visits to the park has been to run my dog. She is an impeccably trained border collie. If you have ever been around a border collie, you know that they are active dogs that need to run. I have trained Josie to run laps around the ball fields. She is getting old now and has slowed down. In her prime she would routinely run 20 or so laps around the ball-field as a warm up. After the laps she generally has the energy to do other training exercises. Her record is 50 laps, or about eight miles, at a full run. I have taken her to the dog parks, but there is nothing there that can provide the flat out running that she both enjoys and needs. She is completely disinterested in other dogs and humans.

I stopped bringing a leash to the park years ago. I can honestly say that, in all our visits, not once has my dog been a danger to any person in the park. By training and temperament, Josie rarely acknowledges that there are people in the park. As a border collie, she is all business and her business is running in the patterns that I tell her to.

Josie is a fixture at the park. She is known by the workers and the neighbors. She is admired for her ability and her training. I have received countless compliments on this dog. The number of complaints can be counted on the fingers of one hand. On the rare occasion someone looks nervous or says something, I take Josie out of the park. Periodically, either in Spring Creek Park or in other parks, an animal control person will stop by and tell me about the law. Without exception, those folks have complimented me on Josie's obedience and asked me to take her out of the park. To make life easier for all of us, on those occasions I leave the area. One animal control person said they were responding to complaints about dogs in that particular park. He suggested other places where he would not be patrolling that day.

The animal control workers and the visitors to the park have understood something the city ordinance and the park rangers do not. My dog running in the park is an innocuous and safe activity that improves my life, the dog's life, and is largely a joy to other park visitors.

While I am at the park, I pick up more dog waste than Josie leaves. I collect and throw litter away. Every couple of years I find a stray that has wandered away from home. I track down the owners and return the dog. One time I found and returned a wallet that had been stolen from a truck in the neighborhood. The owner didn’t even know it was missing. When teens rolled one of the trash barrels out onto the frozen lake, strewing garbage along the way, I recruited a couple of fire-fighters from the next door station. We retrieved the trash barrel and picked up the garbage.

Of course, the fact that my dog is well behaved does not make the law bad nor does it excuse me as a lawbreaker. What makes the law bad is that it worsens rather than betters the community. I have spent a lot of time researching human behavior, particularly in cities. We know a lot about what makes good, vibrant communities. One of the most important factors is that neighbors know each other, are aware of their surroundings, and take responsibility for the community. Parks can be marvelous places, but many towns have the problem that their parks are dangerous, particularly in off-hours. One of the major differences between a safe park and a dangerous one is community use. Crime and trouble avoid public view. When good people are near and watching, trouble moves away.

Over the years it has been interesting to see how people use the park. Almost no one simply walks through the park enjoying it. People come because they have a reason. They do what they planned to do, then they leave. Parents who bring their kids to the park stay very close to the playground. The more adventurous will walk over to the ball field to play for a couple of minutes. Ball players stay on or next to the playing field. Walkers and runners tend to either walk in a straight line through the park or they travel the perimeter. Sunbathers invariably pick a spot away from these main uses. Except for dog runners, most of the park is simply unused other than as an attractive backdrop to the other activities. People with dogs off lead almost always stay away from other park users and run their dogs in the open, unused spaces.

I seem to be pretty typical of the people who take their dogs to Spring Creek Park. These are neighborhood people who care about the neighborhood and take care of the park. I know this because I see them every day. I see them talk with each other and pick up after themselves and others. Some of the dogs are on leash, most of them are not. The difference seems to be in the temperament of the owner and the dog. If someone has a dog that will not obey or is flighty, they will only take the dog off lead once or twice. The adventure of screaming at your dog while chasing it through the public park or nearby neighborhood is a powerful deterrent to letting untrained animals off lead. People with leashed dogs behave more like walkers. They move through the park rather than spend time in it.

I am sure the Parks Department receives complaints about dogs running wild through the park. Many people feel threatened by dogs on the loose and many people simply do not like the idea that others violate the law. If the city banned blue flowers, I am sure it would receive many calls about the law violators with blue flowers in their yards.

In terms of actual danger or harm, I cannot speak with authority because I do not have the figures. I can speak from my experience and observations. Fossil Creek Park just opened. It is a beautiful park and heartening in terms of its design. For some time parks have been dumbed down to make sure that no visitor can be injured. Fossil Creek Park seems to have very thoughtfully constructed play areas, but ones where the users must take responsibility for their actions. The skate park at Fossil Creek is marvelous and seems to be heavily used. Even without seeing the figures, I can guarantee that there are more injuries in that skate park in a single week than dogs have caused in Spring Creek Park in the past ten years. Of course there is a difference between me falling off my own skateboard and being accosted by a strange dog. One I control, the other I do not. Whenever a skateboarder lets his board fly and it hits someone else or a dog owner has a dog that is out of control, they should be held accountable. But control is the problem, not the activity.

The Parks Department is in an awkward position. There is a law on the books. Until the law is changed, they must either ticket the offenders or turn a blind eye. I would prefer that the law is changed, but this is an area where emotion runs high. Not many politicians will stick a neck out for a fight that will gain them nothing but animosity, regardless of the facts. We, unfortunately, live in a time where many areas of civil behavior have been written into law. Most of them are well intended, but many of them are violated routinely. If a policeman decided to ticket every lawbreaker he saw, he could never get to a violent crime scene. He would spend all his time ticketing speeders, jaywalkers, and litterers along the way. When laws are numerous they are, and always will be, selectively enforced. I suggest that the Park Rangers would be better employed by spending more time getting to know the neighbors who do the real policing in the parks and less time driving them away by ticketing them for harmless activities.

Bad Smells and Complexity

I work with software. A few years ago a couple of influential folks, Kent Beck and Martin Fowler, started using the term "bad smell" to describe a situation where something doesn't seem quite right. When you pour a glass of milk, you may catch an odor. Everything may be fine, but you probably don't want to gulp it down until you have investigated a little further. The notion of bad smells can be used in any complex human endeavor. Experienced people quickly recognize when something is off, even if they cannot express exactly what it is.

Sometimes it is hard to figure out whether something is just complex or if the investigators are simply lost. Migraine headaches are a good example. A few years ago I started getting occasional migraine auras though, thankfully, not the headaches. When I researched the topic I found that there were several proposed causes and any number of proposed treatments or ameliorations. Sufferers look for triggers of the headaches. Different people have proposed different triggers. Trigger lists include: changes in air pressure, bright light, smells, foods including meat, wine, chocolate, beans, cheese, pickled or marinated food, nuts, herring, figs, raisins, citrus, tea, coffee, chicken livers. Of course the old standby "stress" is included in the trigger list.

Lists like the migraine triggers are a bad smell. If you stop any person on the street, you will undoubtedly find they have been exposed to one or more of these triggers in the past 24 hours. It may be that triggers are quite individual and that different people have a small set of different triggers. I find it more likely that people in pain are grasping at straws to find something that might enable them to avoid the pain in the future.

Sometimes complexity is not a bad smell, but just complexity. For a long time I have been interested in questions that different people answer differently. I live in a place without a lot of crime. A number of years ago I asked folks whether they locked their doors when they were in their house. I thought the answer might give an indication of risk tolerance. Sure enough, some people answered always, some answered never. One person said he only locked the doors when he was home because he cared about his own safety, not that of his possessions. I don't know if this was a truthful answer, but it was an interesting point of view.

It is an open question whether we can use some relatively small set of traits to characterize our behavior. Even identifying traits that influence behavior is problematic.

For about a century western society has been very interested in measuring "intelligence" (http://en.wikipedia.org/wiki/Intelligence_testing). A Welsh friend of mine used to say that people in the United States have a peculiar fascination with reducing complex phenomena to a single number. Certainly intelligence testing fits this pattern. It is not clear what "intelligence" is and how many aspects of it there are (http://en.wikipedia.org/wiki/Theory_of_multiple_intelligences), but that doesn't stop us from measuring and assigning a single number.

There have also been attempts to measure both moral beliefs (Moral Judgment Test) and religiosity (The internal structure of the Post-Critical Belief scale). As it turns out, these may not be correlated (Religiosity, moral attitudes and moral competence).

Psychologists have tried a number of personality characterizations. There are classifications of pathologies and personality disorders (http://en.wikipedia.org/wiki/Personality_disorder). Some of this is clearly cultural. For example, homosexuality was listed for a time as a personality disorder. There are tests, Minnesota Multiphasic Personality Inventory for example, to help diagnose these disorders.

There are also more general classifications. The Myers Briggs Type Indicator test is based on Jung's analysis of perception (sensation and intuition) and judgment (thinking and feeling). Based on this, four dichotomies were established: Extroversion/Introversion, Sensing/iNtuition, Thinking/Feeling, Judging/Perceiving. I always joke that Myers Briggs test is much better than astrology because astrology has only 12 categories to explain us, but Myers Briggs has 16.

The "Big Five" model comes from linguistic analysis. The thought is that essentials of personality are expressed in language so an analysis of language can lead to a understanding of personality. The big five are: openness, conscientiousness, extroversion, agreeableness and neuroticism.

In looking at all of this I don't detect that much bad science or even many bad smells. I come to the conclusion that people are just complicated. The theories project this complexity onto a number of different spaces, but none of them fully encompass us. We are just too complicated for simple analysis.

Heart Rate Monitors and Training

This is something I wrote some time ago, originally to help my daughter who got a heart rate monitor. Periodically I change it slightly and send to someone. I post it here because it may be of more general interest.
--------
How you train depends on what you want to accomplish. Genetically people tend toward a predominance of one or the other of two muscle fibers: Fast twitch or slow twitch muscles. The ratio of these is not fixed. Slow twitch fibers can be recruited to be fast twitch, but there is a genetic diffference that tends people toward one or the other. Those who have lots of fast twitch fibers tend to be sprinters. Those with lots of slow twitch tend to be better at endurance. Related to the sprinter/endurance dichotomy are several metabolic pathways for producing energy. There are aerobic (with oxygen) and anaerobic pathways. The anaerobic pathways tend to be good for very short duration activities, a few minutes or less. I have never been tested, but I am the anti-sprinter, so I suspect I tend toward the slow twitch. At any rate, my goals tend to be fitness, endurance and speed (strength) over periods of about an hour. I choose an hour because that tends to be the amount of time I can manage for exercise on any given day. This puts me in the aerobic, endurance camp. I have no idea how sprinters (or their cousins, weight lifters) train.
Several things limit athletic performance. One of them is oxygen. A second is Lactate buildup. Another is muscle strength. In a sense, muscle strength is the simplest to train for. Muscles get stronger as they adapt to stress. To make your muscles stronger you stress them to the point of minor damage, then allow them to recover. Both stress and recovery are important. No stress, no need for adaptation. No rest, no chance for the damage to be repaired and the muscles to get stronger. In addition to simple strength there is also technique. Every activity can be done more or less efficiently. The more efficient you are, the less energy is required for a given result. This is good.
The oxygen/lactate systems are more complicated because there is so much going on. Every person has a maximum rate at which oxygen can be absorbed through the lungs into the bloodstream, pumped by the heart to the body and utilized by the muscles and other organs. This can be measured as your VO2 max. If you exceed your VO2 max value, by definition you can’t provide enough oxygen to support the work you are doing and something either has to stop or go anaerobic. The point at which you start going anaerobic is called your anaerobic threshold. The best thing you can do to improve your VO2 max is to choose your parents well. However, even the genetically ungifted can improve their VO2 max through training.
When you produce energy anaerobically, one of the by products is lactic acid. If the lactic acid accumulates in the muscle it changes the PH and the muscle loses the ability to contract. Of course none of this is black and white. Even when you are exercising mainly aerobically, some lactic acid is produced. You should be able to increase performance by improving your capacity to get oxygen to your muscles, improving your ability to remove lactic acid from your muscles when it occurs, and by making your muscles more tolerant of lactic acid when it is present. Lactic acid itself is burned oxidatively by mitochondria to produce energy. That is one of the ways that lactic acid is removed. See http://home.hia.no/~stephens/lacthres.htm.. Since this post was originally written I have spoken to some exercise physiologists. What happens when lose our ability to continue at a given level of exertion is not clear. It is NOT a simple matter of lactic acid and no particular metabolite has been identified. There may even be a mental control designed to protect the organism.
Two other factors in oxygen transport are your maximum heart rate and your resting heart rate. There is a speed past which your heart will not beat. The maximum differs from person to person and is also dependent on age. You lose roughly one beat per minute per year as you get older. Your resting heart rate serves as an indicator of heart efficiency. Take your resting heart rate while lying down, preferably in the morning before getting up or right before bed. At rest, your body still demands a certain amount of oxygen. The fewer heart beats needed to satisfy this requirement, the more efficient the heart is. As you get more fit, your heart simply delivers more blood per beat.
What does this have to do with training with a heart rate monitor? Your heart rate roughly corresponds with the amount of oxygen being demanded by your body. This, in turn correlates with the energy your body is putting out. The harder you work, the faster your heart beats. In an endurance sense your heart rate is limited on one end by your resting heart rate and on the other end by your anaerobic threshold. The range is expanded by making your heart pump more per beat (lowering the resting heart rate) and raising your anaerobic threshold (increasing oxygen delivery and lactate removal).
As long as you aren’t "overtraining", a heart rate monitor allows you to judge just how hard you are working. Surprisingly this can be more useful on easy days than on hard days. Using the monitor you can get a sense of your real anaerobic threshold (it hurts and you can’t sustain for long). You can gauge effort by finding how close you are to either your anaerobic threshold or to your maximum heart rate. This is usually expressed as a percentage (90% of maximum heart rate). Be aware that these measures are different because your anaerobic threshold is below your maximum heart rate. However, your anaerobic threshold is something you can figure out. Your maximum heart rate is difficult to determine. I knew my predicted one was wrong when I looked down at the monitor one day and saw a number above my "maximum".
Conventional wisdom is that training should be a mix of stress and recovery. Recovery does not mean inactivity. It means activity at a level below that which stresses you. Most of your time should be spent in recovery. Exercise at a recovery level can be performed for a longer duration than exercise at stress levels. This recovery not only allows you rebuild your stressed muscle, it increases endurance and gives you a chance to work on efficiency. Although technique is critically important, efficiency is not synonymous with technique. I believe some of it is a physiologic adaptation.
Stress levels for your heart lung system are right around the anaerobic threshold. To work on raising the anaerobic threshold, lactate recovery, and increasing VO2 max, intervals are the prescribed medicine. To stress the aerobic system rather than just using your sprint abilities, the intervals should be at least a couple of minutes long. These are very painful and it is recommended that you not do this more than about once a week.
A step below intervals is a hard workout. Ignoring my own advice about recovery, these are typically what I do on the bike. For a hard workout I basically work for an hour at just below my anaerobic threshold. For me on the bike this is around 90% of my predicted maximum heart rate. My excuse is that I am not an athlete and don’t have enough time to recover as an athlete would (easy days). I recover through inactivity. There are many days where I simply don’t get any exercise and it is uncommon for me to have the same exercise activity two days in a row.
For athletes and sensible people who actually schedule easy recovery workouts, the heart rate monitor is most useful. The trick is to set a level before you start. It is very easy to have a recovery workout turn into yet another hard workout. The heart rate monitor gives you a number. If that number goes too high, you ease off regardless of how you feel. I find this to be very difficult. Several sources give target recovery workout rates of about 70% of your max heart rate. I find it hard to give advice that I don’t follow, but conventional wisdom would probably have you do as many recovery workouts as you do hard workouts and intervals. Hard days should be followed by recovery days. I do not know much about recovery time. For example, I don’t know how long it takes muscles to heal from the micro-tears induced by a hard workout (but I think it is on the order of one to two days). This is probably what should govern your recovery schedule for any given muscle set.
My goal is general health and fitness, so I don’t tune my training toward any particular event or activity. I’m on maintenance and (I hope) gradual improvement regime. If you are tuning toward an event or season, there are longer cycles (weeks and months) of buildup and relaxation you should use. At that point you should find a coach.
All this assumes that your heart is a pretty good gauge of effort. For the most part it is. After using the heart rate monitor for some time on the bike and blades, I find that it correlates pretty well with perceived effort, that is, how hard it feels that I’m working. This isn’t always true though. It is not true when I’m boosting or dropping my heart rate. For example, increasing exertion to go from a heart rate of 150 to 155 can feel worse than a steady state of 165. Dropping from 170 to 165 can feel like heaven.
A second factor that can change heart rate is over-stressing. When I measure my resting heart rate after a hard workout day, it tends to be high. I think this is part of the normal recovery. If you over-stress for days or weeks, your resting heart rate will remain high and your total capacity to work will be reduced. Resting heart rate is both an indicator of general fitness (low is good) and of "overtraining" (goes up and stays up over a period of days). I think resting heart rate also goes up during illness.