Tag Archives: model

Error in Aetna’s Model Pricing Model

It’s not often that a model gets front page attention at the Wall Street Journal.  Last Friday, WSJ placed an article on Aetna’s pricing model trouble on the front page of the Marketplace section.  According to the article, Aetna had to pull a request for increasing prices for California policyholders when they discovered “miscalculations” in the monthly premiums.  It’s not clear from the article if there were multiple errors or the magnitude of the errors.  This story comes a few months after another insurer, Wellpoint, received similar press about how external auditors found mathematical errors in their filing.

How can this happen?  You would expect that with something so important, there would be ways to make sure that “model errors” would be caught and fixed BEFORE things get filed and rate increases are announced.  (Approximately 765,000 Wellpoint and Aetna policyholders would have been affected by the “wrong” price increase had the errors not been found.)

I have not worked with either company, but I have seen several kinds of similar mistakes in pricing/cost models.  The fact that many mathematical models have mistakes is not a surprise for those who study how models are built and used.  Ray Panko, whose research focuses on spreadsheet errors, has collected data from several authors that show that 88% of spreadsheets have errors.  Furthermore, Panko has found that 1% of all spreadsheet formulas are in error.  One percent may seem small… until you consider how “big” one mistake can be.  Can you imagine how many formulas must be in Wellpoint or Aetna’s model?

I recall talking to folks at NASA about failure rates for the 10,000 components they may have on a Space Shuttle.  One tenth of one percent failure (0.1%) is still 10 components.  Some components have backups or may be non-essential, but many components are essential to the survival of the crew and the success of the mission.

Think about your last “big model”.  How many errors do you think it has?  Would you be open to letting others audit it?

What is a “good model”?

Chances are, you have worked with models.  You may build and run complex models spanning many years or detailing lots of “steps” or “lines”.  You may have also used some simple models, such as a hand-drawn map that tells you how to get to the park.

A model is simply “some representation of reality”.  You have a real product line and a set of salespeople.  They take real orders from real customers.  If all goes well, you will receive real dollars (or Euros) for the sale.  So you have a speadsheet for projecting sales revenue associated with all this.  After a hard day at work, you may go running at a nearby park, and a simple hand-drawn map may suffice in getting you there.

Both the “sales projection spreadsheet” and the “hand drawn map to the park” are models.  There are two key features of a model, any model, that can be illustrated through the above examples.

1. A model must be constructed with more or less a specific question in mind.
2. A model is “good” or “bad” in light of this purpose.

So a hand-scribbled model, not drawn to scale, with some streets that are not labeled may indeed suffice for your manoeuvring through the neighborhood and finding the park.  But if you wanted to lay down utility lines and plan some street-ripping construction, you would want a different model, one that shows more specific dimensions and perhaps what kind of surface materials you are dealing with.

In many of the modeling and analytics work I do, I get asked, “how accurate is it?” or “how much data is in it?”.  I believe that the questions are valid, but the first questions to ask are: “what are we trying to solve?”.

Before you build a model, think about what you are trying to do.  Who is the audience?  How will the model (or the results) be used? What kind of questions will people ask?  Then we can go about discussing “level of detail” or “what kind of data”.