Monday, December 28, 2009

Differential Response or Uplift Modeling

Some time before the holidays, we received the following inquiry from a reader:

Dear Data Miners,



I’ve read interesting arguments for uplift modeling (also called incremental response modeling) [1], but I’m not sure how to implement it. I have responses from a direct mailing with a treatment group and a control group. Now what? Without data mining, I can calculate the uplift between the two groups but not for individual responses. With the data mining techniques I know, I can identify the ‘do not disturbs,’ but there’s more than avoiding mailing that group. How is uplift modeling implemented in general, and how could it be done in R or Weka?



[1] http://www.stochasticsolutions.com/pdf/CrossSell.pdf

I first heard the term "uplift modeling" from Nick Radcliffe, then of Quadstone. I think he may have invented it. In our book, Data Mining Techniques, we use the term "differential response analysis." It turns out that "differential response" has a very specific meaning in the child welfare world, so perhaps we'll switch to "incremental response" or "uplift" in the next edition. But whatever it is called, you can approach this problem in a cell-based fashion without any special tools. Cell-based approaches divide customers into cells or segments in such a way that all members of a cell are similar to one another along some set of dimensions considered to be important for the particular application. You can then measure whatever you wish to optimize (order size, response rate, . . .) by cell and, going forward, treat the cells where treatment has the greatest effect.

Here, the quantity  to measure is the difference in response rate or average order size between treated and untreated groups of otherwise similar customers. Within each cell, we need a randomly selected treatment group and a randomly selected control group; the incremental response or uplift is the difference in average order size (or whatever) between the two. Of course some cells will have higher or lower overall average order size, but that is not the focus of incremental response modeling. The question is not "What is the average order size of women between 40 and 50 who have made more than 2 previous purchases and live in a neighborhood where average household income is two standard deviations above the regional average?" It is "What is the change in order size for this group?"

Ideally, of course, you should design the segmentation and assignment of customers to treatment and control groups before the test, but the reader who submitted the question has already done the direct mailing and tallied the responses. Is it now too late to analyze incremental response?  That depends: If the control group is a true random control group and if it is large enough that it can be partitioned into segments that are still large enough to provide statistically significant differences in order size, it is not too late. You could, for instance, compare the incremental response of male and female responders.

A cell-based approach is only useful if the segment definitions are such that incremental response really does vary across cells. Dividing customers into male and female segments won't help if men and women are equally responsive to the treatment. This is the advantage of the special-purpose uplift modeling software developed by Quadstone (now Portrait Software). This tool builds a decision tree where the splitting criteria is maximizing the difference in incremental response. This automatically leads to segments (the leaves of the tree) characterized by either high or low uplift.  That is a really cool idea, but the lack of such a tool is not a reason to avoid incremental response analysis.

Labels: , , ,

2 Comments:

Anonymous Jeff Allard said...

Here is a link to a paper which explains a simple approach (you could modify to use something other than logistic regression if you chose):

http://www.sigkdd.org/explorations/issues/4-2-2002-12/lo.pdf

December 29, 2009 8:44 AM  
Anonymous Patrick Surry said...

We (Quadstone, now Portrait) originally used the term "differential response analysis" as well. The earliest ref I'm aware of is Radcliffe & Surry, "Differential Response Analysis: Modeling True Response by Isolating the Effect of a Single Action", Credit Scoring & Credit Control VI, 1999. We soon switched to "Uplift" as a more approachable term, see for example Radcliffe, "Using Control Groups to Target on Predicted Lift: Building and Assessing Uplift Models", DMA Journal, 2007.

The attractive thing about the tree-based approach is that it directly models lift: each time it considers a potential segmentation it can assess the significance of the observed lift in each segment. In contrast, Lo's approach (and many related regression-based variants) are indirect, in that they involve creating separate predictions for the response rate in the treated and control groups and then estimating lift by subtraction. We find these approaches aren't always robust in practice: my intuition for why being that they rely on the error terms in the two predictions canceling in a nice way, which isn't always the case.

We're collecting references and resources on uplift modeling at our website (http://www.portraitsoftware.com/uplift-modeling), soon to include a new discussion forum.

patrick.surry at portraitsoftware.com

January 6, 2010 4:34 AM  

Post a Comment

Links to this post:

Create a Link

<< Home