Logistic Regression with R

Logistic Regression

It is used to predict the result of a categorical dependent variable based on one or more continuous or categorical independent variables. In other words, it is multiple regression analysis but with a dependent variable is categorical.

Examples

1. An employee may get promoted or not based on age, years of experience, last performance rating etc. We are trying to calculate the factors that affects promotion. In this case, two possible categories in dependent variable : "Promoted" and "Not Promoted".

2. We are interested in knowing how variables, such as age, sex, body mass index, effect blood pressure (sbp). In this case, two possible categories in dependent variable : "High Blood Pressure" and "Normal Blood Pressure".

Algorithm

Logistic regression is based on Maximum Likelihood (ML) Estimation which says coefficients should be chosen in such a way that it maximizes the Probability of Y given X (likelihood). With ML, the computer uses different "iterations" in which it tries different solutions until it gets the maximum likelihood estimates. Fisher Scoring is the most popular iterative method of estimating the regression parameters.

logit(p) = b0 + b1X1 + b2X2 + ------ + bk Xk

where logit(p) = loge(p / (1-p))

Take exponential both the sides

p : the probability of the dependent variable equaling a "success" or "event".

Logistic Regression Curve

Interpretation of Logistic Regression Estimates

If X increases by one unit, the log-odds of Y increases by k unit, given the other variables in the model are held constant.
In logistic regression, the odds ratio is easier to interpret. That is also called Point estimate. It is exponential value of estimate.

For Continuous Predictor

An unit increase in years of experience increases the odds of getting a job by a multiplicative factor of 4.27, given the other variables in the model are held constant. In other words, the odds of getting a job are increased by 327% (4.27-1)*100 for an unit increase in years of experience.

For Binary Predictor

The odds of a person having years of experience getting a job are 4.27 times greater than the odds of a person having no experience. 

Note :  To calculate 5 unit increase, 4.27 ^ 5 (instead of multiplication).

Magnitude : If you want to compare the magnitudes of positive and negative effects, simply take the inverse of the negative effects. For example, if Exp(B) = 2 on a positive effect variable, this has the same magnitude as variable with Exp(B) = 0.5 = ½ but in the opposite direction.

Odd Ratio (exp of estimate) less than 1 ==> Negative relationship (It means negative coefficient value of estimate coefficients)

1. The logit transformation of the outcome variable has a linear relationship with the predictor variables. The one way to check the assumption is to categorize the independent variables. Transform the numeric variables to 10/20 groups and then check whether they have linear or monotonic relationship.
2. No multicollinearity problem. No high correlationship between predictors.
3. No influential observations (Outliers).
4. Large Sample Size - It requires atleast 10 observations per independent variable.

C = Area under Curve = %concordant + (0.5 * %tied) 

It looks at maximum difference between distribution of cumulative events and cumulative non-events.

#Read Data File

mydata <- read.csv("http://www.ats.ucla.edu/stat/data/binary.csv")

#Summary

summary(mydata)

#Cross Tab

xtabs(~admit + rank, data = mydata)

#Data Preparation

mydata$rank <- factor(mydata$rank)

# Split data into training (70%) and validation (30%)

dt = sort(sample(nrow(mydata), nrow(mydata)*.7))

train<-mydata[dt,]

val<-mydata[-dt,] 

# Check number of rows in training and validation data sets

nrow(train)

nrow(val)

#Run Logistic Regression

mylogistic <- glm(admit ~ ., data = train, family = "binomial")

summary(mylogistic)$coefficient

#Stepwise Logistic Regression

mylogit = step(mylogistic)

#Logistic Regression Coefficient

summary.coeff0 = summary(mylogit)$coefficient

#Calculating Odd Ratios

OddRatio = exp(coef(mylogit))

summary.coeff = cbind(Variable = row.names(summary.coeff0), OddRatio, summary.coeff0)

row.names(summary.coeff) = NULL

#Get Standardized Coefficients

stdz.coff <- function (regmodel) 

{ b <- summary(regmodel)$coef[-1,1]

  sx <- sapply(regmodel$model[-1], sd)

  sy <- sapply(regmodel$model[1], sd)

  beta <- b * sx /  sy

  return(beta)

}

std.Coeff = data.frame(Standardized.Coeff = stdz.coff(mylogit))

std.Coeff = cbind(Variable = row.names(std.Coeff), std.Coeff)

row.names(std.Coeff) = NULL

#Final Summary Report

final = merge(summary.coeff, std.Coeff, by = "Variable", all.x = TRUE)

#Prediction

pred = predict(mylogit,val, type = "response")

finaldata = cbind(val, pred)

#Storing Model Performance Scores

library(ROCR)

pred_val <-prediction(pred ,finaldata$admit)

# Maximum Accuracy and prob. cutoff against it

acc.perf <- performance(pred_val, "acc")

ind = which.max( slot(acc.perf, "y.values")[[1]])

acc = slot(acc.perf, "y.values")[[1]][ind]

cutoff = slot(acc.perf, "x.values")[[1]][ind]

# Print Results

print(c(accuracy= acc, cutoff = cutoff))

# Calculating Area under Curve

perf_val <- performance(pred_val,"auc")

perf_val

# Plotting Lift curve

plot(performance(pred_val, measure="lift", x.measure="rpp"), colorize=TRUE)

# Plot the ROC curve

perf_val2 <- performance(pred_val, "tpr", "fpr")

plot(perf_val2, col = "green", lwd = 1.5)

#Calculating KS statistics

ks1.tree <- max(attr(perf_val2, "y.values")[[1]] - (attr(perf_val2, "x.values")[[1]]))

ks1.tree

Interpret Results :

1. Odd Ratio of GRE (Exp value of GRE Estimate) = 1.002 implies for a one unit increase in gre, the odds of being admitted to graduate school increase by a factor of 1.002.