Package 'caret'

Description

Conversion functions for class confusionMatrix

Usage

## S3 method for class 'confusionMatrix'
as.matrix(x, what = "xtabs", ...)

Arguments

Details

For as.table, the cross-tabulations are saved. For as.matrix, the three object types are saved in matrix format.

Value

A matrix or table

Author(s)

Max Kuhn

Examples

###################
## 2 class example

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)),
                levels = rev(lvs))
pred <- factor(
               c(
                 rep(lvs, times = c(54, 32)),
                 rep(lvs, times = c(27, 231))),
               levels = rev(lvs))

xtab <- table(pred, truth)

results <- confusionMatrix(xtab)
as.table(results)
as.matrix(results)
as.matrix(results, what = "overall")
as.matrix(results, what = "classes")

###################
## 3 class example

xtab <- confusionMatrix(iris$Species, sample(iris$Species))
as.matrix(xtab)

Description

Aggregate several neural network models

Usage

avNNet(x, ...)

## S3 method for class 'formula'
avNNet(
  formula,
  data,
  weights,
  ...,
  repeats = 5,
  bag = FALSE,
  allowParallel = TRUE,
  seeds = sample.int(1e+05, repeats),
  subset,
  na.action,
  contrasts = NULL
)

## Default S3 method:
avNNet(
  x,
  y,
  repeats = 5,
  bag = FALSE,
  allowParallel = TRUE,
  seeds = sample.int(1e+05, repeats),
  ...
)

## S3 method for class 'avNNet'
print(x, ...)

## S3 method for class 'avNNet'
predict(object, newdata, type = c("raw", "class", "prob"), ...)

Arguments

Details

Following Ripley (1996), the same neural network model is fit using different random number seeds. All the resulting models are used for prediction. For regression, the output from each network are averaged. For classification, the model scores are first averaged, then translated to predicted classes. Bagging can also be used to create the models.

If a parallel backend is registered, the foreach package is used to train the networks in parallel.

Value

For avNNet, an object of "avNNet" or "avNNet.formula". Items of interest in #' the output are:

Author(s)

These are heavily based on the nnet code from Brian Ripley.

References

Ripley, B. D. (1996) Pattern Recognition and Neural Networks. Cambridge.

See Also

nnet, preProcess

Examples

data(BloodBrain)
## Not run: 
modelFit <- avNNet(bbbDescr, logBBB, size = 5, linout = TRUE, trace = FALSE)
modelFit

predict(modelFit, bbbDescr)

## End(Not run)

Description

bag provides a framework for bagging classification or regression models. The user can provide their own functions for model building, prediction and aggregation of predictions (see Details below).

Usage

bag(x, ...)

bagControl(
  fit = NULL,
  predict = NULL,
  aggregate = NULL,
  downSample = FALSE,
  oob = TRUE,
  allowParallel = TRUE
)

## Default S3 method:
bag(x, y, B = 10, vars = ncol(x), bagControl = NULL, ...)

## S3 method for class 'bag'
predict(object, newdata = NULL, ...)

## S3 method for class 'bag'
print(x, ...)

## S3 method for class 'bag'
summary(object, ...)

## S3 method for class 'summary.bag'
print(x, digits = max(3, getOption("digits") - 3), ...)

ldaBag

plsBag

nbBag

ctreeBag

svmBag

nnetBag

Arguments

Format

An object of class list of length 3.

An object of class list of length 3.

An object of class list of length 3.

An object of class list of length 3.

An object of class list of length 3.

An object of class list of length 3.

Details

The function is basically a framework where users can plug in any model in to assess the effect of bagging. Examples functions can be found in ldaBag, plsBag , nbBag, svmBag and nnetBag. Each has elements fit, pred and aggregate.

One note: when vars is not NULL, the sub-setting occurs prior to the fit and #' predict functions are called. In this way, the user probably does not need to account for the #' change in predictors in their functions.

When using bag with train, classification models should use type = "prob" #' inside of the predict function so that predict.train(object, newdata, type = "prob") will #' work.

If a parallel backend is registered, the foreach package is used to train the models in parallel.

Value

bag produces an object of class bag with elements

Author(s)

Max Kuhn

Examples

## A simple example of bagging conditional inference regression trees:
data(BloodBrain)

## treebag <- bag(bbbDescr, logBBB, B = 10,
##                bagControl = bagControl(fit = ctreeBag$fit,
##                                        predict = ctreeBag$pred,
##                                        aggregate = ctreeBag$aggregate))




## An example of pooling posterior probabilities to generate class predictions
data(mdrr)

## remove some zero variance predictors and linear dependencies
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .95)]

## basicLDA <- train(mdrrDescr, mdrrClass, "lda")

## bagLDA2 <- train(mdrrDescr, mdrrClass,
##                  "bag",
##                  B = 10,
##                  bagControl = bagControl(fit = ldaBag$fit,
##                                          predict = ldaBag$pred,
##                                          aggregate = ldaBag$aggregate),
##                  tuneGrid = data.frame(vars = c((1:10)*10 , ncol(mdrrDescr))))

Description

A bagging wrapper for multivariate adaptive regression splines (MARS) via the earth function

Usage

bagEarth(x, ...)

## Default S3 method:
bagEarth(x, y, weights = NULL, B = 50, summary = mean, keepX = TRUE, ...)

## S3 method for class 'formula'
bagEarth(
  formula,
  data = NULL,
  B = 50,
  summary = mean,
  keepX = TRUE,
  ...,
  subset,
  weights = NULL,
  na.action = na.omit
)

## S3 method for class 'bagEarth'
print(x, ...)

Arguments

Details

The function computes a Earth model for each bootstap sample.

Value

A list with elements

Author(s)

Max Kuhn (bagEarth.formula is based on Ripley's nnet.formula)

References

J. Friedman, “Multivariate Adaptive Regression Splines” (with discussion) (1991). Annals of Statistics, 19/1, 1-141.

See Also

earth, predict.bagEarth

Examples

## Not run: 
library(mda)
library(earth)
data(trees)
fit1 <- earth(x = trees[,-3], y = trees[,3])
set.seed(2189)
fit2 <- bagEarth(x = trees[,-3], y = trees[,3], B = 10)

## End(Not run)

Description

A bagging wrapper for flexible discriminant analysis (FDA) using multivariate adaptive regression splines (MARS) basis functions

Usage

bagFDA(x, ...)

## Default S3 method:
bagFDA(x, y, weights = NULL, B = 50, keepX = TRUE, ...)

## S3 method for class 'formula'
bagFDA(
  formula,
  data = NULL,
  B = 50,
  keepX = TRUE,
  ...,
  subset,
  weights = NULL,
  na.action = na.omit
)

## S3 method for class 'bagFDA'
print(x, ...)

Arguments

Details

The function computes a FDA model for each bootstap sample.

Value

A list with elements

Author(s)

Max Kuhn (bagFDA.formula is based on Ripley's nnet.formula)

References

J. Friedman, “Multivariate Adaptive Regression Splines” (with discussion) (1991). Annals of Statistics, 19/1, 1-141.

See Also

fda, predict.bagFDA

Examples

library(mlbench)
library(earth)
data(Glass)

set.seed(36)
inTrain <- sample(1:dim(Glass)[1], 150)

trainData <- Glass[ inTrain, ]
testData  <- Glass[-inTrain, ]


set.seed(3577)
baggedFit <- bagFDA(Type ~ ., trainData)
confusionMatrix(data = predict(baggedFit, testData[, -10]),
                reference = testData[, 10])

Description

Mente and Lombardo (2005) develop models to predict the log of the ratio of the concentration of a compound in the brain and the concentration in blood. For each compound, they computed three sets of molecular descriptors: MOE 2D, rule-of-five and Charge Polar Surface Area (CPSA). In all, 134 descriptors were calculated. Included in this package are 208 non-proprietary literature compounds. The vector logBBB contains the concentration ratio and the data fame bbbDescr contains the descriptor values.

Value

Source

Mente, S.R. and Lombardo, F. (2005). A recursive-partitioning model for blood-brain barrier permeation, Journal of Computer-Aided Molecular Design, Vol. 19, pg. 465-481.

Description

These classes can be used to estimate transformations and apply them to existing and future data

Usage

BoxCoxTrans(y, ...)

## Default S3 method:
BoxCoxTrans(
  y,
  x = rep(1, length(y)),
  fudge = 0.2,
  numUnique = 3,
  na.rm = FALSE,
  ...
)

## S3 method for class 'BoxCoxTrans'
print(x, newdata, digits = 3, ...)

## S3 method for class 'BoxCoxTrans'
predict(object, newdata, ...)

Arguments

Details

BoxCoxTrans function is basically a wrapper for the boxcox function in the MASS library. It can be used to estimate the transformation and apply it to new data.

expoTrans estimates the exponential transformation of Manly (1976) but assumes a common mean for the data. The transformation parameter is estimated by directly maximizing the likelihood.

If any(y <= 0) or if length(unique(y)) < numUnique, lambda is not estimated and no transformation is applied.

Value

Both functions returns a list of class of either BoxCoxTrans or expoTrans with elements

BoxCoxTrans also returns:

The predict functions returns numeric vectors of transformed values

Author(s)

Max Author

References

Box, G. E. P. and Cox, D. R. (1964) An analysis of transformations (with discussion). Journal of the Royal Statistical Society B, 26, 211-252. Manly, B. L. (1976) Exponential data transformations. The Statistician, 25, 37 - 42.

See Also

boxcox, preProcess, optim

Examples

data(BloodBrain)

ratio <- exp(logBBB)
bc <- BoxCoxTrans(ratio)
bc

predict(bc, ratio[1:5])

ratio[5] <- NA
bc2 <- BoxCoxTrans(ratio, bbbDescr$tpsa, na.rm = TRUE)
bc2

manly <- expoTrans(ratio)
manly

Description

For classification models, this function creates a 'calibration plot' that describes how consistent model probabilities are with observed event rates.

Usage

calibration(x, ...)

## Default S3 method:
calibration(x, ...)

## S3 method for class 'formula'
calibration(
  x,
  data = NULL,
  class = NULL,
  cuts = 11,
  subset = TRUE,
  lattice.options = NULL,
  ...
)

## S3 method for class 'calibration'
print(x, ...)

## S3 method for class 'calibration'
xyplot(x, data = NULL, ...)

## S3 method for class 'calibration'
ggplot(data, ..., bwidth = 2, dwidth = 3)

Arguments

Details

calibration.formula is used to process the data and xyplot.calibration is used to create the plot.

To construct the calibration plot, the following steps are used for each model:

1. The data are split into cuts - 1 roughly equal groups by their class probabilities

2. the number of samples with true results equal to class are determined

3. the event rate is determined for each bin

xyplot.calibration produces a plot of the observed event rate by the mid-point of the bins.

This implementation uses the lattice function xyplot, so plot elements can be changed via panel functions, trellis.par.set or other means. calibration uses the panel function panel.calibration by default, but it can be changed by passing that argument into xyplot.calibration.

The following elements are set by default in the plot but can be changed by passing new values into xyplot.calibration: xlab = "Bin Midpoint", ylab = "Observed Event Percentage", type = "o", ylim = extendrange(c(0, 100)),xlim = extendrange(c(0, 100)) and panel = panel.calibration

For the ggplot method, confidence intervals on the estimated proportions (from binom.test) are also shown.

Value

calibration.formula returns a list with elements:

xyplot.calibration returns a lattice object

Author(s)

Max Kuhn, some lattice code and documentation by Deepayan Sarkar

See Also

xyplot, trellis.par.set

Examples

## Not run: 
data(mdrr)
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .5)]


inTrain <- createDataPartition(mdrrClass)
trainX <- mdrrDescr[inTrain[[1]], ]
trainY <- mdrrClass[inTrain[[1]]]
testX <- mdrrDescr[-inTrain[[1]], ]
testY <- mdrrClass[-inTrain[[1]]]

library(MASS)

ldaFit <- lda(trainX, trainY)
qdaFit <- qda(trainX, trainY)

testProbs <- data.frame(obs = testY,
                        lda = predict(ldaFit, testX)$posterior[,1],
                        qda = predict(qdaFit, testX)$posterior[,1])

calibration(obs ~ lda + qda, data = testProbs)

calPlotData <- calibration(obs ~ lda + qda, data = testProbs)
calPlotData

xyplot(calPlotData, auto.key = list(columns = 2))

## End(Not run)

Description

Ancillary functions for univariate feature selection

Usage

caretSBF

anovaScores(x, y)

gamScores(x, y)

Arguments

Format

An object of class list of length 5.

Details

More details on these functions can be found at http://topepo.github.io/caret/feature-selection-using-univariate-filters.html.

This page documents the functions that are used in selection by filtering (SBF). The functions described here are passed to the algorithm via the functions argument of sbfControl.

See sbfControl for details on how these functions should be defined.

anovaScores and gamScores are two examples of univariate filtering functions. anovaScores fits a simple linear model between a single feature and the outcome, then the p-value for the whole model F-test is returned. gamScores fits a generalized additive model between a single predictor and the outcome using a smoothing spline basis function. A p-value is generated using the whole model test from summary.Gam and is returned.

If a particular model fails for lm or gam, a p-value of 1 is returned.

Author(s)

Max Kuhn

See Also

sbfControl, sbf, summary.Gam

Description

Kuiper (2008) collected data on Kelly Blue Book resale data for 804 GM cars (2005 model year).

Value

Source

Kuiper, S. (2008). Introduction to Multiple Regression: How Much Is Your Car Worth?, Journal of Statistics Education, Vol. 16 http://jse.amstat.org/jse_archive.htm#2008.

Description

This function computes the class centroids and covariance matrix for a training set for determining Mahalanobis distances of samples to each class centroid.

Usage

classDist(x, ...)

## Default S3 method:
classDist(x, y, groups = 5, pca = FALSE, keep = NULL, ...)

## S3 method for class 'classDist'
predict(object, newdata, trans = log, ...)

Arguments

Details

For factor outcomes, the data are split into groups for each class and the mean and covariance matrix are calculated. These are then used to compute Mahalanobis distances to the class centers (using predict.classDist The function will check for non-singular matrices.

For numeric outcomes, the data are split into roughly equal sized bins based on groups. Percentiles are used to split the data.

Value

for classDist, an object of class classDist with elements:

For predict.classDist, a matrix with columns for each class. The columns names are the names of the class with the prefix dist.. In the case of numeric y, the class labels are the percentiles. For example, of groups = 9, the variable names would be dist.11.11, dist.22.22, etc.

Author(s)

Max Kuhn

References

Forina et al. CAIMAN brothers: A family of powerful classification and class modeling techniques. Chemometrics and Intelligent Laboratory Systems (2009) vol. 96 (2) pp. 239-245

See Also

mahalanobis

Examples

trainSet <- sample(1:150, 100)

distData <- classDist(iris[trainSet, 1:4],
                      iris$Species[trainSet])

newDist <- predict(distData,
                   iris[-trainSet, 1:4])

splom(newDist, groups = iris$Species[-trainSet])

Description

Calculates a cross-tabulation of observed and predicted classes with associated statistics.

Usage

confusionMatrix(data, ...)

## Default S3 method:
confusionMatrix(
  data,
  reference,
  positive = NULL,
  dnn = c("Prediction", "Reference"),
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

## S3 method for class 'matrix'
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

## S3 method for class 'table'
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

Arguments

Details

The functions requires that the factors have exactly the same levels.

For two class problems, the sensitivity, specificity, positive predictive value and negative predictive value is calculated using the positive argument. Also, the prevalence of the "event" is computed from the data (unless passed in as an argument), the detection rate (the rate of true events also predicted to be events) and the detection prevalence (the prevalence of predicted events).

Suppose a 2x2 table with notation

The formulas used here are:

$Sensitivity = A/(A+C)$

$Specificity = D/(B+D)$

$Prevalence = (A+C)/(A+B+C+D)$

$PPV = (sensitivity * prevalence)/((sensitivity*prevalence) + ((1-specificity)*(1-prevalence)))$

$NPV = (specificity * (1-prevalence))/(((1-sensitivity)*prevalence) + ((specificity)*(1-prevalence)))$

$Detection Rate = A/(A+B+C+D)$

$Detection Prevalence = (A+B)/(A+B+C+D)$

$Balanced Accuracy = (sensitivity+specificity)/2$

$Precision = A/(A+B)$

$Recall = A/(A+C)$

$F1 = (1+beta^2)*precision*recall/((beta^2 * precision)+recall)$

where beta = 1 for this function.

See the references for discussions of the first five formulas.

For more than two classes, these results are calculated comparing each factor level to the remaining levels (i.e. a "one versus all" approach).

The overall accuracy and unweighted Kappa statistic are calculated. A p-value from McNemar's test is also computed using mcnemar.test (which can produce NA values with sparse tables).

The overall accuracy rate is computed along with a 95 percent confidence interval for this rate (using binom.test) and a one-sided test to see if the accuracy is better than the "no information rate," which is taken to be the largest class percentage in the data.

Value

a list with elements

Note

If the reference and data factors have the same levels, but in the incorrect order, the function will reorder them to the order of the data and issue a warning.

Author(s)

Max Kuhn

References

Kuhn, M. (2008), “Building predictive models in R using the caret package, ” Journal of Statistical Software, (doi:10.18637/jss.v028.i05).

Altman, D.G., Bland, J.M. (1994) “Diagnostic tests 1: sensitivity and specificity,” British Medical Journal, vol 308, 1552.

Altman, D.G., Bland, J.M. (1994) “Diagnostic tests 2: predictive values,” British Medical Journal, vol 309, 102.

Velez, D.R., et. al. (2008) “A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction.,” Genetic Epidemiology, vol 4, 306.

See Also

as.table.confusionMatrix, as.matrix.confusionMatrix, sensitivity, specificity, posPredValue, negPredValue, print.confusionMatrix, binom.test

Examples

###################
## 2 class example

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)),
                levels = rev(lvs))
pred <- factor(
               c(
                 rep(lvs, times = c(54, 32)),
                 rep(lvs, times = c(27, 231))),
               levels = rev(lvs))

xtab <- table(pred, truth)

confusionMatrix(xtab)
confusionMatrix(pred, truth)
confusionMatrix(xtab, prevalence = 0.25)

###################
## 3 class example

confusionMatrix(iris$Species, sample(iris$Species))

newPrior <- c(.05, .8, .15)
names(newPrior) <- levels(iris$Species)

confusionMatrix(iris$Species, sample(iris$Species))

Description

Using a train, rfe, sbf object, determine a confusion matrix based on the resampling procedure

Usage

## S3 method for class 'train'
confusionMatrix(
  data,
  norm = "overall",
  dnn = c("Prediction", "Reference"),
  ...
)

Arguments

Details

When train is used for tuning a model, it tracks the confusion matrix cell entries for the hold-out samples. These can be aggregated and used for diagnostic purposes. For train, the matrix is estimated for the final model tuning parameters determined by train. For rfe, the matrix is associated with the optimal number of variables.

There are several ways to show the table entries. Using norm = "none" will show the aggregated counts of samples on each of the cells (across all resamples). For norm = "average", the average number of cell counts across resamples is computed (this can help evaluate how many holdout samples there were on average). The default is norm = "overall", which is equivalento to "average" but in percentages.

Value

a list of class confusionMatrix.train, confusionMatrix.rfe or confusionMatrix.sbf with elements

Author(s)

Max Kuhn

See Also

confusionMatrix, train, rfe, sbf, trainControl

Examples

data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

knnFit <- train(TrainData, TrainClasses,
                method = "knn",
                preProcess = c("center", "scale"),
                tuneLength = 10,
                trControl = trainControl(method = "cv"))
confusionMatrix(knnFit)
confusionMatrix(knnFit, "average")
confusionMatrix(knnFit, "none")

Description

From Sutherland, O'Brien, and Weaver (2003): "A set of 467 cyclooxygenase-2 (COX-2) inhibitors has been assembled from the published work of a single research group, with in vitro activities against human recombinant enzyme expressed as IC50 values ranging from 1 nM to >100 uM (53 compounds have indeterminate IC50 values)."

Details

The data are in the Supplemental Data file for the article.

A set of 255 descriptors (MOE2D and QikProp) were generated. To classify the data, we used a cutoff of $2^2.5$ to determine activity

Value

Source

Sutherland, J. J., O'Brien, L. A. and Weaver, D. F. (2003). Spline-Fitting with a Genetic Algorithm: A Method for Developing Classification Structure-Activity Relationships, Journal of Chemical Information and Computer Sciences, Vol. 43, pg. 1906-1915.

Description

A series of test/training partitions are created using createDataPartition while createResample creates one or more bootstrap samples. createFolds splits the data into k groups while createTimeSlices creates cross-validation split for series data. groupKFold splits the data based on a grouping factor.

Usage

createDataPartition(
  y,
  times = 1,
  p = 0.5,
  list = TRUE,
  groups = min(5, length(y))
)

createFolds(y, k = 10, list = TRUE, returnTrain = FALSE)

createMultiFolds(y, k = 10, times = 5)

createTimeSlices(y, initialWindow, horizon = 1, fixedWindow = TRUE, skip = 0)

groupKFold(group, k = length(unique(group)))

createResample(y, times = 10, list = TRUE)

Arguments

Details

For bootstrap samples, simple random sampling is used.

For other data splitting, the random sampling is done within the levels of y when y is a factor in an attempt to balance the class distributions within the splits.

For numeric y, the sample is split into groups sections based on percentiles and sampling is done within these subgroups. For createDataPartition, the number of percentiles is set via the groups argument. For createFolds and createMultiFolds, the number of groups is set dynamically based on the sample size and k. For smaller samples sizes, these two functions may not do stratified splitting and, at most, will split the data into quartiles.

Also, for createDataPartition, very small class sizes (<= 3) the classes may not show up in both the training and test data

For multiple k-fold cross-validation, completely independent folds are created. The names of the list objects will denote the fold membership using the pattern "Foldi.Repj" meaning the ith section (of k) of the jth cross-validation set (of times). Note that this function calls createFolds with list = TRUE and returnTrain = TRUE.

Hyndman and Athanasopoulos (2013)) discuss rolling forecasting origin techniques that move the training and test sets in time. createTimeSlices can create the indices for this type of splitting.

For Group k-fold cross-validation, the data are split such that no group is contained in both the modeling and holdout sets. One or more group could be left out, depending on the value of k.

Value

A list or matrix of row position integers corresponding to the training data. For createTimeSlices subsamples are named by the end index of each training subsample.

Author(s)

Max Kuhn, createTimeSlices by Tony Cooper

References

http://topepo.github.io/caret/data-splitting.html

Hyndman and Athanasopoulos (2013), Forecasting: principles and practice. https://otexts.com/fpp2/

Examples

data(oil)
createDataPartition(oilType, 2)

x <- rgamma(50, 3, .5)
inA <- createDataPartition(x, list = FALSE)

plot(density(x[inA]))
rug(x[inA])

points(density(x[-inA]), type = "l", col = 4)
rug(x[-inA], col = 4)

createResample(oilType, 2)

createFolds(oilType, 10)
createFolds(oilType, 5, FALSE)

createFolds(rnorm(21))

createTimeSlices(1:9, 5, 1, fixedWindow = FALSE)
createTimeSlices(1:9, 5, 1, fixedWindow = TRUE)
createTimeSlices(1:9, 5, 3, fixedWindow = TRUE)
createTimeSlices(1:9, 5, 3, fixedWindow = FALSE)

createTimeSlices(1:15, 5, 3)
createTimeSlices(1:15, 5, 3, skip = 2)
createTimeSlices(1:15, 5, 3, skip = 3)

set.seed(131)
groups <- sort(sample(letters[1:4], size = 20, replace = TRUE))
table(groups)
folds <- groupKFold(groups)
lapply(folds, function(x, y) table(y[x]), y = groups)

Description

Given two numeric vectors of data, the mean squared error and R-squared are calculated. For two factors, the overall agreement rate and Kappa are determined.

Usage

defaultSummary(data, lev = NULL, model = NULL)

postResample(pred, obs)

twoClassSummary(data, lev = NULL, model = NULL)

mnLogLoss(data, lev = NULL, model = NULL)

multiClassSummary(data, lev = NULL, model = NULL)

prSummary(data, lev = NULL, model = NULL)

Arguments

Details

postResample is meant to be used with apply across a matrix. For numeric data the code checks to see if the standard deviation of either vector is zero. If so, the correlation between those samples is assigned a value of zero. NA values are ignored everywhere.

Note that many models have more predictors (or parameters) than data points, so the typical mean squared error denominator (n - p) does not apply. Root mean squared error is calculated using sqrt(mean((pred - obs)^2. Also, $R^2$ is calculated wither using as the square of the correlation between the observed and predicted outcomes when form = "corr". when form = "traditional",

$R^2 = 1-\frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

. Mean absolute error is calculated using mean(abs(pred-obs)).

defaultSummary is the default function to compute performance metrics in train. It is a wrapper around postResample. The first argument is data, which is data.frame with columns named obs and pred for the observed and predicted outcome values (either numeric data for regression or character values for classification). The second argument is lev, a character string that has the outcome factor levels or NULL for a regression model. The third parameter is model, which can be used if a summary metric is specific to a model function. If other columns from the data are required to compute the summary statistics, but should not be used in the model, the recipe method for train can be used.

twoClassSummary computes sensitivity, specificity and the area under the ROC curve. mnLogLoss computes the minus log-likelihood of the multinomial distribution (without the constant term):

$-logLoss = \frac{-1}{n}\sum_{i=1}^n \sum_{j=1}^C y_{ij} \log(p_{ij})$

where the y values are binary indicators for the classes and p are the predicted class probabilities.

prSummary (for precision and recall) computes values for the default 0.50 probability cutoff as well as the area under the precision-recall curve across all cutoffs and is labelled as "AUC" in the output. If assumes that the first level of the factor variables corresponds to a relevant result but the lev argument can be used to change this.

multiClassSummary computes some overall measures of for performance (e.g. overall accuracy and the Kappa statistic) and several averages of statistics calculated from "one-versus-all" configurations. For example, if there are three classes, three sets of sensitivity values are determined and the average is reported with the name ("Mean_Sensitivity"). The same is true for a number of statistics generated by confusionMatrix. With two classes, the basic sensitivity is reported with the name "Sensitivity".

To use twoClassSummary and/or mnLogLoss, the classProbs argument of trainControl should be TRUE. multiClassSummary can be used without class probabilities but some statistics (e.g. overall log loss and the average of per-class area under the ROC curves) will not be in the result set.

Other functions can be used via the summaryFunction argument of trainControl. Custom functions must have the same arguments asdefaultSummary.

The function getTrainPerf returns a one row data frame with the resampling results for the chosen model. The statistics will have the prefix "Train" (i.e. "TrainROC"). There is also a column called "method" that echoes the argument of the call to trainControl of the same name.

Value

A vector of performance estimates.

Author(s)

Max Kuhn, Zachary Mayer

References

Kvalseth. Cautionary note about $R^2$. American Statistician (1985) vol. 39 (4) pp. 279-285

See Also

trainControl

Examples

predicted <-  matrix(rnorm(50), ncol = 5)
observed <- rnorm(10)
apply(predicted, 2, postResample, obs = observed)

classes <- c("class1", "class2")
set.seed(1)
dat <- data.frame(obs =  factor(sample(classes, 50, replace = TRUE)),
                  pred = factor(sample(classes, 50, replace = TRUE)),
                  class1 = runif(50))
dat$class2 <- 1 - dat$class1

defaultSummary(dat, lev = classes)
twoClassSummary(dat, lev = classes)
prSummary(dat, lev = classes)
mnLogLoss(dat, lev = classes)

Description

A set of lattice functions are provided to plot the resampled performance estimates (e.g. classification accuracy, RMSE) over different subset sizes.

Usage

## S3 method for class 'rfe'
densityplot(x, data = NULL, metric = x$metric, ...)

Arguments

Details

By default, only the resampling results for the optimal model are saved in the rfe object. The function rfeControl can be used to save all the results using the returnResamp argument.

If leave-one-out or out-of-bag resampling was specified, plots cannot be produced (see the method argument of rfeControl)

Value

A lattice plot object

Author(s)

Max Kuhn

See Also

rfe, rfeControl, histogram, densityplot, xyplot, stripplot

Examples

## Not run: 
library(mlbench)
n <- 100
p <- 40
sigma <- 1
set.seed(1)
sim <- mlbench.friedman1(n, sd = sigma)
x <- cbind(sim$x,  matrix(rnorm(n * p), nrow = n))
y <- sim$y
colnames(x) <- paste("var", 1:ncol(x), sep = "")

normalization <- preProcess(x)
x <- predict(normalization, x)
x <- as.data.frame(x, stringsAsFactors = TRUE)
subsets <- c(10, 15, 20, 25)

ctrl <- rfeControl(
                   functions = lmFuncs,
                   method = "cv",
                   verbose = FALSE,
                   returnResamp = "all")

lmProfile <- rfe(x, y,
                 sizes = subsets,
                 rfeControl = ctrl)
xyplot(lmProfile)
stripplot(lmProfile)

histogram(lmProfile)
densityplot(lmProfile)

## End(Not run)

Description

Sutherland and Weaver (2004) discuss QSAR models for dihydrofolate reductase (DHFR) inhibition. This data set contains values for 325 compounds. For each compound, 228 molecular descriptors have been calculated. Additionally, each samples is designated as "active" or "inactive".

Details

The data frame dhfr contains a column called Y with the outcome classification. The remainder of the columns are molecular descriptor values.

Value

Source

Sutherland, J.J. and Weaver, D.F. (2004). Three-dimensional quantitative structure-activity and structure-selectivity relationships of dihydrofolate reductase inhibitors, Journal of Computer-Aided Molecular Design, Vol. 18, pg. 309-331.

Description

Methods for making inferences about differences between models

Usage

## S3 method for class 'resamples'
diff(
  x,
  models = x$models,
  metric = x$metrics,
  test = t.test,
  confLevel = 0.95,
  adjustment = "bonferroni",
  ...
)

## S3 method for class 'diff.resamples'
summary(object, digits = max(3, getOption("digits") - 3), ...)

compare_models(a, b, metric = a$metric[1])

Arguments

Details

The ideas and methods here are based on Hothorn et al. (2005) and Eugster et al. (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from confLevel to 1-((1-confLevel)/p) here p is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

compare_models is a shorthand function to compare two models using a single metric. It returns the results of t.test on the differences.

Value

An object of class "diff.resamples" with elements:

or...

An object of class "summary.diff.resamples" with elements:

...or (for compare_models) an object of class htest resulting from t.test.

Author(s)

Max Kuhn

References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

See Also

resamples, dotplot.diff.resamples, densityplot.diff.resamples, bwplot.diff.resamples, levelplot.diff.resamples

Examples

## Not run: 
#load(url("http://topepo.github.io/caret/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))

difs <- diff(resamps)

difs

summary(difs)

compare_models(rpartFit, ctreeFit)

## End(Not run)

Description

A lattice dotplot is created from an object of class varImp.train.

Usage

dotPlot(x, top = min(20, dim(x$importance)[1]), ...)

Arguments

Value

an object of class trellis.

Author(s)

Max Kuhn

See Also

varImp, dotplot

Examples

data(iris)
TrainData <- iris[,1:4]
TrainClasses <- iris[,5]

knnFit <- train(TrainData, TrainClasses, "knn")

knnImp <- varImp(knnFit)

dotPlot(knnImp)

Description

Lattice functions for visualizing resampling result differences between models

Usage

## S3 method for class 'diff.resamples'
dotplot(x, data = NULL, metric = x$metric[1], ...)

Arguments

Details

densityplot and bwplot display univariate visualizations of the resampling distributions. levelplot displays the matrix of pair-wide comparisons. dotplot shows the differences along with their associated confidence intervals.

Value

a lattice object

Author(s)

Max Kuhn

See Also

resamples, diff.resamples, bwplot, densityplot, xyplot, splom

Examples

## Not run: 
#load(url("http://topepo.github.io/caret/exampleModels.RData"))

resamps <- resamples(list(CART = rpartFit,
                          CondInfTree = ctreeFit,
                          MARS = earthFit))
difs <- diff(resamps)

dotplot(difs)

densityplot(difs,
            metric = "RMSE",
            auto.key = TRUE,
            pch = "|")

bwplot(difs,
       metric = "RMSE")

levelplot(difs, what = "differences")


## End(Not run)

Description

downSample will randomly sample a data set so that all classes have the same frequency as the minority class. upSample samples with replacement to make the class distributions equal

Usage

downSample(x, y, list = FALSE, yname = "Class")

Arguments

Details

Simple random sampling is used to down-sample for the majority class(es). Note that the minority class data are left intact and that the samples will be re-ordered in the down-sampled version.

For up-sampling, all the original data are left intact and additional samples are added to the minority classes with replacement.

Value

Either a data frame or a list with elements x and y.

Author(s)

Max Kuhn

Examples

## A ridiculous example...
data(oil)
table(oilType)
downSample(fattyAcids, oilType)

upSample(fattyAcids, oilType)

Description

dummyVars creates a full set of dummy variables (i.e. less than full rank parameterization)

Usage

dummyVars(formula, ...)

## Default S3 method:
dummyVars(formula, data, sep = ".", levelsOnly = FALSE, fullRank = FALSE, ...)

## S3 method for class 'dummyVars'
print(x, ...)

## S3 method for class 'dummyVars'
predict(object, newdata, na.action = na.pass, ...)

contr.ltfr(n, contrasts = TRUE, sparse = FALSE)

class2ind(x, drop2nd = FALSE)

Arguments

Details

Most of the contrasts functions in R produce full rank parameterizations of the predictor data. For example, contr.treatment creates a reference cell in the data and defines dummy variables for all factor levels except those in the reference cell. For example, if a factor with 5 levels is used in a model formula alone, contr.treatment creates columns for the intercept and all the factor levels except the first level of the factor. For the data in the Example section below, this would produce:

 (Intercept) dayTue dayWed dayThu dayFri daySat daySun
           1      0      0      0      0      0      0
           1      0      0      0      0      0      0
           1      0      0      0      0      0      0
           1      0      1      0      0      0      0
           1      0      1      0      0      0      0
           1      0      0      0      1      0      0
           1      0      0      0      0      1      0
           1      0      0      0      0      1      0
           1      0      0      0      1      0      0

In some situations, there may be a need for dummy variables for all the levels of the factor. For the same example:

 dayMon dayTue dayWed dayThu dayFri daySat daySun
      1      0      0      0      0      0      0
      1      0      0      0      0      0      0
      1      0      0      0      0      0      0
      0      0      1      0      0      0      0
      0      0      1      0      0      0      0
      0      0      0      0      1      0      0
      0      0      0      0      0      1      0
      0      0      0      0      0      1      0
      0      0      0      0      1      0      0

Given a formula and initial data set, the class dummyVars gathers all the information needed to produce a full set of dummy variables for any data set. It uses contr.ltfr as the base function to do this.

class2ind is most useful for converting a factor outcome vector to a matrix (or vector) of dummy variables.

Value

The output of dummyVars is a list of class 'dummyVars' with elements

The predict function produces a data frame.

class2ind returns a matrix (or a vector if drop2nd = TRUE).

contr.ltfr generates a design matrix.

Author(s)

contr.ltfr is a small modification of contr.treatment by Max Kuhn

References

https://cran.r-project.org/doc/manuals/R-intro.html#Formulae-for-statistical-models

See Also

model.matrix, contrasts, formula

Examples

when <- data.frame(time = c("afternoon", "night", "afternoon",
                            "morning", "morning", "morning",
                            "morning", "afternoon", "afternoon"),
                   day = c("Mon", "Mon", "Mon",
                           "Wed", "Wed", "Fri",
                           "Sat", "Sat", "Fri"),
                           stringsAsFactors = TRUE)

levels(when$time) <- list(morning="morning",
                          afternoon="afternoon",
                          night="night")
levels(when$day) <- list(Mon="Mon", Tue="Tue", Wed="Wed", Thu="Thu",
                         Fri="Fri", Sat="Sat", Sun="Sun")

## Default behavior:
model.matrix(~day, when)

mainEffects <- dummyVars(~ day + time, data = when)
mainEffects
predict(mainEffects, when[1:3,])

when2 <- when
when2[1, 1] <- NA
predict(mainEffects, when2[1:3,])
predict(mainEffects, when2[1:3,], na.action = na.omit)


interactionModel <- dummyVars(~ day + time + day:time,
                              data = when,
                              sep = ".")
predict(interactionModel, when[1:3,])

noNames <- dummyVars(~ day + time + day:time,
                     data = when,
                     levelsOnly = TRUE)
predict(noNames, when)

head(class2ind(iris$Species))

two_levels <- factor(rep(letters[1:2], each = 5))
class2ind(two_levels)
class2ind(two_levels, drop2nd = TRUE)

Description

These functions can be used for a single train object or to loop through a number of train objects to calculate the training and test data predictions and class probabilities.

Usage

extractPrediction(
  models,
  testX = NULL,
  testY = NULL,
  unkX = NULL,
  unkOnly = !is.null(unkX) & is.null(testX),
  verbose = FALSE
)

extractProb(
  models,
  testX = NULL,
  testY = NULL,
  unkX = NULL,
  unkOnly = !is.null(unkX) & is.null(testX),
  verbose = FALSE
)

## S3 method for class 'train'
predict(object, newdata = NULL, type = "raw", na.action = na.omit, ...)

Arguments

Details

These functions are wrappers for the specific prediction functions in each modeling package. In each case, the optimal tuning values given in the tuneValue slot of the finalModel object are used to predict.

To get simple predictions for a new data set, the predict function can be used. Limits can be imposed on the range of predictions. See trainControl for more information.

To get predictions for a series of models at once, a list of train objects can be passes to the predict function and a list of model predictions will be returned.

The two extraction functions can be used to get the predictions and observed outcomes at once for the training, test and/or unknown samples at once in a single data frame (instead of a list of just the predictions). These objects can then be passes to plotObsVsPred or plotClassProbs.

Value

For predict.train, a vector of predictions if type = "raw" or a data frame of class probabilities for type = "prob". In the latter case, there are columns for each class.

For predict.list, a list results. Each element is produced by predict.train.

For extractPrediction, a data frame with columns:

For extractProb, a data frame. There is a column for each class containing the probabilities. The remaining columns are the same as above (although the pred column is the predicted class)

Author(s)

Max Kuhn

References

Kuhn (2008), “Building Predictive Models in R Using the caret” (doi:10.18637/jss.v028.i05)

See Also

plotObsVsPred, plotClassProbs, trainControl

Examples

## Not run: 

knnFit <- train(Species ~ ., data = iris, method = "knn",
                trControl = trainControl(method = "cv"))

rdaFit <- train(Species ~ ., data = iris, method = "rda",
                trControl = trainControl(method = "cv"))

predict(knnFit)
predict(knnFit, type = "prob")

bothModels <- list(knn = knnFit,
                   tree = rdaFit)

predict(bothModels)

extractPrediction(bothModels, testX = iris[1:10, -5])
extractProb(bothModels, testX = iris[1:10, -5])
  
## End(Not run)

Description

A shortcut to produce lattice graphs

Usage

featurePlot(
  x,
  y,
  plot = if (is.factor(y)) "strip" else "scatter",
  labels = c("Feature", ""),
  ...
)

Arguments

Details

This function “stacks” data to get it into a form compatible with lattice and creates the plots

Value

An object of class “trellis”. The ‘update’ method can be used to update components of the object and the ‘print’ method (usually called by default) will plot it on an appropriate plotting device.

Author(s)

Max Kuhn

Examples

x <- matrix(rnorm(50*5),ncol=5)
y <- factor(rep(c("A", "B"),  25))

trellis.par.set(theme = col.whitebg(), warn = FALSE)
featurePlot(x, y, "ellipse")
featurePlot(x, y, "strip", jitter = TRUE)
featurePlot(x, y, "box")
featurePlot(x, y, "pairs")

Description

Specific engines for variable importance on a model by model basis.

Usage

filterVarImp(x, y, nonpara = FALSE, ...)

Arguments

Details

The importance of each predictor is evaluated individually using a “filter” approach.

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance. For multi-class outcomes, the problem is decomposed into all pair-wise problems and the area under the curve is calculated for each class pair (i.e class 1 vs. class 2, class 2 vs. class 3 etc.). For a specific class, the maximum area under the curve across the relevant pair-wise AUC's is used as the variable importance measure.

For regression, the relationship between each predictor and the outcome is evaluated. An argument, nonpara, is used to pick the model fitting technique. When nonpara = FALSE, a linear model is fit and the absolute value of the $t$-value for the slope of the predictor is used. Otherwise, a loess smoother is fit between the outcome and the predictor. The $R^2$ statistic is calculated for this model against the intercept only null model.

Value

A data frame with variable importances. Column names depend on the problem type. For regression, the data frame contains one column: "Overall" for the importance values.

Author(s)

Max Kuhn

Examples

data(mdrr)
filterVarImp(mdrrDescr[, 1:5], mdrrClass)

data(BloodBrain)

filterVarImp(bbbDescr[, 1:5], logBBB, nonpara = FALSE)
apply(bbbDescr[, 1:5],
      2,
      function(x, y) summary(lm(y~x))$coefficients[2,3],
      y = logBBB)

filterVarImp(bbbDescr[, 1:5], logBBB, nonpara = TRUE)

Description

This function searches through a correlation matrix and returns a vector of integers corresponding to columns to remove to reduce pair-wise correlations.

Usage

findCorrelation(
  x,
  cutoff = 0.9,
  verbose = FALSE,
  names = FALSE,
  exact = ncol(x) < 100
)

Arguments

Details

The absolute values of pair-wise correlations are considered. If two variables have a high correlation, the function looks at the mean absolute correlation of each variable and removes the variable with the largest mean absolute correlation.

Using exact = TRUE will cause the function to re-evaluate the average correlations at each step while exact = FALSE uses all the correlations regardless of whether they have been eliminated or not. The exact calculations will remove a smaller number of predictors but can be much slower when the problem dimensions are "big".

There are several function in the subselect package (leaps, genetic, anneal) that can also be used to accomplish the same goal but tend to retain more predictors.

Value

A vector of indices denoting the columns to remove (when names = TRUE) otherwise a vector of column names. If no correlations meet the criteria, integer(0) is returned.

Author(s)

Original R code by Dong Li, modified by Max Kuhn

See Also

leaps, genetic, anneal, findLinearCombos

Examples

R1 <- structure(c(1, 0.86, 0.56, 0.32, 0.85, 0.86, 1, 0.01, 0.74, 0.32, 
                  0.56, 0.01, 1, 0.65, 0.91, 0.32, 0.74, 0.65, 1, 0.36,
                  0.85, 0.32, 0.91, 0.36, 1), 
                .Dim = c(5L, 5L))
colnames(R1) <- rownames(R1) <- paste0("x", 1:ncol(R1))
R1

findCorrelation(R1, cutoff = .6, exact = FALSE)
findCorrelation(R1, cutoff = .6, exact = TRUE)
findCorrelation(R1, cutoff = .6, exact = TRUE, names = FALSE)


R2 <- diag(rep(1, 5))
R2[2, 3] <- R2[3, 2] <- .7
R2[5, 3] <- R2[3, 5] <- -.7
R2[4, 1] <- R2[1, 4] <- -.67

corrDF <- expand.grid(row = 1:5, col = 1:5)
corrDF$correlation <- as.vector(R2)
levelplot(correlation ~ row + col, corrDF)

findCorrelation(R2, cutoff = .65, verbose = TRUE)

findCorrelation(R2, cutoff = .99, verbose = TRUE)

Description

Enumerate and resolve the linear combinations in a numeric matrix

Usage

findLinearCombos(x)

Arguments

Details

The QR decomposition is used to determine if the matrix is full rank and then identify the sets of columns that are involved in the dependencies.

To "resolve" them, columns are iteratively removed and the matrix rank is rechecked.

The trim.matrix function in the subselect package can also be used to accomplish the same goal.

Value

a list with elements:

Author(s)

Kirk Mettler and Jed Wing (enumLC) and Max Kuhn (findLinearCombos)

See Also

trim.matrix

Examples

testData1 <- matrix(0, nrow=20, ncol=8)
testData1[,1] <- 1
testData1[,2] <- round(rnorm(20), 1)
testData1[,3] <- round(rnorm(20), 1)
testData1[,4] <- round(rnorm(20), 1)
testData1[,5] <- 0.5 * testData1[,2] - 0.25 * testData1[,3] - 0.25 * testData1[,4]
testData1[1:4,6] <- 1
testData1[5:10,7] <- 1
testData1[11:20,8] <- 1

findLinearCombos(testData1)

testData2 <- matrix(0, nrow=6, ncol=6)
testData2[,1] <- c(1, 1, 1, 1, 1, 1)
testData2[,2] <- c(1, 1, 1, 0, 0, 0)
testData2[,3] <- c(0, 0, 0, 1, 1, 1)
testData2[,4] <- c(1, 0, 0, 1, 0, 0)
testData2[,5] <- c(0, 1, 0, 0, 1, 0)
testData2[,6] <- c(0, 0, 1, 0, 0, 1)

findLinearCombos(testData2)

Description

Return a string representing the ‘bagEarth’ expression.

Usage

## S3 method for class 'bagEarth'
format(x, file = "", cat = TRUE, ...)

Arguments

Value

A character representation of the bagged earth object.

See Also

earth

Examples

a <- bagEarth(Volume ~ ., data = trees, B= 3)
format(a)

# yields:
# (
#   31.61075 
#   +  6.587273 * pmax(0,  Girth -   14.2) 
#   -  3.229363 * pmax(0,   14.2 -  Girth) 
#   - 0.3167140 * pmax(0,     79 - Height) 
#   +
#    22.80225 
#   +  5.309866 * pmax(0,  Girth -     12) 
#   -  2.378658 * pmax(0,     12 -  Girth) 
#   +  0.793045 * pmax(0, Height -     80) 
#   - 0.3411915 * pmax(0,     80 - Height) 
#   +
#    31.39772 
#   +   6.18193 * pmax(0,  Girth -   14.2) 
#   -  3.660456 * pmax(0,   14.2 -  Girth) 
#   + 0.6489774 * pmax(0, Height -     80) 
# )/3

Description

Built-in functions related to genetic algorithms

These functions are used with the functions argument of the gafsControl function. More information on the details of these functions are at http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html.

Most of the gafs_* functions are based on those from the GA package by Luca Scrucca. These functions here are small re-writes to work outside of the GA package.

The objects caretGA, rfGA and treebagGA are example lists that can be used with the functions argument of gafsControl.

In the case of caretGA, the ... structure of gafs passes through to the model fitting routine. As a consequence, the train function can easily be accessed by passing important arguments belonging to train to gafs. See the examples below. By default, using caretGA will used the resampled performance estimates produced by train as the internal estimate of fitness.

For rfGA and treebagGA, the randomForest and bagging functions are used directly (i.e. train is not used). Arguments to either of these functions can also be passed to them though the gafs call (see examples below). For these two functions, the internal fitness is estimated using the out-of-bag estimates naturally produced by those functions. While faster, this limits the user to accuracy or Kappa (for classification) and RMSE and R-squared (for regression).

Usage

gafs_initial(vars, popSize, ...)

gafs_lrSelection(population, fitness, r = NULL, q = NULL, ...)

gafs_spCrossover(population, fitness, parents, ...)

gafs_raMutation(population, parent, ...)

gafs_nlrSelection(population, fitness, q = 0.25, ...)

gafs_rwSelection(population, fitness, ...)

gafs_tourSelection(population, fitness, k = 3, ...)

gafs_uCrossover(population, parents, ...)

Arguments

Value

The return value depends on the function.

Author(s)

Luca Scrucca, gafs_initial, caretGA, rfGA and treebagGA by Max Kuhn

References

Scrucca L (2013). GA: A Package for Genetic Algorithms in R. Journal of Statistical Software, 53(4), 1-37.

https://cran.r-project.org/package=GA

http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html

See Also

gafs, gafsControl

Examples

pop <- gafs_initial(vars = 10, popSize = 10)
pop

gafs_lrSelection(population = pop, fitness = 1:10)

gafs_spCrossover(population = pop, fitness = 1:10, parents = 1:2)


## Not run: 
## Hypothetical examples
lda_ga <- gafs(x = predictors,
               y = classes,
               gafsControl = gafsControl(functions = caretGA),
               ## now pass arguments to `train`
               method = "lda",
               metric = "Accuracy"
               trControl = trainControl(method = "cv", classProbs = TRUE))

rf_ga <- gafs(x = predictors,
              y = classes,
              gafsControl = gafsControl(functions = rfGA),
              ## these are arguments to `randomForest`
              ntree = 1000,
              importance = TRUE)
	
## End(Not run)

Description

Supervised feature selection using genetic algorithms

Usage

## Default S3 method:
gafs(
  x,
  y,
  iters = 10,
  popSize = 50,
  pcrossover = 0.8,
  pmutation = 0.1,
  elite = 0,
  suggestions = NULL,
  differences = TRUE,
  gafsControl = gafsControl(),
  ...
)

## S3 method for class 'recipe'
gafs(
  x,
  data,
  iters = 10,
  popSize = 50,
  pcrossover = 0.8,
  pmutation = 0.1,
  elite = 0,
  suggestions = NULL,
  differences = TRUE,
  gafsControl = gafsControl(),
  ...
)

Arguments

Details

gafs conducts a supervised binary search of the predictor space using a genetic algorithm. See Mitchell (1996) and Scrucca (2013) for more details on genetic algorithms.

This function conducts the search of the feature space repeatedly within resampling iterations. First, the training data are split be whatever resampling method was specified in the control function. For example, if 10-fold cross-validation is selected, the entire genetic algorithm is conducted 10 separate times. For the first fold, nine tenths of the data are used in the search while the remaining tenth is used to estimate the external performance since these data points were not used in the search.

During the genetic algorithm, a measure of fitness is needed to guide the search. This is the internal measure of performance. During the search, the data that are available are the instances selected by the top-level resampling (e.g. the nine tenths mentioned above). A common approach is to conduct another resampling procedure. Another option is to use a holdout set of samples to determine the internal estimate of performance (see the holdout argument of the control function). While this is faster, it is more likely to cause overfitting of the features and should only be used when a large amount of training data are available. Yet another idea is to use a penalized metric (such as the AIC statistic) but this may not exist for some metrics (e.g. the area under the ROC curve).

The internal estimates of performance will eventually overfit the subsets to the data. However, since the external estimate is not used by the search, it is able to make better assessments of overfitting. After resampling, this function determines the optimal number of generations for the GA.

Finally, the entire data set is used in the last execution of the genetic algorithm search and the final model is built on the predictor subset that is associated with the optimal number of generations determined by resampling (although the update function can be used to manually set the number of generations).

This is an example of the output produced when gafsControl(verbose = TRUE) is used:

Fold2 1 0.715 (13)
Fold2 2 0.715->0.737 (13->17, 30.4%) *
Fold2 3 0.737->0.732 (17->14, 24.0%)
Fold2 4 0.737->0.769 (17->23, 25.0%) *

For the second resample (e.g. fold 2), the best subset across all individuals tested in the first generation contained 13 predictors and was associated with a fitness value of 0.715. The second generation produced a better subset containing 17 samples with an associated fitness values of 0.737 (and improvement is symbolized by the *. The percentage listed is the Jaccard similarity between the previous best individual (with 13 predictors) and the new best. The third generation did not produce a better fitness value but the fourth generation did.

The search algorithm can be parallelized in several places:

1. each externally resampled GA can be run independently (controlled by the allowParallel option of gafsControl)

2. within a GA, the fitness calculations at a particular generation can be run in parallel over the current set of individuals (see the genParallel option in gafsControl)

3. if inner resampling is used, these can be run in parallel (controls depend on the function used. See, for example, trainControl)

4. any parallelization of the individual model fits. This is also specific to the modeling function.

It is probably best to pick one of these areas for parallelization and the first is likely to produces the largest decrease in run-time since it is the least likely to incur multiple re-starting of the worker processes. Keep in mind that if multiple levels of parallelization occur, this can effect the number of workers and the amount of memory required exponentially.

Value

an object of class gafs

Author(s)

Max Kuhn, Luca Scrucca (for GA internals)

References

Kuhn M and Johnson K (2013), Applied Predictive Modeling, Springer, Chapter 19 http://appliedpredictivemodeling.com

Scrucca L (2013). GA: A Package for Genetic Algorithms in R. Journal of Statistical Software, 53(4), 1-37. https://www.jstatsoft.org/article/view/v053i04

Mitchell M (1996), An Introduction to Genetic Algorithms, MIT Press.

https://en.wikipedia.org/wiki/Jaccard_index

See Also

gafsControl, predict.gafs, caretGA, rfGA treebagGA

Examples

## Not run: 
set.seed(1)
train_data <- twoClassSim(100, noiseVars = 10)
test_data  <- twoClassSim(10,  noiseVars = 10)

## A short example
ctrl <- gafsControl(functions = rfGA,
                    method = "cv",
                    number = 3)

rf_search <- gafs(x = train_data[, -ncol(train_data)],
                  y = train_data$Class,
                  iters = 3,
                  gafsControl = ctrl)

rf_search
  
## End(Not run)

Description

Control the computational nuances of the gafs and safs functions

Many of these options are the same as those described for trainControl. More extensive documentation and examples can be found on the caret website at http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html#syntax and http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html#syntax.

The functions component contains the information about how the model should be fit and summarized. It also contains the elements needed for the GA and SA modules (e.g. cross-over, etc).

The elements of functions that are the same for GAs and SAs are:

• fit, with arguments x, y, lev, last, and ..., is used to fit the classification or regression model

• pred, with arguments object and x, predicts new samples

• fitness_intern, with arguments object, x, y, maximize, and p, summarizes performance for the internal estimates of fitness

• fitness_extern, with arguments data, lev, and model, summarizes performance using the externally held-out samples

• selectIter, with arguments x, metric, and maximize, determines the best search iteration for feature selection.

The elements of functions specific to genetic algorithms are:

• initial, with arguments vars, popSize and ..., creates an initial population.

• selection, with arguments population, fitness, r, q, and ..., conducts selection of individuals.

• crossover, with arguments population, fitness, parents and ..., control genetic reproduction.

• mutation, with arguments population, parent and ..., adds mutations.

The elements of functions specific to simulated annealing are:

• initial, with arguments vars, prob, and ..., creates the initial subset.

• perturb, with arguments x, vars, and number, makes incremental changes to the subsets.

• prob, with arguments old, new, and iteration, computes the acceptance probabilities

The pages http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html and http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html have more details about each of these functions.

holdout can be used to hold out samples for computing the internal fitness value. Note that this is independent of the external resampling step. Suppose 10-fold CV is being used. Within a resampling iteration, holdout can be used to sample an additional proportion of the 90% resampled data to use for estimating fitness. This may not be a good idea unless you have a very large training set and want to avoid an internal resampling procedure to estimate fitness.

The search algorithms can be parallelized in several places:

1. each externally resampled GA or SA can be run independently (controlled by the allowParallel options)

2. within a GA, the fitness calculations at a particular generation can be run in parallel over the current set of individuals (see the genParallel)

3. if inner resampling is used, these can be run in parallel (controls depend on the function used. See, for example, trainControl)

4. any parallelization of the individual model fits. This is also specific to the modeling function.

It is probably best to pick one of these areas for parallelization and the first is likely to produces the largest decrease in run-time since it is the least likely to incur multiple re-starting of the worker processes. Keep in mind that if multiple levels of parallelization occur, this can effect the number of workers and the amount of memory required exponentially.

Usage

gafsControl(
  functions = NULL,
  method = "repeatedcv",
  metric = NULL,
  maximize = NULL,
  number = ifelse(grepl("cv", method), 10, 25),
  repeats = ifelse(grepl("cv", method), 1, 5),
  verbose = FALSE,
  returnResamp = "final",
  p = 0.75,
  index = NULL,
  indexOut = NULL,
  seeds = NULL,
  holdout = 0,
  genParallel = FALSE,
  allowParallel = TRUE
)

safsControl(
  functions = NULL,
  method = "repeatedcv",
  metric = NULL,
  maximize = NULL,
  number = ifelse(grepl("cv", method), 10, 25),
  repeats = ifelse(grepl("cv", method), 1, 5),
  verbose = FALSE,
  returnResamp = "final",
  p = 0.75,
  index = NULL,
  indexOut = NULL,
  seeds = NULL,
  holdout = 0,
  improve = Inf,
  allowParallel = TRUE
)

Arguments

Value

An echo of the parameters specified

Author(s)

Max Kuhn

References

http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html, http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html

See Also

safs, safs, , caretGA, rfGA, treebagGA, caretSA, rfSA, treebagSA

Description

Data from Dr. Hans Hofmann of the University of Hamburg.

Details

These data have two classes for the credit worthiness: good or bad. There are predictors related to attributes, such as: checking account status, duration, credit history, purpose of the loan, amount of the loan, savings accounts or bonds, employment duration, Installment rate in percentage of disposable income, personal information, other debtors/guarantors, residence duration, property, age, other installment plans, housing, number of existing credits, job information, Number of people being liable to provide maintenance for, telephone, and foreign worker status.

Many of these predictors are discrete and have been expanded into several 0/1 indicator variables

Source

UCI Machine Learning Repository

Description

Placeholder.

Usage

getSamplingInfo(method = NULL, regex = TRUE, ...)

Arguments

Details

Placeholder.

Value

A list

Description

These functions plot the resampling results for the candidate subset sizes evaluated during the recursive feature elimination (RFE) process

Usage

## S3 method for class 'rfe'
ggplot(
  data = NULL,
  mapping = NULL,
  metric = data$metric[1],
  output = "layered",
  ...,
  environment = NULL
)

## S3 method for class 'rfe'
plot(x, metric = x$metric, ...)

Arguments

Details

These plots show the average performance versus the subset sizes.

Value

a lattice or ggplot object

Note

We using a recipe as an input, there may be some subset sizes that are not well-replicated over resamples. The 'ggplot' method will only show subset sizes where at least half of the resamples have associated results.

Author(s)

Max Kuhn

References

Kuhn (2008), “Building Predictive Models in R Using the caret” (doi:10.18637/jss.v028.i05)

See Also

rfe, xyplot, ggplot

Examples

## Not run: 
data(BloodBrain)

x <- scale(bbbDescr[,-nearZeroVar(bbbDescr)])
x <- x[, -findCorrelation(cor(x), .8)]
x <- as.data.frame(x, stringsAsFactors = TRUE)

set.seed(1)
lmProfile <- rfe(x, logBBB,
                 sizes = c(2:25, 30, 35, 40, 45, 50, 55, 60, 65),
                 rfeControl = rfeControl(functions = lmFuncs,
                                         number = 200))
plot(lmProfile)
plot(lmProfile, metric = "Rsquared")
ggplot(lmProfile)

## End(Not run)

Description

This function takes the output of a train object and creates a line or level plot using the lattice or ggplot2 libraries.

Usage

## S3 method for class 'train'
ggplot(
  data = NULL,
  mapping = NULL,
  metric = data$metric[1],
  plotType = "scatter",
  output = "layered",
  nameInStrip = FALSE,
  highlight = FALSE,
  ...,
  environment = NULL
)

## S3 method for class 'train'
plot(
  x,
  plotType = "scatter",
  metric = x$metric[1],
  digits = getOption("digits") - 3,
  xTrans = NULL,
  nameInStrip = FALSE,
  ...
)

Arguments

Details

If there are no tuning parameters, or none were varied, an error is produced.

If the model has one tuning parameter with multiple candidate values, a plot is produced showing the profile of the results over the parameter. Also, a plot can be produced if there are multiple tuning parameters but only one is varied.

If there are two tuning parameters with different values, a plot can be produced where a different line is shown for each value of of the other parameter. For three parameters, the same line plot is created within conditioning panels/facets of the other parameter.

Also, with two tuning parameters (with different values), a levelplot (i.e. un-clustered heatmap) can be created. For more than two parameters, this plot is created inside conditioning panels/facets.

Author(s)

Max Kuhn

References

Kuhn (2008), “Building Predictive Models in R Using the caret” (doi:10.18637/jss.v028.i05)

See Also

train, levelplot, xyplot, stripplot, ggplot

Examples

## Not run: 
library(klaR)
rdaFit <- train(Species ~ .,
                data = iris,
                method = "rda",
                control = trainControl(method = "cv"))
plot(rdaFit)
plot(rdaFit, plotType = "level")

ggplot(rdaFit) + theme_bw()


## End(Not run)

Description

A set of lattice functions are provided to plot the resampled performance estimates (e.g. classification accuracy, RMSE) over tuning parameters (if any).

Usage

## S3 method for class 'train'
histogram(x, data = NULL, metric = x$metric, ...)

Arguments

Details

By default, only the resampling results for the optimal model are saved in the train object. The function trainControl can be used to save all the results (see the example below).

If leave-one-out or out-of-bag resampling was specified, plots cannot be produced (see the method argument of trainControl)

For xyplot and stripplot, the tuning parameter with the most unique values will be plotted on the x-axis. The remaining parameters (if any) will be used as conditioning variables. For densityplot and histogram, all tuning parameters are used for conditioning.

Using horizontal = FALSE in stripplot works.

Value

A lattice plot object

Author(s)

Max Kuhn

See Also

train, trainControl, histogram, densityplot, xyplot, stripplot

Examples

## Not run: 

library(mlbench)
data(BostonHousing)

library(rpart)
rpartFit <- train(medv ~ .,
                  data = BostonHousing,
                  "rpart", 
                  tuneLength = 9,
                  trControl = trainControl(
                    method = "boot", 
                    returnResamp = "all"))

densityplot(rpartFit,
            adjust = 1.25)

xyplot(rpartFit,
       metric = "Rsquared",
       type = c("p", "a"))

stripplot(rpartFit,
          horizontal = FALSE,
          jitter = TRUE)


## End(Not run)

Description

Fit a linear regression model using independent components

Usage

## S3 method for class 'formula'
icr(formula, data, weights, ..., subset, na.action, contrasts = NULL)

## Default S3 method:
icr(x, y, ...)

## S3 method for class 'icr'
predict(object, newdata, ...)

Arguments

Details

This produces a model analogous to Principal Components Regression (PCR) but uses Independent Component Analysis (ICA) to produce the scores. The user must specify a value of n.comp to pass to fastICA.

The function preProcess to produce the ICA scores for the original data and for newdata.

Value

For icr, a list with elements

Author(s)

Max Kuhn

See Also

fastICA, preProcess, lm

Examples

data(BloodBrain)

icrFit <- icr(bbbDescr, logBBB, n.comp = 5)

icrFit

predict(icrFit, bbbDescr[1:5,])

Description