Package 'caret'

Description

Conversion functions for class confusionMatrix

Usage

## S3 method for class 'confusionMatrix'
as.matrix(x, what = "xtabs", ...)

Arguments

Details

For as.table, the cross-tabulations are saved. For as.matrix, the three object types are saved in matrix format.

Value

A matrix or table

Author(s)

Max Kuhn

Examples

###################
## 2 class example

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)), levels = rev(lvs))
pred <- factor(
  c(
    rep(lvs, times = c(54, 32)),
    rep(lvs, times = c(27, 231))
  ),
  levels = rev(lvs)
)

xtab <- table(pred, truth)

results <- confusionMatrix(xtab)
as.table(results)
as.matrix(results)
as.matrix(results, what = "overall")
as.matrix(results, what = "classes")

###################
## 3 class example

xtab <- confusionMatrix(iris$Species, sample(iris$Species))
as.matrix(xtab)

Description

Aggregate several neural network models

Usage

avNNet(x, ...)

## S3 method for class 'formula'
avNNet(
  formula,
  data,
  weights,
  ...,
  repeats = 5,
  bag = FALSE,
  allowParallel = TRUE,
  seeds = sample.int(1e+05, repeats),
  subset,
  na.action,
  contrasts = NULL
)

## Default S3 method:
avNNet(
  x,
  y,
  repeats = 5,
  bag = FALSE,
  allowParallel = TRUE,
  seeds = sample.int(1e+05, repeats),
  ...
)

## S3 method for class 'avNNet'
print(x, ...)

## S3 method for class 'avNNet'
predict(object, newdata, type = c("raw", "class", "prob"), ...)

Arguments

Details

Following Ripley (1996), the same neural network model is fit using different random number seeds. All the resulting models are used for prediction. For regression, the output from each network are averaged. For classification, the model scores are first averaged, then translated to predicted classes. Bagging can also be used to create the models.

If a parallel backend is registered, the foreach package is used to train the networks in parallel.

Value

For avNNet, an object of "avNNet" or "avNNet.formula". Items of interest in the output are:

• model: a list of the models generated from nnet::nnet()

• repeats: an echo of the model input

• names: if any predictors had only one distinct value, this is a character string of the remaining columns. Otherwise a value of NULL

Author(s)

These are heavily based on the nnet code from Brian Ripley.

References

Ripley, B. D. (1996) Pattern Recognition and Neural Networks. Cambridge.

See Also

nnet::nnet(), preProcess()

Other preprocessing: BoxCoxTrans(), dummyVars(), findCorrelation(), nearZeroVar(), pcaNNet(), preProcess(), spatialSign()

Examples

data(BloodBrain)
modelFit <- avNNet(bbbDescr, logBBB, size = 5, linout = TRUE, trace = FALSE)
modelFit

predict(modelFit, bbbDescr)

Description

bag provides a framework for bagging classification or regression models. The user can provide their own functions for model building, prediction and aggregation of predictions (see Details below).

Usage

bag(x, ...)

bagControl(
  fit = NULL,
  predict = NULL,
  aggregate = NULL,
  downSample = FALSE,
  oob = TRUE,
  allowParallel = TRUE
)

## Default S3 method:
bag(x, y, B = 10, vars = ncol(x), bagControl = NULL, ...)

## S3 method for class 'bag'
predict(object, newdata = NULL, ...)

## S3 method for class 'bag'
print(x, ...)

## S3 method for class 'bag'
summary(object, ...)

## S3 method for class 'summary.bag'
print(x, digits = max(3, getOption("digits") - 3), ...)

ldaBag

plsBag

nbBag

ctreeBag

svmBag

nnetBag

Arguments

Details

The function is basically a framework where users can plug in any model in to assess the effect of bagging. Examples functions can be found in ldaBag, plsBag , nbBag, svmBag and nnetBag. Each has elements fit, pred and aggregate.

One note: when vars is not NULL, the sub-setting occurs prior to the fit and predict functions are called. In this way, the user probably does not need to account for the change in predictors in their functions.

When using bag with train(), classification models should use type = "prob" inside of the predict function so that predict.train(object, newdata, type = "prob") will work.

If a parallel backend is registered, the foreach package is used to train the models in parallel.

Value

bag produces an object of class bag with elements:

• fits: a list with two sub-objects: the fit object has the actual model fit for that bagged samples and the vars object is either NULL or a vector of integers corresponding to which predictors were sampled for that model

• control: a mirror of the arguments passed into bagControl

• call: the call

• B: the number of bagging iterations

• dims: the dimensions of the training set

Author(s)

Max Kuhn

Examples

## A simple example of bagging conditional inference regression trees:
data(BloodBrain)

treebag <- bag(
  bbbDescr,
  logBBB,
  B = 10,
  bagControl = bagControl(
    fit = ctreeBag$fit,
    predict = ctreeBag$pred,
    aggregate = ctreeBag$aggregate
  )
)

## An example of pooling posterior probabilities to generate class predictions
data(mdrr)

## remove some zero variance predictors and linear dependencies
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .95)]

basicLDA <- train(mdrrDescr, mdrrClass, "lda")

bagLDA2 <- train(
  mdrrDescr,
  mdrrClass,
  "bag",
  B = 10,
  bagControl = bagControl(
    fit = ldaBag$fit,
    predict = ldaBag$pred,
    aggregate = ldaBag$aggregate
  ),
  tuneGrid = data.frame(vars = c((1:10) * 10, ncol(mdrrDescr)))
)

Description

A bagging wrapper for multivariate adaptive regression splines (MARS) via the earth function

Usage

bagEarth(x, ...)

## Default S3 method:
bagEarth(x, y, weights = NULL, B = 50, summary = mean, keepX = TRUE, ...)

## S3 method for class 'formula'
bagEarth(
  formula,
  data = NULL,
  B = 50,
  summary = mean,
  keepX = TRUE,
  ...,
  subset,
  weights = NULL,
  na.action = na.omit
)

## S3 method for class 'bagEarth'
print(x, ...)

Arguments

Details

The function computes a Earth model for each bootstap sample.

Value

A list with elements:

• fit: a list of B Earth fits

• B: the number of bootstrap samples

• call: the function call

• x: either NULL or the value of x, depending on the value of keepX

• oob: a matrix of performance estimates for each bootstrap sample

Author(s)

Max Kuhn (bagEarth.formula is based on Ripley's nnet.formula)

References

J. Friedman, “Multivariate Adaptive Regression Splines” (with discussion) (1991). Annals of Statistics, 19/1, 1-141.

See Also

earth::earth(), predict.bagEarth()

Examples

library(mda)
library(earth)
data(trees)
fit1 <- earth(x = trees[, -3], y = trees[, 3])
set.seed(2189)
fit2 <- bagEarth(x = trees[, -3], y = trees[, 3], B = 10)

Description

A bagging wrapper for flexible discriminant analysis (FDA) using multivariate adaptive regression splines (MARS) basis functions

Usage

bagFDA(x, ...)

## Default S3 method:
bagFDA(x, y, weights = NULL, B = 50, keepX = TRUE, ...)

## S3 method for class 'formula'
bagFDA(
  formula,
  data = NULL,
  B = 50,
  keepX = TRUE,
  ...,
  subset,
  weights = NULL,
  na.action = na.omit
)

## S3 method for class 'bagFDA'
print(x, ...)

Arguments

Details

The function computes a FDA model for each bootstap sample.

Value

A list with elements:

• fit: a list of B FDA fits

• B: the number of bootstrap samples

• call: the function call

• x: either NULL or the value of x, depending on the value of keepX

• oob: a matrix of performance estimates for each bootstrap sample

Author(s)

Max Kuhn (bagFDA.formula is based on Ripley's nnet.formula)

References

J. Friedman, “Multivariate Adaptive Regression Splines” (with discussion) (1991). Annals of Statistics, 19/1, 1-141.

See Also

mda::fda(), predict.bagFDA()

Examples

library(mlbench)
library(earth)
data(Glass)

set.seed(36)
inTrain <- sample(1:dim(Glass)[1], 150)

trainData <- Glass[inTrain, ]
testData <- Glass[-inTrain, ]

set.seed(3577)
baggedFit <- bagFDA(Type ~ ., trainData)
confusionMatrix(
  data = predict(baggedFit, testData[, -10]),
  reference = testData[, 10]
)

Description

Mente and Lombardo (2005) develop models to predict the log of the ratio of the concentration of a compound in the brain and the concentration in blood. For each compound, they computed three sets of molecular descriptors: MOE 2D, rule-of-five and Charge Polar Surface Area (CPSA). In all, 134 descriptors were calculated. Included in this package are 208 non-proprietary literature compounds. The vector logBBB contains the concentration ratio and the data fame bbbDescr contains the descriptor values.

Value

:

• bbbDescr: data frame of chemical descriptors

• logBBB: vector of assay results

Source

Mente, S.R. and Lombardo, F. (2005). A recursive-partitioning model for blood-brain barrier permeation, Journal of Computer-Aided Molecular Design, Vol. 19, pg. 465-481.

Description

These classes can be used to estimate transformations and apply them to existing and future data

Usage

BoxCoxTrans(y, ...)

## Default S3 method:
BoxCoxTrans(
  y,
  x = rep(1, length(y)),
  fudge = 0.2,
  numUnique = 3,
  na.rm = FALSE,
  ...
)

## S3 method for class 'BoxCoxTrans'
print(x, newdata, digits = 3, ...)

## S3 method for class 'BoxCoxTrans'
predict(object, newdata, ...)

Arguments

Details

BoxCoxTrans function is basically a wrapper for the MASS::boxcox() function in the MASS library. It can be used to estimate the transformation and apply it to new data.

expoTrans estimates the exponential transformation of Manly (1976) but assumes a common mean for the data. The transformation parameter is estimated by directly maximizing the likelihood.

If any(y <= 0) or if length(unique(y)) < numUnique, lambda is not estimated and no transformation is applied.

Value

Both functions returns a list of class of either BoxCoxTrans or expoTrans with elements

• lambda: estimated transformation value

• fudge: value of fudge

• n: number of data points used to estimate lambda

• summary: the results of summary(y)

• ratio: max(y)/min(y)

• skewness: sample skewness statistic

• fudge: value of fudge

The predict functions returns numeric vectors of transformed values

Author(s)

Max Author

References

Box, G. E. P. and Cox, D. R. (1964) An analysis of transformations (with discussion). Journal of the Royal Statistical Society B, 26, 211-252. Manly, B. L. (1976) Exponential data transformations. The Statistician, 25, 37 - 42.

See Also

MASS::boxcox(), preProcess(), optim()

Other preprocessing: avNNet(), dummyVars(), findCorrelation(), nearZeroVar(), pcaNNet(), preProcess(), spatialSign()

Examples

data(BloodBrain)

ratio <- exp(logBBB)
bc <- BoxCoxTrans(ratio)
bc

predict(bc, ratio[1:5])

ratio[5] <- NA
bc2 <- BoxCoxTrans(ratio, bbbDescr$tpsa, na.rm = TRUE)
bc2

manly <- expoTrans(ratio)
manly

Description

For classification models, this function creates a 'calibration plot' that describes how consistent model probabilities are with observed event rates.

Usage

calibration(x, ...)

## Default S3 method:
calibration(x, ...)

## S3 method for class 'formula'
calibration(
  x,
  data = NULL,
  class = NULL,
  cuts = 11,
  subset = TRUE,
  lattice.options = NULL,
  ...
)

## S3 method for class 'calibration'
print(x, ...)

## S3 method for class 'calibration'
xyplot(x, data = NULL, ...)

## S3 method for class 'calibration'
ggplot(data, ..., bwidth = 2, dwidth = 3)

Arguments

Details

calibration.formula is used to process the data and xyplot.calibration is used to create the plot.

To construct the calibration plot, the following steps are used for each model:

1. The data are split into cuts - 1 roughly equal groups by their class probabilities

2. the number of samples with true results equal to class are determined

3. the event rate is determined for each bin

xyplot.calibration produces a plot of the observed event rate by the mid-point of the bins.

This implementation uses the lattice function lattice::xyplot(), so plot elements can be changed via panel functions, lattice::trellis.par.set() or other means. calibration uses the panel function panel.calibration() by default, but it can be changed by passing that argument into xyplot.calibration.

The following elements are set by default in the plot but can be changed by passing new values into xyplot.calibration: xlab = "Bin Midpoint", ylab = "Observed Event Percentage", type = "o", ylim = extendrange(c(0, 100)),xlim = extendrange(c(0, 100)) and panel = panel.calibration

For the ggplot method, confidence intervals on the estimated proportions (from stats::binom.test()) are also shown.

Value

calibration.formula returns a list with elements:

• data: the data used for plotting

• cuts: the number of cuts

• class: the event class

• probNames: the names of the model probabilities

xyplot.calibration returns a lattice object

Author(s)

Max Kuhn, some lattice code and documentation by Deepayan Sarkar

See Also

lattice::xyplot(), lattice::trellis.par.set()

Examples

data(mdrr)
mdrrDescr <- mdrrDescr[, -nearZeroVar(mdrrDescr)]
mdrrDescr <- mdrrDescr[, -findCorrelation(cor(mdrrDescr), .5)]

inTrain <- createDataPartition(mdrrClass)
trainX <- mdrrDescr[inTrain[[1]], ]
trainY <- mdrrClass[inTrain[[1]]]
testX <- mdrrDescr[-inTrain[[1]], ]
testY <- mdrrClass[-inTrain[[1]]]

library(MASS)

ldaFit <- lda(trainX, trainY)

testProbs <- data.frame(
  obs = testY,
  lda = predict(ldaFit, testX)$posterior[, 1]
)

calibration(obs ~ lda, data = testProbs)

calPlotData <- calibration(obs ~ lda, data = testProbs)
calPlotData

xyplot(calPlotData, auto.key = list(columns = 2))

Description

Ancillary functions for univariate feature selection

Usage

caretSBF

anovaScores(x, y)

gamScores(x, y)

Arguments

Details

More details on these functions can be found at http://topepo.github.io/caret/feature-selection-using-univariate-filters.html.

This page documents the functions that are used in selection by filtering (SBF). The functions described here are passed to the algorithm via the functions argument of sbfControl().

See sbfControl() for details on how these functions should be defined.

anovaScores and gamScores are two examples of univariate filtering functions. anovaScores fits a simple linear model between a single feature and the outcome, then the p-value for the whole model F-test is returned. gamScores fits a generalized additive model between a single predictor and the outcome using a smoothing spline basis function. A p-value is generated using the whole model test from gam::summary.Gam() and is returned.

If a particular model fails for lm or gam, a p-value of 1 is returned.

Author(s)

Max Kuhn

See Also

sbfControl(), sbf(), gam::summary.Gam()

Description

Kuiper (2008) collected data on Kelly Blue Book resale data for 804 GM cars (2005 model year).

Value

:

• cars: data frame of the suggested retail price (column Price) and various characteristics of each car (columns Mileage, Cylinder, Doors, Cruise, Sound, Leather, Buick, Cadillac, Chevy, Pontiac, Saab, Saturn, convertible, coupe, hatchback, sedan and wagon)

Source

Kuiper, S. (2008). Introduction to Multiple Regression: How Much Is Your Car Worth?, Journal of Statistics Education, Vol. 16 http://jse.amstat.org/jse_archive.htm#2008.

Description

This function computes the class centroids and covariance matrix for a training set for determining Mahalanobis distances of samples to each class centroid.

Usage

classDist(x, ...)

## Default S3 method:
classDist(x, y, groups = 5, pca = FALSE, keep = NULL, ...)

## S3 method for class 'classDist'
predict(object, newdata, trans = log, ...)

Arguments

Details

For factor outcomes, the data are split into groups for each class and the mean and covariance matrix are calculated. These are then used to compute Mahalanobis distances to the class centers (using predict.classDist The function will check for non-singular matrices.

For numeric outcomes, the data are split into roughly equal sized bins based on groups. Percentiles are used to split the data.

Value

for classDist, an object of class classDist with elements:

• values: a list with elements for each class. Each element contains a mean vector for the class centroid and the inverse of the class covariance matrix

• classes: a character vector of class labels

• pca: the results of stats::prcomp() when pca = TRUE

• call: the function call

• p: the number of variables

• n: a vector of samples sizes per class

For predict.classDist, a matrix with columns for each class. The columns names are the names of the class with the prefix dist.. In the case of numeric y, the class labels are the percentiles. For example, of groups = 9, the variable names would be dist.11.11, dist.22.22, etc.

Author(s)

Max Kuhn

References

Forina et al. CAIMAN brothers: A family of powerful classification and class modeling techniques. Chemometrics and Intelligent Laboratory Systems (2009) vol. 96 (2) pp. 239-245

See Also

stats::mahalanobis()

Examples

trainSet <- sample(1:150, 100)

distData <- classDist(iris[trainSet, 1:4], iris$Species[trainSet])

newDist <- predict(distData, iris[-trainSet, 1:4])

splom(newDist, groups = iris$Species[-trainSet])

Description

Calculates a cross-tabulation of observed and predicted classes with associated statistics.

Usage

confusionMatrix(data, ...)

## Default S3 method:
confusionMatrix(
  data,
  reference,
  positive = NULL,
  dnn = c("Prediction", "Reference"),
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

## S3 method for class 'matrix'
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

## S3 method for class 'table'
confusionMatrix(
  data,
  positive = NULL,
  prevalence = NULL,
  mode = "sens_spec",
  ...
)

Arguments

Details

The functions requires that the factors have exactly the same levels.

For two class problems, the sensitivity, specificity, positive predictive value and negative predictive value is calculated using the positive argument. Also, the prevalence of the "event" is computed from the data (unless passed in as an argument), the detection rate (the rate of true events also predicted to be events) and the detection prevalence (the prevalence of predicted events).

Suppose a 2x2 table with notation

The formulas used here are:

$Sensitivity = A/(A+C)$

$Specificity = D/(B+D)$

$Prevalence = (A+C)/(A+B+C+D)$

$PPV = (sensitivity * prevalence)/((sensitivity*prevalence) + ((1-specificity)*(1-prevalence)))$

$NPV = (specificity * (1-prevalence))/(((1-sensitivity)*prevalence) + ((specificity)*(1-prevalence)))$

$Detection Rate = A/(A+B+C+D)$

$Detection Prevalence = (A+B)/(A+B+C+D)$

$Balanced Accuracy = (sensitivity+specificity)/2$

$Precision = A/(A+B)$

$Recall = A/(A+C)$

$F1 = (1+beta^2)*precision*recall/((beta^2 * precision)+recall)$

where beta = 1 for this function.

See the references for discussions of the first five formulas.

For more than two classes, these results are calculated comparing each factor level to the remaining levels (i.e. a "one versus all" approach).

The overall accuracy and unweighted Kappa statistic are calculated. A p-value from McNemar's test is also computed using stats::mcnemar.test() (which can produce NA values with sparse tables).

The overall accuracy rate is computed along with a 95 percent confidence interval for this rate (using stats::binom.test()) and a one-sided test to see if the accuracy is better than the "no information rate," which is taken to be the largest class percentage in the data.

Value

a list with elements:

• table: the results of table on data and reference

• positive: the positive result level

• overall: a numeric vector with overall accuracy and Kappa statistic values

• byClass: the sensitivity, specificity, positive predictive value, negative predictive value, precision, recall, F1, prevalence, detection rate, detection prevalence and balanced accuracy for each class. For two class systems, this is calculated once using the positive argument

Note

If the reference and data factors have the same levels, but in the incorrect order, the function will reorder them to the order of the data and issue a warning.

Author(s)

Max Kuhn

References

Kuhn, M. (2008), "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, 28(5), 1-26. (doi:10.18637/jss.v028.i05).

Altman, D.G., Bland, J.M. (1994) "Diagnostic tests 1: sensitivity and specificity," British Medical Journal, vol 308, 1552.

Altman, D.G., Bland, J.M. (1994) "Diagnostic tests 2: predictive values," British Medical Journal, vol 309, 102.

Velez, D.R., et. al. (2008) "A balanced accuracy function for epistasis modeling in imbalanced datasets using multifactor dimensionality reduction.," Genetic Epidemiology, vol 4, 306.

See Also

as.table.confusionMatrix(), as.matrix.confusionMatrix(), sensitivity(), specificity(), posPredValue(), negPredValue(), print.confusionMatrix(), stats::binom.test()

Other performance: defaultSummary(), negPredValue(), recall()

Examples

###################
## 2 class example

lvs <- c("normal", "abnormal")
truth <- factor(rep(lvs, times = c(86, 258)), levels = rev(lvs))
pred <- factor(
  c(
    rep(lvs, times = c(54, 32)),
    rep(lvs, times = c(27, 231))
  ),
  levels = rev(lvs)
)

xtab <- table(pred, truth)

confusionMatrix(xtab)
confusionMatrix(pred, truth)
confusionMatrix(xtab, prevalence = 0.25)

###################
## 3 class example

confusionMatrix(iris$Species, sample(iris$Species))

newPrior <- c(.05, .8, .15)
names(newPrior) <- levels(iris$Species)

confusionMatrix(iris$Species, sample(iris$Species))

Description

Using a train(), rfe(), sbf() object, determine a confusion matrix based on the resampling procedure

Usage

## S3 method for class 'train'
confusionMatrix(
  data,
  norm = "overall",
  dnn = c("Prediction", "Reference"),
  ...
)

Arguments

Details

When train() is used for tuning a model, it tracks the confusion matrix cell entries for the hold-out samples. These can be aggregated and used for diagnostic purposes. For train(), the matrix is estimated for the final model tuning parameters determined by train(). For rfe(), the matrix is associated with the optimal number of variables.

There are several ways to show the table entries. Using norm = "none" will show the aggregated counts of samples on each of the cells (across all resamples). For norm = "average", the average number of cell counts across resamples is computed (this can help evaluate how many holdout samples there were on average). The default is norm = "overall", which is equivalento to "average" but in percentages.

Value

a list of class confusionMatrix.train, confusionMatrix.rfe or confusionMatrix.sbf with elements

• table: the normalized matrix

• norm: an echo fo the call

• text: a character string with details about the resampling procedure (e.g. "Bootstrapped (25 reps) Confusion Matrix"

Author(s)

Max Kuhn

See Also

confusionMatrix(), train(), rfe(), sbf(), trainControl()

Examples

data(iris)
TrainData <- iris[, 1:4]
TrainClasses <- iris[, 5]

knnFit <- train(
  TrainData,
  TrainClasses,
  method = "knn",
  preProcess = c("center", "scale"),
  tuneLength = 10,
  trControl = trainControl(method = "cv")
)
confusionMatrix(knnFit)
confusionMatrix(knnFit, "average")
confusionMatrix(knnFit, "none")

Description

From Sutherland, O'Brien, and Weaver (2003): "A set of 467 cyclooxygenase-2 (COX-2) inhibitors has been assembled from the published work of a single research group, with in vitro activities against human recombinant enzyme expressed as IC50 values ranging from 1 nM to >100 uM (53 compounds have indeterminate IC50 values)."

Details

The data are in the Supplemental Data file for the article.

A set of 255 descriptors (MOE2D and QikProp) were generated. To classify the data, we used a cutoff of $2^2.5$ to determine activity

Value

:

• cox2Descr: the descriptors

• cox2IC50: the IC50 data used to determine activity

• cox2Class: the categorical outcome ("Active" or "Inactive") based on the $2^2.5$ cutoff

Source

Sutherland, J. J., O'Brien, L. A. and Weaver, D. F. (2003). Spline-Fitting with a Genetic Algorithm: A Method for Developing Classification Structure-Activity Relationships, Journal of Chemical Information and Computer Sciences, Vol. 43, pg. 1906-1915.

Description

A series of test/training partitions are created using createDataPartition while createResample creates one or more bootstrap samples. createFolds splits the data into k groups while createTimeSlices creates cross-validation split for series data. groupKFold splits the data based on a grouping factor.

Usage

createDataPartition(
  y,
  times = 1,
  p = 0.5,
  list = TRUE,
  groups = min(5, length(y))
)

createFolds(y, k = 10, list = TRUE, returnTrain = FALSE)

createMultiFolds(y, k = 10, times = 5)

createTimeSlices(y, initialWindow, horizon = 1, fixedWindow = TRUE, skip = 0)

groupKFold(group, k = length(unique(group)))

createResample(y, times = 10, list = TRUE)

Arguments

Details

For bootstrap samples, simple random sampling is used.

For other data splitting, the random sampling is done within the levels of y when y is a factor in an attempt to balance the class distributions within the splits.

For numeric y, the sample is split into groups sections based on percentiles and sampling is done within these subgroups. For createDataPartition, the number of percentiles is set via the groups argument. For createFolds and createMultiFolds, the number of groups is set dynamically based on the sample size and k. For smaller samples sizes, these two functions may not do stratified splitting and, at most, will split the data into quartiles.

Also, for createDataPartition, very small class sizes (<= 3) the classes may not show up in both the training and test data

For multiple k-fold cross-validation, completely independent folds are created. The names of the list objects will denote the fold membership using the pattern "Foldi.Repj" meaning the ith section (of k) of the jth cross-validation set (of times). Note that this function calls createFolds with list = TRUE and returnTrain = TRUE.

Hyndman and Athanasopoulos (2013)) discuss rolling forecasting origin techniques that move the training and test sets in time. createTimeSlices can create the indices for this type of splitting.

For Group k-fold cross-validation, the data are split such that no group is contained in both the modeling and holdout sets. One or more group could be left out, depending on the value of k.

Value

A list or matrix of row position integers corresponding to the training data. For createTimeSlices subsamples are named by the end index of each training subsample.

Author(s)

Max Kuhn, createTimeSlices by Tony Cooper

References

http://topepo.github.io/caret/data-splitting.html

Hyndman and Athanasopoulos (2013), Forecasting: principles and practice. https://otexts.com/fpp2/

See Also

Other resampling: resamples()

Examples

data(oil)
createDataPartition(oilType, 2)

x <- rgamma(50, 3, .5)
inA <- createDataPartition(x, list = FALSE)

plot(density(x[inA]))
rug(x[inA])

points(density(x[-inA]), type = "l", col = 4)
rug(x[-inA], col = 4)

createResample(oilType, 2)

createFolds(oilType, 10)
createFolds(oilType, 5, FALSE)

createFolds(rnorm(21))

createTimeSlices(1:9, 5, 1, fixedWindow = FALSE)
createTimeSlices(1:9, 5, 1, fixedWindow = TRUE)
createTimeSlices(1:9, 5, 3, fixedWindow = TRUE)
createTimeSlices(1:9, 5, 3, fixedWindow = FALSE)

createTimeSlices(1:15, 5, 3)
createTimeSlices(1:15, 5, 3, skip = 2)
createTimeSlices(1:15, 5, 3, skip = 3)

set.seed(131)
groups <- sort(sample(letters[1:4], size = 20, replace = TRUE))
table(groups)
folds <- groupKFold(groups)
lapply(folds, function(x, y) table(y[x]), y = groups)

Description

Given two numeric vectors of data, the mean squared error and R-squared are calculated. For two factors, the overall agreement rate and Kappa are determined.

Usage

defaultSummary(data, lev = NULL, model = NULL)

postResample(pred, obs)

twoClassSummary(data, lev = NULL, model = NULL)

mnLogLoss(data, lev = NULL, model = NULL)

multiClassSummary(data, lev = NULL, model = NULL)

prSummary(data, lev = NULL, model = NULL)

Arguments

Details

postResample is meant to be used with apply across a matrix. For numeric data the code checks to see if the standard deviation of either vector is zero. If so, the correlation between those samples is assigned a value of zero. NA values are ignored everywhere.

Note that many models have more predictors (or parameters) than data points, so the typical mean squared error denominator (n - p) does not apply. Root mean squared error is calculated using ⁠sqrt(mean((pred - obs)^2⁠. Also, $R^2$ is calculated wither using as the square of the correlation between the observed and predicted outcomes when form = "corr". when form = "traditional",

$R^2 = 1-\frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$

. Mean absolute error is calculated using mean(abs(pred-obs)).

defaultSummary is the default function to compute performance metrics in train(). It is a wrapper around postResample. The first argument is data, which is data.frame with columns named obs and pred for the observed and predicted outcome values (either numeric data for regression or character values for classification). The second argument is lev, a character string that has the outcome factor levels or NULL for a regression model. The third parameter is model, which can be used if a summary metric is specific to a model function. If other columns from the data are required to compute the summary statistics, but should not be used in the model, the recipe method for train() can be used.

twoClassSummary computes sensitivity, specificity and the area under the ROC curve. mnLogLoss computes the minus log-likelihood of the multinomial distribution (without the constant term):

$-logLoss = \frac{-1}{n}\sum_{i=1}^n \sum_{j=1}^C y_{ij} \log(p_{ij})$

where the y values are binary indicators for the classes and p are the predicted class probabilities.

prSummary (for precision and recall) computes values for the default 0.50 probability cutoff as well as the area under the precision-recall curve across all cutoffs and is labelled as "AUC" in the output. If assumes that the first level of the factor variables corresponds to a relevant result but the lev argument can be used to change this.

multiClassSummary computes some overall measures of for performance (e.g. overall accuracy and the Kappa statistic) and several averages of statistics calculated from "one-versus-all" configurations. For example, if there are three classes, three sets of sensitivity values are determined and the average is reported with the name ("Mean_Sensitivity"). The same is true for a number of statistics generated by confusionMatrix(). With two classes, the basic sensitivity is reported with the name "Sensitivity".

To use twoClassSummary and/or mnLogLoss, the classProbs argument of trainControl() should be TRUE. multiClassSummary can be used without class probabilities but some statistics (e.g. overall log loss and the average of per-class area under the ROC curves) will not be in the result set.

Other functions can be used via the summaryFunction argument of trainControl(). Custom functions must have the same arguments asdefaultSummary.

The function getTrainPerf returns a one row data frame with the resampling results for the chosen model. The statistics will have the prefix "Train" (i.e. "TrainROC"). There is also a column called "method" that echoes the argument of the call to trainControl() of the same name.

Value

A vector of performance estimates.

Author(s)

Max Kuhn, Zachary Mayer

References

Kvalseth. Cautionary note about $R^2$. American Statistician (1985) vol. 39 (4) pp. 279-285

See Also

trainControl()

Other performance: confusionMatrix(), negPredValue(), recall()

Examples

predicted <- matrix(rnorm(50), ncol = 5)
observed <- rnorm(10)
apply(predicted, 2, postResample, obs = observed)

classes <- c("class1", "class2")
set.seed(1)
dat <- data.frame(
  obs = factor(sample(classes, 50, replace = TRUE)),
  pred = factor(sample(classes, 50, replace = TRUE)),
  class1 = runif(50)
)
dat$class2 <- 1 - dat$class1

defaultSummary(dat, lev = classes)
twoClassSummary(dat, lev = classes)
prSummary(dat, lev = classes)
mnLogLoss(dat, lev = classes)

Description

A set of lattice functions are provided to plot the resampled performance estimates (e.g. classification accuracy, RMSE) over different subset sizes.

Usage

## S3 method for class 'rfe'
densityplot(x, data = NULL, metric = x$metric, ...)

Arguments

Details

By default, only the resampling results for the optimal model are saved in the rfe object. The function rfeControl() can be used to save all the results using the returnResamp argument.

If leave-one-out or out-of-bag resampling was specified, plots cannot be produced (see the method argument of rfeControl())

Value

A lattice plot object

Author(s)

Max Kuhn

See Also

rfe(), rfeControl(), lattice::histogram(), lattice::densityplot(), lattice::xyplot(), lattice::stripplot()

Examples

library(mlbench)
n <- 100
p <- 40
sigma <- 1
set.seed(1)
sim <- mlbench.friedman1(n, sd = sigma)
x <- cbind(sim$x, matrix(rnorm(n * p), nrow = n))
y <- sim$y
colnames(x) <- paste("var", 1:ncol(x), sep = "")

normalization <- preProcess(x)
x <- predict(normalization, x)
x <- as.data.frame(x, stringsAsFactors = TRUE)
subsets <- c(10, 15, 20, 25)

ctrl <- rfeControl(
  functions = lmFuncs,
  method = "cv",
  verbose = FALSE,
  returnResamp = "all"
)

lmProfile <- rfe(x, y, sizes = subsets, rfeControl = ctrl)
xyplot(lmProfile)
stripplot(lmProfile)

histogram(lmProfile)
densityplot(lmProfile)

Description

Sutherland and Weaver (2004) discuss QSAR models for dihydrofolate reductase (DHFR) inhibition. This data set contains values for 325 compounds. For each compound, 228 molecular descriptors have been calculated. Additionally, each samples is designated as "active" or "inactive".

Details

The data frame dhfr contains a column called Y with the outcome classification. The remainder of the columns are molecular descriptor values.

Value

:

• dhfr: data frame of chemical descriptors and the activity values

Source

Sutherland, J.J. and Weaver, D.F. (2004). Three-dimensional quantitative structure-activity and structure-selectivity relationships of dihydrofolate reductase inhibitors, Journal of Computer-Aided Molecular Design, Vol. 18, pg. 309-331.

Description

Methods for making inferences about differences between models

Usage

## S3 method for class 'resamples'
diff(
  x,
  models = x$models,
  metric = x$metrics,
  test = t.test,
  confLevel = 0.95,
  adjustment = "bonferroni",
  ...
)

## S3 method for class 'diff.resamples'
summary(object, digits = max(3, getOption("digits") - 3), ...)

compare_models(a, b, metric = a$metric[1])

Arguments

Details

The ideas and methods here are based on Hothorn et al. (2005) and Eugster et al. (2008).

For each metric, all pair-wise differences are computed and tested to assess if the difference is equal to zero.

When a Bonferroni correction is used, the confidence level is changed from confLevel to 1-((1-confLevel)/p) here p is the number of pair-wise comparisons are being made. For other correction methods, no such change is used.

compare_models is a shorthand function to compare two models using a single metric. It returns the results of stats::t.test() on the differences.

Value

An object of class "diff.resamples" with elements:

• call: the call

• difs: a list for each metric being compared. Each list contains a matrix with differences in columns and resamples in rows

• statistics: a list of results generated by test

• adjustment: the p-value adjustment used

• models: a character string for which models were compared.

• metrics: a character string of performance metrics that were used

• call: the call

• table: a list of tables that show the differences and p-values

...or (for compare_models) an object of class htest resulting from stats::t.test().

Author(s)

Max Kuhn

References

Hothorn et al. The design and analysis of benchmark experiments. Journal of Computational and Graphical Statistics (2005) vol. 14 (3) pp. 675-699

Eugster et al. Exploratory and inferential analysis of benchmark experiments. Ludwigs-Maximilians-Universitat Munchen, Department of Statistics, Tech. Rep (2008) vol. 30

See Also

resamples(), dotplot.diff.resamples(), densityplot.diff.resamples(), bwplot.diff.resamples(), levelplot.diff.resamples()

Examples

load(url("http://topepo.github.io/caret/exampleModels.RData"))

resamps <- resamples(list(
  CART = rpartFit,
  CondInfTree = ctreeFit,
  MARS = earthFit
))

difs <- diff(resamps)

difs

summary(difs)

compare_models(rpartFit, ctreeFit)

Description

A lattice lattice::dotplot() is created from an object of class varImp.train.

Usage

dotPlot(x, top = min(20, dim(x$importance)[1]), ...)

Arguments

Value

an object of class trellis.

Author(s)

Max Kuhn

See Also

varImp(), lattice::dotplot()

Examples

data(iris)
TrainData <- iris[, 1:4]
TrainClasses <- iris[, 5]

knnFit <- train(TrainData, TrainClasses, "knn")

knnImp <- varImp(knnFit)

dotPlot(knnImp)

Description

Lattice functions for visualizing resampling result differences between models

Usage

## S3 method for class 'diff.resamples'
dotplot(x, data = NULL, metric = x$metric[1], ...)

Arguments

Details

densityplot and bwplot display univariate visualizations of the resampling distributions. levelplot displays the matrix of pair-wide comparisons. dotplot shows the differences along with their associated confidence intervals.

Value

a lattice object

Author(s)

Max Kuhn

See Also

resamples(), diff.resamples(), lattice::bwplot(), lattice::densityplot(), lattice::xyplot(), lattice::splom()

Examples

load(url("http://topepo.github.io/caret/exampleModels.RData"))

resamps <- resamples(list(
  CART = rpartFit,
  CondInfTree = ctreeFit,
  MARS = earthFit
))
difs <- diff(resamps)

dotplot(difs)

densityplot(difs, metric = "RMSE", auto.key = TRUE, pch = "|")

bwplot(difs, metric = "RMSE")

levelplot(difs, what = "differences")

Description

downSample will randomly sample a data set so that all classes have the same frequency as the minority class. upSample samples with replacement to make the class distributions equal

Usage

downSample(x, y, list = FALSE, yname = "Class")

Arguments

Details

Simple random sampling is used to down-sample for the majority class(es). Note that the minority class data are left intact and that the samples will be re-ordered in the down-sampled version.

For up-sampling, all the original data are left intact and additional samples are added to the minority classes with replacement.

Value

Either a data frame or a list with elements x and y.

Author(s)

Max Kuhn

Examples

## A ridiculous example...
data(oil)
table(oilType)
downSample(fattyAcids, oilType)

upSample(fattyAcids, oilType)

Description

dummyVars creates a full set of dummy variables (i.e. less than full rank parameterization)

Usage

dummyVars(formula, ...)

## Default S3 method:
dummyVars(formula, data, sep = ".", levelsOnly = FALSE, fullRank = FALSE, ...)

## S3 method for class 'dummyVars'
print(x, ...)

## S3 method for class 'dummyVars'
predict(object, newdata, na.action = na.pass, ...)

contr.ltfr(n, contrasts = TRUE, sparse = FALSE)

class2ind(x, drop2nd = FALSE)

Arguments

Details

Most of the stats::contrasts() functions in R produce full rank parameterizations of the predictor data. For example, stats::contr.treatment() creates a reference cell in the data and defines dummy variables for all factor levels except those in the reference cell. For example, if a factor with 5 levels is used in a model formula alone, stats::contr.treatment() creates columns for the intercept and all the factor levels except the first level of the factor. For the data in the Example section below, this would produce:

 (Intercept) dayTue dayWed dayThu dayFri daySat daySun
           1      0      0      0      0      0      0
           1      0      0      0      0      0      0
           1      0      0      0      0      0      0
           1      0      1      0      0      0      0
           1      0      1      0      0      0      0
           1      0      0      0      1      0      0
           1      0      0      0      0      1      0
           1      0      0      0      0      1      0
           1      0      0      0      1      0      0

In some situations, there may be a need for dummy variables for all the levels of the factor. For the same example:

 dayMon dayTue dayWed dayThu dayFri daySat daySun
      1      0      0      0      0      0      0
      1      0      0      0      0      0      0
      1      0      0      0      0      0      0
      0      0      1      0      0      0      0
      0      0      1      0      0      0      0
      0      0      0      0      1      0      0
      0      0      0      0      0      1      0
      0      0      0      0      0      1      0
      0      0      0      0      1      0      0

Given a formula and initial data set, the class dummyVars gathers all the information needed to produce a full set of dummy variables for any data set. It uses contr.ltfr as the base function to do this.

class2ind is most useful for converting a factor outcome vector to a matrix (or vector) of dummy variables.

Value

The output of dummyVars is a list of class 'dummyVars' with elements

• call: the function call

• form: the model formula

• vars: names of all the variables in the model

• facVars: names of all the factor variables in the model

• lvls: levels of any factor variables

• sep: NULL or a character separator

• terms: the stats::terms.formula() object

• levelsOnly: a logical

The predict function produces a data frame. class2ind returns a matrix (or a vector if drop2nd = TRUE). contr.ltfr generates a design matrix.

Author(s)

contr.ltfr is a small modification of stats::contr.treatment() by Max Kuhn

References

https://cran.r-project.org/doc/manuals/R-intro.html#Formulae-for-statistical-models

See Also

stats::model.matrix(), stats::contrasts(), stats::formula()

Other preprocessing: BoxCoxTrans(), avNNet(), findCorrelation(), nearZeroVar(), pcaNNet(), preProcess(), spatialSign()

Examples

when <- data.frame(
  time = c(
    "afternoon",
    "night",
    "afternoon",
    "morning",
    "morning",
    "morning",
    "morning",
    "afternoon",
    "afternoon"
  ),
  day = c("Mon", "Mon", "Mon", "Wed", "Wed", "Fri", "Sat", "Sat", "Fri"),
  stringsAsFactors = TRUE
)

levels(when$time) <- list(
  morning = "morning",
  afternoon = "afternoon",
  night = "night"
)
levels(when$day) <- list(
  Mon = "Mon",
  Tue = "Tue",
  Wed = "Wed",
  Thu = "Thu",
  Fri = "Fri",
  Sat = "Sat",
  Sun = "Sun"
)

## Default behavior:
model.matrix(~day, when)

mainEffects <- dummyVars(~ day + time, data = when)
mainEffects
predict(mainEffects, when[1:3, ])

when2 <- when
when2[1, 1] <- NA
predict(mainEffects, when2[1:3, ])
predict(mainEffects, when2[1:3, ], na.action = na.omit)

interactionModel <- dummyVars(~ day + time + day:time, data = when, sep = ".")
predict(interactionModel, when[1:3, ])

noNames <- dummyVars(~ day + time + day:time, data = when, levelsOnly = TRUE)
predict(noNames, when)

head(class2ind(iris$Species))

two_levels <- factor(rep(letters[1:2], each = 5))
class2ind(two_levels)
class2ind(two_levels, drop2nd = TRUE)

Description

These functions can be used for a single train object or to loop through a number of train objects to calculate the training and test data predictions and class probabilities.

Usage

extractPrediction(
  models,
  testX = NULL,
  testY = NULL,
  unkX = NULL,
  unkOnly = !is.null(unkX) & is.null(testX),
  verbose = FALSE
)

extractProb(
  models,
  testX = NULL,
  testY = NULL,
  unkX = NULL,
  unkOnly = !is.null(unkX) & is.null(testX),
  verbose = FALSE
)

## S3 method for class 'train'
predict(object, newdata = NULL, type = "raw", na.action = na.omit, ...)

Arguments

Details

These functions are wrappers for the specific prediction functions in each modeling package. In each case, the optimal tuning values given in the tuneValue slot of the finalModel object are used to predict.

To get simple predictions for a new data set, the predict function can be used. Limits can be imposed on the range of predictions. See trainControl() for more information.

To get predictions for a series of models at once, a list of train() objects can be passes to the predict function and a list of model predictions will be returned.

The two extraction functions can be used to get the predictions and observed outcomes at once for the training, test and/or unknown samples at once in a single data frame (instead of a list of just the predictions). These objects can then be passes to plotObsVsPred() or plotClassProbs().

Value

For predict.train, a vector of predictions if type = "raw" or a data frame of class probabilities for type = "prob". In the latter case, there are columns for each class.

For predict.list, a list results. Each element is produced by predict.train.

For extractPrediction, a data frame with columns:

• obs: the observed training and test data

• pred: predicted values

• model: the type of model used to predict

• object: the names of the objects within models. If models is an un-named list, the values of object will be "Object1", "Object2" and so on

• dataType: "Training", "Test" or "Unknown" depending on what was specified

For extractProb, a data frame. There is a column for each class containing the probabilities. The remaining columns are the same as above (although the pred column is the predicted class)

Author(s)

Max Kuhn

References

Kuhn (2008), “Building Predictive Models in R Using the caret” (doi:10.18637/jss.v028.i05)

See Also

plotObsVsPred(), plotClassProbs(), trainControl()

Other train: ggplot.train(), print.train(), train(), trainControl(), varImp()

Examples

## Not run: 

knnFit <- train(
  Species ~ .,
  data = iris,
  method = "knn",
  trControl = trainControl(method = "cv")
)

rdaFit <- train(
  Species ~ .,
  data = iris,
  method = "rda",
  trControl = trainControl(method = "cv")
)

predict(knnFit)
predict(knnFit, type = "prob")

bothModels <- list(knn = knnFit, tree = rdaFit)

predict(bothModels)

extractPrediction(bothModels, testX = iris[1:10, -5])
extractProb(bothModels, testX = iris[1:10, -5])
  
## End(Not run)

Description

A shortcut to produce lattice graphs

Usage

featurePlot(
  x,
  y,
  plot = if (is.factor(y)) "strip" else "scatter",
  labels = c("Feature", ""),
  ...
)

Arguments

Details

This function “stacks” data to get it into a form compatible with lattice and creates the plots

Value

An object of class “trellis”. The ⁠update' method can be used to update components of the object and the ⁠print' method (usually called by default) will plot it on an appropriate plotting device.

Author(s)

Max Kuhn

Examples

x <- matrix(rnorm(50 * 5), ncol = 5)
y <- factor(rep(c("A", "B"), 25))

trellis.par.set(theme = col.whitebg(), warn = FALSE)
featurePlot(x, y, "ellipse")
featurePlot(x, y, "strip", jitter = TRUE)
featurePlot(x, y, "box")
featurePlot(x, y, "pairs")

Description

Specific engines for variable importance on a model by model basis.

Usage

filterVarImp(x, y, nonpara = FALSE, ...)

Arguments

Details

The importance of each predictor is evaluated individually using a “filter” approach.

For classification, ROC curve analysis is conducted on each predictor. For two class problems, a series of cutoffs is applied to the predictor data to predict the class. The sensitivity and specificity are computed for each cutoff and the ROC curve is computed. The trapezoidal rule is used to compute the area under the ROC curve. This area is used as the measure of variable importance. For multi-class outcomes, the problem is decomposed into all pair-wise problems and the area under the curve is calculated for each class pair (i.e class 1 vs. class 2, class 2 vs. class 3 etc.). For a specific class, the maximum area under the curve across the relevant pair-wise AUC's is used as the variable importance measure.

For regression, the relationship between each predictor and the outcome is evaluated. An argument, nonpara, is used to pick the model fitting technique. When nonpara = FALSE, a linear model is fit and the absolute value of the $t$-value for the slope of the predictor is used. Otherwise, a loess smoother is fit between the outcome and the predictor. The $R^2$ statistic is calculated for this model against the intercept only null model.

Value

A data frame with variable importances. Column names depend on the problem type. For regression, the data frame contains one column: "Overall" for the importance values.

Author(s)

Max Kuhn

Examples

data(mdrr)
filterVarImp(mdrrDescr[, 1:5], mdrrClass)

data(BloodBrain)

filterVarImp(bbbDescr[, 1:5], logBBB, nonpara = FALSE)
apply(
  bbbDescr[, 1:5],
  2,
  function(x, y) summary(lm(y ~ x))$coefficients[2, 3],
  y = logBBB
)

filterVarImp(bbbDescr[, 1:5], logBBB, nonpara = TRUE)

Description

This function searches through a correlation matrix and returns a vector of integers corresponding to columns to remove to reduce pair-wise correlations.

Usage

findCorrelation(
  x,
  cutoff = 0.9,
  verbose = FALSE,
  names = FALSE,
  exact = ncol(x) < 100
)

Arguments

Details

The absolute values of pair-wise correlations are considered. If two variables have a high correlation, the function looks at the mean absolute correlation of each variable and removes the variable with the largest mean absolute correlation.

Using exact = TRUE will cause the function to re-evaluate the average correlations at each step while exact = FALSE uses all the correlations regardless of whether they have been eliminated or not. The exact calculations will remove a smaller number of predictors but can be much slower when the problem dimensions are "big".

Value

A vector of indices denoting the columns to remove (when names = TRUE) otherwise a vector of column names. If no correlations meet the criteria, integer(0) is returned.

Author(s)

Original R code by Dong Li, modified by Max Kuhn

See Also

findLinearCombos()

Other preprocessing: BoxCoxTrans(), avNNet(), dummyVars(), nearZeroVar(), pcaNNet(), preProcess(), spatialSign()

Examples

R1 <- structure(
  c(
    1,
    0.86,
    0.56,
    0.32,
    0.85,
    0.86,
    1,
    0.01,
    0.74,
    0.32,
    0.56,
    0.01,
    1,
    0.65,
    0.91,
    0.32,
    0.74,
    0.65,
    1,
    0.36,
    0.85,
    0.32,
    0.91,
    0.36,
    1
  ),
  .Dim = c(5L, 5L)
)
colnames(R1) <- rownames(R1) <- paste0("x", 1:ncol(R1))
R1

findCorrelation(R1, cutoff = .6, exact = FALSE)
findCorrelation(R1, cutoff = .6, exact = TRUE)
findCorrelation(R1, cutoff = .6, exact = TRUE, names = FALSE)

R2 <- diag(rep(1, 5))
R2[2, 3] <- R2[3, 2] <- .7
R2[5, 3] <- R2[3, 5] <- -.7
R2[4, 1] <- R2[1, 4] <- -.67

corrDF <- expand.grid(row = 1:5, col = 1:5)
corrDF$correlation <- as.vector(R2)
levelplot(correlation ~ row + col, corrDF)

findCorrelation(R2, cutoff = .65, verbose = TRUE)

findCorrelation(R2, cutoff = .99, verbose = TRUE)

Description

Enumerate and resolve the linear combinations in a numeric matrix

Usage

findLinearCombos(x)

Arguments

Details

The QR decomposition is used to determine if the matrix is full rank and then identify the sets of columns that are involved in the dependencies.

To "resolve" them, columns are iteratively removed and the matrix rank is rechecked.

Value

a list with elements:

• linearCombos: If there are linear combinations, this will be a list with elements for each dependency that contains vectors of column numbers.

• remove: a list of column numbers that can be removed to counter the linear combinations

Author(s)

Kirk Mettler and Jed Wing (enumLC) and Max Kuhn (findLinearCombos)

Examples

testData1 <- matrix(0, nrow = 20, ncol = 8)
testData1[, 1] <- 1
testData1[, 2] <- round(rnorm(20), 1)
testData1[, 3] <- round(rnorm(20), 1)
testData1[, 4] <- round(rnorm(20), 1)
testData1[, 5] <- 0.5 *
  testData1[, 2] -
  0.25 * testData1[, 3] -
  0.25 * testData1[, 4]
testData1[1:4, 6] <- 1
testData1[5:10, 7] <- 1
testData1[11:20, 8] <- 1

findLinearCombos(testData1)

testData2 <- matrix(0, nrow = 6, ncol = 6)
testData2[, 1] <- c(1, 1, 1, 1, 1, 1)
testData2[, 2] <- c(1, 1, 1, 0, 0, 0)
testData2[, 3] <- c(0, 0, 0, 1, 1, 1)
testData2[, 4] <- c(1, 0, 0, 1, 0, 0)
testData2[, 5] <- c(0, 1, 0, 0, 1, 0)
testData2[, 6] <- c(0, 0, 1, 0, 0, 1)

findLinearCombos(testData2)

Description

Return a string representing the ‘bagEarth’ expression.

Usage

## S3 method for class 'bagEarth'
format(x, file = "", cat = TRUE, ...)

Arguments

Value

A character representation of the bagged earth object.

See Also

earth::earth()

Examples

a <- bagEarth(Volume ~ ., data = trees, B = 3)
format(a)

# yields:
# (
#   31.61075
#   +  6.587273 * pmax(0,  Girth -   14.2)
#   -  3.229363 * pmax(0,   14.2 -  Girth)
#   - 0.3167140 * pmax(0,     79 - Height)
#   +
#    22.80225
#   +  5.309866 * pmax(0,  Girth -     12)
#   -  2.378658 * pmax(0,     12 -  Girth)
#   +  0.793045 * pmax(0, Height -     80)
#   - 0.3411915 * pmax(0,     80 - Height)
#   +
#    31.39772
#   +   6.18193 * pmax(0,  Girth -   14.2)
#   -  3.660456 * pmax(0,   14.2 -  Girth)
#   + 0.6489774 * pmax(0, Height -     80)
# )/3

Description

Built-in functions related to genetic algorithms

These functions are used with the functions argument of the gafsControl() function. More information on the details of these functions are at http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html.

Most of the ⁠gafs_*⁠ functions are based on those from the GA package by Luca Scrucca. These functions here are small re-writes to work outside of the GA package.

The objects caretGA, rfGA and treebagGA are example lists that can be used with the functions argument of gafsControl().

In the case of caretGA, the ... structure of gafs() passes through to the model fitting routine. As a consequence, the train() function can easily be accessed by passing important arguments belonging to train() to gafs(). See the examples below. By default, using caretGA will used the resampled performance estimates produced by train() as the internal estimate of fitness.

For rfGA and treebagGA, the randomForest and bagging functions are used directly (i.e. train() is not used). Arguments to either of these functions can also be passed to them though the gafs() call (see examples below). For these two functions, the internal fitness is estimated using the out-of-bag estimates naturally produced by those functions. While faster, this limits the user to accuracy or Kappa (for classification) and RMSE and R-squared (for regression).

Usage

gafs_initial(vars, popSize, ...)

gafs_lrSelection(population, fitness, r = NULL, q = NULL, ...)

gafs_spCrossover(population, fitness, parents, ...)

gafs_raMutation(population, parent, ...)

gafs_nlrSelection(population, fitness, q = 0.25, ...)

gafs_rwSelection(population, fitness, ...)

gafs_tourSelection(population, fitness, k = 3, ...)

gafs_uCrossover(population, parents, ...)

Arguments

Value

The return value depends on the function.

Author(s)

Luca Scrucca, gafs_initial, caretGA, rfGA and treebagGA by Max Kuhn

References

Scrucca L (2013). GA: A Package for Genetic Algorithms in R. Journal of Statistical Software, 53(4), 1-37.

https://cran.r-project.org/package=GA

http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html

See Also

gafs(), gafsControl()

Examples

pop <- gafs_initial(vars = 10, popSize = 10)
pop

gafs_lrSelection(population = pop, fitness = 1:10)

gafs_spCrossover(population = pop, fitness = 1:10, parents = 1:2)

## Hypothetical usage (not run):
## lda_ga <- gafs(x = predictors,
##                y = classes,
##                gafsControl = gafsControl(functions = caretGA),
##                method = "lda",
##                metric = "Accuracy",
##                trControl = trainControl(method = "cv",
##                                         classProbs = TRUE))
##
## rf_ga <- gafs(x = predictors,
##               y = classes,
##               gafsControl = gafsControl(functions = rfGA),
##               ntree = 1000,
##               importance = TRUE)

Description

Supervised feature selection using genetic algorithms

Usage

## Default S3 method:
gafs(
  x,
  y,
  iters = 10,
  popSize = 50,
  pcrossover = 0.8,
  pmutation = 0.1,
  elite = 0,
  suggestions = NULL,
  differences = TRUE,
  gafsControl = gafsControl(),
  ...
)

## S3 method for class 'recipe'
gafs(
  x,
  data,
  iters = 10,
  popSize = 50,
  pcrossover = 0.8,
  pmutation = 0.1,
  elite = 0,
  suggestions = NULL,
  differences = TRUE,
  gafsControl = gafsControl(),
  ...
)

Arguments

Details

gafs() conducts a supervised binary search of the predictor space using a genetic algorithm. See Mitchell (1996) and Scrucca (2013) for more details on genetic algorithms.

This function conducts the search of the feature space repeatedly within resampling iterations. First, the training data are split be whatever resampling method was specified in the control function. For example, if 10-fold cross-validation is selected, the entire genetic algorithm is conducted 10 separate times. For the first fold, nine tenths of the data are used in the search while the remaining tenth is used to estimate the external performance since these data points were not used in the search.

During the genetic algorithm, a measure of fitness is needed to guide the search. This is the internal measure of performance. During the search, the data that are available are the instances selected by the top-level resampling (e.g. the nine tenths mentioned above). A common approach is to conduct another resampling procedure. Another option is to use a holdout set of samples to determine the internal estimate of performance (see the holdout argument of the control function). While this is faster, it is more likely to cause overfitting of the features and should only be used when a large amount of training data are available. Yet another idea is to use a penalized metric (such as the AIC statistic) but this may not exist for some metrics (e.g. the area under the ROC curve).

The internal estimates of performance will eventually overfit the subsets to the data. However, since the external estimate is not used by the search, it is able to make better assessments of overfitting. After resampling, this function determines the optimal number of generations for the GA.

Finally, the entire data set is used in the last execution of the genetic algorithm search and the final model is built on the predictor subset that is associated with the optimal number of generations determined by resampling (although the update function can be used to manually set the number of generations).

This is an example of the output produced when gafsControl(verbose = TRUE) is used:

Fold2 1 0.715 (13)
Fold2 2 0.715->0.737 (13->17, 30.4%) *
Fold2 3 0.737->0.732 (17->14, 24.0%)
Fold2 4 0.737->0.769 (17->23, 25.0%) *

For the second resample (e.g. fold 2), the best subset across all individuals tested in the first generation contained 13 predictors and was associated with a fitness value of 0.715. The second generation produced a better subset containing 17 samples with an associated fitness values of 0.737 (and improvement is symbolized by the *. The percentage listed is the Jaccard similarity between the previous best individual (with 13 predictors) and the new best. The third generation did not produce a better fitness value but the fourth generation did.

The search algorithm can be parallelized in several places:

1. each externally resampled GA can be run independently (controlled by the allowParallel option of gafsControl())

2. within a GA, the fitness calculations at a particular generation can be run in parallel over the current set of individuals (see the genParallel option in gafsControl())

3. if inner resampling is used, these can be run in parallel (controls depend on the function used. See, for example, trainControl())

4. any parallelization of the individual model fits. This is also specific to the modeling function.

It is probably best to pick one of these areas for parallelization and the first is likely to produces the largest decrease in run-time since it is the least likely to incur multiple re-starting of the worker processes. Keep in mind that if multiple levels of parallelization occur, this can effect the number of workers and the amount of memory required exponentially.

Value

an object of class gafs

Author(s)

Max Kuhn, Luca Scrucca (for GA internals)

References

Kuhn M and Johnson K (2013), Applied Predictive Modeling, Springer, Chapter 19 http://appliedpredictivemodeling.com

Scrucca L (2013). GA: A Package for Genetic Algorithms in R. Journal of Statistical Software, 53(4), 1-37. https://www.jstatsoft.org/article/view/v053i04

Mitchell M (1996), An Introduction to Genetic Algorithms, MIT Press.

https://en.wikipedia.org/wiki/Jaccard_index

See Also

gafsControl(), predict.gafs(), caretGA(), rfGA() treebagGA()

Examples

set.seed(1)
train_data <- twoClassSim(100, noiseVars = 10)
test_data <- twoClassSim(10, noiseVars = 10)

## A short example
ctrl <- gafsControl(functions = rfGA, method = "cv", number = 3)

rf_search <- gafs(
  x = train_data[, -ncol(train_data)],
  y = train_data$Class,
  iters = 3,
  gafsControl = ctrl
)

rf_search

Description

Control the computational nuances of the gafs() and safs() functions

Many of these options are the same as those described for trainControl(). More extensive documentation and examples can be found on the caret website at http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html#syntax and http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html#syntax.

The functions component contains the information about how the model should be fit and summarized. It also contains the elements needed for the GA and SA modules (e.g. cross-over, etc).

The elements of functions that are the same for GAs and SAs are:

• fit, with arguments x, y, lev, last, and ..., is used to fit the classification or regression model

• pred, with arguments object and x, predicts new samples

• fitness_intern, with arguments object, x, y, maximize, and p, summarizes performance for the internal estimates of fitness

• fitness_extern, with arguments data, lev, and model, summarizes performance using the externally held-out samples

• selectIter, with arguments x, metric, and maximize, determines the best search iteration for feature selection.

The elements of functions specific to genetic algorithms are:

• initial, with arguments vars, popSize and ..., creates an initial population.

• selection, with arguments population, fitness, r, q, and ..., conducts selection of individuals.

• crossover, with arguments population, fitness, parents and ..., control genetic reproduction.

• mutation, with arguments population, parent and ..., adds mutations.

The elements of functions specific to simulated annealing are:

• initial, with arguments vars, prob, and ..., creates the initial subset.

• perturb, with arguments x, vars, and number, makes incremental changes to the subsets.

• prob, with arguments old, new, and iteration, computes the acceptance probabilities

The pages http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html and http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html have more details about each of these functions.

holdout can be used to hold out samples for computing the internal fitness value. Note that this is independent of the external resampling step. Suppose 10-fold CV is being used. Within a resampling iteration, holdout can be used to sample an additional proportion of the 90\ use for estimating fitness. This may not be a good idea unless you have a very large training set and want to avoid an internal resampling procedure to estimate fitness.

The search algorithms can be parallelized in several places:

1. each externally resampled GA or SA can be run independently (controlled by the allowParallel options)

2. within a GA, the fitness calculations at a particular generation can be run in parallel over the current set of individuals (see the genParallel)

3. if inner resampling is used, these can be run in parallel (controls depend on the function used. See, for example, trainControl())

4. any parallelization of the individual model fits. This is also specific to the modeling function.

It is probably best to pick one of these areas for parallelization and the first is likely to produces the largest decrease in run-time since it is the least likely to incur multiple re-starting of the worker processes. Keep in mind that if multiple levels of parallelization occur, this can effect the number of workers and the amount of memory required exponentially.

Usage

gafsControl(
  functions = NULL,
  method = "repeatedcv",
  metric = NULL,
  maximize = NULL,
  number = ifelse(grepl("cv", method), 10, 25),
  repeats = ifelse(grepl("cv", method), 1, 5),
  verbose = FALSE,
  returnResamp = "final",
  p = 0.75,
  index = NULL,
  indexOut = NULL,
  seeds = NULL,
  holdout = 0,
  genParallel = FALSE,
  allowParallel = TRUE
)

safsControl(
  functions = NULL,
  method = "repeatedcv",
  metric = NULL,
  maximize = NULL,
  number = ifelse(grepl("cv", method), 10, 25),
  repeats = ifelse(grepl("cv", method), 1, 5),
  verbose = FALSE,
  returnResamp = "final",
  p = 0.75,
  index = NULL,
  indexOut = NULL,
  seeds = NULL,
  holdout = 0,
  improve = Inf,
  allowParallel = TRUE
)

Arguments

Value

An echo of the parameters specified

Author(s)

Max Kuhn

References

http://topepo.github.io/caret/feature-selection-using-genetic-algorithms.html, http://topepo.github.io/caret/feature-selection-using-simulated-annealing.html

See Also

safs(), safs(), , caretGA(), rfGA(), treebagGA(), caretSA(), rfSA(), treebagSA()

Other feature-selection: predict.gafs(), rfe(), sbf()

Description

Data from Dr. Hans Hofmann of the University of Hamburg.

Details

These data have two classes for the credit worthiness: good or bad. There are predictors related to attributes, such as: checking account status, duration, credit history, purpose of the loan, amount of the loan, savings accounts or bonds, employment duration, Installment rate in percentage of disposable income, personal information, other debtors/guarantors, residence duration, property, age, other installment plans, housing, number of existing credits, job information, Number of people being liable to provide maintenance for, telephone, and foreign worker status.

Many of these predictors are discrete and have been expanded into several 0/1 indicator variables

Source

UCI Machine Learning Repository

Description

Placeholder.

Usage

getSamplingInfo(method = NULL, regex = TRUE, ...)

Arguments

Details

Placeholder.

Value

A list

Description

These functions plot the resampling results for the candidate subset sizes evaluated during the recursive feature elimination (RFE) process

Usage

## S3 method for class 'rfe'
ggplot(
  data = NULL,
  mapping = NULL,
  metric = data$metric[1],
  output = "layered",
  ...,
  environment = NULL
)

## S3 method for class 'rfe'
plot(x, metric = x$metric, ...)

Arguments

Details

These plots show the average performance versus the subset sizes.

Value

a lattice or ggplot object

Note

We using a recipe as an input, there may be some subset sizes that are not well-replicated over resamples. The ggplot method will only show subset sizes where at least half of the resamples have associated results.

Author(s)

Max Kuhn

References

Kuhn (2008), “Building Predictive Models in R Using the caret” (doi:10.18637/jss.v028.i05)

See Also

rfe(), lattice::xyplot(), ggplot2::ggplot()

Examples

data(BloodBrain)

x <- scale(bbbDescr[, -nearZeroVar(bbbDescr)])
x <- x[, -findCorrelation(cor(x), .8)]
x <- as.data.frame(x, stringsAsFactors = TRUE)

set.seed(1)
lmProfile <- rfe(
  x,
  logBBB,
  sizes = c(2:25, 30, 35, 40, 45, 50, 55, 60, 65),
  rfeControl = rfeControl(functions = lmFuncs, number = 10)
)
plot(lmProfile)
plot(lmProfile, metric = "Rsquared")
ggplot(lmProfile)

Description

This function takes the output of a train() object and creates a line or level plot using the lattice or ggplot2 libraries.

Usage

## S3 method for class 'train'
ggplot(
  data = NULL,
  mapping = NULL,
  metric = data$metric[1],
  plotType = "scatter",
  output = "layered",
  nameInStrip = FALSE,
  highlight = FALSE,
  ...,
  environment = NULL
)

## S3 method for class 'train'
plot(
  x,
  plotType = "scatter",
  metric = x$metric[1],
  digits = getOption("digits") - 3,
  xTrans = NULL,
  nameInStrip = FALSE,
  ...
)

Arguments

Details

If there are no tuning parameters, or none were varied, an error is produced.

If the model has one tuning parameter with multiple candidate values, a plot is produced showing the profile of the results over the parameter. Also, a plot can be produced if there are multiple tuning parameters but only one is varied.

If there are two tuning parameters with different values, a plot can be produced where a different line is shown for each value of of the other parameter. For three parameters, the same line plot is created within conditioning panels/facets of the other parameter.

Also, with two tuning parameters (with different values), a levelplot (i.e. un-clustered heatmap) can be created. For more than two parameters, this plot is created inside conditioning panels/facets.

Author(s)

Max Kuhn

References

Kuhn, M. (2008), "Building Predictive Models in R Using the caret Package," Journal of Statistical Software, 28(5), 1-26. (doi:10.18637/jss.v028.i05).

See Also

train(), lattice::levelplot(), lattice::xyplot(), lattice::stripplot(), ggplot2::ggplot()

Other train: extractPrediction(), print.train(), train(), trainControl(), varImp()

Examples

library(klaR)
rdaFit <- train(
  Species ~ .,
  data = iris,
  method = "rda",
  control = trainControl(method = "cv")
)
plot(rdaFit)
plot(rdaFit, plotType = "level")

ggplot(rdaFit) + theme_bw()

Description

A set of lattice functions are provided to plot the resampled performance estimates (e.g. classification accuracy, RMSE) over tuning parameters (if any).

Usage

## S3 method for class 'train'
histogram(x, data = NULL, metric = x$metric, ...)

Arguments

Details

By default, only the resampling results for the optimal model are saved in the train object. The function trainControl() can be used to save all the results (see the example below).

If leave-one-out or out-of-bag resampling was specified, plots cannot be produced (see the method argument of trainControl())

For xyplot and stripplot, the tuning parameter with the most unique values will be plotted on the x-axis. The remaining parameters (if any) will be used as conditioning variables. For densityplot and histogram, all tuning parameters are used for conditioning.

Using horizontal = FALSE in stripplot works.

Value

A lattice plot object

Author(s)

Max Kuhn

See Also

train(), trainControl(), lattice::histogram(), lattice::densityplot(), lattice::xyplot(), lattice::stripplot()

Examples

library(mlbench)
data(BostonHousing)

library(rpart)
rpartFit <- train(
  medv ~ .,
  data = BostonHousing,
  "rpart",
  tuneLength = 9,
  trControl = trainControl(
    method = "boot",
    returnResamp = "all"
  )
)

densityplot(rpartFit, adjust = 1.25)

xyplot(rpartFit, metric = "Rsquared", type = c("p", "a"))

stripplot(rpartFit, horizontal = FALSE, jitter = TRUE)

Description

Fit a linear regression model using independent components

Usage

## S3 method for class 'formula'
icr(formula, data, weights, ..., subset, na.action, contrasts = NULL)

## Default S3 method:
icr(x, y, ...)

## S3 method for class 'icr'
predict(object, newdata, ...)

Arguments

Details

This produces a model analogous to Principal Components Regression (PCR) but uses Independent Component Analysis (ICA) to produce the scores. The user must specify a value of n.comp to pass to fastICA::fastICA().

The function preProcess() to produce the ICA scores for the original data and for newdata.

Value

For icr, a list with elements:

• model: the results of stats::lm() after the ICA transformation

• ica: pre-processing information

• n.comp: number of ICA components

• names: column names of the original data

Author(s)

Max Kuhn

See Also

fastICA::fastICA(), preProcess(), stats::lm()

Examples

data(BloodBrain)

icrFit <- icr(bbbDescr, logBBB, n.comp = 5)

icrFit

predict(icrFit, bbbDescr[1:5, ])

Description

The function performs the opposite of which converting a set of integers to a binary vector

Usage

index2vec(x, vars, sign = FALSE)

Arguments

Value

a numeric vector

Author(s)

Max Kuhn

Examples

index2vec(x = 1:2, vars = 5)
index2vec(x = 1:2, vars = 5, sign = TRUE)

Description

$k$-nearest neighbour classification that can return class votes for all classes.

Usage

knn3(x, ...)

## S3 method for class 'formula'
knn3(formula, data, subset, na.action, k = 5, ...)

## S3 method for class 'data.frame'
knn3(x, y, k = 5, ...)

## S3 method for class 'matrix'
knn3(x, y, k = 5, ...)

## S3 method for class 'knn3'
print(x, ...)

## S3 method for class 'knn3'
predict(object, newdata, type = c("prob", "class"), ...)

knn3Train(train, test, cl, k = 1, l = 0, prob = TRUE, use.all = TRUE)