Bestnormalize Package
Bestnormalize Package
1
2 bestNormalize-package
R topics documented:
bestNormalize-package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
arcsinh_x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
autotrader . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
bestNormalize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
binarize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
boxcox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
exp_x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
lambert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
log_x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
no_transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
orderNorm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
plot.bestNormalize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
sqrt_x . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
yeojohnson . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Index 23
Description
The bestNormalize package provides several normalizing transformations, and introduces a new
transformation based off of the order statistics, orderNorm. Perhaps the most useful function is
bestNormalize, which attempts all of these transformations and picks the best one based off of a
goodness of fit statistic.
Author(s)
See Also
Useful links:
• https://github.com/petersonR/bestNormalize
arcsinh_x 3
Description
Perform a arcsinh(x) transformation
Usage
arcsinh_x(x, standardize = TRUE)
Arguments
x A vector to normalize with with x
standardize If TRUE, the transformed values are also centered and scaled, such that the
transformation attempts a standard normal
object an object of class ’arcsinh_x’
newdata a vector of data to be (potentially reverse) transformed
inverse if TRUE, performs reverse transformation
... additional arguments
Details
arcsinh_x performs an arcsinh transformation in the context of bestNormalize, such that it creates
a transformation that can be estimated and applied to new data via the predict function.
The function is explicitly: log(x + sqrt(x^2 + 1))
Value
A list of class arcsinh_x with elements
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Description
A dataset containing the prices and other attributes of over 6000 cars in the Minneapolis area.
Usage
autotrader
Format
A data frame with 6283 rows and 10 variables:
price price, in US dollars
Car_Info Raw description from website
Link hyperlink to listing (must be appended to https://www.autotrader.com/)
Make Car manufacturer
Year Year car manufactured
Location Location of listing
Radius Radius chosen for search
mileage mileage on vehicle
status used/new/certified
model make and model, separated by space
Source
https://www.autotrader.com/
bestNormalize 5
Description
Performs a suite of normalizing transformations, and selects the best one on the basis of the Pearson
P test statistic for normality. The transformation that has the lowest P (calculated on the transformed
data) is selected. See details for more information.
Usage
bestNormalize(x, standardize = TRUE, allow_orderNorm = TRUE,
allow_lambert_s = FALSE, allow_lambert_h = FALSE,
out_of_sample = TRUE, cluster = NULL, k = 10, r = 5,
loo = FALSE, warn = TRUE, quiet = FALSE)
Arguments
x A vector to normalize
standardize If TRUE, the transformed values are also centered and scaled, such that the
transformation attempts a standard normal. This will not change the normality
statistic.
allow_orderNorm
set to FALSE if orderNorm should not be applied
allow_lambert_s
Set to TRUE if the lambertW of type "s" should be applied (see details)
allow_lambert_h
Set to TRUE if the lambertW of type "h" should be applied (see details)
out_of_sample if FALSE, estimates quickly in-sample performance
cluster name of cluster set using makeCluster
k number of folds
r number of repeats
loo should leave-one-out CV be used instead of repeated CV? (see details)
warn Should bestNormalize warn when a method doesn’t work?
quiet Should a progress-bar not be displayed for cross-validation progress?
object an object of class ’bestNormalize’
6 bestNormalize
Details
bestNormalize estimates the optimal normalizing transformation. This transformation can be per-
formed on new data, and inverted, via the predict function.
This function currently estimates the Yeo-Johnson transformation, the Box Cox transformation (if
the data is positive), the log_10(x+a) transformation, the square-root (x+a) transformation, and the
arcsinh transformation. a is set to max(0, -min(x) + eps) by default. If allow_orderNorm == TRUE
and if out_of_sample == FALSE then the ordered quantile normalization technique will likely be
chosen since it essentially forces the data to follow a normal distribution. More information on the
orderNorm technique can be found in the package vignette, or using ?orderNorm.
Repeated cross-validation is used by default to estimate the out-of-sample performance of each
transformation if out_of_sample = TRUE. While this can take some time, users can speed it up by
creating a cluster via the parallel package’s makeCluster function, and passing the name of this
cluster to bestNormalize via the cl argument. For best performance, we recommend the number
of clusters to be set to the number of repeats r. Care should be taken to account for the number of
observations per fold; to small a number and the estimated normality statistic could be inaccurate,
or at least suffer from high variability.
As of version 1.3, users can use leave-one-out cross-validation as well for each method by setting
loo to TRUE. This will take a lot of time for bigger vectors, but it will have the most accurate estimate
of normalization efficacy. Note that if this method is selected, arguments k, r are ignored. This
method will still work in parallel with the cl argument.
NOTE: Only the Lambert technique of type = "s" (skew) ensures that the transformation is consis-
tently 1-1, so it is the only method currently used in bestNormalize(). Use type = "h" or type =
’hh’ at risk of not having this estimate 1-1 transform. These alternative types are effective when
the data has exceptionally heavy tails, e.g. the Cauchy distribution. Additionally, as of v. 1.2.0,
Lambert of type "s" is not used by default in bestNormalize() since it uses multiple threads on
some Linux systems, which is not allowed on CRAN checks. Set allow_lambert_s = TRUE in order
to test this transformation as well. Note that the Lambert of type "h" can also be done by setting
allow_lambert_h = TRUE, however this can take significantly longer to run.
Value
A list of class bestNormalize with elements
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
See Also
Examples
x <- rgamma(100, 1, 1)
## Not run:
# With Repeated CV
BN_obj <- bestNormalize(x)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)
all.equal(x2, x)
## End(Not run)
## Not run:
# With leave-one-out CV
BN_obj <- bestNormalize(x, loo = TRUE)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)
all.equal(x2, x)
## End(Not run)
# Without CV
BN_obj <- bestNormalize(x, allow_orderNorm = FALSE, out_of_sample = FALSE)
BN_obj
p <- predict(BN_obj)
x2 <- predict(BN_obj, newdata = p, inverse = TRUE)
all.equal(x2, x)
8 binarize
binarize Binarize
Description
This function will perform a binarizing transformation, which could be used as a last resort if the
data cannot be adequately normalized. This may be useful when accidentally attempting normaliza-
tion of a binary vector (which could occur if implementing bestNormalize in an automated fashion).
Note that the transformation is not one-to-one, in contrast to the other functions in this package.
Usage
binarize(x, location_measure = "median")
Arguments
x A vector to binarize
location_measure
which location measure should be used? can either be "median", "mean", "mode",
a number, or a function.
object an object of class ’binarize’
newdata a vector of data to be (reverse) transformed
inverse if TRUE, performs reverse transformation
... additional arguments
Value
A list of class binarize with elements
x.t transformed original data
x original data
method location_measure used for original fitting
location estimated location_measure
n number of nonmissing observations
norm_stat Pearson’s P / degrees of freedom
The predict function with inverse = FALSE returns the numeric value (0 or 1) of the transforma-
tion on newdata (which defaults to the original data).
If inverse = TRUE, since the transform is not 1-1, it will create and return a factor that indicates
where the original data was cut.
boxcox 9
Examples
x <- rgamma(100, 1, 1)
binarize_obj <- binarize(x)
(p <- predict(binarize_obj))
Description
Usage
Arguments
Details
boxcox estimates the optimal value of lambda for the Box-Cox transformation. This transformation
can be performed on new data, and inverted, via the predict function.
The function will return an error if a user attempt to transform nonpositive data.
10 exp_x
Value
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
References
Box, G. E. P. and Cox, D. R. (1964) An analysis of transformations. Journal of the Royal Statistical
Society B, 26, 211-252.
See Also
boxcox
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Description
Usage
Arguments
Details
exp_x performs a simple exponential transformation in the context of bestNormalize, such that it
creates a transformation that can be estimated and applied to new data via the predict function.
Value
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
12 lambert
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Description
Perform Lambert’s W x F transformation and center/scale a vector to attempt normalization via the
LambertW package.
Usage
Arguments
Details
lambert uses the LambertW package to estimate a normalizing (or "Gaussianizing") transformation.
This transformation can be performed on new data, and inverted, via the predict function.
NOTE: The type = "s" argument is the only one that does the 1-1 transform consistently, and so
it is the only method currently used in bestNormalize(). Use type = "h" or type = ’hh’ at risk
of not having this estimate 1-1 transform. These alternative types are effective when the data has
exceptionally heavy tails, e.g. the Cauchy distribution.
Additionally, sometimes (depending on the distribution) this method will be unable to extrapolate
beyond the observed bounds. In these cases, NaN is returned.
Value
A list of class lambert with elements
x.t transformed original data
x original data
mean mean after transformation but prior to standardization
sd sd after transformation but prior to standardization
tau.mat estimated parameters of LambertW::Gaussianize
n number of nonmissing observations
norm_stat Pearson’s P / degrees of freedom
standardize was the transformation standardized
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
References
Georg M. Goerg (2016). LambertW: An R package for Lambert W x F Random Variables. R
package version 0.6.4.
Georg M. Goerg (2011): Lambert W random variables - a new family of generalized skewed distri-
butions with applications to risk estimation. Annals of Applied Statistics 3(5). 2197-2230.
Georg M. Goerg (2014): The Lambert Way to Gaussianize heavy-tailed data with the inverse of
Tukey’s h transformation as a special case. The Scientific World Journal.
See Also
Gaussianize
Examples
## Not run:
x <- rgamma(100, 1, 1)
all.equal(x2, x)
## End(Not run)
Description
Perform a log_b (x+a) normalization transformation
Usage
log_x(x, a = NULL, b = 10, standardize = TRUE, eps = 0.001,
warn = TRUE)
Arguments
x A vector to normalize with with x
a The constant to add to x (defaults to max(0, -min(x) + eps))
b The base of the log (defaults to 10)
standardize If TRUE, the transformed values are also centered and scaled, such that the
transformation attempts a standard normal
eps The allowed error in the expression for the selected a
warn Should a warning result from infinite values?
object an object of class ’log_x’
newdata a vector of data to be (potentially reverse) transformed
inverse if TRUE, performs reverse transformation
... additional arguments
Details
log_x performs a simple log transformation in the context of bestNormalize, such that it creates a
transformation that can be estimated and applied to new data via the predict function. The param-
eter a is essentially estimated by the training set by default (estimated as the minimum possible to
some extent epsilon), while the base must be specified beforehand.
no_transform 15
Value
A list of class log_x with elements
x.t transformed original data
x original data
mean mean after transformation but prior to standardization
sd sd after transformation but prior to standardization
a estimated a value
b estimated base b value
n number of nonmissing observations
norm_stat Pearson’s P / degrees of freedom
standardize was the transformation standardized
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Description
Perform an identity transformation. Admittedly it seems odd to have a dedicated function to essen-
tially do I(x), but it makes sense to keep the same syntax as the other transformations so it plays
nicely with them. As a benefit, the bestNormalize function will also show a comparable normaliza-
tion statistic for the untransformed data.
Usage
no_transform(x, standardize = FALSE, warn = TRUE)
Arguments
x A vector
standardize If TRUE, the transformed values are centered and scaled
warn Should a warning result from infinite values?
object an object of class ’no_transform’
newdata a vector of data to be (potentially reverse) transformed
inverse if TRUE, performs reverse transformation
... additional arguments
Details
no_transform creates a identity transformation object that can be applied to new data via the
predict function.
Value
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
orderNorm 17
Description
The Ordered Quantile (ORQ) normalization transformation, orderNorm(), is a rank-based proce-
dure by which the values of a vector are mapped to their percentile, which is then mapped to the
same percentile of the normal distribution. Without the presence of ties, this essentially guarantees
that the transformation leads to a uniform distribution.
The transformation is:
Where Φ refers to the standard normal cdf, rank(x) refers to each observation’s rank, and length(x)
refers to the number of observations.
By itself, this method is certainly not new; the earliest mention of it that I could find is in a 1947
paper by Bartlett (see references). This formula was outlined explicitly in Van der Waerden, and
expounded upon in Beasley (2009). However there is a key difference to this version of it, as
explained below.
Using linear interpolation between these percentiles, the ORQ normalization becomes a 1-1 trans-
formation that can be applied to new data. However, outside of the observed domain of x, it is
unclear how to extrapolate the transformation. In the ORQ normalization procedure, a binomial
glm with a logit link is used on the ranks in order to extrapolate beyond the bounds of the original
domain of x. The inverse normal CDF is then applied to these extrapolated predictions in order
to extrapolate the transformation. This mitigates the influence of heavy-tailed distributions while
preserving the 1-1 nature of the transformation. The extrapolation will provide a warning unless
warn = FALSE.) However, we found that the extrapolation was able to perform very well even on
data as heavy-tailed as a Cauchy distribution (paper to be published).
This transformation can be performed on new data and inverted via the predict function.
Usage
orderNorm(x, ..., warn = TRUE)
Arguments
x A vector to normalize
... additional arguments
18 orderNorm
Value
A list of class orderNorm with elements
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
References
Bartlett, M. S. "The Use of Transformations." Biometrics, vol. 3, no. 1, 1947, pp. 39-52. JSTOR
www.jstor.org/stable/3001536.
Van der Waerden BL. Order tests for the two-sample problem and their power. 1952;55:453-458.
Ser A.
Beasley TM, Erickson S, Allison DB. Rank-based inverse normal transformations are increasingly
used, but are they merited? Behav. Genet. 2009;39(5): 580-595. pmid:19526352
See Also
boxcox, lambert, bestNormalize, yeojohnson
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
plot.bestNormalize 19
Description
Plots transformation functions for objects produced by the bestNormalize package
Usage
## S3 method for class 'bestNormalize'
plot(x, inverse = FALSE, bounds = NULL,
cols = c("green3", 1, 2, 4:6), methods = c("boxcox", "yeojohnson",
"orderNorm", "lambert_s", "lambert_h"), leg_loc = "top", ...)
Arguments
x a fitted transformation
inverse if TRUE, plots the inverse transformation
bounds a vector of bounds to plot for the transformation
cols a vector of colors to use for the transforms (see details)
methods a vector of transformations to plot
leg_loc the location of the legend on the plot
... further parameters to be passed to plot and lines
Details
The plots produced by the individual transformations are simply plots of the original values by the
newly transformed values, with a line denoting where transformations would take place for new
data.
For the bestNormalize object, this plots each of the possible transformations run by the original
call to bestNormalize. The first argument in the "cols" parameter refers to the color of the chosen
transformation.
20 sqrt_x
Description
Perform a sqrt (x+a) normalization transformation
Usage
sqrt_x(x, a = NULL, standardize = TRUE, eps = 0.001)
Arguments
x A vector to normalize with with x
a The constant to add to x (defaults to max(0, -min(x) + eps))
standardize If TRUE, the transformed values are also centered and scaled, such that the
transformation attempts a standard normal
eps The allowed error in the expression for the selected a
object an object of class ’sqrt_x’
newdata a vector of data to be (potentially reverse) transformed
inverse if TRUE, performs reverse transformation
... additional arguments
Details
sqrt_x performs a simple square-root transformation in the context of bestNormalize, such that it
creates a transformation that can be estimated and applied to new data via the predict function.
The parameter a is essentially estimated by the training set by default (estimated as the minimum
possible to some extent epsilon), while the base must be specified beforehand.
Value
A list of class sqrt_x with elements
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Description
Perform a Yeo-Johnson Transformation and center/scale a vector to attempt normalization
Usage
yeojohnson(x, eps = 0.001, standardize = TRUE, ...)
Arguments
x A vector to normalize with Yeo-Johnson
eps A value to compare lambda against to see if it is equal to zero
standardize If TRUE, the transformed values are also centered and scaled, such that the
transformation attempts a standard normal
... Additional arguments that can be passed to the estimation of the lambda param-
eter (lower, upper)
object an object of class ’yeojohnson’
newdata a vector of data to be (reverse) transformed
inverse if TRUE, performs reverse transformation
22 yeojohnson
Details
yeojohnson estimates the optimal value of lambda for the Yeo-Johnson transformation. This trans-
formation can be performed on new data, and inverted, via the predict function.
The Yeo-Johnson is similar to the Box-Cox method, however it allows for the transformation of
nonpositive data as well. The step_YeoJohnson function in the recipes package is another useful
resource (see references).
Value
A list of class yeojohnson with elements
The predict function returns the numeric value of the transformation performed on new data, and
allows for the inverse transformation as well.
References
Yeo, I. K., & Johnson, R. A. (2000). A new family of power transformations to improve normality
or symmetry. Biometrika.
Max Kuhn and Hadley Wickham (2017). recipes: Preprocessing Tools to Create Design Matrices.
R package version 0.1.0.9000. https://github.com/topepo/recipes
Examples
x <- rgamma(100, 1, 1)
all.equal(x2, x)
Index
Gaussianize, 13
lambert, 12, 18
log_x, 14
no_transform, 15
orderNorm, 7, 17
plot.bestNormalize, 19
plot.boxcox (plot.bestNormalize), 19
plot.lambert (plot.bestNormalize), 19
plot.orderNorm (plot.bestNormalize), 19
plot.yeojohnson (plot.bestNormalize), 19
predict.arcsinh_x (arcsinh_x), 3
predict.bestNormalize (bestNormalize), 5
predict.binarize (binarize), 8
predict.boxcox (boxcox), 9
predict.exp_x (exp_x), 10
predict.lambert (lambert), 12
predict.log_x (log_x), 14
predict.no_transform (no_transform), 15
predict.orderNorm (orderNorm), 17
predict.sqrt_x (sqrt_x), 20
predict.yeojohnson (yeojohnson), 21
print.arcsinh_x (arcsinh_x), 3
print.bestNormalize (bestNormalize), 5
23