Conditional Correlations
Source:vignettes/conditional-correlations.Rmd
conditional-correlations.RmdIntroduction
Conditional correlations identify conditions under
which two numeric variables are strongly related. In
nuggets, the basic scheme is:
xvar ~ yvar | condition
This reads as: the variables xvar and yvar
are highly correlated in the sub-data satisfying the given
condition.
For example:
study_time ~ test_score | hard_exam
This means that for difficult exams, the amount of study time is strongly related to the obtained test score.
The dig_correlations() function searches over generated
conditions and computes correlations for all selected combinations of
numeric variables. It supports the standard correlation methods
implemented by stats::cor.test(),
namely Pearson, Kendall, and Spearman correlations.
Before going further, load the packages used in this vignette:
For a broader overview of the package workflow, see
vignette("nuggets").
A Small Working Dataset
dig_correlations() expects logical condition
columns and numeric variables to correlate.
The built-in iris dataset is convenient for this, because
it already contains several numeric measurements and one factor column
that can be transformed into condition predicates.
In the next example, we:
- convert
Speciesinto dummy logical predicates withpartition(), - add two logical helper conditions,
- add two derived numeric variables that will be useful in examples.
iris_corr <- iris |>
mutate(long_sepal = Sepal.Length >= median(Sepal.Length),
wide_petal = Petal.Width >= median(Petal.Width),
sepal_ratio = Sepal.Length / Sepal.Width,
petal_ratio = Petal.Length / Petal.Width) |>
partition(Species)
head(iris_corr, n = 3)
#> # A tibble: 3 × 11
#> Sepal.Length Sepal.Width Petal.Length Petal.Width long_sepal wide_petal
#> <dbl> <dbl> <dbl> <dbl> <lgl> <lgl>
#> 1 5.1 3.5 1.4 0.2 FALSE FALSE
#> 2 4.9 3 1.4 0.2 FALSE FALSE
#> 3 4.7 3.2 1.3 0.2 FALSE FALSE
#> sepal_ratio petal_ratio `Species=setosa` `Species=versicolor`
#> <dbl> <dbl> <lgl> <lgl>
#> 1 1.46 7 TRUE FALSE
#> 2 1.63 7 TRUE FALSE
#> 3 1.47 6.5 TRUE FALSE
#> `Species=virginica`
#> <lgl>
#> 1 FALSE
#> 2 FALSE
#> 3 FALSEThe dummy columns created from Species and the logical
helper columns can be used to generate conditions, while the original
and derived numeric columns can be used in the correlation tests.
For more information on preparing data for pattern extraction, see
vignette("data-preparation").
Basic Conditional Correlation Search
The simplest search chooses condition predicates and numeric variables, then limits the condition length and support:
corr_basic <- dig_correlations(iris_corr,
condition = where(is.logical),
xvars = c(Sepal.Length, Sepal.Width, sepal_ratio),
yvars = c(Petal.Length, Petal.Width, petal_ratio),
min_length = 0,
max_length = 2,
min_support = 0.2)
corr_basic |>
arrange(desc(abs(estimate))) |>
head(n = 6)
#> # A tibble: 6 × 10
#> condition support xvar yvar estimate
#> <chr> <dbl> <chr> <chr> <dbl>
#> 1 {} 1 Sepal.Length Petal.Length 0.872
#> 2 {Species=virginica,wide_petal} 0.333 Sepal.Length Petal.Length 0.864
#> 3 {Species=virginica} 0.333 Sepal.Length Petal.Length 0.864
#> 4 {long_sepal,Species=virginica} 0.313 Sepal.Length Petal.Length 0.846
#> 5 {} 1 sepal_ratio Petal.Length 0.838
#> 6 {} 1 Sepal.Length Petal.Width 0.818
#> p_value method alternative n
#> <dbl> <chr> <chr> <int>
#> 1 1.04e-47 Pearson's product-moment correlation two.sided 150
#> 2 6.30e-16 Pearson's product-moment correlation two.sided 50
#> 3 6.30e-16 Pearson's product-moment correlation two.sided 50
#> 4 6.86e-14 Pearson's product-moment correlation two.sided 47
#> 5 1.01e-40 Pearson's product-moment correlation two.sided 150
#> 6 2.33e-37 Pearson's product-moment correlation two.sided 150
#> condition_length
#> <int>
#> 1 0
#> 2 2
#> 3 1
#> 4 2
#> 5 0
#> 6 0The result is a tibble where each row represents one discovered pattern. The main columns are:
-
condition- the generated condition, -
support- relative frequency of the condition, -
xvar,yvar- the correlated variable pair, -
estimate- the correlation coefficient, -
p_value- significance of the test, -
n- number of rows in the corresponding sub-data, -
alternative,method- additional information about the test.
Controlling the Search Space
The condition, xvars, and
yvars arguments accept tidyselect
expressions. This makes it easy to restrict the search to specific
groups of columns.
As with other dig_*() functions, the search can be
controlled with min_length, max_length,
min_support, max_support, and
max_results.
For example, the following search uses only species predicates as conditions and correlates all sepal variables against all petal variables:
corr_species <- dig_correlations(iris_corr,
condition = starts_with("Species"),
xvars = starts_with("Sepal"),
yvars = starts_with("Petal"),
min_length = 1,
max_length = 1,
min_support = 0.3)
head(corr_species, n = 6)
#> # A tibble: 6 × 10
#> condition support xvar yvar estimate p_value
#> <chr> <dbl> <chr> <chr> <dbl> <dbl>
#> 1 {Species=setosa} 0.333 Sepal.Length Petal.Length 0.267 0.0607
#> 2 {Species=setosa} 0.333 Sepal.Length Petal.Width 0.278 0.0505
#> 3 {Species=setosa} 0.333 Sepal.Length petal_ratio -0.194 0.178
#> 4 {Species=setosa} 0.333 Sepal.Width Petal.Length 0.178 0.217
#> 5 {Species=setosa} 0.333 Sepal.Width Petal.Width 0.233 0.104
#> 6 {Species=setosa} 0.333 Sepal.Width petal_ratio -0.132 0.362
#> method alternative n condition_length
#> <chr> <chr> <int> <int>
#> 1 Pearson's product-moment correlation two.sided 50 1
#> 2 Pearson's product-moment correlation two.sided 50 1
#> 3 Pearson's product-moment correlation two.sided 50 1
#> 4 Pearson's product-moment correlation two.sided 50 1
#> 5 Pearson's product-moment correlation two.sided 50 1
#> 6 Pearson's product-moment correlation two.sided 50 1Choosing the Correlation Method
The method argument selects which correlation
coefficient is used:
-
"pearson"for linear relationships, -
"spearman"for monotone relationships based on ranks, -
"kendall"for rank-based association with Kendall’s tau.
Under the hood, every combination uses stats::cor.test()
with the corresponding method argument. The
estimate, statistic, p_value, and
confidence-interval columns in the result map directly to the values
returned by cor.test().
Here is the same basic search with Spearman correlation. As we are
O.K. with non-exact p-values, we can set exact = FALSE
(otherwise a warning is issued for small sub-data sizes):
corr_spearman <- dig_correlations(iris_corr,
condition = where(is.logical),
xvars = c(Sepal.Length, Sepal.Width),
yvars = c(Petal.Length, Petal.Width),
method = "spearman",
exact = FALSE,
min_length = 1,
max_length = 1,
min_support = 0.2)
head(corr_spearman, n = 6)
#> # A tibble: 6 × 10
#> condition support xvar yvar estimate p_value
#> <chr> <dbl> <chr> <chr> <dbl> <dbl>
#> 1 {wide_petal} 0.567 Sepal.Length Petal.Length 0.704 5.62e-14
#> 2 {wide_petal} 0.567 Sepal.Length Petal.Width 0.473 4.87e- 6
#> 3 {wide_petal} 0.567 Sepal.Width Petal.Length 0.302 4.91e- 3
#> 4 {wide_petal} 0.567 Sepal.Width Petal.Width 0.399 1.56e- 4
#> 5 {long_sepal} 0.513 Sepal.Length Petal.Length 0.654 1.08e-10
#> 6 {long_sepal} 0.513 Sepal.Length Petal.Width 0.440 6.14e- 5
#> method alternative n condition_length
#> <chr> <chr> <int> <int>
#> 1 Spearman's rank correlation rho two.sided 85 1
#> 2 Spearman's rank correlation rho two.sided 85 1
#> 3 Spearman's rank correlation rho two.sided 85 1
#> 4 Spearman's rank correlation rho two.sided 85 1
#> 5 Spearman's rank correlation rho two.sided 77 1
#> 6 Spearman's rank correlation rho two.sided 77 1If you are interested specifically in positive or negative
relationships, use the alternative argument with values
"greater" or "less".
Correlations on the Whole Dataset
If condition = NULL, dig_correlations()
computes correlations on the whole dataset only. This is useful when you
want the same output structure without searching over conditions.
corr_whole <- dig_correlations(
iris_corr,
condition = NULL,
xvars = starts_with("Sepal"),
yvars = starts_with("Petal")
)
corr_whole
#> # A tibble: 9 × 10
#> condition support xvar yvar estimate p_value
#> <chr> <dbl> <chr> <chr> <dbl> <dbl>
#> 1 {} 1 Sepal.Length Petal.Length 0.872 1.04e-47
#> 2 {} 1 Sepal.Length Petal.Width 0.818 2.33e-37
#> 3 {} 1 Sepal.Length petal_ratio -0.574 1.64e-14
#> 4 {} 1 Sepal.Width Petal.Length -0.428 4.51e- 8
#> 5 {} 1 Sepal.Width Petal.Width -0.366 4.07e- 6
#> 6 {} 1 Sepal.Width petal_ratio 0.368 3.71e- 6
#> 7 {} 1 sepal_ratio Petal.Length 0.838 1.01e-40
#> 8 {} 1 sepal_ratio Petal.Width 0.754 8.92e-29
#> 9 {} 1 sepal_ratio petal_ratio -0.612 8.78e-17
#> method alternative n condition_length
#> <chr> <chr> <int> <int>
#> 1 Pearson's product-moment correlation two.sided 150 0
#> 2 Pearson's product-moment correlation two.sided 150 0
#> 3 Pearson's product-moment correlation two.sided 150 0
#> 4 Pearson's product-moment correlation two.sided 150 0
#> 5 Pearson's product-moment correlation two.sided 150 0
#> 6 Pearson's product-moment correlation two.sided 150 0
#> 7 Pearson's product-moment correlation two.sided 150 0
#> 8 Pearson's product-moment correlation two.sided 150 0
#> 9 Pearson's product-moment correlation two.sided 150 0This is effectively a convenient way to compute a structured set of pairwise correlation tests between selected columns.
Notes on Interpretation
When reading discovered correlations, it is useful to keep the following points in mind:
- a large absolute
estimateindicates a stronger relationship, -
p_valuereflects statistical evidence for the chosen alternative, but should be interpreted with caution due to multiple comparisons (see below), -
supportandndescribe how much data contributed to the pattern, - longer conditions describe more specific contexts, but usually with smaller support.
Conditional correlations are therefore most useful when the relationship between two variables changes across subgroups and would be hidden in a single global correlation.
Multiple Comparisons
When dig_correlations() is applied to a dataset, it
simultaneously tests correlations for a potentially large number of
condition–variable-pair combinations. Each test produces a
p_value, but interpreting any individual
p_value as if it were the result of a single pre-planned
test is misleading: if hundreds of tests are performed at level 0.05,
several false discoveries are expected by chance alone.
The patterns returned by dig_correlations() are
therefore best understood as generated hypotheses -
promising associations that deserve further scrutiny - rather than as
confirmed findings. This is known as the problem of simultaneous
statistical inference or multiple comparisons.
A standard remedy is to adjust the p-values to control either the family-wise error rate (FWER) or the false discovery rate (FDR):
- FWER-controlling methods (e.g. Bonferroni, Holm) ensure that the probability of making any false discovery across all tests stays below a chosen threshold. They are conservative when the number of tests is large.
- FDR-controlling methods (e.g. Benjamini–Hochberg, abbreviated BH) allow a small proportion of discoveries to be false positives. This is less stringent than FWER control and retains more patterns in exploratory analyses.
R’s built-in p.adjust() function supports both families.
The example below applies Holm correction (FWER) and Benjamini–Hochberg
correction (FDR) to the result of a search:
corr_basic$p_holm <- p.adjust(corr_basic$p_value, method = "holm")
corr_basic$p_bh <- p.adjust(corr_basic$p_value, method = "BH")
corr_basic[, c("condition", "xvar", "yvar", "p_value", "p_holm", "p_bh")]
#> # A tibble: 90 × 6
#> condition xvar yvar p_value p_holm p_bh
#> <chr> <chr> <chr> <dbl> <dbl> <dbl>
#> 1 {} Sepal.Length Petal.Length 1.04e-47 9.35e-46 9.35e-46
#> 2 {} Sepal.Length Petal.Width 2.33e-37 2.05e-35 6.98e-36
#> 3 {} Sepal.Length petal_ratio 1.64e-14 1.34e-12 1.64e-13
#> 4 {} Sepal.Width Petal.Length 4.51e- 8 3.48e- 6 2.90e- 7
#> 5 {} Sepal.Width Petal.Width 4.07e- 6 3.01e- 4 2.16e- 5
#> 6 {} Sepal.Width petal_ratio 3.71e- 6 2.78e- 4 2.09e- 5
#> 7 {} sepal_ratio Petal.Length 1.01e-40 9.02e-39 4.56e-39
#> 8 {} sepal_ratio Petal.Width 8.92e-29 7.76e-27 2.01e-27
#> 9 {} sepal_ratio petal_ratio 8.78e-17 7.47e-15 1.32e-15
#> 10 {wide_petal} Sepal.Length Petal.Length 3.68e-18 3.16e-16 6.62e-17
#> # ℹ 80 more rowsAfter adjustment, you can filter by the corrected p-values:
corr_basic[corr_basic$p_bh < 0.05, ]
#> # A tibble: 54 × 12
#> condition support xvar yvar estimate p_value
#> <chr> <dbl> <chr> <chr> <dbl> <dbl>
#> 1 {} 1 Sepal.Length Petal.Length 0.872 1.04e-47
#> 2 {} 1 Sepal.Length Petal.Width 0.818 2.33e-37
#> 3 {} 1 Sepal.Length petal_ratio -0.574 1.64e-14
#> 4 {} 1 Sepal.Width Petal.Length -0.428 4.51e- 8
#> 5 {} 1 Sepal.Width Petal.Width -0.366 4.07e- 6
#> 6 {} 1 Sepal.Width petal_ratio 0.368 3.71e- 6
#> 7 {} 1 sepal_ratio Petal.Length 0.838 1.01e-40
#> 8 {} 1 sepal_ratio Petal.Width 0.754 8.92e-29
#> 9 {} 1 sepal_ratio petal_ratio -0.612 8.78e-17
#> 10 {wide_petal} 0.567 Sepal.Length Petal.Length 0.774 3.68e-18
#> method alternative n condition_length
#> <chr> <chr> <int> <int>
#> 1 Pearson's product-moment correlation two.sided 150 0
#> 2 Pearson's product-moment correlation two.sided 150 0
#> 3 Pearson's product-moment correlation two.sided 150 0
#> 4 Pearson's product-moment correlation two.sided 150 0
#> 5 Pearson's product-moment correlation two.sided 150 0
#> 6 Pearson's product-moment correlation two.sided 150 0
#> 7 Pearson's product-moment correlation two.sided 150 0
#> 8 Pearson's product-moment correlation two.sided 150 0
#> 9 Pearson's product-moment correlation two.sided 150 0
#> 10 Pearson's product-moment correlation two.sided 85 1
#> p_holm p_bh
#> <dbl> <dbl>
#> 1 9.35e-46 9.35e-46
#> 2 2.05e-35 6.98e-36
#> 3 1.34e-12 1.64e-13
#> 4 3.48e- 6 2.90e- 7
#> 5 3.01e- 4 2.16e- 5
#> 6 2.78e- 4 2.09e- 5
#> 7 9.02e-39 4.56e-39
#> 8 7.76e-27 2.01e-27
#> 9 7.47e-15 1.32e-15
#> 10 3.16e-16 6.62e-17
#> # ℹ 44 more rowsIn a large exploratory search you may prefer the FDR approach (BH) because it keeps more patterns visible while still limiting the expected fraction of false discoveries. Use FWER control (Holm) when you need stronger guarantees.
Related Tools
dig_correlations() is a specialized wrapper around
dig_grid(). If you need custom statistics on pairs of
variables under generated conditions, see
vignette("custom-patterns") for the more general
workflow.
For interactive inspection of discovered patterns, you can also use:
explore(corr_basic, iris_corr)Summary
This vignette showed how to search for conditional correlations with
nuggets:
- prepare logical condition predicates and numeric variables in one dataset,
- use
dig_correlations()to search over generated conditions, - select condition and variable columns with tidyselect expressions,
- control the search with support, condition length, and result limits,
- choose a correlation method appropriate for the data and interpretation.
For related material, see:
-
vignette("data-preparation")for creating condition predicates, -
vignette("contrast-patterns")for condition-dependent statistical differences, -
vignette("custom-patterns")for custom grid-based analyses, -
vignette("nuggets")for the package overview.