Pluggable classification backends • xplus

xplus() historically drove every pseudo-labeling iteration and the final refit through glmnet::cv.glmnet(). The development version turns that learner into a pluggable slot: any model that can fit weighted binary or fractional targets and return probabilities can drive the PLUS loop, and the elastic-net fit remains the default. This article describes the backend contract, the built-in learners, and the evaluation benchmark.

Built-in backends

Five backends ship with the package. Pass a name or a configured constructor to the backend argument of xplus():

backend	learner	notes
`"glmnet"`	`glmnet::cv.glmnet()`	default; honors `alpha` and `cv_measure`, records the CV measure in `history`
`"ranger"`	`ranger::ranger()`	probability forest for binary targets, regression forest for soft pseudo-labels; OOB `fit_predict`
`"gam"`	`mgcv::gam()`	one smoothing spline per feature; `max_features` guards the dimension cost
`"xgboost"`	`xgboost::xgb.train()`	logistic/regression objectives by target; out-of-fold `fit_predict`
`"lightgbm"`	`lightgbm::lgb.train()`	fastest tree backend; out-of-fold `fit_predict`

Constructors expose learner hyperparameters with parsnip-harmonized names, e.g. xplus_backend_ranger(trees = 1000, mtry = 4) or xplus_backend_xgboost(trees = 200, learn_rate = 0.05). ranger, mgcv, xgboost, and lightgbm are in Suggests, so install them to use those backends. A sixth path wraps any parsnip model spec directly:

library(parsnip)
backend <- rand_forest(trees = 500) |>
  set_engine("ranger", importance = "impurity") |>
  xplus_backend_parsnip()
fit <- xplus(x, y, backend = backend)

xplus_backend_parsnip() flips the spec’s mode per fit — classification on binary iterative targets, regression on the fractional final ones — maps case weights to hardhat weight vectors, injects engine seeds so xplus(seed = ...) stays reproducible, and reuses out-of-bag scores when the engine stores them. Specs whose model lacks a regression mode fail with a clear message at the final stage.

library(xplus)
sim <- simulate_pu_data(n = 400, p = 8, boundary = "nonlinear",
                        prevalence = 0.4, label_propensity = 0.5, seed = 11)
unl <- sim$y == 0

fit_glmnet <- xplus(sim$x, sim$y, backend = "glmnet", max_iter = 30, seed = 3)
fit_ranger <- xplus(sim$x, sim$y, backend = xplus_backend_ranger(trees = 300),
                    max_iter = 30, seed = 3)

data.frame(
  backend = c("glmnet", "ranger"),
  auc_unlabeled = c(auc(sim$z[unl], as.numeric(predict(fit_glmnet))[unl]),
                    auc(sim$z[unl], as.numeric(predict(fit_ranger))[unl])),
  cutoff = c(fit_glmnet$cutoff, fit_ranger$cutoff),
  n_iter = c(fit_glmnet$n_iter, fit_ranger$n_iter)
)
#>   backend auc_unlabeled   cutoff n_iter
#> 1  glmnet     0.5000000 0.581250     12
#> 2  ranger     0.8932671 0.140136      9

On a nonlinear boundary the elastic-net logistic fit collapses toward chance while the forest tracks the radial structure; on linear boundaries the ranking reverses, which is exactly why the learner is now a choice rather than a constant.

The backend contract

xplus_backend() defines the minimal estimator interface — fit weighted (x, y) data, then predict probabilities — shared by caret modelInfo entries, sklearn fit/predict_proba estimators, and mlr3 Learners:

field	signature	role
`fit`	`fit(x, y, foldid, weights)`	fit binary or fractional targets; attach `attr(model, "xplus_cv_measure")` to report `"deviance"` or `"auc"` in `history`
`predict`	`predict(model, newx)`	finite probabilities in `[0, 1]` per row
`importance`	`importance(model)`	optional named feature scores for `get_importance()`
`fit_predict`	`fit_predict(model, x)`	optional honest predictions on the fitted rows (see below)
`uses_cv`, `parameters`	—	metadata: CV usage and tunable parameters (caret `modelInfo` style), the latter reserved for a future tuning bridge

constant <- xplus_backend(
  name = "constant",
  fit = function(x, y, foldid, weights) stats::weighted.mean(y, weights),
  predict = function(model, newx) rep(model, nrow(newx))
)
fit <- xplus(sim$x, sim$y, backend = constant, max_iter = 3)
fit$backend$name
#> [1] "constant"

Honest training predictions

A subtlety of the contract: xplus() stores training predictions in object$pred_y and calibrates the classification cutoff as the qq quantile of positive predictions. High-capacity learners memorize their own training targets, so in-sample scores for labeled positives sit far above anything reachable — with a forest, the cutoff landed above every unlabeled score and F1 collapsed to zero.

Backends may therefore declare fit_predict(model, x): honest estimates on the rows they were fitted on, in the same order. The ranger backend supplies out-of-bag predictions; rows never out-of-bag fall back to predict. The iterative pseudo-label fits deliberately keep plain predict, since the self-training update relies on in-sample scores, but the stored predictions and cutoff are honest — and predict(fit) without newx returns those stored scores for non-glmnet backends, so training-data evaluation matches what the model produces on unseen data.

Benchmark

inst/benchmarks/backends/compare_backends.R fits every backend on simulate_pu_data() boundaries (linear, interaction, nonlinear) and scores predictions against the hidden labels, overall and inside the unlabeled pool:

Rscript inst/benchmarks/backends/compare_backends.R [outdir] [reps] [quick]

Representative results (3 replicates, n = 1000, p = 12, prevalence 0.4, label propensity 0.5):

backend	boundary	AUC (unlabeled)	F1 (unlabeled)	seconds
glmnet	linear	0.96	0.79	0.4
ranger	linear	0.92	0.68	2.0
gam	linear	0.95	0.79	22.0
xgboost	linear	0.85	0.59	3.0
lightgbm	linear	0.91	0.67	2.9
glmnet	interaction	0.72	0.48	0.4
ranger	interaction	0.74	0.48	2.1
gam	interaction	0.66	0.47	163.0
xgboost	interaction	0.72	0.47	2.8
lightgbm	interaction	0.74	0.49	2.9
glmnet	nonlinear	0.50	0.27	0.5
ranger	nonlinear	0.91	0.69	2.2
gam	nonlinear	0.93	0.77	23.9
xgboost	nonlinear	0.83	0.55	1.3
lightgbm	nonlinear	0.89	0.65	0.9

ranger is competitive on every boundary and roughly ten times faster than gam, lightgbm is the fastest learner after glmnet, and glmnet remains the cheapest option while still winning cleanly on linear structure. The boosting backends work out of the box but their default hyperparameters are untuned — the nonlinear AUC gap versus gam is exactly what a tuning pass should close.

The honest cutoff transfers to held-out data: ranger reaches test-set F1 of 0.76–0.80 with the same stored cutoff, which the memorized in-sample calibration could not.

Model selection without negatives

xplus_grid() fits every candidate on shared bootstrap seeds and ranks them by pu_score() — the positive-vs-unlabeled AUC of the honest stored predictions, which orders learners consistently under SCAR labeling:

xplus_grid(x, y, c("glmnet", "ranger", "lightgbm"))
xplus_grid(x, y, xplus_backend_xgboost,
           params = expand.grid(trees = c(100, 300), learn_rate = c(0.05, 0.15)))

On the lacsSample dataset (600 curated PU text documents reduced to a 300-term TF-IDF matrix, 153 positives vs 447 unlabeled) the PU ranking recovers the expected text-model ordering — glmnet 0.999 > ranger 0.966 > lightgbm 0.958 > xgboost 0.938 — and ranks xgboost hyperparameter settings correctly (300 trees at learn_rate 0.05 beat the smaller, faster settings). Linear learners win on sparse text; forests win on curved boundaries. The leaderboard makes that choice an empirical one.

The same pattern holds on open binary datasets PU-ified with pu_mask() — on UCI spambase (ranger 0.852 ≈ glmnet 0.845 > xgboost 0.807) and mushroom (xgboost 0.843 > lightgbm 0.836 > glmnet 0.818), the winner changes with the feature geometry, and every ranking was produced without ever reading the hidden negatives. Covertype (40k-row subsample of the UCI forest-cover data, lodgepole pine vs rest) confirms it again — xgboost 0.756 > ranger 0.744 > lightgbm 0.740 > glmnet 0.698 — three datasets, three different leaderboards.

Tuning with tidymodels

pu_xplus() registers xplus as a first-class parsnip model, so the whole tidymodels tuning stack drives the backend contract directly. Engine parameters become tune() placeholders, and any yardstick metric computed on the observed labels is PU-valid — the labeled positives vs the unlabeled-as-negative AUC is exactly pu_score():

library(parsnip); library(tune); library(finetune)
library(workflows); library(rsample); library(yardstick)

wf <- workflow(y ~ ., spec = pu_xplus(backend = "ranger", seed = 42,
                                      trees = tune(), mtry = tune()))
tune_grid(wf, resamples = vfold_cv(d, v = 5, strata = y),
          grid = 10, metrics = metric_set(roc_auc))

tune_sim_anneal(wf, resamples = vfold_cv(d, v = 5, strata = y),
                param_info = parameters(trees(c(50, 500)),
                                        finalize(mtry(c(1, 10)), d)),
                iter = 20, metrics = metric_set(roc_auc))

Registration is automatic when parsnip is installed; the wrapper enforces the PU label contract (factor 0/1, data-frame or matrix predictors), keeps xplus() seeds explicit, and filters backend-foreign arguments with a warning so mixed-backend grids remain usable.

Head-to-head against published PU methods

xplus_elkan_noto() and xplus_naive() express classical PU baselines on the same backend contract — Elkan-Noto (2008) rescaled P-vs-U classification and the naive U-as-negative classifier — so every method shares the honest prediction machinery. On held-out test labels (SCAR masking, label_propensity = 0.5, 70/30 stratified split, mean of 3 reps):

dataset	published SOTA	best ours	margin
mushroom	PU ET 99.70%	elkan_noto–ranger 99.99%	+0.29
covtype	PU ET 76.51%	elkan_noto–ranger 81.10%, xplus–xgboost 80.86%	+4.59
spambase	(TED)ⁿ 89.40%	xplus–glmnet 91.24%, elkan_noto–glmnet 90.97%	+1.84

The naive baseline collapses everywhere (≤77%), confirming that PU methods recover most of what supervised discrimination offers. Two findings matter beyond the leaderboard: on near-separable data (mushroom) the PLUS quantile cutoff leaves accuracy on the table — its oracle threshold reaches 100% where the SCAR cutoff yields 94% — while Elkan-Noto’s propensity rescaling closes that gap; and xplus edges Elkan-Noto where pseudo-label iteration pays (spambase), so the two are complementary rather than competing. Published numbers come from slightly different protocols, so treat exact deltas as indicative rather than definitive.

What is next

With tune/finetune integrated, the open direction is benchmark depth: held-out evaluation across more PU-ified datasets, convergence-aware stopping for boosting backends, and comparison against PU baselines such as Elkan–Noto rescaling — all on top of the honest-prediction infrastructure the backend contract already provides.