Embeddings - Synthefy Docs

Every time Nori looks at a row, it builds a dense vector for it, a learned representation shaped by how that row relates to the target and to the rest of your table. Pull those vectors out and they become features you can drop into anything downstream.

Downstream models

Feed the vectors to a kNN, linear probe, or gradient-boosting head, often matching Nori’s own accuracy with a model you fully control.

Similarity & search

Nearest-neighbor lookup in embedding space finds rows that are behaviorally alike, not just numerically close.

Clustering

Cluster on the embeddings to segment rows by what actually drives the target.

Visualization

Project to 2-D and watch the target structure appear. See the gallery below.

Nori is regression-only, so every embedding is conditioned on a continuous target. Extraction runs on the local synthefy-nori package (not the hosted API) and needs no API key. Weights download from Hugging Face on first use.

Extract embeddings in four steps

Install the package

The base install is all you need to extract vectors and fit downstream heads (scikit-learn is a core dependency).

pip install synthefy-nori

Fit and pull out-of-fold vectors

NoriEmbedding is a scikit-learn transformer. With n_fold >= 2, fit_transform returns leak-free training embeddings (each row embedded by a model that never saw its label), and transform embeds unseen rows with the final full-data model.

from synthefy_nori import NoriEmbedding

embedder  = NoriEmbedding(n_fold=5, shuffle=True, random_state=0)
train_emb = embedder.fit_transform(X_train, y_train)   # out-of-fold
test_emb  = embedder.transform(X_test)                 # full-data model

Collapse the ensemble to a 2-D matrix

Both calls return a 3-D array (n_estimators, n_samples, embed_dim), one slice per preprocessing pipeline in the inference ensemble. Average over axis 0 (or pick one member) for a matrix any estimator can consume.

Z_train = train_emb.mean(axis=0)   # (n_train, embed_dim)
Z_test  = test_emb.mean(axis=0)    # (n_test,  embed_dim)

Use them as features

The result is a plain 2-D matrix, so any scikit-learn estimator takes it from here, for example a nearest-neighbor regressor:

from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score

knn = KNeighborsRegressor(n_neighbors=10).fit(Z_train, y_train)
print(r2_score(y_test, knn.predict(Z_test)))

Pick your extraction API

NoriEmbedding (recommended)
get_embeddings (low-level)

The sklearn transformer. Handles the fold logic and the final full-data fit for you, so training embeddings stay leak-free.

Method	Returns
`fit_transform(X_train, y_train)`	training embeddings (out-of-fold when `n_fold >= 2`)
`transform(X)`	embeddings for unseen rows, always from the full-data model

from synthefy_nori import NoriEmbedding, NoriRegressor

# Hand it a pre-built estimator to reuse a config across the many fold-fits
# (e.g. skip compilation for lots of small folds).
embedder = NoriEmbedding(
    n_fold=5, shuffle=True, random_state=0,
    model=NoriRegressor(compile_model=False),
)
Z_train = embedder.fit_transform(X_train, y_train).mean(axis=0)
Z_test  = embedder.transform(X_test).mean(axis=0)

transform never returns cached training embeddings, even if the input equals the training set. For training-row vectors use fit_transform (or read embedder.train_embeddings_).

Direct control over which rows get embedded against which context. Call it on a fitted NoriRegressor; data_source selects what comes back.

`data_source`	Embeds
`"test"` (default)	the query rows you pass as `X`
`"train"`	the stored context rows, so `X` is ignored and can be omitted

from synthefy_nori import NoriRegressor

model     = NoriRegressor().fit(X_train, y_train)
test_emb  = model.get_embeddings(X_test, data_source="test")   # query rows
train_emb = model.get_embeddings(data_source="train")          # context rows

These come from the full-data model, so "train" rows encode their own labels. For leak-free training embeddings, use NoriEmbedding with n_fold >= 2.

Leak-free or fast: choose per use

The n_fold setting is the one decision that matters. It trades a bit of compute for an honest downstream score.

Out-of-fold (n_fold ≥ 2), for when you train on the rows

K-fold cross-validation: each training row is embedded by a fold model that did not have it in context, so the vector never leaks that row’s own label. Unseen rows are then embedded by a single model fit on the full training set. This is what gives a downstream probe an unbiased estimate of generalization (arXiv:2502.17361).

NoriEmbedding(n_fold=5, shuffle=True, random_state=0)

Vanilla (n_fold = 0), for a quick one-shot embed

One model is fit on the whole training set and used for every row. It is cheaper (a single fit instead of n_fold + 1), but the training embeddings encode their own labels. Fine for visualization or embedding brand-new rows; don’t fit a downstream model on vanilla training embeddings and trust the score.

NoriEmbedding(n_fold=0)

See it on real data

Because the vectors are organized by the target, a 2-D t-SNE of Nori embeddings separates rows by their target value far more cleanly than a t-SNE of the raw features. Both runnable examples below produce exactly these plots. Clone the repo, install the eval extra, and run:

pip install "synthefy-nori[eval]"   # adds matplotlib (+ openml for TabArena)

Synthetic dataset
Real dataset (TabArena)

examples/embedding_synthetic.py runs the workflow on a small synthetic regression target: it extracts embeddings, runs a kNN probe, and draws the t-SNE. Fast, with no data download.

python embedding_synthetic.py

synthetic regression: train=280 test=120 features=5
  get_embeddings -> (n_estimators, n_samples, embed_dim) = (8, 120, 128)
=== R2 ===
  kNN probe on embeddings  : 0.9954
  Nori native head (ref.)  : 0.9995

t-SNE of raw features vs Nori embeddings on a synthetic dataset

examples/embedding_tabarena.py runs the same workflow on a real TabArena regression dataset (a Ridge probe here). Pass --dataset to switch, or --list to see the options.

python embedding_tabarena.py --dataset wine_quality

wine_quality: train=3000 test=1500 features=12
  get_embeddings -> (n_estimators, n_samples, embed_dim) = (8, 1500, 128)
=== R2 ===
  Ridge probe on embeddings  : 0.5168
  Nori native head (ref.)    : 0.5148

t-SNE of raw features vs Nori embeddings on the wine_quality dataset

Rolling your own plot is three lines once you have Z_test:

from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler

emb_2d = TSNE(n_components=2, init="pca", random_state=0).fit_transform(
    StandardScaler().fit_transform(Z_test)
)   # scatter emb_2d colored by y_test

API reference

NoriEmbedding parameters

Parameter	Default	Description
`n_fold`	`0`	CV folds. `0` = vanilla (single model); `>= 2` = out-of-fold training embeddings.
`model`	`None`	A pre-configured `NoriRegressor`. Built at `fit` time when `None`.
`shuffle`	`False`	Shuffle the K-fold split.
`random_state`	`None`	Seed for the split when `shuffle=True`.

After fit, the transformer exposes model_ (the full-data model) and train_embeddings_ (out-of-fold when n_fold >= 2).

Downstream head snippets

A standardized ridge probe (good default for the higher-dimensional embeddings of real data):

from sklearn.linear_model import RidgeCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

probe = make_pipeline(StandardScaler(), RidgeCV(alphas=(0.1, 1.0, 10.0, 100.0)))
probe.fit(Z_train, y_train)

The same Z_train / Z_test matrices feed clustering (KMeans), nearest- neighbor search (NearestNeighbors), or any 2-D projection.

Shapes at a glance: embeddings are (n_estimators, n_samples, embed_dim). Average over axis=0 (or select [0]) before handing them to a 2-D estimator.

Downstream models

Similarity & search

Clustering

Visualization

​Extract embeddings in four steps

​Pick your extraction API

​Leak-free or fast: choose per use

​See it on real data

​API reference

Extract embeddings in four steps

Pick your extraction API

Leak-free or fast: choose per use

See it on real data

API reference