Large Tables

Nori predicts in context, so your table is input: every call reads all of X_train and keeps a per-layer key/value cache over those rows. That cache — not the model itself — is what fills a GPU on a big table. Nori handles it by walking a ladder of fallbacks, keeping the cache at full precision while it fits and stepping down only when it must. memory_policy lets you decide which steps it may take, and tells you afterwards which one it used. memory_policy governs how the cache holds your context. Above a row-count threshold, large_context_policy governs which context rows a call actually uses — routing across one or more Nori calls instead of trimming randomly or paying the VRAM cost of the whole table. See context routing below. The same fields work locally and over the hosted API.

from synthefy_nori import MemoryPolicy, NoriRegressor

model = NoriRegressor(model="nori-30m", memory_policy=MemoryPolicy(cache_dtype="int8"))
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(model.memory_report_["rung"])   # which rung actually ran

from synthefy import MemoryPolicy, SynthefyNoriClient

client = SynthefyNoriClient(
    api_key="YOUR_API_KEY", mode="remote", model="synthefy/nori-30m"
)
predictions = client.predict(
    X_train, y_train, X_test,
    memory_policy=MemoryPolicy(cache_dtype="int8"),
)
print(client.last_memory_report["rung"])   # which rung actually ran

curl -X POST https://inference.baseten.co/predict \
  -H "Authorization: Bearer $SYNTHEFY_NORI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "synthefy/nori-30m",
    "X_train": [[0.0, 1.0], [1.0, 0.0]],
    "y_train": [0.1, 0.9],
    "X_test": [[0.3, 0.7]],
    "memory_policy": {"cache_dtype": "int8"}
  }'

Locally this needs synthefy-nori 0.13.0 or newer; see Installation.

Quickstart

Omit memory_policy entirely and nothing changes — the defaults are what almost every call should use. They are: cache at full precision (bf16), let it drop to int8 only if that is what keeps it on the GPU, spend at most 40% of VRAM on it, offload to host RAM rather than give up, and shrink the context only as a last resort. In our tests, changing chunking or batching introduced only small rounding differences, though predictions were not always bit-for-bit identical. Reach for a policy when you want a different trade. A preset covers the common three:

NoriRegressor(model="nori-30m", memory_policy="exact")        # disable cache quantization
NoriRegressor(model="nori-30m", memory_policy="max_context")  # fit the largest table
NoriRegressor(model="nori-30m", memory_policy="off")          # no cache at all

Which lever saves the most memory

Ranked by how much GPU memory each one frees, most first. Figures are from a 400-row × 8-feature context, whose full-precision cache is 0.0122 GiB — yours scales with rows and features, but the ratios hold.

Lever	GPU memory	Costs you
`offload_to_host=True` (default) or a small `gpu_budget_*`	drops to 0 — the cache leaves the GPU entirely	latency. Streaming it back per layer measured ~4× slower on this table. Identical to the same resident cached computation.
`cache_dtype="int8"`	~1.9× smaller (0.0122 → 0.0065 GiB)	accuracy, but barely: \|ΔR²\| ≈ 6e-6
`context_row_chunk=N`	bounds the build spike, not the final cache	speed and possible floating-point rounding differences. This is the one that fixes an out-of-memory error while the cache is being created
`elements_budget=N`	bounds the per-pass working set	more, smaller passes
`cache=False`	0 — no cache exists	a lot of speed: the context is re-read for every batch

Note that int8 is ~1.9× and not 4×: the saving is against bf16, not fp32, and the per-row scales cost a little back.

Row chunking

Use context_row_chunk when a call dies building the cache rather than holding it — the projection over all N rows at once is the spike, and this processes N a slice at a time. It is the lever for “it OOMs on fit”, where a smaller budget would not help.

NoriRegressor(
    model="nori-30m",
    memory_policy=MemoryPolicy(context_row_chunk=2048),
)

Nori also reaches for this automatically after an out-of-memory retry, so you rarely have to set it — see the warning in Good to know about its numerics.

Capping the budget on a shared GPU

Budgets are fractions of your hardware by default, so one setting travels between cards. Use the absolute form when a hard ceiling is the actual requirement, such as a GPU you share with another process:

NoriRegressor(
    model="nori-30m",
    memory_policy=MemoryPolicy(gpu_budget_absolute_gb=8.0),
)

Refusing a shortened context

Shrinking the context is the last resort on the ladder: Nori reaches for it only when every other fallback still leaves the request too large, and it never fails outright if there is any way to serve it — see Quickstart. Make that an error instead when a silently shortened context would invalidate your results:

NoriRegressor(
    model="nori-30m",
    memory_policy=MemoryPolicy(allow_subsample=False),
)

allow_subsample=False raises before any prediction runs, rather than returning one computed on fewer rows than you asked for. Either way, check memory_report_["dropped_context_rows"].

How it works

One predict reads the context once and saves a key/value vector per layer, per feature group, per context row. Query rows then attend to that cache instead of re-reading the table. The cache is what grows with your data:

cache bytes ≈ layers × groups × context_rows × 2 × embed_dim × bytes_per_value

Query rows are processed in batches, and the cache is what makes the second batch cheap. That is also why it is only built when there is a second batch — a query set small enough to fit one pass has nothing to reuse it across, and reports no_cache. That is normal, not a degradation. When the cache does not fit, Nori takes the cheapest step that works:

Rung	What happens	Accuracy
`resident_bf16`	full-precision cache in GPU memory	unchunked cached baseline
`resident_int8`	quantized so it stays on the GPU	~1.9× smaller, \|ΔR²\| ≈ 6e-6
`offload_bf16`	full-precision cache in host RAM, streamed back per layer	identical to the same resident cached computation; slower
`offload_int8`	quantized and in host RAM	as int8
`context_row_chunk`	cache built in row chunks after an out-of-memory retry	see note below
`plain_loop`	no cache; the context is re-read for every batch	rounding can differ from cached execution; context may be subsampled
`no_cache`	the cached path did not apply	rounding can differ from cached execution

Only the int8 rungs quantize the cache. Ordinary BF16 offloading moves bytes unchanged: its reference is the same resident cached computation with the same context, query chunks, projection chunks, ensemble batching, model, and numerical backend. The policy can change batching eligibility, so selecting offload_bf16 alone does not guarantee identical end-to-end predictions. Changing chunk sizes, using streaming, or switching between cached and uncached execution can change floating-point rounding even without quantization. Subsampling changes the context itself. The exact preset disables cache quantization. It does not guarantee identical predictions across execution paths or prevent subsampling. Use allow_subsample=False separately when every context row must be retained.

Reading the report

memory_report_ locally, last_memory_report on the client, memory_report in the raw response. It carries the resolved policy plus what was decided:

Field	Meaning
`rung`	which fallback ran
`est_cache_gb`	full-precision cache footprint for this context
`resident_gb`	what stayed in GPU memory — `0` if offloaded or uncached
`query_chunk`	query rows per forward pass
`dropped_context_rows`	context rows discarded to fit — check this
`clamped`	hosted only — fields the server capped rather than honouring verbatim
`notes`	hosted only — remarks about the policy you sent

The last two are added by the server when it echoes the report, so they are present over the API and absent from memory_report_ locally. dropped_context_rows is the one number worth watching: it is the only accuracy loss here that is not a rounding-level effect. Set allow_subsample=False to make that case an error instead.

Reaching the cached path over the API

Coming soon. The cache is built only when your query set spans more than one batch, and today that threshold is more query rows than a single hosted request can carry — so a hosted call reports no_cache however much data you send. We are raising the hosted request limit; once it lifts, a normal-sized request reaches the cached path on its own, with nothing extra to set.

Choosing context rows above a threshold

memory_policy decides how the cache holds your context; large_context_policy decides which rows go into it once X_train exceeds large_context_threshold (50,000 rows by default). Above that threshold, a policy routes the request across one or more Nori calls instead of the cache silently trimming a random subset to fit. Off by default — large_context_policy=None keeps behavior byte-for-byte identical to today. The same built-in policy specs work in local, hosted, and SageMaker modes:

from synthefy_nori import NoriRegressor

model = NoriRegressor(model="nori-30m", large_context_policy="cluster_route")
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(model.large_context_report_["nori_calls"])   # how many calls it made

from synthefy import SynthefyNoriClient

client = SynthefyNoriClient(
    api_key="YOUR_API_KEY", mode="remote", model="synthefy/nori-30m"
)
predictions = client.predict(
    X_train, y_train, X_test,
    large_context_policy="cluster_route",
)
print(client.last_large_context_report["nori_calls"])

curl -X POST https://inference.baseten.co/predict \
  -H "Authorization: Bearer $SYNTHEFY_NORI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "synthefy/nori-30m",
    "X_train": [[0.0, 1.0], [1.0, 0.0]],
    "y_train": [0.1, 0.9],
    "X_test": [[0.3, 0.7]],
    "large_context_policy": "cluster_route"
  }'

`cluster_route` — recommended

Clusters the query rows into groups (8 by default) and gives each group its own local context pool, one Nori call per group. It is the only policy with full coverage across a broad benchmark sweep — best on nearly half the tables tested, and it never regressed below a single shared window.

Choosing between them

Here are some commonly used built-in policies:

Policy	Nori calls	Use when
`random`	1	the cheapest fallback — one shared context window, no routing
`cluster_route`	up to 8	the default choice for most tables
`cluster_route_g4`	up to 4	cheaper than `cluster_route` on tables it was measured against; not a safe blanket default
`safeboost`	one per context shard	opt-in residual boosting with an out-of-fold guard; prefer it to `boost`
`boost`	one per stage	only after validating it on your table; measured regressions can be severe

Hosted and SageMaker support built-ins from the installed Nori version. The complete current list and configuration options live in src/synthefy_nori/inference/policies.py in the Synthefy Nori repository. Custom Python policies are local-only.

Reading the report

large_context_report_ locally, last_large_context_report on the client, large_context_report in the raw response — present whenever a policy was requested, even below the threshold, so a deployment silently ignoring the field is distinguishable from an ordinary small-table request.

Field	Meaning
`applied`	whether the table was above the threshold
`policy`	the built-in policy string the server resolved and ran
`threshold` / `seed`	the bounded controls used for this call
`window`	context rows per Nori call
`nori_calls`	internal Nori calls this request made
`shards_available`	full policy windows the table contains
`reused_train_state`	always `false` over the hosted API — see Good to know

Configuration

Presets

Preset	Equivalent to	Use when
`exact`	`allow_quantization=False`	cache quantization must be disabled; offload instead
`max_context`	`cache_dtype="int8"`	fit the largest table you can
`off`	`cache=False`	isolating a suspected cache problem

Precision

Field	Default	Meaning
`cache`	`True`	build a cache at all
`cache_dtype`	`"bf16"`	precision the cache starts at
`allow_quantization`	`True`	may drop to int8 if bf16 will not stay resident

Budgets

Budgets are fractions of your hardware, so one setting travels from a laptop GPU to an H200. The *_absolute_gb overrides exist for a shared GPU, where a hard ceiling is the requirement rather than a share.

Field	Default	Meaning
`gpu_budget_frac`	`0.4`	share of total VRAM the cache may occupy
`gpu_budget_absolute_gb`	`None`	hard VRAM ceiling in GiB, overriding the fraction
`offload_to_host`	`True`	may move the cache to host RAM
`host_budget_frac`	`0.25`	share of host RAM offload may claim
`host_budget_absolute_gb`	`None`	hard host-RAM ceiling in GiB

Over the hosted API the host-RAM budgets are capped to what the container survives, and a cap is reported in clamped rather than applied silently.

Chunking and context

Field	Default	Meaning
`elements_budget`	derived from VRAM	cap on (context + query) rows × features per pass
`adaptive_query_chunk`	`True`	halve the query batch and retry on an out-of-memory error
`context_row_chunk`	`None`	build the cache in chunks of this many context rows
`allow_subsample`	`True`	permit dropping context rows as a last resort

large_context_policy

Field	Default	Meaning
`large_context_policy`	`None`	a built-in name or parameter string; `None` disables routing
`large_context_threshold`	`50000`	context-row count strictly above which the policy engages (1 to 10,000,000)
`large_context_seed`	`0`	deterministic policy seed (0 to 2**32 − 1)

large_context_cache_entries (NoriRegressor only, default 1) keeps more than one policy’s context cache resident between calls on the same estimator. It is not part of the shared client/hosted contract: hosted requests are one-shot, so there is nothing to retain between them.

Good to know

Incoherent settings fail loudly. Asking for a cache-only field alongside cache=False is rejected rather than ignored, because a policy spelled correctly that silently does nothing is worse than an error. Settings that work but probably are not what you meant are honoured and explained in notes.

The report is not a policy. Feeding a memory_report back in as memory_policy is rejected: those are decided outputs, and reusing them as configuration would skip the coherence checks.

Row chunking is mathematically equivalent, but not guaranteed bit-identical. Changing the projection shape can change floating-point accumulation order. A single chunk uses the unchunked projection shape; smaller chunks may return slightly different predictions. Row chunking can also be reached automatically after an out-of-memory retry. The size of any difference depends on the model, inputs, precision, and backend; it is not a fixed error bound.

Not on Thinking models. A Nori Thinking deployment rejects both memory_policy and large_context_policy with a 400. It predicts through a test-time-compute wrapper with no separate fit/predict step for a policy to attach to, and manages its own inference memory — so a policy passed in from outside would not reach the run that produces the answer, and accepting it silently would be worse than refusing it.

Where a cache cannot help. Preprocessing (polynomial features and the adaptive SVD) runs before any of this and has its own memory cost, which memory_policy does not govern. A table that fails there fails regardless of the policy.

large_context_policy is point-output only. output_type="mean" and "median" are supported; "quantiles" and "full" fail before inference, because a routed policy combines point predictions from several Nori calls and has no single combined predictive distribution.

Hosted requests fail closed on a stale deployment. The response always carries large_context_report when a policy was requested, even below the threshold. If a deployment omits it or reports a different policy, threshold, or seed, the client raises rather than silently returning an ordinary, unrouted prediction that looks valid.

One-shot everywhere. Every call — local, hosted, or SageMaker — supplies and fits X_train again; there is no hidden upload-once/query-many cache across requests. reused_train_state in the report reflects reuse within one client-owned NoriRegressor instance in local mode only, and is always false over the hosted API.

Get Started

Guides

Integration

Quickstart

Which lever saves the most memory

Row chunking

Capping the budget on a shared GPU

Refusing a shortened context

How it works

Reading the report

Reaching the cached path over the API

Choosing context rows above a threshold

`cluster_route` — recommended

Choosing between them

Reading the report

Configuration

Good to know

Next steps

Hosted client

Missing values

​Quickstart

​Which lever saves the most memory

​Row chunking

​Capping the budget on a shared GPU

​Refusing a shortened context

​How it works

​Reading the report

​Reaching the cached path over the API

​Choosing context rows above a threshold

​cluster_route — recommended

​Choosing between them

​Reading the report

​Configuration

​Good to know

​Next steps

Hosted client

Missing values

Quickstart

Which lever saves the most memory

Row chunking

Capping the budget on a shared GPU

Refusing a shortened context

How it works

Reading the report

Reaching the cached path over the API

Choosing context rows above a threshold

`cluster_route` — recommended

Choosing between them

Reading the report

Configuration

Good to know

Next steps