Large Tables

Nori predicts in context, so your table is input: every call reads all of X_train and keeps a per-layer key/value cache over those rows. That cache — not the model itself — is what fills a GPU on a big table. Nori handles it by walking a ladder of fallbacks, keeping the cache at full precision while it fits and stepping down only when it must. memory_policy lets you decide which steps it may take, and tells you afterwards which one it used. The same fields work locally and over the hosted API.

from synthefy_nori import MemoryPolicy, NoriRegressor

model = NoriRegressor(model="nori-6m", memory_policy=MemoryPolicy(cache_dtype="int8"))
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(model.memory_report_["rung"])   # which rung actually ran

from synthefy import MemoryPolicy, SynthefyNoriClient

client = SynthefyNoriClient(
    api_key="YOUR_API_KEY", mode="remote", model="synthefy/nori-6m"
)
predictions = client.predict(
    X_train, y_train, X_test,
    memory_policy=MemoryPolicy(cache_dtype="int8"),
)
print(client.last_memory_report["rung"])   # which rung actually ran

curl -X POST https://inference.baseten.co/predict \
  -H "Authorization: Api-Key $SYNTHEFY_NORI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "synthefy/nori-6m",
    "X_train": [[0.0, 1.0], [1.0, 0.0]],
    "y_train": [0.1, 0.9],
    "X_test": [[0.3, 0.7]],
    "memory_policy": {"cache_dtype": "int8"}
  }'

Locally this needs synthefy-nori 0.13.0 or newer; see Installation.

Quickstart

Omit memory_policy entirely and nothing changes — the defaults are what almost every call should use. They are: cache at full precision (bf16), let it drop to int8 only if that is what keeps it on the GPU, spend at most 40% of VRAM on it, offload to host RAM rather than give up, and shrink the context only as a last resort. In short: stay exact while that is possible, and never fail outright if there is any way to serve the request. Reach for a policy when you want a different trade. A preset covers the common three:

NoriRegressor(model="nori-6m", memory_policy="exact")        # never trade accuracy
NoriRegressor(model="nori-6m", memory_policy="max_context")  # fit the largest table
NoriRegressor(model="nori-6m", memory_policy="off")          # no cache at all

Which lever saves the most memory

Ranked by how much GPU memory each one frees, most first. Figures are from a 400-row × 8-feature context, whose full-precision cache is 0.0122 GiB — yours scales with rows and features, but the ratios hold.

Lever	GPU memory	Costs you
`offload_to_host=True` (default) or a small `gpu_budget_*`	drops to 0 — the cache leaves the GPU entirely	latency. Streaming it back per layer measured ~4× slower on this table. Bit-exact.
`cache_dtype="int8"`	~1.9× smaller (0.0122 → 0.0065 GiB)	accuracy, but barely: \|ΔR²\| ≈ 6e-6
`context_row_chunk=N`	bounds the build spike, not the final cache	a little speed. This is the one that fixes an out-of-memory error while the cache is being created
`elements_budget=N`	bounds the per-pass working set	more, smaller passes
`cache=False`	0 — no cache exists	a lot of speed: the context is re-read for every batch

Note that int8 is ~1.9× and not 4×: the saving is against bf16, not fp32, and the per-row scales cost a little back.

Row chunking

Use context_row_chunk when a call dies building the cache rather than holding it — the projection over all N rows at once is the spike, and this processes N a slice at a time. It is the lever for “it OOMs on fit”, where a smaller budget would not help.

NoriRegressor(
    model="nori-6m",
    memory_policy=MemoryPolicy(context_row_chunk=2048),
)

Nori also reaches for this automatically after an out-of-memory retry, so you rarely have to set it — see the warning in Good to know about its numerics.

Capping the budget on a shared GPU

Budgets are fractions of your hardware by default, so one setting travels between cards. Use the absolute form when a hard ceiling is the actual requirement, such as a GPU you share with another process:

NoriRegressor(
    model="nori-6m",
    memory_policy=MemoryPolicy(gpu_budget_absolute_gb=8.0),
)

Refusing a shortened context

By default Nori will drop context rows rather than fail. When a quietly shortened context would invalidate your results, make it an error instead:

NoriRegressor(
    model="nori-6m",
    memory_policy=MemoryPolicy(allow_subsample=False),
)

Either way, check memory_report_["dropped_context_rows"].

How it works

One predict reads the context once and saves a key/value vector per layer, per feature group, per context row. Query rows then attend to that cache instead of re-reading the table. The cache is what grows with your data:

cache bytes ≈ layers × groups × context_rows × 2 × embed_dim × bytes_per_value

Query rows are processed in batches, and the cache is what makes the second batch cheap. That is also why it is only built when there is a second batch — a query set small enough to fit one pass has nothing to reuse it across, and reports no_cache. That is normal, not a degradation. When the cache does not fit, Nori takes the cheapest step that works:

Rung	What happens	Accuracy
`resident_bf16`	full-precision cache in GPU memory	exact
`resident_int8`	quantized so it stays on the GPU	~1.9× smaller, \|ΔR²\| ≈ 6e-6
`offload_bf16`	full-precision cache in host RAM, streamed back per layer	exact, slower
`offload_int8`	quantized and in host RAM	as int8
`context_row_chunk`	cache built in row chunks after an out-of-memory retry	see note below
`plain_loop`	no cache; the context is re-read for every batch	exact, much slower
`no_cache`	the cached path did not apply	exact

Only the int8 rungs trade accuracy, and they are reached only when full precision will not fit. Offloading moves bytes rather than approximating.

Reading the report

memory_report_ locally, last_memory_report on the client, memory_report in the raw response. It carries the resolved policy plus what was decided:

Field	Meaning
`rung`	which fallback ran
`est_cache_gb`	full-precision cache footprint for this context
`resident_gb`	what stayed in GPU memory — `0` if offloaded or uncached
`query_chunk`	query rows per forward pass
`dropped_context_rows`	context rows discarded to fit — check this
`clamped`	hosted only — fields the server capped rather than honouring verbatim
`notes`	hosted only — remarks about the policy you sent

The last two are added by the server when it echoes the report, so they are present over the API and absent from memory_report_ locally. dropped_context_rows is the one number worth watching: it is the only accuracy loss here that is not a rounding-level effect. Set allow_subsample=False to make that case an error instead.

Reaching the cached path over the API

Coming soon. The cache is built only when your query set spans more than one batch, and today that threshold is more query rows than a single hosted request can carry — so a hosted call reports no_cache however much data you send. We are raising the hosted request limit; once it lifts, a normal-sized request reaches the cached path on its own, with nothing extra to set.

Configuration

Presets

Preset	Equivalent to	Use when
`exact`	`allow_quantization=False`	accuracy must not move; offload instead
`max_context`	`cache_dtype="int8"`	fit the largest table you can
`off`	`cache=False`	isolating a suspected cache problem

Precision

Field	Default	Meaning
`cache`	`True`	build a cache at all
`cache_dtype`	`"bf16"`	precision the cache starts at
`allow_quantization`	`True`	may drop to int8 if bf16 will not stay resident

Budgets

Budgets are fractions of your hardware, so one setting travels from a laptop GPU to an H200. The *_absolute_gb overrides exist for a shared GPU, where a hard ceiling is the requirement rather than a share.

Field	Default	Meaning
`gpu_budget_frac`	`0.4`	share of total VRAM the cache may occupy
`gpu_budget_absolute_gb`	`None`	hard VRAM ceiling in GiB, overriding the fraction
`offload_to_host`	`True`	may move the cache to host RAM
`host_budget_frac`	`0.25`	share of host RAM offload may claim
`host_budget_absolute_gb`	`None`	hard host-RAM ceiling in GiB

Over the hosted API the host-RAM budgets are capped to what the container survives, and a cap is reported in clamped rather than applied silently.

Chunking and context

Field	Default	Meaning
`elements_budget`	derived from VRAM	cap on (context + query) rows × features per pass
`adaptive_query_chunk`	`True`	halve the query batch and retry on an out-of-memory error
`context_row_chunk`	`None`	build the cache in chunks of this many context rows
`allow_subsample`	`True`	permit dropping context rows as a last resort

Good to know

Incoherent settings fail loudly. Asking for a cache-only field alongside cache=False is rejected rather than ignored, because a policy spelled correctly that silently does nothing is worse than an error. Settings that work but probably are not what you meant are honoured and explained in notes.

The report is not a policy. Feeding a memory_report back in as memory_policy is rejected: those are decided outputs, and reusing them as configuration would skip the coherence checks.

context_row_chunk is not bit-identical. A single chunk reproduces the unchunked result exactly, but a smaller chunk differs at around 4e-3 on a 512-row query set — the projection’s accumulation order depends on the chunk size. It is reached automatically after an out-of-memory retry, so a call that retries can return slightly different numbers.

Not on Thinking models. A Nori Thinking deployment rejects memory_policy with a 400. It predicts through a test-time-compute wrapper that manages its own inference memory, so a policy passed in from outside would not reach the run that produces the answer — and accepting it silently would be worse than refusing it.

Where a cache cannot help. Preprocessing (polynomial features and the adaptive SVD) runs before any of this and has its own memory cost, which memory_policy does not govern. A table that fails there fails regardless of the policy.

Get Started

Guides

Integration

Quickstart

Which lever saves the most memory

Row chunking

Capping the budget on a shared GPU

Refusing a shortened context

How it works

Reading the report

Reaching the cached path over the API

Configuration

Good to know

Next steps

Hosted client

Missing values

​Quickstart

​Which lever saves the most memory

​Row chunking

​Capping the budget on a shared GPU

​Refusing a shortened context

​How it works

​Reading the report

​Reaching the cached path over the API

​Configuration

​Good to know

​Next steps

Hosted client

Missing values

Quickstart

Which lever saves the most memory

Row chunking

Capping the budget on a shared GPU

Refusing a shortened context

How it works

Reading the report

Reaching the cached path over the API

Configuration

Good to know

Next steps