> ## Documentation Index
> Fetch the complete documentation index at: https://docs.synthefy.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Large Tables
> Serve context tables larger than GPU memory. Nori walks a ladder of fallbacks, and memory_policy lets you choose which ones it may use.
Nori predicts in context, so your table is *input*: every call reads all of
`X_train` and keeps a per-layer key/value cache over those rows. That cache —
not the model itself — is what fills a GPU on a big table. Nori handles it by
walking a ladder of fallbacks, keeping the cache at full precision while it fits
and stepping down only when it must. `memory_policy` lets you decide which steps
it may take, and tells you afterwards which one it used.
The same fields work locally and over the hosted API.
```python Local theme={null}
from synthefy_nori import MemoryPolicy, NoriRegressor
model = NoriRegressor(model="nori-6m", memory_policy=MemoryPolicy(cache_dtype="int8"))
model.fit(X_train, y_train)
predictions = model.predict(X_test)
print(model.memory_report_["rung"]) # which rung actually ran
```
```python Hosted theme={null}
from synthefy import MemoryPolicy, SynthefyNoriClient
client = SynthefyNoriClient(
api_key="YOUR_API_KEY", mode="remote", model="synthefy/nori-6m"
)
predictions = client.predict(
X_train, y_train, X_test,
memory_policy=MemoryPolicy(cache_dtype="int8"),
)
print(client.last_memory_report["rung"]) # which rung actually ran
```
```bash cURL theme={null}
curl -X POST https://inference.baseten.co/predict \
-H "Authorization: Api-Key $SYNTHEFY_NORI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "synthefy/nori-6m",
"X_train": [[0.0, 1.0], [1.0, 0.0]],
"y_train": [0.1, 0.9],
"X_test": [[0.3, 0.7]],
"memory_policy": {"cache_dtype": "int8"}
}'
```
Locally this needs `synthefy-nori` 0.13.0 or newer; see
[Installation](/setup/installation).
## Quickstart
**Omit `memory_policy` entirely and nothing changes** — the defaults are what
almost every call should use. They are: cache at full precision (`bf16`), let it
drop to `int8` only if that is what keeps it on the GPU, spend at most 40% of
VRAM on it, offload to host RAM rather than give up, and shrink the context only
as a last resort. In short: stay exact while that is possible, and never fail
outright if there is any way to serve the request.
Reach for a policy when you want a different trade. A preset covers the common
three:
```python theme={null}
NoriRegressor(model="nori-6m", memory_policy="exact") # never trade accuracy
NoriRegressor(model="nori-6m", memory_policy="max_context") # fit the largest table
NoriRegressor(model="nori-6m", memory_policy="off") # no cache at all
```
### Which lever saves the most memory
Ranked by how much GPU memory each one frees, most first. Figures are from a
400-row × 8-feature context, whose full-precision cache is 0.0122 GiB — yours
scales with rows and features, but the ratios hold.
| Lever | GPU memory | Costs you |
| ---------------------------------------------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `offload_to_host=True` (default) or a small `gpu_budget_*` | drops to **0** — the cache leaves the GPU entirely | latency. Streaming it back per layer measured \~4× slower on this table. Bit-exact. |
| `cache_dtype="int8"` | **\~1.9× smaller** (0.0122 → 0.0065 GiB) | accuracy, but barely: \|ΔR²\| ≈ 6e-6 |
| `context_row_chunk=N` | bounds the **build** spike, not the final cache | a little speed. This is the one that fixes an out-of-memory error *while* the cache is being created |
| `elements_budget=N` | bounds the per-pass working set | more, smaller passes |
| `cache=False` | **0** — no cache exists | a lot of speed: the context is re-read for every batch |
Note that int8 is \~1.9× and not 4×: the saving is against `bf16`, not `fp32`,
and the per-row scales cost a little back.
### Row chunking
Use `context_row_chunk` when a call dies building the cache rather than holding
it — the projection over all N rows at once is the spike, and this processes N a
slice at a time. It is the lever for "it OOMs on `fit`", where a smaller budget
would not help.
```python theme={null}
NoriRegressor(
model="nori-6m",
memory_policy=MemoryPolicy(context_row_chunk=2048),
)
```
Nori also reaches for this automatically after an out-of-memory retry, so you
rarely have to set it — see the warning in [Good to know](#good-to-know) about
its numerics.
### Capping the budget on a shared GPU
Budgets are fractions of your hardware by default, so one setting travels
between cards. Use the absolute form when a hard ceiling is the actual
requirement, such as a GPU you share with another process:
```python theme={null}
NoriRegressor(
model="nori-6m",
memory_policy=MemoryPolicy(gpu_budget_absolute_gb=8.0),
)
```
### Refusing a shortened context
By default Nori will drop context rows rather than fail. When a quietly
shortened context would invalidate your results, make it an error instead:
```python theme={null}
NoriRegressor(
model="nori-6m",
memory_policy=MemoryPolicy(allow_subsample=False),
)
```
Either way, check `memory_report_["dropped_context_rows"]`.
## How it works
One `predict` reads the context once and saves a key/value vector per layer, per
feature group, per context row. Query rows then attend to that cache instead of
re-reading the table. The cache is what grows with your data:
```
cache bytes ≈ layers × groups × context_rows × 2 × embed_dim × bytes_per_value
```
Query rows are processed in batches, and the cache is what makes the second
batch cheap. That is also why it is only built when there *is* a second batch —
a query set small enough to fit one pass has nothing to reuse it across, and
reports `no_cache`. That is normal, not a degradation.
When the cache does not fit, Nori takes the cheapest step that works:
| Rung | What happens | Accuracy |
| ------------------- | --------------------------------------------------------- | ------------------------------ |
| `resident_bf16` | full-precision cache in GPU memory | exact |
| `resident_int8` | quantized so it stays on the GPU | \~1.9× smaller, \|ΔR²\| ≈ 6e-6 |
| `offload_bf16` | full-precision cache in host RAM, streamed back per layer | exact, slower |
| `offload_int8` | quantized *and* in host RAM | as int8 |
| `context_row_chunk` | cache built in row chunks after an out-of-memory retry | see note below |
| `plain_loop` | no cache; the context is re-read for every batch | exact, much slower |
| `no_cache` | the cached path did not apply | exact |
Only the int8 rungs trade accuracy, and they are reached only when full
precision will not fit. Offloading moves bytes rather than approximating.
### Reading the report
`memory_report_` locally, `last_memory_report` on the client, `memory_report` in
the raw response. It carries the resolved policy plus what was decided:
| Field | Meaning |
| ---------------------- | ----------------------------------------------------------------------- |
| `rung` | which fallback ran |
| `est_cache_gb` | full-precision cache footprint for this context |
| `resident_gb` | what stayed in GPU memory — `0` if offloaded or uncached |
| `query_chunk` | query rows per forward pass |
| `dropped_context_rows` | context rows discarded to fit — **check this** |
| `clamped` | *hosted only* — fields the server capped rather than honouring verbatim |
| `notes` | *hosted only* — remarks about the policy you sent |
The last two are added by the server when it echoes the report, so they are
present over the API and absent from `memory_report_` locally.
`dropped_context_rows` is the one number worth watching: it is the only accuracy
loss here that is not a rounding-level effect. Set `allow_subsample=False` to
make that case an error instead.
### Reaching the cached path over the API
**Coming soon.** The cache is built only when your query set spans more than
one batch, and today that threshold is more query rows than a single hosted
request can carry — so a hosted call reports `no_cache` however much data you
send. We are raising the hosted request limit; once it lifts, a normal-sized
request reaches the cached path on its own, with nothing extra to set.
## Configuration
| Preset | Equivalent to | Use when |
| ------------- | -------------------------- | --------------------------------------- |
| `exact` | `allow_quantization=False` | accuracy must not move; offload instead |
| `max_context` | `cache_dtype="int8"` | fit the largest table you can |
| `off` | `cache=False` | isolating a suspected cache problem |
| Field | Default | Meaning |
| -------------------- | -------- | ----------------------------------------------- |
| `cache` | `True` | build a cache at all |
| `cache_dtype` | `"bf16"` | precision the cache starts at |
| `allow_quantization` | `True` | may drop to int8 if bf16 will not stay resident |
Budgets are fractions of your hardware, so one setting travels from a laptop GPU
to an H200. The `*_absolute_gb` overrides exist for a shared GPU, where a hard
ceiling is the requirement rather than a share.
| Field | Default | Meaning |
| ------------------------- | ------- | ------------------------------------------------- |
| `gpu_budget_frac` | `0.4` | share of total VRAM the cache may occupy |
| `gpu_budget_absolute_gb` | `None` | hard VRAM ceiling in GiB, overriding the fraction |
| `offload_to_host` | `True` | may move the cache to host RAM |
| `host_budget_frac` | `0.25` | share of host RAM offload may claim |
| `host_budget_absolute_gb` | `None` | hard host-RAM ceiling in GiB |
Over the hosted API the host-RAM budgets are capped to what the container
survives, and a cap is reported in `clamped` rather than applied silently.
| Field | Default | Meaning |
| ---------------------- | ----------------- | --------------------------------------------------------- |
| `elements_budget` | derived from VRAM | cap on (context + query) rows × features per pass |
| `adaptive_query_chunk` | `True` | halve the query batch and retry on an out-of-memory error |
| `context_row_chunk` | `None` | build the cache in chunks of this many context rows |
| `allow_subsample` | `True` | permit dropping context rows as a last resort |
## Good to know
**Incoherent settings fail loudly.** Asking for a cache-only field alongside
`cache=False` is rejected rather than ignored, because a policy spelled
correctly that silently does nothing is worse than an error. Settings that work
but probably are not what you meant are honoured and explained in `notes`.
**The report is not a policy.** Feeding a `memory_report` back in as
`memory_policy` is rejected: those are decided outputs, and reusing them as
configuration would skip the coherence checks.
**`context_row_chunk` is not bit-identical.** A single chunk reproduces the
unchunked result exactly, but a smaller chunk differs at around 4e-3 on a
512-row query set — the projection's accumulation order depends on the chunk
size. It is reached automatically after an out-of-memory retry, so a call that
retries can return slightly different numbers.
**Not on Thinking models.** A [Nori Thinking](/nori/quickstart) deployment
rejects `memory_policy` with a `400`. It predicts through a test-time-compute
wrapper that manages its own inference memory, so a policy passed in from
outside would not reach the run that produces the answer — and accepting it
silently would be worse than refusing it.
**Where a cache cannot help.** Preprocessing (polynomial features and the
adaptive SVD) runs before any of this and has its own memory cost, which
`memory_policy` does not govern. A table that fails there fails regardless of
the policy.
## Next steps
Auth, modes, and the request contract for calling Nori over the API.
Pass NaNs straight through — what the model does with them.