> ## Documentation Index
> Fetch the complete documentation index at: https://docs.synthefy.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Relational Data
> Predict from linked database tables with Nori-Rel: connect, see the graph, predict, evaluate, save the pipeline, and read the features it built.
Nori-Rel connects to linked database tables, builds features at each past date
from what had happened by then, and has Nori predict what happens next. You
describe the target in one line of PQL, and nothing is trained.
Nori-Rel connects to PostgreSQL, Snowflake, and Databricks. It needs Python
3.11, 3.12, or 3.13. Install `nori-rel` 0.4.3 or later with the extra for your
database and the `local` extra, as each tab under Connect shows; see
[Installation](/setup/installation). Local mode runs Nori in your process and
downloads its weights from Hugging Face on first use. A GPU is faster but not
required. The outputs on this page come from the public
[RelBench `rel-f1`](https://relbench.stanford.edu/datasets/rel-f1/) Formula 1
database.
## Build a database from files
Skip this section if your data is already in related tables with declared keys.
If your data is a folder of CSV or Excel exports, the
[`nori-rel-prepare-data`](https://github.com/Synthefy/nori-rel/tree/main/skills/nori-rel-prepare-data)
skill walks an AI agent through turning it into a PostgreSQL database Nori-Rel
can use. The agent profiles the files, designs tables with declared keys, and
leaves out columns that would leak the future, such as a customer's lifetime
total. Before it creates anything, it shows you every table, column, and
relationship it plans and waits for your confirmation.
```bash theme={null}
git clone --depth 1 https://github.com/Synthefy/nori-rel
cp -r nori-rel/skills/nori-rel-prepare-data ~/.claude/skills/
```
Then ask Claude Code to prepare your files for Nori-Rel. With another agent,
give it the folder's `SKILL.md` as its instructions.
## Connect
Keep anything that identifies your account in environment variables. Nori-Rel
reads them in your process and never sends them anywhere.
Replace every value marked `# yours` with your own: the user, role, key path,
host, warehouse, database, and schema shown are examples. Register the time
column of each event table. Tables without one, such as `drivers`, are
entities.
```bash theme={null}
pip install "nori-rel[database,local]>=0.4.3"
# yours: user, password, host, and database
export DATABASE_URL="postgresql://YOUR_USER:YOUR_PASSWORD@db.example.com:5432/analytics?sslmode=require"
```
```python theme={null}
from nori_rel import EnvironmentCredential, NoriRelClient
client = NoriRelClient(model="nori-30m")
source = client.connect(
engine="postgresql",
credential=EnvironmentCredential(variable="DATABASE_URL"),
schema_name="f1", # yours: the schema that holds your tables
time_columns={
"races": "date",
"results": "date",
"qualifying": "date",
"standings": "date",
"constructor_results": "date",
"constructor_standings": "date",
},
)
print(client.test_connection(source).status.value)
```
```bash theme={null}
pip install "nori-rel[snowflake,local]>=0.4.3"
# yours: account identifier, user, and the path to that user's private key
export SNOWFLAKE_ACCOUNT="your-org-your-account"
export SNOWFLAKE_AUTH_JSON='{"user": "YOUR_USER", "authenticator": "SNOWFLAKE_JWT", "private_key_file": "/path/to/your_rsa_key.p8"}'
```
```python theme={null}
import os
from nori_rel import EnvironmentCredential, NoriRelClient, SnowflakeConnection
client = NoriRelClient(model="nori-30m")
source = client.connect(
engine="snowflake",
connection=SnowflakeConnection(
account=os.environ["SNOWFLAKE_ACCOUNT"],
warehouse="COMPUTE_WH", # yours
database="NORI_REL_DOCS", # yours: the database that holds your tables
role="YOUR_ROLE", # yours: a role your user has that can read them
),
credential=EnvironmentCredential(variable="SNOWFLAKE_AUTH_JSON"),
schema_name="F1", # yours
time_columns={
"races": "date",
"results": "date",
"qualifying": "date",
"standings": "date",
"constructor_results": "date",
"constructor_standings": "date",
},
)
print(client.test_connection(source).status.value)
```
```bash theme={null}
pip install "nori-rel[databricks,local]>=0.4.3"
# yours: workspace host, SQL warehouse path, and service-principal secret
export DATABRICKS_SERVER_HOSTNAME="your-workspace.cloud.databricks.com"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/your-warehouse-id"
export DATABRICKS_AUTH_JSON='{"client_id": "...", "client_secret": "..."}'
```
```python theme={null}
import os
from nori_rel import DatabricksConnection, EnvironmentCredential, NoriRelClient
client = NoriRelClient(model="nori-30m")
source = client.connect(
engine="databricks",
connection=DatabricksConnection(
server_hostname=os.environ["DATABRICKS_SERVER_HOSTNAME"],
http_path=os.environ["DATABRICKS_HTTP_PATH"],
catalog="analytics", # yours
),
credential=EnvironmentCredential(variable="DATABRICKS_AUTH_JSON"),
schema_name="f1", # yours
time_columns={
"races": "date",
"results": "date",
"qualifying": "date",
"standings": "date",
"constructor_results": "date",
"constructor_standings": "date",
},
)
print(client.test_connection(source).status.value)
```
```
ready
```
Connect with a role that can only read. Nori-Rel needs to read the schema and
the tables you predict from, and nothing else.
If you don't know which column holds each table's event time, connect once
without `time_columns` and print every table's columns with their database
types:
```python theme={null}
for table in client.discover(source).tables:
print(table.name)
for column in table.columns:
print(f" {column.name}: {column.data_type}")
```
With `rel-f1` in PostgreSQL, this prints the following (three of the nine
tables shown):
```
drivers
driverId: BIGINT
driverRef: TEXT
code: TEXT
forename: TEXT
surname: TEXT
dob: TIMESTAMP
nationality: TEXT
races
raceId: BIGINT
year: BIGINT
round: BIGINT
circuitId: BIGINT
name: TEXT
date: TIMESTAMP
time: TEXT
results
resultId: BIGINT
raceId: BIGINT
driverId: BIGINT
constructorId: BIGINT
number: DOUBLE PRECISION
grid: BIGINT
position: DOUBLE PRECISION
positionOrder: BIGINT
points: DOUBLE PRECISION
laps: BIGINT
milliseconds: DOUBLE PRECISION
fastestLap: DOUBLE PRECISION
rank: DOUBLE PRECISION
statusId: BIGINT
date: TIMESTAMP
```
Pick the date or timestamp column that records when each row happened, add it
to `time_columns`, and connect again. Here that is `date` in `races` and
`results`. `races.time` is text, so it can't be a time column. `drivers.dob` is
a timestamp but it's a date of birth, so `drivers` stays an entity. Type names
follow your database.
## See the graph
`discover` returns the tables, keys, and links Nori-Rel found. It reads the
primary and foreign keys your database declares. Snowflake and Databricks
record keys without enforcing them, and Nori-Rel checks them before it uses
them. If your tables declare no keys, pass `primary_keys=` and `foreign_keys=`
to `connect`.
This helper turns the result into a Mermaid diagram:
```python theme={null}
graph = client.discover(source)
def get_mermaid_er_diagram(graph) -> str:
"""Return a Mermaid ER diagram: keys, links and time columns of each table."""
lines = ["erDiagram"]
for table in graph.tables:
for key in table.foreign_keys:
lines.append(f' {key.referred_table} ||--o{{ {table.name} : "{key.columns[0]}"')
for table in graph.tables:
linked = {column for key in table.foreign_keys for column in key.columns}
lines.append(f" {table.name} {{")
for column in table.columns:
if column.name in table.primary_key:
lines.append(f" key {column.name} PK")
elif column.name in linked:
lines.append(f" key {column.name} FK")
elif column.name == table.time_column:
lines.append(f" time {column.name}")
lines.append(" }")
return "\n".join(lines)
print(get_mermaid_er_diagram(graph))
```
Paste the output into any Mermaid renderer. For `rel-f1` it draws nine tables:
`results`, `qualifying`, and `standings` each link a driver to a race, and
`drivers` has no time column because a driver is an entity rather than an
event.
```mermaid theme={null}
erDiagram
constructors ||--o{ constructor_results : "constructorId"
races ||--o{ constructor_results : "raceId"
constructors ||--o{ constructor_standings : "constructorId"
races ||--o{ constructor_standings : "raceId"
constructors ||--o{ qualifying : "constructorId"
drivers ||--o{ qualifying : "driverId"
races ||--o{ qualifying : "raceId"
circuits ||--o{ races : "circuitId"
constructors ||--o{ results : "constructorId"
drivers ||--o{ results : "driverId"
races ||--o{ results : "raceId"
drivers ||--o{ standings : "driverId"
races ||--o{ standings : "raceId"
circuits {
key circuitId PK
}
constructor_results {
key constructorResultsId PK
key raceId FK
key constructorId FK
time date
}
constructor_standings {
key constructorStandingsId PK
key raceId FK
key constructorId FK
time date
}
constructors {
key constructorId PK
}
drivers {
key driverId PK
}
qualifying {
key qualifyId PK
key raceId FK
key driverId FK
key constructorId FK
time date
}
races {
key raceId PK
key circuitId FK
time date
}
results {
key resultId PK
key raceId FK
key driverId FK
key constructorId FK
time date
}
standings {
key driverStandingsId PK
key raceId FK
key driverId FK
time date
}
```
## Set up the prediction
This example predicts each driver's average finishing position over the next
60 days. In PQL that is one line. Four dates and one column set up the rest:
* **`anchor_time`** is the date you predict from. Features for the drivers you
score read only rows dated up to it.
* **`anchor_times`** are past dates Nori learns from. At each one, Nori-Rel
builds every driver's features from rows dated up to that date and a label
from the 60 days after it, so each date adds one example per driver. Each
must be earlier than `anchor_time`.
* **`observed_through`** is how far your data is complete. A past date is used
only if its whole 60-day window ends by the earlier of `anchor_time` and
`observed_through`, so a partly loaded month can't produce wrong labels. The
latest timestamp in the database can't stand in for it, because a partly
loaded period still has rows.
* **`entity_created_at`** names the column in the entity table (here
`drivers`) that says when each entity came into existence. A driver is left
out of every past date before it, and a driver with no value is left out
entirely. Most databases have a `created_at` column for this; `rel-f1` has
none, so the date of birth stands in.
```python theme={null}
from datetime import UTC, datetime, timedelta
# The date to predict from: features for the scored drivers read rows up to it.
anchor_time = datetime(2004, 7, 1, tzinfo=UTC)
request = dict(
# The database registered with connect.
source=source,
# For every driver, the mean of results.positionOrder over the 60 days
# after the date: (date, date + 60 days].
query="PREDICT AVG(results.positionOrder,0,60,days) FOR EACH drivers.driverId",
# Twelve past dates, 60 days apart, covering the two years before
# anchor_time. Each gives one example per driver: features up to the date,
# label from the 60 days after it.
anchor_times=[anchor_time - timedelta(days=60 * k) for k in range(1, 13)],
# The prediction date. Every entry in anchor_times must be earlier.
anchor_time=anchor_time,
# The data is complete up to here, so every past date's 60-day window that
# ends by July 2004 has a trustworthy label.
observed_through=anchor_time,
# Required: a date column on drivers saying when each driver exists. A
# driver counts at a date only if born by then; rel-f1 has no signup date.
entity_created_at="dob",
# Cap on example rows built from the past dates. The default, 5,000, keeps
# the most recent dates first and drops the oldest; 12,000 keeps them all.
max_context_rows=12_000,
# Return predictions, metadata, the reusable pipeline and the feature
# matrix, instead of the predictions table alone.
return_details=True,
)
```
Then predict. `feature_strategy=` picks how Nori-Rel builds the features, and
`on_event` prints each stage as it happens. Pick a tab: each runs the same
`request`. The rest of this page uses the NoriDFS result.
```python theme={null}
details = client.predict(
**request,
# Follow the foreign keys from each driver and aggregate what they
# reach, within a budget of 48 features.
feature_strategy="noridfs",
# Called for each stage as it happens: reading the database snapshot,
# planning the features, and predicting. Leave it out to run quietly.
on_event=lambda event: print(f"{event.type}: {event.message}"),
)
print(details.features.shape[1], "features")
print(details.predictions.head())
```
```
status: Run started: noridfs search.
progress: Started reading the database snapshot.
progress: Finished reading the database snapshot in 41.3 s.
progress: Started planning features.
progress: Finished planning features in 0.3 s.
progress: Started predicting.
progress: Finished predicting in 9.1 s.
status: Run completed.
48 features
entity_id anchor_time prediction
0 0 2004-07-01 00:00:00+00:00 10.931919
1 1 2004-07-01 00:00:00+00:00 11.153991
2 2 2004-07-01 00:00:00+00:00 10.949479
3 3 2004-07-01 00:00:00+00:00 9.111845
4 4 2004-07-01 00:00:00+00:00 11.130487
```
NoriDFS follows the foreign keys from the driver, including two hops away
(the races and constructors behind each result), and fills a budget of 48
features. It takes the prediction date's fields and the driver's own
columns first, then counts, recency, and ages of linked rows, then means of
value columns until the budget runs out. It computes them in pandas, not
SQL.
```python theme={null}
details = client.predict(
**request,
# COUNT, MIN, MAX, MEAN and STD of every column in the tables that
# point at the driver, each written as SQL and run on DuckDB.
feature_strategy="fastdfs",
# Called for each stage as it happens: reading the database snapshot,
# planning the features, and predicting. Leave it out to run quietly.
on_event=lambda event: print(f"{event.type}: {event.message}"),
)
print(details.features.shape[1], "features")
print(details.predictions.head())
```
```
status: Run started: fastdfs search.
progress: Started reading the database snapshot.
progress: Finished reading the database snapshot in 48.1 s.
progress: Started planning features.
progress: Finished planning features in 2.6 s.
progress: Started predicting.
progress: Finished predicting in 2.4 s.
status: Run completed.
63 features
entity_id anchor_time prediction
0 0 2004-07-01 00:00:00+00:00 14.455626
1 1 2004-07-01 00:00:00+00:00 11.711873
2 2 2004-07-01 00:00:00+00:00 14.455626
3 3 2004-07-01 00:00:00+00:00 9.434957
4 4 2004-07-01 00:00:00+00:00 14.455626
```
FastDFS is the default when you leave out `feature_strategy`. It takes
COUNT, MIN, MAX, MEAN, and STD of every column in the tables that point at
the driver: 63 features over `qualifying`, `results`, and `standings`. It
does not reach `races` or use the driver's own columns, but it does include
the mean of `results.positionOrder`, each driver's past average finish.
FastDFS writes each feature as a SQL aggregation and runs it on a local
DuckDB copy of the snapshot.
```bash theme={null}
# Add agentic to the extras for your database; Snowflake here.
pip install "nori-rel[snowflake,local,agentic]>=0.4.3"
```
```python theme={null}
from nori_rel import AgenticSearch
from nori_rel.feature.agentic.usage import TokenPrices
search = AgenticSearch.claude( # reads ANTHROPIC_API_KEY
model="claude-sonnet-5-5",
prices=TokenPrices(input=3.0, output=15.0, cache_read=0.3, cache_write_5m=3.75, cache_write_1h=6.0),
max_cost_usd=3.0, # stop the agent once its token cost reaches $3
)
details = client.predict(
**request,
# An agent writes SQL features on top of NoriDFS.
feature_strategy=search,
# Prints each stage, plus a line for every program the agent scores.
on_event=lambda event: print(f"{event.type}: {event.message}"),
)
keys = ("agentic_model", "agentic_trials", "agentic_cost_usd", "agentic_fallback_reason")
print({key: details.metadata[key] for key in keys})
```
The trial events, with the spend updates between them left out:
```
metric: Trial 1/24: MAE 2.769 (baseline 2.507); not a significant gain.
metric: Trial 2/24: MAE 2.72 (baseline 2.507); not a significant gain.
metric: Trial 3/24: MAE 2.63 (baseline 2.507); not a significant gain.
metric: Trial 4/24: MAE 2.623 (baseline 2.507); not a significant gain.
metric: Trial 5/24: MAE 2.687 (baseline 2.507); not a significant gain.
metric: Trial 6/24: MAE 2.528 (baseline 2.507); not a significant gain.
metric: Trial 7/24: MAE 2.576 (baseline 2.507); not a significant gain.
metric: Trial 8/24: MAE 2.5 (baseline 2.507); not a significant gain.
metric: Trial 9/24: MAE 2.684 (baseline 2.507); not a significant gain.
metric: Trial 10/24: MAE 2.501 (baseline 2.507); not a significant gain.
metric: Trial 11/24: MAE 2.714 (baseline 2.507); not a significant gain.
metric: Trial 12/24: MAE 2.511 (baseline 2.507); not a significant gain.
metric: Trial 13/24: MAE 2.481 (best 2.481, baseline 2.507); new best.
metric: Trial 14/24: MAE 2.479 (best 2.481, baseline 2.507); not a significant gain.
warning: No program beat the baseline; using noridfs.
```
```
{'agentic_model': 'claude-sonnet-5-5', 'agentic_trials': 14, 'agentic_cost_usd': 0.739491, 'agentic_fallback_reason': 'no_program_confirmed'}
```
Sonnet 5.5 wrote 14 SQL feature programs in 238 seconds, for \$0.74. Trial
13 lowered the holdout error from 2.507 to 2.481, but did not hold up on the
separate confirmation rows, so the prediction fell back to NoriDFS alone:
the same 48 features and the same predictions as the NoriDFS tab. When a
program is kept, its columns appear in `get_feature_table()` with `source`
set to `"agent"` and the agent's SQL in `sql`.
An agent writes SQL feature programs on top of the NoriDFS features. Each
program runs locally, and Nori-Rel keeps it only if it beats NoriDFS alone
on held-out rows; otherwise the prediction uses NoriDFS. The agent sends
your schema, column profiles with some raw values, and samples of its query
results to the model provider you configure. The
[`nori-rel` README](https://github.com/Synthefy/nori-rel#readme) lists
exactly what it sends. Set your prices in `TokenPrices` so the cost cap is
in dollars.
The 12 past dates cover the two years before July 2004. Every driver born by a
date is a candidate example for it, 10,284 candidates in all. Only 190 of them
raced in the 60 days after their date, so only those have a label, and Nori
learns from those 190. `max_context_rows=12_000` keeps every candidate in play;
the default of 5,000 fills from the most recent dates and would drop the oldest
ones. `entity_created_at="dob"` keeps each driver out of dates before their
birth.
The events show where the time goes. Reading the snapshot takes most of it,
and that depends on your warehouse; planning features and running Nori take a
few seconds. To run in the background, call `client.start_predict(...)` with
the same arguments, then read `run.events()` or stop it with `run.cancel()`.
## Save the pipeline
`details.pipeline` records how the prediction was made: the task, the features
NoriDFS planned, and the model settings. It holds no credentials, rows, or
model weights. Save it to predict again later without a new feature search:
```python theme={null}
details.pipeline.save("driver_position.json")
```
## See the features
`details.features` is the matrix Nori scored: a pandas DataFrame with one row
per driver, indexed by `entity_id`, and one column per feature. Each column
name says what it reads: `results.grid.mean` is the mean of `results.grid` over
the driver's results rows, and `qualifying.constructors.count` counts the
constructors reached through the driver's qualifying rows. Transposed, each
row is a feature and each column a driver:
```python theme={null}
print(details.features.head().T)
```
```
entity_id 0 1 ... 3 4
anchor.entity_key 0 1 ... 3 4
anchor.cutoff_epoch_days 12600.0 12600.0 ... 12600.0 12600.0
anchor.cutoff_year 2004.0 2004.0 ... 2004.0 2004.0
anchor.cutoff_month 7.0 7.0 ... 7.0 7.0
anchor.cutoff_day 1.0 1.0 ... 1.0 1.0
anchor.cutoff_day_of_week 3.0 3.0 ... 3.0 3.0
drivers.dob 7115.0 9914.0 ... 8373.0 8291.0
qualifying.count 0.0 32.0 ... 26.0 0.0
qualifying.constructors.count 0.0 32.0 ... 26.0 0.0
results.count 0.0 75.0 ... 42.0 0.0
qualifying.races.count 0.0 32.0 ... 26.0 0.0
standings.count 0.0 72.0 ... 42.0 0.0
results.constructors.count 0.0 75.0 ... 42.0 0.0
results.races.count 0.0 75.0 ... 42.0 0.0
standings.races.count 0.0 72.0 ... 42.0 0.0
qualifying.recency NaN 12.0 ... 12.0 NaN
qualifying.constructors.recency NaN 12.0 ... 12.0 NaN
results.recency NaN 11.0 ... 11.0 NaN
qualifying.races.recency NaN 11.0 ... 11.0 NaN
standings.recency NaN 11.0 ... 11.0 NaN
results.constructors.recency NaN 11.0 ... 11.0 NaN
results.races.recency NaN 11.0 ... 11.0 NaN
standings.races.recency NaN 11.0 ... 11.0 NaN
qualifying.mean_age NaN 469.84375 ... 290.653846 NaN
qualifying.constructors.mean_age NaN 469.84375 ... 290.653846 NaN
results.mean_age NaN 815.813333 ... 602.333333 NaN
qualifying.races.mean_age NaN 468.84375 ... 289.653846 NaN
standings.mean_age NaN 849.152778 ... 602.333333 NaN
results.constructors.mean_age NaN 815.813333 ... 602.333333 NaN
results.races.mean_age NaN 815.813333 ... 602.333333 NaN
standings.races.mean_age NaN 849.152778 ... 602.333333 NaN
qualifying.oldest_age NaN 1573.0 ... 1006.0 NaN
qualifying.constructors.oldest_age NaN 1573.0 ... 1006.0 NaN
results.oldest_age NaN 1572.0 ... 1215.0 NaN
qualifying.races.oldest_age NaN 1572.0 ... 1005.0 NaN
standings.oldest_age NaN 1572.0 ... 1215.0 NaN
results.constructors.oldest_age NaN 1572.0 ... 1215.0 NaN
results.races.oldest_age NaN 1572.0 ... 1215.0 NaN
standings.races.oldest_age NaN 1572.0 ... 1215.0 NaN
qualifying.number.mean NaN 12.375 ... 8.5 NaN
results.fastestLap.mean NaN 34.666667 ... 27.0 NaN
qualifying.races.round.mean NaN 7.5 ... 7.576923 NaN
standings.points.mean NaN 3.777778 ... 16.47619 NaN
results.races.round.mean NaN 8.453333 ... 7.952381 NaN
standings.races.round.mean NaN 8.819444 ... 7.952381 NaN
qualifying.position.mean NaN 13.40625 ... 8.538462 NaN
results.grid.mean NaN 12.653333 ... 12.714286 NaN
qualifying.races.year.mean NaN 2002.78125 ... 2003.269231 NaN
[48 rows x 5 columns]
```
## Replay the pipeline
Load the saved pipeline, in this process or another one, and predict with it.
It reads the database again and rebuilds the same features with the saved plan,
with no feature search:
```python theme={null}
from nori_rel import NoriRelPipeline
pipeline = NoriRelPipeline.load("driver_position.json")
replayed = client.predict(
# The database to read; the pipeline checks it has the saved schema.
source=source,
# The saved task, features and model settings.
pipeline=pipeline,
# The date to predict from. The past dates Nori learns from keep their
# spacing before it, as in the first run.
as_of=anchor_time,
# The data is complete up to here.
observed_through=anchor_time,
)
same = replayed["prediction"].equals(details.predictions["prediction"])
print(len(replayed), "predictions, identical to the first run:", same)
```
```
857 predictions, identical to the first run: True
```
Replayed from the same date, it returns the same 857 predictions as the first
run. Pass a later `as_of` to predict a new date with the same recipe. Each
replay reads fresh rows from the database.
## Evaluate the predictions
To check the pipeline, backtest it: replay it from past dates whose next 60
days are already in the database, then compare with what happened. This loop
replays the saved `pipeline` at eight dates, two months apart, starting in July
2003\. Each replay builds its context and features only from rows dated before
its `as_of`, with the same features and settings every time. For a baseline,
it uses each driver's average finish over the year before the date, a simple
"recent form" guess.
The actual values come from a query on the same database. `connection` is any
DB-API connection to it, such as `psycopg.connect(...)` or
`snowflake.connector.connect(...)`. Quote column names the way your database
does.
```python theme={null}
import pandas as pd
def get_average_finish_by_driver(connection, start, end) -> pd.Series:
"""Return each driver's mean finishing position over results in (start, end]."""
cursor = connection.cursor()
cursor.execute(
'SELECT "driverId", AVG("positionOrder") FROM "results" '
'WHERE "date" > %s AND "date" <= %s GROUP BY "driverId"',
(start, end),
)
return pd.Series({driver: float(value) for driver, value in cursor.fetchall()})
rows = []
for k in range(8):
anchor = datetime(2003, 7, 1, tzinfo=UTC) + timedelta(days=60 * k)
# Replay the saved pipeline as if it were this date: the data counts as
# complete up to it, and the past dates it learns from move back with it.
predicted = client.predict(
source=source,
pipeline=pipeline,
as_of=anchor,
observed_through=anchor,
).set_index("entity_id")["prediction"]
actual = get_average_finish_by_driver(connection, anchor, anchor + timedelta(days=60))
last_year = get_average_finish_by_driver(connection, anchor - timedelta(days=365), anchor)
scored = pd.DataFrame({"actual": actual, "nori": predicted}).dropna()
scored["last_year_average"] = last_year.reindex(scored.index).fillna(last_year.median())
rows.append(scored)
evaluation = pd.concat(rows)
print(len(evaluation), "drivers scored")
for name in ("nori", "last_year_average"):
print(name, round((evaluation[name] - evaluation["actual"]).abs().mean(), 3))
```
```
131 drivers scored
nori 2.592
last_year_average 2.763
```
Two of the eight dates fall in the off-season, with no race in the next 60
days, so 131 predictions across six dates have an actual value to compare
with. Nori misses by 2.59 finishing places on average, against 2.76 for the
recent-form baseline. That is a gap of 0.17 places, but on 131 predictions it
is within noise: a bootstrap 95% interval for the gap runs from -0.08 to 0.43,
and Nori is closer on 65 of the 131. Run the same comparison on your own data,
with more dates, before you rely on a difference this small.
The backtest gives the same errors as predicting each date from scratch with
`feature_strategy="noridfs"`, because NoriDFS plans the same features at each
of these dates. A pipeline that holds an agent's program replays that SQL
unchanged, so backtesting it before the date of its search tests a program
chosen with later data.
## Good to know
* Each prediction reads every table as of one moment. Snowflake uses Time
Travel and Databricks pins Delta table versions, so those tables need
history; views are not supported.
* Local mode predicts numeric targets. A yes/no target such as
`COUNT(results.*,0,60,days)=0` needs hosted mode.
* Your database bills the reads, including the key checks that run before
extraction.
## Next steps
The `nori-rel` extras for PostgreSQL, Snowflake, and Databricks.
SHAP values, feature interactions, and partial dependence.
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.