> ## Documentation Index > Fetch the complete documentation index at: https://docs.synthefy.com/llms.txt > Use this file to discover all available pages before exploring further. # Nori Fine-Tuning on Amazon SageMaker > Fine-tune Nori on your own table in your AWS account, deploy it as an asynchronous SageMaker endpoint, and predict with the Synthefy client. `SynthefyNoriFineTuner` fine-tunes Nori on your table in your own AWS account, deploys the result as a SageMaker asynchronous endpoint and gives you a `SynthefyNoriClient` for it. To serve the released models as they are, see [Nori Inference on Amazon SageMaker](/nori/sagemaker). ## What you get * A Nori model fine-tuned on your table. Fine-tuning is regression only. * An asynchronous SageMaker endpoint in your account that serves only that model, under the name you choose. * A training report that scores the fine-tuned model and the base model on the same rows. The training job, the endpoint and your data stay in your account. No Synthefy service sits in front of them, and Synthefy can neither see nor stop them. ## Before you start **1. Install the client** ```bash theme={null} pip install -U "synthefy[aws]>=7.3.0" ``` **2. Subscribe to Synthefy - Nori Fine-Tuning** Open the Synthefy - Nori Fine-Tuning listing in AWS Marketplace and choose **View purchase options**. ![Synthefy - Nori Fine-Tuning product page in AWS Marketplace with View purchase options](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/product-page.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=d0a5ec03c1075ab0d6d7f9509befdd52) Review the pricing and terms, then choose **Subscribe** at the bottom of the page. ![Synthefy - Nori Fine-Tuning purchase options page with Subscribe highlighted](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/purchase-options.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=5ddce9d1b32091be0c3bd70e731bb5bb) **3. Copy the Algorithm ARN for your region** Once the subscription is active, choose **Launch your software** on the same page. Under **Algorithm ARNs**, copy the ARN for your region (the **JSON** view lists every region). To come back to it later, open the listing and choose **View purchase options**, then **Launch your software**. Use that region for the bucket, the training job and the endpoint: the client refuses an Algorithm ARN from another region before any AWS call. You don't need the page's **Create training job** button: the client starts the job. ![Launch page for Synthefy - Nori Fine-Tuning with the Algorithm ARNs panel and its copy button highlighted](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/algorithm-arn.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=3d275d23e3ac56dbbaa5971e95f52973) **4. Create the execution role** SageMaker runs the training job and the endpoint as this role. **Console** In the IAM console choose **Roles**, then **Create role**. Choose **AWS service**, then **SageMaker** as the use case, and continue: the role gets AWS's `AmazonSageMakerFullAccess` policy. Name it `nori-fine-tune-execution` and choose **Create role**. **AWS CLI** ```bash theme={null} aws iam create-role --role-name nori-fine-tune-execution --assume-role-policy-document \ '{"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Principal": {"Service": "sagemaker.amazonaws.com"}, "Action": "sts:AssumeRole"}]}' aws iam attach-role-policy --role-name nori-fine-tune-execution \ --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess ``` **5. Choose a bucket** Use an S3 bucket in that region whose name contains `sagemaker`, such as `sagemaker-us-east-1-ACCOUNT_ID`. `AmazonSageMakerFullAccess` grants object access to buckets named that way, so a bucket named otherwise is refused. To create one: ```bash theme={null} aws s3 mb s3://sagemaker-us-east-1-ACCOUNT_ID --region us-east-1 ``` The identity that runs your code needs SageMaker and S3 access too, and `AmazonSageMakerFullAccess` covers every call the client makes. To grant both identities less, use the [least-privilege policies](#least-privilege-iam-policies). > [!NOTE] > > A new AWS account can have no quota for GPU instances. SageMaker then refuses > the job or the endpoint before anything bills, and the error names the quota to > raise in the Service Quotas console, such as > `ml.g5.xlarge for training job usage`. ## Fine-tune a model The snippets on this page run in order in one Python session. Replace `ACCOUNT_ID`, the Algorithm ARN and the target column `y` with your own. ### Prepare your table Fine-tuning is regression only, on a CSV table with one header row. * Every column the job trains on must be numeric. Features may have missing values; the target must be finite in every row. * Encode categorical columns as numbers before you upload, and keep that encoding for prediction: a text column fails the job. * The training table needs at least 1,111 rows, or 1,000 with a validation channel. Above 200,000 rows or 30 million cells (rows x features), the job trains on a subsample drawn with its `seed`. `training_data` is one object ending in `.csv` or a folder ending in `/`. Under a folder, the job reads every uncompressed `.csv` file in it and its sub-folders, and those files must share one header. `validation_data` is optional, takes the same form and has exactly the training table's columns: the job validates on its rows and never trains on them. The walkthrough uses three files with the same columns: `train.csv`, the table the job trains on; `validation.csv`, optional; and `new.csv`, rows the job never sees, for the prediction below. Bring your own, or let the first snippet write a synthetic example of each: ```python theme={null} import logging from pathlib import Path import boto3 import numpy as np import pandas as pd from synthefy import FineTuneHyperparameters, FineTuneJobRequest, SynthefyNoriFineTuner logging.basicConfig(level=logging.INFO) # wait() logs each status change REGION = "us-east-1" ALGORITHM_ARN = "arn:aws:sagemaker:us-east-1:..." # copied from your subscription, for REGION ROLE_ARN = "arn:aws:iam::ACCOUNT_ID:role/nori-fine-tune-execution" BUCKET, TARGET = "sagemaker-us-east-1-ACCOUNT_ID", "y" def example_table(rows: int, seed: int) -> pd.DataFrame: """A synthetic regression table: eight numeric features and a target no straight line fits.""" rng = np.random.default_rng(seed) x = rng.normal(size=(rows, 8)) weights = np.random.default_rng(1234).normal(size=4) # fixed, so every table samples the same function y = np.sin(1.5 * x[:, 0]) + 0.5 * x[:, 1] ** 2 + x[:, 2] * x[:, 3] + np.tanh(x[:, 4:] @ weights / 2) y = y + 0.1 * rng.normal(size=rows) return pd.DataFrame(x, columns=[f"f{i}" for i in range(1, 9)]).assign(**{TARGET: y}) if not any(Path(name).exists() for name in ("train.csv", "validation.csv", "new.csv")): held_out = example_table(500, seed=1) example_table(2000, seed=0).to_csv("train.csv", index=False) held_out.iloc[:400].to_csv("validation.csv", index=False) held_out.iloc[400:].to_csv("new.csv", index=False) s3 = boto3.client("s3", region_name=REGION) s3.upload_file("train.csv", BUCKET, "nori/data/train/train.csv") s3.upload_file("validation.csv", BUCKET, "nori/data/validation/validation.csv") # optional table = pd.read_csv("train.csv") features = [column for column in table.columns if column != TARGET] # the order the job trains on ``` ### Start the job The request is checked before any AWS call: the S3 locations, the hyperparameters, an instance type the base model fits and a runtime that covers every child. A data path without its trailing slash (`.../train`, which S3 would also match `train-old/` against) or a base too large for the instance is refused here, with the reason, rather than minutes into a billed job. ```python theme={null} tuner = SynthefyNoriFineTuner(region_name=REGION) request = FineTuneJobRequest( algorithm=ALGORITHM_ARN, role_arn=ROLE_ARN, training_data=f"s3://{BUCKET}/nori/data/train/", validation_data=f"s3://{BUCKET}/nori/data/validation/", # optional: the job validates on these rows output_path=f"s3://{BUCKET}/nori/output/", hyperparameters=FineTuneHyperparameters( target_column=TARGET, model_name="ft-acme-churn-v1", # the name your fine-tuned model will answer to base_model="nori-6m", bag=2, # the number of children the job trains, 1 to 8; the fine-tuned model is built from them steps=30, ), ) print(request.instance_type, request.max_runtime_s) # ml.g5.xlarge 4500 job = tuner.fine_tune(request) print(job.job_name) # keep it: every later step can start from this name ``` `bag=2, steps=30` is a small run: the job takes 3 to 5 minutes when capacity is available. Every hyperparameter, with its default: | Hyperparameter | Default | Meaning | | - | - | - | | `target_column` | required | the column of the training table to predict | | `model_name` | required | the name the fine-tuned model answers to, and the endpoint's name later: `ft-` followed by 1 to 60 lowercase letters, digits and `-`, the first a letter or digit; a released model's name is refused | | `base_model` | `nori-6m` | the released Nori model to fine-tune: `nori-6m` or `nori-100m` | | `task` | `regression` | only regression is supported | | `feature_columns` | every column but the target | the feature columns, as a list | | `bag` | 8 | the number of children the job trains, 1 to 8; the fine-tuned model is built from them, and each child has its own steps and time budget | | `steps` | 100 | maximum training steps per child | | `patience` | 25 | a child stops after this many steps without improvement | | `budget_s` | 900 | wall-clock cap per child, in seconds | | `lr` | 1e-5 | learning rate | | `seed` | 0 | the job's random seed | ### Follow the job ```python theme={null} job = tuner.wait(job.job_name) print(job.status, job.billable_seconds) # Completed 149 if job.status == "Failed": raise SystemExit(job.failure_reason) for line in tuner.get_logs(job.job_name): # the job's own log lines print(line) curves = pd.DataFrame(tuner.get_training_curves(job.job_name)) # columns child, step, train_loss, val_mse ``` `get_training_curves()` gives one row per child and validated step: ![Validation error per validated step for each child of a fine-tuning job](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/training-curves.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=f3f5205a7e3b50d3aa60a9ba96f48616) The job also appears under **Training jobs** in the SageMaker AI console: ![A completed fine-tuning job in the Training jobs list of the SageMaker AI console](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/training-job.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=b2d24ffa0807edc2780d9e72cbf76632) `tuner.stop(job.job_name)` stops a job. It still writes a model from the training done so far when any of it improved on the base, and ends `Stopped` with `cut_short` true in the report. A job keeps no checkpoints and cannot resume; a new job starts again from the base. ### Read the result ```python theme={null} report = tuner.get_training_report(job.job_name) print(report.val_mse_before, report.val_mse_after) # the base model and the fine-tuned model, same rows ``` Lower is better: the distance from `val_mse_before` to `val_mse_after` is what fine-tuning gained on your table, so always read the two together. Both are mean squared errors in standardized target units, scored on the same rows. Without a validation channel those rows are held out of training (10 % of the table, up to 4,096 rows); with one, they are the channel's rows, which also steer the training, so `val_mse_after` runs a little optimistic there. The report also names `n_val_rows`, `n_rows_used`, `cut_short` and `model_version`. ## Access the fine-tuned model The job writes `model.tar.gz` under `output_path`, at `job.artifact_uri`, with the model, its registry entry and `training_report.json`. The model holds what changed relative to the base, so you serve it through an endpoint, not on its own. It stays in your bucket until you delete it. > [!WARNING] > > Never expire `nori/output/` as a whole: the model package reads its model from > there. From another session, pick the job up by its name; `deploy()` takes the name directly: ```python theme={null} tuner = SynthefyNoriFineTuner(region_name=REGION) job = tuner.get_status("nori-ft-acme-churn-v1-...") # the name fine_tune() printed print(job.status, job.artifact_uri) ``` ## Deploy and predict ### Deploy ```python theme={null} endpoint = tuner.deploy(job) # asynchronous endpoint on ml.g5.xlarge, named after model_name print(endpoint.endpoint_name, endpoint.status) # ft-acme-churn-v1 InService ``` `deploy()` makes a model package from the job, a SageMaker model, an endpoint config and the endpoint, and returns once it is InService, in 4 to 6 minutes. Pass `instance_type=` for another endpoint type from the [instance table](#instance-types-and-quotas). The endpoint appears under **Endpoints** in the console: ![A fine-tuned model's asynchronous endpoint InService in the SageMaker AI console](https://mintcdn.com/synthefy/Ii0mTvAILfqq21nD/assets/sagemaker/fine-tuning/endpoint.png?fit=max\&auto=format\&n=Ii0mTvAILfqq21nD\&q=85\&s=f0f7fa2b16c8afba909472757643a9fc) > [!WARNING] > > The endpoint bills every hour from creation until `undeploy()`, whether or not > you send requests. Prices per instance type and region are on > [SageMaker AI pricing](https://aws.amazon.com/sagemaker/ai/pricing/). ### Predict ```python theme={null} client = tuner.create_client(endpoint, s3_staging_uri=f"s3://{BUCKET}/nori/invocations/") new_rows = pd.read_csv("new.csv") # rows the job never saw: same columns, same order X_test, truth = new_rows[features].head(20), new_rows[TARGET].head(20) context = table.sample(n=500, random_state=0) prediction = client.predict( X_train=context[features], y_train=context[TARGET], X_test=X_test, categorical_columns=None, as_pandas=True ) assert client.last_model_version == report.model_version # the fine-tuned model answered print(pd.DataFrame({"prediction": prediction.values, "truth": truth.values}).round(3).head(8)) ``` `create_client()` builds the same `SynthefyNoriClient` every Nori user has, in SageMaker mode, bound to this endpoint and the fine-tuned model's name. The endpoint does not store your table, so every `predict()` sends context rows; 500 keeps the example small. Send the feature columns in training order, without the target. `new.csv` holds rows the job never saw, so the comparison with `truth` is an honest one; the validation rows would flatter it, since they also steered the training. The endpoint serves only its own model and refuses a request naming another, even a valid `ft-` name. ### Several models and endpoints A job trains one model, and an endpoint serves the one model it was deployed from: a request naming another model is refused. For several models, run one job per `model_name` and deploy each: ```python theme={null} endpoints = {} for name, target in [("ft-acme-churn-v1", "churn"), ("ft-acme-ltv-v1", "ltv")]: hp = FineTuneHyperparameters(target_column=target, model_name=name) # max_runtime_s=None: the copy derives its runtime from these hyperparameters per_model = request.model_copy(update={"hyperparameters": hp, "max_runtime_s": None}) job = tuner.wait(tuner.fine_tune(per_model).job_name) endpoints[name] = tuner.deploy(job) # one endpoint per model, named after it ``` Jobs run at the same time when your training quota allows it (see the quotas below). One model can sit behind several endpoints: pass a different `endpoint_name` to each `deploy()` call (the default is the model's name). The model package is reused, each endpoint gets its own async output prefix under `output_path`, and each bills on its own. Separate endpoints isolate workloads, for example one per team or per environment. For throughput, add instances to one endpoint instead: each instance handles one request at a time and extra requests wait in the endpoint's queue, so N instances answer N requests at once. `deploy()` creates the endpoint with one instance; raise the count from the SageMaker AI console (the endpoint's configuration) or with `aws sagemaker update-endpoint` and a new endpoint config, within your endpoint quota. ### Switch it off ```python theme={null} tuner.undeploy(endpoint) # or tuner.undeploy("ft-acme-churn-v1") from another session tuner.close() ``` `undeploy()` deletes the endpoint, its endpoint configs and the SageMaker model, and keeps the model package for a later `deploy()`. Expire `nori/invocations/` and `nori/output/ft-acme-churn-v1/async/` after at least a day. ## When something goes wrong | You see | What to do | | - | - | | `deploy()` or `get_training_report()` raises `FineTuningError` saying the `Stopped` job has no model artifact | Nothing had improved on the base when the job stopped, so it wrote no model. Start a new job. | | `Failed`, and `failure_reason` names your table or settings | The job refused the input once its instance was up. Fix it and start a new job; `tuner.get_logs()` has any traceback. | | `InProgress` in `Pending` for a long time with the message `Training job waiting for capacity`, or `Failed` with `CapacityError:` | AWS has no spare capacity for that type. The wait is unbilled and can last hours. Stop it and start a new job on another type. | | `wait()` raises `APITimeoutError` | It waited the time limit plus 7,200 s. The job still runs: call `wait()` again, or `stop()`. | | `deploy()` raises `FineTuningError` naming `InsufficientInstanceCapacity` | AWS found no instance of that type in time. Call `tuner.undeploy("ft-acme-churn-v1")`, then `tuner.deploy(job, instance_type=...)` with another endpoint type; it reuses the model package. | | `deploy()` raises `APITimeoutError` | The endpoint was not InService within `timeout_s` (3,600 s by default) and is still starting and billing. Call `tuner.undeploy("ft-acme-churn-v1")`, then deploy again with a larger `timeout_s`. | ## Reference ### Instance types and quotas You need quota for one training instance and one endpoint instance. A job's default type follows its base model: | Use | Instance types | Default | | - | - | - | | Train `nori-6m` | `ml.g5.xlarge`, `ml.g5.2xlarge`, `ml.g6.xlarge` (24 GB); `ml.g6e.xlarge`, `ml.g6e.2xlarge` (48 GB) | `ml.g5.xlarge` | | Train `nori-100m` | `ml.g6e.xlarge`, `ml.g6e.2xlarge` only | `ml.g6e.xlarge` | | Endpoint | `ml.g5.xlarge`, `ml.g5.2xlarge`, `ml.g6.xlarge` | `ml.g5.xlarge` | The quotas are named like `ml.g5.xlarge for training job usage` and `ml.g5.xlarge for endpoint usage`. At 0 SageMaker refuses the create call (`ResourceLimitExceeded`) before anything bills, and the client raises `FineTuningError` naming the quota. The same quotas set how much runs at once: N jobs on `ml.g5.xlarge` need its training quota at N or more, and N endpoint instances of a type need its endpoint quota at N or more. New accounts often start at 0 for GPU types; request what you need in the Service Quotas console before the first job, since approval can take a few days. Check your quota: ```bash theme={null} aws service-quotas list-service-quotas --service-code sagemaker --region us-east-1 \ --query "Quotas[?contains(QuotaName, 'ml.g5.xlarge')].[QuotaName,Value]" --output table # lists the training and the endpoint quota; for nori-100m, run it again with ml.g6e.xlarge ``` ### Least-privilege IAM policies For production, replace `AmazonSageMakerFullAccess` with two policies that grant each identity only what it uses. With them the bucket can have any name. The execution role's policy, saved as `execution-policy.json` with `BUCKET`, `REGION`, `ACCOUNT_ID` and `ALGORITHM_ARN` replaced, lets: * the job read `nori/data/` and write `nori/output/` * the endpoint read `nori/invocations/` and write under `nori/output/` * SageMaker describe the Algorithm and your model package as this role The model package is named after the job, `nori-ft-...` unless you set `job_name`; if you set it, use its prefix in this policy. ```json execution-policy.json theme={null} {"Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::BUCKET"}, {"Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::BUCKET/nori/data/*"}, {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:AbortMultipartUpload"], "Resource": ["arn:aws:s3:::BUCKET/nori/output/*", "arn:aws:s3:::BUCKET/nori/invocations/*"]}, {"Effect": "Allow", "Action": ["logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams"], "Resource": ["arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/sagemaker/TrainingJobs*", "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/sagemaker/Endpoints/*"]}, {"Effect": "Allow", "Action": "cloudwatch:PutMetricData", "Resource": "*"}, {"Effect": "Allow", "Action": ["sagemaker:DescribeAlgorithm", "sagemaker:DescribeModelPackage"], "Resource": ["ALGORITHM_ARN", "arn:aws:sagemaker:REGION:ACCOUNT_ID:model-package/nori-ft-*"]}]} ``` Put it on the role in place of the managed policy: ```bash theme={null} aws iam put-role-policy --role-name nori-fine-tune-execution --policy-name nori --policy-document file://execution-policy.json aws iam detach-role-policy --role-name nori-fine-tune-execution --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess ``` The identity that runs your code needs the policy below, saved as `caller-policy.json` with `BUCKET`, `REGION` and `ACCOUNT_ID` replaced. `s3:ListBucket` makes a missing object read as not found, which the client relies on while it waits for a result. With `kms_key_id`, the key's policy must let both identities use it. ```json caller-policy.json theme={null} {"Version": "2012-10-17", "Statement": [ {"Effect": "Allow", "Resource": "*", "Action": [ "sagemaker:CreateTrainingJob", "sagemaker:DescribeTrainingJob", "sagemaker:StopTrainingJob", "sagemaker:DescribeAlgorithm", "sagemaker:CreateModelPackage", "sagemaker:DescribeModelPackage", "sagemaker:DeleteModelPackage", "sagemaker:CreateModel", "sagemaker:DescribeModel", "sagemaker:DeleteModel", "sagemaker:CreateEndpointConfig", "sagemaker:DescribeEndpointConfig", "sagemaker:ListEndpointConfigs", "sagemaker:DeleteEndpointConfig", "sagemaker:CreateEndpoint", "sagemaker:DescribeEndpoint", "sagemaker:DeleteEndpoint", "sagemaker:InvokeEndpointAsync"]}, {"Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::ACCOUNT_ID:role/nori-fine-tune-execution", "Condition": {"StringEquals": {"iam:PassedToService": "sagemaker.amazonaws.com"}}}, {"Effect": "Allow", "Action": ["logs:DescribeLogStreams", "logs:GetLogEvents"], "Resource": ["arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/sagemaker/TrainingJobs", "arn:aws:logs:REGION:ACCOUNT_ID:log-group:/aws/sagemaker/TrainingJobs:*"]}, {"Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::BUCKET"}, {"Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::BUCKET/nori/*"}]} ``` For a role: ```bash theme={null} aws iam put-role-policy --role-name YOUR_ROLE --policy-name nori-caller --policy-document file://caller-policy.json ``` ### Alarms An EventBridge rule on SageMaker's training job state changes can email you when a job ends `Failed` or `Stopped`, and CloudWatch alarms on `InvocationFailures` and `ApproximateAgeOfOldestRequest` watch an endpoint you keep running. The client's README has the commands, under [Alarms](https://github.com/Synthefy/synthefy-nori/blob/main/libs/synthefy/README.md#alarms). ## Next steps > **[Inference on Amazon SageMaker](/nori/sagemaker)** > > Serve the released Nori models from one SageMaker endpoint, without fine-tuning. > **[Python Client](/nori/client)** > > DataFrames, categorical encoding, retries, and distribution outputs. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.