SageMaker Model Endpoints: How to Choose, Deploy and Shut Down

Getting a model to run in a notebook is the easy part.

The hard part is getting it to serve predictions reliably at 2 AM when traffic spikes, without you having to do anything about it. That’s the gap between a demo and a deployed system, and it’s the gap most ML learners never close.

This article covers how to close it on Amazon SageMaker AI: which of its four ways to serve a model fits your traffic, how a model becomes a live endpoint, what it costs you while nobody is calling it, and how to take it all down again.

Why local inference doesn’t scale

Running model.predict() in a notebook has three production problems:

1. It’s tied to your machine. The notebook closes, the service stops. You can’t call it from another service or expose it as an API without significant extra work.
2. It doesn’t handle concurrency. A single Python process serving predictions one at a time is fine for prototypes. Real traffic requires batching, queuing, or horizontal scaling.
3. There’s no observability. No latency metrics, no error rates, no auto-restart on crash.

A SageMaker endpoint solves all three. It runs your model in a managed container, puts it behind an HTTPS API that only callers with the right IAM permission can reach, and can add or remove instances based on CloudWatch metrics.

Four ways SageMaker serves a model

SageMaker AI offers four inference options, and AWS publishes a hard limit for each. These limits decide the choice more often than price does, so start with them (AWS: inference options):

Real-time endpoints run on instances you choose (CPU or GPU) and stay up until you delete them. That is what makes them fast and what makes them expensive when idle.

Serverless endpoints need no instance type. You pick a memory size from 1 GB to 6 GB, and SageMaker gives the container CPU in proportion. You pay for the compute time each request uses, billed by the millisecond, plus the data processed. There is a list of things serverless cannot do: no GPUs, no VPC configuration, no multi-model endpoints, no data capture, no Model Monitor and no multiple production variants (AWS: Serverless Inference). If your model needs a GPU, serverless is off the table before price comes into it.

Asynchronous endpoints put requests in a queue, read the input from S3 and write the answer back to S3. They are the right home for a document-processing or video model that takes minutes per request, which would hit the 60-second ceiling anywhere else. Because the caller isn’t waiting, you can let the instance count drop to zero between jobs.

Batch transform creates no endpoint at all. It starts instances, runs your model over a dataset in S3, writes the results and shuts down. For a nightly scoring job or for checking a model against a test set, it avoids paying for an endpoint that sits idle all day.

How to choose

Ask three questions in order:

1. Is someone waiting on the answer? If not, use batch transform for a whole dataset, or asynchronous inference for requests that arrive one at a time.
2. Does one request fit in 60 seconds and 4 MB, and does the model run without a GPU? If yes and traffic comes in bursts with quiet gaps, use serverless. If any answer is no, use real-time (or asynchronous, if the request is too large or too slow).
3. Is the traffic steady enough to keep an instance busy? A real-time endpoint that’s busy most of the day is usually cheaper per request than serverless. One that’s idle most of the day is paying for nothing.

Serving more than one model

Two features matter once you have more than one model, or more than one version of the same model:

• Production variants. One endpoint can split traffic between two or more variants, for example 90% to the current model and 10% to a new one. This is SageMaker’s built-in A/B test and a safe way to roll out a new model version.
• Multi-model endpoints. Many models that use the same framework share one fleet of instances and one serving container. SageMaker loads each model from S3 the first time it is called, keeps it cached, and unloads the least used when memory runs short (AWS: multi-model endpoints). This suits per-customer models in a SaaS product. The cost is a cold start the first time a rarely used model is called.

The 4 steps from trained model to live endpoint

Step 1: Package the model artifact

SageMaker expects your model as a model.tar.gz file in S3. For Hugging Face Hub models, the SageMaker SDK can skip this and have the container pull the model from the Hub at start-up. For custom models, you package the weights and any pre-processing code into the tarball.

Step 2: Define the model container

SageMaker provides ready-made Docker images for common frameworks (PyTorch, TensorFlow, Hugging Face). You pick the one that matches your model. If your model needs extra dependencies, you extend one of these images and push it to Amazon ECR. The model (artifact plus container plus IAM role) becomes a SageMaker model record.

Step 3: Create the endpoint configuration

The endpoint configuration names the model, the instance type, the starting instance count and, if you want an A/B test, how traffic splits between production variants. For serverless it holds the memory size and maximum concurrency instead.

Step 4: Deploy the endpoint

SageMaker starts the instances, pulls your model, starts the container and checks that it is healthy. From then on you call it through the SageMaker Runtime API (invoke_endpoint in boto3) by endpoint name. The call is signed with your IAM credentials, so the endpoint is not open to the internet. To serve a public website, put API Gateway or a Lambda function in front of it.

Those are three separate records: model, endpoint configuration and endpoint. Keep that in mind, because cleaning up means deleting all three.

Cold starts, and what actually fixes them

A real-time endpoint has its model loaded and waiting, so a request never waits for start-up. The exception is scaling out: a new instance has to start and load the model before it takes traffic, while the existing instances keep serving. That is why the scaling policy in the script below adds capacity quickly and removes it slowly.

A serverless endpoint has no compute running when it is idle. The first request after a quiet spell, or a burst that goes past the current number of running copies, waits while SageMaker starts one. AWS doesn’t give a fixed number for this. It depends on model size, how long the model takes to download and how long the container takes to start. Measure it with the OverheadLatency CloudWatch metric instead of guessing.

The fix AWS documents is Provisioned Concurrency: you keep a set number of serverless copies warm, ready to answer in milliseconds, and pay for that warm capacity whether or not it’s used. It suits predictable bursts, such as a feature everyone uses at 9 AM. If you need warm capacity all day, compare the price with a small real-time endpoint first.

End project: deploy a Hugging Face model to a real-time endpoint with autoscaling, then tear it down

This script deploys distilbert-base-uncased-finetuned-sst-2-english (a sentiment classifier) to a SageMaker real-time endpoint, sets up autoscaling, runs three test predictions, and then deletes everything it made once you press Enter. It works with any AWS account that has the IAM permissions from the previous article.

#!/usr/bin/env python3
"""
Deploy a HuggingFace model to SageMaker real-time endpoint with autoscaling.

Requirements:
  pip install "sagemaker<3" boto3
  (SageMaker Python SDK v3 replaced HuggingFaceModel with ModelBuilder;
   this script uses the v2 API, so pin it.)

Environment:
  AWS_DEFAULT_REGION set, or pass region_name below.
  SAGEMAKER_EXECUTION_ROLE_ARN — ARN of your SageMaker execution role.
"""

import os
import json
import time
import boto3
import sagemaker
from sagemaker.huggingface import HuggingFaceModel

# --- Config ---
REGION = os.environ.get("AWS_DEFAULT_REGION", "ap-south-1")
ROLE_ARN = os.environ["SAGEMAKER_EXECUTION_ROLE_ARN"]
ENDPOINT_NAME = "sentiment-classifier-v1"

# HuggingFace model hub ID
HF_MODEL_ID = "distilbert-base-uncased-finetuned-sst-2-english"
HF_TASK = "text-classification"

# Instance type: ml.m5.large is cheapest for CPU inference
# Use ml.g4dn.xlarge for GPU if latency is critical
INSTANCE_TYPE = "ml.m5.large"
INITIAL_INSTANCE_COUNT = 1

# Autoscaling config
MIN_CAPACITY = 1
MAX_CAPACITY = 3
TARGET_INVOCATIONS_PER_INSTANCE = 50   # scale out when avg > 50 req/min per instance


def deploy_endpoint():
    sess = sagemaker.Session(boto_session=boto3.Session(region_name=REGION))

    # HuggingFaceModel pulls directly from the Hub — no S3 upload needed
    hf_model = HuggingFaceModel(
        model_data=None,          # None = pull from Hub at runtime
        env={
            "HF_MODEL_ID": HF_MODEL_ID,
            "HF_TASK": HF_TASK,
        },
        role=ROLE_ARN,
        transformers_version="4.37",
        pytorch_version="2.1",
        py_version="py310",
        sagemaker_session=sess,
    )

    print(f"Deploying {HF_MODEL_ID} to endpoint '{ENDPOINT_NAME}'...")
    predictor = hf_model.deploy(
        initial_instance_count=INITIAL_INSTANCE_COUNT,
        instance_type=INSTANCE_TYPE,
        endpoint_name=ENDPOINT_NAME,
        wait=True,
    )
    print(f"Endpoint deployed: {ENDPOINT_NAME}")
    return predictor


def configure_autoscaling(endpoint_name):
    client = boto3.client("application-autoscaling", region_name=REGION)

    resource_id = f"endpoint/{endpoint_name}/variant/AllTraffic"

    # Register the endpoint as a scalable target
    client.register_scalable_target(
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
        MinCapacity=MIN_CAPACITY,
        MaxCapacity=MAX_CAPACITY,
    )

    # Target tracking policy: scale based on invocations per instance
    client.put_scaling_policy(
        PolicyName=f"{endpoint_name}-scaling-policy",
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
        PolicyType="TargetTrackingScaling",
        TargetTrackingScalingPolicyConfiguration={
            "TargetValue": TARGET_INVOCATIONS_PER_INSTANCE,
            "PredefinedMetricSpecification": {
                "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
            },
            "ScaleInCooldown": 300,   # seconds before scaling in
            "ScaleOutCooldown": 60,   # seconds before scaling out
        },
    )
    print(f"Autoscaling configured: {MIN_CAPACITY}–{MAX_CAPACITY} instances")


def test_endpoint(endpoint_name):
    runtime = boto3.client("sagemaker-runtime", region_name=REGION)

    test_inputs = [
        "This product is absolutely fantastic, highly recommend!",
        "Terrible experience, the item broke after one day.",
        "It works as described, nothing more nothing less.",
    ]

    print("\nRunning test predictions:")
    for text in test_inputs:
        payload = json.dumps({"inputs": text})
        response = runtime.invoke_endpoint(
            EndpointName=endpoint_name,
            ContentType="application/json",
            Body=payload,
        )
        result = json.loads(response["Body"].read())
        label = result[0]["label"]
        score = round(result[0]["score"], 3)
        print(f"  Input:  {text[:60]}")
        print(f"  Output: {label} (confidence: {score})\n")


def tear_down(predictor, endpoint_name):
    """Remove everything deploy_endpoint() and configure_autoscaling() created."""
    client = boto3.client("application-autoscaling", region_name=REGION)
    resource_id = f"endpoint/{endpoint_name}/variant/AllTraffic"

    # Scaling first, so nothing tries to resize an endpoint that is going away
    client.delete_scaling_policy(
        PolicyName=f"{endpoint_name}-scaling-policy",
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    )
    client.deregister_scalable_target(
        ServiceNamespace="sagemaker",
        ResourceId=resource_id,
        ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    )

    # Deleting the endpoint stops the instance charges. It does NOT delete the
    # endpoint configuration or the model record, so remove those too.
    predictor.delete_endpoint(delete_endpoint_config=True)
    predictor.delete_model()
    print(f"Deleted endpoint, endpoint config and model for '{endpoint_name}'")


def main():
    predictor = deploy_endpoint()
    configure_autoscaling(ENDPOINT_NAME)
    test_endpoint(ENDPOINT_NAME)

    print("\nEndpoint is live. Invoke it from any service with IAM permission:")
    print(f"  Endpoint name: {ENDPOINT_NAME}")
    print(f"  Region:        {REGION}")

    input("\nPress Enter to tear everything down and stop billing...")
    tear_down(predictor, ENDPOINT_NAME)


if __name__ == "__main__":
    main()
Copy

Run it with python deploy.py. Deployment takes several minutes. After that you have a live endpoint you can call from anything in AWS with the right IAM permission, such as a Lambda function, an EC2 instance or an ECS container, using invoke_endpoint().

The autoscaling policy watches invocations per instance per minute. If traffic triples, SageMaker adds instances, up to three. When traffic drops, it waits five minutes before removing any, so a short lull doesn’t trigger a scale-in right before the next spike.

What it costs, and how to stop paying

A real-time endpoint is billed for its instances for as long as the endpoint exists, whether or not anyone calls it. That is the single most common surprise on a learner’s AWS bill. Instance prices differ by type and region, so look yours up on the SageMaker AI pricing page before you deploy, and multiply by the hours you expect to leave it running.

Stopping the charge is a delete, but deleting an endpoint is not the whole job. AWS is explicit that deleting an endpoint does not delete the endpoint configuration or the model record (AWS: delete endpoints and resources). Those two don’t run instances, but they pile up, and an old configuration makes it easy to recreate an expensive endpoint by accident. The tear_down() function in the script removes, in order:

1. The scaling policy, and the endpoint’s registration as something to scale
2. The endpoint, which stops the instance charges
3. The endpoint configuration
4. The model record

Two things survive even that: the model artifact in S3 and any custom image in ECR. Deleting the model record never touches them. Both are ordinary storage charges, small but not zero, so delete them when you’re done with the project.

From the command line, the same clean-up is:

aws sagemaker delete-endpoint --endpoint-name sentiment-classifier-v1
aws sagemaker delete-endpoint-config --endpoint-config-name sentiment-classifier-v1
aws sagemaker list-models --name-contains huggingface   # find the model name
aws sagemaker delete-model --model-name <model-name-from-the-list>
Copy

(The SDK names the endpoint configuration after the endpoint; the model gets a generated name, which is why you list it first.)

Knowing how to tear infrastructure down is as important as knowing how to create it. If you’re still deciding whether you need SageMaker at all, or whether a managed model behind Amazon Bedrock would do, this comparison of the two walks through the decision.

Build on this in the AWS + ML path →