Master

Deploying a private model safely

Deploying a local model to a server or team means securing it, monitoring it, and keeping updates safe. Privacy and access control become important.

What you will see

Health check: ok Max latency observed: 25.0s Alert fired above threshold: True

About 1 min of reading

Try It

Deploy a private model with auth, network isolation, and monitoring

Only let trusted services authenticate and reach the model.

def authenticate(token): valid_tokens = {"trusted-service-token"} return token in valid_tokens print() print("Deploying a private model safely") token = "trusted-service-token" print("Trusted service authenticates: %s" % authenticate(token)) untrusted = "unknown-token" print("Unknown service blocked: %s" % (not authenticate(untrusted)))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Deploying a private model safely Trusted service authenticates: True Unknown service blocked: True

Monitor latency and memory with alerts.

def check_health(latency_seconds, memory_gb): if latency_seconds > 10: return "slow" if memory_gb > 20: return "memory high" return "ok" print("Health check: %s" % check_health(3.0, 10.0)) latencies = [2.0, 4.0, 25.0, 3.0] print("Max latency observed: %.1fs" % max(latencies)) print("Alert fired above threshold: True" if max(latencies) > 10 else "Alert fired above threshold: False")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Health check: ok Max latency observed: 25.0s Alert fired above threshold: True

Data Flow

A model is placed on a dedicated, firewalled machine, access is controlled, and monitoring keeps it healthy for real users.

1 Isolate the server

↓

2 Control access

↓

3 Monitor usage

↓

4 Plan updates

Why this works

Access control

Only let trusted services talk to the model. Use authentication and a private network so nobody external can reach it or read its logs.

Monitoring

Watch how long answers take and how much memory is used. A model that slows to a crawl can break the apps that depend on it, so alerts matter.

Examples

Your own model server

Run the model on a dedicated machine
Put it behind a firewall
Allow only trusted apps to connect

Now a team can use the private model without exposing data to the internet.

Health checks and retries

Client -> health check /health
   down -> try next replica
   up -> route traffic

A load balancer sends requests only to healthy instances. If one model server crashes, traffic automatically fails over to a working replica, so users never see a broken page.

Logging without leaking data

Log: request time, token count, status
Do NOT log: the prompt or the answer
   Mask user IDs instead

Logs tell you something is wrong, but they can also leak private data. Never store prompts or answers to disk. Record timing and load only, and keep any identifiers hashed.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Optimising speed and memory

Previous Lesson

Optimising speed and memory