Udemy – LLM Serving, Deployment & Scaling – Ollama, vLLM & Ray Serve

A warm welcome to LLM Serving, Deployment & Scaling – Ollama, vLLM & Ray Serve course by Uplatz.

What is LLM Serving?

LLM serving is the process of running a trained language model and making it available to applications or users for inference.

The model has already been trained. Serving begins when a system loads the model into memory and accepts requests such as:

• Summarise this document.
• Answer this question.
• Generate a product description.
• Continue this conversation.

A typical serving flow is:

• Application sends a request.
• The LLM-serving engine receives it.
• The model performs inference.
• The model generates output tokens.
• The response is returned to the application.

When a request reaches the serving system:

• The input text is converted into tokens.
• The model processes the input tokens.
• The model generates output tokens, usually one at a time.
• The tokens are converted back into readable text.
• The response is returned to the application.