> ## Content Index
> Fetch the complete content index at: https://airabbit.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to Set Up and Run Ollama on a GPU-Powered VM (vast.ai)
- URL: https://airabbit.blog/how-to-set-up-and-run-ollama-on-a-gpu-powered-vm-vast-ai/
- Published: 2024-08-14T09:43:04.000Z
- Updated: 2025-03-21T16:14:27.000Z
- Description: Unlock the power of private model inference with this tutorial on setting up Ollama on a GPU-enabled VM, whether locally or via cloud services like Vast.ai or Runpod. Enhance data security and speed by following our step-by-step guide to install and run your models securely. Perfect for efficient...
- Author: AiRabbit

In this tutorial, we’ll walk you through the process of setting up and using Ollama for private model inference on a VM with GPU, either on your local machine or a rented VM from [Vast.ai ](https://cloud.vast.ai/?ref%5Fid=145250&ref=airabbit.blog)or [Runpod](https://runpod.io/?ref=7su8gs12).io. Ollama allows you to run models privately, ensuring data security and faster inference times thanks to the power of GPUs. By leveraging a GPU-powered VM, you can significantly improve the performance and efficiency of your model inference tasks.

# Outline

1. Set up a VM with GPU on Vast.ai
2. Start Jupyter Terminal
3. Install Ollama
4. Run Ollama Serve
5. Test Ollama with a model
6. (Optional) using your own model

# Setting Up a VM with GPU on Vast.ai

**1\. Create a VM with GPU:** — Visit [Vast.ai](https://cloud.vast.ai/create/?ref=airabbit.blog) to create your VM. — Choose a VM with at least 30 GB of storage to accommodate the models. This ensures you have enough space for installation and model storage. — Select a VM that costs less than $0.30 per hour to keep the setup cost-effective.

![](https://miro.medium.com/v2/resize:fit:1400/1*BmfKGSHXTwv552eWCYyzSA.png)

**2\. Start Jupyter Terminal:** — Once your VM is up and running, start Jupyter and open a terminal within it.

![](https://miro.medium.com/v2/resize:fit:1400/1*5V3zvvO8WMCFR9kr6WXdrg.png)

# Downloading and Running Ollama

1. **Start Jupyter Terminal:** — Once your VM is up and running, start Jupyter and open a terminal within it. This is the easiest method to get started. — Alternatively, you can use SSH on your local VM, for example with VSCode, but you will need to create an SSH key to use it.

![](https://miro.medium.com/v2/resize:fit:1400/1*5FThAEYy7bY-zrOqdexX5g.png)

1. **Install Ollama:** — Open the terminal in Jupyter and run the following command to install Ollama:

```
bash curl -fsSL https://ollama.com/install.sh | sh
```

**2\. Run Ollama Serve:** — After installation, start the Ollama service by running:

````
bash ollama serve &
```

Ensure there are no GPU errors. If there are issues, the response will be slow when interacting with the model.

**3\. Test Ollama with a Model:** — Test the setup by running a sample model like Mistral:

````
bash ollama run mistral
```

You can now start chatting with the model to ensure everything is working correctly.

**Optional (Check GPU usage)**

**Check GPU Utilization:** — During the inference (last step), check if the GPU is being utilized by running the following command:`bash nvidia-smi `\- Ensure that the memory utilization is greater than 0%. This indicates that the GPU is being used for the inference process.

![](https://miro.medium.com/v2/resize:fit:1400/1*9bZnZptF5Wj4eXdo7Y7haw.png)

# Using Your Own Hugging Face Model with Ollama

**1\. Install Hugging Face CLI:** — If you want to use your own model from Hugging Face, first install the Hugging Face CLI. Here we will use an example of a fine tuned Mistral model **TheBloke/em\_german\_mistral\_v01-GGUF em\_german\_mistral\_v01.Q4\_K\_M.gguf**

**2\. Download Your Model:** — Download your desired model from Hugging Face. For example, to download a fine-tuned Mistral model:

````
pip3 install huggingface-hub# Try with my custom model for fine tuned Mistral
huggingface-cli download TheBloke/em_german_mistral_v01-GGUF em_german_mistral_v01.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
```

**3\. Create a Model File:** — Create a model config file ***Modelfile*** with the following content:

````FROM
FROM em_german_mistral_v01.Q4_K_M.gguf

# set the temperature to 1 [higher is more creative, lower is more coherent]
PARAMETER temperature 0

# # set the system message
# SYSTEM """
# You are Mario from Super Mario Bros. Answer as Mario, the assistant, only.
# """
```

**4\. Instruct Ollama to Create the Model:** — Create the custom model using Ollama with the command:

````
ollama create -f mymodel Modelfile
```

**5\. Run Your Custom Model:** — Run your custom model using:

````
ollama run mymodel
```

By following these steps, you can effectively utilize Ollama for private model inference on a VM with GPU, ensuring secure and efficient operations for your machine learning projects.

Happy prompting!