How to Fine Tune Mistral on TextGen WebUI

In this tutorial, I will guide you through the process of fine-tuning a Mistral model using TextGen WebUI. By the end of this guide, you'll be able to configure the necessary formats, generate training data, and apply LoRA adapters to get your model fine-tuned to answer in German for fun! You can adapt this process for your own use cases.

For this tutorial, we'll generate around 400 datasets using GPT-4, demonstrating how easy and cost-effective it is to fine-tune a model. We'll be using a rented GPU on Vast.ai. For more information on setting up Vast.ai, you can check out my other tutorials.

Step 1: Clone the TextGen WebUI Repository

First, navigate to the TextGen WebUI GitHub repository and clone the repository to your local machine. This can be done using the following command:

git clone https://github.com/oobabooga/text-generation-webui.git

After cloning, navigate to the directory and run the startup script:

cd text-generation-webui
./start_linux.sh

When prompted to select a GPU, choose the NVIDIA option (option A).

Step 2: Create a New Data Format for Mistral

For this tutorial, we will use the model teknium/OpenHermes-2.5-Mistral-7B. You can choose any other Mistral model as well. To create a new data format for Mistral, follow these steps:

Create a file named mistral.json and place it in the /training/format/ directory. The content of the file should be:

{
    "instruction,output": "<s>[INST] %instruction% [/INST] %output%"
}

After saving the file, reload the UI to apply the new format.

Step 3: Generate Test Data

Now, we need to create a prompt to generate the test data. This step involves using GPT-4 to generate around 400 datasets. Below is an example prompt to generate the test data:

Step 4: Load Data Format and Configure Training Parameters

Next, load the Mistral data format and the dataset in the Training tab. Set two configuration parameters for the LoRA training:

• Number of epochs
• Addition of the EOS token by the trainer

Once configured, click "Start LoRA Training" as shown in the screenshot below:

Step 5: Apply the LoRA Adapter

After the training is complete, go to the model tab and reload the model. Then apply the LoRA adapter as depicted below:

Ensure that you see the message “Successfully applied the LoRAs”.

Step 6: Configure Inference Parameters

Before testing the model, you need to change the instruction template for Mistral. Navigate to the Parameters tab and select Mistral from the drop-down menu as shown below:

Step 7: Test the Model

Now, it's time to test your model. Go to the chat tab, select "Instruction", and enter any prompt. The model should respond in German, as specified in the training data. Here is an example:

More Tips

If you plan to perform more training sessions, make sure to always unload and reload the model before applying new LoRAs. This ensures that the LoRAs are applied correctly.

Happy prompting!