Optimising LLMs: A Practical Guide to Fine-Tuning

Adapting a large language model to a specific objective can be achieved through fine-tuning. This approach enhances a model's ability to adhere to instructions, manage specialised workflows, or format outputs consistently. However, the process demands significant data, computational resources, and rigorous testing. In many scenarios, optimising the prompt or selecting a different model may be sufficient.

This guide outlines the circumstances in which fine-tuning is beneficial, contrasts it with alternative methods, and highlights key considerations for fine-tuning a model such as Qwen3.8-27B on local infrastructure.

What Is Fine-Tuning?

Fine-tuning involves further training an existing model using a dataset of examples. Rather than building from scratch, you leverage a pretrained model and adjust its parameters to boost performance for a specific task or response type.

For instance, a general-purpose language model can generate code, explain errors, and answer technical queries. By fine-tuning it with high-quality examples of your own coding standards and review processes, the model can better align with those conventions.

It is important to note that fine-tuning does not guarantee increased intelligence or superior performance across all tasks. Outcomes rely on the base model, the quality of the training data, and the training methodology. A poorly curated dataset can actually diminish the model's utility.

Why Fine-Tune a Model?

The primary motivation for fine-tuning is to enhance a model's reliability for a specific task that is critical to your workflow.

Consistent output: Train the model to generate reports, structured responses, or code in a specific format.
Specialised tasks: Enhance performance on narrow tasks, such as classifying technical issues or reviewing code against defined conventions.
Instruction following: Instruct the model to follow a specific process or respond according to a consistent set of requirements.
Repeated workloads: Adapt the model for frequently occurring tasks, where even modest improvements can yield significant value.

Fine-tuning is particularly worth exploring when you possess numerous examples of the desired behaviour and the base model consistently fails in similar ways. By training on these examples, you can measure whether performance improves.

Conversely, it is less effective when the issue stems from missing information, ambiguous instructions, or a base model that lacks the necessary capabilities. Fine-tuning cannot reliably overcome all inherent limitations.

Fine-Tuning vs. Prompting vs. RAG

Fine-tuning is not the sole method for improving a language model. Prompting and retrieval-augmented generation (RAG) often address issues with less effort.

Prompting

Prompting involves providing the model with instructions, examples, and context during inference. If you require a specific tone, adherence to a response template, or execution of a straightforward task, this is the recommended starting point. It requires neither model training nor dataset preparation.

Retrieval-Augmented Generation (RAG)

RAG feeds relevant information to the model as it generates answers. This is ideal when the model needs access to internal documentation, technical manuals, or dynamic information. You can update the knowledge source rather than retraining the model whenever data changes.

Fine-Tuning

Fine-tuning alters the model's learned behaviour through additional training. Consider this approach when prompting alone fails to produce consistent results, or when repeated tasks benefit from specialised training examples.

These methods are also complementary. A fine-tuned model can leverage RAG to retrieve current information, while prompting provides task-specific instructions for each individual request.

When Is Fine-Tuning Worth It?

Successful fine-tuning requires a suitable dataset, adequate training resources, and a robust evaluation strategy. Ensure it addresses a genuine problem before committing resources.

A practical strategy involves testing the base model on a representative set of tasks using well-crafted prompts. Document errors and identify recurring patterns. If failures result from missing information, implement RAG. If instructions are unclear, refine the prompt. If the model consistently struggles with a specific behaviour despite clear instructions and context, fine-tuning may be a viable option.

Also evaluate the frequency of the task. Improvements to rarely used workflows may not justify the training costs, whereas slight improvements across thousands of repeated requests can be highly valuable.

Finally, benchmark the results against a more capable base model. Using a stronger model might resolve the issue without training, though it may incur higher hardware or API costs.

Choosing a Fine-Tuning Method

Various methods exist for fine-tuning a model. Key differences lie in the number of parameters updated and the computational memory required for training.

Full fine-tuning: Updates all model parameters. This offers extensive control but generally demands significantly more memory and compute power.
LoRA: Freezes the base model weights and trains a smaller set of additional parameters known as adapters. This reduces the number of trainable parameters, making the process more practical.
QLoRA: Utilises a quantised base model while training adapters. This further reduces memory requirements, though supported architectures and configurations may vary.

For many local experiments, LoRA or QLoRA represents a sensible starting point. You can train and evaluate an adapter without updating every parameter in the original model. The optimal method depends on the specific model, available hardware, and the degree of adaptation required.

Can You Fine-Tune Qwen3.8-27B on Local Hardware?

Qwen3.8-27B is an example of an open-weight model attractive for coding, research, and other specialised workloads. Fine-tuning it locally offers control over training data and the deployment environment, but a model of this scale requires careful hardware planning.

With approximately 27 billion parameters, loading weights in BF16 alone necessitates around 54 GB of memory. Training requires additional memory for activations and intermediate data. Full fine-tuning introduces further demands due to gradients and optimiser states.

LoRA reduces the number of trainable parameters, while QLoRA can lower the memory footprint of the base model. Neither eliminates the need for memory to handle activations, sequence lengths, and training batches. Actual requirements vary based on the training framework and configuration.

Before acquiring hardware, verify support for the exact model architecture and training method. Then, estimate VRAM requirements, dataset size, expected training frequency, and storage needs. A GPU capable of running a quantised model for inference may lack the memory necessary for fine-tuning.

If fine-tuning is an occasional requirement, renting a GPU desktop may be more practical than purchasing a dedicated machine. However, if you train regularly or require the entire workload to remain on your network, dedicated on-premise hardware is likely the better fit.

How to Tell Whether Fine-Tuning Worked

A lower training loss does not inherently indicate a superior model. The model may simply be memorising training examples or losing capability on tasks outside the dataset.

Maintain a separate test set that is excluded from training. Compare the original and fine-tuned models on identical prompts using consistent evaluation criteria. Assess accuracy, consistency, instruction adherence, and performance on examples distinct from the training data.

For coding tasks, this involves verifying whether generated code passes tests and follows your conventions. For structured outputs, it means measuring whether the model consistently produces valid responses in the required format.

If the fine-tuned model does not improve on key metrics, revisit the dataset and training approach. You may find that a refined prompt, RAG, or a different base model offers a more effective solution.

Running Fine-Tuned Models on Your Own Hardware

Fine-tuning is only one component of the workflow. You must also run the resulting model, manage resources, and maintain the surrounding environment. The optimal deployment strategy depends on usage frequency and the level of control required.

DaDesktop provides cloud desktops with dedicated GPU resources, available for rent by the day. For continuous workloads, we also design and manage custom on-premise hardware tailored to your requirements. This includes GPU-based systems for machine learning and CPU-based systems for development and other compute tasks.

Discover more about our local LLM solution and machine learning solution. For additional context, refer to our guides on running local LLMs and understanding VRAM requirements.

Have questions about your specific use case? Contact us and we will guide you in the right direction.