Asking how to train an LLM usually means one of four quite different projects, with costs separated by orders of magnitude. Pretraining a model from scratch is a research-scale undertaking almost no organisation should attempt. Continued pretraining adapts an existing model to a specialised domain. Fine-tuning teaches a model a task or style using modest examples. Adapter methods do the same while updating only a fraction of the parameters. This guide covers what each involves, what data and evaluation each requires, and how to choose between them.
Understanding the Four Approaches
The approaches differ in how much of the model they change, which determines cost, data requirement, and risk. Full pretraining builds capability from nothing. Continued pretraining extends an existing model’s knowledge into a domain. Fine-tuning adjusts behaviour on a specific task. Adapter methods achieve similar behavioural change while leaving most weights untouched. Matching approach to objective prevents both the waste of over-engineering and the disappointment of expecting behaviour change from a method that cannot deliver it.
Pretraining From Scratch
Building a foundation model requires enormous compute, vast datasets, and specialist expertise. It is justified only for organisations whose core business is model development, not for applying models to a domain.
Continued Pretraining on Domain Text
Feeding large volumes of unlabelled domain text to an existing model extends its familiarity with specialised vocabulary and conventions. This suits fields where general models handle terminology poorly.
Supervised Fine-Tuning on Examples
Providing input and output pairs teaches a specific task or response style. This is the approach most business applications need, and it works with hundreds to thousands of examples.
Parameter-Efficient Adapters
Techniques updating a small fraction of parameters achieve most of the benefit of fine-tuning at far lower compute cost, while allowing several task-specific variants over one base model.
Choosing Based on the Gap You Are Closing
Missing knowledge points toward continued pretraining or retrieval. Wrong behaviour or format points toward fine-tuning. Diagnosing the gap correctly determines whether LLM development effort produces results.
Preparing Data That Produces Good Behaviour
Training data determines what the model learns, including every inconsistency it contains. This is more consequential in fine-tuning than in traditional machine learning, because the model absorbs style, tone, and reasoning patterns alongside correct answers. A dataset assembled from whatever examples were convenient teaches the model to imitate that inconsistency. Careful, smaller datasets built to a clear specification consistently outperform larger ones collected without one.
Defining the Target Behaviour Precisely
Write down what a good response looks like before collecting examples. Ambiguity here produces a dataset that pulls the model in several directions simultaneously.
Keeping Examples Consistent in Style
Variation in tone, structure, and detail across examples teaches the model that any of them is acceptable. Consistency matters more than volume for behavioural training.
Covering the Range of Real Inputs
Include the awkward, ambiguous, and out-of-scope cases the model will meet. Training only on clean examples produces confident failures on everything else.
Teaching Refusal and Uncertainty
Include examples where the correct response is declining or expressing uncertainty. Models trained only on confident answers produce confident answers regardless of whether they should.
Building a Repeatable Data Pipeline
Datasets need regeneration as requirements change. Establishing extraction and preparation through data engineering capability makes iteration practical rather than manual.
Evaluating Whether Training Actually Worked
Evaluation is where most training projects are weakest, because it is harder than it looks and less interesting than the training itself. Loss metrics indicate that learning occurred, not that behaviour improved on anything you care about. Without a held-out test set and a defined judgement of quality, teams end up assessing output by impression, which reliably favours the newest model regardless of whether it is better.
Holding Out a Test Set
Reserve examples the model never sees during training. Evaluating on training data measures memorisation and produces confidence unsupported by production behaviour.
Defining Quality Criteria Explicitly
Decide what correct means for your task, including format, accuracy, and tone. Without written criteria, evaluation becomes preference and comparisons between versions become unreliable.
Comparing Against a Prompting Baseline
Always measure against the untuned model with good prompting. Fine-tuning frequently produces less improvement than expected, and knowing this before deployment saves ongoing cost.
Testing for Regression
Improving one task can degrade others the model previously handled. Test broader capability rather than only the target behaviour, particularly after continued pretraining.
Monitoring Performance in Production
Behaviour on real inputs differs from test performance. Continuous observation through MLOps services catches drift that pre-deployment evaluation cannot anticipate.
Weighing Cost, Maintenance, and Alternatives
Training carries ongoing obligations that teams underestimate at the outset. A tuned model is a version you now maintain, re-evaluate against newer base models, and retrain as requirements change. Base models improve rapidly, and a fine-tune that outperformed the general model at the time is frequently overtaken within months. Accounting for this maintenance burden, rather than only the initial training cost, is what makes the build decision realistic.
Counting Total Cost Rather Than Compute
Compute is often the smaller expense. Data preparation, evaluation design, and ongoing maintenance typically dominate, and omitting them produces budgets that fail early.
Planning for Base Model Obsolescence
Newer base models may surpass your tuned version. Architect so that switching is practical, and re-evaluate periodically rather than assuming your tune remains superior.
Considering Retrieval and Prompting First
Many objectives are met without training at all. Establishing what prompting and grounding achieve, informed by AI consulting services, avoids unnecessary projects.
Assessing Deployment and Serving Costs
Self-hosted models carry infrastructure and operational cost that hosted inference does not. Include serving in the comparison rather than only training.
Building Internal Capability Deliberately
Training creates an asset requiring people who can maintain it. Access to machine learning engineers determines whether the model improves or slowly becomes a liability.
Frequently Asked Questions
Should I train my own LLM?
Almost certainly not from scratch. Fine-tuning or adapting an existing model achieves what most organisations need at a fraction of the cost. Pretraining is justified only where model development is your core business rather than an application of it.
How much data do I need to fine-tune?
Hundreds to a few thousand high-quality, consistent examples for most tasks. Consistency of style and coverage of real input variation matter considerably more than volume, and a smaller careful dataset usually beats a larger inconsistent one.
What is the difference between fine-tuning and continued pretraining?
Continued pretraining feeds large volumes of unlabelled domain text to extend the model’s familiarity with a field. Fine-tuning uses labelled input and output pairs to teach a specific task or response style. They address different gaps.
Will fine-tuning stop the model making things up?
Not reliably. Fine-tuning shapes behaviour and format rather than guaranteeing factual accuracy. Grounding responses in retrieved source content addresses factual reliability more directly than training on additional examples does.
How do I know if training improved anything?
Compare against a held-out test set and against the untuned model with good prompting, using written quality criteria. Impression-based comparison consistently favours the newest version regardless of whether it actually performs better.
How often do tuned models need retraining?
Whenever requirements change, when performance drifts on production data, and when a newer base model may have surpassed your tuned version. Treating a tune as permanent is how organisations end up maintaining an outdated model.
