How to Train AI on Your Own Data: A Simple Guide

I am a student who over the past few years has learned all of this through trial and error through breaking things, scouring documentation, and asking plenty of stupid questions. This guide is what I wished had been handed to me from the get-go. I have not tried each and every one of these examples on each and every type of GPU, and will be completely clear on where my work is verified and where it is not.

The real problem: generic models don’t know your world

Pre-trained models are amazing. They can craft essays, describe code, or even write a pretty good birthday poem. Ask them something more specific about your company’s internal refund policy, notes you wrote down yourself, or any obscure legal document, and they’ll either guess or come up with something.

Closing that gap is what “training AI on your own data” means to bridge. However, the term itself is somewhat deceptive, since it implies at least three very distinct concepts: prompt engineering, retrieval-augmented generation (RAG), and fine-tuning. They address different issues, have different costs associated with them and are prone to failure in different ways. Picking the wrong one is a beginners’ mistake number one.

One of the most important lessons I’ve learned: don’t fine-tune a model to make it learn facts. Fine-tuning is used for learning behaviour style, formatting, tone, tasks specifics. Facts belong in retrieval. It’s something multiple independent sources agree on.

A short history: how we got here

It’s important to review the evolution of the field in order to understand what makes training complicated.

2018-2019: The BERT era. With BERT, people proved that it was possible to pre-train a big model using unlabeled text, and then fine-tune it using one more layer only for specific tasks such as question answering or sentiment analysis. This way, people introduced the first “pre-train, then fine-tune” paradigm. It was cheap to fine-tune because the only thing you had to change was a tiny task-specific head.

2022: Instruction tuning and RLHF. With the release of InstructGPT paper, OpenAI showed that a small model can be fine-tuned on human demonstrations and preference rankings to become better than the huge base model. This is the paradigm used to train current chatbots: supervised fine-tuning (SFT) followed by reinforcement learning from human feedback (RLHF). The point here is that alignment instruction following and helpfulness of a model require special data.

2021-2023: The Efficiency Revolution. Fine-tuning of large models remained inefficient. LoRA (Low-Rank Adaptation), introduced in 2021, froze the weights of the original model and performed training using small adapter matrices, decreasing the number of trainable parameters by up to 10,000x in comparison with the fine-tuning of GPT-3 175B. And QLoRA (2023) introduced 4-bit quantization for the base model, which made it possible to fine-tune a 65B-parameters model using a single 48GB GPU maintaining the 16-bit fine-tuning performance. Custom training suddenly became accessible not only for labs that had clusters of A100s.

Present day. The ecosystem has matured. Hugging Face’s peft, trl, and transformers are the de facto standard open-source library suite. Unsloth and other utilities reduced the amount of VRAM required for fine-tuning even further. OpenAI, Google, and Mistral offer managed APIs for fine-tuning without the need to possess any GPUs. RAG tooling evolved into an entire industry on its own. But while the barrier to entry has never been lower the decisions have never been harder to make.

Start by asking whether you really need to train something at all.

Instead of picking up a training script, go through the following decision tree first. This will save you several weeks.

Step 1: Prompt engineering

Are you able to reach 80% of what you want with improved prompts, including examples inside the prompt, or temperature change? This can be done instantly and for free. The vast majority of problems that arise from “model doesn’t understand my domain” could often be solved with more concrete instructions.

Step 2: Retrieval-augmented generation (RAG)

Is there a need to provide the model with access to certain information (be it changing, citable, etc.)? RAG retrieves the appropriate documents from your knowledge base and feeds them to the prompt at query time. No modification of the model needed. The most common choice for enterprise knowledge applications and much cheaper to update than a fine-tuned model.

Step 3: Fine-tuning

Are there certain output formats, tonalities, or limited tasks you want the model to perform on a repeatable basis, without having to give it detailed instructions each time? Fine-tuning is the process to use here, modifying the weights of the model to make such behavior default.

The uncomfortable truth about facts

Fine-tuning is not an effective means of teaching the model new facts the way you might wish. The model can remember certain examples used in training but cannot be updated like a database. If your information changes on a weekly basis, your fine-tuned model will go out of date from the second your training ends. RAG keeps up-to-date because you’re updating the index, not the weights.

To be fair, one provider (Unsloth) claims that fine-tuning can mimic the capabilities of RAG in certain situations. Here’s that perspective for the sake of completeness, but the independent consensus is clear: use RAG for knowledge and fine-tuning for behavior. In doubt, choose RAG.

RAG vs. fine-tuning: what each one actually does

Dimension	RAG	Fine-tuning
What changes	Nothing in the model; the prompt gets extra context	The model’s weights are updated
Best for	Facts, documents, policies, anything that changes	Style, format, tone, narrow task behaviour
Data freshness	Real-time if you update the index	Frozen at training time; retraining needed for updates
Upfront cost	Medium (pipeline + vector store)	Medium–high (data prep + training compute)
Update cost	Low (re-index)	High (retrain)
Traceability	Can cite retrieved sources	No source attribution
Risk	Bad retrieval → bad answers	Catastrophic forgetting, overfitting

While this table is simplified, it reflects the essence of the problem. According to the guidance provided by Microsoft, RAG optimization is geared towards context optimization, while the optimization of the model is achieved through fine-tuning. The systems employed in production in 2026 employ both methods.

How fine-tuning actually works (without the math degree)

Supervised fine-tuning (SFT)

However, the most common type of fine-tuning is SFT (supervised fine-tuning). It involves feeding the model examples of inputs and expected outputs, like customer queries and their correct answers, which then adjust their parameters to output something similar. This is how most fine-tuning tutorials operate.

The efficiency trick: LoRA and QLoRA

Full fine-tuning updates all the parameters of the model. 7 billion for the 7B-parameter model quite an amount.

LoRA fixes the original weights and adds two matrices in each layer. It does not update the whole weight matrix but rather learns the low-rank update. As the authors of the LoRA paper proved, this can decrease the number of trainable parameters up to 10,000 times while keeping the same or better quality of fine-tuning compared to full fine-tuning. Usually, LoRA trains only less than 1% of the model’s parameters.

QLoRA goes one step further: it quantizes the fixed base model to 4-bit and trains LoRA adapters on top. Using QLoRA, the authors managed to fine-tune 65B model using a single 48GB GPU without sacrificing 16-bit precision. If you use consumer GPUs, then QLoRA is the technique you need to train a 7B model in 8-12GB of VRAM.

What you actually have: a tiny adapter file, usually just a few megabytes in size, which is used over the existing base model. You can switch the adapter, share it, and still retain the original base model. That’s the reason why LoRA is so popular it’s modular and cheap.

What you need before you start

Data (The aspect everyone undervalues)

Quality always trumps quantity. A few hundred good examples will do better than thousands of bad examples. The conventional file format used for SFT is JSONL (JSON Lines): an example per line with an array of messages comprising system, user, and assistant turns.

There are five things that are essential for data quality: consistency, accuracy, relevance, diversity, and proper formatting. Your examples contradict each other? The model will learn contradictory things. Your ideal answers have mistakes? The model will learn those mistakes. There is no algorithmic solution to poor data.

Hardware

VRAM will be the major limitation here. The below figures are approximate and will depend on model size, batch size, sequence length, etc.:

• 8GB VRAM: Possible fine-tuning of a small 7B model with QLoRA. Expect small batch sizes and slow performance.
• 12GB VRAM: Will give you better comfort working with 7B models with QLoRA. The range where things become “viable” for many consumer GPUs.
• 16-24GB VRAM: More options with larger models and batch sizes. You may start using LoRA (instead of QLoRA) with 7B models in some cases.
• 48+ GB VRAM: Will let you use QLoRA with larger models, up to 65B, according to the authors of the original paper.

The above figures are consistent throughout different guides, but there are no guarantees that they will work for you.

Software stack

The canonical stack using open source software components is:

• PyTorch for the low-level computations.
• Hugging Face Transformers to load the model and tokenizer.
• PEFT for LoRA and parameter-efficient tuning in general.
• TRL for the supervised fine-tuning trainers.
• bitsandbytes for quantization (QLoRA).
• Accelerate for device handling.

Unsloth is an optimisation layer that potentially lowers the VRAM footprint and speeds up the training on consumer hardware.

A realistic QLoRA workflow (conceptual)

Please, take this as a guide, but don’t assume that you can use it out of the box without testing it on your machine. Official documentation on Hugging Face and Unsloth contains all you need to know about working solutions.

1. Install the stack. pip install transformers peft trl bitsandbytes accelerate datasets
2. Quantize base model using BitsAndBytesConfig with load_in_4bit=True and bnb_4bit_quant_type="nf4".
3. Configure LoRA. Use LoraConfig with rank parameter (r=16, for example), lora_alpha, and target modules (e.g., q_proj, v_proj).
4. Prepare dataset. Make it in the format of JSONL file containing arrays of messages.
5. Set up trainer. Use SFTTrainer from TRL and configure training arguments (learning rate, number of epochs (usually 1-3), batch size, gradient accumulation).
6. Train. Observe the change of validation loss. Interrupt training once the validation loss stops decreasing and begins to increase – it’s an indicator of overfitting.
7. Evaluate. Training loss won’t tell you much. Validate the model on unseen data and make sure that it works fine in general case (catastrophic forgetting check).
8. Save and load adapter. Adapter is small, while base model stays intact.

The most frequent mistake that has occurred when I worked on my initial projects was neglecting Step 7. It is nice to see that your training loss is decreasing, but it means absolutely nothing regarding how well your model is performing.

A realistic RAG workflow (conceptual)

RAG is usually the better way to go. This is what it looks like:

1. Gather your documents. Whatever form these documents take: PDFs, text files, markdowns, database entries these are the sources of your knowledge.
2. Chunk them. Into chunks of a few hundred tokens each. Chunking is very important for RAG; poor chunking will lead to poor retrieval.
3. Embed the chunks. Using an embedding model, e.g. sentence-transformer, or a vector embedding service such as Cohere, Anthropic, or a host of other providers, turn each chunk into a vector.
4. Store the vectors. In some kind of vector database: FAISS, Chroma, Qdrant, Pinecone, Weaviate anything that works.
5. Then, at query time: embed the question asked by the user, retrieve the most relevant chunks, and feed them to the LLM as context.
6. Answer the question. The LLM uses the retrieved context to generate the answer. If citations are desired, mapping from answer back to context is possible.

With LangChain or LlamaIndex, it’s possible to create a basic RAG pipeline in a day. It won’t be production-ready, but it will help understand retrieval quality better than any blog post could. What’s more important, once your documents change, all you have to do is re-index; there’s no need for retraining.

Common mistakes and limitations (read this before you train)

1. Catastrophic Forgetting

In fine-tuning on a small dataset, the model tends to lose some of its generalisation capabilities. This phenomenon is referred to as catastrophic forgetting. Although LoRA and QLoRA mitigate the problem compared to fully fine-tuning the model, it remains a concern. The mitigation techniques used are: reducing the number of epochs; applying low learning rates; including some out-of-distribution (general) data in training; and testing the performance of the model on some unrelated tasks. As indicated by the authors of the LoRA paper, parameter-efficient approaches tend to interfere with pre-existing knowledge in the model.

2. Overfitting

In fine-tuning the model for too many epochs, the model learns to memorise your data and does not develop the generalisation capability. The standard fine-tuning process requires about 1-3 epochs. Keep monitoring the loss on the validation set and stop if the loss starts increasing.

3. Data Leakage and Contamination

If the validation data appears in your training data, your evaluation is irrelevant. Properly split your data. If you are fine-tuning on some public benchmarks, confirm if the benchmark is part of the data used to pre-train the base model (often difficult to ascertain).

4. Legal and Privacy Issues

If your training data includes personal information, you should have a legal basis for processing it according to the GDPR and other privacy laws. Consent can serve as a legal ground, but there is also legitimate interest, subject to certain criteria. Utilization of publicly available data is not always a lawful action in relation to AI training. The European Data Protection Board has explicitly stated that purpose limitation principle exists: data gathered for one purpose cannot be used later for training purposes without having a proper legal ground. Health data, financial data, and data on children require legal consultations. I am not a lawyer.

5. Cost traps

Fine-tuning costs are not limited to the training phase itself. There are other associated costs, such as:

• Labelling or curating data (usually the biggest hidden cost).
• Experimenting with failed runs.
• The costs of hosting the fine-tuned model (managed APIs charge an extra cost per token).
• Re-training if the base model evolves or there is a requirement change.

Figures tend to differ quite a bit, from a few dollars per million tokens for training with managed APIs for small models, to higher amounts for larger models, and the cost per token of inference for fine-tuned models will be higher than that of the base model.

Conclusion

The process of training the model using your own data has become more affordable than ever before, but it’s also more tempting to spend time on something unnecessary. If one looks at the evolution of models in the field – from BERT, through InstructGPT and LoRA to QLoRA one would see how the process of adaptation becomes cheaper and more modular, but not magical.

For the beginners, I would recommend to build a RAG pipeline first. Not only it teaches what is important in terms of data, retrieval quality and evaluation, but also solves one of the most common problems: when the model doesn’t know anything about the domain of interest. In case of necessity of consistent behavior try a fine-tuning of a small model with QLoRA on a consumer-grade GPU. Keep the size of the dataset small and clean. And don’t trust anyone who claims that fine-tuning will replace your database.

It’s a lesson I’m still learning myself. The discipline changes quickly, and half of what I’ve written here will be outdated in a year from now. However, the framework for decision making facts versus behaviour, RAG versus tuning, evaluate before you scale remains the same. Start with that.

Explore Our AI Category. And if you are reading it up to here, leave a sweet comment to motivate us to write blog everyday.