Reference

Terms and examples, spelled out.

The jargon used across the Journal, explained once, properly — plus a couple of concrete examples instead of more abstract description.

◇Ollama: Ollama is a command-line tool (with a small local API) for downloading and running open-weight language models on your own hardware, without sending anything to a remote server. It's the base layer most of the local-model posts on this site build on top of.
◇Quantization: Quantization reduces the numerical precision used to store a model's weights, for example from 16-bit down to 4-bit. The model gets smaller and faster to run, at the cost of some accuracy.
✣Agent: An agent wraps a language model with the ability to take actions: reading and writing files, running shell commands, calling APIs, deciding what to do next based on the result.
◇Context length: How much text, measured in tokens, a model can 'see' at once, including your prompt and its own output so far.
◇Benchmark: A repeatable test used to compare how different models perform at a specific task.
⬡Markdown: A plain-text formatting syntax (.md files) for structuring documents without needing a specific app to open them. See the example below.
◇LLM: Large Language Model, the general term for the type of AI model behind tools like ChatGPT, Claude, or the local models discussed here.
◇System prompt: A system prompt is the set of instructions given to a model before the conversation starts, defining its role, tone, rules and boundaries for the whole session. It's set once by whoever configures the assistant, not typed by the user for each message.
◇API key: An API key is a private string a program or account holder sends along with each request to a hosted API, proving they're allowed to use it and letting the provider bill or rate-limit by that key. Treat one like a password, since anyone holding it can rack up usage on your account.
◇Chain-of-thought: Chain-of-thought is when a model writes out its intermediate reasoning steps before giving a final answer instead of jumping straight to a conclusion. Prompting a model to think step by step is a simple way to trigger this, and it often improves accuracy on multi-step problems.
◇CSS: CSS, short for Cascading Style Sheets, is the language that controls how a website looks. It handles colors, spacing, fonts and layout, while HTML provides the actual content and structure underneath.
◇Deploy: Deploying means taking code that works on your own machine and putting it live where other people can actually reach it, for example pushing a website to a real server. On this site that step happens through Cloudflare's wrangler deploy command.
◇Diff: A diff shows exactly what changed between two versions of a file, line by line, instead of making you compare the whole thing by eye. It's one reason plain text formats like Markdown are pleasant to work with, since a diff of a .md file is just readable text.
◇Fine-tuning: Fine-tuning means taking an already-trained model and training it further on a smaller, specific dataset so it gets better at one particular task or style. It's a heavier, more involved alternative to writing a better prompt or system prompt.
◇Function calling: Function calling is when a model doesn't just reply with text but outputs a structured request to run a specific function or tool, like checking a weather API or querying a database, then uses the result to continue. It's the technical mechanism that makes agents possible.
◇Git: Git is a version control system that tracks every change made to a set of files over time, so you can see exactly what changed and when, and roll back if something breaks. It keeps a project's history intact instead of relying on filenames like final_v3_reallyfinal.docx.
◇Guardrails: Guardrails are rules or filters layered around a model to keep its behavior inside acceptable limits, for example blocking certain topics or refusing to leak its own system instructions. They sit alongside the system prompt rather than replacing it, catching cases the prompt alone doesn't cover.
◇Hallucination: A hallucination is when a model states something false with full confidence, inventing a fact, a source or a detail that simply isn't true. It happens because the model is predicting plausible-sounding text rather than looking anything up, so it's worth double-checking anything that matters.
◇Inference: Inference is the moment a trained model produces an answer to a specific input, as opposed to training, the earlier and much longer process of teaching the model in the first place. Every time you send a prompt and get a reply, that's inference happening.
◇LoRA: LoRA, short for Low-Rank Adaptation, is a lightweight way to fine-tune a model by training a small set of additional weights instead of adjusting the whole model. It needs far less hardware and time than full fine-tuning, which is why it's popular for local and hobby setups.
◇MCP: MCP, short for Model Context Protocol, is a standard way for a model or agent to connect to external tools, files and data sources without needing custom integration code for each one. It aims to do for agent tooling what a universal plug does for electronics.
◇Model card: A model card is the documentation a model's creator publishes alongside it, covering what it was trained on, known limitations, intended use cases and license. It's the closest thing to a nutrition label for an AI model, worth reading before trusting one with anything important.
◇Open-weight model: An open-weight model has its trained parameters published and downloadable, so anyone can run it on their own hardware, unlike a closed model such as GPT-4 that only exists behind a company's API. Ollama exists specifically to run open-weight models locally.
◇Parameters: Parameters are the internal numbers a model learns during training and uses to produce its output. The total count, like the 7B in a 7B model, is a rough proxy for its size and capability. More parameters usually mean better results but also more memory and compute to run it.
◇Prompt injection: Prompt injection is when text from an untrusted source, like a webpage or a document, contains hidden instructions trying to hijack a model or agent into doing something other than what the user actually asked for. It's a real risk for anything that lets a model read external content.
◇RAG: RAG, short for Retrieval-Augmented Generation, means a model looks up relevant information from an external source, like a folder of documents, before generating its answer, rather than relying only on what it learned during training. It's the idea behind letting a model answer questions about your own notes instead of just general knowledge.
◇Repository: A repository, usually shortened to repo, is the folder of files and their full history tracked by a version control system like Git, often hosted somewhere like GitHub so others can see or contribute to it. Every project on this site lives in its own repo.
◇Sampling: Sampling settings like top-p and top-k control how a model picks its next word out of all the plausible options, rather than always taking the single most likely one. Lower values make output more focused, higher values make it more varied, similar to what temperature does through a different mechanism.
◇Temperature: Temperature is a setting that controls how predictable or varied a model's output is. A low value produces safe, repetitive answers, a high value produces more surprising, sometimes less coherent ones. It's usually one of the first knobs people try when output feels too flat or too chaotic.
◇Terminal: A terminal, also called a command line, is a text-based way to interact directly with a computer by typing commands instead of clicking through menus and windows. Most local AI tools, including Ollama, are run and controlled from a terminal.
◇Token: A token is the small chunk of text a model actually processes, often a word, part of a word, or a punctuation mark. Tokenization is the step that breaks input into these pieces before the model ever sees it, which is why context length is measured in tokens rather than words.
◇Vector embedding: A vector embedding turns a piece of text into a list of numbers that captures its meaning, so texts with similar meaning end up with similar numbers even if they share no words. This is what makes it possible to search a pile of notes by meaning instead of just by keyword.
◇VRAM: VRAM is the dedicated memory on a graphics card, and it's usually the hard limit on which local models you can actually run, since the whole model needs to fit in it to run at a reasonable speed. This is the practical reason a smaller, more quantized model sometimes beats a bigger one on the same machine.
◇Zero-shot / few-shot prompting: Zero-shot prompting asks a model to do a task with no examples at all, relying purely on the instructions. Few-shot prompting gives it a small handful of examples of the input and output wanted first. Few-shot usually gets more consistent results when the task is unusual or the exact format matters.
⬡Obsidian AI Vault: Obsidian AI Vault is one of my own projects, listed as "soon" under Currently Building on the homepage. It's a local system that indexes an Obsidian notes vault with embeddings, so notes can be searched by meaning instead of by exact keyword, all running on my own machine.
ϟAI Control Center: AI Control Center is one of my own projects, a local dashboard for launching, monitoring and organizing AI tools, models and agents running on my Mac. It's listed under Currently Building on the homepage, already a working prototype rather than just an idea.
◇nomic-embed-text: nomic-embed-text is a small open-weight embedding model that runs locally through Ollama, turning text into vector embeddings for tasks like semantic search. It's the model Obsidian AI Vault currently uses to index notes.
◇Open WebUI: Open WebUI is a self-hosted, browser-based chat interface for models running through Ollama. It gives them a ChatGPT-style front end instead of the command line. Settings like context length are configured per model through a Modelfile, a plain text config file Ollama reads when loading that model.
◇pipx: pipx installs and runs Python command-line tools in their own isolated environment, without touching a project's regular pip install or requiring a virtual environment to be set up by hand. It's now the recommended way to install Local LLM Matrix, mainly because it sidesteps most of the setup problems that used to show up as messy venv issues.
◇ComfyUI: ComfyUI is a node-based interface for running image generation workflows locally, most often with Stable Diffusion-style models. Instead of one fixed pipeline you wire together nodes for loading a model, sampling and post-processing yourself, which makes it flexible but also means progress and error reporting vary a lot between workflows.
◇Reinforcement learning: Reinforcement learning is a training method where a model learns from trial and reward rather than from labeled examples alone, taking an action and getting a signal for whether the outcome was good or bad. It's part of how models like ChatGPT get tuned to follow instructions and prefer better answers, often called RLHF when the reward signal comes from human feedback.
◇Distillation: Distillation means training a smaller model to mimic the outputs of a larger one, so it picks up most of the bigger model's ability at a fraction of the size and cost. It's one of the main ways today's fast, cheap models get built without training something from scratch.
◇Weights: Weights are the specific numbers inside a model that get adjusted during training and stored as the result, the same numbers quantization compresses and a fine-tuning run changes. The term often gets used interchangeably with parameters, though weights technically means the connections themselves rather than the full count.
◇Validation loss: Validation loss is a score measured on data the model didn't train on, used to check whether training is actually helping it generalize rather than just memorizing. A validation loss that stops improving, or starts rising, while training loss keeps dropping is the classic sign of overfitting.

Examples

Less description, more showing.

What a real .md note looks like

A short frontmatter block for metadata, then plain Markdown underneath. Nothing exotic, that's the point.

---
title: Quantization notes
tags: [local-models, ollama]
date: 2026-07-05
status: draft
---

# Why Q4 was good enough here

Tested Q4_K_M vs Q8_0 on a 7B model for summarization.
Barely any quality difference for this task, but Q4 runs
noticeably faster on 16GB of RAM.

## Next
- [ ] Try the same comparison on a 13B model
- [ ] Check if longer context changes the outcome

Takeaway: metadata up top, plain text below. Anything that can open a text file can open this, no export step required, ever.

← Back to the glossary term

Same task, two different prompts

The same request to summarize benchmark results, written for a plain chat LLM versus an agent that can read files and run commands.

Normal chat LLM

Summarize the three benchmark runs below in 3 bullet points, each under 15 words, focused only on what changed between runs.

Run 1: 7B model, Q4, 12.4s avg response, 82% pass rate
Run 2: 7B model, Q8, 18.1s avg response, 84% pass rate
Run 3: 13B model, Q4, 21.7s avg response, 89% pass rate

Agent

Read the three benchmark result files in /results/, write a 3-bullet summary of what changed between them into SUMMARY.md, and run `node validate.js` to confirm the output format is correct before finishing.

Takeaway: the two prompts aren't that different in length or effort to write. The real difference shows up after you hit enter. The chat LLM hands back text for you to read and act on yourself, while the agent goes and does the multi-step work itself, reading the real files, writing a real file, and checking its own output before calling it done.

← Back to the glossary term

System prompt vs. a regular prompt

Same idea as the agent example above, one level earlier: a system prompt is set once and shapes every reply in the session, a regular prompt is the one-off request inside it.

System prompt (set once)

You are a calm, precise German business email assistant. Always reply in the formal "Sie" form, keep emails under 150 words, never use exclamation marks, and always close with a clear next step.

Regular prompt (this message)

Write a follow-up email to a supplier who hasn't confirmed our delivery date yet.

Takeaway: the system prompt doesn't change per message, and that's what makes replies consistent across a whole conversation. Change the system prompt and every future reply shifts tone, even though the regular prompts stay exactly the same.

← Back to the glossary term

A Custom GPT is a saved system prompt, not a new model

OpenAI's Custom GPTs (and similar "projects" or "personas" elsewhere) look like a special product, but under the hood it's the same system-prompt idea from above, just saved behind a name and an icon so nobody has to retype it.

Name: Invoice Checker
Description: Reviews supplier invoices for missing fields and math errors.

Instructions (system prompt):
  Check invoices for: correct VAT calculation, a valid invoice
  number, a matching PO reference, and a due date within 30 days.
  Flag anything missing or wrong as a short bullet list, do not
  rewrite the invoice yourself.

Knowledge files: vat_rules_de.pdf, po_format_reference.pdf
Model: same underlying model as regular chat

Takeaway: it's the same model and the same system-prompt mechanism as the example above, just saved once, given a name, and optionally handed a couple of reference files. Nothing about the model itself changes when you "make a Custom GPT."

← Back to the glossary term