Frequently Asked Questions

A client who runs a furniture shop wrote to us pretty annoyed because the chatbot we’d just installed “took forever” to answer. We grabbed a phone and timed it: 4.2 seconds from the moment the user hit enter to the first word showing up. To us, after months of staring at logs for this stuff, that didn’t sound like much. To someone used to WhatsApp, where the reply shows up almost instantly, it is.

Why an AI chatbot takes so long to answer has a fairly concrete explanation: it depends on the size of the model you’re using, how much context you’re sending with each request, and whether it has to call an external tool (a database, a catalog, another API) before it can reply. A large model, carrying a long conversation history, with an inventory lookup thrown in, can easily take more than three seconds even when everything is working exactly as it should.

Model size matters more than people expect. When we ran the same flow on GPT-4 and on a smaller model built for simple tasks, the difference went from 4 seconds down to 900 milliseconds for questions like “do you have this table in oak?”. The large model was still needed for the odd cases, the ones mixing several conditions or requiring a bit of actual reasoning. For everything else, we were paying extra for precision nobody needed at that point.

What the client actually cared about, though, wasn’t the exact number — it was how it felt. And there’s a trick that changes the perception without touching the real latency: showing the reply word by word as it arrives (what’s usually called streaming), instead of waiting for the full text and dumping it on screen all at once. Psychologically, seeing that something is “already happening” is tolerated far better than a blank screen with a spinner.

There are other things that actually cut the real time, not just the feeling of it. Trimming the system prompt down to what’s strictly needed helps, since every extra token gets processed before anything gets generated. Caching results from external calls that don’t change by the minute — prices, rough stock levels, shipping policies — also removes the slowest part, which is almost never the model itself but the round trip to some other system; it’s the same logic we go into when explaining why you sometimes don’t see changes on a website, just applied to data instead of pages. And using a fast model as a first filter, one that only escalates to the big model when the question actually calls for it, is basically what we did for the furniture shop: same inbox, two speeds depending on what’s needed.

The furniture client is still running that mixed setup months later. What we haven’t fully figured out, and we won’t dress it up, is exactly where to draw the line between “this is simple, send it down the fast path” and “this needs the big model”: every time that rule gets tweaked, some new question shows up that doesn’t quite fit in either drawer.

→ Direct link to this question

A client called us a couple of months ago, pretty annoyed: the customer-service chatbot we’d built for them had started recommending competitors’ products. It wasn’t a model glitch. It was the system prompt, badly written, that never told the bot what to do when it didn’t know the answer.

A chatbot’s system prompt is the block of instructions sent to the model before every conversation, invisible to whoever’s typing, and it sets the bot’s role, tone and limits; change a handful of sentences there and the same model can go from sounding like a generic bot to acting like the assistant for one specific brand.

In practice it’s a text file, sometimes two paragraphs, sometimes four pages. We put things in it like “you’re this store’s assistant, you only talk about products in this catalog” or “if you don’t know the shipping cost, say so, don’t make up a number.” That’s where 80% of the real work is when we build an agent or chatbot for a client: not in picking the model, in tuning that text against weird conversations a normal user would never type in a demo.

What nobody tells you starting out is that the same prompt behaves differently depending on the model. We had one that worked great on GPT-4 and turned evasive on Claude with the exact same instructions, word for word. Had to rewrite half the instruction just because of that provider switch.

And there’s a limit we sometimes forget to flag for the client: a system prompt isn’t security. A patient user can get the bot to ignore its instructions. Stopping that takes more than a nicely written paragraph up front: filters outside the prompt, backend validation, limits that don’t rely on the model “remembering” to behave.

→ Direct link to this question

A few months back I built the internal search for an online store that sells fishing gear, a client whose whole site we already look after. Someone typed “spinning rod” into the search box and got nothing back, even though the product was listed as “fishing rod for spinning” and was sitting right there in stock, photos and all. Old-school search compares words: if the query doesn’t roughly match the title or description letter for letter, you get zero results. I swapped the search engine for one built on embeddings and the problem was gone by the end of the afternoon, not after a week of hand-feeding it synonyms one by one.

An embedding is a list of numbers, a vector, usually with several hundred or a few thousand positions, that captures what a piece of text means rather than the letters it’s made of. “Spinning rod” and “fishing rod for spinning” end up as vectors that sit very close together even though they barely share any words, and that numerical closeness is what lets you search by meaning instead of by literal match.

To generate them I used OpenAI’s text-embedding-3-small: you send it text, it hands back a vector of 1,536 numbers, at around $0.02 per million tokens, so running the whole catalogue (about 4,000 products) through it cost pennies. I stored the vectors in Postgres using the pgvector extension, five lines of SQL for the index, and from there I could compare each search query’s vector against the products’ vectors and keep the closest ones. No new infrastructure, no bolting on an external service just for this.

What caught me off guard was how badly this explains itself to a client. I said “the system searches by meaning” and got a look like I was selling snake oil. What actually landed was screenshots, before and after: same search term, zero results with the old search, three relevant products with the new one. That took ten seconds to sink in; the theory wouldn’t have landed in ten minutes.

Embeddings are also the piece that lets a chatbot answer using a company’s own documents without the model ever being trained on them: you turn every chunk of a manual or a PDF into its own vector, and when someone asks a question, you pull the chunks whose vectors sit closest to that question and hand those to the model as context, as much as fits inside its context window. That’s the technical foundation of what most people call RAG, even though the term almost never comes up when I’m actually talking to a client about it.

What I do weigh carefully is which embedding model to use, because it’s not a one-size-fits-all choice: some are cheaper and faster, others cost more and pick up on finer distinctions, like unusual synonyms or the jargon of one specific industry. For the fishing catalogue the small model was more than enough; for a legal search tool with very specific terminology it probably wouldn’t be. I still don’t have a fixed rule for deciding that upfront. For now I just test both against a real sample of the client’s own searches before settling on one.

→ Direct link to this question

A couple of months back I asked a “regular” model to check the logic on a discount system where the rules kept stepping on each other (coupon, seasonal offer, and an already-reduced price, all three at once), and it handed back an answer that was confident, well-written, and wrong. I switched to the same provider’s reasoning mode, the exact same question, and it took fifteen seconds instead of two. This time it caught the odd edge case where two conditions cancelled each other out.

A reasoning model is an AI that, before writing the answer you actually see, works through an internal chain of intermediate steps, trying out paths and dropping the ones that don’t add up, instead of answering on the first try the way a regular chatbot does. That costs more time and more tokens, but it noticeably improves accuracy on tasks with several chained logical steps: code with crossed dependencies, contracts with clauses that contradict each other, that sort of thing.

And that’s exactly the catch: thinking harder isn’t free. With most providers you’re also paying for the internal reasoning tokens, the ones you never see on screen, so the same question can end up costing quite a bit more than with a regular model. For a customer-service question like “do you ship to Portugal?” that’s money down the drain: the standard model answers just as well in a fraction of the time. Which is exactly where understanding how tokens get counted actually helps you decide which model to run where, not just when to switch on reasoning.

We use it in a fairly surgical way.

For marketing copy, summarizing an email, or tagging a support ticket, a mid-tier standard model does the job, no question. For reviewing an architecture change before touching anything, chasing a bug that’s been fighting us for two days, or reading through a contract before a client sees it, we flip to reasoning mode without a second thought. The difference shows up most when the problem has a hidden trap: the regular model steps on it with total confidence, the reasoning one sometimes sees it coming and says so out loud before answering, something like “wait, this has a weird edge case.”

What I still haven’t figured out is when to hand that choice, fast model or model that thinks, over to the system itself instead of deciding it by hand every time. Some providers already offer an automatic router that picks for you based on the question, but the few times we’ve tried it in production it got it right less often than we’d have liked. So for now, we’re still the ones flipping the switch.

→ Direct link to this question

A few weeks ago a client messaged us at eleven at night, pretty rattled: he’d just gotten an alert from his AI provider saying usage had spiked in the last hour, something like two hundred dollars in forty minutes. Nobody at his company had touched anything. When we checked the project’s GitHub repo — public, because he’d pushed it himself without asking us, “to show a partner” — there it was, in a commit from three weeks earlier: the API key pasted straight into the code, no environment variables, nothing. Someone, or some scraping bot, had found it and was using it to generate who knows what on his dime.

An AI API key is basically a password that identifies your account to the model provider (OpenAI, Anthropic, Google, whoever) every time your app asks it for something. If that key ends up in someone else’s hands, they can use the model as if they were you, and the bill lands on you: the provider has no way of knowing it wasn’t you, unless you catch it fast enough to revoke the key.

What we did with this client was the usual playbook: revoke the key from the provider’s dashboard the moment we saw the alert, generate a new one, and this time put it in the server’s environment variables instead of code that gets pushed to a repo. That part took ten minutes. What took longer was convincing him that his partner didn’t need to see the actual source to understand what we’d built — a recorded demo would’ve done the job — and that a “private but with the key baked in” repo isn’t as private as it sounds, since any future collaborator, or a badly handled fork, inherits it just the same.

The bill, for what it’s worth, got refunded after we filed a fraud claim with the provider. That doesn’t always happen. It depends on the provider’s policy and how fast you report it, so it’s not something worth counting on.

Since then we’ve applied one rule on every project that we used to just assume everyone already knew, without actually checking: no key ever goes in the code, all of them live in an environment file that Git itself ignores, and production keys get stored separately, in the host’s secrets manager, never in a shared document or a WhatsApp thread about the project (we’ve seen that last one more than once). It’s the same lesson we took away from auditing our own server’s security: most scares don’t come from a sophisticated attack, they come from a key someone left lying around without noticing.

What I still haven’t figured out is how to explain this to a client without it sounding like a scolding, especially when the mistake came from someone with good intentions who just wanted to show off their project.

→ Direct link to this question

A client of ours who sells lamps and home decor online got in touch a few weeks ago. She wanted to automate part of the returns process: when a buyer sends a photo saying “it arrived broken,” she wanted the system to look at that image and decide whether to approve the return, instead of someone on her team opening every email and checking the picture by hand.

That’s exactly what multimodal AI does: it understands text and images (sometimes audio or video too) in the same conversation, without needing a separate system for each type of content. You send a photo along with a question, and it answers based on what it actually sees, the way you’d describe a picture to someone over the phone.

Until a couple of years ago, building something like this meant stitching pieces together: a vision model to classify the image, OCR if there was text in the photo, and then a separate language model to draft a response using whatever the other two handed it. Three systems, three places where something could break. With a multimodal model like Claude or GPT-4o, all of that collapses into a single call: the image sits in the same context window as the text, and the model reasons over both at once.

We had a working prototype for the client in an afternoon, literally. We fed it the photo plus a prompt with her return policy’s conditions, and it comes back with whether there’s visible damage, what kind, and a short note for the team. It handles obvious breakage well (a shattered piece of glass is easy to catch) and struggles more with subtler stuff: a small scratch on a matte finish gets flagged some of the time and missed the rest. We still keep a human checking before anything gets approved automatically. The AI cuts down the work of eyeballing every photo one by one. It doesn’t replace the final call.

What almost nobody asks upfront, and should, is what happens to those photos once they leave your server. They go to a third-party provider’s API, same as with any language model, so if a customer uploads something with visible personal details (an address on a label, a face in the background) it needs the same care as any other data you hand over to an AI. In this case we told the client upfront that return photos go through that process, and put it in writing in her policy, not as a footnote.

We haven’t tried feeding audio into the same flow yet, though the same client has already asked whether something similar could work for the voice notes people sometimes send explaining what happened. Technically, sure, it could. Whether it’s worth building for the volume of returns she gets is a question I don’t have a clean answer to.

→ Direct link to this question

A client wrote to me a few weeks ago, pretty annoyed: he’d asked ChatGPT to summarize the same document twice, with the exact same prompt, copied and pasted, and got two different summaries back. He thought the tool was broken. It isn’t. It works like that by design, and it took me a while to accept that too, the first time we built an integration with a model for a client and the results in testing just wouldn’t come out the same twice.

A normal program is deterministic: same input, same output, every time. A spreadsheet doesn’t have good days and bad days. A language model doesn’t work that way underneath. When it generates text, at every single step it computes a list of candidate words with a probability attached to each one, then picks one from among the most likely, not always the single most likely word, but one sampled at random from within that range. That’s controlled by a setting almost every API exposes called “temperature”: higher means more variety (and more risk of wandering off); lower, or set to zero on some models, means answers much closer to each other, though almost never a hundred percent identical. In one line: an AI gives different answers to the same question because it doesn’t always pick the single most likely word — it samples among several candidates based on that randomness setting, so some variation isn’t a bug, it’s part of the design.

Why build it this way, when it makes the tool feel unreliable? Because the same mechanism that makes two summaries come out different is what lets the model avoid sounding the same every time, lets it finish a sentence naturally instead of repeating the statistically flattest word combination over and over. Without that randomness a chatbot would sound like a machine, not like something imitating a real conversation.

The trouble shows up when someone, us included the first time, uses a model for a task that actually needs to be deterministic. We built an automation for a client that read invoices out of PDFs and pulled the amount, date and supplier using an LLM. It worked fine in testing. In production, every so often the exact same invoice format would come back with a field misread that had come out perfect the time before. It wasn’t a bug in our code, it was the model being a model. We dropped the temperature to the minimum and added a validation layer with fixed rules, the amount has to be a valid number, the date has to actually exist, and the problem got a lot better. It didn’t disappear entirely. With these tools, “almost always right” is not the same as “always right”, and you need to design knowing which one you actually need for each piece of the process.

For normal use, asking something, getting ideas, having it help draft an email, this variability doesn’t matter at all, it even helps. The problem is when someone builds an automated business process assuming an LLM behaves like a spreadsheet formula. It doesn’t. The sooner you have that straight while designing the automation, the fewer surprises with weird data you’ll run into six months after switching it on and walking away.

→ Direct link to this question

Last autumn a real estate client asked us for something that sounded simple: an assistant that could answer questions about their earnest-money contracts, just by uploading the PDF straight into the chat. We tried it first on a short contract, eight pages, and it worked great — questions about deadlines, the penalty clause, who keeps the deposit if the deal falls through. All fine.

Then we tried one of their genuinely long contracts, the kind dragging technical annexes and a whole history of amendments: forty-odd pages. The model started answering with total confidence about clauses that didn’t exist. Not slightly wrong — it was inventing entire article numbers.

That’s where the context window comes in, a concept almost no client has heard of until it blows up in their face. Every AI model has a limit on how much text it can “hold in mind” at once, and that limit is measured in tokens, not pages or words (a token is roughly three or four characters). A model with a 128,000-token window can handle, roughly, some 300 pages of normal text. Sounds generous. But a scanned PDF, with badly converted tables or embedded images, can end up costing two or three times the tokens it looks like it should, and that’s exactly where the margin quietly runs out.

And here’s the part that actually matters: when you go over the window, the model doesn’t throw a clean “this doesn’t fit” error. Depending on the tool, it truncates the document wherever it can, or silently summarizes chunks without telling you, or just fills the gaps with whatever seems statistically likely given the pattern of the text. That’s what happened to our client: the model never had the end of the contract in its active window, and instead of saying “I don’t know,” it improvised.

The fix wasn’t to grab a model with a bigger window (they exist, some with a million tokens, but they cost more and run slower). We went with RAG instead: rather than dumping the whole contract in one go, the system first pulls the fragments relevant to the actual question, and only those go into the context window. Less elegant than “upload the PDF and ask anything,” but a lot more reliable, and cheaper too, since you only pay for the tokens each query actually uses.

If someone tries to sell you a “no limits” AI assistant for long documents, ask exactly what happens when a document doesn’t fit whole. If the answer is vague, they probably haven’t tested it against a real forty-page case yet.

→ Direct link to this question

A few months back a client in the legal sector asked us for something very specific: they wanted an assistant to help their team search through thousands of pages of case files, but they flat-out refused to let any of those documents leave their server. Not to OpenAI, not to Anthropic, not to anyone. That’s exactly where the difference between “cloud” AI and self-hosted open-source AI comes in.

When you talk to ChatGPT or Claude, your questions travel to a server run by OpenAI or Anthropic, get processed there, and the answer comes back to you. Convenient, fast, and you never have to think about infrastructure. The catch is that for some businesses, sending sensitive information to a third party simply isn’t an option — whether because of a contract, industry regulation, or the client just not trusting it. In one line: running AI in the cloud means sending your data to a third-party server (OpenAI, Anthropic, Google) in exchange for convenience; running it on your own servers means that data never leaves your network, in exchange for building and paying for the infrastructure yourself.

That’s where open-source (or “open-weight”, the more precise term) models come in: Meta’s Llama, Mistral, Qwen. Models you can download and run on your own hardware, so not a single word of what you ask ever leaves your network. With tools like Ollama or vLLM, setting one up isn’t nearly as complicated as it sounds — a couple of days of work, not weeks.

What does cost real money is the hardware. For the law firm’s project we ended up renting a GPU machine from a cloud provider (buying your own graphics cards is something we almost always advise against), and even then the monthly bill ran about four times what using the Anthropic API would have cost for the same volume of queries. And honestly, the quality of the answers was worse: Llama does a decent job, but it’s not in the same league as Claude or GPT-4 on complex reasoning over long text.

So the real question isn’t “open source, yes or no”. It’s “which matters more to me, confidentiality or quality”. For most of our clients the answer is clear: using a serious provider’s API, with a solid data processing agreement in place, ends up cheaper, faster to set up, and gives better results. The law firm was one of the rare cases with no way around it: their own lawyers had confidentiality obligations that ruled out any middleman, however good the contract on paper.

There’s a middle ground people don’t talk about as much: some providers (Azure OpenAI, Amazon Bedrock) offer the same closed models but with contractual guarantees that your data isn’t used for training and isn’t shared with anyone else. For a good number of clients worried about privacy, that solves the problem without standing up your own servers or giving up quality. We’ve yet to explain this to someone without watching their shoulders drop a little.

What I honestly don’t know is what happens in two years, once open models close the quality gap with the closed ones. Maybe “self-hosted or not” stops being a question of sacrificing quality and turns into a purely operational cost calculation. Or maybe the big models just stay one step ahead forever. We’ll see.

→ Direct link to this question

A logistics client wrote to us a couple of months ago, pretty on edge: he’d read that a major AI provider had hiked prices for certain enterprise customers with no warning, and wanted to know if the same thing could happen to the assistant we’d built him for sorting delivery incidents. I told him the truth: yes, it could. We hadn’t talked about that risk when we signed the project. Neither had he. Neither had we.

Bringing an AI model into a business means taking on a new dependency, and a serious one. It’s not like depending on hosting, which you can swap out in an afternoon by copying files from one place to another. Once an assistant has been running for months on a specific model, the instructions you give it (the “prompt”) are tuned to that particular model’s quirks: how it reads dates, how much it rambles if you don’t set a limit, what examples it needs to stay on script. Switching providers isn’t swapping an API key. It’s re-testing half the assistant’s logic from scratch. In short: if your business depends on a specific AI provider and that provider raises its price, switches models, or drops support, migrating isn’t just swapping an API key — it’s reworking the prompt and re-testing a good chunk of the assistant’s logic.

It happened to us for real with an insurance client. The model we were using got deprecated in its old version, and we had three weeks to migrate the assistant to the new one, on a deadline set by the provider, not by us. The general logic still worked, but the tone changed: the new model was drier, gave shorter answers, and we had to rework the whole prompt to get it sounding like it used to. Three days of work nobody had budgeted for.

Since then I try to explain this on every new project that leans on AI in a serious way, not as a shop-window feature. Two things help a lot. One: don’t put business logic inside the prompt if you can avoid it. Hard rules (what price applies, what deadline has to be met) belong in ordinary code, and the model only handles the language part. That way, if the model changes tomorrow, what breaks is the tone, not the calculation. Two: write the code so that switching providers means swapping out one function, not rewriting the whole application. Nothing fancy, but few clients ask for it, because nobody’s told them they need to.

What I’m still not sure about is how much of this is worth building in from day one on a small project. That abstraction layer costs hours the client sees on the invoice, and for a simple assistant at a small business it might be paying for insurance you’ll never need to use. For the insurance client, looking back, it was worth it. For the delivery-incidents one, honestly, I still don’t know.

→ Direct link to this question