Generative AI is easy to try and difficult to integrate. An employee can get a first answer in seconds, but an organisation must answer harder questions: which data may be used, who verifies the result, what the service really costs and what happens when it is wrong?
The right starting point is therefore not a model choice. It is a precise, frequent and measurable problem that can show whether AI creates an improvement. This guide provides a method for moving from an attractive demonstration to an operational service.
Start with a task, not a universal assistant
A request to "deploy AI across the business" provides no success criterion. Reduce it to a task: summarising a file before a meeting, preparing a support response, finding an internal procedure or producing a first documentation draft.
A strong initial use case usually has four properties:
- it happens often enough to justify integration work;
- a person already knows how to recognise a good answer;
- an error remains detectable and recoverable;
- the required data is identified.
Frequency is not enough. Automating a rare but highly risky task may require more control than the work it saves. A repetitive and reversible activity, by contrast, can be tested quickly without delegating a critical decision.
Before building a prototype, write a short brief: intended user, starting situation, data used, expected output, final decision and consequence of an error. This becomes the foundation of the evaluation.
Choose between a hosted service, an open model and internal development
A hosted service reduces time to first use. It provides the interface, infrastructure and updates, but its data terms, location, retention and export options still require review.
An open-weight model provides more control over hosting and the technical chain. It does not remove costs or dependencies. The organisation must operate inference, updates, observability, evaluations and security. Our analysis of open models and integration choices explains this transfer of responsibility.
Internal development makes sense when differentiated business logic must remain under control. It does not necessarily mean training a model. An internal architecture can orchestrate an external provider, document retrieval and validation rules while retaining ownership of the product.
Compare the complete cost: subscription or tokens, integration, supervision, tests, support, incidents and the ability to switch provider. The cheapest model per request is not always the cheapest system.
Protect data before the first real use
The first rule is to classify information before writing a policy. Public, internal, confidential, personal and trade-secret data cannot follow the same path.
For every use case, document:
- the data sent to the service;
- its retention period;
- whether it may be used to improve models;
- subprocessors and hosting regions;
- who can inspect inputs and outputs;
- deletion and incident procedures.
A technical filter is more reliable than instructions alone. Secrets, credentials, personal data and unauthorised documents should be removed before a model call. Logs must remain useful for diagnosis without recreating a sensitive database.
Agents introduce another risk because they can act. The French data protection authority highlights autonomy, persistent memory and interaction with multiple services. Permissions should therefore be minimal, temporary where possible, and separated between reading, proposing and executing. Our guide to AI agents in technical teams explains why human approval remains part of the system.
Measure quality with real cases
A positive impression during a demo is not an evaluation. Build a representative set from real work, then define what a good output must contain and what it must never do.
Metrics depend on the task:
- factual accuracy and required elements;
- percentage of answers requiring correction;
- actual time saved, including review;
- escalation rate to a person;
- critical errors and misplaced confidence;
- end-user satisfaction.
Measure quality by segment. An average can hide a system that performs well on simple requests and fails dangerously on rare cases. Keep incomplete, conflicting and boundary examples in the test set.
Our method for evaluating a chatbot before production provides a starting point. Rerun those tests when the model, prompt, document source or connected tool changes.
Calculate cost at process level
The price of a request is easy to measure; the cost of the process is not. Include inference, retrieval, tool calls, trace storage, evaluation and human review.
Track at least:
- cost per completed task, not only per message;
- context volume;
- number of attempts or chained calls;
- human review time;
- cost of errors and rework;
- abandoned-task rate.
A shorter, structured answer can be more economical than a cheaper model that requires several corrections. Set a ceiling per user, team or process so that an agent loop cannot turn a functional incident into a financial one.
Establish lightweight but real governance
Governance does not need to begin with a complex committee. It first needs visible responsibilities. For each system, name:
- a business owner responsible for the need;
- a technical owner responsible for operation;
- a person or function responsible for data and risk;
- a channel for reporting errors;
- a review date.
Maintain an AI use register, even a simple one. It should record the provider, use case, data, users, controls and decision to continue or stop. The NIST framework offers four useful functions for structuring this work: govern, map, measure and manage.
The EU AI Act introduces obligations that depend on the organisation's role and the system's risk. Legal classification should not be improvised from a generic list: context, purpose and affected people matter. A register at least makes it possible to identify which systems require review.
A four-stage deployment plan
1. Frame
Choose one task, a limited user group and three success measures. Classify the data and describe unacceptable errors.
2. Test
Build an evaluation set before heavily customising the system. Compare it with the current process, not an ideal demonstration.
3. Pilot
Open use within a controlled scope. Record costs, corrections, incidents and feedback. Give users a simple way to challenge or escalate an output.
4. Industrialise or stop
Automate controls, formalise support and prepare a fallback. If the gain disappears once review is included, stop or reduce the scope. A pilot that concludes "no" may prevent the most expensive investment.
The useful decision
A mature AI strategy does not try to place an assistant everywhere. It selects tasks where probabilistic output can be controlled, corrected and measured. Value then comes from integration into real work: authorised data, appropriate tools, clear responsibility and a path back to a person.
The first deliverable should not be a chatbot. It should be a use-case brief, a test set and an explicit data decision. Those three elements remain useful regardless of the model ultimately selected.
