10 Practical Learnings from Building Domain-Specific AI Solutions

Creating an impressive AI demo is easy these days. Getting a real AI feature to run reliably in production is an entirely different challenge. At Open Community Experience 2026 (OCX26) in Brussels, Philip Langer and Stefan Dirix shared the lessons we have collected over several years of building domain-specific AI systems for production: from voice interaction agents that handle real customer calls, to AI integrations inside specialised IDEs and modelling tools, to the Theia AI platform itself.

The full recording from the Open Community for AI track is now available:

What “Domain-Specific” Actually Means

This talk is not about generic chat products like ChatGPT. Domain-specific AI is the layer around an LLM that is unique to your product: access to your data, actions inside your tool, and integrations with your existing systems. The lessons below follow the three phases where these systems are built and broken: design, development, and production.

Design: Vision, Use Cases, and User Workflow

1. Define an AI vision before you pick features

A free-form chat is easy to ship but challenging to get right due to its open-endedness. Restricted UIs are more constrainted, but easier in production: a terminal agent that can only produce commands, AI-powered code completion that users barely notice, or an API layer that lets external AI tools interact with your product. If you are unsure where to start, a chat interface is the right place to start for exploration; the infrastructure carries over to whatever you build next.

2. Pick use cases that only you can build

The interesting use cases are the ones a generic LLM cannot do without you: access to proprietary data, actions and unique features inside your tool, integrations into existing workflows. Anything else is something any competitor with an API key could ship as well. A practical technique we use in customer workshops: build draft agents with a tailored system prompt and a few mocked tools in Theia AI’s tool sketchpad in interactive brainstorming sessions. You can explore use cases live before writing any production code.

3. Design the user workflow around correctness criteria

For every output the agent produces, ask how you would tell whether it is correct. If correctness is objective, expose a tool the LLM can call to validate itself. If correctness is subjective, design the human-in-the-loop step deliberately. If human rework is cheap, the human review can be handled with simple approval interfaces. If human rework tends to get expensive, introducing a proper plan artifact the human user can correct before the AI is creating the bulk of the work simplifies the review process, as it may happen on a higher level. If the review is very expensive, provide a well-designed review interface with interactive annotations yielding a rich reject signal to the AI, so the user does not have to re-review the whole result in detail again and again. Good steering and review tooling is part of the value your product adds: it is what makes the human efficient.

4. Build a dedicated API for the LLM

A recurring trap is reusing the same API your UI was built against as the API for the LLM. JSON schemas, validation rules, visibility logic, all dumped into the prompt. With strong models, this works for a simple demo. However, consistency collapses fast: the LLM forgets to check visibility, picks wrong paths through the schema, or misreads validation rules.

The fix is to pre-process raw data into an optimized presentation model the LLM can read and easily understand, expose typed and focused tools instead of generic ones, and make sure tool results do not flood the context window. The same idea works in the other direction too: tools that let the LLM act through your UI, like highlighting a node in a diagram, are capabilities that helps users a lot and that no external agent provider can replicate.

Development: Evaluation, Orchestration, Context

5. Evaluate continuously, do not tune in the dark

Every team building on LLMs eventually faces the same anti-pattern: someone tweaks the prompt because “it felt better”, commits it, and breaks another use case next week. Two complementary strategies prevent this:

• Deterministic oracles for tool calls and structured output. If a use case requires a specific tool to be called, write a unit test for it. Collect usage statistics from real runs to derive a baseline test set.
• LLM-as-judge for free-form text, scoring outputs on clarity, hallucination, and other dimensions. Frameworks like Promptfoo make this straightforward.

With evaluations in place, trying a new model becomes a real experiment with a proper basis for taking a decision too, instead of just guessing it works better than the old one.

6. Know when to switch from code orchestration to an agentic flow

For a fixed set of workflows, a code-orchestrated chain of LLM calls is the easiest starting point. You control the structure, isolate non-determinism, and use structured outputs for clean handoffs.

The trap is sticking with code orchestration too long. As you add classification, routing, and branching to support more and more use cases with your code-orchestrated flows, the orchestration code grows faster than the agent improves. In our experience, switching earlier to an agentic flow often pays off, even though the trade-off is real: you lose determinism and gain new context-management problems.

7. Treat context management as a first-class design problem

The moment you let an LLM drive the workflow, context management becomes your biggest challenge. Every tool you expose changes the trajectory of the agent, whether the tool is needed in this turn or not. There is no single solution, and best practices shift every few months. Plan to mix several techniques and to redesign the mix over time:

• Dynamic system messages that change as the task progresses.
• Adding or removing tools depending on the current state.
• Shared files as handover artifacts between turns or agents.
• Deferred or lazy tool loading.
• Sandboxed code execution where the LLM writes code instead of chaining tool calls.

8. Use multi-agent setups for context management

Multi-agent setups are useful, mostly as a tool for managing context. Splitting agents along human job titles (architect, developer, tester, reviewer), even though intuitive, however, rarely gives better results. Authority and responsibility, the things that make a human team work, are irrelevant to agents.

The actual reason to use multiple agents is context management. A sub-agent runs with its own system prompt, its own tools, and its own conversation, and the parent agent only sees the distilled result.

A pattern that works well: start sequentially, with files as the handover artifact. Files are easy to inspect, edit, persist, and reuse to start a new flow. In the Theia IDE, for example, an architect agent delegates code exploration to a dedicated explorer agent, then hands a clean implementation plan to a coder agent. Each agent works on a focused context, and the human can review the plan before code is written.

Production: Where Most of the Work Actually Is

9. Optimise for precision under composition

The old 80/20 rule turns into something closer to 96/4 with AI, and there is no skipping the remaining 4 percent. The hardest part is precision under composition. Five LLM-driven steps at 90% success rate each give you a combined success rate of around 60%. That means, in 40% of cases, the agent often creates more work than it saves.

A concrete example from our voice interaction agents: OpenAI’s real-time audio API is great in demos because it preserves emotion and lowers latency. In production it consumes so much context on audio nuance that core task precision drops. For an agent whose job is to book appointments or close tickets, the trade-off was not worth it.

10. Plan for monitoring, providers, and operations from day one

Three operational realities catch teams often unprepared when going live:

• Providers are not drop-in replacements. Even between supposedly OpenAI-compatible APIs, behaviour differs in subtle ways: tool-calling quirks, caching semantics, output formatting. Outright failures are easy to spot. Silent drift is the bigger risk, for example a switch that disables prompt caching and quietly multiplies your bill. Self-hosting is another step up in complexity and usually even needs a dedicated team to run this infrastructure.
• Instrumentation is mandatory. Shipping the system and waiting for bug reports does not work for AI. You need a path from “user reports something weird” to “concrete test case in the eval set”. For our voice agents, we built a monitoring tool that stores transcripts, auto-classifies calls by topic and outcome, and lets domain experts jump from a failed call directly into the LLM context that produced it, with one click to extract the scenario into the evaluation suite.
• Security and cost are real concerns. API keys for third-party LLMs in production are effectively unbounded credit; leaking one can be expensive. Misuse happens: malicious users will try to repurpose your LLM tokens for their own goals. For tool and IDE vendors, piggybacking on existing user subscriptions (Copilot, ChatGPT, and similar) can avoid the work to manage rate limits, billing, and a lot of operational overhead yourself.

Established platforms with monitoring, observability, and infrastructure built in help here, which is also why we extracted our voice interaction stack into a reusable platform.

Conclusion

The right choices depend on your domain and your constraints. There is no universal architecture or model choice, and no shortcut around evaluation, monitoring, and context management. Moving from prototype to production is where experience counts and where most of the real engineering happens, and it is worth planning for that from the start.

Want to Build Something Like This?

If your team is working on domain-specific AI, either inside a tool or as a standalone system, we’d love to help:

👉 Domain-Specific AI Solutions: design and implementation of AI agents deeply integrated into business applications and workflows.

👉 AI Interaction Agents: voice and chat agents that complete real tasks against your existing systems.

👉 AI-Native Tool and IDE Development: custom AI agents, capabilities, and workflows on top of Theia AI.

👉 Contact us to discuss your use case.

💼 Follow us: EclipseSource on LinkedIn

🎥 Subscribe to our YouTube channel: EclipseSource on YouTube