Developer builds local AI agent from scratch, learns hard lessons
Quentin Merle spent a month building Vibrisse Agent — a local-first autonomous AI agent in Python/LangGraph — and found that "vibe coding" collapses under the weight of real architectural complexity.
Score breakdown
Practitioners building agentic systems should note that context discipline and software engineering fundamentals — not model selection — are what prevent complex AI-assisted projects from collapsing into unmaintainable code.
- 01Vibrisse Agent is built in Python with LangGraph and uses a local-first hybrid architecture running via Ollama or vLLM.
- 02The agent integrates MCP tools including GitHub, SQLite, Context7, and Fetch for real-world tool use.
- 03Multimodal vision is implemented using Gemma 4 to analyze UI live.
Quentin Merle set out to build Vibrisse Agent — not to automate his job, but to understand how AI agents actually work under the hood. The result is a local-first autonomous agent built in Python with LangGraph, designed to run primarily on-device via Ollama or vLLM, with the ability to dynamically delegate tasks to cloud providers like Groq or OpenRouter based on complexity. The agent's feature set was deliberately ambitious: MCP (Model Context Protocol) integration connecting it to GitHub, SQLite, Context7, and Fetch; multimodal vision using Gemma 4; a dynamic prompting system with an onboarding wizard; and Ghost Mode, which lets users direct the agent from inline source code comments (e.g., `// @vibrisse: refactor this loop`) without switching windows.
The most painful problem was what he calls "Lost in the Middle": without disciplined session management, the model would forget architectural decisions made hours earlier, causing the codebase to drift and break.
The central lesson Merle draws is that "vibe coding" — writing a prompt and letting the AI generate a full application — works for simple projects but fails for systems requiring strict context management and state persistence. The most painful problem was what he calls "Lost in the Middle": without disciplined session management, the model would forget architectural decisions made hours earlier, causing the codebase to drift and break. His fix was purely a matter of software engineering discipline — maintaining strict `ROADMAP.md` session files, explicit architectural notes, and regular refactoring phases to enforce separation of concerns. He also found that LLMs excel at writing isolated functions but are poor at designing and maintaining global architecture, and that without his 15 years of web development experience, the project would have become an unmaintainable monolith. The post concludes that productive AI collaboration requires deep experience — not to replace AI, but to know when to challenge it.
Key facts
- 01Vibrisse Agent is built in Python with LangGraph and uses a local-first hybrid architecture running via Ollama or vLLM.
- 02The agent integrates MCP tools including GitHub, SQLite, Context7, and Fetch for real-world tool use.
- 03Multimodal vision is implemented using Gemma 4 to analyze UI live.
- 04Ghost Mode lets users direct the agent via inline source code comments (e.g., `// @vibrisse: refactor this loop`).
- 05The biggest technical challenge was context loss — the 'Lost in the Middle' problem — solved through disciplined session files and a ROADMAP.md, not a better model.
- 06Merle argues LLMs are good at isolated functions but poor at global architecture design and maintenance.
- 07He cites a Stack Overflow stat that around 76% of developers now use AI.
Topics
Summary and scoring are generated automatically from the original article. We always link back to the publisher and never republish images or paywalled content. Last processed Jun 8, 2026 · 15:36 UTC. How this works →