How I'm building my own Jarvis without writing a single line of code

It started with a conversation.

An acquaintance, someone who works in the same field, was talking about knowledge management: the feeling that you are constantly picking things up, reading and hearing them, and that none of it lands anywhere. He had thought about it a lot more than I had. He wasn't just building an app but looking at broader knowledge applications, and he already had a working set-up with Obsidian as the front end for his captured knowledge. A more traditional approach, but a considered one. I don't know exactly what he has built, and his approach deserves more than a passing remark.

I recognised the frustration at once. For years I wrote everything down in a notebook. That wasn't because I didn't organise, because I did, only in a medium where you can't search and where you can't make connections unless you reread everything yourself. You can't make more of a notebook. All the insights, the articles you've read and the podcasts that shaped your thinking pile up without connecting to each other. They are dead weight once the notebook is full and disappears into a drawer.

The digital alternatives I had tried didn't appeal to me. With Obsidian, Notion and an iPad with Apple Pencil I kept having to adapt my way of working to the tool instead of the other way round.

That conversation brought me closer to the problem but didn't give me a direction. That came afterwards, from a YouTube video about the concept of a "second brain". It didn't introduce a new idea, since I already knew the problem, but it gave me another way of looking at it and tackling it.

The key choice was to use Claude itself as the system, and not to build a traditional app.

What I built and what it became

What I have built since differs from what I had in mind beforehand, and it has become richer.

The foundation is a SQLite database, .claude/data/brain.db. I am not a database designer, but SQLite is the only structure that is local, private, searchable and extensible all at once. I don't write Python myself and I don't write SQL myself. With Bert, my technical infrastructure manager and one of the AI agents in the team, I was still able to design the architecture and have it built. There are tables for documents, tags and relations between documents, and a full-text search index based on SQLite FTS5. On top of that there is a semantic search layer: Bert integrated Ollama with the nomic-embed-text model, which generates an embedding for every knowledge atom, a mathematical representation of its meaning. Those embeddings are stored through sqlite-vec, an extension for vector similarity search. Search now works through hybrid retrieval: FTS5 for exact keywords, embeddings for semantic proximity, merged through RRF fusion. In concrete terms, 87 of the 135 atoms already have embeddings. You can now search on what something means and not only on what it literally says. From the start, the schema was set up for connections and not just for storage.

There is also a Python MCP server (brain_mcp_server.py), built by Sven, which connects Claude Code to the database. That layer is what makes the system coherent: every agent can read and write the database through the same tools, with the same structure. On top of that, a Python script (hn_rss_watcher.py) runs daily through launchd, the macOS task scheduler, fetching RSS feeds from Hacker News, de Volkskrant, NRC and the New York Times. Finn, the audio/video specialist, can process podcasts and YouTube videos through yt-dlp and Whisper transcription.

So the system does consist of code. I just don't write it myself.

A team of AI agents

In parallel with the technical infrastructure I built a team of AI agents, each with a name, a persona, an area of expertise and an agent definition as a markdown file in .claude/agents/.

The team started on 11 April 2026 with four specialists. Kenny became team lead; his only role is to delegate and he never carries out substantive tasks himself. Bert took infrastructure, Lara the document processing and Iris the knowledge strategy layer. Then came Daan (academic papers), Noor (business documents), Sven (the MCP integration layer), Henny (HR, creating new team members) and Max (research when expertise is missing).

On 15 April Vera and Finn joined: Vera as strategic adviser, Finn for audio and video ingestion. Today, 16 April, a ghostwriter joined to tell this story.

Not every team member runs on the same model. Vera works on Opus, the heaviest Claude model, for strategic decisions where depth of reasoning and nuance count. Most of the other specialists run on Sonnet, and simple tasks (exploration, HR, housekeeping) on Haiku. That is deliberate model routing, because running everything through the most expensive model makes a system like this unaffordable at scale.

In five days, eleven team members were added. Each exists as a fully worked-out profile: name, persona, area of expertise, guardrails for what they do and don't do, and instructions on how they communicate. At the start of every session Iris reads the open tasks from the backlog. She also curates a newsletter every day. The first run was today, 16 April 2026: 8 atoms stored, 16 items logged in the actuality log, a full curation report and a newsletter of fewer than a thousand words.

What went wrong

It hasn't all gone smoothly, and I want to share that.

Agent consistency is harder than it looks. You record behaviour in agent profiles, in memory files and in the CLAUDE.md configuration, and you update something after a session. In the next session it turns out that an agent still does something that was not intended: carrying something out itself when it should have delegated, creating a relation in the database that contains process metadata instead of substantive connections, or writing a document nobody asked for. Then you correct it, record it again and test it. Even so, you have to go back regularly to evaluate whether the model still holds.

What I have learned is that recording behaviour is an ongoing process. Every time you discover something new about how agents behave, you have to decide whether it deserves a structural change, in the profile, in the CLAUDE.md or in the backlog. Kenny keeps a persistent task register in .claude/backlog.md. That file survives session changes, context compaction and crashes, and is the only source of truth for what is still open.

The system itself is also growing faster than I had foreseen, and that isn't always comfortable. We started with written notes, then came podcasts and videos, and then the daily newsletter based on RSS. Every step opened the next door. I find it hard to see the end of it, but that isn't a requirement either.

Where it stands now

Today the database holds 135 knowledge atoms, spread over five clusters: AI & LLM, Knowledge Management & PKM, Work & Consultancy, Architecture & Technology, Finance & Investing. Besides individual notes there are also syntheses: Daan writes integrative atoms about clusters of three or more related documents. Iris maintains a schema document that describes the structure of the knowledge base itself.

The system now knows who I am, what I read and how I think, and that forms the foundation.

The way of working, "building without writing code", is a wonderful concept to me, with that caveat. I don't have the feeling that I am handing work off. Rather the opposite: I can focus on the things that matter. Architecture choices, priorities and direction stay with me. The technical execution is a means and not an end.

That feels different from what I expected. I thought "having AI build it" would feel like dependence. It feels more like working with specialists who are better than I am in their own domain, but who need my direction to make something worthwhile.

What I actually want

This is not yet the end point. I am thinking of Jarvis, Tony Stark's digital assistant, which is not a robot but a voice. An assistant that knows what is going on on several fronts at once (your work, your finances, what you read yesterday, last week's conversation), that doesn't wait for you to ask a question but thinks along when you miss something for a moment, and that has context and also uses it.

The acquaintance with whom this began built an app, and I chose a different direction. Whether the direction I chose is the better one, I don't know. The outcome, a system that thinks about me, that makes connections I wouldn't make myself and that operates daily without me having to be actively involved, is richer than what I had in mind after that first conversation.

Every discovery came in several steps.