Building an AI Agent for a Proprietary Enterprise Platform

From Specs to Custom Dashboards

A Repeatable Process, Ready to Automate

The Problem

DashReady (name changed) helps organisations manage their data — covering collection, storage, workflows, and dashboards.

Every new client project requires a custom setup: a developer manually translating a requirements document into platform config code, then deploying it. The same process, repeated every time.

Why Existing AI Tools Failed

DashReady's platform is entirely proprietary, with no public docs or examples in AI training data. Off-the-shelf tools produce output that looks plausible but breaks in subtle, hard-to-detect ways.

The Goal

Build an AI agent that takes a spec as input, generates the config code, and deploys it to a live dashboard.

The Approach

Built iteratively:

• Started with prompt engineering using docs + examples, using held-out examples to test blind
• Scraped 86 internal docs (~500k tokens) after discovering incomplete public docs
• Replaced a bloated single prompt with on-demand retrieval tools
• Added a code validation layer to catch errors before deployment
• Reverse-engineered the upload API to automate deployment
• Wired everything into a single pipeline: generate → validate → fix → deploy

The Result

Deployed two projects end to end with no manual intervention. It showed that AI can work meaningfully within a proprietary platform.

That said, it's a proof of concept, not a finished product. It's best used today as a developer augmentation tool while the team reviews and refines each step.

What's Next

A structured evals dashboard for systematic quality improvement, and a simple chatbot interface so non-developers can use it directly.

Learning the Platform

Understanding the Manual Process

Before building anything, I needed to understand how a technical team member performs this task manually.

I asked the DashReady team to walk me through it as if explaining to an intern — covering what the inputs are, what the output code looks like, and how one is converted into the other. I also asked for a few example input/output pairs and any relevant documents.

They shared five example workflow files, some basic documentation, and a walkthrough video.

Getting to Grips with the Problem

Reading through the files alone wasn't enough to fully understand the process. I fed everything into an AI and asked detailed questions — but even that didn't bring complete clarity.

So I decided to learn by getting started with a simple use case.

First Attempt at Code Generation

For my first prompt, I loaded all the files in the prompt's context (around 30k tokens) and asked the model to generate a sample workflow.

The output looked reasonable — but I had no way to verify it. Without deep knowledge of the platform, I couldn't tell if it was correct or broken.

Using Examples as Ground Truth

I had 5 examples, each containing a known input specification paired with its correct output workflow. Since I couldn't verify correctness myself, I used these as my judge: feed the model an input, then compare its output against the known answer.

Since I did not have a complete understanding of the code, I used AI to cross-check the model's output with the known answer.

Progressive Testing

I tested with progressively more context in the prompt:

The Feedback Loop

Every run revealed new errors. Every error became a rule added to the prompt. By documenting failures immediately and converting them into explicit rules, I avoided repeating the same mistakes and steadily improved output quality. Even before running any code.

The held-out example proved to be the most valuable test. It forced a genuine blind evaluation — the answer wasn't in the prompt. Rotating which example was held out across all five runs surfaced a wider range of failure modes, making the final rule set more robust.

Key learning: A good prompt structure consists of Examples, Docs and Rules.

But there was a catch: all of this only told me the output code looked correct. It didn't tell me whether it actually worked.

Building a Test Environment

Getting Access to a Test Server

All of the previous experiments only told me the output code looked correct. It didn't tell me whether it actually worked.

I couldn't validate the generated code without actually running it — so I requested access to a test server to deploy and run the code.

Deploying the Code

To test the code, I had to manually package it into a zip file and upload it through the platform's UI. I uploaded the example file and it worked, but it was tedious and slow.

I looked for an API to automate the upload — but found nothing documented.

Reverse Engineering the Upload API

So I improvised. I opened browser DevTools, triggered an upload, and watched the network tab. I found the API call, grabbed the endpoint and session token, and replicated it in a Python script.

Now uploads were programmatic — a small but meaningful reduction in friction.

First Results (and New Problems)

The first few uploads returned a blank page. No error, no output.

That was still progress: I'd gone from unverifiable output to a real test environment. But debugging was the next challenge — even with logs enabled, the errors were cryptic.

Uncovering the Full Documentation

The Docs Were Incomplete

The errors and blank pages from the test server made one thing clear: the information I had wasn't enough.

The basic docs the DashReady team had shared described the platform in general terms — but specific field names, structural rules, and internal conventions were all missing.

Finding the Internal Docs

After some digging, I discovered that DashReady had a full internal documentation site, hidden behind a login. I requested access.

Once in, the gaps filled fast. Everything I had been guessing at was written down.

Scraping the Docs

Manually copying pages worked at first — but there were far too many. So I wrote a scraper to pull everything down programmatically.

With the new docs, I regenerated the output and uploaded it to the test server. The errors still didn't resolve.

A Hidden Problem: Missing Code Examples

I cross-checked a few downloaded files against the documentation site. The text had come through fine — but every code example was missing.

The scraper hadn't captured them because they were stored in a different format.

I rewrote the scraper to handle code blocks explicitly, pulled everything again, and verified the output carefully this time.

86 docs. ~500k tokens. Now I had the full context.

The Lesson

At every step, it's important to understand what's happening and verify the output carefully. A missed error early on costs a lot of time later.

Note: In some organisations, there could be restrictions on accessing proprietary documents and scraping everything, even if everything is being securely stored for the use by agent. In this case, I would build a function to securely access the required documentation when required. But it will still require retrieving the list of available documents along with brief details of what it contains.

From Monolithic Prompt to On-Demand Retrieval

The Problem with More Context

Stuffing everything into one prompt was a useful starting point — it helped establish whether the AI could generate DashReady code at all and build intuition. But with docs now spanning 86 files and 500k tokens, a single prompt was no longer practical.

The issues were threefold: the context exceeded what a single prompt can handle, repeated runs were slow and expensive, and too much information made it harder — for both humans and AI — to find what's actually relevant.

A Better Approach: On-Demand Retrieval

Instead of loading everything upfront, I gave the AI the ability to look up what it needs, when it needs it. I created three simple tools:

• list_docs — see what's available
• read_doc — open a specific document
• search_docs — find documents by keyword

But creating the functions was only half the job. The AI also needed a way to actually use them.

Function Calling and the Tool Loop

Function calling is what allows an AI to use tools — discrete pieces of code it can invoke on demand. Tools can retrieve information, interact with systems, or take actions.

This is what separates a model that generates text from an agent that accomplishes tasks.

Crucially, the AI can use tools in a loop: reason, call a function, inspect the result, and self-correct — as many times as needed.

The Result

The prompt shifted from being a library — holding everything — to being a librarian: knowing what exists and where to find it.

The impact was dramatic. Prompt size dropped from 120k tokens to 3k. A 98% reduction.

Catching Errors Before They Reach the Server

The Problem: Silent Failures

Early on, a mistake in the generated code produced a blank screen — no error message, no hint of what went wrong. Debugging meant manually checking the logs, browser console, comparing the AI's output against known examples, and cross-referencing the platform docs. It was slow and unreliable.

Why Rules Alone Weren't Enough

Once errors were identified, they were added as rules to the prompt — to prevent the same mistakes recurring. But two problems emerged:

• Some rules were wrong, based on incomplete understanding of the platform
• Others were correct, but the AI ignored them during generation

A DashReady developer reviewed the prompt and rules and corrected them. He offered one additional suggestion: rules are necessary, but not sufficient. A second layer was needed.

The Code Review Function

The solution was a Python script that checks the generated code before it ever reaches the server. Rather than waiting for a blank screen, it returns a specific list of errors — immediately after generation.

That error list is then fed back into the next generation step, alongside the previous output, allowing the AI to self-correct.

Two Layers, Working Together

Rules prevented known errors during generation. The review function caught what slipped through after generation.

Neither was sufficient alone. Together, they formed a self-correcting loop.

A System That Learns From Failures

Neither the rules nor the review function were built all at once. Both evolved over time:

• Each new failure produced a new rule in the prompt and
• Each repeated failure produced a new check in the review function

By the end of the initial iterations, nine rules existed — every one traceable to a specific failure, none guessed upfront.

A Step Towards Self-Correcting Pipeline

Previously, an error mid-process meant stopping entirely — requiring manual intervention and starting over manually. With the review function in place, the agent could handle errors automatically,

The Result

Two complete workflows were generated, validated, and deployed — with no manual intervention in between. The DashReady team reviewed both outputs and the feedback was clean. The proof of concept was complete.

The proof of concept established that the approach works. But some errors still required a developer to diagnose, and the output could only be fully validated by someone with deep platform knowledge.

How It Works Best Today: Augmentation not Automation

The agent handles the repetitive mechanical work:

• Hand-search documentation
• Manually write config files
• Zip and upload through the browser
• Debug a blank screen with no error message

The developer keeps it on track by handling the judgement calls: reviewing output, catching edge cases, and knowing when something is subtly wrong.

Iterative building and Pair Programming

The original framing was simple: take a spec, generate code, deploy it. In practice, specs are rarely complete. PRDs describe intent — field names, structural conventions, and edge cases don't live in a document. The AI fills the gaps with guesses, and large one-shot generations create a large surface area for hard-to-diagnose failures.

The more effective approach is iterative building:

"Add a checkbox as read-only." "Move this field to the review step."

Each prompt is small and targeted. The agent retrieves the current state, makes the specific change, re-uploads. The developer reviews, then drives the next step.

Each iteration is small and verifiable. Failures are contained.

What Comes Next

More automation — as error patterns become better understood and a structured evals framework matures, more steps can be handed off with confidence. The manual process doesn't disappear, but its scope narrows over time.

A simpler interface — the agent currently runs inside a coding environment, limiting who can use it. The next step is a plain chatbot: paste in a spec, get back a deployed workflow. No Python, no terminal — usable by anyone on the team or even by the client. The interface would also let the team add new docs and examples as the platform evolves, keeping the knowledge base current without touching the code.

Reflections

Figuring It Out By Building

Every time I hit a wall, I'd discover something new. Internal docs. Server access. Code examples. Each one came only after I realised I needed it.

The organisation wasn't being difficult. They handed things over when I asked. But they were cautious — internal docs, proprietary code, server access aren't things you share lightly with an outsider.

I was tempted to think: if I mapped everything out upfront, figured out what I needed to build, get access from day one, it would have saved everyone's time. But that assumes I already knew the shape of the problem. The key insight is navigating the unknowns quickly by building — not trying to eliminate them.

An NDA and an agreement builds trust to move fast — but both sides need to accept that requirements will emerge through building, not before it.

Retrieval Architecture Tradeoff

The retrieval system I build was deliberately simple: list documents, read a document, search by keyword. No vector databases, no embeddings, no complex pipelines.

But simple doesn't mean perfect. A few challenges surfaced quickly:

• Large documents were hitting rate limits when loaded whole.
• Truncation meant information in the latter half of long docs was silently dropped.
• Keyword search missed relevant content that was a bit far around

A natural fix would be generating summaries, metadata, and keyword indexes for each document up front — so retrieval can be better.

It wasn't a solved problem. But it was functional, and functional was enough to move ahead.

The Evaluation Bottleneck

Building the pipeline was the easier half. Knowing whether it worked — that was the harder problem.

Syntax errors were catchable. Uploads were verifiable. But whether the output actually matched what a client dashboard needed required domain knowledge I didn't have. Every real evaluation had to go through the DashReady team: generate, send it over, wait for feedback.

Without being able to evaluate the output myself, I was generating and guessing. Which points to a structural problem: I can't improve what I can't measure.

The evals process and dashboard gives the DashReady team a structured place to batch-test, annotate, and track quality over time. They have the domain knowledge to judge output accurately; the dashboard gives them a systematic way to apply it.

Evals process:

• Run a batch of queries through the pipeline
• Review the output
• Annotate what's correct, what's wrong, and why
• Use that feedback to refine the prompt, the functions, or the docs

But handing over a dashboard isn't enough. This is a two-sided commitment — the domain expert has to show up and use it. That's where the pipeline becomes reliably good. The path to fuller automation runs through iteratively improving the agent.

Anatomy of an AI Agent

The Building Blocks of an AI Agent

Every part of this system — looking up docs, generating code, checking for errors, deploying — is built from the same three ingredients:

• A prompt (instructions for the AI)
• A set of functions (tools it can use)
• A knowledge base (information it can look up)

When all four components — retrieval, generation, validation, deployment — were combined into a single pipeline, these three ingredients were merged into one unified system, with an added layer of orchestration in the prompt to coordinate the steps in the right order.

The Prompt

The prompt is where the agent gets its identity, its task, and its rules of engagement. It had six parts:

Role — A single line establishing expertise: You are an expert developer for the DashReady platform.

Task — What goes in, what comes out. The agent receives a structured workflow specification and is responsible for producing deployed, working code on the platform.

Process — Steps agent may follow to accomplish the task:

1. Search — Look up relevant docs and examples before writing anything
2. Generate — Produce and save the workflow files
3. Validate — Check output, fix errors, repeat until clean
4. Deploy — Upload to the server
5. Test — Confirm the workflow runs on a live instance

Knowledge map — An explicit index of what docs and examples exist and how to access them. Without this, the agent would guess — or worse, hallucinate — what was available. Telling it upfront what exists meant it could retrieve precisely what it needed.

Rules — Output format requirements, plus a growing list of known pitfalls. None were invented upfront. Every rule came from a real failure — caught in logs, traced back to the docs, turned into a constraint.

Function list — All available tools listed explicitly, so the agent always knew what it could call.

The Functions

Ten functions in total - each handling exactly one job:

• Read — list, read, and search docs and examples
• Save — write and edit generated files
• Validate — check output against the platform spec
• Upload — zip and deploy to the server
• Test — create test instances and confirm the workflow runs

The agent could decide the order dynamically and call them when required.

The Docs

The docs gave the agent its knowledge — retrieved on demand, not loaded upfront.

• 80+ platform documents, scraped and stored locally
• A library of real input/output workflow examples, used for both retrieval and cross-validation

This is what made approach practical. The agent didn't need everything at once — it needed the right thing at the right moment.

How Solveit Made This Possible

Building this agent required holding a lot in my head at once: platform docs, generated code, server logs, API responses, error messages, and a growing set of rules. Keeping that manageable — without losing the thread — turned out to be as important as the technical work itself.

Solveit handled that.

One place for everything

No switching between a notebook, a terminal, an API client, and a chat window. Emails, docs, code, logs, errors, and AI — all in one place. Every piece of the project lived in the same environment it was built in.

Same environment, start to finish

The same place used to understand the problem was the same place used to build and run the solution. Learning, building, debugging, and executing the final pipeline — no handoffs, no translation between tools.

A second brain throughout

AI wasn't just generating code. It was helping make sense of things: synthesising docs, interpreting cryptic errors, cross-checking outputs, refining the problem definition. That kind of ongoing sense-making was only possible because the AI had full context at every step.

The result: faster iteration, less friction, and a tighter loop between understanding and building.