This is the last part in my series of “learnings from building an AI coding agent”. Here are the previous two posts (in case you have read them yet) :
- Part 1 : Building LLM Powered Coding Agent as a PM : Key Learnings (1/3)
- Part 2 : How I built an “AI Evals” system for my product (a Coding Agent) - (2/3)
Table of Contents
Open Table of Contents
Building “AI-Assisted App Migration” feature workflow with AI-First principles
What is “AI-Assisted App Migration” feature
It was a feature we built in the AI coding agent at Freshworks that would take the outdated code of an app code base and migrate it “magically” to the latest supported developer platform version. It was important for our marketplace customers and business to ensure developers adopt latest platform versions which was not only more secure, making the apps secure, but also was feature rich in building diverse types of apps use-cases.
Here is how the feature worked
Of-course the risk was high! We were changing 10s of files in an existing working app codebase which developers had written and understood very well! As a developer myself, I would hardly trust an AI (which is non-deterministic duh!) to make changes to my codebase since I am responsible for the code and a working app!
How I baked trust in the workflow and built some principles along the way..
Principle 1 : Radical Transparency & Communication
Does the user know what’s happening at all times? Do they understand what the AI is doing and how it’s arriving at its response?
We see this principle everywhere, driverless cars showing the how the car is navigating the environment with rich designed interface, or LLMs showing their reasoning traces on what they are doing, how they are planning and arriving at the answer!
When users understand what’s happening, why it’s happening, and how the AI system is achieving their goals, you build extraordinary trust. This transparency acts as a trust safety net : even when AI doesn’t deliver the expected output, users are willing to iterate and try again, adjusting what’s within their control (prompts, seeking help, contacting support, etc.).
Real-World Example: AI-Assisted App Migration
Few steps we took to bake trust into the feature workflow :
-
Pushing real-time status messages in the chat vs generic loading spinner on what is happening to ensure developer is always aware of what is happening”. Migration starting… analyzing files… migration in progress… migration completed.” This isn’t just status updates, it’s transparency in action.
-
Explicit permission for every change : Ensuring “no change” gets committed without developer reviews them. Our messaging actively nudged users with “Let’s review the changes,” providing clear assurance that no code would be modified without explicit permission
-
Honest communication about AI limitations We encouraged users to thoroughly test their apps after accepting changes, explicitly reminding them that AI can make mistakes and that they must verify everything works before submitting to app marketplaces.
Principle 2 : Human always in control
Human always in control : They can start, stop, undo, and do anything to shape the AI behaviour at run time. Basically fully in control at all times.
Users trust “systems” when they feel safe in applying AI and feel in control that they can apply, revert go back or go forward as they like.
One of my first interactions with an early user was a wake-up call. She said, “I’m not sure if I want to apply these code changes, what if they’re wrong and you replace my old working code!”
That was a light bulb moment for me on how much risk I am putting the user into! Basically taking away their working code and replacing it with the non-deterministic AI generated code!
Few steps we took to make user feel in control :
- Comprehensive review system : Presenting changes as a change-log or Git PR, giving users complete control to review and modify generated code before accepting any changes
- Granular selection control : They can select all files, or any number of files they want to merge! Basically, whatever felt right for their situation
- Migration report : Most importantly, every migration generated a “migration-report.html” showing all the file changes in the diff format. That meant the changelog (their old code and proposed changes) were always available on demand in case anything goes wrong!
Note : A revert and checkpoint system would have been perfect but that takes time to build and hence we chose the shortest path!
Principle 3 : Design for high latency if you have to…
From the beginning, we faced an important decision: should we build synchronous or asynchronous workflows? Based on evals, Large apps could take 10-15 minutes for migration, with an average processing time of 5 minutes. Rather than forcing users to sit and wait, we chose asynchronous workflows from day one.
Our Approach: Start the migration, then go do your work. We’ll notify you when the migration is complete so you can return and resume where you left off.
Impact - Developer Love ❤️❤️
- Developers were praising productivity gains in development and the ability to migrate apps to new platform versions seamlessly with AI assistance
- “Positive feedback rate” almost doubled from 30% to 54% even while feedback volume jumped 87% > - indicating both higher satisfaction and broader adoption
- Continually improve “Response acceptance rate” - a critical milestone showing developers trust and actively adopt AI-generated suggestions
- Developer engagement accelerated 32.8% QoQ - reflecting sustained and growing usage patterns
Again, this was far from perfect and what I was satisfied with as a good experience for my users! But was a great learning process (rather intro) for building AI-first workflows!