--- title: "How I (Currently) Map Our AI Tool Stack for Dev, Research, Production and Non-Tech" type: post url: "https://robotpaper.ai/how-i-map-our-ai-tool-stack-for-dev-research-production-and-non-tech-in-2025/" canonical: "https://robotpaper.ai/how-i-map-our-ai-tool-stack-for-dev-research-production-and-non-tech-in-2025/" markdown_url: "https://robotpaper.ai/how-i-map-our-ai-tool-stack-for-dev-research-production-and-non-tech-in-2025.md" published: "2025-09-23T12:21:30.000+00:00" updated: "2026-10-22T14:32:29.000+00:00" tags: ["MLOps", "Tools"] author: "Roy Osherove" author_links: ["https://robotpaper.ai", "https://x.com/royosherove", "https://www.linkedin.com/in/osherove/", "https://forms.gle/jQSx5KoXvrVA1JCK7", "https://robotpaper.ai/about-roy.md"] description: "A bird's-eye map of the AI tools we use: chat, coding tools and agents, research studios, gateways, guardrails, prompt monitoring, agents, model serving." usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original post)." license: "https://creativecommons.org/licenses/by/4.0/" --- # How I (Currently) Map Our AI Tool Stack for Dev, Research, Production and Non-Tech TLDR; see main image of this post, and this is the structure: ```ascii AI Stack │ ├── Day-to-Day Chat │ ├── Development Centric │ ├── AI Coding Tools │ └── Background Coding Agents │ ├── Research Centric │ ├── AI Studio (Research) │ └── Model Training, Eval & Lifecycle │ ├── Production Facing │ ├── AI Gateways │ ├── AI Guardrails │ ├── Prompt Mgmt & Monitoring │ └── Agentic Frameworks │ └── Model Serving ├── Self-Hosted └── Managed ``` I like zooming out of things. And at some point I realized we don't have a good bird's eye view of all of the AI tools and frameworks we use every day at work. In fact, we didn't even have proper categories for how and where we use each tool. It's all a big mess and one excel file of expenses only gets you so far. Especially if you want to present this stuff to other people, explain why we need to focus on a specific category of tooling, or how our AI adoption is going in various "verticals" in the company, and for what purposes. This might change next month (probably will) but here are the current categories and sub categories we ended up using for now. I'm sure there are more you can think of so p[lease drop me a line](https://robotpaper.ai/about/) and tell me. ## Day to Day Chat AI Tools ![Day-to-day chat AI tools row: OpenAI Desktop, Claude Desktop and Gemini Desktop](https://robotpaper.ai/content/images/2025/09/image-51.png) These "client" tools are used both by techies and non techies at the company for various purposes. We usually use the "teams" version of things, and make sure data sharing for training is off of course since we need to be HIPAA compliant. (we have [signed BAAs like this one](https://help.openai.com/en/articles/8660679-how-can-i-get-a-business-associate-agreement-baa-with-openai-for-the-api-services) with our vendors) - Desktop AI Chat Client: [OpenAI ChatGPT Teams](https://openai.com/index/introducing-chatgpt-team/) - Desktop AI Chat Client: [Claude Teams](https://claude.ai/download) - Potential to check in the future: [Gemini Chat](https://gemini.google.com/) in browser or desktop ## Developer Centric AI Tools ![Developer-centric AI tools map: AI coding tools and background coding agents](https://robotpaper.ai/content/images/2025/09/image-52.png) These tools are used day-to-day by developers either directly (in the IDE or CLI) or indirectly (fire and forget background agents) or passively (background bug or review agents) ### AI Coding Tools for Developers Every developer gets to choose ONE of the following licenses on a monthly basis for day to day dev work: - CLI: [Claude Code](https://github.com/anthropics/claude-code) (via anthropic console) - CLI: [Gemini CLI](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstart) (via vertex studio) - IDE: [Cursor](https://www.cursor.so/) - IDE Addon: [Jetbrains AI Addon](https://www.jetbrains.com/ai/) (we have a lot of people, mainly research that don't want to leave the pycharm comfort for cursor. the AI addon is *acceptable* as far as I'm concerned, even though it doesn't offer the amount of features cursor has, but better something than nothing) - IDE Addon: [Co-Pilot](https://github.com/features/copilot) \- Since it supports most IDEs it's an option, though I feel it misses a lot of nice features. - TBD : [Windsurf](https://windsurf.com/download), [Cascade IDE/Addon](https://windsurf.com/cascade), [Cline](https://github.com/cline/cline), [AWS Q Developer](https://aws.amazon.com/q/) ### Change your mind (and license) every month Because things in the AI world are moving so fast, we're not setting up yearly licenses for these tools, but we renew each month. Monthly subs allow our developers and researchers to "change their mind" or for our recommendations to change on a monthly basis and we can cancel and switch across tools without paying for something people aren't using anymore. We're paying the premium so we can keep up with things. ### Developer Centric AI Background Agents I initially thought that ***fire-and-forget*** background agents will play a bigger role for me and others as a developer in our org, but I still like doing things in a more "eyes-on" approach. Sending off a BG agent to do some real bug fix or adding tests and then reviewing a huge PR with all the back and forth it will surely create, will take the same if not more time , and parallelizing this work for me or others might just create a bigger bottlenecks of PRs to review of AI slop that should have been prevented as it was created, not after the fact. That said, ***passive*** background agents such as Cursor BugBot have been *very useful*. yes they do have some false-positives but they find real bugs and it really helps to find real stuff early. - [Claude github Actions](https://docs.claude.com/en/docs/claude-code/github-actions): We have been experimenting with this and it seems pretty cool as a way to pay less for cursor bug bot (which is $40 per developer per month **on top** of the regular cursor licenses - sheesh!). It works really well, but if you don't have github enterprise, you need to install and maintain it per repo , so it's all about scale of team and repos) - [Cursor BugBot](https://cursor.com/docs/bugbot): A very professional product. Produces good results. Very expensive in my opinion, and needs more justification when you can install claude github actions and pay per token. Takes less messing around than github actions, and nice dashboards. - [CoPilot github agent](https://docs.github.com/en/copilot/concepts/coding-agent/coding-agent): Nice, but the results were not as good as the first two options here. Lots of noise and false positives, but the Developer experience is nice and simple. Not justifying the cost in my opinion. (this was tested 6 months ago, so things could have changed dramatically meanwhile. I should try it again!) - TBD: [OpenAI Codex Agent](https://openai.com/codex/), [CodeRabbit](https://www.coderabbit.ai/), [Google Jules](https://jules.google/), [AWS Q Developer](https://aws.amazon.com/q/developer/) ## Research Centric AI Tools ![Research-centric AI tools map: AI studio platforms and model training, evaluation and lineage tools](https://robotpaper.ai/content/images/2025/09/image-53.png) Our researchers use a combination of tools for day to day research and for managing the models and algorithms they end up creating and optimizing. ### AI Studio for Researchers These are all the various "console" and "playground" tools from Anthropic, OpenAI , Vertex, Bedrock and Vertex. Each one is being used for testing specific models with specific prompt versions. - AI Studios: [OpenAI Platform,](https://platform.openai.com/playground) [Anthropic Console](https://console.anthropic.com/) (now renamed to Claude Console), Google [Vertex AI Studio](https://console.cloud.google.com/vertex-ai/studio/freeform) (since we also use Gemini models), and [AWS Bedrock Playground](https://docs.aws.amazon.com/bedrock/latest/userguide/playgrounds.html) ### Model Training, Eval and Lifecycle tools for researchers We originally used manually managed EC2 GPU based instances with scripts to do most of the training, and since moved to using [ClearML](https://clear.ml/) for model eval and promotion, however, we are slowly integrating [Langfuse](https://langfuse.com/docs/prompt-management/overview) for prompt management and monitoring. We're still scared of putting it all in one place. ## Production Facing AI Stack ![Production-facing AI stack map: AI gateways, guardrails, prompt management, agentic frameworks and model serving](https://robotpaper.ai/content/images/2025/09/image-57.png) I keep changing (mostly adding) categories here, as things are slowly splitting into patterns. ### AI Gateways for Production LLM Use ![AI gateways for production LLM use: LiteLLM, OpenRouter, Portkey, Kong AI Gateway and others](https://robotpaper.ai/content/images/2025/09/image-62.png) AI gateways help centralize all the LLM related API calls into one central place in our infrastructure instead of having multiple services in our infrastructure manage their own API keys, quotas, rate limits, caching, security, model routing and more. They also help with centralized monitoring, applying AI guardrails across the entire infrastructure or product line, cost & budget management, and changing underlying models and infrastructure relatively simply without interrupting dependent services. AI gateways basically offer all the benefits of creating a single-responsibility micro-service that is internally managed instead of creating multiple sprawling dependencies and duplicate code across our product line and services. There are also some disadvantages to them, for example, a centralized AI gateway for an LLM must expose some centralized, generic API that will support all of the use cases clients need. If there are feature gaps between the that AI API exposes vs specific features that might exist only for specific LLM services, say, Gemini API, but not in OpenAI API, then you either must find a gateway that supports this new API, or have an API that reduces to the lowest common denominator but also ask your clients to use the 3rd party API directly, which defeats the purposes of using an AI gateway. Anyway, we're still exploring how to solve this problem in a maintainable and sustainable way, I'll write here about our efforts soon. - We're currently starting to use self-hosted [LiteLLM](https://www.litellm.ai/), which seems to offer great value and many features out of the box. The biggest contender for us is (maybe self-hosted) [PortKey](https://portkey.ai/) since it offers lots of extra enterprise-y features which would reduce the need for LangFuse , for example. - TBD: [PortKey](https://portkey.ai/), [OpenRouter](https://openrouter.ai/), [Kong AI Gateway](https://konghq.com/products/kong-ai-gateway), [llm-bridge](https://github.com/supermemoryai/llm-bridge) (seems a bit too early in its lifecycle?) ### AI Guardrails for production use ![AI guardrails tools for production: Guardrails AI, NeMo Guardrails and Bedrock Guardrails](https://robotpaper.ai/content/images/2025/09/image-61.png) We're still early in our journey for AI guardrails, so we have some custom internal code and tools in our services today, but all created in-house. Still early to tell, but [GuardRails.AI](https://www.guardrailsai.com/) seems to be the strongest contender for us to move to right now and we'll be starting integration soon as a POC. - TBD: [NeMo Guardrails](https://github.com/NVIDIA-NeMo/Guardrails), [Coralogix Guardrails](https://coralogix.com/platform/ai-guardrails/), others? ### Prompt Management & Monitoring ![Prompt management and monitoring tools: Langfuse, LangSmith and Portkey](https://robotpaper.ai/content/images/2025/09/image-60.png) We started, like others with hardcoded prompts, later configurable via JSON, later database oriented, and eventually started looking at [LangFuse](https://langfuse.com/) for prompt management. Currently in the early phases of testing it out in our non prod environments. Langfuse also provides powerful monitoring and observability for our prompts which is pretty nice. Allowing changing or comparing the results of different prompts is pretty powerful, as well as separating the text itself from where it is used allows for better maintenance over time (single responsibility principle). - TBD: [LangSmith](https://www.langchain.com/langsmith), [Portkey](https://portkey.ai/) (also provides prompt features) ### Agentic Frameworks for AI Production use ![Agentic frameworks for production: CrewAI, Temporal, LangChain agents, AWS AgentCore, Pydantic agents, Strands and more](https://robotpaper.ai/content/images/2025/09/image-59.png) Other than using [pure LLM relatively-deterministic](https://robotpaper.ai/zero-touch-algorithm-handoffs-how-we-ship-ml-algorithms-to-prod-without-rewrites-or-ml-engineers/) workflows, we're experimenting with both creating our own internal agentic loop for various workflows using [Temporal](https://temporal.io/) workflows with Claude Code, as well as using tools like [CrewAI](https://www.crewai.com/). - TBD: [Langchain Agents,](https://github.com/langchain-ai/langchain) [AWS AgentCore](https://aws.amazon.com/bedrock/agentcore/), [Pydantic Agents](https://github.com/pydantic/pydantic-ai), [AWS Strands SDK](https://github.com/strands-agents/sdk-python), [Vercel AI SDK](https://github.com/vercel/ai), [Microsoft AutoGen SDK](https://github.com/microsoft/autogen) ### Model Serving We serve both LLM based and NLP based ML models, with a combination of both self-hosted and managed services. ![Model serving map comparing self-hosted options (EC2, SageMaker) with managed APIs like Bedrock, Vertex AI and OpenAI](https://robotpaper.ai/content/images/2025/09/image-58.png) #### Self Hosted ML Model Service We used to self host directly on EC2, then moved to AWS ECS (we're not K8s centric yet), and slowly moving into hosting any custom or 3rd party models on [AWS SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/realtime-endpoints.html) endpoints. We manage this deployment and syncing via ClearML and CI jobs. #### Managed ML Model Service We use a host of 3rd party managed services for other models, namely LLMs like Claude GPT and Gemini, but also use a combination of Bedrock hosted models from the same companies for the purposes of multi-region compliance such as EU regions with HIPAA and GDPR [compliance](https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html). - AWS Bedrock: To host compliant region specific models - Google Vertex AI : for Gemini models (both API and endpoints in EU etc. - Anthropic: Directly console access for research and APIs when we need the latest feature, or for routing balance - OpenAI: Same as Claude - [Groq](https://console.groq.com/docs/models): for optimized access to some specific models - [Oracle OCI](https://www.oracle.com/artificial-intelligence/): For hosting models for training and evals before getting into prod. ## Summary This is a rather simplistic and always changing map of what categories we use and some of the services we use. It keeps changing and I'd love to hear how you map your own stuff, if you have any other categories I should include. [Send me a note](https://robotpaper.ai/about/). --- ## About the author **Roy Osherove** - Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery. ### Sites - [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership. - [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts. - [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin. - [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model. - [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps. - [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes). ### Social - X: [https://x.com/royosherove](https://x.com/royosherove) - LinkedIn: [https://www.linkedin.com/in/osherove/](https://www.linkedin.com/in/osherove/) ### Contact - Contact form: [https://forms.gle/jQSx5KoXvrVA1JCK7](https://forms.gle/jQSx5KoXvrVA1JCK7) ### License Content on robotpaper.ai is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): reuse, quote and adapt with attribution (Roy Osherove, link to the original post). More about Roy and his other sites and projects: [https://robotpaper.ai/about-roy.md](https://robotpaper.ai/about-roy.md)