Software Architect - RAG Based Agentic AI Platform

Introduction

At IBM Software, we transform client challenges into solutions. Building the world’s leading AI-powered, cloud-native products that shape the future of business and society. Our legacy of innovation creates endless opportunities for IBMers to learn, grow, and make an impact on a global scale.

Working in Software means joining a team fueled by curiosity and collaboration. You’ll work with diverse technologies, partners, and industries to design, develop, and deliver solutions that power digital transformation. With a culture that values innovation, growth, and continuous learning, IBM Software places you at the heart of IBM’s product and technology landscape.

Here, you’ll have the tools and opportunities to advance your career while creating software that changes the world.

Your Role And Responsibilities

Navi is IBM/HashiCorp's agentic AI platform, spanning multiple product pillars (Assistant, Insight, GTM, with a fourth in development) and handling tens of thousands of queries per month for field, customer-facing, and go-to-market teams. We're looking for a Band 10 AI Architect to serve as the end-to-end technical co-owner of Navi's architecture — someone equally comfortable designing an intuitive user experience, architecting agentic reasoning flows, engineering the underlying data pipelines, and making platform-level infrastructure decisions.

This is a full-stack AI architecture role: you won't hand off between specialists — you'll be expected to move fluently across every layer of the stack, from UI/UX through to deployment infrastructure, and to write production code yourself where needed.

Product ownership across Navi is currently divided among a small group of architects, each with deep context in different areas of the platform. This role requires someone who can operate independently while also actively collaborating with those architects — brainstorming design options, reconciling different perspectives, and converging on the best overall solution rather than optimizing for their own area in isolation.

What You'll Be Working On

UI/UX Layer

  • Architect intuitive, trustworthy interfaces for surfacing AI-generated answers, confidence signals, and source citations to non-technical business users
  • Design interaction patterns that make agentic reasoning transparent (e.g., how confidence scoring, source grounding, and escalation paths are presented to end users)

Agentic AI Layer

  • Design and evolve multi-agent orchestration and reasoning flows (current stack includes Flowise and Langflow), including tool use, retrieval strategies, and prompt/system-message architecture
  • Own the model-grounding strategy — ensuring answers are anchored in vetted, governed content sources rather than open-ended generation
  • Design and refine confidence-scoring mechanisms, including evaluating and improving on current LLM-self-assessed scoring approaches
  • Lead exploration and adoption of MCP (Model Context Protocol) to improve system portability and support broader enterprise integration
  • Integrate agentic workflows with enterprise orchestration platforms (e.g., watsonx Orchestrate)

Data Layer

  • Architect data ingestion pipelines from diverse, mixed-reliability sources (formal validated designs, structured GTM/CRM data, informal internal knowledge)
  • Design and improve retrieval infrastructure (vector search/RAG, e.g. Pinecone), including retrieval quality, chunking strategy, and reducing retrieval-driven answer variance
  • Partner with data science on predictive modeling initiatives (e.g., churn prediction), including data readiness, feature pipelines, and drift monitoring
  • Own data governance and content-scoring frameworks that rank source reliability and trigger low-confidence alerts to users

Platform Layer

  • Make infrastructure and re-platforming decisions as Navi scales across IBM business units, including cloud environment strategy (e.g., Azure to Cirrus transitions)
  • Architect for reliability, determinism, and reproducibility in LLM-based systems (e.g., addressing answer inconsistency through temperature control, retrieval stability, and regression testing)
  • Design for compliance, security, and governance requirements specific to enterprise AI deployment at IBM scale
  • Evaluate and integrate new AI infrastructure/tooling as the ecosystem evolves

What You'll Improve

Beyond maintaining what exists, you're expected to actively advance Navi's current capabilities, including:

  • Reducing answer inconsistency and improving reproducibility across agentic responses
  • Strengthening confidence-scoring rigor beyond current LLM self-assessment approaches
  • Improving text-to-SQL / structured-query reliability for analytics-style queries
  • Advancing the re-platforming and MCP-adoption effort to support enterprise-wide governance and portability goals
  • Building more robust content governance and feedback loops with content owners and subject-matter experts

Preferred Education

Master's Degree

Required Technical And Professional Expertise

  • 20+ years of software/AI/data science engineering experience, including significant hands-on architecture experience
  • Demonstrated ability to design and ship production AI applications end-to-end — not just prototype-level work
  • Deep, practical expertise in LLM application architecture: RAG/retrieval systems, agentic/multi-agent frameworks, prompt and system-message design, and orchestration tooling
  • Strong data engineering background: pipeline design, vector databases, data quality/governance frameworks
  • Experience with cloud infrastructure and platform architecture at enterprise scale
  • Proficiency across the full stack — able to write production code in UI, backend, and data/ML layers as needed, not purely a "whiteboard architect"
  • Strong communication skills — able to represent architecture decisions to both engineering teams and executive stakeholders
  • Proven ability to collaborate effectively with other architects on a shared platform — brainstorming openly, reconciling differing viewpoints, and building consensus around the best overall solution rather than working in isolation

Preferred Technical And Professional Experience

  • Experience with Flowise, Langflow, or comparable low-code/agentic orchestration tools
  • Familiarity with Model Context Protocol (MCP) or similar interoperability standards
  • Experience with enterprise orchestration platforms (e.g., watsonx Orchestrate)
  • Background in applying AI to GTM, customer success, or sales analytics use cases
  • Prior experience navigating large-enterprise governance, compliance, and re-platforming initiatives
Back to blog

Other Jobs To Apply

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...