Summary
The Agentic AI Developer designs, builds, and operates agentic AI process automations and reusable
LLM‑powered components on our Azure-first stack. Partnering with Business Technology Partners (BTPs)
and teams across the company, you will convert business requirements and process maps into
production-grade automations that are observable, governed, and reusable. A core responsibility is
rigorous quality control: you will develop evaluation workflows (test prompts, regression suites),
implement hallucination detection and mitigation, measure and reduce bias, enforce guardrails and
schema validation, and ensure outputs are grounded, reliable, compliant, and auditable—consistent
with SEC governance and U.S. legal requirements for registered financial advisors.
Who you are
You’re a hands-on builder fluent in modern AI tooling who can translate business intent into robust
agentic workflows. You communicate clearly with technical and non-technical stakeholders, execute
with precision, and bring a strong quality mindset to everything you release. You balance speed with
governance, instrument your solutions for observability and cost control, and document decisions for
traceability. You are adaptable, eager to learn new technologies, and able to manage multiple projects
under time constraints.
What you’ll do
ï· Design agentic AI systems that automate multi-step workflows; orchestrate LLM tools/skills and
APIs to deliver reliable outcomes.
ï· Build reusable components (prompt libraries, tools/functions, skills, RAG pipelines, evaluation
harnesses) for enterprise reuse across teams.
ï· Integrate LLMs (e.g., ChatGPT, Claude) with Azure services (Azure OpenAI, Azure AI
Search/Vector, Functions, Logic Apps) and Microsoft 365/Copilot Studio.
ï· Use n8n (and/or Power Automate) for orchestration, eventing, and integrations across line-of-
business systems.
ï· Implement RAG (retrieval‑augmented generation) with vector search and grounding/citation
policies; enforce structured outputs (JSON schemas).
ï· Build evaluation & QC: red-team test prompts, regression suites, groundedness checks, bias
assessments, output validation, cost/latency monitoring.
ï· Instrument observability (logging, trace, metrics), set SLOs (e.g., first‑pass accuracy,
hallucination rate, bias metrics, latency, cost‑per‑task), and implement guardrails.
ï· Follow intake and prioritization through BTPs/supervisor; collaborate directly with business
stakeholders on requirements and acceptance tests.
ï· Ensure security & compliance (PII handling, RBAC, data minimization, audit trails) aligned to SEC
governance and company policies (e.g., prompt/response archiving).
ï· Create documentation and runbooks; contribute to code reviews and best practices; support
end‑users and BTPs with adoption and training.
ï· Troubleshoot and resolve issues related to AI automations and integrations.
ï· Stay current on agentic AI frameworks and Azure/M365 capabilities; propose improvements to
platform, tooling, and standards.
ï· Other duties, responsibilities, and special projects as assigned.
What you’ll bring
Experience
ï· 5+ years professional programming experience; 2+ years building AI tool based solutions (LLM
APIs, Copilot, ChatGPT/Claude, orchestration).
ï· Proven delivery of production automations on Azure and Microsoft 365 ecosystems.
ï· At least 2 years of experience developing solutions with AI tools (e.g., Copilot, ChatGPT, Claude,
n8n, Foundry).
Technical skills (core)
ï· Languages: Python, C#, R (plus familiarity with JavaScript/TypeScript helpful).
ï· Frameworks/tools: LangChain, LangGraph, LlamaIndex, AutoGen (or similar agent frameworks);
Copilot Studio, GitHub, n8n.
ï· Azure: Azure OpenAI, Azure AI Search/Vector, Functions, Logic Apps, Key Vault, Storage,
DevOps/GitHub Actions.
ï· RAG pipelines, vector databases (e.g., Azure AI Search; familiarity with Pinecone/Weaviate a
plus).
ï· API design & integration, structured outputs (JSON schema), function/tool calling.
ï· Observability & cost control: logging, tracing, metrics, rate/usage monitoring.
Quality control & evaluation (explicit)
ï· Ability to design and run test prompts and regression suites covering normal, edge, and
adversarial scenarios.
ï· Skill in hallucination detection/mitigation (groundedness checks, citation requirements, retrieval
confidence thresholds, fallback policies).
ï· Experience measuring and reducing bias (fairness metrics, scenario/corpus checks, cohort A/B
testing, safe‑prompting).
ï· Output validation using schema enforcement, tool/function results verification, and
business‑rule assertions.
ï· Building evaluation harnesses with clear SLOs/KPIs (first‑pass accuracy, hallucination rate, bias
indicators, latency, cost‑per‑task) and reporting dashboards.
ï· Implementing guardrails (content filters, allow/deny lists, sensitive-topic controls) and red‑team
testing before deployment.
ï· Documenting test plans, evaluation results, and traceability from requirements to acceptance
criteria.
Collaboration & compliance
ï· Strong communication with BTPs/end‑users; aptitude for translating requirements into technical
designs and acceptance tests.
ï· Working knowledge of SEC governance and U.S. regulatory expectations for AI usage in financial
advisory contexts; commitment to Responsible AI practices.
ï· Familiarity with enterprise archiving/monitoring of prompts and outputs per policy (e.g.,
Smarsh, Theta Lake) and change‑control practices.
Nice to have
ï· Experience with real‑estate AI platforms (e.g., V7, Kolena, Cadastral) or domain workflows.
ï· Knowledge of GenAIOps/MLOps concepts for lifecycle and evaluation (not model training).
ï· Containers (Docker) and CI/CD; Power Platform experience.
Additional Information for Remote Role
ï· Work Hours: Must have at least 4 hours overlap with Mountain Time zone (Denver, CO).
ï· Communication: Strong internet connection and reliable communication tools are essential.
ï· Remote Work Environment: Must have a dedicated workspace free from distractions to ensure
productivity.
ï· Cultural Sensitivity: Ability to work effectively with a diverse team and understand cultural
differences.
