This abstract was part of the 2026 UCSF AI and Education Symposium
UCSF’s greatest asset is its people, yet connecting trainees with the right mentors across 4,000+ faculty remains a significant challenge. Traditional keyword searches often fail due to query ambiguity and information overload; a search for "mentorship" on UCSF Profiles currently returns nearly 800 results, forcing students to manually sift through profiles or abandon the search.
We propose a Show-and-Tell session where we'll demonstrate AI Mentor Match, a prototype developed to bridge the semantic gap between a prospective mentee's intent and the results hidden in UCSF Profiles.
We'll plan to first showcase a live demo currently being integrated with the UCSF Profiles team.
- Input: The tool accepts a simple "one-liner" from a mentee (e.g., "I am a resident interested in medical education scholarship regarding health equity").
- Process: The system generates broad search queries to retrieve a wide pool of candidates via the UCSF Profiles API, then utilizes a Large Language Model (LLM) to semantically analyze and rank these profiles against the user's specific intent.
- Output: The user receives a curated, ranked list of potential mentors with AI-generated summaries explaining why they are a good match.
Discussion:
- We will discuss findings from our development phase, where we evaluated the system against a "Gold Standard" dataset of mentor matches curated by Lekshmi Santhosh, MD. As UCSF's Associate Chair of People Development and Mentorship she was a natural choice to use as the domain expert to create "Gold Standards."
- We discovered that human-curated matches often rely on non-intuitive natural language connections that would be nearly impossible to guess as a keyword queries. Consequently, our tool does not rely on precise search strings; instead, it casts a wide net and relies on the LLM's natural language capabilities to find the "mentor in the haystack."
- We are creating an LLM-as-a-judge system to create a robust evaluation metric that can be tracked over time.
Tools and Resources:
- This project utilizes OpenAI's API for testing, and in production will use UCSF’s Versa API. The architecture is designed for sustainability and low cost (utilizing models like the -mini series of GPT where possible). We aim to continue to reduce the cost of each search, without any cost optimization each search costs about 25-50 cents.
Adaptability:
- While designed for mentorship, the development approach is highly adaptable. The key teachable steps are: (1) find a single domain expert to provide gold standard matches, (2) iteratively test the system by hand to understand initially how difficult / easy it will be to achieve the result you desire, (3) start with building evaluations first, so you know if your system is improving or not (e.g. LLM-as-a-judge).
Contact
Gregory Ow, [email protected]
Lekshmi Santhosh, [email protected]