A Human-Like, Real-Time Healthcare AI Voice Scheduler Powered by Amazon Nova Sonic

Summary

INNERLUXES built a HIPAA-compliant AI voice agent that automates healthcare appointment scheduling through natural, human-like conversations. It uses the Amazon Nova Sonic speech-to-speech model together with the LiveKit Media Server to handle same-call confirmations with no backend latency, and is projected to reduce scheduling costs by at least 50%.

Where Existing AI Voice Agents Fall Short

Most appointment-scheduling AI voice agents on the market rely on carefully balancing several models — one converts speech to text, another interprets the language and generates a response, and a third turns text back into speech. This multi-step setup often produces robotic-sounding responses and slow, fragmented conversations. Such assistants ignore the natural pauses when a patient hesitates, can't handle polite turn-taking, and don't adapt to variations in speaking style, prosody, and tone, so interactions become awkward, impersonal, and error-prone.

The same limitations complicate real-time data exchange with hospital systems such as scheduling tools, EHRs, and practice management software (PMS). As a result, some assistants rely on manual entry or batch processes to update scheduling details and can't check physician availability in real time during the call.

A HIPAA-Compliant Voice Agent That Overcomes These Limitations

To address the drawbacks of both human-operated scheduling and delay-prone multi-step AI agents, INNERLUXES developed a fully customizable, HIPAA-compliant AI voice assistant for healthcare appointment scheduling. Built on Amazon's Nova Sonic speech-to-speech model, the agent holds natural conversations with patients to book, reschedule, and cancel appointments, and can also make outbound rescheduling calls on behalf of providers. It exchanges data with hospital systems — EHR, CRM, and PMS — in real time through FHIR-based APIs.

The assistant integrates Amazon Nova Sonic with the LiveKit Media Server. Because Nova Sonic supports bidirectional streaming through a single API, the agent can listen and respond in real time without delays or awkward pauses, and its function-calling support lets it interact directly with hospital systems during the call — accessing the schedule, checking provider availability, and updating appointment data on the fly.

The agent is estimated to reduce appointment booking time by 40%, cut call abandonment by 30%, and lower operational costs by at least 50%. Because it can handle multiple calls at once, it is expected to process 70% more calls per hour than a patient service representative.

Core Capabilities

The voice assistant's capabilities can be adapted to a provider's needs — for example, multilingual scheduling, the option to book a virtual consultation, or pre-appointment instructions for specific appointment types such as a fasting blood test. Core capabilities include:

• Inbound calls. Answers incoming patient calls and processes booking, rescheduling, cancellation, and clarification requests.
• Outbound calls. Makes calls triggered by system events or rules to confirm, cancel, or reschedule appointments.
• Identity verification. Verifies patient identity on both inbound and outbound calls, escalating to a patient service representative if verification fails.
• Real-time provider availability. Queries integrated systems such as PMS or EHR to confirm provider availability in real time.
• Appointment data management. Creates, reads, updates, and deletes appointment data in the integrated systems, as authorized by role-based access control.

How the Agent Works in a Clinical Setting

When a patient calls, the agent uses Nova Sonic's natural-language capabilities to ask whether they want to make, reschedule, or cancel an appointment. It then verifies identity by asking for the patient's name, date of birth, and the last four digits of their Social Security number, comparing them against the integrated systems (CRM, EHR, or PMS). If verification fails, the call is forwarded to a patient service representative.

Once identity is verified, the agent asks for the preferred physician (by specialty or name), date, and time, checks the slot's availability in real time within the integrated scheduling system, books the appointment, saves it back to the system, and generates a unique booking ID that it reads out to the patient. Through FHIR integration, it can dynamically create, read, update, and delete appointment data across patient records, appointment calendars, and room-availability trackers, and can make outbound confirmation or rescheduling calls to patients who have given express consent.

Reference Architecture

INNERLUXES's principal architects provided a reference architecture for the agent; the specific technologies and services can be adapted to each provider's requirements and budget. The architecture supports integration with leading agentic AI models, including Amazon Nova Sonic, OpenAI's GPT-4o, and Google's Gemini 2.5 Pro.

The agent is deployed in a HIPAA-compliant Amazon Virtual Private Cloud (VPC), integrates with hospital systems via FHIR-based APIs, and routes all interactions through LiveKit for real-time processing. Incoming calls reach a dedicated number via the Amazon Chime SDK, which handles the telephony connection, and are forwarded to the LiveKit Media Server over SIP trunking. The Media Server creates a "room" per call, adds the patient and the agent as participants, and merges their audio into a real-time conversational session, recording the dialogue to an AWS S3 bucket for transcripts, compliance checks, analytics, and model improvement.

All conversations pass through Amazon Bedrock Guardrails, an AI firewall that monitors the dialogue for anomalous or malicious behavior, verifies patient intent against the agent's authorized scope, and enforces HIPAA compliance by filtering responses to prevent disclosure of protected health information (PHI). The system scales horizontally — each agent instance handles one call at a time, but instances can be spawned dynamically to manage high call volumes. The agent follows configurable, prompt-style instructions, and each action is handled by a dedicated tool — for example, a verify-identity tool that queries the EHR, an availability tool that retrieves real-time provider slots, and a booking tool that schedules the appointment directly in the EHR.

Security and HIPAA Compliance

The solution's security and compliance framework rests on these core components:

• The system runs within an audited, encrypted, access-controlled Amazon VPC.
• Amazon Bedrock Guardrails acts as an AI firewall, filtering all agent–patient interactions in real time to prevent disclosure of sensitive data and detect suspicious input such as prompt injection.
• AWS CloudTrail logs all API calls and keeps detailed audit trails of system and user actions.
• Amazon CloudWatch monitors performance metrics and alerts the security team to suspicious changes.
• AWS Security Hub evaluates controls against NIST SP 800-53 and recommends fixes for identified gaps.
• Amazon Macie uses machine learning to discover, classify, and protect sensitive data in S3, flagging unencrypted or over-shared buckets and triggering remediation.
• VPC endpoints keep all agent-to-AWS communication on the private AWS network, eliminating public-internet exposure.

For data exchange and access control, FHIR securely integrates the agent with EHR, CRM, and PMS; identity verification uses the patient's name, date of birth, and last four SSN digits, redirecting to a representative on failure; role-based access control lets the agent schedule without direct access to clinical or billing data; and encryption is enforced at rest (AES-256) and in transit (TLS 1.2) across stored data and voice communications.

Estimated Results

Unlike most AI voice agents — which rely on multiple models and sound slow and robotic — INNERLUXES's scheduling assistant supports natural, human-like conversations tuned to the tone, rhythm, emotion, speed, and speaking style of each patient. It is estimated to:

• Reduce appointment booking time by 40%.
• Cut call abandonment rates by 30%.
• Lower operational costs by at least 50%.
• Process 70% more calls per hour than a patient service representative by handling multiple calls simultaneously.

Technologies and Tools

Technologies: AI, GenAI, LLM, Cloud.

Tools and platforms: Amazon Bedrock, Amazon Nova Sonic, Amazon Bedrock Guardrails, AWS Polly, AWS Transcribe, Amazon Elastic Container Service (ECS), VPC Endpoints, LiveKit SDK, LiveKit Media Server, Amazon Chime SDK, AWS CloudTrail, Amazon CloudWatch, AWS Security Hub, Amazon Macie.