Computer-Vision Neurocognitive Assessment Platforms for Stroke Care

Summary

We build bedside neurocognitive assessment platforms for stroke care. A platform of this kind guides a patient through a fixed sequence of motor, sensory and language tasks, records responses with a camera and microphone, derives objective measures from those recordings, and presents them to a clinician who confirms or overrides the scoring. The aim is consistency and repeatability between assessments, not autonomy.

The Challenge

Neurological assessment after stroke is repetitive, time-sensitive and rater-dependent. Real wards defeat naive assumptions. Blankets, bed rails, lines and monitors occlude limbs, beds are reclined at varying angles so a pose normalisation tuned for standing subjects breaks, lighting is uneven and often behind the patient, and camera placement varies with every setup. Comprehension failure and language impairment are easily confused when instructions are given in a second language. Landmark models degrade unevenly across skin tone, age, glasses, head coverings and bed position, and an unnoticed subgroup failure is a patient safety issue rather than a metric. Scoring that informs neurological diagnosis is regulated software, so agreement with expert raters, not accuracy alone, is the standard it has to meet.

The Solution

The two-screen architecture

Two synchronised surfaces are needed.

• The patient-facing screen presents an animated three-dimensional presenter that demonstrates each task, with instructions shown visually and spoken in the patient own language.
• The clinician screen handles authentication, patient selection, exam initiation, live view of responses, scoring review and report generation.
• The two are kept in lockstep through a persistent bidirectional connection over a shared session state machine, with clock alignment, because response latency is itself a recorded measure.

Vision, audio and scoring pipeline

Video is captured, frames normalised, and landmarks estimated for face and body, giving facial mesh points and skeletal keypoints per frame. Landmarks are normalised for camera distance and torso orientation, then fed to temporal models, recurrent or temporal convolutional networks over landmark sequences, which classify whether a requested movement occurred and how it was performed. Concrete measures include limb elevation and drift over a held interval, symmetry indices across the facial midline for weakness, gaze deviation, movement onset latency and smoothness. Speech tasks add recognition and acoustic features for articulation. Task selection mirrors the structure of established stroke scales, covering level of consciousness, gaze, visual fields, facial palsy, motor arm and leg, ataxia, sensory, language, articulation and extinction, so results map onto scoring clinicians already trust.

Records, integration and data handling

The data model is exam sessions containing item-level scores, derived measures, confidence values and links to the media that produced them. Raw video is sensitive and usually needs a short, explicit retention window with derived landmarks kept in its place. Results belong in the wider record through standard clinical resources so serial assessments appear in the patient timeline. Storage is encrypted, access is role-scoped and audited, and identifiers never appear in logs or filenames.

Clinical validity and safe degradation

Regulated software means a documented development lifecycle, risk analysis, and validation against blinded expert raters reporting agreement statistics and limits of agreement, characterised across the subgroups above. Consent for video recording is a distinct requirement from consent for treatment, and both are captured in the record. Operationally, the system needs a calibration and framing step, explicit detection of occlusion or unusable capture, and graceful degradation to manual scoring for individual items rather than failure of a whole exam. A patient who cannot follow a prompt is clinically informative, so no-response is a recorded outcome rather than an error.

What This Delivers

Clinicians get repeatable, item-level assessment with objective measures behind each score, presented as a proposal to confirm or override rather than an automated verdict. Serial exams become comparable, because tasks are presented identically each time. Ward reality is handled explicitly: occlusion and poor framing are detected, individual items fall back to manual scoring, and results reach the wider clinical record.

Technologies and Tools

• Synchronised patient and clinician surfaces over a persistent bidirectional connection with a shared session state machine
• An animated three-dimensional presenter with multilingual visual and spoken instructions
• Facial mesh and skeletal keypoint estimation, with pose and distance normalisation
• Recurrent and temporal convolutional models over landmark sequences
• Speech recognition with acoustic articulation features
• Encrypted, role-scoped and audited storage with short retention for raw media
• Standard clinical resources for integration with the wider record