Co-written with Morteza Hosseini

Background

Earlier this year, Professor Vijay Janapa Reddy and Kai Kleinbard at Harvard University introduced SocratiQ, an AI-powered educational assistant designed to tackle the challenges of personalized learning at scale. SocratiQ helps students engage with the Machine Learning Systems textbook by offering tailored explanations, adaptive assessments, and interactive conversations.

These features are powered by Large Language Models (LLMs) served via available cloud-based inference services.

SocratiQ offers several features designed to enhance engagement and personalize learning:

• Level-based personalization: Adapts explanations and quizzes to each student’s prior experience with the content.
• Interactive chat: Enables conversational learning.
• Bounded learning: Uses in-context prompting to provide responses based on textbook material.
• Knowledge graph: Tracks student progress and concept mastery.
• Gamification: Encourages engagement through streaks, badges, and quiz milestones.
• Engagement heatmap: Visualizes user activity and learning patterns.

As educational content continues to shift toward digital formats and students increasingly rely on online resources, integrating AI tools directly into textbooks is becoming a practical and impactful reality. This shift presents an exciting opportunity for the Qualcomm Dragonwing AI On-Prem Appliance solution to help educational applications like SocratiQ to move beyond cloud dependency and bring near-edge inferencing into the classroom – AI tutor in the box.

Why near-edge matters for education

Transitioning to near-edge solutions offers several key benefits for students and educators:

• Offline access: Students can continue learning without an internet connection, especially important in remote or bandwidth-constrained environments.
• Enhanced privacy: Local inference eliminates the need to transmit student data to the cloud.
• Low latency: Real-time responses without network delays.
• Reliability: Less  risk of downtime due to connectivity issues or data center availability.
• Total cost of ownership: Lower TCO compared to cloud-based services over the long term.

System architecture

To support near-edge deployment and eliminate reliance on cloud-based models, we restructured the SocratiQ architecture to run entirely on premises. This involved replacing all API calls to external cloud-hosted LLMs with local inference using the vLLM backend, which hosts opensource models directly on the Qualcomm Cloud AI 100 Ultra hardware in the Dragonwing On-Prem Appliance.

• Web interface: JavaScript app embedded via Shadow DOM
• Local database: Stores user preferences and quiz progress using IndexedDB
• Local FastAPI server: Routes prompts and communicates with the vLLM backend
• vLLM backend: Powers AI content generation using speculative decoding for high-throughput, memory-efficient inference
• Model flexibility: Depending on the coursebook's content and language, hosted models can easily be switched to best support the educational objectives.