KioskVue
✨Sketch It, Fetch It, Get It✨
BRIEF INTRO:
• KioskVue is an AI-powered retail kiosk system that enhances the
shopping experience by integrating real-time drawing/handwriting
recognition and product recommendations.
• This project leverages Google Generative AI (Gemini API) and
searches for the object in given database/CSV file for occurrence
and displays products in real-time.
Situation
💡 Retail shopping often lacks accessibility, personalization, and efficiency.
• Customers struggle to find the exact products they need.
• Often it can be difficult to visually describe an object by words.
• Many users prefer drawing the object, but most systems can’t process it
efficiently.
🚀 Accessibility Challenges:
• Non-Verbal Individuals: People who cannot speak can write their requests using
the AirWrite panel, enabling effective communication in retail environments.
• Elderly & Disabled Users: Offers a more accessible way to interact with kiosks
without requiring complex touch or voice interactions.
Task
1️⃣ AirWrite Interface: Captures and processes freehand drawings or
handwritten text, converting them into digital input.
2️⃣ Understands real-time drawn/handwritten input using Generative AI
(Google Gemini API).
3️⃣ Extracts meaningful keywords from the description.
4️⃣ Instantly searches a store inventory (CSV-based) to display relevant
products.
5️⃣ Improves accessibility for users with speech impairments,
disabilities, or visual challenges.
Action
1️⃣ Initialize Trackbars for Color Detection
•Create a window with trackbars to adjust HSV values for color detection.
•Users can modify Hue, Saturation, and Value thresholds dynamically.
2️⃣ Set Up Data Structures for Drawing
•Use deque to store colored points for smooth drawing.
•Maintain separate lists for Blue, Green, Red, and Yellow strokes.
3️⃣ Create Canvas (Paint Window)
•Initialize a blank white canvas (paintWindow).
•Draw color selection buttons (Blue, Green, Red, Yellow) and a "CLEAR" option.
4️⃣ Initialize Webcam & Setup Environment
•Load the default webcam using cv2.VideoCapture(0).
•Flip the frame horizontally for a mirror-like view.
•Convert the captured frame to HSV color space for better color detection.
5️⃣ Trackbar Configuration for Color Detection
•Create trackbars to adjust the upper and lower HSV values dynamically.
•Retrieve HSV values in real-time for detecting the pointer.
6️⃣ Draw Color Selection Buttons on Screen
•Display buttons for different colors (Blue, Green, Red, Yellow).
•Using a precise prompt:
•Instead of asking for an explanation, you directly request the object name (e.g., "Just tell what
•object is in the image that can be bought from a shopping store. No explanations.").
•This avoids unnecessary text and ensures concise results.
•Restricting token count (max_output_tokens=7).
•Used Regex to filter out stopwords.
OPTIMIZATION-
-Track word frequency across multiple inputs.
-Identify frequently occurring but unimportant words.
-Automatically update the stopwords list based on user feedback.
Result
1️⃣ User opens the webpage
2️⃣ window.onload = loadProducts; runs automatically
3️⃣ fetch("/get_products") sends a GET request to Flask
4️⃣ Flask receives request → reads output.json → sends JSON response
5️⃣ JavaScript receives the data → updates the webpage dynamically
1️⃣ Initially Used an Array – Faced Challenges:
• I started with an array because it's simple and provides O(1) indexing for quick access.
• However, I realized that insertions and deletions in the middle were costly (O(n)).
• Since AirWrite required frequent insertions and deletions from both ends, an array was
inefficient.
2️⃣ Shifted to a Doubly Linked List (DLL) – Faced Challenges:
• Then, I moved to a doubly linked list to improve insertion and deletion times.
• DLL allows O(1) insertion and deletion from both ends.
• However, it increased memory overhead due to storing extra pointers, and random
access became O(n) instead of O(1) like an array.
• Also, managing pointers increased complexity.
3️⃣ Finally Used a Deque – Solved All Issues:
• I finally used deque (double-ended queue) because it provides:
✅ O(1) insertions and deletions from both ends.
✅ Efficient memory usage (implemented using a dynamic circular array).
✅ Easier management compared to DLL (no manual pointer handling).
