Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

5. 3D interaction techniques.

MIMP PYQ: Explain 3D Manipulation techniques in AR. AND Explain 3D

Manipulation tasks in AR.

3D manipulation in AR allows users to interact with virtual objects overlaid onto the real world. The goal
is to make this interaction intuitive, efficient, and natural. Below is a comprehensive breakdown of 3D
manipulation tasks and techniques used in AR environments.

3D Manipulation Tasks in AR
These are the fundamental actions users perform to interact with virtual objects:

a) Selection
●​ Choosing a specific virtual object for manipulation.
●​ Techniques: Touch-based selection, raycasting, gaze-based selection.

b) Translation (Moving an Object)

●​ Moving an object from one place to another in 3D space.
●​ Methods: Dragging (touch-based), mid-air gestures, device-based motion.

c) Rotation (Changing Orientation)

●​ Adjusting the angle or direction of an object.
●​ Methods: Two-finger rotation (touch-based), wrist movements, virtual hand interaction.

d) Scaling (Resizing Objects)

●​ Increasing or decreasing the size of an object.
●​ Methods: Pinch-to-zoom, distance-based scaling (device movement).

e) Placement
●​ Positioning an object within the AR environment while considering real-world constraints.
●​ Example: Placing a virtual chair on a floor surface in an AR furniture app.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

3D Manipulation Techniques in AR
These techniques define how users interact with virtual objects:

a) Touch-Based Interaction
●​ Used in handheld AR (smartphones, tablets).
●​ Challenges: Mapping 2D touch gestures to 3D interactions.
●​ Examples:
○​ Dragging to move objects.
○​ Pinching to scale.
○​ Rotating fingers to adjust orientation.

b) Mid-Air Gesture-Based Interaction

●​ Uses hand and finger tracking (without touching a screen).
●​ Requires depth sensors and AI for tracking.
●​ Examples:
○​ Grabbing virtual objects with hand movements.
○​ Rotating and scaling using finger gestures.
○​ Waving a hand to trigger functions.

c) Device-Based Interaction
●​ Uses the device’s movement and orientation to manipulate objects.
●​ Examples:
○​ Tilting the device to rotate an object.
○​ Moving the device forward/backward to scale objects.

d) Raycasting
●​ Projects a virtual ray from the user’s device or hand into the AR environment to select and
manipulate distant objects.
●​ Common in AR headsets (HoloLens, Magic Leap).

e) Virtual Hand
●​ Displays a virtual hand within the AR scene to interact with objects naturally.
●​ Mimics real-world hand movements for an immersive experience.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

f) Physics-Based Manipulation
●​ Objects respond to real-world physics (gravity, collision, friction).
●​ Example: Dropping a virtual ball that rolls on a table realistically.

Key Considerations for Effective 3D Manipulation in AR

To ensure a smooth user experience, AR systems must consider:
●​ Intuitive Interaction – Should feel natural and easy to learn.
●​ Precision – Users should be able to make accurate adjustments.
●​ Usability – Techniques should be simple and efficient.
●​ Context Awareness – The AR system should adapt to the user’s environment.

4. Real-World Applications
●​ AR Interior Design (IKEA Place, Houzz AR) – Users place and manipulate furniture in their real
space.
●​ Medical Training (HoloLens Surgery Simulation) – Surgeons practice 3D anatomical
manipulations.
●​ AR Gaming (Pokémon GO, Snapchat Lenses) – Players interact with virtual objects in the real
world.
●​ Manufacturing & Prototyping – Engineers manipulate virtual models before production.

MIMP PYQ: Interaction Techniques for 3D Manipulation in AR.

When discussing interaction techniques for 3D manipulation in AR, we explore how users select,
move, rotate, and scale virtual objects within an augmented reality environment. These techniques
should be intuitive, precise, and context-aware to ensure a seamless user experience.

Core Interaction Techniques for 3D Manipulation

A. Touch-Based Interaction
💡 Common in mobile AR applications (smartphones, tablets).
●​ Users interact with virtual objects using on-screen touch gestures.
●​ Challenge: Mapping 2D touch inputs to 3D interactions.
●​ Examples:
○​ Dragging → Moving objects across the AR scene.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

○​ Pinching → Scaling objects up or down.

○​ Rotating → Changing an object’s orientation with multi-touch gestures.

B. Gesture-Based Interaction (Mid-Air Gestures)

💡 Uses hand and finger tracking to allow direct manipulation.
●​ Provides a more natural and immersive experience.
●​ Requires advanced hand tracking sensors (e.g., HoloLens, Leap Motion).
●​ Examples:
○​ Grabbing → Picking up and moving virtual objects with a grabbing motion.
○​ Hand Orientation → Rotating or scaling objects by changing hand positions.
○​ Waving Hand → Triggering UI elements or animations.

C. Device-Based Interaction
💡 Uses the movement and orientation of the AR device to manipulate objects.
●​ Common in handheld AR (mobile AR) and headset-based AR.
●​ Uses sensors like the accelerometer and gyroscope to detect motion.
●​ Examples:
○​ Tilting Device → Rotating objects.
○​ Moving Device Closer/Farther → Scaling objects.

D. Raycasting ("Laser Pointer" Technique)

💡 Projects an invisible ray from the user’s hand or device into the AR space.
●​ Ideal for selecting distant objects in AR headsets and controller-based AR.
●​ Used in VR/AR headsets like HoloLens, Magic Leap, Oculus.
●​ Example:
○​ Pointing a virtual "laser" to select, move, or resize an object.

E. Virtual Hand Interaction

💡 Displays a virtual representation of the user’s hand within the AR space.
●​ Provides a more direct and natural way of interacting with objects.
●​ Example:
○​ A virtual hand mimics real movements, allowing the user to grab, push, or rotate objects.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

Key Considerations for AR Interaction

●​ Intuitive Design – Users should feel comfortable and natural when interacting.
●​ Precision & Accuracy – Objects should respond accurately to user inputs.
●​ Contextual Awareness – AR systems should understand real-world surroundings.
●​ Usability & Comfort – Interaction should be ergonomic for long-term use.

Real-World Applications
●​ AR Interior Design (IKEA Place, Houzz AR) – Uses touch and mid-air gestures for object
placement. 🏠
●​ AR Gaming (Pokémon GO, Snapchat Lenses) – Uses touch-based and device movement
🎮
for interaction.
●​ Medical AR Training (HoloLens Surgery Simulations) – Uses virtual hand interaction for
precision tasks. 👨‍⚕️
●​ Industrial & Engineering AR (Prototyping, Manufacturing) – Uses raycasting and indirect
touch for precise control. 🏗

The effectiveness of 3D manipulation in AR depends on selecting the right interaction technique based
on hardware capabilities and user needs. From touch-based gestures in mobile AR to virtual hands
and raycasting in AR headsets, these techniques define how users engage with augmented reality
environments.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

IMP PYQ: What is a Manipulation Task? Various Approaches in AR/VR.

In Augmented Reality (AR) and Virtual Reality (VR), a manipulation task refers to the interaction
between a user and a virtual 3D object to modify its position, orientation, or size. This is essential for
object control, spatial understanding, and immersive user experiences.

Core Manipulation Tasks

🔹 Translation (Moving an Object)
●​ Shifting an object from one location to another in 3D space.
●​ Example: Dragging a virtual chair in an AR furniture app.

🔹 Rotation (Changing Orientation)

●​ Adjusting an object’s orientation along X, Y, or Z axes.
●​ Example: Rotating a 3D car model to view different angles.

🔹 Scaling (Resizing an Object)

●​ Increasing or decreasing an object’s size proportionally.
●​ Example: Pinching to zoom in/out on a 3D model in AR.

🔹 Selection (Choosing an Object)

●​ Identifying and highlighting a specific object for further interaction.
●​ Example: Tapping on an AR object to activate its movement controls.

Various Approaches to Manipulation in AR/VR

Manipulation approaches vary based on the hardware, user interface, and application needs. The
main categories include:

A. Direct Manipulation
💡 Users interact directly with virtual objects as if they were physically touching them.
●​ Virtual Hand Interaction → A virtual hand mimics real-world hand movements to grab, move,
and rotate objects.
●​ Touchscreen Interaction → Users use multi-touch gestures on mobile devices to manipulate
objects.
●​ Example: Pinching to scale or dragging to move a virtual object in an AR app.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

B. Indirect Manipulation
💡 Uses tools or interfaces instead of direct physical interaction.
●​ Ray Casting ("Laser Pointer" Method) → Projects a ray to select and manipulate distant
objects.
●​ Widgets & UI Controls → Uses on-screen buttons/sliders to move or rotate objects.
●​ Teleoperation → Remote manipulation via joysticks or controllers (e.g., robotic arm control in
AR).
●​ Example: Using an AR headset controller to point at and rotate objects.

C. Gesture-Based Manipulation
💡 Users manipulate objects using hand or body gestures detected by sensors.
●​ Hand Gestures → Grabbing, swiping, or pinching gestures to move, scale, and rotate objects.
●​ Body Tracking → Moving the head or body to interact with objects (common in AR gaming).
●​ Example: Pinching in mid-air to select and move a holographic object in HoloLens AR.

D. Device-Based Manipulation
💡 The device's motion and orientation determine object manipulation.
●​ Tilting the device → Rotates objects in AR.
●​ Moving the device closer/farther → Scales objects dynamically.
●​ Example: Using a smartphone's gyroscope and accelerometer to control 3D models in AR
apps.

Key Considerations for Effective Manipulation in AR/VR

●​ Precision: Ensuring accurate control over object movements.
●​ Intuition: Making interactions feel natural and user-friendly.
●​ Efficiency: Minimizing effort while maximizing effectiveness.
●​ Context Awareness: Adapting interactions to real-world surroundings.

Real-World Applications of Manipulation Techniques

●​ AR Interior Design (IKEA Place, Houzz AR) – Uses touch and gesture-based manipulation
to move furniture in a room.🏠
●​ Medical AR (HoloLens Surgery Simulations) – Uses virtual hand interaction for precise 3D
anatomical manipulations. 👨‍⚕️
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

●​ Gaming (Pokémon GO, AR Filters) – Uses device-based and touch-based manipulation for
in-game object control.🎮
●​ Industrial AR (Prototyping & Manufacturing) – Uses ray casting and indirect manipulation
for design adjustments. 🏗

Manipulation tasks in AR/VR involve selecting, moving, rotating, and scaling virtual objects. These
tasks can be performed through direct, indirect, gesture-based, or device-based interactions,
depending on hardware capabilities and user needs. The goal is to create intuitive, precise, and
efficient interactions for a seamless AR/VR experience.

MIMP PYQ: Explain input devices with respect to 3D Manipulation.

Input devices in 3D manipulation play a crucial role in interacting with virtual objects, allowing users
to move, rotate, and scale objects in Augmented Reality (AR) and Virtual Reality (VR) environments.
These devices translate user movements into precise digital interactions, making 3D manipulation
intuitive and immersive.

1. Hand Tracking Devices (e.g., Leap Motion, HoloLens, Meta Quest Hand
Tracking)
💡 Uses cameras and sensors to track hand movements for direct interaction.
🔹 Functionality:
●​ Uses infrared cameras or depth sensors to capture hand and finger movements.
●​ Detects position, gestures, and orientation in real time.
●​ Translates hand actions into 3D interactions without requiring a physical controller.

🔹 Application in 3D Manipulation:
●​ Natural interaction → Users can grab, move, rotate, and scale virtual objects using real-world
gestures.
●​ Gesture-based controls → Pinching, swiping, or opening/closing hands can trigger various
commands.
●​ No need for controllers → Fully hands-free operation enhances immersion.

🔹 Advantages:
●​ Highly intuitive and immersive interaction.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

●​ Enables fine-grained control over 3D objects.

●​ Reduces the need for external controllers.

🔹 Disadvantages:
●​ Requires high processing power for accurate tracking.
●​ Lighting conditions and occlusions can impact accuracy.
●​ Can be less precise than controller-based manipulation.

2. 6DOF (Six Degrees of Freedom) Controllers (e.g., Oculus Touch, HTC Vive,
PlayStation VR2 Controllers)

💡 Provides highly accurate tracking of position and orientation for precise manipulation.
🔹 Functionality:
●​ Tracks movement in six degrees of freedom (6DOF):​
🔸 3 for position (X, Y, Z) → Moving forward/backward, left/right, up/down.​
🔸 3 for rotation (pitch, yaw, roll) → Tilting, turning, and rolling.
●​ Includes buttons, joysticks, and triggers for additional input.
●​ Uses sensors and tracking cameras to detect real-time motion.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

🔹 Application in 3D Manipulation:
●​ Precise translation & rotation → Objects can be moved, rotated, and placed with great
accuracy.
●​ Selection and pointing → Controllers allow users to aim and manipulate distant objects.
●​ Multi-functional interaction → Buttons and triggers enable grabbing, scaling, and tool
switching.

🔹 Advantages:
●​ High precision and accuracy in 3D manipulation.
●​ More control options (buttons, triggers, joysticks).
●​ Reliable tracking in different environments.

🔹 Disadvantages:
●​ Requires physical controllers, which may limit natural movement.
●​ Learning curve → Users need to get familiar with the button layout.
●​ Can feel less immersive than direct hand tracking.

Comparison: Hand Tracking vs. 6DOF Controllers

Feature Hand Tracking Devices 6DOF Controllers

Interaction Direct hand gestures Buttons, triggers, joysticks

Type

Precision Lower (depends on tracking quality) High (very precise movements)

Ease of Use Natural & intuitive Requires learning controller layout

Immersion High (hands-free) Moderate (needs a physical device)

Best Use Case Simple, intuitive tasks (e.g., object Complex manipulation (e.g., VR modeling,
picking) gaming)

Both hand tracking devices and 6DOF controllers offer powerful interaction methods for 3D
manipulation in AR/VR. Hand tracking provides an intuitive, controller-free experience, while 6DOF
controllers deliver precision and versatility for more complex tasks.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

PYQ: Explain 3D manipulation interface wrt techniques in AR.

A 3D Manipulation Interface in Augmented Reality (AR) allows users to interact with virtual objects
placed in a real-world environment. The goal is to make manipulating objects intuitive and seamless,
ensuring the interface adapts to user actions and surroundings.

Key Aspects of 3D Manipulation Interfaces in AR

●​ The main challenge is to blend virtual objects with the real world in a natural way.
●​ The interface should be aware of the user's surroundings for precise placement and
interaction.

Interaction Techniques in 3D Manipulation

1. Touch-Based Interfaces (Mobile AR)
📌 Common in smartphone/tablet AR apps​
✅ Uses screen gestures (tap, pinch, swipe) for manipulation
●​ Dragging → Moves objects across the screen (Translation).
●​ Pinching → Scales objects (Zoom In/Out).
●​ Rotating fingers → Rotates objects.

Example: IKEA Place app lets users position furniture in AR using touch gestures.

2. Gesture-Based Interfaces (Hand Tracking AR)

📌 Uses hand and finger tracking for natural, mid-air interaction.​
✅ Provides a more immersive experience
●​ Grabbing gestures → Pick up and move objects.
●​ Two-hand scaling → Resize objects dynamically.
●​ Hand rotation → Changes object orientation.

Example: HoloLens 2 lets users interact with virtual objects using pinching and grabbing gestures.

3. Spatial Interfaces (Ray Casting & Virtual Tools)

📌 Uses spatial understanding for precise manipulation.​
✅ Allows users to interact with distant objects.
●​ Ray Casting → Projects a virtual "laser" from the hand/controller to select objects.
●​ Virtual Widgets/Handles → Appear on objects for resizing or rotation.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

Example: Magic Leap allows ray-based selection of objects in 3D space.

4. Device-Based Interfaces (Using AR Device Motion)

📌 Uses device movement (AR headset/smartphone) for control.​
✅ Translates physical device motion into object interaction.
●​ Tilting the device → Rotates objects.
●​ Moving closer or farther → Scales objects dynamically.

🔹 Example: Google ARCore lets users reposition AR objects by moving their phone.

Key Design Considerations for AR Interfaces

●​ Intuitive Feedback → Use visual highlights or haptic feedback to confirm actions.
●​ Context Awareness → The AR system should adapt based on the user’s environment.
●​ Precision & Accuracy → Ensure smooth and responsive object manipulation.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

●​ Usability → The interface should be easy to learn and use.

Visualizing 3D Manipulation Interfaces in AR

Here are three real-world scenarios demonstrating different manipulation techniques:
📱 Touch-Based AR (Mobile Apps):
●​ A user sees their living room through a smartphone.
●​ A virtual furniture model is placed in AR.
●​ The user drags, pinches, and rotates the furniture on the screen.

🖐 Gesture-Based AR (Hand Tracking):

●​ A user wearing an AR headset sees a floating 3D model.
●​ They grab, rotate, and resize the model with their hands.

🎯 Spatial AR (Ray Casting & Virtual Tools):

●​ A user selects a virtual wall in an AR room using a ray from their hand.
●​ Virtual handles appear, allowing them to resize the wall.

A 3D manipulation interface in AR should make interacting with virtual objects as natural as

real-world handling. By combining touch, gestures, spatial interactions, and device movement, AR
systems create a seamless user experience.

PYQ: Discuss Hybrid techniques wrt 3D Manipulation Tasks

Hybrid techniques in 3D manipulation tasks combine multiple interaction methods to create more
intuitive, efficient, and adaptable user experiences. No single technique is ideal for all situations, so
hybrid approaches integrate different input methods to leverage their strengths while mitigating
weaknesses.

Key Benefits
●​ Enhanced Precision & Control: Combining precise controllers with natural hand interactions.
●​ Contextual Adaptability: Adapts to different tasks by switching between interaction modes.
●​ Improved User Experience: Supports diverse user preferences and skill levels.

Optimized Hybrid Combinations for 3D Manipulation

1. Touch + Hand Gestures
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

✅ Best For: AR applications, 3D modeling, and design software.

●​ Large-scale object manipulation: Use touch gestures (pinch, swipe) for broad placement and
scaling.
●​ Fine adjustments: Use hand gestures (twisting, pointing) for detailed rotations and
modifications.
🔹 Example: In an AR furniture placement app, users place and resize objects using touch, then rotate
or fine-tune their position using hand gestures.

2. 6DOF Controllers + Hand Tracking

✅ Best For: VR environments, gaming, and medical training simulations.
●​ Precision control: 6DOF controllers handle delicate movements (e.g., surgery simulations, CAD
modeling).
●​ Natural interaction: Hand tracking allows grabbing, pointing, and gesturing.
🔹 Example: A VR sculpting app lets users shape objects with a controller’s precision but also smooth or
mold surfaces with their hands naturally.

3. Ray Casting + Direct Manipulation

✅ Best For: Virtual environments, object selection, and architectural design.
●​ Fast selection: Ray casting enables users to point and grab distant objects.
●​ Detailed adjustments: Direct manipulation allows users to move, rotate, or resize objects with
high accuracy.
🔹 Example: In a VR interior design tool, users select a couch from across the room using ray casting,
then adjust its position and rotation using direct hand movements.

4. 2D Inputs (Mouse & Keyboard) + 3D Inputs (Stylus & VR Controllers)

✅ Best For: Medical imaging, 3D modeling, and engineering applications.
●​ Quick navigation: 2D inputs (mouse, keyboard) for UI interactions, menu selections, and rough
positioning.
●​ Depth precision: 3D stylus or VR controllers for depth-aware drawing, sculpting, or medical scan
manipulation.
🔹 Example: In a medical imaging application, a doctor uses a mouse to navigate 2D MRI scans but
switches to a 3D stylus for volumetric analysis.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

Key Considerations for Seamless Hybrid Interaction

●​ Seamless Transitions: Switching between interaction methods should be intuitive and
automatic.
●​ Clear Feedback: Provide visual, haptic, or auditory cues to indicate the active interaction mode.
●​ Task-Specific Optimization: The combination should be tailored to the specific manipulation
task, ensuring efficiency and ease of use.

Hybrid techniques in 3D manipulation offer a powerful way to enhance usability, efficiency, and
adaptability in interactive systems. By strategically combining different input methods, users can benefit
from both precision and natural interaction, leading to a more fluid and effective manipulation experience.

PYQ: What is 3D computer graphics & also discuss the rendering process
3D computer graphics involve generating three-dimensional images from digital data, creating virtual
scenes that can be displayed on a 2D screen. This technology is widely used across multiple industries:

Applications of 3D Computer Graphics

●​ Entertainment: Used in movies, video games, and animation for immersive visuals.
●​ Architecture & Design: Helps in creating realistic visualizations of buildings and products.
●​ Medical Imaging: Generates 3D models of organs and tissues for diagnostics and surgery.
●​ Engineering & Simulation: Assists in designing and testing mechanical parts and complex
systems.

The Rendering Process in 3D Graphics : Rendering is the process of transforming a 3D

scene into a 2D image by simulating lighting, textures, and materials.

1. Modeling
●​ The creation of 3D objects using software like Blender, Maya, or 3ds Max.
●​ Defines object geometry, shape, and size.
●​ Includes polygonal modeling, NURBS modeling, and sculpting techniques.

2. Shading & Texturing

●​ Texturing: Applying image-based materials (e.g., wood, metal, or glass) to objects for realism.
●​ Shading: Simulating how light interacts with surfaces, affecting their appearance.
●​ Common shading techniques:
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

○​ Flat Shading: Basic color per polygon.

○​ Gouraud Shading: Smooth gradients across surfaces.
○​ Phong Shading: More realistic lighting with reflections.

3. Lighting
●​ Placement of virtual light sources (e.g., point lights, spotlights, directional lights).
●​ Determines shadows, reflections, and highlights.
●​ Advanced techniques like global illumination and ambient occlusion improve realism.

4. Rendering Techniques
This is the step where the 3D scene is converted into a 2D image. Two main techniques are used:

●​ Ray Tracing:
○​ Simulates the real behavior of light by tracing rays as they interact with surfaces.
○​ Produces highly realistic shadows, reflections, and refractions.
○​ Used in movies, high-end graphics, and modern real-time rendering (RTX in
gaming).
●​ Rasterization:
○​ Converts 3D objects into pixels using a process called scan conversion.
○​ Faster than ray tracing, making it ideal for real-time applications like video games.
○​ Used in game engines (Unity, Unreal Engine) and GPU-based rendering (DirectX,
OpenGL).

5. Post-Processing
●​ Final touch-ups to enhance image quality.
●​ Common effects include:
○​ Anti-aliasing: Smooths jagged edges.
○​ Motion Blur: Adds realism to fast-moving objects.
○​ Depth of Field: Simulates camera focus effects.
○​ Color Grading & Bloom: Adjusts tone, contrast, and glow effects.

3D computer graphics play a crucial role in various industries, creating lifelike and engaging visuals.
The rendering process is at the heart of 3D graphics, using complex algorithms to transform raw 3D
models into high-quality images. With advancements in AI-driven rendering, real-time ray tracing, and
GPU acceleration, 3D graphics continue to evolve, pushing the boundaries of realism and performance.
Prepared by : Divya Kaurani​ ​ ​ ​ Augmented and Virtual Reality​

See U In The Next Unit ! 🖐️