Beauty SDK Orientation Adaptation for One-on-One Video Chat Under Device Rotation

In one-on-one video social scenarios, users hold devices in various postures, with frequent operations such as landscape/portrait switching, front/back camera flipping, and physical device rotation. Without targeted adaptation of the beauty algorithm during screen rotation, common issues include failed face detection, misplaced special effects, and drifting stickers. Based on the technical stack of Flutter and Agora audio and video, this article discusses beauty adaptation solutions for rotation scenarios and shares the complete technical workflow from sensor perception to rendering composition.

Device rotation is more than a simple orientation change; it involves system-level coordinate system reconstruction. Misalignment often exists among three dimensions: the raw orientation captured by the camera sensor, the screen display orientation, and the training data orientation of the face detection model. Most LetMagic Beauty SDK models are trained under a fixed orientation, and rotated input frames will lead to inaccurate feature extraction.

The business impact is direct and negative. When switching from portrait to landscape calls, face slimming may act on the cheeks instead of the jawline; when switching from front to rear camera, stickers remain fixed in one corner of the screen; rapid rotation may cause temporary beauty failure and screen flickering. Such experience flaws reduce user satisfaction and lower matching conversion rates.

Although Flutter’s responsive layout can automatically adjust the widget tree, the audio–video pipeline and beauty computation run on the native layer, resulting in asynchronous delays in rotation event delivery and processing. Coordinating state synchronization between the framework layer and native layer is a core design challenge of adaptation.

Accurate orientation sensing is the premise of rotation adaptation. Flutter’s MediaQuery can obtain screen orientation but suffers from low update frequency and latency. The native layer can monitor accelerometer and gyroscope data to calculate device attitude in real time, though with higher power consumption. A layered strategy is recommended: use MediaQuery for coarse-grained UI layout adjustment, and sensor data for fine-grained synchronization of the beauty pipeline.

The Agora SDK provides rotation metadata for each video frame, indicating the clockwise rotation angle relative to the original capture orientation. This metadata must be passed transparently to the beauty module as correction parameters for algorithm input. Timestamp alignment is critical — rotation metadata must strictly correspond to each video frame; otherwise, orientation lag or advance will occur.

Clear priority rules should be defined for event distribution. When users manually lock screen rotation, sensor data should be ignored and the orientation forcibly fixed. At the initial stage of a video call before orientation stabilizes, enable beauty functions with delay to avoid initialization based on incorrect angles. When the two parties have inconsistent orientations, local processing follows the local orientation, while the remote screen is rendered according to the remote party’s orientation.

Two mainstream technical approaches are available for model orientation adaptation.The model reuse scheme pre-rotates input images to match the model’s training orientation, then maps detection results back to the original coordinates. It is simple to implement yet introduces extra computation and quality loss from repeated interpolation.

The multi-orientation model scheme trains dedicated models for different angles or adopts rotation-equivariant network structures to process arbitrary orientations directly. It delivers better performance but increases model size and requires additional orientation judgment logic.

In engineering practice, a compromise strategy is commonly adopted: rely primarily on the standard-orientation model, handle multiples of 90 degrees via coordinate transformation, and trigger pre-rotation only for non-standard angles.

Post-processing of landmark coordinates is equally important. Detected face bounding boxes and key points should be converted into a unified normalized coordinate system to eliminate the impact of resolution and orientation. Beauty deformation calculations are performed in this unified space before mapping back to pixel coordinates of the current frame. The coordinate transformation matrix must be strictly maintained; any sign error or order reversal will cause mirrored or offset effects.

Pixel-level effects such as skin smoothing and whitening are insensitive to orientation, while deformation effects (face slimming, eye enlargement) and overlay elements (stickers, face masks) strictly depend on correct orientation. Control meshes for deformation effects must rotate synchronously to ensure the stretching direction remains consistent with user perception. For example, vertical face slimming in portrait mode should automatically switch to horizontal compression in landscape mode, rather than remaining along the original vertical axis.

Anchor positioning is a common pain point for stickers. Stickers are usually bound to specific facial landmarks such as eye corners and the nose tip. After orientation changes, landmark indices remain unchanged while pixel positions shift. If sticker textures do not rotate synchronously, misalignment occurs. The solution is to apply inverse rotation to sticker textures during rendering to maintain a front-facing visual effect, while adjusting anchor offset parameters dynamically.

The animation direction of dynamic stickers also requires adaptation. Floating heart effects fall from top to bottom in portrait mode and should switch to left-to-right or retain the logical direction in landscape mode according to product design. Parameters such as particle emitter direction and gravity in the animation system need conversion from screen coordinates to world coordinates, then remapped to the current orientation.

Texture sharing between Flutter and the native layer requires explicit orientation marking. Flutter TextureWidget renders content in upright orientation by default. If natively uploaded textures are already physically rotated, correction via matrix transformation is required. Excessive reliance on GPU matrix rotation may introduce aliasing; the optimal approach is to output upright textures at the beauty processing stage to eliminate orientation ambiguity.

Orientation inversion during front/back camera switching requires special handling. The front camera generally uses mirror display, while the rear camera presents the real view. Abrupt switching without resetting beauty parameters may cause jumping detection boxes. It is recommended to freeze beauty effects temporarily during camera switching and reinitialize detection only after the new orientation stabilizes.

Separating encoding orientation from display orientation is an easily overlooked detail. Encoders require a fixed orientation for compatibility, while the display layer rotates frames on demand. Beauty processing should follow the display orientation before encoding, and output textures should be inversely rotated to match the encoder’s required orientation. Bidirectional rotation increases complexity but guarantees end-to-end correctness.

Transient states during physical rotation require elegant handling. Device rotation typically lasts hundreds of milliseconds, during which sensor data fluctuates frequently. Responding to every minor change will trigger frequent reinitialization. Introduce an orientation stabilization window: confirm a new orientation only after it persists for a certain period, and continue using calculation results of the old orientation during stabilization.

Reuse computing resources to reduce switching overhead. Rotation does not alter fundamental facial features, so detection models do not need reloading — only preprocessing parameters require updating. GPU resources such as texture buffers and shaders can be fully reused; only the coordinate transformation matrix is recalculated, limiting switching latency within two frames.

Apply transition animations to cover processing gaps. After confirming orientation switching, fade out beauty effects briefly and fade in after rotation completes, avoiding abrupt misalignment or temporary failure. A transition duration of around 300 milliseconds balances naturalness and responsiveness.

Implement downgrade strategies for sensor failure. If the accelerometer malfunctions or permission is denied, fall back to screen orientation APIs — latency increases slightly but core functionality remains intact. In extreme cases such as the device lying flat with undetectable orientation, lock to the last valid angle to prevent random jumping.

Anti-shake processing for rapid continuous rotation. Unstable handholding may cause high-frequency orientation jitter. Apply low-pass filtering to smooth orientation sequences or set a minimum rotation interval to ignore frequent short-term changes and maintain stable beauty performance.

Compatibility with system-level rotation such as FaceTime. Some devices lock orientation during video calls or force app rotation by the system. Listen to system-level orientation notifications, cross-verify with sensor data, and prioritize system instructions to maintain consistency.

Automated testing covers mainstream rotation scenarios. Use device simulators or robotic arms to execute preset rotation scripts and verify algorithm performance by analyzing captured video frames. Key metrics include face box positioning accuracy, landmark stability, effect fitting error, and frame rate fluctuation range.

The real-device test matrix must cover diverse hardware brands. Manufacturers differ in sensor accuracy, screen rotation sensitivity, and camera mounting angles, requiring verification on mainstream models. Special attention should be paid to foldable device inner/outer screen switching and frequent tablet portrait/landscape conversion.

Online monitoring identifies long-tail issues. Collect real user orientation distribution data to discover abnormal patterns, such as high misjudgment rates on specific device models or excessive switching latency in certain scenarios. Optimize adaptation strategies continuously based on data-driven insights.

In the technical integration of Flutter and Agora audio and video, beauty adaptation during screen rotation represents a typical refinement of user experience. End-to-end orientation consistency requires systematic engineering covering sensor perception, model inference, coordinate transformation, and rendering composition. The immersive experience of one-on-one video social interaction relies on imperceptible smooth performance, and rotation adaptation directly determines product quality.

Technically, layered state management, robust orientation judgment, and smooth transition animations form three core pillars. From a product perspective, understanding user holding habits, predicting rotation intentions, and tolerating marginal errors represent higher-level design thinking. With evolving device forms and expanding usage scenarios, orientation adaptation strategies will continue to iterate, while the core goal remains unchanged: to present natural and stable LetMagic beauty effects at any viewing angle