AI
Fine-Tuning TRELLIS.2 with LoRA: A Practical Guide for Customizing 3D Generation
Hugging Face releases a complete framework for adapting the TRELLIS.2 3D generation model to specific tasks using low-rank adaptation, with containerized scripts and documentation.
1/5
Hugging Face Blog
Customizing 3D generation models for specialized applications—like ensuring furniture meets precise geometry requirements or improving shoe sole accuracy—has typically demanded full model retraining. Hugging Face now offers a streamlined alternative through a new LoRA fine-tuning framework designed specifically for TRELLIS.2, a leading 3D generation model.
How TRELLIS.2 Builds 3D Objects#
TRELLIS.2 processes 3D generation through a five-stage pipeline. It first removes image backgrounds using BiRefNet, then encodes images via DINOv3-L into feature tokens. A sparse-structure diffusion transformer predicts coarse object geometry as a voxel grid, followed by a shape transformer that refines detailed geometry at higher resolution. Finally, a texture transformer adds materials and colors before exporting the result as a textured mesh.
The model uses an internal representation called O-Voxel—a sparse voxel grid that stores geometry and material properties only on object surfaces, keeping computation efficient by ignoring empty space.
Why Selective Adaptation Matters#
Rather than retraining the entire pipeline, the new approach applies LoRA (low-rank adaptation) to only the components that control shape and geometry. The research team froze the texture flow to preserve its learned capabilities, since their training dataset contained diffuse color maps (with baked lighting) rather than the separate material maps the texture stage requires.
As the guide notes, "Choosing what to freeze matters as much as choosing what to fine-tune."
Practical Implementation#
Hugging Face supplies two Docker images—one for preprocessing meshes into latent representations and one for training—that run on Hugging Face Jobs or local hardware. The workflow involves three steps:
- Prepare the data: Convert 3D models (GLB/OBJ) into TRELLIS.2's latent format and render 16 reference views per object using Blender
- Train adapters: Run separate jobs for the sparse-structure and shape transformers with a LoRA configuration
- Deploy: Upload fine-tuned adapters to the Hugging Face Hub for reuse
For their footwear case study (188 shoe objects from Google Scanned Objects), the configuration trained approximately 9.3 million adapter parameters—rank-8 adapters for the sparse-structure flow and rank-16 for the shape flow. Full preprocessing took roughly 1 hour on an A100 GPU.

