Initial commit

.gitignore

@@ -0,0 +1,57 @@
+# Compiled source #
+###################
+*.o
+*.so
+
+# Packages #
+############
+# it's better to unpack these files and commit the raw source
+# git has its own built in compression methods
+*.7z
+*.dmg
+*.gz
+*.iso
+*.jar
+*.rar
+*.tar
+*.zip
+*.mtl
+
+
+# OS generated files #
+######################
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# 3D data #
+############
+*.mat
+# *.pkl
+# *.obj
+*.dat
+*.npz
+
+# python file #
+############
+*.pyc
+__pycache__
+build
+
+# pixie data
+###
+*.obj
+*.yaml
+*.pkl
+*.npy
+# data
+internal
+pixie-env
+eval-script
+results
+scripts
+

Doc/docs/getting_started.md

@@ -0,0 +1,33 @@
+## Getting started
+
+Start by cloning the repo:
+```bash
+git clone https://github.com/YadiraF/PIXIE
+cd PIXIE
+```  
+
+#### Requirements
+  * Python 3.7 (numpy, skimage, scipy, opencv, kornia)  
+  * PyTorch >= 1.6 
+    You can run 
+    ```bash
+    pip install -r requirements.txt
+    ```
+      Or create a separate virtual environment by running:  
+    ```bash
+    bash install_conda.sh
+    ```
+    or 
+    ```bash
+    bash install_pip.sh
+    ```
+    For visualization, we use our [rasterizer](https://github.com/YadiraF/PIXIE/tree/master/pixielib/utils/rasterizer) that uses pytorch JIT Compiling Extensions. 
+    If there occurs a compiling error, you can install [pytorch3d](https://github.com/facebookresearch/pytorch3d/blob/master/INSTALL.md) instead and set --rasterizer_type=pytorch3d when running the demos. 
+
+#### Pre-trained model and data
+  * Download [SMPL-X Model](http://smpl-x.is.tue.mpg.de/). Choose 'SMPL-X 2020' to download 'SMPLX_NEUTRAL_2020.npz', put it into `./data`  
+  * Download [PIXIE data](http://pixie.is.tue.mpg.de/): 
+      * the pre-trained model 'pixie_model.tar' and place it into `./data`   
+      * the utilities file `utilities.zip` and extract its contents into `./data`  
+  * (Optional) Follow the instructions for the [Albedo model](https://github.com/TimoBolkart/BFM_to_FLAME) to get 'FLAME_albedo_from_BFM.npz'. Put it into `./data`  
+  * (Optional) Clone and prepare [DECA](https://github.com/YadiraF/DECA)

Doc/docs/notes.md

@@ -0,0 +1,39 @@
+## Discussion
+
+  * **Moderation**: The weight from moderators tell us how much we should trust
+    the feature (information) from body image or part (head/hands) crops, the
+    animation figure shows the visualization of the weights: the lighter color
+    means it trust body more.  
+  * **Face from SMPL-X model**: In PIXIE, we use a common shape variable for the body and the face.
+    The shape space of SMPL-X is learned from 
+    3800 real full body scans and captures the correlation between body and
+    face shape. In the samples script we support the option of predicting 
+    the shape of the full body from a face-only image. If you are only interested in
+    getting accurate face shape,
+    we suggest to try other face-specifc work, such as [DECA](https://github.com/YadiraF/DECA).  
+ * **Model conversion**: If you want to convert the predicted SMPL-X body to a different body model, 
+    e.g. SMPL, please take a look [here](https://github.com/vchoutas/smplx/blob/master/transfer_model/README.md)
+
+### Limitations:
+
+  * **Cropping matters:** Even though we already did a lot of data
+    augmentations durining training, the results will still vary a bit due to
+    cropping differences for the input images. For
+    simplicity, we chose Faster-RCNN as a person detector, 
+    while in the paper we used OpenPose keypoints to compute a bounding box.  
+  * **Perspective projection**: We use a scaled-orthographic/weak-perspective camera model in PIXIE, which
+      does not work well for images with strong perpective distortion.  
+  * **Misalignment issue**: For regression works that output model
+    (SMPL/SMPL-X) parameters, misalignment with the person in the image is always an issue.
+    This can be improved by using PIXIE results as initialization for an optimization method
+    like [SMPLify-X](https://github.com/vchoutas/smplify-x)
+    to refine the pose. Note that, the moderator weight could also be utilized
+    as a confidence measure during optimization.  
+  * **Speed**: The main bottleneck of PIXIE is the need for three separate encoders
+    for the body, head and hand images. Changing the current
+    backbone, i.e. Resnet50 or HRNet, to a lighter one, like MobileNet, should
+    accelerate inference at the cost of performance. We will attempt to provide different
+    options when we release the **training** code. 
+  * **Texture**: Similar to [DECA](https://github.com/YadiraF/DECA), we rely
+    upon the Basel face model for our albedo space. Its lack of ethnic
+    diversity in the albedo causes the model to often compensate for skin tone with lighting.

LICENSE

@@ -0,0 +1,90 @@
+License
+
+Software Copyright License for non-commercial scientific research purposes
+Please read carefully the following terms and conditions and any accompanying documentation before you download
+and/or use the PIXIE model, data and software, (the "Model & Software"), including 3D meshes, software, and scripts.
+By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use
+of this github repository), you acknowledge that you have read these terms and conditions, understand them, and
+agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use
+the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights
+under this License
+
+Ownership / Licensees
+The Model & Software and the associated materials has been developed at the
+Max Planck Institute for Intelligent Systems (hereinafter "MPI").
+
+Any copyright or patent right is owned by and proprietary material of the
+Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter
+collectively “Max-Planck”)
+
+hereinafter the “Licensor”.
+
+License Grant
+Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:
+
+	• To install the Model & Software on computers owned, leased or otherwise controlled by you and/or your organization;
+	• To use the Model & Software for the sole purpose of performing peaceful non-commercial scientific research, 
+    non-commercial education, or non-commercial artistic projects;
+	• To modify, adapt, translate or create derivative works based upon the Model & Software.
+
+Any other use, in particular any use for commercial, pornographic, military, or surveillance purposes is prohibited. 
+This includes, without limitation, incorporation in a commercial product, use in a commercial service, 
+or production of other artefacts for commercial purposes. 
+
+The Model & Software may not be used to create fake, libelous, misleading, or defamatory content of any kind excluding 
+analyses in peer-reviewed scientific research. 
+The Model & Software may not be reproduced, modified and/or made available in any form to any third party 
+without Max-Planck’s prior written permission.
+
+The Model & Software may not be used for pornographic purposes or to generate pornographic material whether
+commercial or not. This license also prohibits the use of the Model & Software to train methods/algorithms/neural
+networks/etc. for commercial use of any kind. By downloading the Model & Software, you agree not to reverse engineer it.
+
+No Distribution
+The Model & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered
+for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive
+purposes only.
+
+Disclaimer of Representations and Warranties
+You expressly acknowledge and agree that the Model & Software results from basic research, is provided “AS IS”,
+may contain errors, and that any use of the Model & Software is at your sole risk.
+LICENSOR MAKES NO REPRESENTATIONS
+OR WARRANTIES OF ANY KIND CONCERNING THE MODEL & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY
+LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor
+makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of
+the Model & Software, (ii) that the use of the Model & Software will not infringe any patents, copyrights or other
+intellectual property rights of a third party, and (iii) that the use of the Model & Software will not cause any
+damage of any kind to you or a third party.
+
+Limitation of Liability
+Because this Model & Software License Agreement qualifies as a donation, according to Section 521 of the German
+Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only.
+If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee
+for the resulting damage.
+
+Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have
+arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be
+liable in accordance with the German Product Liability Act in the event of product liability. The foregoing
+applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall 
+be excluded. Patent claims generated through the usage of the Model & Software cannot be directed towards the copyright holders.
+The Model & Software is provided in the state of development the licensor defines. If modified or extended by
+Licensee, the Licensor makes no claims about the fitness of the Model & Software and is not responsible
+for any problems such modifications cause.
+
+No Maintenance Services
+You understand and agree that Licensor is under no obligation to provide either maintenance services,
+update services, notices of latent defects, or corrections of defects with regard to the Model & Software.
+Licensor nevertheless reserves the right to update, modify, or discontinue the Model & Software at any time.
+
+Defects of the Model & Software must be notified in writing to the Licensor with a comprehensible description
+of the error symptoms. The notification of the defect should enable the reproduction of the error.
+The Licensee is encouraged to communicate any use, results, modification or publication.
+
+Publications using the Model & Software
+You acknowledge that the Model & Software is a valuable scientific resource and agree to appropriately reference
+the following paper in any publication making use of the Model & Software.
+
+Commercial licensing opportunities
+For commercial uses of the Model & Software, please send email to [email protected]
+
+This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.