PawFACS: Leveraging Semi-Supervised Learning for Pet Facial Action Recognition

Nowadays, Pets are regarded as integral family members because they provide emo- tional support and bonding, yet understanding and assessing their emotions can be chal- lenging. Pet facial action detection, which helps determine their emotional states, re- mains a significant and unsolved research issue. The lack of labeled data, however, presents a challenge, as manually creating labels for various species and breeds is labor- intensive. Drawing from DogFACS and CatFACS standards, we present a multi-label classification system for pet facial action units (AUs) in dogs and cats. Our approach centers on two main objectives: (i) a semi-supervised pseudo-labeling strategy to address the lack of labeled pet facial AU data, and (ii) providing annotations for 8,918 images with multiple pet facial action labels to assist future research in this field. We introduce a semi-supervised learning framework that utilizes a small portion of precisely labeled data, gradually extracting reliable samples from the unlabeled data pool. Various deep neural models based on transformers and convolutional neural networks are evaluated on the CatVsDog benchmark dataset. Our comparison study reveals that SwinTransformer- Tiny outperformed other models with an accuracy of 88.93 % on the CatVsDog dataset. Our effort attempts to bridge the gap between pet and human facial action understanding.

My work at Samsung R&D Institute Banglore as a PhD Intern. A patent has been filed for this work.

Paper Link

(Unfortunately they did not put my name in the paper, maybe because I was not a full-time employee.)