Unraveling the Mysteries of Deepfakes: A Journey Through Innovation and Iteration

Once upon a time, in the realm of artificial intelligence, I embarked on a quest to tackle one of the most perplexing challenges of our digital age: detecting deep fakes. Armed with determination and a thirst for knowledge, I delved into the depths of research papers, algorithms, and datasets, ready to face whatever obstacles lay ahead.

For those of you new to term deep fakes: It can refer to computer-generated images of human subjects that do not exist in real life.

To view the project for yourself CLICK HERE

Approach 1: The Quest for Low-Level Cues

My journey began with a noble idea: to seek out the subtle traces left behind by the creators of artificial images. With the guidance of esteemed mentors like ResNET 50 and ResNET 18, I set out to train models capable of discerning between real and fake images. But alas, my optimism soon met its match.

As the days turned into weeks, and the epochs multiplied, I encountered unforeseen challenges. Despite my valiant efforts, the classifiers faltered when faced with images birthed from unfamiliar generative techniques. It became clear that a new path must be forged if I were to succeed in my mission.

Flaws with Approach 1

1. ⏳ Time Consumption: Training the models proved to be a time-consuming endeavor, with accuracy not entirely guaranteed.
2. 🔄 Limited Generalization: The classifiers struggled to generalize beyond the specific generation techniques used during training.
3. 🎭 Classification Conundrum: A peculiar phenomenon emerged wherein real images were misclassified as fake when generated using different techniques.

Approach 2: Pioneering Generalization

Undeterred by setbacks, I charted a new course toward enlightenment. Instead of confining myself to the narrow confines of specific generation methods, I embraced the power of generalization. By harnessing the might of pretrained backbones and an arsenal of classifiers, I ventured forth into uncharted territory.

Implementation Details

• 📂 Datasets Exploration: Laion vs LDM100, ImageNet vs LDM200, bigGAN Real vs bigGAN Fake.
• 🎨 Transformation Tactics: From Gaussian Blur to Jitter, each transformation revealed its impact on model robustness.
• 🧠 Backbone Models: DINO ViT-B/16, DINO ResNET50, CLIP ViT-B/16.
• 📊 Classifier Selection: A plethora of classifiers, from Decision Trees to SVMs, were put to the test.
• 🌀 Dimensionality Reduction Dilemma: To reduce or not to reduce? PCA and autoencoding offered varying degrees of success.

Findings and Inference

1. Impact of Transformations: Despite challenges with Gaussian Blur, certain transformations proved beneficial for model robustness.
2. Dimensionality Reduction Insights: Autoencoding emerged as a promising alternative to traditional methods.
3. Classifier Comparisons: SVM, Linear Probing, and Linear Discriminant Analysis showcased superior performance.
4. Backbone Brilliance: CLIP ViT-B/16 emerged as the champion, surpassing its counterparts in all cases.

Beyond the Horizon: Pushing the Boundaries

But my journey did not end with mere revelations. Nay, I sought to push the boundaries of possibility even further. Through the fusion of multiple backbones and the pursuit of cross-dataset generalization, I embarked on a quest for excellence.

Further Experimentation

• 🌐 Backbone Combinations: DINO ViT-B/16 with DINO ResNET50, CLIP ViT-B/16 with DINO ResNET50, and more.
• 🛠 Model Optimization: Randomized jitter and blur were introduced, yet the quest for optimization persisted.
• 🧭 Cross-Dataset Testing: Testing models trained on one dataset over another revealed intriguing insights into generalization.

Inference and Reflection

Despite valiant efforts, the optimal solution remained elusive. CLIP ViT-B/16 continued to reign supreme, yet challenges persisted in achieving universal generalization. However, each setback served as a stepping stone towards greater understanding and innovation.

Results and Conclusion: Triumph Amidst Adversity

In the end, amidst the trials and tribulations, victory was mine to claim. Through perseverance and a willingness to adapt, I unearthed the optimal path forward. With a generalized backbone, SVM or LDA as my steadfast companions, and the wisdom gleaned from my journey, I achieved unparalleled accuracy in the detection of deepfakes.

Key Results

1. 🏆 Triumphant Discoveries: The best accuracy achieved soared to new heights:
   • Test accuracy: 98.1875% with train accuracy at 98.75% when trained and tested over combined datasets.
   • For GANs: Generalization prevailed, achieving accuracy of 98.875% when tested across various datasets.

Future Scope: Towards New Horizons

As the sun sets on one chapter of my journey, another beckons on the horizon. The future holds untold possibilities: from harnessing text embeddings for image classification to unlocking the full potential of autoencoders. With each new endeavor, I embrace the unknown, for it is in the pursuit of knowledge that true innovation thrives.

Exploring New Frontiers

• 🔍 Text Embeddings Exploration: Unveiling the semantic information within images holds promise for future endeavors.
• 🔄 Autoencoder Advancements: Further enhancements in autoencoder hyperparameters could unlock untapped potential.
• 📈 Larger Datasets Utilization: Scaling up datasets promises to further enhance model robustness and generalization.

Join me as we embark on this odyssey together, charting a course towards a future where technology serves as a beacon of truth and enlightenment.