DEV Community: Achal Jain

Unraveling the Mysteries of Deepfakes: A Journey Through Innovation and Iteration

Achal Jain — Fri, 12 Apr 2024 15:41:01 +0000

Once upon a time, in the realm of artificial intelligence, I embarked on a quest to tackle one of the most perplexing challenges of our digital age: detecting deep fakes. Armed with determination and a thirst for knowledge, I delved into the depths of research papers, algorithms, and datasets, ready to face whatever obstacles lay ahead.

For those of you new to term deep fakes: It can refer to computer-generated images of human subjects that do not exist in real life.

To view the project for yourself CLICK HERE

Approach 1: The Quest for Low-Level Cues

My journey began with a noble idea: to seek out the subtle traces left behind by the creators of artificial images. With the guidance of esteemed mentors like ResNET 50 and ResNET 18, I set out to train models capable of discerning between real and fake images. But alas, my optimism soon met its match.

As the days turned into weeks, and the epochs multiplied, I encountered unforeseen challenges. Despite my valiant efforts, the classifiers faltered when faced with images birthed from unfamiliar generative techniques. It became clear that a new path must be forged if I were to succeed in my mission.

Flaws with Approach 1

⏳ Time Consumption: Training the models proved to be a time-consuming endeavor, with accuracy not entirely guaranteed.
🔄 Limited Generalization: The classifiers struggled to generalize beyond the specific generation techniques used during training.
🎭 Classification Conundrum: A peculiar phenomenon emerged wherein real images were misclassified as fake when generated using different techniques.

Approach 2: Pioneering Generalization

Undeterred by setbacks, I charted a new course toward enlightenment. Instead of confining myself to the narrow confines of specific generation methods, I embraced the power of generalization. By harnessing the might of pretrained backbones and an arsenal of classifiers, I ventured forth into uncharted territory.

Implementation Details

📂 Datasets Exploration: Laion vs LDM100, ImageNet vs LDM200, bigGAN Real vs bigGAN Fake.
🎨 Transformation Tactics: From Gaussian Blur to Jitter, each transformation revealed its impact on model robustness.
🧠 Backbone Models: DINO ViT-B/16, DINO ResNET50, CLIP ViT-B/16.
📊 Classifier Selection: A plethora of classifiers, from Decision Trees to SVMs, were put to the test.
🌀 Dimensionality Reduction Dilemma: To reduce or not to reduce? PCA and autoencoding offered varying degrees of success.

Findings and Inference

Impact of Transformations: Despite challenges with Gaussian Blur, certain transformations proved beneficial for model robustness.
Dimensionality Reduction Insights: Autoencoding emerged as a promising alternative to traditional methods.
Classifier Comparisons: SVM, Linear Probing, and Linear Discriminant Analysis showcased superior performance.
Backbone Brilliance: CLIP ViT-B/16 emerged as the champion, surpassing its counterparts in all cases.

Beyond the Horizon: Pushing the Boundaries

But my journey did not end with mere revelations. Nay, I sought to push the boundaries of possibility even further. Through the fusion of multiple backbones and the pursuit of cross-dataset generalization, I embarked on a quest for excellence.

Further Experimentation

🌐 Backbone Combinations: DINO ViT-B/16 with DINO ResNET50, CLIP ViT-B/16 with DINO ResNET50, and more.
🛠 Model Optimization: Randomized jitter and blur were introduced, yet the quest for optimization persisted.
🧭 Cross-Dataset Testing: Testing models trained on one dataset over another revealed intriguing insights into generalization.

Inference and Reflection

Despite valiant efforts, the optimal solution remained elusive. CLIP ViT-B/16 continued to reign supreme, yet challenges persisted in achieving universal generalization. However, each setback served as a stepping stone towards greater understanding and innovation.

Results and Conclusion: Triumph Amidst Adversity

In the end, amidst the trials and tribulations, victory was mine to claim. Through perseverance and a willingness to adapt, I unearthed the optimal path forward. With a generalized backbone, SVM or LDA as my steadfast companions, and the wisdom gleaned from my journey, I achieved unparalleled accuracy in the detection of deepfakes.

Key Results

🏆 Triumphant Discoveries: The best accuracy achieved soared to new heights:
- Test accuracy: 98.1875% with train accuracy at 98.75% when trained and tested over combined datasets.
- For GANs: Generalization prevailed, achieving accuracy of 98.875% when tested across various datasets.

Future Scope: Towards New Horizons

As the sun sets on one chapter of my journey, another beckons on the horizon. The future holds untold possibilities: from harnessing text embeddings for image classification to unlocking the full potential of autoencoders. With each new endeavor, I embrace the unknown, for it is in the pursuit of knowledge that true innovation thrives.

Exploring New Frontiers

🔍 Text Embeddings Exploration: Unveiling the semantic information within images holds promise for future endeavors.
🔄 Autoencoder Advancements: Further enhancements in autoencoder hyperparameters could unlock untapped potential.
📈 Larger Datasets Utilization: Scaling up datasets promises to further enhance model robustness and generalization.

Join me as we embark on this odyssey together, charting a course towards a future where technology serves as a beacon of truth and enlightenment.

🏆 Exciting News: Winning Bronze Medal at Inter IIT Tech Meet 12.0! 🏆

Achal Jain — Sat, 06 Jan 2024 14:20:29 +0000

Ahoy, fellow tech enthusiasts! Buckle up as I take you on a rollercoaster ride through my quantum adventure at the Inter IIT Tech Meet 12.0 in Madras, India. Picture this: 12+ events, high prep, mid prep, low prep, and a sprinkle of quantum magic. This was my second time in the competition, and guess what? We clinched the bronze medal! 🥉

The Background: What is Inter IIT Tech Meet?

Inter IIT - as its name suggests is a competition between IITs
Tech Meet - There are mainly 3 competitions (Cultural, Sports and Technical). Tech refers to the technical meet

There I was, just an ordinary day, sifting through the mundane sea of emails when BAM! I see this mail announcing the 12th edition of the Inter IIT Tech Meet. A competition in which amazing companies like Adobe and DevRev, and various others all in different domains offer problem statements that are to be solved by the students of IITs, with each IIT sending a representative team for their college. The competition is quite fierce with each college trying to prove their name and their technical skills (with all of them being IITs!).

I was very excited, and I knew I wanted to be part of the problem for AI/ML by DevRev. The interview takes place .... I don't get selected. I go for my next best interest (Algo-Trading Challenge by Zelta) .... I don't get selected.

I say lets give a last try for Quantum Computing Challenge by Mphasis. A few days pass and I am selected !!! (At this point I am pretty good at classical computing but still a beginner with quantum computing). The college forms a squad of 6 members - selected via each participant giving a proposal and then selecting the best ones from those.

The Challenge: Airline Woes and Quantum Rescues

So, there we were, a squad of six tech enthusiasts who had just met each other. The challenge was given - "Solving for passenger recovery during airline disruptions using Quantum Computing." Easy, right? Wrong! In just 15-20 days, we had to tackle this NP Hard problem and optimize like there was no tomorrow. Moreover, our squad was formed with us not having worked with each other before this! What's more, this is a field which is still emerging and has very limited literature for it, so you can't just ChatGPT it!!!

The Holiday that Wasn't 🏝️🚫

Our college vacations started, and while friends scattered like dandelion seeds in the wind, our team stayed on the campus. The campus, usually buzzing with life, transformed into a deserted tech haven. Wake up, work, work, work, sleep – rinse and repeat. Dedication level: 100!

Quantum Hesitation to Revelation 🤔➡️🌟

Confession time: I initially shrugged off quantum computing, thinking we could solve things the good old classical way. Oh, how wrong I was! A meeting with company reps showed they wanted quantum based solution and nothing else would qualify. From clueless to having some starting point – we started our journey

The Odyssey of Quantum Quirks 🌀

We start off by thinking lets make ourselves up to date with the current standards by reading some research papers. I read through a crazy ton of research papers trying to reach the current standards, also reading the same paper multiple times (realizing only after like 10-15 minutes of reading that I have already seen this). At this point we have no idea of what to do. We want to do everything. We just want to win.

We work.... We find a breakthrough solution .... We work on implementing it .... We ask, why are we doing this? This won't work !! .... Back to square one.

Again and again we went through the above process.

"I know let's ask the professors of our college," I thought. "Bravo," everyone said. The next thing we were doing was scheduling meetings with various domain experts.

Now we had some starting point. We split our problem into two and solved each separately. We would find a quantum and classical solution for both and then merge the best combination. I was assigned so solve classical solution for both the subproblems (since my teammates were just better than me quantum computing).

I was still thinking in the back of my head, "We don't need quantum computing, classical algorithms are just better in this case." My team captain then went ahead and challenged me to solve the problem classically faster (as a joke), than they could with a quantum algorithm.

I took that to heart and began my journey. My first sub-problem was, "Find alternate flight paths, subject to constraints that can change dynamically, when a given set of flights are cancelled." Now at first thought this reminded me of NP Hard problems like travelling salesman problem, and like anyone else would, I tried mapping my problem - It did not work. I started implementing algorithms like Ant Colony Optimization (ACO), Genetic Algorithms (GA), Bee Colony Optimization (BCO), Simulated Annealing and many other algorithms (All of which I learned on the way). I then created a solution with a mixture of these. It worked fine but required too much time (too many iterations for convergence). We decided that in the first sub-problem we will go with quantum approach.

I accepted my defeat at that time, but in the back of my head I was thinking of other algorithms.

A few days passed when I suddenly had this thought, "Why not take an assumption - Any alternate flight path shall have only K stopovers (k<2 logically)." I also thought, lets try a simple recursion based algorithm. It will take a long time since recursion cause exponential time complexity, but lets do it anyways.

I code the solution (without telling my teammates). Eureka!!! My code is better than any quantum approach !! It is fast and more accurate than the quantum solution. Why? Taking the assumption that maximum k stopovers are allowed reduced the time complexity from O(N!) -> O(k! * N) to O(N). Since (K!) turned out to be constant. So we now switched to choosing classical approach for pathfinding. (It was a bit slower but it had other benefits - can solve for all paths with 100% accuracy, while quantum approach solves for top k paths with lower accuracy)

Now, moving ahead to the 2nd subproblem "Reallocate affected passengers to the paths found while minimizing delay, checking available seats minimum cost for company, getting multiple solutions, flexible business rules and on and on". I knew I couldn't classically solve this faster than a quantum algorithm.

So I sat with my team mates and understood their current scenario. My captain had this idea to map the problem to the Knap-Sack problem, but since their are multiple paths, we map to multi-knapsack problem. This was fine, but we ran into some issues. There were some constraints we were not able to factor in (not easy to add constraints in quantum. We need to make the QUBO handle these constraints).

I asked my leader to explain the equation he made (cost function). I jokingly asked him, "why not add a new factor alpha, and make this factor handle everything". Then we just looked and each other and thought - That is exactly what we need. We created the factor alpha whose job to separate solutions spaces but putting then far form each other subject to the constraints of the classes.

So we now mapped our problem to Class Constrained Multi Knapsack Problem. This is a problem that has never been solved before via quantum computing (at least in all literature that we found).

Through this we now felt some confidence that we can actually win this competition - while we started with no idea what is even happening.

The Impromptu Symphony 🎶

As if our journey needed more drama, the presentation became a last-minute train escapade. Picture this: me and three team members, crafting our presentation in the train to IIT Madras. Presenting the entire project without any practice. Improv at its finest!

The Overall Experience 💡🍽️

Thorough this unbelievable experience I learnt a lot.

Some times simple is just better. You don't always have to think outside the box
Asking around always helps. Never hesitate to ask. People will be there to help you
Time management is very very important. (Making then presentation at the last minute was probably not a good idea)
Always make sure to work as a team. Keep a good rapport with your team will always help. Always try to explain your thoughts to someone, this will clear your own thoughts
Never Give UP !!!
And a bunch of technical skills like, Algorithms Design 🤖, OOPS, Quantum Algorithms (QAOA, Quantum Annealing, etc) 🌌, Nature-Inspired Algorithms (Ant Colony Optimization, Genetic Algorithms) 🐜, Vectorization of Algorithms using Pandas 🐼, Class Constrained Multi Knapsack Problem using Quantum Computing 🎒, In-depth Algorithm Analysis 📊, QUBO, Linear Programming

Participating in the Inter IIT Tech Meet 12.0 was more than a competition; it was a cosmic adventure of challenges, breakthroughs, and camaraderie. A huge shout out to my professors and teammates for being the real MVPs. As the curtain falls on this quantum saga, I carry forward the lessons learned and skills earned into the ever-evolving tech cosmos.

#InterIITTechMeet #InterIIT #Tech #Coding #Quantum #QuantumComputing #QuantumAlgorithm #Mphasis #MphasisFoundation #Challenge #Software #SoftwareDevelopment #Victory #IITMandi #IIT #QuantumAdventures #TechTales #BronzeVictory #TeamTechies #CodeCraftersUnleashed