Cross-Platform Fake Profile Detection: Experimental Machine Learning Graduation Project

Introduction

After a long journey at the Higher School of Computer Science in Sidi Bel Abbès, I finally graduated on September 22, 2025.

In this blog, I will explain the steps I followed to build my graduation project, which was: “An experimental ML system for robust fake profile detection across social platforms using multi-strategy feature selection and model benchmarking”, using metaheuristic and machine learning methods.

Table of Contents

• Introduction
• The Journey
• The Problem I Tried to Solve
• Data & Sources
  • Research Papers
  • Datasets
  • Metaheuristic Research Papers
• Methodology Overview
• System Implementation
  • Step 1: Generic Metaheuristic Engine
  • Step 2: Data Preprocessing and Cleaning
  • Step 3: Feature Selection
  • Step 4: Machine Learning
    • Step 4.1: Algorithms
    • Step 4.2: Splits
    • Step 4.3: Validation Strategy
    • Step 4.4: Metrics & Evaluation Strategy
    • Step 4.5: Hyperparameter Optimization
    • Step 4.6: Recap
  • Step 5: Feature Scoring
  • Step 6: Retraining with Best Features and Best Algorithm
• Results
• Discussion
• Deployment
• Improvements for Future Work
• Conclusion

The journey

Although it was a long journey that took about 8 months of serious work, it’s the period where I evolved the most during my academic path.

It may look simple, you just have to follow this steps:

1. try something
2. criticize it
3. improve it
4. and repeat

But in reality, it’s much harder than that.

You constantly:

• question your own work
• go back and redo things from scratch
• test multiple strategies that don’t always work
• and spend hours waiting for results that might not always satisfies you

The first months felt like I wasn’t making real progress.
I was implementing metaheuristics in a generic, reusable way, but it didn’t produce immediate results. I was also trying to figure out the best direction to follow, which made things even more confusing.

But looking back, those were actually the months where I progressed the most.

They built the foundation for everything that came after. Once that part was done, the next steps became much easier and much faster.

The Problem I Tried to solve

There is a large number of research papers on the topic of fake profile detection across social networks. These works have contributed significantly to the field, both in terms of methods and results.

However, from the analysis of many of them, I noticed a common pattern.

Most of the papers I studied work under relatively limited settings:

• they focus on one or two social platforms
• they use one or two datasets
• they test a small number of heuristics or optimization methods (if any)
• and they usually rely on one or two data split strategies

In most cases, each study explores a specific and constrained configuration, but rarely the problem in a broader and more systematic way.

So the question that came to me was: What if we go further?

What if instead of evaluating models in a limited experimental setup, we try to build a more general and experimental system, that:

• works across multiple social platforms
• uses multiple datasets
• explores multiple metaheuristic optimization strategies
• tests different data splits
• and provides a more rigorous and comparative benchmarking of models

Data & Sources

Research papers

I analysed 6 research papers on 3 main platforms:

1. Twitter (X now)
2. Facebook
3. Instagram

All of them expose labeled datasets created manually by researchers and use machine learning or deep learning methods. They also report performance scores, which allowed me to:

• understand existing approaches
• identify gaps and limitations
• reuse datasets for fair comparison

Datasets

I worked on 5 datasets in total:

• 1 Facebook dataset
• 1 Twitter dataset
• 3 Instagram datasets

This multi-platform setup allowed me to evaluate how models behave across different social media environments and test their generalization ability.

All datasets were collected via platform APIs and manually labeled in the original studies.

Metaheuristic research papers

In addition to machine learning methods, I explored a wide range of metaheuristic optimization algorithms inspired by different real-world systems and games.

These algorithms are often designed as search strategies that simulate natural or social behaviors, such as competition, cooperation, or exploration.

I studied the following approaches:

• Most Valuable Player Algorithm (MVPA)
• Multi-Objective Volleyball Premier League (MOVPL)
• Shell Game Optimization (SGO)
• Tug of War Optimization (TWO)
• Ring Toss Game Based Optimization
• Puzzle Optimization Algorithm (POA)
• Mixed Leader Based Optimizer (MLBO)
• Soccer-Inspired Metaheuristics
• Tiki-Taka Algorithm (TTA)
• Running City Game Optimizer (RCGO)
• Golf Optimization Algorithm (GOA)
• Hide Objects Game Optimization (HOGO)
• Football Game Based Optimization (FGBO)
• Billiards-inspired Optimization Algorithm (BOA)
• Kho-Kho Optimization Algorithm (KKO)
• Orientation Search Algorithm (OSA)
• Dice Game Optimizer (DGO)
• Binary Orientation Search Algorithm (BOSA)

These methods may look very different on the surface, but they all share a common goal: explore a search space efficiently to find near-optimal solutions

Methodology Overview

Once the problem and data were clear, I designed the following pipeline to structure the entire system:

• labeled input data is collected
• data is cleaned
• data is preprocessed
• feature selection is applied
• dataset is split into train/test (with different split strategies)
• models are trained and tested
• cross-validation is applied for more reliable evaluation
• hyperparameter optimization is performed
• best combination is selected:
  • best features
  • best algorithm
  • best split strategy
• final model is retrained using the best configuration
• final model is deployed

[Labeled Data]
      |
      v
[Cleaning & Preprocessing]
      |
      v
[Feature Selection]
      |
      v
[Train / Test + Cross Validation + Splits]
      |
      v
[Model Benchmarking]
      |
      v
[Best Features + Best Model + Best Split]
      |
      v
[Final Retraining]
      |
      v
[Deployment (Hugging Face)]

System Implementation

Step 1: Generic Metaheuristic Engine

I started by implementing metaheuristic algorithms in a generic way, so I could reuse the same implementation for feature selection, scoring, and machine learning evaluation.

During this step, you don’t really feel the progress, since it takes a lot of time (more than a month in my case because of the number of algorithms). But once this foundation is done, the progress on the next steps becomes much faster and smoother.

Step 2: Data preprocessing and cleaning

I started working on this step in parallel with the first step in order to save time later in the process. The data preprocessing and cleaning phase was not identical for all datasets, as each one had its own structure, but in general it followed a common pipeline.

First, the usual preprocessing steps were applied, such as handling missing values and removing empty or irrelevant entries.

Then, for datasets containing raw profile names, I tried to extract as much useful information as possible. This included encoding the text and adding additional engineered features such as:

• number of characters in the profile name
• number of words in the profile name

The idea was that even simple textual patterns could carry useful signals for fake profile detection.

Finally, all features were normalized so that each attribute would have the same scale and contribute fairly to the learning process, avoiding bias toward features with larger numerical ranges.

Step 3: Feature selection

• Variance Threshold (0.7)
• SelectKBest
• RFE (Recursive Feature Elimination)
• Sequential Feature Selection
• Mutual Information (MI) (combined with all implemented metaheuristics)
• Gini Index (combined with all implemented metaheuristics)
• Fisher Score (combined with all implemented metaheuristics)

In addition to these classical feature selection methods, I also integrated metaheuristic optimization techniques into the feature selection process.

The main idea was to go beyond standard deterministic selection methods and explore the feature space in a more flexible and intelligent way, allowing the model to discover feature combinations that traditional approaches might miss.

To make this exploration efficient and controlled, I designed a custom stopping strategy:

• The optimization is executed in blocks of 100 iterations
• After each block, I keep the best solution found so far
• Then I launch the next block of 100 iterations

Now comes the stopping logic:

• If the solution does not improve for 3 consecutive blocks, the process stops and the best solution is kept
• Otherwise, it continues until a maximum of 10 blocks, after which the best overall solution is selected

In short, this acts as a controlled exploration mechanism: the algorithm is given multiple opportunities to improve, but it is stopped early when it clearly starts to stagnate, avoiding unnecessary computation while preserving performance.

Step 4: Machine Learning

Step 4.1: Algorithms

I started this step by identifying the most performing algorithms for each dataset.

The idea was simple: before applying feature selection and optimization, I wanted to evaluate all models using the full feature space. This allowed me to understand which algorithms naturally perform better depending on the dataset, so I could later combine the strongest models with the most relevant features.

At the same time, I explored different hyperparameter configurations for each algorithm to ensure a fair and robust comparison.

• Random forest

n_estimators
max_depth
min_samples_split
min_samples_leaf
criterion
max_features

• Logistic regression

C
penalty
solver
class_weight
max_iter

• Naive bayes

alpha
fit_prior
class_prior

• Linear SVC

C
penalty
loss
dual
tol
class_weight
max_iter

• XGBoost

max_depth
learning_rate
n_estimators
subsample
colsample_bytree
scale_pos_weight

• Deep Forest (Cascade forest)

n_estimators
max_depth
min_samples_split
min_samples_leaf

This initial benchmarking step helped me identify strong baseline models, which were later refined through feature selection and optimization.

Step 4.2: Splits

We often encounter the famous 80/20 split, where 80% of the data is used for training and 20% for testing. But how can we be sure that this split is the most optimal one?

I decided to go deeper and test multiple splits, almost like a detective: I tried 50/50, 60/40, 70/30, the classic 80/20, and finally 90/10.

Is there a difference? Yes, they tend to give different results. Is 80/20 always the most optimal one?
The answer is not always, as we will see in the results section just a bit later.

Step 4.3: Validation Strategy

To make the evaluation more reliable, I used cross-validation.

The idea is simple. Let’s take the example of an 80/20 split:

• Instead of doing a single split (80% training / 20% testing)
• The dataset is divided into multiple parts (folds)

For example, with 5-fold cross-validation:

1. The dataset is split into 5 equal parts
2. At each iteration:
   • 4 parts (≈80%) are used for training
   • 1 part (≈20%) is used for testing
3. This process is repeated 5 times, each time changing the test part

So every sample:

• is used for training multiple times
• is used for testing at least once

How is the data divided?

The data is divided into k equal subsets, called folds.

let’s take the example of 5 folds

Fold 1 → Test | Train Train Train Train  
Fold 2 → Train | Test Train Train Train  
Fold 3 → Train | Train | Test Train Train  
Fold 4 → Train | Train Train | Test Train  
Fold 5 → Train | Train Train Train | Test

Why I did this step

At first, it might look like an unnecessary step.
Why not just split the data once and train/test directly?

The problem is that a single split can be biased. Some samples may be easier or harder than others, which can lead to misleading results.

With cross-validation:

• the model is evaluated on different subsets of the data
• the results become more stable and reliable
• and the evaluation better reflects the model’s ability to generalize

In other words, instead of depending on one “lucky” split, this approach gives a more robust estimation of performance across the entire dataset.

Step 4.4: Metrics & Evaluation Strategy

What are all the metrics i recorded To evaluate model performance, I recorded a wide range of metrics for two main reasons:

1. To ensure that everything was working correctly (no missing steps, no overfitting, and stable training behavior)
2. To guarantee a fair comparison with other research papers, since not all of them report the same metrics—even though I focused my optimization on precision

The metrics I recorded were:

• Accuracy
• Precision (precision_score)
• Recall (recall_score)
• F1-score (f1_score)
• Cohen’s Kappa (cohen_kappa_score)
• Balanced Accuracy (balanced_accuracy_score)
• Confusion Matrix
• Best Hyperparameters
• Training Time (s)
• Testing Time (s)
• Validation Time (s)

This combination allowed me to evaluate the models from multiple perspectives: performance quality, robustness, and computational cost.

Why precision

At first glance, one might ask: why choose precision instead of accuracy or F1-score? After all, accuracy gives a global view of performance, and F1-score balances precision and recall.

However, in the context of fake profile detection, the cost of errors is not symmetric.

If a fake profile is classified as real, the consequences can be serious:

• users may trust malicious accounts
• misinformation can spread more easily
• in some cases (e-commerce, social engineering), it can lead to real harm

This is why precision becomes the most critical metric in this context. It directly measures how many predicted “real profiles” are actually real. In other words, the higher the precision, the fewer fake profiles are incorrectly labeled as trustworthy.

So instead of optimizing for general correctness, I focused on minimizing this specific and risky type of error.

Step 4.5: Hyperparameter Optimization

I experimented with several optimization approaches, including well-known ones such as:

• Grid Search
• Randomized Search
• Bayesian Optimization

But I didn’t stop there. I also used metaheuristics with a custom stopping condition (the one we explained in the feature selection).

A brief recap of it’s details:

• I run the optimization in blocks of 100 iterations
• After each block, I keep the best solution found
• Then I launch another block of 100 iterations

Now comes the stopping logic:

• If the solution does not improve for 3 consecutive blocks, I stop and keep the best result
• Otherwise, I continue until I reach a maximum of 10 blocks, and then select the best solution overall

So in short, it’s like a controlled exploration: I give the algorithm multiple chances to improve, but I stop early if it clearly stagnates.

The stopping condition was based on precision not improving we just explained the why above.

Step 4.6: Recap

so here is the recap of all the training part

                    ┌────────────────────────────┐
                    │   Full Feature Dataset     │
                    └─────────────┬──────────────┘
                                  │
                                  v
                    ┌────────────────────────────┐
                    │  Model Benchmarking        │
                    │  (Multiple Algorithms)     │
                    │  RF / LR / SVC / XGB / NB  │
                    │  Deep Forest               │
                    └─────────────┬──────────────┘
                                  │
                                  v
                    ┌────────────────────────────┐
                    │ Hyperparameter Tuning      │
                    │ (Grid / Random / Bayesian  │
                    │ + Metaheuristics)          │
                    └─────────────┬──────────────┘
                                  │
                                  v
                    ┌────────────────────────────┐
                    │ Evaluation Strategy        │
                    │ - Train/Test Splits        │
                    │ - Multiple split ratios    │
                    │ - Cross Validation (k-fold)│
                    └─────────────┬──────────────┘
                                  │
                                  v
                    ┌────────────────────────────┐
                    │ Metrics Computation        │
                    │ Acc / Prec / Rec / F1      │
                    │ Kappa / Balanced Acc       │
                    │ Confusion Matrix           │
                    │ Time metrics               │
                    └─────────────┬──────────────┘
                                  │
                                  v
                    ┌────────────────────────────┐
                    │ Best Model Selection       │
                    │ + Best Hyperparameters     │
                    │ + Best Split Strategy      │
                    └────────────────────────────┘

Step 5: Feature scoring

You might ask the question: if I tested many approaches, did I use all of them for the final selection?

The answer is no. But then how did I get the final best feature set?

I followed a simple approach: I scored the features. At the beginning, all features have a score of 0. Every time a feature is selected by an algorithm, its score is incremented by 1.

At the end of all iterations, each feature has a final score. I simply sorted the features based on this score, and I got the overall best feature set.

Step 6: Retraining with the best feature and best algorithm

Now that I had identified both the best feature set and the best performing algorithm, the next step was to combine them into a final optimized model.

To build the most efficient version of the system, I retrained the model using only the top-ranked features, selected through the scoring method. In my case, I chose the top 5 features, which allowed me to reduce complexity while keeping strong predictive power.

Then, I retrained the best-performing algorithm using this reduced feature set. This step was important because it:

• ensures the model is trained only on the most informative signals
• reduces noise coming from less relevant features
• improves generalization and stability
• makes the final model lighter and faster to evaluate

As a result, I obtained a final model that is not only high-performing, but also simpler and easier to test and deploy in practice.

Results

Dataset	Accuracy	Precision	F1-score
Facebook	0.9880	0.9953	0.9953
Twitter	0.9200	0.9078	0.9212
Instagram (Dataset 1)	0.9378	0.9706	0.9333
Instagram (Dataset 2)	0.9378	0.9381	0.9333
Instagram (Dataset 3)	0.9880	0.9953	0.9953

Discussion

For sure, the “everything law” still doesn’t exist, as well as a universal best model in machine learning. However, I noticed some patterns in social media labeled datasets: tree-based algorithms often give the best results (like XGBoost and Random Forest), and in many cases, grid search is one of the most reliable methods for hyperparameter optimization.

Deployment

I deployed the final model on Hugging Face, choosing the best-performing version trained with the full feature set (I deployed the best of each platform which means one for facebook, one for X and one for Instagram).

The goal of this step was to make the model accessible and easy to test in a real-world setting, without requiring any local setup or complex dependencies.

By deploying the complete model (with all features), I ensured that:

• no information loss occurs due to feature reduction
• users can directly test realistic predictions
• the pipeline remains consistent from training to deployment

This step marked the transition from a research-oriented system to a usable and interactive machine learning application.

Improvements for Future Work

• I initially created separate Google Colab notebooks for each dataset, which led to some code repetition. I started with one dataset and then replicated the workflow for the others. In hindsight, I would design a shared module from the beginning to make the process more scalable and reusable.
• I also started the experiments on Google Colab, but long training times and occasional disconnections made it less convenient. I later moved to a local Jupyter Notebook setup, and if I were to redo it, I would start directly with a local environment and GitHub for better version control and stability.

Conclusion

This project investigated fake profile detection across Facebook, Twitter, and Instagram using a full experimental machine learning pipeline. It combined multiple datasets, several classification algorithms, diverse feature selection techniques, and different optimization strategies to evaluate model performance in a systematic and comparable way.

A wide range of metrics was used, with a particular focus on precision due to the high cost of false positives in real-world scenarios. Cross-validation and multiple train/test splits were applied to ensure robust and reliable evaluation across datasets.

The results show that well-tuned ensemble methods such as Random Forest and Gradient Boosting achieve strong performance across platforms (in 4 out of 5 datasets). Metaheuristic-based feature selection sometimes provided some improvements, but did not consistently outperform classical methods (in 3 of the 5 datasets we tried the classical methods outperformed and were less time consuming), highlighting that sometimes simple strategies give better results.

Overall, this work demonstrates that robust fake profile detection can be effectively achieved through a well-designed classical ML pipeline combined with rigorous experimental evaluation across multiple datasets.