Random Forest Hit 99.7% and Still Missed the Rare Attack

My model hit 99.7% weighted accuracy on the test set. I nearly closed the notebook there. Then I looked at the one class that actually matters most and it had missed almost 40% of it.

That gap is the entire post. I built a Random Forest to classify network traffic into normal activity and four attack families, using an NSL-KDD-style dataset, and the numbers below are honest ones from my own run, not the textbook figures. The good news came easily. The bad news took a confusion matrix to find, and it's the more useful of the two.

What a Random Forest actually is

A Random Forest is a pile of decision trees that vote, instead of you trusting any single tree.

A single decision tree is a flowchart you could draw by hand: is src_bytes under 60? Is the error rate over 0.8? Follow the branches down to a leaf and that leaf gives you an answer. Trees are easy to read and easy to overtrain. Give one tree the full dataset and it will happily memorise the noise along with the pattern, then fall apart on anything new.

A forest fixes that by building hundreds of trees that each see a different version of the truth, then averaging their votes.

The row sampling has a name, bootstrapping: draw as many rows as the dataset has, but with replacement, so each tree's training set is a slightly different reshuffle with some rows repeated and others missing. Combine that with only offering each split a random handful of features (scikit-learn's default is roughly the square root of the total, which is what I used) and no two trees end up alike. One tree overfitting to a quirk in the data gets outvoted by 99 trees that never saw that quirk.

That suits this problem, because network traffic is messy and high-dimensional: forty-odd features per connection, some numeric, some categorical, most of them correlated with each other. A forest handles that kind of data without much preparation, and it doesn't demand the features be scaled or normally distributed the way some algorithms do.

Many biased guesses, averaged, beat one guess that tried to be perfect. It's the same intuition behind asking five colleagues to review a finding instead of trusting your own first read.

Using it for anomaly detection, specifically

There are two ways to point a Random Forest at security data, and it's worth being precise about which one this post does.

I mention this because "anomaly detection" gets used loosely, and the two approaches fail differently. A classifier trained on known attacks will confidently miss a brand new attack family, because it has literally never seen anything like it and has no notion of "unfamiliar, therefore suspicious". That's worth knowing before you deploy either one on faith.

The dataset, and being straight about mine

NSL-KDD is a cleaned-up successor to the 1999 KDD Cup dataset: duplicate rows removed, class balance improved, and it's the standard benchmark this exact lab exercise gets taught on. Each row is one network connection with 41 features (protocol, service, byte counts, error rates, login attempts, and a set of statistics about recent connections to the same host) plus a label naming the traffic as normal or one of dozens of specific attacks.

Here's an honest complication: I went to fetch the dataset and hit a dead end that has nothing to do with my setup. UNB's own hosting page for it currently reads "we apologize, this dataset is no longer available" (mirrors exist on Kaggle, but I wasn't going to build a post's numbers around a copy I couldn't verify). Rather than write around figures I hadn't actually produced, I built a synthetic set in the same shape: the same 41 columns, the same five-way split (Normal, DoS, Probe, Privilege escalation, Access), and the same lopsided class sizes real intrusion datasets have. I checked that shape against what UNB publish about the real class balance, and small classes staying rare is exactly the pattern their own statistics show. I'd rather tell you that plainly than dress synthetic numbers up as the original dataset's.

# fetching the real dataset (the approach, if you have network access)
import requests, zipfile, io

url = "https://academy.hackthebox.com/storage/modules/292/KDD_dataset.zip"
response = requests.get(url)
z = zipfile.ZipFile(io.BytesIO(response.content))
z.extractall('.')

The dataset comes as a flat text file with no header row, so you supply the column names yourself:

columns = [
    'duration', 'protocol_type', 'service', 'flag', 'src_bytes', 'dst_bytes',
    'land', 'wrong_fragment', 'urgent', 'hot', 'num_failed_logins', 'logged_in',
    'num_compromised', 'root_shell', 'su_attempted', 'num_root', 'num_file_creations',
    'num_shells', 'num_access_files', 'num_outbound_cmds', 'is_host_login', 'is_guest_login',
    'count', 'srv_count', 'serror_rate', 'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate',
    'same_srv_rate', 'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count',
    'dst_host_same_srv_rate', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate',
    'dst_host_srv_diff_host_rate', 'dst_host_serror_rate', 'dst_host_srv_serror_rate',
    'dst_host_rerror_rate', 'dst_host_srv_rerror_rate', 'attack', 'level'
]
df = pd.read_csv('KDD+.txt', names=columns)

My run had 4,818 rows: 3,200 normal, 950 DoS, 420 probe, 210 access, and just 38 privilege escalation. Keep that last number in your head. It's the whole plot.

Two ways to label the same traffic

Binary is the simple version: was this connection normal, or was it an attack? One column, 0 or 1.

df['attack_flag'] = df['attack'].apply(lambda a: 0 if a == 'normal' else 1)

Multi-class keeps the attack type, grouped into four families used across most NSL-KDD research:

dos_attacks = ['apache2','back','land','neptune','mailbomb','pod',
               'processtable','smurf','teardrop','udpstorm','worm']
probe_attacks = ['ipsweep','mscan','nmap','portsweep','saint','satan']
privilege_attacks = ['buffer_overflow','loadmodule','perl','ps','rootkit','sqlattack','xterm']
access_attacks = ['ftp_write','guess_passwd','http_tunnel','imap','multihop','named',
                   'phf','sendmail','snmpgetattack','snmpguess','spy',
                   'warezclient','warezmaster','xclock','xsnoop']

def map_attack(attack):
    if attack in dos_attacks: return 1
    if attack in probe_attacks: return 2
    if attack in privilege_attacks: return 3
    if attack in access_attacks: return 4
    return 0

df['attack_map'] = df['attack'].apply(map_attack)

Multi-class is the more useful label for a defender. Knowing "this is an attack" tells you to look. Knowing "this is a probe" versus "this is privilege escalation" tells you how fast to move and who to page.

Getting it model-ready

Two feature types, two treatments. protocol_type and service are categorical text (tcp, http, ftp_data), so they get one-hot encoded exactly like the protocol column in Topic 6: a binary column per category, so the model never mistakes "udp" for being numerically bigger than "tcp".

features_to_encode = ['protocol_type', 'service']
encoded = pd.get_dummies(df[features_to_encode])

The rest are already numbers: byte counts, connection counts, and a set of rate features between 0 and 1 that summarise recent behaviour to the same host, like same_srv_rate (what fraction of recent connections used the same service) and serror_rate (what fraction came back with a connection error). Those rate features carry most of the signal, which will matter again later.

numeric_features = [
    'duration', 'src_bytes', 'dst_bytes', 'wrong_fragment', 'urgent', 'hot',
    'num_failed_logins', 'num_compromised', 'root_shell', 'su_attempted',
    'num_root', 'num_file_creations', 'num_shells', 'num_access_files',
    'num_outbound_cmds', 'count', 'srv_count', 'serror_rate',
    'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate', 'same_srv_rate',
    'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count',
    'dst_host_same_srv_rate', 'dst_host_diff_srv_rate',
    'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate',
    'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'dst_host_rerror_rate',
    'dst_host_srv_rerror_rate'
]
train_set = encoded.join(df[numeric_features])
multi_y = df['attack_map']

My run ended up with 49 feature columns after encoding: the categorical columns expand out, the numeric ones stay as they are.

Splitting it three ways, and keeping the classes honest

Same 60/20/20 shape as Topic 6: train, tune on a validation slice, touch the test set exactly once.

train_X, test_X, train_y, test_y = train_test_split(
    train_set, multi_y, test_size=0.2, random_state=1337, stratify=multi_y)

multi_train_X, multi_val_X, multi_train_y, multi_val_y = train_test_split(
    train_X, train_y, test_size=0.3, random_state=1337, stratify=train_y)

I added stratify=multi_y, which the original lab exercise doesn't use, and on a dataset with a 38-row class it isn't optional. Without it, a plain random split can hand your test set six privilege escalation rows or none at all, and every metric you compute afterwards is noise dressed up as a result. With stratification, each split keeps the same 3,200:950:420:210:38 proportions as the full set, just scaled down.

train 2697  val 1157  test 964
train class counts: [1792  532  235   21  117]
val class counts:   [768  228  101    9   51]
test class counts:  [640  190   84    8   42]

Twenty-one privilege escalation rows to learn from, in the whole training set. Hold that thought.

Training

from sklearn.ensemble import RandomForestClassifier

rf_model_multi = RandomForestClassifier(random_state=1337)
rf_model_multi.fit(multi_train_X, multi_train_y)

That's the whole call. random_state pins the random bootstrapping and feature selection so your run is reproducible, which matters more than it sounds like it should: without it, re-running the exact same code gives you a slightly different forest and slightly different numbers every time, which makes debugging painful.

The results that looked great

Look only at the weighted row and you'd sign off on this model without a second thought. 99.7% across the board is the kind of number that ends a meeting. It's also almost entirely a story about Normal, DoS and Probe, because those three classes alone make up 914 of the 964 test rows. The weighted average is doing exactly what its name says: weighting by how common each class is, and privilege escalation is not common.

Where the three misses actually went

A confusion matrix earns its keep exactly here, because a single F1 score can't tell you what the model confused. This one can.

test_conf_matrix = confusion_matrix(test_y, test_multi_predictions)
sns.heatmap(test_conf_matrix, annot=True, fmt='d', cmap='Blues',
            xticklabels=class_labels, yticklabels=class_labels)

Every single mistake in the entire test set is on that one row, and all three land in the same column: Access. That's not scattered confusion, it's a specific, explainable mix-up, and it makes sense once you think about what privilege escalation and unauthorised access actually look like on the wire. Both start from a session that's already logged in. Both involve a comparatively normal-looking connection with a handful of suspicious actions buried inside it, rather than a flood of obviously hostile traffic like DoS. With only 21 privilege examples to learn the difference from, the forest leaned on the wrong handful of tells for three of them.

Why the rare class loses, structurally

This isn't bad luck, and it isn't specific to Random Forest. It's what happens whenever one class is both rare and subtle, and it's worth having a mental model for, because "just add more data" is not always available to you on a real engagement.

DoS traffic is a firehose: hundreds of connections, sky-high error rates, a flag pattern that looks nothing like normal use. There's no shortage of examples and the signal is loud. Privilege escalation is one user, already authenticated, doing a handful of unusual things inside an otherwise ordinary-looking session. Quiet signal, and a tenth as much data to learn it from. Ask a model to find a needle when you've only shown it twenty-one needles to begin with, and some needles get missed. This is the exact same lesson as the accuracy-versus-recall problem in Topic 6, wearing a different outfit: the rarest thing is usually the thing you most needed to catch, and it's the thing your headline metric is worst at reporting on.

What the forest actually paid attention to

Random Forest gives you something a lot of models don't: a ranked list of which features it leaned on. I got mine by shuffling one feature at a time and watching how much accuracy dropped, which is the same idea as scikit-learn's built-in feature_importances_.

importances = pd.Series(rf_model_multi.feature_importances_, index=train_set.columns)
importances.sort_values(ascending=False).head(10)

The top two are both about repetition patterns to a destination host, not about any single connection's own byte count or duration. That tracks with how the four attack families actually behave: DoS and Probe both leave a fingerprint in "how many connections, how repetitive, to how many services", because they involve dozens or hundreds of connections happening close together. A single privilege escalation attempt doesn't touch those host-level statistics much at all, it's one session, so the features that dominate the forest's decisions are exactly the ones that class barely moves. That's a second, independent reason it's the hardest class here, not just the rarest.

What you'd actually do about it

Not "collect more data" and stop there, though that's the honest first answer if you can get it. A few things that don't require a bigger dataset:

• Class weighting. RandomForestClassifier(class_weight='balanced') tells the model to penalise a mistake on a rare class more heavily than a mistake on a common one, instead of treating every row equally.
• Look at recall per class, always, not just the weighted row. A one-line habit: print classification_report in full and actually read the small classes, every time.
• A lower decision threshold for the classes that matter most. Random Forest gives you predict_proba, not just a hard label. You can choose to flag "privilege escalation" on weaker evidence than you'd need for "probe", because a missed escalation is worse than an extra alert.
• Route by consequence, not by frequency. If privilege escalation is rare but severe, it earns tighter review even at 8 examples in your test set. Common and mild can tolerate a coarser net.

I haven't tried the class-weighting option against this exact dataset yet, and I'd genuinely like to see whether it closes the gap on the 38-row class or just moves the mistakes somewhere else. That's on my list for a follow-up.

Saving it, and the same caution as last time

import joblib

joblib.dump(rf_model_multi, 'network_anomaly_detection_model.joblib')

Same note as Topic 7: joblib is pickle underneath, so loading a model file runs code. Fine for something you trained yourself, not fine for a random download, and worth remembering the moment an IDS pipeline starts pulling "community" models from somewhere you don't control.

The number I nearly believed

I nearly stopped at the top-line number, and I think that's the most useful admission in this whole post. 99.7% weighted accuracy is a genuinely good-looking result, and it would have been an easy one to screenshot and move on from. It took deliberately going class by class to find the one number that mattered more than the headline.

What I'm still turning over is how much of this generalises past my synthetic set. Real privilege escalation traffic might carry a stronger or weaker signal than what I generated, and I won't know until I run this against the genuine NSL-KDD file. If you've trained on the real dataset and seen a similar precision/recall split on the rare classes, I'd like to compare notes.

Next post I'm switching from tables to images: training a CNN to sort malware by what its bytes look like as a picture.

References

• NSL-KDD dataset page, Canadian Institute for Cybersecurity (the dataset's home; direct download is currently unavailable from UNB, Kaggle carries mirrors)
• Tavallaee, Bagheri, Lu & Ghorbani, "A Detailed Analysis of the KDD CUP 99 Data Set" (2009) (the paper NSL-KDD comes from)
• scikit-learn: RandomForestClassifier

FAQ

How does a Random Forest detect network anomalies?

It trains many decision trees, each on a random bootstrap sample of the traffic with a random subset of features at every split, then lets them vote. A connection gets flagged when most trees agree its statistics (byte counts, error rates, connection counts) look more like an attack pattern than normal traffic.

What is the NSL-KDD dataset?

An improved version of the 1999 KDD Cup dataset, with duplicate records removed and better class balance, used to benchmark intrusion detection systems. Each row is a network connection with 41 features plus a label naming the traffic as normal or one of several named attacks.

Why is weighted accuracy misleading for intrusion detection?

It averages performance across classes in proportion to how common each one is. In my run the model missed 3 of 8 privilege escalation attempts, a 62.5% recall, while weighted accuracy still read 99.7%, because that rare class barely moves an average dominated by thousands of normal and DoS rows.

Why is privilege escalation the hardest class to detect?

It is rare in training data and, unlike a DoS flood, it looks statistically similar to a normal logged-in session: reasonable byte counts, a valid login, ordinary connection counts. The signal is a handful of features like root_shell and su_attempted, easy for the model to under-learn from very few examples.

What is feature importance in a Random Forest?

A score for how much each input feature contributes to the model's decisions, measured here by how much accuracy drops when that feature's values are shuffled. In my run dst_host_same_srv_rate and dst_host_srv_count mattered most, both describing how repetitive a connection's targets are.

Related reading

• AI in InfoSec, from Raw Logs to a Model (Topic 6, the accuracy-versus-recall problem this post revisits)
• Naive Bayes and How a Spam Filter Thinks (Topic 7, a simpler classifier with the opposite precision/recall priority)
• Supervised Learning Algorithms (Topic 2, decision trees and ensembles from first principles)
• Browse the whole AI / LLMs track