AI in InfoSec: 86.6% Accuracy, Zero Attacks Caught

Five topics in and I've been building up theory. This one I actually ran. I generated two thousand rows of network logs, deliberately made a mess of them, then tried to train something that could spot the dodgy traffic. I expected to spend the evening picking an algorithm. I spent it fixing a CSV.

That turned out to be the lesson. The model bit is about eight lines. Everything before it, working out which rows are lying to you, is the job. And the bit that genuinely caught me out came at the end: my first score was 86.6% accurate and it had found precisely nothing.

Everything below runs on a laptop, in one folder, with two libraries. You can follow along and get your own numbers.

Two libraries, and which one you actually need

Python owns this space for a boring reason: the libraries are good and they all speak the same data formats. For our purposes there are two that matter.

My honest take after a few evenings of this: people reach for the neural network far too early. On tabular security data a random forest or a plain logistic regression will usually match a small network, train in a second, and let you explain to a human why it flagged something. That last part matters more in security than it does anywhere else.

So start with scikit-learn, and reach for PyTorch when the data stops being rows and columns.

The lab, in one file

You need a messy dataset to practise on, and real logs you can publish are hard to come by. So make one. This writes 2,000 network events with an honest amount of rubbish sprinkled through it.

pip install pandas numpy scikit-learn

Now the generator. Each row is one network event: who sent it, where it went, which protocol, how many bytes, and a threat level from 0 (normal) to 2 (nasty).

import numpy as np, pandas as pd
rng = np.random.default_rng(1337)

protos = ['TCP','TLS','SSH','POP3','DNS','HTTPS','SMTP','FTP','UDP','HTTP']
port_for = {'TCP':445,'TLS':443,'SSH':22,'POP3':110,'DNS':53,
            'HTTPS':443,'SMTP':25,'FTP':21,'UDP':161,'HTTP':80}

rows = []
for i in range(1, 2001):
    p = rng.choice(protos, p=[.10,.10,.08,.04,.12,.22,.06,.04,.10,.14])
    b = float(rng.lognormal(mean=7.5, sigma=1.4))   # bytes, heavily skewed
    if   p in ('SSH','FTP') and b > 4000: t = 2     # big transfer over admin protocol
    elif p == 'DNS' and b > 9000:         t = 2     # looks like DNS tunnelling
    elif p in ('SSH','FTP','SMTP') and b > 1200: t = 1
    elif b > 30000:                       t = 1
    else:                                 t = 0
    ip = f"{rng.integers(10,220)}.{rng.integers(0,255)}.{rng.integers(0,255)}.{rng.integers(1,254)}"
    rows.append([i, ip, port_for[p], p, int(b), t])

df = pd.DataFrame(rows, columns=['log_id','source_ip','destination_port',
                                 'protocol','bytes_transferred','threat_level'])
df = df.astype(object)

def corrupt(col, values, n):                        # break it on purpose
    idx = rng.choice(df.index, size=n, replace=False)
    df.loc[idx, col] = rng.choice(values, size=n)

corrupt('source_ip',         ['MISSING_IP','INVALID_IP','999.14.2.1'], 104)
corrupt('destination_port',  ['STRING_PORT','UNUSED_PORT','70000'],     96)
corrupt('protocol',          ['UNKNOWN','ICMPv9'],                      61)
corrupt('bytes_transferred', ['NON_NUMERIC','-1'],                      77)
corrupt('threat_level',      ['?','9'],                                 68)

df.to_csv('demo_logs.csv', index=False)

The corruption isn't me being cute. Every one of those values is something I've seen come out of a real log pipeline: a sensor that writes a placeholder when it can't resolve a field, a port column that sometimes holds a service name, an analyst who typed ? because they weren't sure. Real data is worse than this, honestly.

Look at it before you touch it

What this step is: reading the shape of the data before you decide anything about it.

Load it into a pandas DataFrame, which is just a table in memory with named columns, and ask it three questions.

import pandas as pd

data = pd.read_csv("demo_logs.csv")
data.info()                  # columns, types, how many non-null
print(data.isnull().sum())   # missing values per column

This is the bit that got me, and it's why I'm labouring it. isnull() reports nothing missing, because nothing is missing. Every cell has a value. It's just that some of those values are the literal string MISSING_IP, which pandas is perfectly happy to store.

The tell is in the Dtype column. destination_port and bytes_transferred should be integers. They came in as object, which is pandas saying "there's text in here somewhere". One bad string in a column of 2,000 numbers turns the whole column into text.

Missing is easy. Wrong is the problem.

Missing data announces itself. Wrong data sits there looking like data. So write a small check per column that answers "could this value actually exist?" and see what falls out.

import re

IP_RE = re.compile(r'^((25[0-5]|2[0-4]\d|[01]?\d?\d)\.){3}(25[0-5]|2[0-4]\d|[01]?\d?\d)$')
VALID_PROTOCOLS = ['TCP','TLS','SSH','POP3','DNS','HTTPS','SMTP','FTP','UDP','HTTP']

def bad_ip(v):
    return not bool(IP_RE.match(str(v)))

def bad_port(v):
    try:    return not (0 <= int(v) <= 65535)
    except (TypeError, ValueError): return True

def bad_bytes(v):
    try:    return int(v) < 0
    except (TypeError, ValueError): return True

def bad_threat(v):
    try:    return not (0 <= int(v) <= 2)
    except (TypeError, ValueError): return True

bad = {
    "source_ip":         data[data['source_ip'].apply(bad_ip)],
    "destination_port":  data[data['destination_port'].apply(bad_port)],
    "protocol":          data[~data['protocol'].isin(VALID_PROTOCOLS)],
    "bytes_transferred": data[data['bytes_transferred'].apply(bad_bytes)],
    "threat_level":      data[data['threat_level'].apply(bad_threat)],
}
for col, rows in bad.items():
    print(f"{col:<18} {len(rows):>4} bad rows")

Each check is deliberately dull. The IP one is a regex that only matches four numbers of 0 to 255. The port one tries to cast to an integer and asks whether it lands in 0 to 65535, so STRING_PORT fails on the cast and 70000 fails on the range. Protocol is checked against a list you control. Here's what came back.

That gap between 406 and 376 is the thing to notice. A row with one bad cell is a bad row. Corruption spread thinly across five columns costs you far more than the same amount concentrated in one.

Drop it, or fix it

Two options, and they're not equal.

The first move for either path is the same: turn every flavour of junk into a single, consistent "nothing here" marker, NaN. Once the mess is standardised, everything downstream gets simple.

import numpy as np

JUNK = ['MISSING_IP','INVALID_IP','STRING_PORT','UNUSED_PORT',
        'NON_NUMERIC','?','UNKNOWN','ICMPv9','9']
df = df.replace(JUNK, np.nan)

# force the numeric columns to be numeric; anything that won't cast becomes NaN
for c in ['destination_port','bytes_transferred','threat_level']:
    df[c] = pd.to_numeric(df[c], errors='coerce')

# and catch the values that are the right type but still impossible
df['source_ip'] = df['source_ip'].where(df['source_ip'].astype(str).str.match(IP_RE))
df.loc[(df['destination_port'] < 0) | (df['destination_port'] > 65535), 'destination_port'] = np.nan
df.loc[df['bytes_transferred'] < 0, 'bytes_transferred'] = np.nan
df.loc[~df['threat_level'].isin([0,1,2]), 'threat_level'] = np.nan

errors='coerce' is the useful flag there. It says "try to make this a number, and if you can't, put a NaN in and carry on" instead of throwing. Now isnull().sum() finally tells the truth: 104, 96, 61, 77, 68.

Then fill the gaps. Median for the numbers because it ignores extreme values, most common value for the categorical column, and a sentinel for the IP.

df['destination_port']  = df['destination_port'].fillna(df['destination_port'].median())
df['bytes_transferred'] = df['bytes_transferred'].fillna(df['bytes_transferred'].median())
df['protocol']          = df['protocol'].fillna(df['protocol'].mode()[0])
df['source_ip']         = df['source_ip'].fillna('0.0.0.0')

# the label is different. Never guess it.
df = df.dropna(subset=['threat_level'])
df['threat_level'] = df['threat_level'].astype(int)

If you use scikit-learn's helper instead of pandas, the shape is the same:

from sklearn.impute import SimpleImputer

num_imputer = SimpleImputer(strategy='median')
df[['destination_port','bytes_transferred']] = num_imputer.fit_transform(
    df[['destination_port','bytes_transferred']])

There's also KNNImputer, which fills a hole by looking at the most similar rows rather than the whole column. It's smarter and slower. On a column like destination_port, where the value is basically decided by the protocol, it's a genuinely better guess. I've not used it enough to tell you when it stops being worth the compute.

The order that works: standardise the junk to NaN, impute the features, then drop the rows with no label.

Make it model-ready

Clean isn't the same as usable. Two more things have to happen before a model can read this.

Protocol is text, and models can't read

What one-hot encoding is: turning one text column into a set of yes/no columns, one per possible value.

The lazy alternative is numbering the categories: TCP is 1, TLS is 2, SSH is 3. Do that and the model quietly learns that SSH is three times TCP, and that TLS sits between them, which is nonsense. One-hot avoids it by giving every protocol its own column holding 1 or 0.

encoded = pd.get_dummies(df['protocol'], prefix='protocol').astype(int)
X = pd.concat([df[['destination_port','bytes_transferred']], encoded], axis=1)
y = df['threat_level']

Two columns in, twelve columns out: the two numeric ones plus ten protocol flags. That's the trade. One-hot is clean but it grows your data sideways, and on a column with thousands of distinct values (source IPs, user agents) it will bury you. That's where hashing or frequency encoding comes in, and it's on my list to actually try.

Scikit-learn's version, if you want it inside a pipeline:

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder(handle_unknown='ignore', sparse_output=False)
encoded = encoder.fit_transform(df[['protocol']])

handle_unknown='ignore' matters more than it looks. It's what stops your model exploding in production the first time a protocol appears that wasn't in the training data. Which, in security, happens constantly.

One column was drowning out everything else

What skew is: most values bunched at one end, a handful stretching miles off to the other.

bytes_transferred in my run: median 1,829 bytes, maximum 270,107. Half the rows sit under two kilobytes and one row is a quarter of a megabyte. That's normal for network data and it's poison for a model, because the huge values dominate the maths and the differences between all the small rows get flattened to nothing.

The fix is one line. log1p takes the logarithm after adding 1, which compresses the big values hard and the small values barely at all. The 1 is there so a row with 0 bytes doesn't blow up, since log(0) is undefined.

df['bytes_transferred'] = np.log1p(df['bytes_transferred'])

Nothing is thrown away here, which is worth saying because it surprised me. The ordering is untouched: the biggest row is still the biggest row. You've only changed the spacing so the model can see the small differences that were being squashed.

Scaling is the related habit. StandardScaler shifts every column to a mean of 0 and a spread of 1 so that a column measured in bytes doesn't outweigh a column measured in port numbers purely because the digits are bigger. MinMaxScaler squeezes everything into 0 to 1. RobustScaler does the same job as StandardScaler but ignores extreme values while working out the middle, which is handy when you've got outliers you want to keep.

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)   # fit ONLY on training data
X_test  = scaler.transform(X_test)        # then apply the same numbers

Split it three ways, and stop touching one of them

Three sets, three jobs.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=1337, stratify=y)

X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=1337, stratify=y_train)

The 0.25 on the second split looks wrong until you work it out. It's 25% of the 80% you had left, which is 20% of the original. So you end up with 60 / 20 / 20.

stratify=y is not optional on data like this. It forces every split to keep the same class proportions as the full set. Without it, with attacks at roughly 5% of rows, a bad shuffle can hand your test set almost no attacks at all and your score becomes meaningless. My split came out with 1,158 training rows, 387 validation and 387 test, and the test set held 335 normal events, 32 low threat and 20 high threat.

Train the thing

After all that, this is the part everyone thinks is machine learning.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=2000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Three lines. Create, fit, predict. Swap LogisticRegression for RandomForestClassifier or SVC and those same three lines still work, which is the single best thing about scikit-learn's design. Trying a different algorithm costs you one word.

Before you trust the number, cross-validation is worth a mention. Instead of one train/test split it cuts the data into five folds, trains five times, and reports five scores. If those five scores are all over the place, your single split was luck.

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X_train, y_train, cv=5)
print(scores.mean(), scores.std())

Accuracy is a liar

Here's where the evening got interesting. My class balance after cleaning was 1,673 normal, 161 low threat, 98 high threat. So attacks are about 13% of the data, and in a real network they'd be a fraction of that.

Now build the laziest possible model: one that ignores its input entirely and says "normal" every single time.

86.56% would look perfectly respectable on a slide. It caught nothing. That's not a quirk of my toy dataset, it's the defining property of security data: the thing you care about is rare, and any metric that rewards getting the common case right will let a useless model through.

The four numbers, in plain words

Say the model flags something as an attack.

• Accuracy is how often it's right overall. Correct calls divided by all calls. Fine when your classes are balanced. Actively misleading when they aren't.
• Precision is how often a flag is real. Of everything it called an attack, what proportion actually was? Low precision means your analysts spend their week on false alarms.
• Recall is how much it found. Of all the real attacks, what proportion did it catch? Low recall means things walked past you and nobody noticed.
• F1 combines precision and recall into one number, leaning towards the worse of the two. It's what you quote when you need a single figure and you don't want to be able to cheat it.

The lazy model's precision on attacks is undefined and its recall is 0, because it never flagged anything. F1 collapses to 0. That's the point of F1: you can't game it by being cautious.

Now the trained model, same data, same split.

Look at the accuracy row on its own and the trained model is a modest improvement. Look at recall and it's the difference between a security control and a decoration. Ten points of accuracy hides the fact that one of these finds 44 attacks and the other finds none.

The word macro is doing work there too. Macro F1 averages the three classes equally, so the tiny high-threat class counts as much as the huge normal one. Weighted averaging counts each class by its size, which drags the score straight back towards the majority and hides the same problem accuracy does. On security data, quote macro.

The confusion matrix, which is the one I'd actually look at

Six false alarms out of 335 normal events, so analysts aren't drowning. Three real threats missed. And that bottom-left 1 is the one I'd be chasing: a high-threat event the model called completely normal. Not "probably fine", not downgraded to low. Normal. In an alerting pipeline that's the one that ends up in an incident report.

That's why the matrix beats the summary numbers. F1 gives you 0.8983. The matrix tells you which mistake you're making and lets you go and read those rows.

Two more you'll meet: AUC measures how well the model separates classes across every possible threshold, which is useful when you're deciding how trigger-happy to make an alert. Matthews correlation coefficient is the one to reach for when classes are badly imbalanced, because it only scores well if the model does well on both sides.

The PyTorch version, and whether it's worth it

Same problem in PyTorch, so you can see the difference in effort. If you've read Topic 4 this will look familiar, because it's the same loop written out by hand.

PyTorch works on tensors, which are arrays that can live on a GPU and remember how they were calculated so gradients can flow back through them.

import torch
import torch.nn as nn

X_t = torch.tensor(X_train, dtype=torch.float32)
y_t = torch.tensor(y_train, dtype=torch.long)

if torch.cuda.is_available():
    X_t, y_t = X_t.to('cuda'), y_t.to('cuda')

For a stack of layers with nothing clever going on, Sequential is enough:

model = nn.Sequential(
    nn.Linear(12, 32),   # 12 features in (2 numeric + 10 protocol flags)
    nn.ReLU(),
    nn.Linear(32, 3),    # 3 threat levels out
)

The moment you want branches, shared layers or anything conditional, you subclass nn.Module instead and write forward() yourself:

class ThreatNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(12, 32)
        self.fc2 = nn.Linear(32, 3)
        self.relu = nn.ReLU()

    def forward(self, x):
        return self.fc2(self.relu(self.fc1(x)))

model = ThreatNet()

Then the loop. This is the whole of training, and it's five steps repeated:

loss_fn   = nn.CrossEntropyLoss()
optimiser = torch.optim.Adam(model.parameters(), lr=0.001)

for epoch in range(200):
    y_hat = model(X_t)              # 1. forward: guess
    loss  = loss_fn(y_hat, y_t)     # 2. how wrong were we
    optimiser.zero_grad()           # 3. clear last round's gradients
    loss.backward()                 # 4. work out which way each weight should move
    optimiser.step()                # 5. move them

    if epoch % 50 == 0:
        print(f"epoch {epoch:>3}  loss {loss.item():.4f}")

Notes on that, in the order they caught me out. zero_grad() is not optional: PyTorch adds new gradients to whatever's already there, so skip it and you're training on a running total of every previous batch. CrossEntropyLoss expects raw scores, not softmax output, because it applies softmax internally. Put a Softmax layer at the end of your model and you've applied it twice, which trains badly and silently. And Adam is the sensible default optimiser: it adapts the step size per weight, where plain SGD uses one rate for everything.

For anything bigger than fits in memory, Dataset and DataLoader handle batching and shuffling:

from torch.utils.data import TensorDataset, DataLoader

loader = DataLoader(TensorDataset(X_t, y_t), batch_size=32, shuffle=True)
for x_batch, y_batch in loader:
    ...

And saving, which has one gotcha worth knowing:

torch.save(model.state_dict(), 'threat_model.pth')

model = ThreatNet()                                    # rebuild the same shape first
model.load_state_dict(torch.load('threat_model.pth'))
model.eval()                                           # switch off dropout / batchnorm training behaviour

state_dict() saves the weights, not the class. You need the code that defines ThreatNet to load it back, which is a good thing: it means loading a model file doesn't execute somebody's arbitrary Python. If you save the whole model object instead, it does. Worth knowing before you download a .pth off the internet.

Now the honest bit. On this dataset, all that gets you roughly what the three-line logistic regression got. Twelve features and 1,932 rows is not a job for a neural network. I wrote it out because the mechanics are worth having in your hands, not because it's the right tool here.

What this is and isn't, for security work

I want to be careful here, because "AI for threat detection" is sold hard and I've just spent an evening finding out how thin the ground is.

What genuinely works: sorting and triaging things nobody has time to read. Ranking alerts so the likely-real ones surface first. Spotting a host whose traffic pattern has changed shape. Clustering similar events so an analyst reads one summary instead of four hundred lines. These are all "reduce the pile" problems and they suit a model well.

What I'd be sceptical of:

• Your labels are probably wrong. Mine were perfect because I generated them. Real threat labels come from alerts that were already triaged by an imperfect tool and an overworked human. The model learns whatever that process believed, mistakes included.
• Networks drift. A model trained on last quarter's traffic gets worse every week as services change. Nothing tells you it's happening except your recall quietly sliding.
• The adversary is not a fixed distribution. Every other machine learning problem has data that doesn't care what your model does. Security has an opponent who will happily reshape their traffic once they work out where your boundary is.
• A number you can't explain is hard to act on. "The model says 0.83" is a difficult thing to put in front of a client. This is why I'd take a decision tree over a network for anything I have to defend in a meeting.

None of that makes it useless. It makes it a filter rather than a verdict, which is a perfectly good thing to be.

The ratio that surprised me

The ratio surprised me. Cleaning and shaping the data took maybe eighty per cent of the time. Choosing and training the model took minutes. Every course teaches it the other way round, and I now think that's backwards.

The other thing I'll carry: never quote accuracy on security data without recall next to it. 86.6% and zero attacks found is going to stick with me for a while.

Next up is the part I've been building towards since Topic 1: attacking these systems rather than building them. Prompt injection, model extraction, poisoning the training data. If you've done AI security testing properly and I'm about to walk into something obvious, tell me now and save me a fortnight. My AI testing notes are where that work will land as I go.

FAQ

Why is accuracy a bad metric for a security model?

Attacks are rare, so a model that labels everything normal still scores well. On my 2,000 row log set that lazy model hit 86.6% accuracy and caught zero of the 52 attacks in the test split. Accuracy rewards the majority class. Recall on the rare class is what tells you whether it found anything.

What is the difference between scikit-learn and PyTorch?

Scikit-learn covers classic machine learning on tabular data: regressions, trees, forests, clustering, plus the cleaning and scoring tools around them. PyTorch builds neural networks, runs on a GPU and gives you the training loop to write yourself. For log rows in a CSV, start with scikit-learn.

Should you drop or impute bad rows in a security dataset?

Impute the features, drop the label. Filling a missing port with the median costs you very little. Guessing a missing threat level teaches the model your guess instead of the truth. On my run, dropping every dirty row lost 376 of 2,000 rows. Imputing kept 1,932.

What does one-hot encoding do to a protocol column?

It replaces one text column with a set of 0 or 1 columns, one per protocol. HTTPS becomes protocol_HTTPS = 1 and every other protocol column 0. It stops the model reading TCP as smaller than UDP, which is what happens if you just number the categories.

Why split data into training, validation and test sets?

The training set fits the model. The validation set is where you compare settings and pick a winner. The test set is touched once, at the very end. If you tune against the test set you are slowly memorising it, and the score you report stops meaning anything.

Related reading

• Supervised Learning Algorithms (Topic 2, the algorithms behind that three-line fit())
• Neural Networks & Backpropagation (Topic 4, the loop the PyTorch section writes out by hand)
• Generative AI, LLMs & Diffusion (Topic 5, what happens when you point all this at making things)
• Browse the whole AI / LLMs track