Malware Image Classification: Repacking Beats the Hash, Not the Texture

I didn't write a single line of malware for this post. I turned it into wallpaper instead, greyscale, twenty-four pixels a side, and asked a neural network to sort it by what it looked like. That sentence still sounds slightly mad to me and it's also a real, published technique with a decade of research behind it.

The short version: I built a small convolutional network from scratch, no PyTorch, no pretrained ResNet50, because this sandbox has no route to the internet to fetch either. It hit 95.65% weighted accuracy across five synthetic malware "families". Then I checked the one family with only 40 examples and found it was right on that class barely more often than a coin flip. Same story as Topic 6, Topic 7 and Topic 8, wearing yet another outfit, and I'm starting to think that's less of a coincidence and more of a rule.

What malware image classification actually is

The trick is to read a malware binary's raw bytes as pixel values instead of as instructions. Do that and a family of malware becomes a family of images that look alike.

This isn't a party trick. It comes from a 2011 paper by Nataraj, Karthikeyan, Jacob and Manjunath that noticed something useful: malware authors reuse code, packers and builder tools within a family, and that reuse shows up as a visual pattern once you lay the bytes out on a grid. A worm variant and its cousin, compiled from the same source with a couple of strings changed, produce images with the same stripes, blocks and blank regions in roughly the same places, even though a byte-for-byte signature match would fail the moment either sample is repacked.

How: take the raw bytes of the file, treat each byte (0 to 255) as a greyscale pixel intensity, and reshape that long strip of numbers into a 2D grid. A bigger file gets a wider image, following a size-to-width lookup table from the original research so the layout stays roughly proportional. No disassembly, no execution, no sandbox detonation required, you're reading the file as data, not running it.

That matters because a lot of evasion effort goes into breaking hash-based and pattern-based signatures. Repacking a binary changes its hash completely. What it does not necessarily change is the coarse visual layout of its sections, because the underlying code and resources are still largely the same, just wrapped differently. A model that learns texture rather than exact bytes has a shot at surviving that kind of superficial evasion.

Put another way, you're no longer asking "have I seen this exact file before". You're asking "does this file look like it came from the same mould as files I've already seen", and that's a much harder question to dodge by changing a few bytes.

What a CNN actually is

A convolutional neural network is built specifically to notice patterns that show up in small local patches of an image, wherever in the image they happen to be.

A plain neural network looks at every pixel independently and has to relearn "this looks like a stripe" separately for every possible position in the image, which is wasteful and needs a lot of data. A convolutional layer instead slides a small filter, say 3x3 pixels, across the whole image, and that same filter is reused at every position. If it learns to detect a horizontal edge, it detects horizontal edges everywhere, not just in the top-left corner where it happened to see one during training.

Pooling is the part that trips people up first, so here's the plain version: after convolution you have a grid of "how strongly did this pattern show up here" scores. Max pooling looks at each small block of that grid, say 2x2, and keeps only the highest score, throwing the rest away. That halves the width and height every time you do it, and it also buys you a bit of tolerance: a stripe pattern that's shifted by a pixel or two still gets picked up by the same pooled region.

Which lands squarely on what we're doing here, because the whole premise of malware-image classification is "does this texture roughly match a family's texture", not "is this the exact same file". Convolution plus pooling is built for exactly that kind of approximate, position-tolerant pattern matching. It's the same reason CNNs are good at recognising a cat regardless of which corner of the photo the cat is standing in.

Stack a few rounds of "detect a local pattern, then shrink and keep the strongest bits" and by the last layer the network isn't looking at raw pixels any more. It's looking at learned combinations of shapes, which is a far better basis for "which family is this" than any single pixel value could ever be.

The dataset, and being straight about mine

The dataset this technique is usually taught on is malimg: roughly 9,300 malware samples pre-converted to greyscale images, spread across 25 real families with names like Adialer.C, Swizzor.gen!E and VB.AT, some families holding thousands of samples and others holding barely twenty. Fetching it means either a Kaggle API key hitting Kaggle's servers or a direct download from the original research mirror, and this sandbox has neither: every outbound request I tried came back as a blocked connection, no exceptions for research datasets.

Rather than write around numbers I hadn't actually produced, I built my own small image dataset with the same shape as the real problem: five made-up "families" (I want to be completely clear these names are mine, not real malware, so nobody mistakes them for an actual threat), 920 images total, deliberately imbalanced the way real malware families are, 320 down to just 40. I gave each family its own generating pattern, stripes, blocks, scattered dots, diagonal waves, plain noise, plus per-image randomness so no two samples in a family are identical, the same way real compiled variants of one family differ in the details while sharing a structure.

That last point matters enough to say plainly: I didn't just make DotSkintrim rare, I made it genuinely resemble BlockFake in texture, because that's what real malware families that share a packer or a builder actually look like to this technique. The original malimg paper reports exactly this kind of confusion between families that share tooling. Building a dataset that's rare but trivially separable would have taught me nothing, so I made the hard part hard on purpose.

How you'd actually build this

The standard modern approach, and the one worth knowing even though I couldn't run it here, uses torchvision to preprocess and a pretrained ResNet50 for transfer learning:

import torch
from torchvision import transforms, datasets, models
from torch.utils.data import DataLoader

transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

train_data = datasets.ImageFolder('malimg/train', transform=transform)
train_loader = DataLoader(train_data, batch_size=32, shuffle=True)

model = models.resnet50(weights='IMAGENET1K_V2')
for param in model.parameters():
    param.requires_grad = False
model.fc = torch.nn.Linear(model.fc.in_features, num_classes)

ResNet50 has already learned to recognise edges, textures and shapes from a million photographs. Freezing its existing layers and only training a new final layer means you're reusing that visual knowledge and just teaching it your specific classes, which needs far less data and time than training from nothing.

I couldn't do that here. No torch, no torchvision, and no network route to download a pretrained checkpoint even if I installed them. So the numbers in this post come from a compact CNN I built myself in plain numpy, two convolutional layers (8 filters, then 16), two pooling layers, and two dense layers on top, trained completely from scratch with no pretrained weights of any kind. It's a smaller, humbler thing than ResNet50 and I want that difference on the record rather than glossed over: everything below is what a from-scratch model can do on this problem, not what transfer learning would do.

# my actual model, plain numpy, no framework
class TinyCNN:
    def __init__(self, n_classes):
        self.conv1 = Conv2D(in_c=1,  out_c=8,  k=3, pad=1)   # 24x24 -> 24x24
        self.pool1 = MaxPool2()                              # 24x24 -> 12x12
        self.conv2 = Conv2D(in_c=8,  out_c=16, k=3, pad=1)   # 12x12 -> 12x12
        self.pool2 = MaxPool2()                              # 12x12 -> 6x6
        self.fc1   = Dense(in_f=16*6*6, out_f=64)
        self.fc2   = Dense(in_f=64, out_f=n_classes)

    def forward(self, X):
        x = relu(self.pool1.forward(self.conv1.forward(X)))
        x = relu(self.pool2.forward(self.conv2.forward(x)))
        x = relu(self.fc1.forward(x.reshape(len(x), -1)))
        return self.fc2.forward(x)

Each image is 24x24, one channel. The first convolution keeps the size the same (thanks to padding) but expands to 8 learned filters, pooling halves it to 12x12, the second convolution expands to 16 filters at 12x12, pooling halves it again to 6x6, and that gets flattened into 576 numbers feeding two dense layers ending in 5 outputs, one score per family. I trained the whole thing with Adam, a common gradient descent variant that adapts its own step size per parameter, at a learning rate of 0.002, in batches of 32, over 18 passes through the training data.

Splitting it and training

An 80/20 stratified split, the same instinct as the 60/20/20 split in Topic 8, keeps each family's proportions intact in both halves so the rarest class doesn't vanish from the test set by bad luck.

train 736  test 184
train class counts: [256 192 160  96  32]
test class counts:  [64 48 40 24  8]

Thirty-two DotSkintrim images to learn a texture from. Keep that number in mind, we'll come back to it.

epoch  1/18  loss 1.1188  train_acc 57.61%
epoch  5/18  loss 0.1301  train_acc 96.74%
epoch 10/18  loss 0.0265  train_acc 99.59%
epoch 14/18  loss 0.0102  train_acc 100.00%
epoch 18/18  loss 0.0049  train_acc 100.00%
fit time: 11.1s

Full marks on the training set by epoch 14, which on its own tells you almost nothing useful. A network with enough capacity will happily memorise 736 images if you let it run long enough, the only number that means anything is what happens on images it never saw during training.

The test set results that looked great

95.65% weighted accuracy is a genuinely good number to see on a first from-scratch model. Four of five families sit at a perfect 1.0000 across the board. If I'd stopped reading at the weighted row, which is exactly what the row is designed to tempt you into doing, I'd have written a much shorter, much less honest post.

Where the misses actually went

Same move as Topic 8: a single F1 score tells you how bad the damage is, a confusion matrix tells you exactly what got confused with what.

This isn't scattered noise, it's a two-way mix-up between exactly the two families I built to overlap. 3 real BlockFake images got called DotSkintrim, and 5 real DotSkintrim images got called BlockFake. Every other family, including WaveYuner with only 24 training images, came out clean. Sample count alone doesn't explain the gap: DotSkintrim lost because it was both rare and genuinely similar to a more common class, and either one on its own would probably have been survivable.

Why a rare, similar-looking family loses structurally

This maps onto something well documented in the actual malimg research, not just my toy version of it: families that share a packer, a builder or a chunk of reused code produce images that genuinely look alike, and the original 2011 paper reports real confusion between such families. If two malware families are built from largely the same code with a different payload stitched in, expecting a texture-based classifier to cleanly separate them is expecting more than the technique promises. It's reading layout, not intent, and layout can be shared on purpose or by accident.

What you'd actually do about it

Not just "get more DotSkintrim samples", although that's the honest first answer if you can get it. A few things that don't require a bigger dataset:

• Data augmentation on the rare class specifically. Small rotations, flips and noise jitter on just the minority family's images, generating more training variety without needing more real samples.
• Class-weighted loss. Penalise a mistake on DotSkintrim more heavily than a mistake on StripeBot during training, instead of treating every image equally regardless of how rare its class is.
• A second, cheap signal alongside the image. File size, section count, or import table hashes, features from the actual PE structure rather than the pixel grid, can break a tie the image alone can't.
• Route the ambiguous cases to a human. If the model's own confidence on a prediction is low, or two classes are close, that's worth flagging for review rather than trusting the top label blind.

I haven't tried the augmentation route against this exact dataset yet and I'd like to, mainly because I'm not actually sure it would help here. Augmenting DotSkintrim makes more versions of images that already resemble BlockFake, it doesn't necessarily teach the model a feature that tells the two apart. That might be a case where the fix has to be architectural or data-driven rather than "more of the same, rotated".

A quick side quest: text instead of images

The material I was working from also covers a smaller exercise using the same toolbox on a completely different kind of data, sentiment classification on written reviews, TF-IDF turning words into numbers the way Topic 7's CountVectorizer did, fed into logistic regression instead of Naive Bayes. I'm not building that one out into its own lab here, this post is long enough and the ground (turning text into vectors, then classifying) is ground I've already covered properly. But it's a good reminder that "AI in security" isn't one technique, it's a toolbox, and the same TF-IDF idea that spots spam words can just as easily spot sentiment, or phishing language, or leaked-credential paste dumps. Worth filing away for whenever I need it.

The lesson that won't go away

I keep expecting the "headline metric hides the class that matters" lesson to stop showing up and it keeps showing up. Logistic regression in Topic 6, Naive Bayes in Topic 7, Random Forest in Topic 8, and now a CNN. Four different model families, one repeated failure mode: rare plus subtle beats big headline numbers every time, and the only way I've found to catch it is to deliberately go and look, because the weighted average will never volunteer it.

What I'm genuinely unsure about is how much of this generalises to a real ResNet50 transfer-learning setup on the actual malimg dataset. Transfer learning brings a huge amount of pretrained visual vocabulary my two-layer network never had, edges, textures and shapes learned from a million unrelated photos, and it's entirely possible that extra vocabulary closes a gap like this one rather than just relocating it. I don't know yet, and I'd rather say that plainly than imply my numpy model settles the question for the full-size approach.

Next up I'm actually turning this track around properly: instead of building classifiers, trying to break one. Prompt injection and data poisoning are next.

References

• Nataraj, Karthikeyan, Jacob & Manjunath, "Malware Images: Visualization and Automatic Classification" (2011) (the paper this whole technique comes from)
• torchvision: ResNet models
• PyTorch: Conv2d documentation

FAQ

How does malware image classification work?

A binary's raw bytes are read as unsigned 8-bit values (0-255) and reshaped into a 2D array, which is then treated as a greyscale image. Malware from the same family tends to keep a similar internal layout, so different samples of one family produce visually similar textures a CNN can learn to recognise.

What is a CNN and how does it classify images?

A convolutional neural network slides small filters across an image to detect local patterns like edges and textures, shrinks the result with pooling, and repeats that a few times before a final dense layer turns the learned features into a class prediction. Early layers learn simple shapes, later layers combine them into more complex ones.

Why do malware families produce similar looking images?

Samples in the same family are usually built from shared code, the same packer, or the same builder tool, so their compiled bytes line up in similar patterns even when individual samples are tweaked to evade signature detection. That structural similarity is what the image conversion is designed to expose.

Why did the rare family get missed so often?

It had only 40 of 920 images and its texture deliberately overlapped another, more common family in my dataset. With few examples and a genuine visual resemblance to borrow mistakes from, the model leaned on the wrong family's pattern, recall on that class came out at 0.375 while overall weighted accuracy read 95.65%.

What is transfer learning and why didn't I use ResNet50 here?

Transfer learning reuses a model already trained on millions of images, like ResNet50, and only retrains its final layers on your data. It needs a downloaded pretrained checkpoint, which this sandbox couldn't fetch with no outbound network, so I built a small CNN from scratch instead and trained it from nothing.

Related reading

• Network Anomaly Detection with Random Forest (Topic 8, the same rare-class problem in a completely different model)
• Naive Bayes and How a Spam Filter Thinks (Topic 7, TF-IDF-style text vectorisation via CountVectorizer)
• AI in InfoSec, from Raw Logs to a Model (Topic 6, where this whole "accuracy lies" thread started)
• Browse the whole AI / LLMs track