MalwareScope

LLM-assisted malware analysis and classification

A self-hosted platform that tells you what a Windows file is, then proves what it does, without the file ever leaving the organisation.

Presented by

Omar Kharrat
Rayen Skhiri

Supervisors

Hachem Barhoumi, 3S
Sourour Mhenni, ISI

Graduation project

National Professional Master's
Information Systems and Infrastructure Security, 2025-2026

Our host: the Cybersecurity department of 3S

An integrator since 1988

Standard Sharing Software builds and runs IT infrastructure for companies in Tunisia, and in Côte d'Ivoire since 2019.

Security is a core business

Firewalls, SIEM, IAM and PAM, and a SOC and NOC that watch customers' systems.

TrustShield, the managed SOC

Built with GlobalNet on an XDR, a SIEM and a SOAR. The technology partners are Palo Alto Networks and Splunk.

That is why MalwareScope plugs into Cortex XDR and Splunk: they are the tools the SOC already uses every day.

A suspicious file reaches the SOC. What is it, and what does it do?

AV-TEST registers more than 450,000 new malicious programs every day. Most are variants built to slip past signatures.

Usual answer	Why it falls short
Reverse engineering by hand	Reliable, but one analyst cannot keep up with the volume.
Antivirus verdict	Signatures and opaque heuristics: hard to explain, easy to evade with a new variant.
Public sandbox	The customer's file is uploaded to a third party.
Self-hosted sandbox	Heavy to set up, and if it is badly isolated it becomes an attack path itself.
Ask a general LLM	Tempting, but in our test it was right only 36% of the time.

Meet MalwareScope

Drop a Windows file. Get its family in seconds, watch it run in a sealed machine, and send the verdict back to the incident. On your own hardware.

2independent verdicts, static and dynamic

21.9 sfrom the request to a running sample

0files sent outside the organisation

Live sandbox session with a 100 out of 100 verdict

One workspace · 1 / 3

A verdict in seconds, then the real behaviour

**Static verdict**family and confidence in seconds, with a written explanation

**Live sandbox**drive the Windows desktop from the browser

One workspace · 2 / 3

Every finding explained, every indicator ready to share

**Causality graph**every finding tied to the process that caused it

**IOCs ready to share**hashes, domains, URLs, defanged, CSV or JSON

One workspace · 3 / 3

Kept as proof, and wired into the SOC

**Signed history**PDF report and an integrity check for each analysis

**Cortex XDR tab**from an incident's file to a verdict on the incident

How it works: two verdicts on the same file

One Windows executable, identified by its SHA-256, never run outside the sandbox

Static verdict

a few seconds, the file is never executed

8 features read from the PE structuresize, entropy, sections, imports, exports, strings

Random Forest we trained on 760 real files

Family and confidence for example trojan, 82%

Dynamic verdict

the file runs in a disposable Windows VM

Isolated KVM machine, live console in the browser

Sysmon telemetry and full network captureDNS, decrypted HTTPS, PCAP

Behaviour score from 0 to 100 with MITRE ATT&CK techniques

The language model never gives a verdict. It writes the explanation of both, in plain English.

A model we trained ourselves, on 760 real executables

trojan319 benign201 ransomware110 spyware51 malware (other)49 coinminer25 worm3

Plus 2 files of unknown family. The set is imbalanced, and that shows in the results.

559 real malware

AgentTesla, Emotet, GandCrab, LokiBot, NanoCore, TrickBot and others, from three public research repositories: jstrosch/malware-samples, theZoo and MalwareBazaar.

The 201 benign files are real tools: Sysinternals, PuTTY, 7-Zip, Notepad++.

Samples stay in encrypted archives and are never pushed to Git.

Eight features, and the model that fits them

Feature	What it tells us
file_size	packed payloads are often large
file_entropy	above 7, likely compressed or encrypted
num_sections	unusual counts point to a packer
avg_section_entropy	where the packed data sits
num_imported_dlls	which Windows libraries it needs
num_imported_functions	few imports: API resolved at run time
num_exports	functions it offers to others
strings_count	readable strings of 5+ characters

Why a Random Forest

Random Forest chosen

200 trees vote. Good results with default settings on a small set, no scaling needed, a real confidence and the weight of each feature.

XGBoost

Strong on millions of files like EMBER. On 760 files it overfits unless it is finely tuned.

k-nearest neighbours

Distance based: the file size in millions of bytes drowns out entropy. No feature importance.

On 152 files it had never seen

89.5%

accuracy over the seven families
weighted F1 0.87

95.4%

benign or malicious
2 false positives, 5 misses

86.8%

mean of a 5-fold cross-validation
84.9% to 88.8% per fold

Ransomware and coinminers are almost perfect. Most errors land in the mixed "malware" class and the rare worm, which the model calls trojan.

With 60 training files the model scores 79.6%, with 608 it reaches 89.5%, and the curve is still rising. Re-running the training script gives exactly the same numbers.

Most useful features: file entropy (0.169), imported functions (0.163), strings (0.138).

Could an LLM alone do the job? We measured it

Same 50 real test files, same information. The LLM (gpt-oss-20b through Groq) had to name the family by itself.

82%

our Random Forest

36%

LLM alone

On the yes-or-no question, the LLM caught almost every malware. It also called all 13 benign files malicious.

A tool that flags every clean file buries the analysts in false alerts. The forest kept 12 of the 13 clean files clean.

So in MalwareScope the trained model decides, and the LLM writes the explanation from the same evidence.

The sandbox never runs a file it cannot watch

Check the cage
the VM sits on a network with no route out, nothing is shared
Test the recorders
the guest makes a marker process and a DNS query with a random nonce
Find the proof
the host looks for that nonce in Sysmon, in the DNS sink and inside the PCAP
Deliver, verify, wait
the file lands on the desktop, SHA-256 checked on both sides, not executed
Run on command
only when the analyst presses Run

Capture service stopped on purpose: the run is refused in 0.5 s and the VM never starts.

A live Windows desktop in the browser, like ANY.RUN, but at home

Real session with our harmless test program, the canary. Played at 1.5x.

The analyst drives the VM through a filtered noVNC console. Verdict, decrypted requests and findings fill in while the sample runs.

69.2 s → 21.9 s

from the request to a running sample, thanks to warm start: a saved, logged-in Windows desktop restored on a fresh disk overlay

40 warm-start sessions out of 40 completed.

Live sandbox · one run

One run, three moments

**01Delivered, not executed**on the guest desktop, SHA-256 matched before and after transfer, capture already recording

**02Running**the analyst pressed Run: the live verdict and the network capture start filling in

**03Verdict**100 / 100, malicious, from 2,219 events, findings mapped to MITRE ATT&CK

One appliance, and every part only gets what it needs

Least privilege. The part exposed to the network has no rights. Only the capture service touches iptables and tcpdump.

One channel to the guest. Files and commands go through the QEMU agent. No shared folder, no clipboard.

One machine per analysis. Each slot has its own network, capture and image, and is destroyed after use.

Twelve layers, each assuming the last one can fail

Contain the sample

so it cannot reach the host or the Internet

L1 QEMU confinementown user, AppArmor, seccompread on the running VMs

L2 Network isolationgate refuses any routed or bridged VM11 tests, ISOLATED OK

L3 Controlled egresssink by default, allow-list, kill switchsink active on every run

Observe correctly

so the sample shows its real behaviour

L4 Anti-evasionhypervisor flag hidden, Dell SMBIOS, real MACPafish: 3 markers gone

L5 Controlled executionverified delivery, one use, clean state provenrefusal in 0.5 s

L6 Untrusted guest dataallow-listed keys, size capsautomated tests

Protect access

so the platform itself is not the way in

L7 Hardened servicesystemd sandbox, no capabilitiesexposure 9.2 → 1.5 / 10

L8 AuthenticationPBKDF2, HMAC tokens, rolesblocked from the 11th try

L9 Web and consolestrict CSP, single-use 30 s ticket, RFB filter11 tests

Prove

so a result can be trusted later

L10 Secretsgenerated on the box, encrypted, write-onlybundle refused with a key

L11 Evidence and auditSHA-256 + HMAC manifest per analysistampering detected

L12 Confidentialitythe file never leaves, Cortex writes off by defaultno secret in API replies

From a Cortex XDR incident to a verdict, inside the SOC

What an analyst does today

an XDR alert flags a file on an endpoint
someone fetches the file by hand
uploads it to a public sandbox, or pays for ANY.RUN through XSOAR
copies the result back into the incident

With MalwareScope

pick the incident and the file in our Cortex tab
the endpoint's agent uploads the file after an explicit confirmation
it runs in our own sandbox, on our own network
the verdict goes back as a comment on the incident

No XSOAR licence, no third-party upload, and every step stays in the audit log.

The loop, call by call, on the Cortex XDR public API

List incidents incidents/get_incidents
severity, alerts, status, last 7 days by default
Files and endpoints of an incident get_incident_extra_data
name, hash, signature, where it was seen
Retrieve the file endpoints/file_retrieval
confirmed by the analyst, then followed with get_action_status
Unpack safely
size caps, no path tricks, SHA-256 must match the incident
Analyse
a normal sandbox session, Run stays manual
Post the verdict incidents/update_incident
score, family, link to the PDF report

The two writes are off by default

File retrieval and the verdict comment touch customers' endpoints and incidents. An administrator turns them on in Settings, with a warning, and each use is audited.

What the client cannot do

There is no isolate, quarantine or remediation function in the code. The API key only needs the "Privileged Responder" role.

Connected to a real Cortex XDR tenant

Customer names, hosts, users and the tenant address are blurred. The incidents and files are real.

Configured, tenant reachable, API key accepted. The tab lists the real incidents of the last seven days, their files with the signature status, and the endpoints where each file was seen.

Where we stand

done on the real tenant
reading incidents, files and endpoints

validated on a simulated tenant
file retrieval and the verdict comment, end to end, with automated tests

Cortex can call us too, and the whole SOC gets the results

MalwareScope as a Cortex integration

Packaged as one YAML file, like the ANY.RUN pack, so a playbook can call our sandbox.

malwarescope-detonate-file	submit and poll for the verdict
malwarescope-get-analysis	full result
malwarescope-get-report-pdf	attach the report
malwarescope-lookup-hash	known verdict for a SHA-256

Tested against our real API: the canary came back malicious, 100/100, in 110 s. Importing it in a SaaS tenant needs a Cortex engine that can reach the appliance.

Exports to the tools around it

Splunk

HTTP Event Collector

Any SIEM

CEF over syslog

Threat intel

STIX 2.1 / TAXII, MISP

Chat, tickets

signed webhooks

Only malicious results are published as indicators, with an expiry and without private addresses. VirusTotal only ever receives the hash.

What TrustShield and its analysts get from it

For the managed SOC service

A sandbox of its own. Customer files are analysed inside the SOC, never uploaded to a public service.
It speaks the stack TrustShield sells. Cortex XDR in, Splunk out, with the same Palo Alto Networks and Splunk partners.
Evidence to hand to a customer. A signed PDF report, the IOCs and the packet capture for every analysis.
Easy to roll out. One ISO and about thirty minutes, on a central box or at a customer's site.

For the analyst on shift

A first answer in seconds, then the real behaviour about twenty seconds after pressing Run.
No copy and paste. The file comes from the endpoint and the verdict goes back onto the incident.
Findings they can defend. Each one names the process behind it and its ATT&CK technique.
No false comfort. When the capture cannot prove a file is clean, the answer is inconclusive.

How it compares with what SOCs use today

	VirusTotal	Hybrid Analysis	ANY.RUN	Cuckoo / CAPE	MalwareScope
Where the file goes	public cloud	public cloud	vendor cloud	stays on site	stays on site
Static verdict	many AV engines	static analysis	dynamic only	no static model	our trained model
Live, interactive VM	no	no	yes	mostly automated	yes
Refuses to run until the capture is proven	no	no	no	no	yes
Cortex XDR	not compared here		through XSOAR	no	built in, no XSOAR
Getting started	nothing to install, private analysis is paid			heavy installation	ISO, about 30 min

From the public descriptions of each product, as summarised in chapter 2 of our report. MalwareScope is smaller in scale: one Windows image, and no API call tracing yet.

What is new in MalwareScope

Capture before execution

The sample only runs once the host has found its own test traffic in Sysmon, the DNS sink and the PCAP. No proof, no run.

Verdicts that admit doubt

A clean result the evidence cannot support becomes inconclusive. What is not collected, like API calls, is shown as not collected.

An LLM in the right job

We measured it against our trained model, 36% against 82%, so it explains the verdict and never gives it.

Cortex XDR without XSOAR

From an incident's file to a verdict on that incident, with endpoint actions off until an administrator turns them on.

Proof you can check later

Every analysis carries a SHA-256 and HMAC manifest, so a changed verdict or screenshot is detected.

A sandbox you can install

An ISO that builds its own Windows analysis machine and checks its isolation before the first run.

It installs like an appliance, and it is tested

30 min

from booting our ISO to a working appliance, analysis machine included, on a fresh 6-core VM

413

automated tests, no failure, run on the appliance with fake guests and a fake libvirt

40 / 40

warm-start sessions completed, 12 / 12 with the final manual policy, 9 / 9 with two at once

Docker was considered and dropped: the web part uses 30 MB, each Windows machine about 3 GB, and a container would need the libvirt socket and network admin rights anyway.

The two verdicts complement each other

File	Static (RF)	Dynamic (sandbox)	What we learn
canary (imitates malware)	trojan, 45.5%	malicious, 21 runs at 100/100	the model hesitates on an unusual file, the behaviour settles it
benign test program	trojan, 48.5%	benign, 14/100	a low-confidence static mistake, fixed by the sandbox
Python installer	not analysed	suspicious, 52/100	installers drop executables from Temp, like droppers
Zadig (USB tool)	trojan, 38%	inconclusive	nothing observable, so no false "benign"
UsbDk driver (.msi)	not analysed	inconclusive	not an executable, reported as such

What is still missing, and what comes next

Limits we know

760 files, imbalanced, labelled by family
sandbox measured with our canary, not yet a batch of real malware
rules are harsh on installers
Windows API calls are not collected
Cortex writes not yet run on a real tenant; HTTP only

Next steps

run a batch of real malware and compare RF, LLM and sandbox on it
one hybrid model on static and behavioural features
use Authenticode signatures in the score
the full Cortex loop on a test endpoint, then the playbook pack
HTTPS in the setup wizard, rename Sysmon, remove the last QEMU markers
implement MFA (multi-factor authentication) in the application

Know what a file is. Prove what it does. Keep it at home.