Revolutionizing AI Security: Static Scans and Red Teams Lead Charge

At a glance

– Vulnerability Concerns: Malicious AI models are turning up more often on public repositories such as Hugging Face. – AI Security Measures: Security firms are using static scans and automated red teaming to find unsafe models. – Key Players: JFrog, Protect AI and ReversingLabs are among the companies building tools, and Google, NIST, OWASP and Databricks have published frameworks. – Ethical Considerations: Teams also have to check models for bias and licensing problems, not only malicious code. – Future of AI Security: Balancing fast AI adoption with security is still hard.

Malicious models on the rise

Companies are pulling AI models from public repositories as casually as they pull open-source packages. Attackers have noticed. JFrog said it had flagged 212 AI models containing some kind of malicious functionality or code, roughly double the count a year earlier, Dark Reading reported in March 2025.

Why models are a supply-chain risk

The risks look a lot like the ones in open-source software. “The way that developers and data scientists use machine learning model is very similar to how developers use open source packages,” Tal Zarfati, lead architect at JFrog Security, told Dark Reading. Some models carry code that runs the moment they are loaded, often hidden in unsafe serialization formats such as Pickle.

Scanning models before they load

Static scanning checks a model’s files for that kind of code before anyone deploys it. On March 4, 2025, JFrog announced a partnership with Hugging Face to scan the machine-learning models hosted there. Scanners have limits, though: researchers at ReversingLabs found two models with malicious code in their Pickle files that slipped past Hugging Face’s automated checks.

Today’s scanners mostly catch the easy cases, code that runs on load. Backdoors that trigger only on a particular prompt are much harder to find.

Red-teaming DeepSeek-R1

Red teaming attacks a model the way an adversary would to see what breaks. Protect AI used its automated red-teaming service on the Chinese model DeepSeek-R1 and found no backdoors or remote code execution flaws, though it warned the model’s outputs might not match many companies’ values. Modified DeepSeek copies uploaded by attackers to Hugging Face did carry backdoor code, the company said.

“Teams should also continue to run model scanning against model version updates, as well as automated red teaming against live applications to maintain the strongest security position,” said Chris King, head of product at Protect AI.

Frameworks to build on

Several frameworks can help companies build a safer process. Google created the Secure AI Framework (SAIF). NIST’s AI Risk Management Framework brings together industry and government thinking. OWASP released an AI Security and Privacy Guide in 2023, and Databricks published the second version of its AI Security Framework, which maps 62 risks to 64 controls.

Provenance matters too. Mandiant’s Muhammad Muneer advised companies to evaluate a model developer’s reputation, make sure the code has been audited and test the model for vulnerabilities before adopting it.

Beyond code: bias and fairness

Security isn’t the only check. A model can be free of malicious code and still produce biased or unfair results, especially in decisions that affect people’s lives.

Due diligence on a model should cover bias, licensing and performance alongside security, so teams know what they are deploying.

Treat models like open-source code

The lesson from the open-source world applies: know where your components come from and keep checking them. Static scans, red teaming and runtime monitoring for problems like prompt injection all have a place.

AI models offer real benefits, and they bring real risks. Companies that scan models, test them like attackers and keep monitoring them in production will catch more problems before they turn into incidents.

Frank Jones, CISSP

Frank Jones has loved computers from the age of 13. Frank got his hacking career started when he downloaded a war dialing program that he used to detect dial up modems in his hometown of Chicago. Frank Jones now works as a JAVA coder and cyber security researcher.