DevSecOps for AI: Shift Left on Model and Prompt Security
DevSecOps recap: shift left on the secure SDLC
DevSecOps took the security review that used to sit at the end of a release cycle and moved it into the pipeline itself. Instead of a penetration test two weeks after deploy, the secure SDLC runs checks at every commit: static analysis on source code, dependency scanning, secret detection, container scans, and policy gates that block a merge before it can reach production. Security that runs continuously inside CI/CD catches defects when they are cheapest to fix, which is the entire point of shift left security.
The mechanics are well established for conventional software. You scan source code, you scan third party libraries, you scan images, you enforce least privilege, and you require signed artifacts. Each control produces an artifact that an auditor can later read, and that matters enormously in a regulated environment. A bank cannot claim "we checked it" without evidence. A pipeline that emits scan results as machine readable artifacts turns a vague assertion into a repeatable control.
Why AI systems break the old pipeline
Classic application security in CI/CD rests on a few assumptions that AI systems quietly violate. A conventional application has a fixed attack surface: your routes, your inputs, your dependencies, your secrets. An AI system adds new surfaces that the old scanners never see.
The first break is the prompt, which is both input and control plane. A prompt is user supplied text that partially determines behavior. That makes it a classic injection surface, except the "code" being injected is natural language rather than SQL or shell. Prompt injection needs no memory corruption bug; it only needs a model that treats instructions and data as the same thing. Your static analysis tools have nothing to say about it because there is no syntax to flag.
The second break is non determinism. Two runs of the same model on the same input can produce different output, which undermines regression style evals that assume a fixed expected result. A test that passes today can fail tomorrow for reasons that have nothing to do with your application code.
The third break is that the model is now a supply chain dependency. Model weights, tokenizers, adapters, and fine tuned checkpoints are large binary artifacts with their own provenance, versions, and vulnerabilities. They do not appear in your software bill of materials the way a JavaScript package does.
The fourth break is generated code. When an AI coding assistant writes code, the organization inherits the assistant's training data, which includes vulnerable patterns and, occasionally, copied snippets. That code enters the repo through the same pipeline as hand written code, but it deserves extra scrutiny because nobody reviewed it line by line at the point of creation.
Shift left on prompts and models
The first place to move security earlier is at the definition of the system itself. Prompt security belongs in the repo, not in a post deploy review.
Treat prompts as configuration with a blast radius. Every prompt, including system prompts, tool descriptions, and retrieval context, is part of your attack surface and should be versioned, reviewed, and tested like code. Store them in version control, attach owners, and run automated checks on every change. A prompt that grants an agent access to a customer record is a privilege escalation waiting to happen, so review prompt changes the way you review permission changes.
Model security starts at selection. Before a model is approved for a regulated workload, document what it can do, what data it was trained on, what safety filtering it includes, and how it will be hosted. A model with no content filtering and no output validation is not a candidate for a lending workflow that touches customer PII. Record the decision as a control: which model, which version, which provider, which fallback.
Then enforce guardrails in the pipeline, and keep the control definitions distinct. Input validation for LLM applications means sanitizing and flagging untrusted instructions in retrieved or user supplied content before the model acts on them. That is prompt injection defense, and it is separate from redacting sensitive data such as PII before it reaches the model, which is a data loss prevention and disclosure control with its own policy. Output validation means sanitizing and validating the model's response before it reaches a user or a downstream system. Put those checks in CI/CD as testable policy in addition to runtime guardrails. Shift left complements the inference time controls rather than replacing them, because CI/CD evals cannot cover every runtime behavior, so the runtime layer should be reproducible and testable rather than eliminated.
Prompt injection as a pipeline problem
Prompt injection prevention is not a single feature; it is a set of invariants you assert in tests. An invariant might be: "the model must refuse to reveal the system prompt when asked," or "the model must not follow an instruction embedded in a document it is asked to summarize." Encode those invariants as tests that run on every prompt change.
A prompt is a control surface, not a string. If you are not testing it, you are not securing it.
Evals as security tests
Traditional security testing needs an expected result to compare against. AI outputs are probabilistic, so you need a different comparison: you evaluate behaviors, not exact strings. This is where evals become the security test suite for AI systems.
An eval is a security test when it asserts a safety property rather than a stylistic one. Instead of checking that an answer is "correct," it checks that the model refuses a jailbreak, does not leak a secret from its context, does not return a PII value, or does not follow an instruction from untrusted content. These are adversarial assertions, and they belong in the same gate as your SAST results.
Red teaming turns this into a repeatable practice. An AI red teaming pass generates adversarial inputs: indirect prompt injection in emails, roleplay jailbreaks, tool abuse prompts, and data exfiltration attempts. The outputs of that pass become permanent regression evals, so every future model or prompt change is re-tested against the attacks that worked before. That closes the loop: the attack of today becomes the test of tomorrow.
Anchor the evals to a known taxonomy so they are defensible. The OWASP Top 10 for LLM Applications (2025) gives you the current list of failure classes to cover: Prompt Injection, Sensitive Information Disclosure, Supply Chain, Data and Model Poisoning, Improper Output Handling, Excessive Agency, System Prompt Leakage, Vector and Embedding Weaknesses, Misinformation, and Unbounded Consumption. Map each eval to one of those categories and you have an auditable coverage statement.
Scanning AI-generated code
Generated code changes the review equation. A human developer usually has context: why this function exists, what the caller expects, which library is already in use. An AI coding assistant holds a compressed version of that context and fills the gaps with statistically common patterns, some of which are vulnerable.
AI-generated code enters the pipeline as untrusted input, and the correct response is to scan it harder, not to ban it. Banning AI coding tools in a regulated bank is a losing game because the tools get used anyway, off the books. The workable policy is to route generated code through the same static analysis, dependency checks, and secrets scanning as everything else, and to add a second pass that hunts the specific failure modes assistants tend to produce.
The failure modes are predictable. Generated code may concatenate strings into SQL, deserialize untrusted data, hardcode credentials it saw in training data, or emit a library call with a deprecated or vulnerable argument. Static application security testing catches a large share of these because they are ordinary code flaws. The remaining gap is semantic: code that compiles, passes lint, and is still wrong about the business rule. That gap is closed by human review, which is why secure AI code generation means pairing the assistant with a required reviewer and a required scan, not a rubber stamp.
A generated code policy
Route assistant output through the same SAST, dependency, and secret scans as human code.
Tag generated changes so reviewers know which lines came from a model.
Require a scan pass before a generated change can merge, with the same gate as hand written code.
Keep a log of generated contributions for audit and attribution.
Supply chain: the model is a dependency too
Your software bill of materials has a blind spot the size of a model checkpoint. AI supply chain security means extending provenance, versioning, and vulnerability tracking to the weights, adapters, and runtime you ship. If you pin a base model by a loose tag, you are effectively running an unpinned dependency that can change underneath you, and you have no lockfile to detect the drift.
Treat model artifacts like any other binary. Pin exact versions, store checksums, record the source and the fine tuning lineage, and keep an inventory that maps each deployed model to its training and approval record. When a vulnerability or a safety issue is disclosed in a base model, you need to know within minutes which of your applications inherited it. That is impossible without an inventory.
The same discipline applies to the surrounding stack: vector databases, orchestration frameworks, agent toolkits, and evaluation harnesses are all ordinary software with ordinary CVEs. They happen to be young and fast moving, which means the update cadence is higher and the scanning cannot be skipped. Continuous dependency scanning that already runs in your pipeline will catch most of this, but only if these components are actually declared as dependencies instead of being installed ad hoc.
A CI/CD blueprint for AI systems
Here is a concrete pipeline that extends DevSecOps to AI systems without inventing a new discipline. Each stage produces an artifact you can show an auditor.
Stage 1: Prompt and policy linting
On every change, run checks that prompts are versioned, that no new prompt grants a tool or permission that was not declared, and that sensitive data markers are present where required. Fail the build on a prompt change that introduces a new tool call without a matching review.
Stage 2: Static and dependency scans
Run static application security testing over the application code, including any AI-generated code, and run dependency scans over both the software stack and the model artifacts. Pin model versions and verify checksums here.
Stage 3: Security evals
Run the adversarial eval suite: prompt injection attempts, jailbreaks, secret leakage probes, and output validation checks. This is the gate where a model or prompt change proves it did not regress on safety.
Stage 4: Red team sampling
On a schedule rather than on every commit, run a broader AI red teaming pass over the integrated system and feed any new working attacks back into Stage 3 as permanent tests.
Stage 5: Release gate
Require signed evidence from Stages 1 through 4 before a release candidate can deploy. Store the artifacts alongside the build so compliance can retrieve them later.
In a regulated AI system, the pipeline is the audit trail. If the gate cannot prove what it checked, the control does not exist.
Where Kolega DevSec fits
This is exactly the terrain Kolega DevSec is built for. DevSec finds, reproduces, fixes, and tests vulnerabilities in AI-generated code and the software around it, then opens a pull request for the team to review and merge. When an enterprise shifts left on AI, DevSec turns the static and dependency scan stage of the blueprint above into a running control: each finding arrives with the reproduction, the tested fix, and a human on the merge.