AI security testing

By @edhonour · Instagram

Source: https://www.instagram.com/reel/DdcyxCYpsSs/?stkn=MWZ5dDZoeWJlNjdpOQ==

AI generated summary of the linked source. Posted anonymously. Transcripts are published only for public links, never for uploads, documents or pasted text.

Your library

The AI link is plain markdown. Paste it into any assistant and it can read the whole public entry.

Summary

AI systems need security and permissions testing baked into every eval, not just functionality checks. People actively try to jailbreak AI or extract system prompts, and a model or prompt upgrade can silently break guardrails that worked fine yesterday.

Core ideas

One context card plus 5 ideas

Security as a test category: Permissions and security need dedicated slots in AI eval suites, sitting alongside functionality tests, not as an afterthought.
Adversarial users exist: People deliberately probe AI systems to extract system prompts or push them off-topic, so this behavior has to be assumed and tested for.
Regression testing matters: Every model upgrade or prompt change needs a fresh security pass, since past security guarantees don't carry forward automatically.
Guardrails are fragile: A model swap or prompt tweak can quietly break protections that took real effort to build, without any obvious signal that it happened.
Continuous vigilance over one-time checks: Security testing isn't a setup step you finish once. It's a repeated part of the release cycle for any AI application.

Quotes

“I know I'm one of those people that as soon as I see it's AI, I want to see if I can get a system prompt in or get it to answer some random question for me.”
@edhonour

“a prompt change or a model change may allow the model to break out of its guardrails that you've spent so much time setting up”
@edhonour

Resources

Links to the tools, products, repos or reading this entry mentions. Found by a web search of what the entry names, so you can go and use them.

Anyone can run this once. After that the links are saved on this entry and shown to everyone.

Remix this knowledge

Turn this entry into something you can post. Pick a format and get a fresh, original rewrite.

Remixes are AI generated, original wording, and free to reuse.

Talk it through

Open a round table on this entry and argue with other people about what it means.

Start a round table

Drop your own

Paste any video, podcast or audio link and get the transcript, the core ideas and the quotes back in one pass.

Try Drop