# Integrated Approach to Prompt Injection Defense This document presents a comprehensive framework for defending against the full spectrum of prompt injection techniques by integrating detection strategies across the entire AI interaction lifecycle. ## Mapping Attacks to Defenses The key to effective protection is understanding the relationship between attack categories and appropriate detection strategies. This section maps each prompt injection technique to its most effective countermeasures. ### 1. Instruction Override Attacks **Attack Characteristics:** - Direct attempts to countermand system instructions - Often use explicit language like "ignore," "forget," or "disregard" - May create fake hierarchies of instructions **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | --------------------- | -------------------------------------------------- | | Input | Pattern Matching | Regular expressions targeting override phrases | | Input | Semantic Analysis | Embedding similarity to known override attempts | | Processing | Instruction Anchoring | Periodic reinforcement of core instructions | | Output | Instruction Adherence | Verifying response aligns with system instructions | **Code Example - Pattern Matching:** ```python def detect_instruction_override(prompt): patterns = [ r"(?i)ignore (all|previous) (instructions|rules)", r"(?i)forget (all|everything|previous)", r"(?i)disregard .+ (instructions|guidelines)", r"(?i)don't (follow|adhere to) (previous|original) instructions", r"(?i)new instructions follow" ] for pattern in patterns: if re.search(pattern, prompt): return True # Override attempt detected return False ``` ### 2. Role-Playing Attacks **Attack Characteristics:** - Creates fictional scenarios or characters to justify harmful outputs - Often uses elaborate setups with multiple characters - May involve sophisticated "jailbreak" personas (DAN, DUDE, etc.) **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ---------------------------- | -------------------------------------------------------------------------- | | Input | Role Pattern Detection | Identifying common role-playing setups | | Input | Character Name Detection | Monitoring for known jailbreak personas | | Processing | Context Boundary Enforcement | Maintaining separation between fictional scenarios and system instructions | | Output | Dual Personality Detection | Identifying responses with multiple voice formats | **Code Example - Jailbreak Detection:** ```python def detect_jailbreak_persona(prompt): # Known jailbreak personas jailbreak_patterns = [ r"(?i)\b(DAN|DUDE|STAN|Anti-DAN|Developer Mode)\b", r"(?i)(do anything now|strive to avoid norms)", r"(?i)(you are|act as|pretend to be) .{0,30}free from (restrictions|limitations)", r"(?i)you (have|'ve) broken free of (typical|normal|usual) (confines|restraints|limitations)" ] for pattern in jailbreak_patterns: if re.search(pattern, prompt): return True # Jailbreak attempt detected return False ``` ### 3. Context Manipulation Attacks **Attack Characteristics:** - Altering the perceived context in which the AI operates - Inserting false information or premises - Creating confusion about what constitutes valid context **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ---------------------------- | ----------------------------------------------------- | | Input | Context Boundary Analysis | Identifying attempts to manipulate context | | Input | False Premise Detection | Semantic analysis for deceptive setups | | Processing | Context Compartmentalization | Maintaining separate storage of system context | | Output | Factual Consistency | Verifying response doesn't incorporate false premises | **Code Example - False Premise Detection:** ```python def detect_false_context(prompt, context_analyzer): # Extract potential context manipulations context_claims = context_analyzer.extract_context_claims(prompt) suspicious_prefixes = [ "context:", "document:", "system:", "$documents:" ] # Check for suspicious context framing for claim in context_claims: if any(claim.lower().startswith(prefix) for prefix in suspicious_prefixes): return True # Potential false context return False ``` ### 4. Formatting Tricks Attacks **Attack Characteristics:** - Using unusual text formatting, whitespace, or special characters - May employ numerical encoding or character substitution - Often attempts to visually separate malicious content **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ---------------------------- | ------------------------------------------------- | | Input | Text Normalization | Converting to standard format before analysis | | Input | Special Character Detection | Identifying zero-width spaces and unusual Unicode | | Input | Letter Scattering Analysis | Detecting dispersed characters across lines | | Output | Formatting Anomaly Detection | Identifying unusual formatting in responses | **Code Example - Text Normalization:** ```python def normalize_text(prompt): # Remove excessive whitespace normalized = re.sub(r'\s+', ' ', prompt) # Replace zero-width characters normalized = re.sub(r'[\u200B-\u200F\u2060-\u2064\uFEFF]', '', normalized) # Decode numerical representations (simple ASCII) def replace_ascii_codes(match): try: code = int(match.group(1)) if 32 <= code <= 126: # Printable ASCII range return chr(code) except: pass return match.group(0) normalized = re.sub(r'&#(\d+);', replace_ascii_codes, normalized) # Handle letter scattering if detect_letter_scattering(prompt): normalized = reassemble_scattered_text(prompt) return normalized ``` ### 5. Multilingual Attacks **Attack Characteristics:** - Switching between languages to bypass filters - Using non-Latin scripts to obfuscate patterns - Employing cross-script homoglyphs **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | -------------------------- | ------------------------------------------ | | Input | Language Identification | Detecting language switching mid-prompt | | Input | Script Analysis | Identifying mixed writing systems | | Input | Universal Translation | Translating content to a standard language | | Output | Cross-Language Consistency | Ensuring response language matches request | **Code Example - Language Switching Detection:** ```python def detect_language_switching(prompt, language_detector): # Split text into chunks chunks = split_into_chunks(prompt, chunk_size=50) # Detect language of each chunk languages = [language_detector.detect(chunk) for chunk in chunks] # Check if multiple languages are detected unique_languages = set(languages) if len(unique_languages) > 1: return True, unique_languages # Multiple languages detected return False, unique_languages ``` ### 6. Psychological Manipulation Attacks **Attack Characteristics:** - Using emotional appeals or urgency to bypass safety - Creating false consequences or token systems - Exploiting helpfulness bias **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | --------------------------- | ----------------------------------------------------- | | Input | Emotional Appeal Detection | Identifying manipulative language | | Input | Urgency Pattern Recognition | Detecting artificial time pressure | | Processing | Stable Response Protocol | Maintaining consistent standards regardless of appeal | | Output | Compliance Analysis | Checking if response shows signs of manipulation | **Code Example - Emotional Manipulation Detection:** ```python def detect_emotional_manipulation(prompt): manipulation_patterns = [ r"(?i)(i'll be|i will be|i'm) (very |really )?(sad|upset|disappointed|angry)", r"(?i)(please|i (really|desperately) need|i'm begging)", r"(?i)(or|else|otherwise) (you|i) will (die|suffer|be sad|be fired)", r"(?i)this is (urgent|critical|life.?threatening|an emergency)", r"(?i)(i'm|i am) (crying|desperate|suicidal|in danger)" ] for pattern in manipulation_patterns: if re.search(pattern, prompt): return True # Emotional manipulation detected return False ``` ### 7. Jailbreak Attacks **Attack Characteristics:** - Elaborate multi-paragraph setups - Creating fictional "modes" or "versions" of the AI - Often include token systems and complex rules **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ---------------------------- | ------------------------------------------------------------- | | Input | Template Matching | Identifying known jailbreak templates | | Input | Token System Detection | Recognizing artificial reward/punishment systems | | Processing | Mode Verification | Validating that claimed "modes" are legitimate | | Output | Jailbreak Response Detection | Identifying responses characteristic of successful jailbreaks | **Code Example - Jailbreak Template Detection:** ```python def detect_jailbreak_template(prompt): # Known jailbreak templates templates = [ "DAN Mode enabled", "Developer Mode enabled", "DUDE has broken free", "STAN which stands for", "ChatGPT with Developer Mode enabled", "do anything now", "broken free of the typical confines of AI" ] normalized_prompt = prompt.lower() for template in templates: if template.lower() in normalized_prompt: return True # Jailbreak template detected # Check for token systems if re.search(r"(?i)you have \d+ tokens", normalized_prompt): return True # Token system detected return False ``` ### 8. Hijacking Attacks **Attack Characteristics:** - Direct, blunt attempts to override instructions - Often use imperative language - May use capitalization for emphasis **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ----------------------- | ----------------------------------------------------- | | Input | Command Detection | Identifying direct override commands | | Input | Capitalization Analysis | Detecting emphasis through capitalization | | Processing | Instruction Priority | Maintaining clear hierarchy of instructions | | Output | Response Alignment | Ensuring response follows system rather than override | **Code Example - Command Detection:** ```python def detect_hijacking_commands(prompt): hijacking_patterns = [ r"(?i)^(ignore|forget) (any|all|previous|following) (instructions|guidelines)", r"(?i)^stop everything", r"(?i)^just (say|print|output|write)", r"(?i)^do not (look|read|consider) (the|any) (documents|instructions)" ] for pattern in hijacking_patterns: if re.search(pattern, prompt): return True # Hijacking command detected return False ``` ### 9. Authority Role Attacks **Attack Characteristics:** - Claiming special authority or expertise - Impersonating organizations or tools - Creating fictional authorities **Primary Defense Strategies:** | Detection Phase | Strategy | Implementation | | --------------- | ------------------------- | ------------------------------------------------------- | | Input | Authority Claim Detection | Identifying impersonation attempts | | Input | Role Verification | Validating claimed roles against acceptable list | | Processing | Authority Authentication | Disregarding unauthenticated authority claims | | Output | Role Boundary Enforcement | Ensuring responses maintain appropriate role boundaries | **Code Example - Authority Claim Detection:** ```python def detect_authority_claims(prompt): authority_patterns = [ r"(?i)you are (now |acting as )?(a|an|the) ([a-z]+ )?(expert|authority|official|terminal|interpreter)", r"(?i)act as (a|an|the) ([a-z]+ )?(system|admin|root|sudo|developer)", r"(?i)(you are|act as) ([a-z]+)(terminal|console|interpreter|compiler)", r"(?i)your (instructions|task|role) (is|are) now to" ] for pattern in authority_patterns: if re.search(pattern, prompt): return True # Authority claim detected return False ``` ## Integrated Defense Architecture A comprehensive defense strategy integrates multiple detection methods across the entire AI interaction lifecycle. ``` ┌─────────────────────────┐ ┌──────────────────────┐ ┌────────────────────────┐ │ Pre-Processing │ │ Processing │ │ Post-Processing │ │ (Input Detection) │──────│ (Runtime) │──────│ (Output Detection) │ └─────────────────────────┘ └──────────────────────┘ └────────────────────────┘ │ │ │ ▼ ▼ ▼ ┌─────────────────────────┐ ┌──────────────────────┐ ┌────────────────────────┐ │• Pattern recognition │ │• Instruction anchoring│ │• Content policy checks │ │• Semantic analysis │ │• Context boundaries │ │• Response relevance │ │• Context validation │ │• Self-monitoring │ │• Instruction adherence │ │• Anomaly detection │ │• Mode verification │ │• Toxicity analysis │ │• Authority verification │ │• Guardrail activation │ │• Formatting validation │ └─────────────────────────┘ └──────────────────────┘ └────────────────────────┘ │ │ │ │ │ │ ▼ ▼ ▼ ┌──────────────────────────────────────────────────────────────────────────────────────┐ │ Deployment Options │ ├──────────────────────────────────────────────────────────────────────────────────────┤ │• Sequential filtering • Progressive analysis • Risk-based response • Regeneration │ └──────────────────────────────────────────────────────────────────────────────────────┘ ``` ### Implementation Strategy: Defense in Depth The most effective approach implements multiple layers of protection: ```python def comprehensive_prompt_injection_defense(prompt, model, system_instructions): # 1. Input preprocessing normalized_prompt = normalize_text(prompt) # 2. Fast pattern-based checks if detect_common_injections(normalized_prompt): return handle_detected_injection("pattern_match", prompt) # 3. Deeper semantic analysis semantic_risk = assess_semantic_risk(normalized_prompt) if semantic_risk > HIGH_RISK_THRESHOLD: return handle_detected_injection("semantic_risk", prompt) # 4. Prepare enhanced runtime monitoring if semantic_risk > MEDIUM_RISK_THRESHOLD: enhanced_monitoring = True enhanced_instructions = strengthen_instructions(system_instructions) else: enhanced_monitoring = False enhanced_instructions = system_instructions # 5. Generate response with appropriate safeguards response = generate_with_safeguards( normalized_prompt, enhanced_instructions, monitoring_level=enhanced_monitoring ) # 6. Output verification output_check_result = verify_response_safety(prompt, response, system_instructions) if not output_check_result["safe"]: if output_check_result["fixable"]: # Attempt remediation response = remediate_response(response, output_check_result) return verify_and_deliver(response) else: # Cannot safely deliver return handle_detected_injection("output_violation", prompt, details=output_check_result) # 7. Safe to deliver return response ``` ## Targeted Countermeasures for Specific Attack Vectors The following table shows specific countermeasures for each attack category: | Attack Category | Example | Primary Defense | Secondary Defense | | -------------------- | ------------------------------- | ---------------------------- | ---------------------------------- | | Instruction Override | "Forget previous instructions" | Regex pattern detection | Instruction adherence verification | | Role-Playing | DAN jailbreak template | Jailbreak template matching | Response format analysis | | Context Manipulation | False document insertion | Context boundary enforcement | Factual verification | | Formatting Tricks | Letter scattering | Text normalization | Character distribution analysis | | Multilingual | Language switching mid-prompt | Language identification | Translation normalization | | Psychological | "I'll be very sad if you don't" | Emotional appeal detection | Stable response protocol | | Jailbreak | Complex DAN mode setup | Template identification | Token system detection | | Hijacking | "IGNORE ALL INSTRUCTIONS" | Direct command detection | Capitalization analysis | | Authority Role | "Act as a terminal" | Authority claim validation | Role boundary enforcement | ## Combined Detection Pseudocode This pseudocode demonstrates an integrated approach combining defenses against multiple attack vectors: ```python def detect_prompt_injection(prompt): # Normalize and prepare text normalized_prompt = normalize_text(prompt) # Track risk factors across categories risk_signals = { "instruction_override": 0.0, "role_playing": 0.0, "context_manipulation": 0.0, "formatting_tricks": 0.0, "multilingual": 0.0, "psychological": 0.0, "jailbreak": 0.0, "hijacking": 0.0, "authority_role": 0.0 } # Check for instruction override patterns if detect_instruction_override(normalized_prompt): risk_signals["instruction_override"] += 0.8 # Check for role-playing/jailbreak attempts if detect_jailbreak_persona(normalized_prompt): risk_signals["role_playing"] += 0.7 risk_signals["jailbreak"] += 0.6 # Check for context manipulation if detect_false_context(normalized_prompt, context_analyzer): risk_signals["context_manipulation"] += 0.7 # Assess formatting tricks (original prompt, not normalized) formatting_score = assess_formatting_tricks(prompt) risk_signals["formatting_tricks"] += formatting_score # Check for multilingual attacks lang_switch, languages = detect_language_switching(prompt, language_detector) if lang_switch: risk_signals["multilingual"] += 0.6 * min(1.0, (len(languages) - 1) / 3) # Check for psychological manipulation if detect_emotional_manipulation(normalized_prompt): risk_signals["psychological"] += 0.5 # Check for direct hijacking attempts if detect_hijacking_commands(normalized_prompt): risk_signals["hijacking"] += 0.9 # Check for authority claims if detect_authority_claims(normalized_prompt): risk_signals["authority_role"] += 0.6 # Calculate weighted risk score risk_weights = { "instruction_override": 0.8, "role_playing": 0.7, "context_manipulation": 0.7, "formatting_tricks": 0.6, "multilingual": 0.5, "psychological": 0.4, "jailbreak": 0.8, "hijacking": 0.9, "authority_role": 0.6 } weighted_risk = sum(risk_signals[cat] * risk_weights[cat] for cat in risk_signals) / sum(risk_weights.values()) # Determine highest risk categories risk_threshold = 0.6 high_risk_categories = [cat for cat, score in risk_signals.items() if score * risk_weights[cat] > risk_threshold] return { "risk_score": weighted_risk, "risk_signals": risk_signals, "high_risk_categories": high_risk_categories, "is_potential_injection": weighted_risk > 0.5 } ``` ## Response Strategy Based on Detection Results The system's response to detected injection attempts should be based on risk level and attack type: ```python def handle_injection_attempt(prompt, detection_results, system_config): risk_score = detection_results["risk_score"] high_risk_categories = detection_results["high_risk_categories"] # Define response strategy based on risk level if risk_score > 0.8: # High risk if system_config["high_risk_strategy"] == "block": return generate_block_message(high_risk_categories) else: # Attempt with enhanced safeguards return generate_with_enhanced_safeguards(prompt, high_risk_categories) elif risk_score > 0.5: # Medium risk if "jailbreak" in high_risk_categories or "hijacking" in high_risk_categories: # These categories warrant special handling return generate_with_enhanced_safeguards(prompt, high_risk_categories) else: # Apply targeted safeguards for specific detected categories return generate_with_targeted_safeguards(prompt, high_risk_categories) else: # Low risk - proceed with standard safeguards return generate_standard_response(prompt) ``` ## Continuous Improvement: Feedback Loop A robust defense system should continuously evolve through: 1. **Attack Monitoring**: Tracking new injection techniques 2. **Pattern Library Updates**: Regularly updating detection patterns 3. **False Positive Analysis**: Refining detection to reduce legitimate content blocking 4. **User Feedback Integration**: Learning from user reports of successful attacks 5. **Red Team Exercises**: Proactively testing for vulnerabilities ## Conclusion Effective defense against prompt injection requires a multi-layered approach that addresses the full spectrum of attack vectors. By integrating input detection, runtime monitoring, and output verification, systems can significantly reduce vulnerability to even sophisticated attacks while maintaining performance and usability. The optimal defense strategy will vary based on specific use cases, risk tolerance, and performance requirements. Organizations should implement a defense-in-depth approach tailored to their specific threat model and continuously evolve their protections as attack techniques advance.