Generative AI Security and Privacy Guidelines

13 national cyber agencies from around the world, led by #ACSC, have collaborated on a guide for secure use of a range of "AI" technologies, and it is definitely worth a read! "Engaging with Artificial Intelligence" was written with collaboration from Australian Cyber Security Centre, along with the Cybersecurity and Infrastructure Security Agency (#CISA), FBI, NSA, NCSC-UK, CCCS, NCSC-NZ, CERT NZ, BSI, INCD, NISC, NCSC-NO, CSA, and SNCC, so you would expect this to be a tome, but it's only 15 pages! It is refreshing to see that the article is not solely focused on LLMs (eg. ChatGPT), but defines Artificial Intelligence to include Machine Learning, Natural Language Processing, and Generative AI (LLMs), while acknowledging there are other sub-fields as well. The challenges identified (with actual real-world examples!) are: 🚩 Data Poisoning of an AI Model: manipulating an AI model's training data, leading to incorrect, biased, or malicious outputs 🚩 Input Manipulation Attacks: includes prompt injection and adversarial examples, where malicious inputs are used to hijack AI model outputs or cause misclassifications 🚩 Generative AI Hallucinations: generating inaccurate or factually incorrect information 🚩 Privacy and Intellectual Property Concerns: challenges in ensuring the security of sensitive data, including personal and intellectual property, within AI systems 🚩 Model Stealing Attack: creating replicas of AI models using the outputs of existing systems, raising intellectual property and privacy issues The suggested mitigations include generic (but useful!) cybersecurity advice as well as AI-specific advice: 🔐 Implement cyber security frameworks 🔐 Assess privacy and data protection impact 🔐 Enforce phishing-resistant multi-factor authentication 🔐 Manage privileged access on a need-to-know basis 🔐 Maintain backups of AI models and training data 🔐 Conduct trials for AI systems 🔐 Use secure-by-design principles and evaluate supply chains 🔐 Understand AI system limitations 🔐 Ensure qualified staff manage AI systems 🔐 Perform regular health checks and manage data drift 🔐 Implement logging and monitoring for AI systems 🔐 Develop an incident response plan for AI systems This guide is a great practical resource for users of AI systems. I would interested to know if there are any incident response plans specifically written for AI systems - are there any available from a reputable source?

If you are an organisation using AI or you are an AI developer, the Australian privacy regulator has just published some vital information about AI and your privacy obligations. Here is a summary of the new guides for businesses published today by the Office of the Australian Information Commissioner which articulate how Australian privacy law applies to AI and set out the regulator’s expectations. The first guide is aimed to help businesses comply with their privacy obligations when using commercially available AI products and help them to select an appropriate product. The second provides privacy guidance to developers using personal information to train generative AI models. GUIDE ONE: Guidance on privacy and the use of commercially available AI products Top five takeaways * Privacy obligations will apply to any personal information input into an AI system, as well as the output data generated by AI (where it contains personal information).  * Businesses should update their privacy policies and notifications with clear and transparent information about their use of AI * If AI systems are used to generate or infer personal information, including images, this is a collection of personal information and must comply with APP 3 (which deals with collection of personal info). * If personal information is being input into an AI system, APP 6 requires entities to only use or disclose the information for the primary purpose for which it was collected. * As a matter of best practice, the OAIC recommends that organisations do not enter personal information, and particularly sensitive information, into publicly available generative AI tools. GUIDE 2: Guidance on privacy and developing and training generative AI models Top five takeaways * Developers must take reasonable steps to ensure accuracy in generative AI models. * Just because data is publicly available or otherwise accessible does not mean it can legally be used to train or fine-tune generative AI models or systems.. * Developers must take particular care with sensitive information, which generally requires consent to be collected. * Where developers are seeking to use personal information that they already hold for the purpose of training an AI model, and this was not a primary purpose of collection, they need to carefully consider their privacy obligations. * Where a developer cannot clearly establish that a secondary use for an AI-related purpose was within reasonable expectations and related to a primary purpose, to avoid regulatory risk they should seek consent for that use and/or offer individuals a meaningful and informed ability to opt-out of such a use. https://lnkd.in/gX_FrtS9

As organizations transition from pilots to enterprise-wide deployment of Generative and Agentic AI, it's crucial to recognize that GAI risks differ significantly from traditional software risks. Towards that, it is important to go back to basics and this publication from 2024 by National Institute of Standards and Technology (NIST)'s Generative AI Profile does a great job! 🌐 Here are the four highest-impact risks and the mitigation actions every organization should implement:- 1. Systemic Risk: Algorithmic Monocultures & Ecosystem-Level Failures When multiple industries depend on the same foundation models, a single unexpected model behavior can lead to correlated failures across the ecosystem. ⚡ Mitigation: - - Build model diversity and avoid single-model dependencies. - Maintain fallback systems and contingency workflows. - Apply stress tests that simulate sector-wide shocks. 2. Human-Originating Risks (Misuse, Over-Trust, Manipulation) Many GAI incidents stem from human behavior, including misuse, over-reliance, indirect prompt injection, and flawed assumptions. ⚡ Mitigation:- - Implement continuous user education on limitations and safe use. - Enforce access controls, privilege separation, and plugin vetting. - Maintain audit trails and logging to identify misuse early. 3. Content Integrity Risks (Hallucinations, Synthetic Media, Provenance Failure) GAI increases the scale and believability of fabricated content, from medical misinformation to deepfake-enabled harms. ⚡ Mitigation:- - Invest in content provenance, watermarking, and metadata tracking. - Require pre-deployment testing for hallucination profiles across contexts. - Use cross-model verification before high-stakes outputs are acted upon. 4. Security Risks (Prompt Injection, Data Leakage, Model Extraction) NIST highlights increasingly sophisticated attack surfaces unique to LLMs: indirect prompt injection, data extraction, and plugin-initiated compromise. ⚡ Mitigation:- - Apply secure-by-design reviews for all LLM integration points. - Red-team regularly using GAI-specific attack methods. - Log inputs/outputs via incident-ready documentation so breaches can be traced. 🔐 The bottom line:- AI risk management is not a technical afterthought, it is now a core capability. Organizations that operationalize governance, provenance, testing, and incident disclosure (NIST’s four focus pillars) will be the ones that deploy AI safely and at scale. 💬 If you’d like to explore Gen AI and Agentic AI risks, practical mitigation strategies, or how to operationalize the NIST AI RMF for your organization, feel free to comment or DM. Let’s build safer AI systems together! #AI #GenAI #AIGovernance #NIST #AIRMF #RiskManagement #AITrust #ResponsibleAI #AILeadership

In January 2024, the National Institute of Standards and Technology (NIST) published its updated report on AI security, called "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations," which now includes a focus on the security of generative AI, addressing attacks on both predictive and generative AI systems. This comprehensive work categorizes various adversarial attack methods, their objectives, and capabilities, along with strategies for their mitigation. It can help put NIST’s AI Risk Management Framework into practice. Attacks on predictive AI systems (see screenshot #1 below): - The report breaks down predictive AI taxonomy into classifications based on attack stages, goals, capabilities, knowledge, and data modality. - Key areas of focus include evasion and poisoning attacks, each with specifics on white-box and black-box attacks, their transferability, and mitigation strategies. - Privacy attacks are dissected into data reconstruction, membership inference, model extraction, and property inference, with proposed mitigations. Attacks on generative AI systems (see screenshot #2 below): - The section on Generative AI Taxonomy from the NIST report outlines attack classifications and specific vulnerabilities within Generative AI systems such as Generative Adversarial Networks (GANs), Generative Pre-trained Transformers (GPTs), and Diffusion Models. - It then delves into the evolution of Generative AI stages of learning, highlighting the shift from traditional models to the pre-training of foundation models using unsupervised learning to capture patterns for downstream tasks. These foundation models are subsequently fine-tuned for specific applications, often by third parties, making them particularly vulnerable to poisoning attacks, even with minimal tampering of training datasets. - The report further explores the deployment phase of generative AI, which exhibits unique vulnerabilities distinct from predictive AI. Notably, it identifies the potential for attackers to exploit data channels for injection attacks similar to SQL injection, the manipulation of model instructions to align LLM behaviors, enhancements through contextual few-shot learning, and the ingestion of runtime data from external sources for application-specific context. - Additionally, it addresses novel security violations specific to Generative AI and details various types of attacks, including AI supply chain attacks, direct and indirect prompt injection attacks, and their mitigations, as well as violations like availability, integrity, privacy compromises, and abuse. For a deeper dive into these findings, including the taxonomy of attacks and their mitigations, visit the full report available at: https://lnkd.in/guR56reH Co-authored by Apostol Vassilev (NIST), Alina Oprea (Northeastern University), Alie Fordyce, and Hyrum Anderson (both from Robust Intelligence) #NIST #aisecurity

AI success isn’t just about innovation - it’s about governance, trust, and accountability. I've seen too many promising AI projects stall because these foundational policies were an afterthought, not a priority. Learn from those mistakes. Here are the 16 foundational AI policies that every enterprise should implement: ➞ 1. Data Privacy: Prevent sensitive data from leaking into prompts or models. Classify data (Public, Internal, Confidential) before AI usage. ➞ 2. Access Control: Stop unauthorized access to AI systems. Use role-based access and least-privilege principles for all AI tools. ➞ 3. Model Usage: Ensure teams use only approved AI models. Maintain an internal “model catalog” with ownership and review logs. ➞ 4. Prompt Handling: Block confidential information from leaking through prompts. Use redaction and filters to sanitize inputs automatically. ➞ 5. Data Retention: Keep your AI logs compliant and secure. Define deletion timelines for logs, outputs, and prompts. ➞ 6. AI Security: Prevent prompt injection and jailbreaks. Run adversarial testing before deploying AI systems. ➞ 7. Human-in-the-Loop: Add human oversight to avoid irreversible AI errors. Set approval steps for critical or sensitive AI actions. ➞ 8. Explainability: Justify AI-driven decisions transparently. Require “why this output” traceability for regulated workflows. ➞ 9. Audit Logging: Without logs, you can’t debug or prove compliance. Log every prompt, model, output, and decision event. ➞ 10. Bias & Fairness: Avoid biased AI outputs that harm users or breach laws. Run fairness testing across diverse user groups and use cases. ➞ 11. Model Evaluation: Don’t let “good-looking” models fail in production. Use pre-defined benchmarks before deployment. ➞ 12. Monitoring & Drift: Models degrade silently over time. Track performance drift metrics weekly to maintain reliability. ➞ 13. Vendor Governance: External AI providers can introduce hidden risks. Perform security and privacy reviews before onboarding vendors. ➞ 14. IP Protection: Protect internal IP from external model exposure. Define what data cannot be shared with third-party AI tools. ➞ 15. Incident Response: Every AI failure needs a containment plan. Create a “kill switch” and escalation playbook for quick action. ➞ 16. Responsible AI: Ensure AI is built and used ethically. Publish internal AI principles and enforce them in reviews. AI without policy is chaos. Strong governance isn’t bureaucracy - it’s your competitive edge in the AI era. 🔁 Repost if you're building for the real world, not just connected demos. ➕ Follow Nick Tudor for more insights on AI + IoT that actually ship.

The UK's Department for Science, Innovation and Technology published a Code of Practice focused specifically on the #cybersecurity of AI. This voluntary Code of Practice takes into consideration that #AI poses security risks different from software, like data poisoning, model obfuscation, indirect prompt injection and operational differences associated with data management. The Code outlines 13 principles separated into five phases: Secure Design 1. Raise awareness of #artificialintelligence security threats and risks. 2. Design your AI system for security as well as functionality and performance. 3. Evaluate the threats and manage the risks to your #AIsystem.  4. Enable human responsibility for AI systems. Secure Development 5. Identify, track and protect your assets. 6. Secure your infrastructure. 7. Secure your #supplychain.  8. Document your data, models, and prompts.  9. Conduct appropriate testing and evaluation.   Secure Deployment 10. Communication and processes associated with End-users and Affected Entities. Secure Maintenance 11. Maintain regular security updates, patches, and mitigations. 12. Monitor your system’s behavior.   Secure End of Life 13. Ensure proper data and model disposal. Even better than the Code is the Implementation Guide to help organizations understand how to meet each provision. The Guide also has examples based on different scenarios of use like a #chatbot app, ML fraud detection, #LLM provider or open-access LLM. 

As lawyers, confidentiality isn’t just important—it’s foundational. Yet as we increasingly integrate generative AI and other tech tools into our practice, maintaining that confidentiality demands attention. Let’s walk through exactly what you need to know to keep client data secure, even as you harness powerful new technologies. 1️⃣ Know Where Your Data Goes Before using any AI-powered tool, clarify how your data is stored, who can access it, and whether it’s shared with third-party services. 2️⃣ Evaluate Data Security Practices Robust data security is non-negotiable. Always verify a vendor’s data security certifications, encryption standards, and access controls. 3️⃣ Limit and Control Your Data Inputs Use only the data you truly need to provide. The more data you input, the higher the confidentiality risk. 4️⃣ Use Built-in Privacy Controls Many reputable AI tools offer privacy modes or confidential environments that ensure your inputs won’t be used for model training or seen by unauthorized personnel. 5️⃣ Regularly Audit and Review Integrating technology is never a “set it and forget it” scenario. Regularly review and audit your chosen tools and their privacy compliance. - I’m Joe Regalia, a law professor and legal writing trainer. Follow me and tap the 🔔 so you won't miss any posts.

Enterprise AI: Governance, Risk, and Compliance Post topic: The Compliance Checklist for Deploying Generative AI As Generative AI adoption accelerates across enterprises, compliance and governance must move from afterthought to core design principle. Here’s a real-world checklist every enterprise team should review before going live with GenAI tools: - Data Privacy & Sovereignty Ensure PII, PHI, and sensitive business data are handled per GDPR, HIPAA, CCPA, and relevant local regulations. Consider regional model hosting where required. - Model Transparency & Explainability Can you explain why the model made a specific decision or output? Regulators and auditors will ask — so should your compliance team. - Human Oversight & Intervention Build workflows for humans to validate critical decisions made by GenAI, especially in regulated industries like finance, healthcare, and legal. - Bias, Fairness & Discrimination Testing Continuously test and document efforts to reduce bias, especially in hiring, lending, diagnostics, or any context where fairness is critical. - Audit Logging & Version Control Maintain logs of prompts, responses, model versions, fine-tuning data, and user interactions. This helps with accountability and rollback during investigations. - Third-Party Risk Management Review contracts and security posture of model vendors (e.g., OpenAI, Anthropic, Azure OpenAI). Check for SLAs, data retention, and liability clauses. - Security & Red-Teaming Simulate attacks like prompt injection, data leakage, jailbreaks. Treat GenAI as a new attack surface that needs constant testing and hardening. - IP & Content Use Policies Ensure generated outputs don’t infringe on copyrights or misuse licensed materials. Define enterprise-wide guidelines for employee use. - Acceptable Use & Internal Policy Enforcement Create clear policies: What tools can be used? For what purposes? By whom? How is employee prompt data used or retained? - Alignment with Responsible AI Principles Align your deployment with your org’s ethical principles around transparency, inclusion, trust, and accountability. ** Final Thought: You don’t need to solve everything at once — but you do need a clear plan, owners, and controls in place before you scale.   #EnterpriseAI #AIGovernance #GenAI #AICompliance #ResponsibleAI #RiskManagement #DataPrivacy #AIPlaybook #CIO #CTO #AIrisk #AIchecklist   Antonio Grasso Antonio Figueiredo Faisal Khan Dr. Ludwig Reinhard Rakesh Darge Fauzia I. Abro Adithyaa Vaasen Aditya Ramnathkar Richard Sturman Phil Fawcett Thorsten L. Taysser Gherfal Faisal Fareed Andy Jiang Khaliq Malik Sara Sanford, Rashim Mogha, Rahil Harihar, Jake George, Gaukhar Zharkeyeva

AI + Privacy New Consumer Report titled "Artificial Intelligence Policy Recommendations" Key Recommendations: Transparency 🔍 Companies must disclose when algorithms are used for important decisions like loans, rentals, promotions, or rate changes. 📝 Companies must explain adverse algorithmic decisions clearly, including how to improve outcomes. Complex unexplainable tools shouldn't be used. 🔬 Algorithm developers must provide access to vetted researchers to understand how tools work and their limitations. ⚖️ Companies must substantiate claims made when marketing their AI products. Fairness 🚫 Algorithmic discrimination should be prohibited, with clarification on how civil rights laws apply to AI development and deployment. 🧪 Independent testing for bias and accuracy should be required before and after deployment of consequential decision-making tools. 🏆 Big Tech shouldn't use AI to unfairly preference their own products when it harms competition. Privacy 📊 Companies should minimize data collection to only what's necessary for requested services. 🔒 Personal data collected by generative AI tools shouldn't be sold or shared with third parties. 👁️ Remote biometric tracking in public spaces should be banned with limited exceptions. Safety 📋 Companies creating consequential or risky tools must conduct risk assessments and make necessary changes. 🗣️ Whistleblower protections are needed for those exposing AI problems that companies won't disclose. ⚠️ Clarify liability for developers who fail to prevent harmful AI uses and unintended consequences. Enforcement + Government Capacity 💰 The FTC and state regulators need additional resources to oversee companies effectively. ⚡ Create legal pathways for individuals harmed by biased algorithms to seek justice when enforcement agencies lack capacity. https://lnkd.in/eHfnJn2C

How do we scale Generative AI without compromising ethics, sustainability, or data integrity? Here are my ten principles: 🔹 Strong Data Foundation: Ensure clean, reliable, and well-structured data to build effective AI systems. 🔹 Bias Mitigation: AI must fairly represent all voices through diverse datasets and rigorous testing. 🔹 Energy Efficiency: Consider the full environmental footprint—carbon, water, and energy consumption—to minimize AI’s impact. 🔹 Transparency: Explainable AI is key to earning user trust by making decisions understandable. 🔹 Data Privacy: Privacy-first design must be prioritized to respect users’ growing data concerns. 🔹 Human Oversight: AI should enhance human judgment, with human-in-the-loop systems ensuring responsible outcomes. 🔹 Guardrails: Implement ethical guardrails to prevent misuse and ensure AI aligns with societal values. 🔹 Collaboration with Regulators: Work closely with regulators like the EU AI Act to ensure compliance and trust. 🔹 Continuous Monitoring and Auditing: Regularly audit AI systems to catch biases and inefficiencies, ensuring ongoing alignment with ethical goals. 🔹 Inclusive Development: Diverse, inclusive teams bring varied perspectives, helping avoid blind spots and foster fair AI. These principles offer a roadmap for scaling AI that is both innovative and responsible, ensuring a balance between growth and ethical standards. #ai #generativeai #responsibleai #genai #ethicalai