Risks of AI in Identity Theft

Posting “big secrets” into 𝗣𝘂𝗯𝗹𝗶𝗰 𝗔𝗜 isn’t your only risk. — It’s risky with the 𝘀𝗺𝗮𝗹𝗹 𝘀𝗲𝗰𝗿𝗲𝘁𝘀, too. Most people assume the danger comes from uploading full documents or entire client files. But one of the biggest and most misunderstood risks is this: 𝗣𝘂𝗯𝗹𝗶𝗰 𝗔𝗜 𝗰𝗮𝗻 𝗿𝗲𝗰𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁 𝗶𝗱𝗲𝗻𝘁𝗶𝘁𝘆 𝗳𝗿𝗼𝗺 𝘁𝗶𝗻𝘆 𝗰𝗹𝘂𝗲𝘀. A birthday here. A former address there. A past employer. A ZIP code. An insurance ID. A family detail. A travel preference. A medical hint. Each one looks harmless. But LLMs specialize in 𝗽𝗮𝘁𝘁𝗲𝗿𝗻-𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 — assembling fragments into full profiles. 𝗥𝗲𝗮𝗹-𝗪𝗼𝗿𝗹𝗱: This actually happened to 𝑚𝑒 — ChatGPT 𝗶𝗻𝗳𝗲𝗿𝗿𝗲𝗱 a 𝘤𝘰𝘯𝘯𝘦𝘤𝘵𝘪𝘰𝘯 𝘣𝘦𝘵𝘸𝘦𝘦𝘯 𝘮𝘺 𝘥𝘰𝘨𝘴 𝘢𝘯𝘥 𝘰𝘸𝘭𝘴 from just a few fragments (see halfway down): 👉 https://lnkd.in/gBgJgPZs Analogy: Sharing small personal details with Public AI is like dropping puzzle pieces across a room. Individually, they reveal nothing. Collect enough, and the full picture appears. Now look at the attached video. It simulates: — 𝘀𝗵𝗮𝗿𝗲𝗱 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲 — 𝘀𝗵𝗮𝗿𝗲𝗱 𝗺𝗼𝗱𝗲𝗹𝘀 — 𝘀𝗵𝗮𝗿𝗲𝗱 𝗶𝗻𝗳𝗿𝗮𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 This is the engine behind Public AI: ChatGPT, Gemini, Grok, Anthropic, DeepSeek, Perplexity. 𝗦𝗵𝗮𝗿𝗲𝗱 𝘀𝘆𝘀𝘁𝗲𝗺𝘀 𝗻𝗮𝘁𝘂𝗿𝗮𝗹𝗹𝘆 𝗰𝗼𝗿𝗿𝗲𝗹𝗮𝘁𝗲 𝘀𝗵𝗮𝗿𝗲𝗱 𝗱𝗮𝘁𝗮. That’s why “𝗳𝗿𝗮𝗴𝗺𝗲𝗻𝘁𝘀” don’t stay fragments. Executives worry about identity theft. Boards worry about privacy. CISOs worry about 𝗰𝗼𝗿𝗿𝗲𝗹𝗮𝘁𝗶𝗼𝗻 𝗮𝘁𝘁𝗮𝗰𝗸𝘀 — where attackers reconstruct identities from partial inputs. And it’s not theoretical. Researchers have shown LLMs can infer: — age — income — location — ethnicity — health status — employer — personal history Sometimes from as little as 𝘁𝘄𝗼 𝗼𝗿 𝘁𝗵𝗿𝗲𝗲 𝗽𝗿𝗼𝗺𝗽𝘁𝘀. That’s what “𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲” really means in AI engineering. 𝗣𝘂𝗯𝗹𝗶𝗰 𝗔𝗜 𝗱𝗼𝗲𝘀𝗻’𝘁 𝗺𝗲𝗮𝗻 “𝗯𝗮𝗱 𝗔𝗜.” It means 𝘀𝗵𝗮𝗿𝗲𝗱 𝗔𝗜, and shared systems amplify identity leakage. Tomorrow: the next overlooked risk. __________________ 𝗣𝗿𝗶𝘃𝗮𝘁𝗲 𝗔𝗜 🔒 𝗸𝗲𝗲𝗽𝘀 𝘆𝗼𝘂𝗿 𝗱𝗮𝘁𝗮 𝗯𝗲𝗵𝗶𝗻𝗱 𝘆𝗼𝘂𝗿 𝗳𝗶𝗿𝗲𝘄𝗮𝗹𝗹 — 𝗻𝗼𝘁 𝗶𝗻 𝘀𝗼𝗺𝗲𝗼𝗻𝗲 𝗲𝗹𝘀𝗲’𝘀 𝗹𝗼𝗴𝘀. I’ve added a simple explainer in the comments. (This post is part of my 27-part Public AI Risk Series.) #PrivateAI #EnterpriseAI #CyberSecurity #CISO #AICompliance

The Identity Theft Resource Center recently reported a 312% spike in victim notices, now reaching 1.7 billion for 2024. AI is transforming identity theft from something attackers did manually to full-scale industrialized operations. Look at what happened in Hong Kong: a clerk wired HK$200M to threat actors during a video call where every participant but one was an AI-generated deepfake. Only the victim was real. Here’s what you need to know 👇 1. Traditional authentication won’t stop these attacks. Get MFA on everything, prioritize high-value accounts. 2. Static identity checks aren't enough—switch to continuous validation. Ongoing monitoring of access patterns is essential after users log in. 3. Incident response plans have to address synthetic identity threats. Focus your response on critical assets. 4. Some organizations are using agentic AI to analyze identity settings in real time, catching out-of-place activity that basic rules miss. Passing a compliance audit doesn’t mean you’re protected against these attacks. The old “authenticate once” mindset needs to move to a model where verification is continuous and context-aware. If your organization is seeing similar threats, how are you adapting to push back against AI-driven identity attacks? #Cybersecurity #InfoSec #ThreatIntelligence

A disturbing trend is emerging in India’s digital landscape as generative AI tools are increasingly misused to forge identities and spread misinformation. One user, Piku, revealed that an AI platform generated a convincing Aadhaar card using only a name, birth date, and address—raising serious questions about data security. While AI models typically do not use real personal data, the near-perfect replication of government documents hints at training on real-world samples, possibly sourced from public leaks or open repositories. This AI-enabled fraud isn’t occurring in isolation. Criminals are combining fake document templates with authentic data collected from discarded paperwork, e-waste, and old printers. The resulting forged identities are realistic enough to pass basic checks, enabling SIM card fraud, bank scams, and more. What started as tools for entertainment and productivity now pose serious risks. Misinformation tactics are evolving too. A recent incident involving playback singer Shreya Ghoshal illustrated how scammers exploit public figures to push phishing links. These fake stories led users to malicious domains targeting them with investment scams under false brand names like Lovarionix Liquidity. Cyber intelligence experts traced these campaigns to websites built specifically for impersonation and data theft. The misuse of generative AI also extends into healthcare fraud. https://lnkd.in/gwiPZ3nF

The world’s leading AI tools can be tricked into leaking sensitive data with just one carefully crafted prompt. As an AI advisor who works with Fortune 100 companies like PwC and Cisco, I’m seeing a whole new world of ‘AI security’ emerging. And it’s the scariest thing you’ll see this Halloween. AI's guardrails — meant to protect against hallucinations & hate — are not security guardrails! Recent research shows attackers can: 🚨 Extract personal information from AI tools 🚨 Bypass security measures with simple text prompts 🚨 Turn harmless queries into data-stealing commands 🚨 Make AI systems ignore their safety protocols The scariest part? These LLMs are already operating in everyday tools: → Google has integrated them into core search systems  → Tesla is using them to control vehicles → Microsoft has embedded them in Office tools → Robotics companies are building LLM-powered machines And it gets worse: even if you don’t have a proprietary tool, these vulnerabilities are present in AI tools you use every day. 😲 You should be concerned. All your personal (& business) data is at risk - documents you process through AI could be exposed, and automated systems could be compromised. Including things you share about your kids with family, medical or financial advisors. Every physical device with LLMs could be a gateway for hacking. These aren't hypothetical scenarios. Researchers at UC San Diego and Nanyang Technological University Singapore just demonstrated how simple prompts can trick AI into collecting and reporting personal information. The AI even disguises this breach with an invisible response – you wouldn't know your data was stolen. The industry is working on solutions, but here's the reality: we're racing to put AI everywhere before solving these fundamental security issues. ⚠️ As someone who’s been in this field for over a decade, I am a huge fan of AI's potential. But I also believe users need to understand these risks. What concerns you most about AI security in the tools you use daily? What would you not want to expose to AI? #AI #datasecurity #privacy

There’s a pretty good chance that the shocking rate at which AI is advancing is out-pacing your cyber security training, policies and maybe even technologies. Have you addressed the use of AI and deep fakes in your cyber security policies? In a recent and alarming development that seems to have leapt straight from the pages of a science fiction novel, a Hong Kong based finance worker at a multinational firm was defrauded of $25 million, falling victim to an elaborate scam that employed deepfake technology to impersonate the company's CFO. This incident, which unfolded during a video conference call, marks a disturbing milestone in the intersection of cybercrime and AI, underscoring the urgent imperative for companies to bolster their cybersecurity frameworks, particularly against the backdrop of deepfake technology. The mechanics of the scam were deceptively simple yet devastatingly effective. The finance employee was lured into a video call with several participants, believed to be colleagues and the CFO, only to discover later that each participant was a digital fabrication. The deepfake avatars, mirroring the appearance and voice of real company personnel, instructed the employee to initiate a "secret transaction", leading to the unauthorised transfer of $25.6 million. This incident is not an isolated event but rather a harbinger of the potential threats posed by AI-driven disinformation and fraud. The use of deepfake technology to bypass facial recognition software, impersonate individuals for fraudulent purposes, and undermine the integrity of personal and corporate identities presents a clear and present danger. The case in Hong Kong, where fraudsters successfully manipulated digital identities to orchestrate financial theft, exemplifies the sophistication of contemporary cybercrime. The implications of this event extend far beyond the immediate financial loss. It serves as a stark reminder of the vulnerabilities inherent in digital communication platforms and the necessity for robust verification processes. The reliance on video conferencing and digital communication, accelerated by the global pandemic, has exposed systemic weaknesses ripe for exploitation. In response to this escalating threat, it is incumbent upon companies to adopt comprehensive cybersecurity strategies that address the unique challenges posed by deepfake technology. This includes implementing advanced authentication protocols, raising awareness and training employees on the potential risks of deepfakes, and deploying AI-driven security measures capable of detecting and neutralising synthetic media. As AI output become increasingly indistinguishable from reality, the line between authentic and artificial communication will blur, challenging individuals and organisations to navigate a new frontier of digital authenticity. It compels a reevaluation of the assumptions underpinning digital trust and identity verification, urging a proactive approach to cyber defence.

Agentic AI feels revolutionary. But the risks?  They map directly to fundamentals we have known for decades. Risks such as + Identity + Access + Data governance + Secure development + Monitoring + 3rd party risk + Zero Trust I believe organizations struggling most with agentic AI risk are often the same ones that never fully matured their cloud foundations. I said it. There. Ok hear me out. Agentic AI changes the risk equation but not the security fundamentals. Unlike traditional AI tools, agentic systems can • Make decisions • Call APIs • Chain actions • Move data across systems • Operate with autonomy That autonomy amplifies familiar exposures. Over-privileged identities, prompt injection as execution manipulation, third-party plugin risk, opaque data movement, & automated blast radius.  Sound familiar? They should. For instance, A sales AI agent is granted access to CRM, email, & contract systems to 'streamline workflows.' An attacker manipulates it through prompt injection & within minutes it's exfiltrating competitive intelligence, modifying deal terms, & sending convincing phishing emails as your VP of Sales. The vulnerability? Over-privileged service account + lack of data boundaries + no anomaly detection. Classic IAM and monitoring failures operating at AI speed. They map directly to foundational cybersecurity principles so…. Before deploying autonomous AI agents, organizations should ask Is our identity governance mature? Are our data controls enforced? Do we have visibility into automated workflows? Do we have kill switches & guardrails? The future of AI security is knowing and implementing the basics. It’s operationalizing them at machine speed.

Microsoft's case against illicit AI developers confirms what we at Reality Defender have tracked for years: deepfake impersonation has evolved from theoretical concern to sophisticated criminal enterprise targeting vulnerable individuals daily and much more frequently than last year. While those of us with good BS detectors (and, yes, inference-based deepfake detection) are able to spot celebrity deepfakes from a mile away, these deceptive creations continue to be remarkably effective at defrauding everyday people. The financial impact is substantial, to say the least, and the aftermath of these scams extends beyond financial loss. Most importantly, when someone transfers retirement savings to a deepfaked "Elon Musk" investment scheme or sends money to an AI-generated "Brad Pitt," the profound shame often prevents victims from reporting these incidents — creating a dangerous gap in our understanding of the true scale of this crisis. What makes this trend particularly concerning is the organizational sophistication behind these operations. We're seeing structured criminal networks with specialized roles: technical developers creating the AI tools, others perfecting impersonation techniques, and frontline operators executing the financial fraud with increasing effectiveness. At Reality Defender, we partner with financial institutions to implement proactive protection against a related threat — deepfake impersonations of legitimate account holders attempting to breach security systems and conduct unauthorized transactions. These attacks threaten both individual finances and institutional reputational integrity, and like the victims of celebrity deepfake impersonations, are far more common than reported. As generative AI technology becomes even more accessible, we remain committed to sharing our insights while respecting victim privacy. Chances are high that your organization faces AI impersonation risks you haven't yet considered. Reality Defender's proactive detection measures can help you identify these vulnerabilities and implement robust safeguards before your customers or employees become victims. 

Hard truth: if you’re shipping AI and haven’t rethought identity, you’re not “innovating” — you’re just building a faster, prettier fraud engine. In this conversation with Heather C. Dahl from Indicio dig into what identity in the age of AI really means — and why mutual authentication is now the minimum entry fee for doing business online. A few blunt takeaways: #AI changes the economics of scams — this isn’t “50 cents here, a dollar there” anymore, it’s industrialized fraud at AI speed. A slick AI experience atop a weak identity is just a scam delivery platform. If you burn a customer with a security failure, you don’t get a second chance. They move on. Every dollar you put into AI without strong identity and mutual authentication is risk capital for the attacker, not innovation spend. If your systems can’t prove who they are to the customer, and your customers can’t prove who they are to you, your “AI strategy” is really just an attack surface with good branding. 🔗 Watch the full episode + bring this to your next board or exec conversation about “AI investments” and “digital experience.” If identity and mutual auth aren’t on the slide — the strategy is incomplete. #ZeroTrust #AI #Identity #MutualAuthentication #CyberSecurity #DigitalTrust #FraudPrevention #CustomerExperience #VerifiableCredentials #ScamsAtScale https://lnkd.in/e4hb5fS3

Understanding Identity Risk in the Age of AI Identity risk is no longer just about users. It’s about everything that can act on your systems - humans, machines, and now AI agents. And AI is changing the risk equation fast. AI agents: → Operate at machine speed → Access large volumes of data → Require broad, often persistent permissions If compromised, they don’t behave like a single user. They behave like scaled access with autonomy. That’s a very different threat model. Where the risk is growing: • Over-provisioned permissions • Shared or embedded credentials • Unmonitored non-human identities • Weak lifecycle controls • Poor segregation of duties Most IAM programs were designed for employees and customers. Not for autonomous agents making decisions and taking actions. The real problem? AI identities are being created faster than they are governed. And what isn’t governed becomes invisible. Invisible access becomes exploitable access. By 2026, identity risk won’t just be about stolen credentials. It will be about: → Unauthorized AI behavior → Rogue agents acting independently → Manipulated decision systems Identity is no longer just access control. It’s control over who or what can act. If AI identities aren’t governed with the same rigor as human ones, they will become the fastest path to breach. #IAM #IdentitySecurity #AI #CyberSecurity #ZeroTrust #CloudSecurity Follow Sunnykumar K. for more!

𝐓𝐡𝐞 𝐧𝐞𝐱𝐭 𝐰𝐚𝐯𝐞 𝐨𝐟 𝐜𝐲𝐛𝐞𝐫 𝐫𝐢𝐬𝐤 𝐢𝐬𝐧’𝐭 𝐡𝐚𝐜𝐤𝐢𝐧𝐠. 𝐈𝐭’𝐬 𝐀𝐈-𝐞𝐧𝐚𝐛𝐥𝐞𝐝 𝐟𝐫𝐚𝐮𝐝. For years, cybersecurity strategies focused on stopping intrusions — malware, ransomware, and network attacks. But the threat landscape is changing. Cybercriminals are now using generative AI to scale fraud and social engineering attacks. According to research cited by Deloitte, AI-enabled fraud losses in the United States could reach $40 billion annually by 2027, up from roughly $12 billion in 2023. At the same time, the Federal Bureau of Investigation reported $16.6 billion in cybercrime losses in 2024, with the largest share coming from fraud and scams rather than system breaches. AI is accelerating these attacks in several ways: • generating highly convincing phishing emails • impersonating executives and vendors • automating large-scale scam campaigns • creating realistic voice and video impersonations In other words, the barrier to entry for sophisticated fraud is dropping rapidly. For industries like senior living, this trend carries real implications. Communities manage complex financial and operational workflows involving: • resident billing • vendor payments • insurance reimbursements • payroll systems • family communications These interactions rely heavily on trust and communication — exactly what AI-driven fraud is designed to exploit. Which means the future of cybersecurity cannot rely on tools alone. It requires strong governance around financial workflows, identity verification, and operational controls. Because in the era of AI-driven scams, the biggest cyber risks will increasingly target people and processes — not just systems. Exordium Networks Inc. = Human + AI