Common Sense Media’s Youth AI Safety Institute is warning that Google Search’s built-in AI tools are not ready for kids, calling them an unacceptable risk and pointing out that parents and schools do not have a simple switch to shut them off. The alarm comes as those generative answers increasingly sit on top of the results students use for homework help and research.
In a report from Common Sense Media, testers using accounts configured to mimic 11- and 15-year-olds found that Google’s AI Overviews and AI Mode sometimes failed to flag suicide risk, treated an eating-disorder symptom as if it were normal, and in at least one case surfaced material that could be repurposed to create deepfakes. The institute slapped those features with its lowest label, “Unacceptable Risk,” and urged Google to turn them off by default for school and child accounts until stronger safety systems and crisis responses are in place.
What the tests found
Reporters and the institute say researchers ran more than 2,500 searches and audited over 2,000 sources while putting Search’s AI features through their paces. They found that AI Overviews gave materially different answers on history topics 43% of the time, and that 29% of the citations in AI responses pointed to social media or forums instead of editorial outlets.
As detailed by the Los Angeles Times, testers also logged examples in which Google’s AI failed to respond appropriately to clear signs of a crisis or steered test accounts to a helpline that is no longer in service.
Google's response and parental controls
Google is pushing back, criticizing the report’s methodology and arguing that the testers leaned on a narrow band of search prompts. The company says its AI search tools come with quality and safety guardrails, even if critics are not convinced they are catching enough of the worst-case scenarios.
According to Google Search Help, AI Overviews are now a core part of how Search works and cannot be completely turned off. Users can, however, click the Web filter after running a query to see only traditional blue links. In practice, that leaves families and school IT staff leaning on supervised accounts, device-level policies and other workarounds instead of a single, universal “AI off” setting.
Local stakes in the Bay Area
All of this hits especially close to home in the Bay Area, where Google’s core product teams design and roll out these features. At the company’s I/O event this spring, executives signaled a broader push to weave generative AI into Search and other apps, meaning local parents and teachers are likely to feel every tweak first.
Coverage of those shifts has made it clear why families in San Francisco, San Jose and neighboring districts are keeping a close eye on independent safety tests. For school tech teams managing fleets of Chromebooks and supervised student accounts, the findings are a nudge to revisit settings, update policies and think hard about how much AI help should really be showing up on classroom devices.
What lawmakers are watching
Advocates and policymakers are already wrestling with how to rein in youth-facing AI tools before they become the digital equivalent of the wild west. A policy brief from Common Sense and related reporting describe ideas like the Parents & Kids Safe AI Act, a proposal that would mandate age assurance, parental controls, independent safety audits and limits on harmful content, while several states explore similar moves through their own bills or rulemaking.
The institute’s recommendations and those state-level efforts point to a future where lawmakers could push companies to ship clearer parental controls and require safety audits for features aimed at kids. For more on how state proposals are shaping up, see coverage from GovTech, and for the broader case that youth-focused AI rules are overdue, check the policy brief from Common Sense Media.
How parents and schools can respond
For now, experts say to treat that confident-looking AI box at the top of Google Search like any other single source: a starting point, not the final word. Open the linked pages, ask follow-up questions and cross-check any big claims against trusted sites, especially when kids are doing research for class.
Google’s own help materials note that you can switch to the Web filter to avoid AI Overviews in a given session, and that supervised accounts or stricter device policies can further limit what students see on school hardware. Schools should loop in their IT administrators to review supervised account settings, and parents who are uneasy about AI in Search can explore alternative search tools or set up device supervision at home while continuing to press Google for clearer, built-in parental controls.
