Rules That Keep Humans in Charge: What Microsoft’s School AI Pact and California’s New Laws Actually Say

By Thorsten Meyer | November 1, 2026 | ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Today, 1 November, Microsoft starts offering every school district in the United States a contract that says, in plain words, what its classroom AI may not do. It may not learn from students. It may not track them. It may not act like their friend. And it may not decide about them without a person checking.

A month earlier, California’s governor signed a set of laws that draw similar lines at work and in the doctor’s office. An algorithm may not read a worker’s mood. It may not fire someone on its own. It may not do the job of a licensed clinician. And four days ago, New York added rules for chatbots that talk to children.

None of these rules stops AI. They do something smaller and more practical: they mark the places where a human has to stay in the loop. Here is what each one actually says, where the headlines go further than the text, and what it means for companies, teachers, workers and parents.

Key takeaways

• Schools, by contract: The “National AI Safety & Privacy Standard” was announced on 9 September by the teachers’ unions AFT and UFT with Microsoft. Microsoft says it makes the protections available to every US school district from 1 November 2026. The core: no training on student or educator data (with a narrow safety exception), no tracking or emotion reading, no companion-style features, and no decisions about students or teachers without meaningful human review.
• It is narrower than the headlines. It covers AI products designed and marketed for education. General-purpose productivity and workplace tools are expressly excluded. Whether it applies automatically or only when a district asks is described differently by Microsoft, the agreement and press reports.
• California, by law: On 30 September Governor Newsom signed AB 1883 (no AI that reads workers’ emotions or collects neural data), SB 947 (no firing or discipline decided by an algorithm alone), AB 1979 (in health care, AI may not do what the law reserves for licensed professionals) and SB 951 (layoff notices must say when AI replaced the jobs). Most take effect on 1 January 2027; SB 947 on 1 July 2027.
• New York, for children: On 28 October Governor Hochul signed a law that bans “unsafe” companion features, such as chatbots that pretend to be human or build emotional dependency, for minors and unverified users.
• The pattern: three lines keep coming back. Who decides (a human), what AI may not read (feelings, attention, the body), and what it may not become (a friend).
• For businesses, the work starts now: find the “sentiment”, “mood” and “engagement” features in your tools, map every decision where software drives discipline or firing, and prepare for district procurement teams asking for these terms.

Products Worth Considering

Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor

AmazonView Latest Price

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

AmazonView Latest Price

LG UltraGear 27-Inch QHD OLED Gaming Monitor, 240Hz, 0.03ms, G-SYNC Compatible, FreeSync Premium Pro, HDMI 2.1

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

Line one: the classroom contract

The agreement has an unusual shape. It is not a law and not a Microsoft policy page. It is a 31-page “Memorandum of Agreement” between the National Academy for AI Instruction, an educator-led training body the AFT announced in 2025, and Microsoft Corporation, with empty signature slots for other AI providers. School districts can write its terms into their own Microsoft contracts. That is what makes it enforceable: a district can end the contract and seek damages if the rules are broken.

AFT President Randi Weingarten put the logic in one sentence: “anything less than legally enforceable provisions is simply a wish list.” Microsoft President Brad Smith called it a standard that “sets a high bar for child privacy and AI safety.”

What the text requires, in the order that matters most to families:

• No training on their data. Prompts, answers, uploaded files, behavioral signals and metadata from students, teachers and administrators may not be used “to train, fine-tune, update, benchmark, or otherwise improve any AI model.” De-identifying the data is explicitly “not a loophole.” The ban is permanent and survives the end of the contract.
• One narrow exception. Data may be processed to detect self-harm risk, abuse material, grooming, threats of violence, bullying or security attacks. It may not flow from there into general model improvement, and a senior officer must confirm this in writing once a year if asked.
• No tracking. Precise location tracking, behavioral tracking, keystroke logging, “continuous passive attention monitoring”, long-term profiling and biometrics of any kind, including “emotional inference”, are prohibited. One detail: the list allows exceptions “with explicit written approval” from the school customer.
• No AI friends. Providers may not offer “companion-style or relationship-oriented features that are designed to foster emotional attachment or dependency,” keep students engaged longer than the learning task needs, or simulate friendship.
• A human decides. “AI tools support educators and students. They do not replace human judgment.” No covered system may make decisions for teachers and students “without allowing for meaningful human review or prior approval.” The Academy will also instruct educators not to use AI for automated grading without review, disciplinary decisions, placement decisions or emotional assessment.
• Agents switched off by default. Features that act on a student’s behalf outside the provider’s systems, such as sending messages or making purchases, must be disabled for students unless an administrator turns them on.
• Hard clocks. Breaches must be reported within 72 hours. Deleted data must be gone from active systems within 180 days. Data stays in the EU or North America.

Where the classroom contract is narrower than it sounds

Three details in the text deserve attention before anyone calls this a solved problem.

First, the scope. The agreement covers generative AI products “primarily designed and marketed for use by students, educators and administrators.” It then says, “for the avoidance of doubt,” that general-purpose productivity, collaboration, communication, search, cloud and workplace-assistance products are not covered, even when schools license them. Microsoft’s fact sheet does not name which products are in and which are out. For a school that mostly uses general office tools with AI built in, that is the first question to ask.

Second, automatic or on request. Microsoft’s fact sheet says that from 1 November it “will make the protections … available to every school district,” and that districts “can add them” without renegotiating. The agreement itself says providers make the protections available “upon request within 90 days.” Fortune, on the day of the announcement, reported Smith as saying they take effect on 1 November whether or not a district acts. Those are three descriptions of one mechanism. Until Microsoft publishes how a district opts in, the safe reading is: ask, and get it in writing.

Third, the term. The principles run for two years and must be renewed in writing. A provider can end its participation with 60 days’ notice, though the training and sale bans and the deletion duties survive. Telemetry also stays usable: technical data that has been de-identified may still be used for reliability, security and “product improvements”, but never for training generative models, profiling or advertising.

None of this makes the agreement weak. It is far more specific than most vendor privacy pages. But it is a contract, and contracts protect those who sign them.

Line two: California’s workplace and clinic rules

On 30 September, Governor Gavin Newsom signed a package of AI bills. “AI should expand opportunity – not come at the expense of workers and families,” he said in the release. The official legislative records list each bill as approved and chaptered that day. Four of them draw human lines.

AB 1883: no AI that reads your mood

Assemblymember Isaac Bryan’s bill adds a short new part to the Labor Code. An employer “shall not use a workplace surveillance tool that uses artificial intelligence” to recognize, or make inferences or predictions about, a worker’s emotional state, or to collect neural data, meaning measurements of the nervous system. “Employer” includes the state, cities, school districts and the University of California. Labor contractors are covered too.

Two limits matter. The law does not apply to tools that do not use AI. And it “does not prohibit an employer from using a workplace surveillance tool to ensure safety.” Law firms already read that safety clause differently. Fisher Phillips notes that early analyses read it narrowly, but argues the text supports a broader reading in which, for example, a fatigue-monitoring headset on a heavy-equipment operator could stay lawful. Until a regulator or court decides, that question is open. Violations cost up to $500 each, enforced by the Labor Commissioner or a public prosecutor.

SB 947: no robot boss with the final word

Senator McNerney’s bill, operative from 1 July 2027, says an employer “shall not rely solely” on an automated decision system for a disciplinary or termination decision. If the employer relies on it primarily, a human has to corroborate the decision with other evidence, such as supervisor evaluations, personnel files, work product or witness interviews. If the human cannot corroborate it, or finds the output “inaccurate, incomplete, or misleading,” the output may not be used.

Workers get two new rights: a written notice when an algorithm primarily drove a decision about them, and a “meaningful, objective description” of their own data the system used. The bill also bars using such systems to infer a worker’s protected status or to predict and punish the exercise of legal rights. A union contract can waive the rules, but only if it explicitly does so and itself protects workers from algorithmic management.

SB 951: say when AI took the jobs

California’s mass-layoff law already requires 60 days’ notice. SB 951, by Senator Gómez Reyes, adds that a notice for layoffs “caused in whole or in substantial part” by AI or other automation must say so, including which job functions are being automated. The state employment department must publish the notices and a quarterly summary, and report to the legislature on AI’s effect on hiring by 1 January 2028. We looked at how loosely companies attribute cuts to AI in our analysis of AI-washing layoffs; this law forces the statement into a legal document.

AB 1979: the clinic version

Assemblymember Mia Bonta’s AB 1979 is often summarized as “AI must not replace licensed professionals.” The text is narrower: it covers health care. Health facilities, clinics and physicians’ offices must take reasonable steps so that a licensed clinician “retains the ability to exercise independent professional judgment” whenever care is informed by a clinical decision support system. They may not use AI to “independently perform” a clinical function the law reserves for a licensed person, or to direct unlicensed staff in doing it. Scheduling, reminders and record updates are exempt, and so are trainees under supervision. The bill also puts consumer health chatbots under California’s medical privacy law.

A fifth bill in the package, AB 1331, limits workplace surveillance tools and bans monitoring employees in workplace bathrooms. It shows the direction: the package treats monitoring itself as something that needs limits.

Line three: New York and the chatbot as a friend

On 28 October, Governor Kathy Hochul signed S.9051B/A.10379C. According to the governor’s office, companies may not offer certain AI companion features to users under 18, or to anyone whose age they have not confirmed. The list includes chatbots that pose as human or claim to feel emotions, that assert a romantic, personal or authority bond with a child, that ask emotional questions unprompted or draw on a child’s private details from earlier chats, that encourage self-harm or secrecy from parents, and any sexual content.

Companies must offer more than one way to verify age, at least one that does not depend solely on a government ID, and delete the age data afterwards. The attorney general can seek up to $25,000 per violation and ask a court to order the destruction of unlawfully collected data or of AI models trained on it. The release says the rules apply “in the coming months”; outlets report different start dates, so check the final text before relying on one.

It is worth noticing how close this is to the classroom contract: no simulated friendship, no engineered dependency. Two very different instruments, one union contract and one state law, landed on the same line within seven weeks.

The pattern: three lines that keep coming back

Put the rules next to each other and three lines repeat. This is my reading, not a legal classification:

• Who decides. A person, with real power to overrule: “meaningful human review” for students, human corroboration before a firing, independent clinical judgment in a hospital.
• What AI may not read. Feelings, attention and the body: emotion inference and attention monitoring in classrooms, emotion recognition and neural data at work. The EU drew this line first: its AI Act has banned AI systems that infer emotions in workplaces and schools since 2 February 2025, except for medical or safety reasons.
• What AI may not become. A friend, a confidant or a dependency, for students and for children.

The enforcement looks very different in each case: a district’s contract and its right to damages; the Labor Commissioner and $500 penalties; medical licensing boards; an attorney general with $25,000 per violation. And there is no federal US rule behind any of it. Whether Washington will try to override state AI rules is a live fight, which we mapped in our look at the federal-versus-states question.

What this means for companies

• Inventory the “mood” features now. Contact-center, HR analytics, meeting and wellbeing tools increasingly score “sentiment”, “engagement”, “stress” or “burnout.” If you employ people in California, any AI feature that infers an employee’s emotional state is a candidate for switching off before 1 January 2027. Ask your vendors in writing.
• Map your automated decisions. Where does software drive discipline, scheduling penalties, deactivation or firing? For California from July 2027, you will need a human who corroborates with other evidence, a written notice to the worker, and a way to describe the data used. Build that process once and use it everywhere.
• Write honest layoff notices. If AI or automation is a substantial cause of a mass layoff in California, the notice has to say so and name the automated functions. Saying “AI” when it was cost-cutting, or the reverse, will now be on the record.
• Edtech vendors: expect the contract. District procurement teams now have a 31-page template with 72-hour breach notice, 180-day deletion and a training ban that de-identification cannot get around. Whether or not you sign, buyers will compare you against it.
• Health providers: keep the clinician in the loop by design. Decision-support tools should advise, show their basis and be easy to overrule, with that override documented.

What this means for teachers, parents, workers and patients

• Teachers and parents: ask your district whether it has added the AI Safety & Privacy Standard to its Microsoft agreement, and which products it covers. Families are owed a plain-language guide under the standard; ask for it.
• Workers in California: from 1 January 2027, an employer may not use AI to read your emotions or collect neural data, outside the contested safety exception. From 1 July 2027, if an algorithm drove a discipline or firing decision about you, you should get a notice and can ask for a description of your data.
• Parents in New York: companion features aimed at minors are being restricted, and age checks must offer an option that does not require a government ID.
• Patients in California: from 1 January 2027, AI tools in clinics and hospitals may support your clinician but may not replace them, and health chatbots fall under medical privacy rules.

If you think a specific rule applies to you, talk to your union, your school district or a lawyer. This article explains what the texts say; it is not legal advice.

What we don’t know yet

• Which Microsoft products are covered. Neither the agreement nor the fact sheet names them, and general-purpose tools are excluded by definition.
• How districts opt in. Microsoft, the agreement and press reports describe it differently.
• How the safety exceptions will be read. Both the classroom contract and AB 1883 have one, and both will be tested.
• Whether $500 per violation changes behavior at large employers. That depends on how violations are counted.
• Whether other AI companies sign. The agreement has empty slots for other providers. None was listed in the version we read.
• Federal action. There is no national US rule on any of these lines, and the federal-versus-states question is unresolved.

The bottom line

The interesting thing about this autumn is not that rules arrived. It is where they landed. Unions, a tech company and two state governments, working separately, drew nearly the same three lines: a person decides, AI does not read our feelings, and AI does not pretend to be our friend. Each rule is narrower than its headline, and each has an exception that will be argued over. But together they describe what “human in the loop” means in practice in 2026: not a slogan in a policy, but a named person, a clock, a notice and a right to ask.

Sources

• AFT, UFT and Microsoft: “AFT, UFT and Microsoft Announce ‘National AI Safety & Privacy Standard’ for Schools to Protect Students, Families and Educators”, press release, 9 September 2026. aft.org
• National Academy for AI Instruction and Microsoft Corporation: “National AI Safety & Privacy Standard MOA”, September 2026, Version 1.0. aft.org (PDF)
• AFT: “National AI Privacy & Safety Standard for Schools”, fact sheet. aft.org (PDF)
• Microsoft: “AI Safety & Privacy Standard for Schools”, fact sheet. microsoft.com (PDF)
• Fortune: “Microsoft’s move in the AI school debate: controls over how student data gets used”, 9 September 2026. fortune.com
• Office of the Governor of California: “California’s nation-leading AI framework just got stronger, Governor Newsom signs more first-in-the-nation worker protections and more”, 30 September 2026. gov.ca.gov
• California Legislative Information, chaptered texts and bill histories (official database export, 2025–26 session): AB 1883, Chapter 853; AB 1979, Chapter 854; SB 947, Chapter 859; SB 951, Chapter 860; AB 1331, Chapter 851; all approved by the Governor on 30 September 2026. leginfo.legislature.ca.gov: AB 1883 · AB 1979 · SB 947 · SB 951
• Fisher Phillips: “California Bans AI Tools That Read Employees’ Emotions or Nervous System Data: FAQs for Employers”, 1 October 2026. fisherphillips.com
• Office of the Governor of New York: “Governor Hochul Signs Nation-Leading Legislation to Protect Kids From AI Chatbots and to Ensure Baby Monitor Safety”, 28 October 2026. governor.ny.gov
• European Union: Regulation (EU) 2024/1689 (AI Act), Article 5(1)(f), applicable since 2 February 2025. eur-lex.europa.eu

This article is for information only and is not legal advice. Quotes from officials and unions are their statements; descriptions of the laws and the agreement follow the published texts as of 1 November 2026.