Wharton – ��

ChatGPT Scores a C+ At the University of Minnesota Law School. Now What?

Greg Toppo — Tue, 07 Feb 2023 20:28:00 +0000

Though computer scientists have been using chatbots to for more than 70 years, 2023 is fast becoming the year in which educators are realizing what artificial intelligence means for their work.

Over the past several weeks, they’ve been putting ’s through its paces on any number of professional-grade exams in law, medicine, and business, among others. The moves seem a natural development just weeks after the groundbreaking, free (for now) chatbot appeared. Now that nearly anyone can play with it, they’re testing how it performs in the real world — and figuring out what that might mean for both teaching skills like writing and critical thinking in K-12, and training young white-collar professionals at the college level.

Most recently, at the University of Minnesota Law School tested it on 95 multiple choice and 12 essay questions from four courses. It passed, though not exactly at the top of its class. The chatbot scraped by with a “low but passing grade” in all four courses, a C+ student.

But don’t get complacent, warned Daniel Schwarcz, a UM professor and one of the study’s authors. The AI earned that C+ “relative to incredibly motivated, incredibly talented students … and it was holding its own.”

Think of it this way, Schwarcz said: Plenty of C+ students at the university go on to graduate and pass the bar exam.

Daniel Schwarcz

ChatGPT debuted less than three months ago, and its respectable performance on several of these tests is forcing educators to quickly rethink how they evaluate students — assigning generic written essays, for instance, now seems like an invitation for fraud.

But it’s also, at a more basic level, forcing educators to reconsider how to help students see the value of learning to think through the material for themselves.

Before he encountered ChatGPT, Schwarcz typically gave open-book exams. What the new technology is making him think more deeply about is whether he was often testing memorization, not thinking. “If that’s the case, I’ve written a bad exam,” he said.

And like Schwarcz, many educators now warn: With improving technology, today’s middling chatbot is tomorrow’s valedictorian.

“If this kind of tool is producing a C+ answer in early 2023,” said Andrew M. Perlman, dean of Suffolk Law School in Boston, “what’s it going to be able to do in 2026?”

Fake studies and ‘human error’

Lawyers aren’t the only professionals in the chatbot’s crosshairs: In January, Christian Terwiesch, a business professor at the University of Pennsylvania’s Wharton School, let it loose on the final exam of Operations Management, a “typical MBA core course” at the nation’s pre-eminent business school.

While the AI made several “surprising” math mistakes, Terwiesch wrote in the, it impressed him with its ability to analyze case studies, among other tasks. “Not only are the answers correct, but the explanations are excellent,” he wrote.

Its final grade: B to B-.

A Wharton colleague, Ethan Mollick, in December that he got the chatbot to write a syllabus for a new course, as well as part of a lecture. And it generated a final assignment with a grading rubric. But its tendency to occasionally deliver erroneous answers from its wide-ranging web searches, Mollick said, makes it more like an “omniscient, eager-to-please intern who sometimes lies to you.”

Indeed, AI tools often create problems of their own. In January, Jeremy Faust, an emergency medicine physician at Brigham and Women’s Hospital in Boston, asked ChatGPT to a 35-year-old woman with chest pains. The patient, he specified, takes birth control pills but has no past medical history.

After a few rounds of back-and-forth, the bot, which Faust cheekily referred to as “Dr. OpenAI,” said she was probably suffering from a pulmonary embolism. When Faust suggested it could also be costochondritis, a painful inflammation of the cartilage that connects rib to breastbone, ChatGPT countered that its diagnosis was supported by research, specifically a 2007 study in the .

Then it offered a citation for a paper that does not exist.

The AI platform has great potential for use in medicine, but has huge pitfalls, says Jeremy Faust, MD

While the journal is real — and a few of the researchers cited have published in it — the bot created the citation out of thin air, Faust wrote. “I’m a little miffed that rather than admit its mistake, Dr. OpenAI stood its ground, and up and confabulated a research paper.”

Confronted with its lie, the AI “said that I must be mistaken,” Faust wrote. “I began to feel like I was and that the computer was HAL-9000, blaming our disagreement on ‘human error.’”

Faust closed his computer.

A scene from “2001: A Space Odyssey,” in which a computer commandeers a space voyage. A Boston emergency room physician who watched recently as a modern AI created a fake medical study to support its diagnosis, said he felt like the astronauts in the movie. (Transcendental Graphics/Getty Images)

‘Proof of original work’

Such bugs haven’t stopped educators from test-driving these tools for students and, in a few cases, for professionals.

Last December, just days after Open AI released ChatGPT, Perlman, the Suffolk dean, presented it with a series of legal prompts. “I was interested in just pushing it to its limits,” he said.

Perlman transcribed its mostly respectable replies and co-authored a with the chatbot.

Andrew M. Perlman

Peter Gault, founder of the AI literacy nonprofit Quill.org, which offers a free AI tool designed to help , said that even if teachers think things are moving fast this winter, the reality is that they are moving even faster than they seem. Case in point: An online “prompt engineering” channel on the social platform Discord, devoted to helping students improve their ChatGPT requests for better, more accurate results, now has about , he said. “There are tens of thousands of students just swapping tips for how to cheat in it,” he said.

Gault’s nonprofit, along with , has already debuted that helps educators sniff out the more formulaic writing that AI typically generates.

While other educators have suggested that future ChatGPT versions could feature a kind of digital watermarking that identifies cut-and-pasted AI text, Gault said that would be easy to circumvent with software that basically launders the text and removes the watermark. He suggested that educators begin thinking now about how they can use tools like Google Docs’ version history to reveal what he calls “proof of original work.”

Peter Gault, founder of Quill.org, talks to students. Gault’s nonprofit uses AI to help students improve their writing. (Courtesy of Peter Gault)

The idea is that educators can see all the writing and revising that go into student essays as they take shape. The typical student, he said, spends nine to 15 hours on a major essay. Google Docs and other tools like it can show that progression. Alternatively, if a student copies and pastes an essay or section from a tool like ChatGPT, he said, the software reveals that the student spent just moments on it.

“We have these tools that can do the thinking for us,” Gault said. “But as the tools get more sophisticated, we just really risk that students are no longer really investing in building intellectual skills. It’s a difficult problem to solve. But I do think it’s worth solving.”

‘Resistance is futile’

Minnesota’s Schwarcz flatly said law schools must train students on tools like ChatGPT and its successors. These tools “are not going away — they’re just going to get better,” he said. “And so in my mind, ultimately as educators, the fundamental thing is to figure out how to train students to use these tools both ethically and effectively.”

Perlman also foresees law schools using tools like ChatGPT and whatever comes next to train lawyers, helping them generate first drafts of legal documents, among other products, as they learn their trade.

In the end, AI could streamline lawyering, allowing attorneys to spend more time practicing “at the top of their license,” Perlman said, engaging in more sophisticated legal work for clients. This, he said, is the part of the job lawyers find most enjoyable — and clients find most valuable.

Eamonn Fitzmaurice/��

It could also make such services more affordable and thus more available, Perlman said. So even as educators focus on the technology’s threat, “I think we are quickly going to have to pivot and think about how we teach students to use these tools to enable them to deliver their services better, faster and cheaper in the future.”Perlman joked that the best way to think about the future of AI in the legal profession is to remember that old “Star Trek” maxim: “ ‘.’ This technology is coming, and I think we ignore it at our peril — and we try to resist at our peril.”

As Biden Signs Waiver Extension, Study Shows School Meals Lower Grocery Costs

Kevin Mahnken — Fri, 24 Jun 2022 11:15:00 +0000

Updated June 27

On June 25, President Biden signed the Keep Kids Fed Act of 2022. The law will extend some school meal waivers through the end of the 2022-23 school year.

With a massive, pandemic-era expansion of free school meals scheduled to expire on June 30, Democrats and Republicans around a possible compromise that would extend the federal program through the summer. Passed , the deal is expected to move through the Senate and be signed by President Biden in the next few days.

Authorized by Congress and the Department of Agriculture over the last two years, widened the category of students eligible to receive breakfast and lunch. Schools providing meals were also offered higher reimbursement rates for the costs of running their programs, as well as the flexibility to serve food off-site and substitute for items lost to supply-chain snags.Those benefits by proponents of renewing the waivers, or even following the pandemic’s end. But language to continue the program into next year was left out of the FY2023 budget signed by the president in March.

Get stories like this delivered straight to your inbox. Sign up for �� Newsletter

In the near term, the could ease classroom hunger and simplify the work of schools in the months to come. But research suggests that greater availability of free meals in public schools actually lowers grocery spending even for those without school-aged children. And at a time of sharply rising food prices, it’s conceivable that the end of the waivers would contribute to further inflation.

In circulated last fall by the National Bureau of Economic Research, academics from the University of Chicago and the Wharton School of the University of Pennsylvania found that an earlier boost to free meals — through the Obama-era Community Eligibility Provision, which allowed certain schools to offer breakfast and lunch to all students without having to process individual applications — caused a significant decline in grocery sales at local retailers. Those chains responded by lowering prices across all their stores, leading nearby households to spend approximately 4.5 percent less in grocery bills in areas where the policy was adopted.

Jessie Handbury, a Wharton economist and one of the paper’s co-authors, called the effects “fairly sizable.”

“Because they’re responding across all their retail locations, the…drop in prices is going to affect all the households in the vicinity of that chain’s stores,” she said. “So you’ll have households that aren’t directly impacted by the demand shock, or that live nowhere near the communities that are taking up universal free lunch, but are still benefiting from it.”

The Community Eligibility Provision was introduced in select states through the Healthy, Hunger-Free Kids Act of 2010, before becoming nationally available in the 2014-15 school year. Participating schools (identified as those where over 40 percent of students qualified for free or reduced-price lunch) could choose to provide such meals to all of their enrolled students, whether they were eligible or not.

To study the effects of the legislation on grocery spending, Handbury and UChicago professor Sarah Moshary gathered information from the National Center for Education Statistics showing school-level participation in the Community Eligibility Provision between the 2011-12 and 2015-16 academic years. They combined that with self-reported grocery purchase figures from the , which collected data from a representative panel of nearly 50,000 American households over the same timeframe.

Finally, the pair added findings from a separate industry tracker of weekly grocery chain sales and sale quantity by product. In the five years under study, the system included responses from over 20,000 stores.

In all, the study found that homes with school-aged children reduced their grocery spending by an average of 7.5 percent (about $200 annually, or roughly two weeks of spending for families included in the sample) when a local school adopted the Community Eligibility Provision — the direct impact of their children receiving more meals for free in school. What’s more, that drop in sales led grocery chains to slash prices not just for the directly affected stores (i.e., the ones located near CEP schools), but in all of their locations. As a consequence, shopping costs in the median ZIP code affected by the policy were reduced by an average of 4.5 percent.

Handbury said it was plausible that a large number of families who were always eligible to receive free meals at school only began taking advantage of them once the provision was adopted. The sudden universality of the program may have reduced the social penalty sometimes referred to as “lunch shaming,” she surmised.

“You could imagine that when it costs money for their child to get lunch at school, they just automatically pack lunch for their children,” Handbury argued. “And when it became free, that was enough to induce them to at least send their kids to try free school lunch. Possibly because there was a reduction in the stigma associated with getting free lunch — or even getting school lunch — it just became what you did.”

Other studies have also shown clear consumer benefits accruing to families impacted by the program. , from researchers at Vanderbilt and the University of Louisville, showed that families with children spent between 5 and 19 percent less on monthly grocery purchases in areas that implemented the Community Eligibility Provision. Low-income households also experienced a meaningful improvement in dietary quality, and fewer were classified as food-insecure, in the wake of CEP adoption.

“The savings of $11 per month (or up to almost $39 for fully exposed ZIP codes) are realistic in magnitude and represent a meaningful change for low-income families that may face especially tight resource constraints,” said Michelle Marcus, one of the paper’s co-authors. “For the average household in our sample with two children, CEP provides about 8.25 additional meals per household for each of the eight academic months.”

Price discounts of that magnitude may not seem like much, but during a period of dramatic inflation — according to the Bureau of Labor Statistics, by over 9 percent between April 2021 and April 2022 — they might make a significant difference. Since the COVID-era meal waivers operate essentially like an enhancement of CEP, Handbury noted, their potential expiration could be expected to have “weekly inflationary effects” on those prices.

That’s partly why advocacy groups are already praising the bipartisan deal to extend the waivers for another school year. Earlier this month, the Food Research and Action Center touting the effects of the Community Eligibility Provision and advocating further flexibility for provision of school nutrition going forward.

In an email to ��, a spokesman for FRAC said the group was “excited about the provisions included in the bill that will support access to summer meals, allow children who are eligible for reduced-price meals to receive free meals, and the additional funding for schools and child care.”

Another group, the School Nutrition Association, was a vital resource at a time when the cost of kitchen essentials like wheat bread and dish gloves had risen by well over 100 percent.

“School nutrition professionals have withstood crippling supply chain breakdowns, rising prices and labor shortages in their efforts to provide students healthy meals, at a time when families are struggling with higher costs. With crucial federal waivers on the verge of expiring, this agreement offers school meal programs a lifeline to help build back toward normal operations.”