Missing Measures

College rating systems once evaluated academic standing and selectivity alone. Now, graduates’ salaries are a major factor in ranking schemes. Should other outcomes be considered in determining a school’s value?

No comments

What’s the most important “return on investment” for a college degree?

By Dawn Cunningham ’85

If you keep an eye on the national reputation of Mills or any other college—perhaps because you’re curious about the value your alma mater’s name adds to your resume, or because you have a college-bound child—you know that the college rankings game has been dominated for decades by one publication: U.S. News & World Report. Each September, U.S. News publishes its “Best Colleges” rankings, in which institutions are scored on such measures as academic reputation, class size, selectivity, and alumni giving. Mills invariably places near the top of its category in the Western region.

But in recent years, the U.S. News methodology has drawn criticism for failing to look beyond admission and campus statistics and factor in the outcomes of a college education. Its rankings don’t reflect, for instance, whether a college’s graduates earn good salaries, repay their student loans on time, or move up the economic ladder. They also don’t consider more qualitative effects, such as whether alumni make a positive impact on society or find satisfaction in their work.

“College is an enormous cost, and families want to know what they will get in return,” says Linda Cohen Turner ’68, a Mills trustee who counsels high school students on selecting a college through her company, The College Choice. “The economic downturn eight years ago—coupled with news about college grads having a hard time finding good jobs—had a lasting effect on their decision-making. Parents, especially, want college to prepare their kids for well-paying careers.”

A college education can boost earning power substantially. The US Department of Education reports: “On average, college graduates earn $1 million more over their lifetimes than high school graduates.” But there’s a great deal of variation in the cost of tuition and in graduates’ earnings, leading to demand for rankings that account for return on investment (ROI).

Since the 2008 recession, several new scoring systems have arisen that give considerable weight to alumni incomes. Among the first were Payscale’s “College Salary Report” and “College ROI Report.” The latter ranks colleges based on the difference between the 20-year median earnings of the college’s bachelor-degree graduates and the 24-year median earnings of high school graduates, minus the total cost of the college degree. (PayScale’s earnings data come from its national survey of US workers; the survey’s data set on Mills graduates, however, is too small for Mills to appear in the ROI report.)

In September 2015, the Department of Education unveiled its College Scorecard website, describing it as “the most comprehensive, reliable data ever published on student outcomes.” The Scorecard does not offer rankings, but instead draws data from IRS records and other federal data systems to show how a college performs on a number of measures compared to national averages. For example, it reports the median earnings of alumni who received federal financial aid as students and the percentage who are paying down their federal student loans; Mills scores above average on both measures.

“America’s Top Colleges 2016,” published in July by Forbes, bases rankings in part on earnings data from both the Scorecard and PayScale. Mills is listed among the colleges it deems worth the investment. Forbes says of its methodology: “We’re not interested in what gets a student into college…. Our sights are set directly on ROI: What are students getting out of college?” The Economist, Washington Monthly, Wall Street Journal/Times Higher Education, Georgetown University, the Brookings Institution, and Niche (a website that rates schools) have also published rankings that incorporate Scorecard data.

Meaning over money

Saleha Ahmed ’16 welcomes the increased focus on alumni incomes. “Financial success and security are big concerns for me and other millennials. We face immense amounts of debt,” she says. “My priority right now has to be making enough money to pay off student loans.” An economics and international relations major, Ahmed is managing investor relations and marketing for her family’s food business while seeking a job in financial analysis.

Yet she also sees the limitations of such earnings-based ratings: “Quantitative metrics like income shouldn’t the only way to measure what makes Mills valuable. You need to look at the impact Mills graduates are making in the world.”

Alice Knudsen, MA ’05, EdD ’07, director of institutional research at the College, agrees. Her surveys to assess students’ educational experiences and outcomes show, she says, that “our students’ goals are to have a meaningful life and to help others.” This holds true even among MBA students. Ilana Golin, who has been involved in admission strategy and career development at Mills’ Lorry I. Lokey School of Business and Public Policy, says: “We ask new students to write down their personal definition of success. Few mention money. Most talk about impact or career fulfillment.”

Knudsen adds, “Many Mills alumnae work for nonprofit organizations or schools that don’t pay as well as corporations. Others work in creative fields where careers take a while to come to fruition. If your concept of ROI is based solely on salary, you’ll miss what Mills is all about.”

In September, a National Public Radio report on college rankings observed, “The earnings method is inherently biased against colleges that educate a lot of future teachers, social workers, and artists, while favoring tech- and engineering-heavy schools.” In fact, about half of the top 25 colleges in PayScale’s ROI report are known primarily for such programs. Students’ earning potential often has more to do with the majors they choose than with the school they attend.

The earnings method is also biased against women’s colleges. As a group, graduates of women’s colleges tend to have lower median earnings than coed school graduates (which include substantial numbers of men), because women tend to be paid less than men for equal work and are more likely to take time off from full-time employment to care for children. In October, New York Times writer James Stewart pointed out how much earnings-based rankings hurt women’s colleges with this stark example: “U.S. News ranks Wellesley College, Hillary Clinton’s alma mater, No. 3 among national liberal arts colleges. It falls to No. 30 on The Wall Street Journal’s rankings, and to No. 201 on PayScale.”

PayScale’s research, furthermore, shows that college graduates who came from low-income households as students earn less, on average, than other alumni. Yet its rankings don’t adjust for this fact, so colleges with a large proportion of low-income students are likely to score lower than those with wealthier students. Low-income students also are more likely to drop out—often because they lack the financial resources to stay in school—so the same colleges may also be demoted in U.S. News and other rankings that factor in student retention rates.

Low-income, high performing

Although colleges that do a good job of educating low-income students are seldom recognized in the rankings, the Department of Education wants to encourage more colleges to enroll and graduate such students. In March, it released “Fulfilling the Promise, Serving the Need: Advancing College Opportunity for Low-Income Students,” a report that uses earnings, retention, and other data from the College Scorecard to highlight the schools that have been most successful in providing students with social mobility.

Mills is one of 13 private colleges cited as exemplary in this effort. Notably, four other women’s colleges also appear on this list: Agnes Scott, Converse, Salem, and Spelman. These colleges enroll a high percentage of students who receive Pell Grants, federal financial aid for low-income students; at Mills, they make up 50 percent of undergraduate students. Pell Grant recipients at these colleges defy the odds, making it through to graduation at the same rate as other students. As alumnae, they see their degrees pay off with a strong ROI.

“All of us in higher education should be part of a problem-solving process to ensure that low-income, underrepresented, and first-generation students can avail themselves of post-secondary opportunities,” says Mills Trustee Judith James ’74. Her own story attests to the value of college as a gateway to social mobility. She first came to Mills during high school with Upward Bound, a college-prep program for students from low-income families and those whose parents don’t hold college degrees. James met both criteria. After high school, she entered Mills with a full scholarship.

She had a rough start. “I came with the fear that I would not be able to perform at the level of Caucasian women from wealthy families,” recalls James, who is black. In addition, she was her mother’s defender against an abusive husband. “During my freshman year, I was at Mills but not totally emotionally present. Who would protect my mother while I was in class?” After her mother finally left her husband, she and James shared an apartment off campus, which allowed James to concentrate on her studies. Meanwhile, she found encouragement from people on campus, especially from her “zipper” (an upperclass peer-mentor), Micheline Beam ’72, and student support services.  “Eventually I got the wind beneath my wings, and I was soaring. I felt like I had an equal chance,” she says. She graduated a semester early and went on to earn master’s and doctoral degrees in education administration.

James has dedicated her career to making higher education available to underrepresented, first-generation students and those from low-income backgrounds —many of whom face the same obstacles to success that she did. Among other roles, she’s been a vice chancellor for student services at a community college, executive director of a county board of education, and a consultant to the Department of Education. These positions enabled her to improve life opportunities for thousands of students—but her earnings haven’t always reflected the scale of her impact.

“When looking at college outcomes, the focus should also be on what alumnae do with their degrees, not disproportionately on the salaries they make,” says James. “It should be on whether underrepresented students are prepared for careers that give them the opportunity to contribute to society.”

The happiness quotient

Mills Trustee Turner agrees: “It’s important for schools like Mills that serve a very diverse population to demonstrate that the outcomes for our students are not only their incomes, but the ability to change their lives, to feel fulfilled, to contribute to the community.”

Leah Ozeroff  works with Knudsen in Mills’ Office of Institutional Research to gather data demonstrating these outcomes. “We survey alumnae on typical measures, like salaries,” she says. “But we also ask how well their education prepared them for a career or a graduate program. We ask about job satisfaction, a measure that’s severely lacking in most of the rankings. And we ask about leadership awards and volunteer work, which indicate a level of fulfillment that you don’t get at when you only talk about salary.”

Mills graduates report high marks in these categories. For instance, 77 percent of undergraduate alumnae are satisfied or very satisfied with their careers, 85 percent say Mills prepared them well for the job market, and 97 percent say Mills prepared them well for further education.

Purdue University, in partnership with Gallup, has also surveyed graduates about life satisfaction and found strong correlations to the quality of learning experiences in college. “If you are a graduate who was emotionally supported during college, it more than doubles your odds of being engaged in work and triples your odds of thriving,” says Brandon Busteed of Gallup.

At Mills’ Lokey School, Golin says: “The educational experience is tremendously important. Do you actually learn the material? Do you feel like a member of a community? Such matters haven’t been part of the rankings conversation.” Yet these are questions that Mills and other colleges study in detail, both because the organizations that accredit them demand it and because such assessment provides data that schools use to improve the curriculum and student life.

A circuitous tale

Sometimes a college’s success with student learning and career outcomes can be demonstrated more powerfully with personal stories, like that of Judith James, than with quantitative data.

Turner says: “When clients ask me how I got to do what I’m doing, I tell my circuitous story. I moved from one kind of opportunity to another, gathering skills and knowledge and translating them into the next situation. I ended up doing work that I didn’t know existed when I was in college. I explain that a college education should give you the mindset to be nimble and flexible and adapt to enormous change, to develop new skills for careers that don’t exist today. To know whether a particular college can provide this, you need to look at how the college’s graduates have adapted to change over the long term. Mills alumnae can all tell stories about this. They can say, ‘Yes, I’ve done all these things.’”

Ahmed recalls how alumnae stories impressed her when she worked as a student caller for Mills’ telephone outreach program. “Alumnae would tell me how they got to where they are,” she says. “They’d talk about how important Mills was to them and how they used their education in their career fields. This really motivated me to make the most of my time in college.” She did so, and now tells her own story about what she got out of Mills: “I studied abroad at Oxford University for a year and I served as vice president of my class for two years,” she says. “I gained leadership and interviewing skills and the ability to clearly present an argument. My whole college experience was transformative.”