How do you know when someone’s learnt something? We think most good educators know it when they see it. This is a proposal for interviews as a means of assessing mastery and granting credit.
Understanding what and how people understand is core to helping people to learn, and therefore to Powderhouse’s work. Despite this, comparatively little effort has been invested in rigorously understanding how people understand (or misunderstand) things, and much of that effort is either embedded in the tacit knowledge of many teachers and tutors or limited by the constraints and approaches of academia.
The Learning Observatory is an initiative aiming to develop the tools, materials, practices, and people required for rigorous, textured understanding and communication of people’s mental models.
Taking inspiration from the work of constructivists like Piaget and timeless models like the viva voce, the Observatory centers observing and interviewing people doing intellectual work. In doing so, the Observatory aims not only to bring much-needed rigor to the work of “evaluation”, but more importantly, to create standard-setting examples for what truly understanding someone else’s mental model looks like, and what true mastery of a concept entails.
The Observatory aims to begin with a simple, concrete offering: Anyone who is seeking traditional academic credit for Common Core Math and/or English Language Arts subjects will be able to request an interview. That interview will involve a problem solving and interview process wherein people tackle problems, describing their solution and thought process to an interviewer, who may follow-up with clarifying questions, related problems, and similar. This entire process will be recorded. In addition to the original interviewer’s evaluation of the interviewee’s understanding, a panel of other interviewers will review the recording and independently evaluate and validate the credit ultimately awarded (or denied). For those interested, the Observatory will also make available 24/7 third-party tutoring support and curated online resources to support people in their studies. Over time, Powderhouse may branch out into creating such resources and/or providing this kind of support directly.
To illustrate the directions for which we hope to set sail from this concrete beginning, these are a few examples of the kinds of projects and applications we hope the Observatory may eventually offer a home to and vehicle for.
Given this context, the rest of this document aims to concisely answer the core questions defining this project.
In the near term, we’d want to:
In the longer term, we’d hope to see the Observatory:
Currently, our best idea about how to make this sustainable involves leveraging the Commonwealth Virtual School (CMVS) structure in Massachusetts under M.G.L. ch.71 §94. This allows for the state to authorize up to ten virtual schools (there are currently two), provided they enroll no more than 2% of all students in Massachusetts (2% of ~913,000 students is ~18,000, and the combined enrollment of the two existing virtual schools is 1,155 + 2,940 = 4,095).
CMVSes receive ~$9,000 per student and are responsible for offering the minimum, full range of services expected of a traditional school (for more on this, refer to the "Key Required Characteristics” excerpted from the application at the end of this document).
One possible plan for launching the CMVS version of the Observatory could look like this:
Powderhouse’s mid-term goals require evaluation of traditional academic skills. In our Innovation School design in Somerville, this arose explicitly through our commitments to “covering” Common Core Math and ELA and ensuring youth passed the MCAS, as well as implicitly through the expectations of families and postsecondary institutions. In our new directions, evaluation of traditional academic skills will remain a priority for similar reasons.
In our experiences with Kindling, it’s become obvious in the near-term, we will face an additional need for this kind of support in youth transitioning from traditional academic contexts which have failed them. The structures in which we’re working with people are not currently broad and deep enough to provide an opportunity to legitimately remediate these failures, especially with the limited timeframe offered by work with older teens and young adults.
There is a tremendous loss of fidelity in various representations of learning education traditionally relies on. In our experience, sitting with someone as they work on something and discussing it (and their reasoning) with them is incredibly illuminating when compared with, e.g. looking at homework assignments (much less a test score or GPA).
In particular, lower fidelity representations hide enormous numbers of false positives: Throughout our work, we regularly encounter people who “do well [enough]” in a traditional academic task who have nearly no real understanding of what they’ve “learned”. (One notable consequence of this is that many adults who aim to work with youth, having mostly succeeded in traditional academic settings, often lack the depth of understanding required to effectively support true understanding in someone else.)
Despite our dramatic differences in perspective and pedagogy, in general we feel that if we were to sit down with a group of typical educators and look, together, at a highly textured example of someone working on a problem, we would very likely agree on the relative extent to which someone understands the concepts and skills entailed by the problem. This seems like an important observation and opportunity.
In the long-run, our approach depends on our ability—and the ability of the adults with whom we work—to take ideas and understanding seriously. At some point, this requires developing a capacity to observe, critique, and ideally represent others’ thinking and working.
As a matter of timing, three particular observations stand-out:
At present, we see three, big red flags:
At the moment, we have to decide whether to pursue this further, which primarily requires understanding whether:
Commonwealth Virtual Schools (CMVS), and from the CMVS Application, the “Key Required Characteristics”
The following summarizes some of the key characteristics of a CMVS.
- Governance: A CMVS is a public school governed by a board of trustees that operates independently of any school district. The CMVS board, upon being granted a certificate by the Board, becomes a public entity authorized by the Commonwealth, subject to the oversight of the Board and the Department, to govern the CMVS. Subject to oversight and approval, the CMVS board has the powers necessary to implement the CMVS including the power to adopt a name for the school, determine the school’s curriculum and annual budget, acquire property for use as a school facility, and receive and disburse funds. A CMVS is a state entity and members of its board are considered to be special state employees under the definition of state ethics laws.
- Equal educational opportunity [M.G.L. ch.71 §94(b)(8)]: As public schools, a CMVS is open to all students. If awarded a certificate, the CMVS must not discriminate on the basis of race, color, national origin, creed, sex, gender identity, ethnicity, sexual orientation, mental or physical disability, age, ancestry, athletic performance, special need, proficiency in the English language or a foreign language, or prior academic achievement.
- Staff: All teachers hired by a CMVS must be licensed in Massachusetts in the areas in which they teach, pursuant to state law. All educators must be evaluated consistent with regulations promulgated by the Board and guidance developed by the Department. In addition to licensed teachers, a CMVS must have a Massachusetts-licensed school nurse (RN), a special education administrator, an attendance officer, and, if the school serves English learners, a licensed English Language Learner (ELL)/English as a Second Language (ESL) teacher.
- Curriculum and instruction: A CMVS must provide curriculum and instruction aligned with the standards contained in the Massachusetts Curriculum Frameworks. A CMVS may choose from available curricula aligned with the frameworks, or develop curricula aligned with these standards that reflect the mission of the school and that are designed to meet the needs of all students enrolled by the CMVS.
- Assessment: All students educated with Massachusetts public funds, including all students enrolled in a CMVS, are required to participate in state assessments. The award of a high school diploma includes the requirement that students earn a Competency Determination (CD) in English language arts, mathematics, Science and Technology/Engineering (STE). ACCESS for ELLs tests are assessed annually to measure the proficiency of ELLs in reading, writing, listening, and speaking English, as well as the progress they are making in learning English.
- Accountability: The Board grants a certificate to the CMVS board to operate a CMVS for a term of 3-5 years, as determined by the Board, after which the CMVS must apply for renewal of its certificate. The guiding areas of virtual school accountability are academic program success, organizational viability, and faithfulness to certificate.
The Performance Criteria for virtual schools articulate the expectations for a CMVS from initial application throughout the certificate term. The criteria provide guidance about how the Board, the Department, and the Commissioner define virtual school success and on what basis the school will be evaluated; and clarify the connection between virtual school accountability and the Massachusetts School and District Accountability System.
- Enrollment restrictions: A statewide CMVS may enroll students from across the state, provided that not more than 2 percent of students statewide attend a CMVS. Enrollment of students in 6 individual online courses that last a full school year count as 1 student enrolled on a fulltime basis for the purpose of this requirement.
- Reporting: Once established, a CMVS is assigned an 8-digit LEA code. Like all public schools, a CMVS is responsible for submitting data and filing reports with the Department in a timely and complete manner in accordance with guidelines published by the Department. The Department, in turn, publishes reports on all public schools in Massachusetts.
- Organizational capacity and experience: A strong CMVS board defines the mission of the school, develops policies and changes them when appropriate, hires qualified staff to manage the school’s day-to-day operations, holds the staff accountable for meeting established goals, and formulates a long-range plan and accountability plan that ensures the school’s continued stability. In addition to its many other responsibilities, the CMVS board must ensure that the school is complying with all applicable state and federal laws and that the board itself is operating in accordance with the rules set out by all applicable Massachusetts laws and regulations. Finally, the CMVS board is responsible for operating the school in accordance with its certificate.
The CMVS board, leadership, and staff of the proposed CMVS must possess a wide variety of skills and qualifications that enable them to develop, open, sustain, operate, and continuously improve an effective school. The CMVS board should be composed of at least five members to conduct business effectively and to provide efficient and effective governance and oversight. In addition, board members must, within a year of their appointment, complete an orientation concerning the responsibilities of their office as defined in the regulations. When recruiting board members, applicants should ensure they:
- possess the experience and qualifications necessary to implement the proposal outlined in the CMVS application;
- possess skills and experience in areas such as online or virtual education, management, finance, development, and law;
- demonstrate the capacity to found and sustain an excellent school; and
- are able to manage public funds effectively and responsibly.
- Public accountability: The CMVS board must operate consistently with laws relating to public accountability, such as the open meeting law, the conflict of interest law, and the financial disclosure law. While applicants may choose to contract with other entities to provide services, state ethics law may limit an individual’s ability to serve on the CMVS board if the individual currently holds or previously held a position at the entity with which the CMVS board is considering partnering or contracting, if the individual’s relatives work at the CMVS or for an entity with which the CMVS board is considering partnering or contracting, or if the individual has some other financial interest in the CMVS or in an entity with which the CMVS board is considering partnering or contracting.
- Learning supports for students in an online environment: Every CMVS must take appropriate and necessary steps to ensure that access to and engagement in the educational program is afforded to every enrolled student. The Department expects virtual schools to have strong protocols, tools, and practices because virtual learning takes place remotely and by definition poses unique challenges. Every CMVS must meet the time and learning requirements of state law unless it requests a waiver of those requirements, supported by research-based evidence and best practices in the field. To request a waiver, the proposed CMVS must respond to the questions in the appropriate attachment and provide a narrative explanation in the section on learning time.
- Nonconsumable instructional supplies: State law requires schools to purchase textbooks and other instructional materials and supplies intended for use and re-use over a period of years. Schools then in turn "loan" those instructional materials free of charge to students, who must return them at the end of the school year. Costly tools such as a tablet or other computer or graphing calculator fall in the category of instructional materials and supplies that, similar to textbooks, are intended for districts to purchase and use and re-use over a period of years. If such technology is required, the school may encourage each student to purchase these devices. Students are likely to do so because they may need those devices for future classes and other use outside of school. A CMVS must, however, be prepared to provide such devices free of charge to students whose families do not choose to buy them or cannot afford to do so. If students need such devices and/or Internet access to complete out-of-school assignments, the CMVS must also provide that access.
- Additional supports and interventions for special populations: A CMVS must provide a program that addresses the unique characteristics of the online learning environment and responds to these unique needs appropriately. Students with special learning needs, including English learners (ELs) and students with disabilities, will require additional supports to ensure they are able to access the whole curriculum, make substantial progress in acquiring the knowledge, skills, and behaviors presented by the curriculum, and in particulate fully in all aspects of the school experience.
- Restriction of enrollment by sending districts (the district where a student resides): The school committee of a sending district may, by vote, to restrict enrollment of its students in a CMVS if the total enrollment of its students in virtual schools exceeds one percent of the total enrollment in its district; provided, however, that no student enrolled in a CMVS is compelled to withdraw as a result of that vote. Only full-time students count towards the one percent threshold. If necessary, the Department will publish an annual list of districts eligible to restrict future full-time enrollment.