How do you know when someone’s learnt something? We think most good educators know it when they see it. This is a proposal for interviews as a means of assessing mastery and granting credit.

Understanding what and how people understand is core to helping people to learn, and therefore to Powderhouse’s work. Despite this, comparatively little effort has been invested in rigorously understanding how people understand (or misunderstand) things, and much of that effort is either embedded in the tacit knowledge of many teachers and tutors or limited by the constraints and approaches of academia.

The Learning Observatory is an initiative aiming to develop the tools, materials, practices, and people required for rigorous, textured understanding and communication of people’s mental models.

Taking inspiration from the work of constructivists like Piaget and timeless models like the viva voce, the Observatory centers observing and interviewing people doing intellectual work. In doing so, the Observatory aims not only to bring much-needed rigor to the work of “evaluation”, but more importantly, to create standard-setting examples for what truly understanding someone else’s mental model looks like, and what true mastery of a concept entails.

The Observatory aims to begin with a simple, concrete offering: Anyone who is seeking traditional academic credit for Common Core Math and/or English Language Arts subjects will be able to request an interview. That interview will involve a problem solving and interview process wherein people tackle problems, describing their solution and thought process to an interviewer, who may follow-up with clarifying questions, related problems, and similar. This entire process will be recorded. In addition to the original interviewer’s evaluation of the interviewee’s understanding, a panel of other interviewers will review the recording and independently evaluate and validate the credit ultimately awarded (or denied). For those interested, the Observatory will also make available 24/7 third-party tutoring support and curated online resources to support people in their studies. Over time, Powderhouse may branch out into creating such resources and/or providing this kind of support directly.

To illustrate the directions for which we hope to set sail from this concrete beginning, these are a few examples of the kinds of projects and applications we hope the Observatory may eventually offer a home to and vehicle for.

  • For years, we’ve talked about why a Sundance Festival for the future of learning would be valuable (see A Sundance for Learning for more on this). The Observatory would make an ideal host for this kind of work.
  • With a network of people whose fluency we’ve verified firsthand, we would be in a position to support peer-tutoring (both for Powderhouse youth, and others), which itself would be a valuable validation of expertise for Powderhouse’s own purposes.
  • Eventually, in claiming and validating Powderhouse’s outcomes along traditional academic lines, Powderhouse would be in a position to essentially deputize third-parties to evaluate youth in an incentive-compatible way while also generating incontrovertible documentation of skill without relying on traditional tests.
  • Over time, this corpus of recordings could represent a unique and valuable resource for those interested in learning and epistemology by providing diverse, structured, firsthand data supporting thinking about thinking by thinking about something, enabling comprehensive, illustrated catalogs of problems, solutions, understandings, and mental models.
  • Relatedly: Through our work on projects like XQ Math (cf. a project-based Algebra 1 curriculum and a proposal for a computational approach to calculus, it has become clear that there is a lot of cruft in traditional conceptions of core academics. Clearing out that cruft requires thoughtful work understanding the heart of a subject’s powerful ideas and reconceptualizing that subject.
  • Even now in Powderhouse’s work, we regularly find ourselves responding ad hoc to needs for thoughtfully curated and enriched materials and activities as youth run into the need for a topic either in the context of a project, interest, or academic goal.
  • Unusual versatility is required of adults interested in doing the kind of work we do with youth. One dimension of that versatility is disciplinary. In our experiences hiring faculty—even very talented faculty—their own relationships to traditional academic disciplines often needed additional support for them to be able to support youth’s interdisciplinary projects. The Observatory’s offerings could be of great use in training and developing faculty, much as it would be in developing youth.

Given this context, the rest of this document aims to concisely answer the core questions defining this project.

What is the purpose?

  1. Develop our ability to observe and describe people’s understanding and mental models
  2. Explore the usefulness of mapping common patterns in people’s understanding (and misunderstanding) as an ingredient in improving our own approach to this work
  3. Curate (and potentially, one day create) practical materials to support the development of core academic skills for adults and youth

Who is the audience?

  1. Adults seeking to master traditional academic content so as to be well-positioned to work with youth around that content
  2. Youth seeking to master (and receive traditional credit for) core academic subjects

What are the results?

In the near term, we’d want to:

  • Offer useful credit and academic preparation to Powderhouse youth,
  • Collect honestly interesting examples of understanding and misunderstanding captured in clear and discussable forms,
  • Validate the claim that calibration with traditional educators would be straightforward,
  • Find that we develop a sense that there is significant, unexplored depth to interviewing as an approach to understanding

In the longer term, we’d hope to see the Observatory:

  • Become a locus of activity around developing deep understandings and representations of people’s mental models,
  • Act as an always-on service Powderhouse youth [and potentially others] access
  • Develop the expertise and depth to offer unique experiences to adults interested in understanding people’s mental models, documenting learning, and similar

What is the plan?

Currently, our best idea about how to make this sustainable involves leveraging the Commonwealth Virtual School (CMVS) structure in Massachusetts under M.G.L. ch.71 §94. This allows for the state to authorize up to ten virtual schools (there are currently two), provided they enroll no more than 2% of all students in Massachusetts (2% of ~913,000 students is ~18,000, and the combined enrollment of the two existing virtual schools is 1,155 + 2,940 = 4,095).

CMVSes receive ~$9,000 per student and are responsible for offering the minimum, full range of services expected of a traditional school (for more on this, refer to the "Key Required Characteristics” excerpted from the application at the end of this document).

One possible plan for launching the CMVS version of the Observatory could look like this:

  1. In Year 1,
    • prototype the basic idea (i.e. curation, interviewing, recording and evaluating) with adults in our gap-year program using this for their traditional academics,
    • explore approaches to funding the establishment of a CMVS (or a partnership with an existing CMVS),
    • and find and on-board one or two people who could lead the operations of this project, developing a first draft application for a CMVS in the case that we choose to start one, or establishing the partnership otherwise
  2. In Year 2,
    • continue our prototyping work, incorporating it into our adult development work (minimally focusing on relevant MTEL tests),
    • begin exploring the kinds of work and understanding enabled by a corpus of recorded interviews (e.g. cataloguing misunderstandings), perhaps partnering with people or groups with a rigorous perspective on interviewing (e.g. the Critical Explorers group)
    • secure authorization to work with youth or an appropriate partnership in a credit-bearing capacity,
  3. In Year 3,
    • launch a CMVS or partnership with a CMVS,
    • hand over day-to-day operational reins to the people we’ve onboarded,
    • focus on enriching and improving the resources and expanding the material covered to include the kinds of courses and credit which seem indefinitely useful to us (e.g. MTEL English and Math; CLEP English, College Algebra, and Statistics; AP Physics, Calculus; SAT)

Why now?

Powderhouse’s mid-term goals require evaluation of traditional academic skills. In our Innovation School design in Somerville, this arose explicitly through our commitments to “covering” Common Core Math and ELA and ensuring youth passed the MCAS, as well as implicitly through the expectations of families and postsecondary institutions. In our new directions, evaluation of traditional academic skills will remain a priority for similar reasons.

In our experiences with Kindling, it’s become obvious in the near-term, we will face an additional need for this kind of support in youth transitioning from traditional academic contexts which have failed them. The structures in which we’re working with people are not currently broad and deep enough to provide an opportunity to legitimately remediate these failures, especially with the limited timeframe offered by work with older teens and young adults.

There is a tremendous loss of fidelity in various representations of learning education traditionally relies on. In our experience, sitting with someone as they work on something and discussing it (and their reasoning) with them is incredibly illuminating when compared with, e.g. looking at homework assignments (much less a test score or GPA).

In particular, lower fidelity representations hide enormous numbers of false positives: Throughout our work, we regularly encounter people who “do well [enough]” in a traditional academic task who have nearly no real understanding of what they’ve “learned”. (One notable consequence of this is that many adults who aim to work with youth, having mostly succeeded in traditional academic settings, often lack the depth of understanding required to effectively support true understanding in someone else.)

Despite our dramatic differences in perspective and pedagogy, in general we feel that if we were to sit down with a group of typical educators and look, together, at a highly textured example of someone working on a problem, we would very likely agree on the relative extent to which someone understands the concepts and skills entailed by the problem. This seems like an important observation and opportunity.

In the long-run, our approach depends on our ability—and the ability of the adults with whom we work—to take ideas and understanding seriously. At some point, this requires developing a capacity to observe, critique, and ideally represent others’ thinking and working.

As a matter of timing, three particular observations stand-out:

  1. 24/7 tutoring services like those offered by Paper and Remind are growing, and are incredibly affordable (Paper charges ~$50/student/month), as are similar, outsourced services for things like special education, nursing, etc.
  2. In the near future, we will begin to confront questions of credit and accreditation with parents, and this would offer an unusually strong form of reassurance which would not significantly alter our model.
  3. And more speculatively, the rapid advance of AI means that minimally, tutoring services will continue to improve and get more affordable. Maximally, there may be interesting applications and explorations which we would be in a unique position to do with an enriched, structured dataset like the corpus which would be created to do this work.

Why shouldn’t we do this?

At present, we see three, big red flags:

  1. Focus ⇒ We are plenty busy developing Powderhouse’s model right now. Unlike much of our work, this is conceivably something we could find people we trust to operate and lead; however, it will entail some serious time and energy.
  2. School-ish-ness ⇒ In its preoccupation with credit, evaluation, and increasing resolution, this has plenty of the hallmarks of a School-ish idea. The two redeeming factors are its focus on epistemology, and the potential for supporting peer-tutoring.
  3. “PBL” v. “Online instruction” anti-pattern ⇒ In nearly every school which has an unusual perspective on learning, models seem to converge to “weird stuff one half of the day” and “normal school the other half.” In “normal school,” this is cloaked in “individualized, online instruction with world class materials”, but is inevitably something like Khan Academy for English, math, and sometimes science. This is not what we’re proposing here, but it moves us closer to that configuration.

Next steps

At the moment, we have to decide whether to pursue this further, which primarily requires understanding whether:

  • something “like” this will eventually be necessary in our work
  • we can find people whom we trust to lead this kind of work,
  • we can either secure a flexible CMVS partnership or plausibly believe that there will be enough budget headroom in establishing a small CMVS ourselves to be worth the hassle

Commonwealth Virtual Schools (CMVS), and from the CMVS Application, the “Key Required Characteristics”

The following summarizes some of the key characteristics of a CMVS.

  • Governance: A CMVS is a public school governed by a board of trustees that operates independently of any school district. The CMVS board, upon being granted a certificate by the Board, becomes a public entity authorized by the Commonwealth, subject to the oversight of the Board and the Department, to govern the CMVS. Subject to oversight and approval, the CMVS board has the powers necessary to implement the CMVS including the power to adopt a name for the school, determine the school’s curriculum and annual budget, acquire property for use as a school facility, and receive and disburse funds. A CMVS is a state entity and members of its board are considered to be special state employees under the definition of state ethics laws.
  • Equal educational opportunity [M.G.L. ch.71 §94(b)(8)]: As public schools, a CMVS is open to all students. If awarded a certificate, the CMVS must not discriminate on the basis of race, color, national origin, creed, sex, gender identity, ethnicity, sexual orientation, mental or physical disability, age, ancestry, athletic performance, special need, proficiency in the English language or a foreign language, or prior academic achievement.
  • Staff: All teachers hired by a CMVS must be licensed in Massachusetts in the areas in which they teach, pursuant to state law. All educators must be evaluated consistent with regulations promulgated by the Board and guidance developed by the Department. In addition to licensed teachers, a CMVS must have a Massachusetts-licensed school nurse (RN), a special education administrator, an attendance officer, and, if the school serves English learners, a licensed English Language Learner (ELL)/English as a Second Language (ESL) teacher.
  • Curriculum and instruction: A CMVS must provide curriculum and instruction aligned with the standards contained in the Massachusetts Curriculum Frameworks. A CMVS may choose from available curricula aligned with the frameworks, or develop curricula aligned with these standards that reflect the mission of the school and that are designed to meet the needs of all students enrolled by the CMVS.
  • Assessment: All students educated with Massachusetts public funds, including all students enrolled in a CMVS, are required to participate in state assessments. The award of a high school diploma includes the requirement that students earn a Competency Determination (CD) in English language arts, mathematics, Science and Technology/Engineering (STE). ACCESS for ELLs tests are assessed annually to measure the proficiency of ELLs in reading, writing, listening, and speaking English, as well as the progress they are making in learning English.
  • Accountability: The Board grants a certificate to the CMVS board to operate a CMVS for a term of 3-5 years, as determined by the Board, after which the CMVS must apply for renewal of its certificate. The guiding areas of virtual school accountability are academic program success, organizational viability, and faithfulness to certificate.

The Performance Criteria for virtual schools articulate the expectations for a CMVS from initial application throughout the certificate term. The criteria provide guidance about how the Board, the Department, and the Commissioner define virtual school success and on what basis the school will be evaluated; and clarify the connection between virtual school accountability and the Massachusetts School and District Accountability System.

  • Enrollment restrictions: A statewide CMVS may enroll students from across the state, provided that not more than 2 percent of students statewide attend a CMVS. Enrollment of students in 6 individual online courses that last a full school year count as 1 student enrolled on a fulltime basis for the purpose of this requirement.
  • Reporting: Once established, a CMVS is assigned an 8-digit LEA code. Like all public schools, a CMVS is responsible for submitting data and filing reports with the Department in a timely and complete manner in accordance with guidelines published by the Department. The Department, in turn, publishes reports on all public schools in Massachusetts.
  • Organizational capacity and experience: A strong CMVS board defines the mission of the school, develops policies and changes them when appropriate, hires qualified staff to manage the school’s day-to-day operations, holds the staff accountable for meeting established goals, and formulates a long-range plan and accountability plan that ensures the school’s continued stability. In addition to its many other responsibilities, the CMVS board must ensure that the school is complying with all applicable state and federal laws and that the board itself is operating in accordance with the rules set out by all applicable Massachusetts laws and regulations. Finally, the CMVS board is responsible for operating the school in accordance with its certificate.

The CMVS board, leadership, and staff of the proposed CMVS must possess a wide variety of skills and qualifications that enable them to develop, open, sustain, operate, and continuously improve an effective school. The CMVS board should be composed of at least five members to conduct business effectively and to provide efficient and effective governance and oversight. In addition, board members must, within a year of their appointment, complete an orientation concerning the responsibilities of their office as defined in the regulations. When recruiting board members, applicants should ensure they:

  • possess the experience and qualifications necessary to implement the proposal outlined in the CMVS application;
  • possess skills and experience in areas such as online or virtual education, management, finance, development, and law;
  • demonstrate the capacity to found and sustain an excellent school; and
  • are able to manage public funds effectively and responsibly.
  • Public accountability: The CMVS board must operate consistently with laws relating to public accountability, such as the open meeting law, the conflict of interest law, and the financial disclosure law. While applicants may choose to contract with other entities to provide services, state ethics law may limit an individual’s ability to serve on the CMVS board if the individual currently holds or previously held a position at the entity with which the CMVS board is considering partnering or contracting, if the individual’s relatives work at the CMVS or for an entity with which the CMVS board is considering partnering or contracting, or if the individual has some other financial interest in the CMVS or in an entity with which the CMVS board is considering partnering or contracting.
  • Learning supports for students in an online environment: Every CMVS must take appropriate and necessary steps to ensure that access to and engagement in the educational program is afforded to every enrolled student. The Department expects virtual schools to have strong protocols, tools, and practices because virtual learning takes place remotely and by definition poses unique challenges. Every CMVS must meet the time and learning requirements of state law unless it requests a waiver of those requirements, supported by research-based evidence and best practices in the field. To request a waiver, the proposed CMVS must respond to the questions in the appropriate attachment and provide a narrative explanation in the section on learning time.
  • Nonconsumable instructional supplies: State law requires schools to purchase textbooks and other instructional materials and supplies intended for use and re-use over a period of years. Schools then in turn "loan" those instructional materials free of charge to students, who must return them at the end of the school year. Costly tools such as a tablet or other computer or graphing calculator fall in the category of instructional materials and supplies that, similar to textbooks, are intended for districts to purchase and use and re-use over a period of years. If such technology is required, the school may encourage each student to purchase these devices. Students are likely to do so because they may need those devices for future classes and other use outside of school. A CMVS must, however, be prepared to provide such devices free of charge to students whose families do not choose to buy them or cannot afford to do so. If students need such devices and/or Internet access to complete out-of-school assignments, the CMVS must also provide that access.
  • Additional supports and interventions for special populations: A CMVS must provide a program that addresses the unique characteristics of the online learning environment and responds to these unique needs appropriately. Students with special learning needs, including English learners (ELs) and students with disabilities, will require additional supports to ensure they are able to access the whole curriculum, make substantial progress in acquiring the knowledge, skills, and behaviors presented by the curriculum, and in particulate fully in all aspects of the school experience.
  • Restriction of enrollment by sending districts (the district where a student resides): The school committee of a sending district may, by vote, to restrict enrollment of its students in a CMVS if the total enrollment of its students in virtual schools exceeds one percent of the total enrollment in its district; provided, however, that no student enrolled in a CMVS is compelled to withdraw as a result of that vote. Only full-time students count towards the one percent threshold. If necessary, the Department will publish an annual list of districts eligible to restrict future full-time enrollment.