Platforms for sharing teaching resources almost all look the same. Platforms for teaching resources also use many of the same quality assessment mechanisms. And business models for these platforms have also seen minimal experimentation. There are new possibilities here.
DESIGN
Virtually every web site for teacher sharing looks the same: a search bar, a list of results, and a collection of documents. But the search paradigm isn’t the only way to organize information on the web. Further, while teachers keep their work mostly in documents, that doesn’t mean it should be sorted and searchable that way. Judges write their decisions in documents, after all, but when lawyers find these decisions online, they’re not posted as Word documents. Why are teacher lessons shared this way? Reimagining the information architecture and access paradigm for education content is an essential first step in creating a professional memory for educators.
Design Solution 1: a topic oriented information architecture
Google won the paradigm wars for organizing the web. In the early days of the web, Google and Yahoo were locked in a war over how to organize the growing number of sites and pages. Google won. The web simply grew too large for directory trees like Yahoo to organize.
But not every mass of information works like the web. Wikipedia organizes information differently than Google: into topic pages where information compiles. Stack Overflow, Quora, and Apple Support organize information around questions with responses ranked by users. Netflix and Amazon organize video content into curated, browsable genres. Each of these manages large quantities of information in different ways. Education can do this, too.
A new approach draws on several organizing paradigms. As a foundation, the current model uses a topic-oriented architecture like Wikipedia. Then, it organizes teaching materials into a simple taxonomy of items within topics. Further, topics are customizable by teachers into subtopics, enabling organic organization and construction of knowledge. Finally, individual curricular objects are duplicable, enabling teachers to copy and refashion them to suit their contexts.
The experience of this is clean, in form and in user experience. Teachers search for topics they are interested in, whether a text or a skill or a character trait. They browse or search for items relevant to their teaching. They join communities of interest.
Teachers have responded enthusiastically. This fundamental change in information organization — topic organization — has been the key feature we have tested with teachers, and they describe this format, plus the minimalist design, as a distinguishing element. It matches the way teachers think when they are searching for materials.
A topic orientation has several advantages
- Topics can be most anything: texts, subject content, skills — anything a teacher teaches. Most commonly teachers search for texts like The Great Gatsby, or events like the Civil Rights Movement, or subject matter like cellular respiration. But topics can also be skills like close reading that align with standards, or pedagogical techniques like harkness discussion, or even more specific areas of interests like First days of class. Each of these topic pages is a home to a collection of approaches to teaching that topic.
- Within topics, information is organized organically by teachers. Teachers create subtopics to structure the lesson materials within a given topic. A novel might be organized by chapter. Historical events might be organized by context, the event, and outcomes. Mathematical topics might be organized by ways to introduce a concept, ways to practice it, and real world examples. These subtopics are generated by teachers as they share their work. As materials accumulate and subtopics grow large, teachers revise and restructure topics based on the nature of the user generated content. This enables flexible organization that both reveals the structure of what people are teaching and makes it easy to discover the most relevant materials.
- Topics are populated with teaching items: questions, activities, assignments, and multimedia. When a teacher visits a topic, they are looking for ways to teach the subject matter of the topic. These granular instructional units are the small pieces of work teachers need to succeed on a daily basis — and reflect the RAND study’s report on what teachers search for online. In this way, Athena starts where the teachers are. (NOTE: Some stakeholders have strong feelings that the granularization of curriculum and swapping out of pieces in the curriculum disrupts the coherence of high-quality, standards-aligned, recursively designed curriculum. This is irrefutable: perfect coherence may be lost. But this revision serves a purpose, typically to make the curriculum more relevant to a class, and there are inevitably gains that come from this as well. Ultimately Athena aims to serve both functions: to provide a format that supports wholesale, integrated curriculum while also providing teachers and instructional designers the tools to swap out pieces of that curriculum in a guided way. If these can be housed in the same environment, then we can close significant gaps in what defines quality.)
- Topics can emerge in response to current events. This is an essential component of the future vision of Athena. When news events transpire, millions of teachers wrestle with how to engage their students meaningfully and healthily with complex and sensitive topics. If two teachers share classroom approaches for engaging students with the news, then suddenly thousands of others are prepared with tools that they can use. This ability for teachers, by sharing knowledge, to help each other guide our nation’s students through our complex times is an increasingly relevant facet of this new vision. Even in the past 18 months, teachers have used the current prototype for this purpose.
- Topic pages enable teachers to gather in communities of interest. Once a topic page has been defined by a teacher, other teachers can visit it, share their materials, and contribute to the organization of the page. Teachers can follow a topic, learning about when materials are added to it. They can post and answer questions with others who visit the topic or follow it, building a community of teachers interest in a topic.
Topic pages provide a literal place to organize and congregate around knowledge. As early as the classical text Rhetorica ad Herennium, humans have understood that memory and perception are influenced deeply by three dimensional space. Organizing information by topic creates a geography — a topography — that doesn’t exist in search results listings. The word “topic” even derives from the Greek “topos,” meaning “place” or “locus.” Teachers can meet at a topic page, join communities of interest, participate in conversations and ask questions. It is like a department office, but much more expansive. The web has always been a place for people to find their tribe. This enables that possibility in education.
Design Solution 2: direct entry of and access to content.
Document intermediated content is unbearably cumbersome online. Teachers must be able, therefore, to directly interact with content, as general users do on Wikipedia. When searching for a writing prompt or an assessment, teachers ought to be able to read that prompt or assessment in a web browser without any document previewing function. This seamless interaction between user and content is a second essential feature of a successful platform, and it based on a key principle:
Frictionless consumption is more important than frictionless contribution. In the late 2000s, the prevailing wisdom was, “You have to make it easy for teachers to share materials.” Companies enabled teachers to upload entire folders, hard drives of materials. These files sat (and continue to sit) disorganized, inconsistently named, and largely untouched on servers. It turns out that acquiring the content is not the important part. Acquiring the content in a consistently formatted, easily presented way is the important part.
This does create barriers to entry, which will slow the rate of user-generated content, but barriers to consumption will bring user participation to a halt. As a result of the slowed rate of content sharing, programmatic engagement and incentive structures will be necessary, but this can create other opportunities, detailed later in “Habituation.”
Design Addendum:
Not essential to the design of a platform for educators, but important in the development of this project, has been the observation that most edtech infantilizes teachers. Technology for teachers is full of the same whistles, bloops, and gamification devices as those for students. Edtech source EdSurge once notably published a quiz: “Can you tell the difference between Pokemon and Edtech startups?” I considered myself pretty discerning until I scored around 50%. Athena takes its name and its design aesthetic from the recognition that teachers ought to feel a part of something historic and stable, something as esteemed as antiquity. Many of our most effective techniques — socratic questioning, debate, experimentation — harken back to the classical era. It is only natural for teachers to feel a great connection to history.
QUALITY
“It’s about matching an instructional action with the right learner at the right time. Instructional design and materials, informed by data and learning science, can focus on the anticipated variability within the target population that will matter most for learning and demonstrating competence with the academic goals.”
David Dockterman (2018)
Good design makes content easily, intuitively navigable. Quality management mechanisms, however, ensure that displayed results are most relevant.
In education, since every context has unique needs, what defines quality changes from location to location. This means that the concept of “best practices” doesn’t apply. Instead, educators recognize that there are effective practices that work in some contexts, and other effective practices that work in others. Even while some core understandings and techniques are universal, the implementation of those understandings and techniques changes from place to place. The job of providing high quality materials, then, becomes the job of providing the most relevant materials for each context.
This challenge grows when quantities of materials proliferate. The platform approach proposed here does not vet content, for example; it does not prevent some posts from going live while permitting others, as this kind of vetting would run counter to the principle of open sharing and creation, of encouraging teachers to collaborate around improving their work. But, this approach does build in quality control measures — or: measures for ensuring teachers find the most relevant materials.
Quality Solutions: three forms of quality assessment
Three techniques are central for assessing quality: affirmations, algorithms, and editors.
Affirmations:
High frequency, low resolution feedback tools like Likes, Favorites, and Claps are the simplest and most common tools for assessing quality. They provide a typically binary snapshot of user sentiment about a tweet, a post, a review, or some other object.
Affirmations can be helpful at scale, but they do face numerous challenges: Often little is known about who or what gives the affirmation, and so it is hard to assess the value or authenticity of the affirmation. Also, algorithms built around affirmations often suffer from first-mover advantage. When an object receives numerous affirmations, it might appear at the top of search results page, leading more users to see it first, assume it is the best, and never review other objects. This suppresses new and potentially superior content, inhibiting innovation and the diffusion of new ideas. Lastly, the low resolution nature of the feedback does little to provide the kind of nuanced reflection that teachers might be looking for and that the platform might find useful for understanding the quality of different materials.
Still, high frequency, low-resolution feedback can prove useful at scale and can surface a version of the wisdom of crowds (Owen, 2018).
Algorithms:
More effective, however, are three algorithmic approaches that work on more nuanced behavior and data: through engagement, network analyses, and school-based data.
Engagement:
The New York Times knows when a reader clicks on a headline, scrolls to the end of an article, shares the article, comments on it, and/or returns to it later. Each of these actions deepens the reader’s engagement with the article, signaling an investment in time and therefore relevance. From a behavioral perspective, this is a more honest assessment of quality. Has the reader invested time in her day? Has he invested his social capital into it by sharing it?
On the current platform model, similar levels of engagement occur. Has a teacher clicked on a writing assignment? Scrolled to the bottom of it? Saved it for later? Commented on it? Made a personal copy? Edited that copy? Shared it with others? Edited it later? Each of these behaviors similarly marks a degree of engagement or relevance that can serve as a proxy for quality.
Items in the database would develop yield ratios over time: what percent of people who clicked on an item scrolled to the end of it? What percent who scrolled saved it for later? What percent who saved it for later returned to it? These behavioral metrics surface authentic user persistence with materials.
Network analyses:
Further, analysis of how users in a network interact with individual objects can enable the identification of new, high quality materials that ought to be promoted to other networks within the system. If a group of users show repeated engagement with each other, such as members of a department might show, and if they each engage certain objects more deeply, then those objects might be promoted to a larger network. If the larger network similarly engages the material deeply, the object might continue to be promoted.
Similarly, as teachers participate more robustly in networked sharing, they will likely take on different personas within the system: consumers (who only browse and take), creators (who create and share), editors (who take, revise, and share again), etc. Particularly as certain creators and editors emerge whose materials are widely engaged, the system can promote their work. And, as teachers develop the option to follow other teachers, the relative social weight of individuals can influence what is promoted as relevant to individual users.
School-based data:
Additionally promising is the ability to leverage school data as a means of recommending materials or assessing the relevance of those materials. As teacher profiles require school identification, and as information about schools is publicly available online, algorithmic analysis can identify what materials were made and assessed positively in schools that are like the schools of other, individual teachers. Teachers at urban schools with a high percentage of low-income students will likely be looking for materials made by or for similar environments, as these might be most relevant. Suburban and rural schools will likely function similarly. Particularly as this data is cross-referenced with previous analyses, the system will have potent techniques for assessing relevance to individual users.
A note on algorithms: Cathy O’Neil’s book Weapons of Maths Destruction (O’Neil, 2016) offers remarkable insight into the strengths and weaknesses of algorithmic models. An essential layer of a successful platform must be the user’s ability openly browse and set filters, tweaking — or at least inspecting — the platform’s relevance strategies. For example, it may make sense to show curriculum from a particular setting to teachers of a similar setting, but those teachers may also want to view and browse materials from dissimilar settings for the purpose of gathering inspiration. Giving teachers the ability to control what they see — or at least being transparent about what is shown — is an essential part of a successful, ethical philosophy.
Editors:
Lastly, nothing is as subtle as the discernment of an expert, human editor. In a bookstore, the Staff Picks table is always full of gems. In the comment section of New York Times articles, the “NYT Picks” section features a healthier range of perspectives than exists in the comments most recommended by readers. Similarly, any system that requires inspection or approval — whether buildings, peer reviewed journals, or passports — these benefit from the ability of human beings to assess context clues and details that aren’t built into an algorithm.
Editors are ideal, though for reasons of time and often of cost, they are not scalable. Still, within limited platforms, opportunities for editors are manifold: as communities of interest emerge around topics, editors can assess and highlight relevant themes, akin to editorial summaries on Rotten Tomatoes; further, external organizations can certify certain curricular objects as modeling a particular pedagogy, such as a project-based learning approach to teaching a subject; similarly, schools or districts could validate certain materials as being approved for certain contexts. Editorial selections provide teachers an additional route to finding the most relevant materials.
A note on reviews and natural language processing: Omitted from this list are text mining and natural language processing. These techniques can be particularly useful for discerning trends across large bodies of text, but until a platform reaches a large scale, they are not as useful, and without other approaches to carry a platform to scale, it will never reach the necessary size. Machine learning, however, may provide an additional layer of effectiveness at surfacing highly relevant materials, but we are some distance away from real utility in this space.
Summary
These three forms of quality assessment and management, taken together, provide meaningful ways of recognizing the key principle that quality is not a clearly defined concept. Rather, quality is relevance, and relevance changes by context. Combining the three quality mechanisms described here will be essential for platform success.
The (Unrealized) Promise of the Internet
Fulfilling the Need: Program Design
Appendix A — Mission: Improvement
Athena: A Prospectus 
Leave a comment