Mistakes were made; or, the atomization of a forums community

When we started our discussion forums nearly three years ago, we thought ourselves pretty clever. The process is straightforward, n’est-ce pas? Vet the options, imagine the use cases, and construct the perfect environment.

So that is what we did. We narrowed down a list of a half-dozen competitors to two or three that best fit our criteria: free and open source; single-sign-on ready; easy to customize and manage; feature-rich; well-supported and credible.

And then we laid everything out just so. People will need to just ask questions! Cool. Make a forum called “Questions”. People will need to report bugs and stuff! Make a forums called “Errors, Bugs, and Obstacles”. What if we want to just share some news? “News” Tips? “Tips”. Suggestions? “Suggestions”.

Wait...people will need to interact with other students in the same level, area of study, and course, right? Right! Solution: make a forum for every single one of those things, arranged hierarchically. It certainly looked and felt beautiful, like a well-organized documents folder on one’s computer.

With good faith and enthusiasm, we had created hundreds of tiny, empty rooms. By the time we realized our mistake a few months ago, plenty of damage had already been done.

As of this writing, there were 383 forums, sub-forums, and, yes, sub-sub-forums, containing 1,893 topics and 7,687 replies from 5,984 people.

Those numbers work out to about 4.9 topics per forum, 4.1 replies per topic, and a total of 1.6 posts per person. In nearly three years, on average, each little room had just five conversations, with five exchanges each, among some sub-set of fifteen people (pretending, for the moment, that people “stand” in just one room).

Generally, those conversations deserved a larger room with more people. Many smaller conversations — and there are many with just one voice speaking into the void, if we can call that a conversation — might have found eager participants in a larger room. Our system, so meticulously designed, had ultimately just thrown up hundreds of walls to isolate visitors and suppress open exchange.

And that is just the part we are directly responsible for.

The software we use was well-supported, but key improvements like better spam control, up/down-voting of posts, @mentions, and threaded replies relied upon idiosyncratic plugins riddled with conflicts. Nor is the administrative backend much better; power to delete spam posts does not necessarily translate into the ability to do so. These problems, coupled with the atomization of the forums community, have made it difficult for staff and the rest of the community to properly keep up.

The story is not all bad, of course. The averages given above obscure the fact of many thriving conversations and useful exchanges. Nevertheless, the emptiness weighs heavily on the community and every unanswered question and undeleted spam post and unreplied-to hello is something worse than a missed opportunity. This is on us; we have created a very large garden to tend, and empowered relatively few people to tend it. (There are a few heroes of the forums, we must say, and those estimable people, one hopes, know who they are. In a stronger community, they would certainly know who they are.)

For the reasons outlined above and many more, we are trialling new forums software that promises to support a much more robust community. On that proposed platform, we have written about the problems with our existing forums and the solutions we hope to see in a newer system:

Part of what I hope this new software could solve is the atomization of conversations; the existing forums carve up content into a bunch of very tiny rooms…it’s likely that a student taking CHEM101 would not bother to open the door, figuratively speaking, of the CHEM102 room, and vice versa. Nevertheless, a question asked by a 101 student can definitely be answered by a more advanced student and, again, vice versa. Indeed, it is perfectly likely that my chemistry question can be quickly answered by a student taking an English course, but in the current forums system, we may never run into one another.

One can, and should, use the search feature, but in my experience it is fairly mediocre on the existing forums — not terribly dependable and does not typically offer useful snippets for me to choose the best results.

Empty rooms also become self-reinforcing; many people swing by, but all decide that the room is empty and move along.

With these beta forums, I hope that having people and content more mashed together will create not confusion and claustrophobia but rather discoverability of content and more easy connection. Also better search. That does not help now, because not many people are here. But such is the idea.

So what happens now? If all goes well, we rebuild (well…build) a thriving community from the ground up. We have a core of experienced, dedicated students. We have eager staff who carry the experience of having made many mistakes. We have a host of new students looking to connect, some because they buy in to our dreams for our community, but many others because they want to learn and know they can do so better with company.

11 thoughts on “Mistakes were made; or, the atomization of a forums community

  1. January 17, 2015

    Shimmy Reply

    Wow that’s awesome!

    What a great feeling it is, when your university (in my case: Saylor Academy) is upgrading.

    Anyway, the most compelling feature I’d love to see implemented on Saylor, is in-course and in-unit track.
    A user should be able to mark subunits as completed and track his overall progress in the Subunit, Unit, Course, and pathway Minor/Major, something like there is in Alison.
    There should also be an option to set the homepage to the recent activity.
    The entire experience should become learning towards a goal. Not just a collection of material.

    Thanks to Saylor’s great team,
    And special thanks for Sean Connor for the extraordinary service quality!

    Shimmy

    • January 20, 2015

      Sean Connor Reply

      Thanks, Shimmy, for the kind acknowledgement!

      Progress tracking is pretty necessary — it is definitely in mind and, I think, becomes possibly with a stronger reliance on Moodle to serve up courses and not just exams.

      Homepage as recent activity is an intriguing idea. Personally, I really like how going to duolingo.com drops me right where I left off — similar, I expect, to what you have in mind.

      Your most important statement is “The entire experience should become learning towards a goal. Not just a collection of material.” This is what we tried to do (and I think did well) with majors and minors. The main trouble with those is that they are both very attractive and very time-consuming, which potentially sets people up to fail. Note, I am not suggesting that the majors/minors are “bad”, just that there is a down side for the student population as a whole. We tried again to do this, I think rather less completely and perfectly, with our “featured tracks”; the problem there is that we did not follow those up with the tools to manage and remain committed to those tracks. In a larger sense, outside the majors system in our eportfolio, we have not explicitly assisted students with articulating and sticking with particular learning/end goals, which almost necessarily will differ from one person to the next.

      That is what we really have to enable…it is not enough to try to get out of the student’s way; we have to help to clear the way, while respecting the directions that people choose for themselves and supporting self-directed learning.

      • January 23, 2015

        Paul Morris Reply

        Sean

        The conversation around Majors/Minors and recognition of structured programmes of study has, of course, been well rehearsed in the (old!) forums following the revamp of the site. I still consider that the Majors, even though few students completed them, spurred many to consider engagement over a series of courses. The current ‘featured tracks’ seem nothing more than a random collection of courses which have, through one route or another, collected possible accreditation.

        While it is certainly true that the scale of effort required to complete a Major might have been off-putting for many, I have argued that a better approach might have been to offer a number of shorter programmes which could have been stepping stones towards the full Majors. By effectively abandoning this structure Saylor seems to have dropped a unique feature just as edX and Coursera adopt a similar concept through their XSeries and Specialisation tracks.

        • January 23, 2015

          Paul Morris Reply

          Drat! I wish there was an edit on these posts. I seem to have lost a closing italic tag…

        • January 23, 2015

          Sean Connor Reply

          One of the problems with majors is the state of some of the “legacy” courses; there is a bait-and-switch aspect to inviting a commitment to a major but then making that impossible to officially fulfill (as, indeed, there is a similar aspect in not having official recognition in place for completing the majors).

          Your points on the positive values of a major are good. Personally, I would like to see “contract” programs — user-defined, but official. E.g. the student selects from a list of pre-defined programs, customizes the chosen program (optional), or defines a wholly original program (editable) which then is officially part of their user account. Majors, then, would be one of the pre-defined programs, among smaller or career- or goal-specific ones.

          In principle, this is possible regardless of what should ultimately become of the legacy courses. My wish for them is to persist and (one hopes) thrive through a model of community maintenance.

          • January 23, 2015

            Paul Morris

            My thoughts have been running along similar lines. Taking the existing Majors as a exemplars, or even as templates, I would see students being able to customise or indeed produce entirely unique combinations of courses within a range of equivalence values (if that phrase makes sense). That is to say, one could specify that a full 4 year equivalent programme requires so many credits (or courses, if all courses are valued equally) with some distribution of 100, 200 etc courses while a 2 year equivalent would be some lesser figure. Shorter programmes of study might be similar to UK undergraduate certificates being about one year’s worth of work.

            This would give the option to construct programmes of at least three different lengths, the shorter, on completion, capable of being rolled into the longer programmes. Now, while it is possible to have a totally open structure (as the Open University does here in the UK with its modular ‘Open’ degree courses) many students would want at least some guidance in how to build their understanding of a subject (which is why the majority of OU students now take ‘named’ degrees which specify a much more constrained choice of courses). This was the core value of the majors–knowing which way to turn next.

            I won’t rehash here the discussions on the legacy courses although I share your hope that the some mechanism for community support can be arranged. I would hope that the old question of certificates for completion of Majors would be subsumed in the ‘contract programs’, should that idea ever come to fruition.

  2. January 23, 2015

    Paul Morris Reply

    The existing forums have two issues; traffic and technical. Taking the second of these first, just for a change, as the posting suggests, there are technical limitations on the implementation of the existing forums. For example, there is no threading (ie you can’t reply to a specific posting) and while this hardly matters when a thread has only on or two posts, the experience on the NASA co-authored Systems course last year highlighted the weakness of the system under any sort of loading. Incidentally, Saylor are not unique in having problems with email platforms; Open2Study have a similar non-threaded environment with the added ‘benefit’ of posts not being presented in chronological order but rather in order of popularity which means replies can appear before or after the posts to which they refer!

    The biggest problem though for the forums is quite simply volume of traffic. The existing structure would be perfect if there were a few hundreds of thousands, or ideally millions, of students actively pursuing courses. As it is, most courses seem to have no more than a handful of students engaged at any time. Now it is inevitable that discussion is going to be much more fragmented where study takes place at the student’s convenience rather than to a stick timetable (this is one of the reasons most MOOC platforms follow a ‘synchronous’ model for their course presentation) but if there were a larger student body this would be much less of an issue.

    The suggestions made in the posting as regards structure are sensible and the way is always open to revert to the more ‘atomised’ structure should volume of posting make that desirable or necessary. Personally, I have found the current traffic levels such that I have been able to regularly trawl through all the whole forum system answering and commenting where able regardless of course (often simply pointing to our old friend the WayBack Machine for broken links) but this really points towards how quiet things can often be here!

    There is a critical mass for active forums where queries receive prompt answers and discussions consist of more than a single post. Below that point forums can be actively discouraging. The key, however, is not entirely to be found in structure but also in attracting more students ‘through the door’ and keeping them engaged long enough for them to become part of a learning community. To that point the introduction of a long awaited Getting Started’ page is a welcome addition but we students can also help by spreading the word about the Saylor Academy to friends, fellow students on other platforms and through social media.

    • January 23, 2015

      Sean Connor Reply

      I faced similar forums problems in a Google Group connected to the first Power Searching with Google course. Although familiar with the function of Groups, I decided after a few minutes that I would just handle things on my own. Higher-ed bloggers looking at MOOCs in the past couple of years have also brought up how the sort-by-popularity option can hurt discovery and wind up burying posts.

      We will inevitably develop categories in the discourse-based forums, but letting the conversations define those categories seems smarter than trying to define conversations. More people in the door is also very necessary. My hope is that greater transparency and proactive communication on our end, combined with frequent invites/reminders (and a clear onboarding structure on the website) will help to do the trick. That reflects what must be an attitudinal change on our end, too; “discussion forums” is staid and “community” can be wishful thinking, but by identifying the forums as the community, or rather its hub/ether/scaffolding/whatever, I think we make progress. To put it more clearly, perhaps: “forums” is distancing, encapsulating; the forum is a place you go. “Community” is here, right where you are. Changing our terminology and technology while committing simultaneously to a corresponding change in attitude feels like a potent mix.

      Not quite a side note: we will have to think seriously about course discussion prompts. The current system seems to encourage the action of “go to this link; compose your response to this question”. It has the feeling almost of a grade-school journaling assignment or a reading response in college. A better system might be: “Here is a question; there is a discussion happening around it at this link. Go see what people are saying and add your two cents.”

      Thank you for the extensive help in forums and for your many contributions of ideas. I feel that good things are afoot, and both support and pushback from the community are going to help to keep us on course.

      • January 24, 2015

        Paul Morris Reply

        I agree, the discussion prompts generally don’t encourage discussion at all but rather the posting of responses–which is quite different. However, how we (note the use of inclusive language!) choose to phrase the prompts remains a moot point so long as there are insufficient numbers of active correspondents. It requires a surprisingly large number of students to actually generate discussions as opposed to monologues.

        An interesting comparison would be to Open2Study who happily make available a good deal of their student data. Most of their courses attract around 1000 students and they have a fairly decent completion rate of around 20%. Courses are short (four weeks) and designed to be accessed in a synchronous fashion. Lecturers make frequent prompts in the videos encouraging students to discuss specific points and some courses include staff posts to initiate those discussions. Nevertheless, it is rare to find discussions ranging beyond one or two replies and many posts have no replies at all. Now it should be said that O2S have truly terrible forum software but the fact remains that even with hundreds of students engaged enough to get through to completion very few take an active part in discussions.

        Some years ago I recall, but can’t track down the source, reading an article that claimed around 10% of forum users ever posted and of those only 10% would ever post a new thread. That is to say only 1% of visitors would start a discussion. Bearing in mind that many students on educational sites (including university VLEs) never actually use the forums in the first place we can easily see what an uphill struggle it can be to foster active participation.

        Having emphasised numbers it must also be recognised that technology and design can, all else being equal, be important factors. I’ve completed well over a hundred online courses in the last year or two and have found the ‘best’ discussions, generally, on Coursera. That is despite their entirely unpretentious software and quite traditional structure but with the benefit of 50-100,000 enrolled students. By contrast, edX courses, with similar numbers of students, tend to have more fragmented and repetitive discussions and less feeling of community. Interestingly, edX software seems to have paid much more attention to integrating the forum software into courses and to have taken a less traditional approach to structure. Perhaps students find the ‘straitjacket’ of hierarchical structures a support rather than a limitation.

        What most platforms lack are community discussion areas, as opposed to course-specific forums. A student, active and committed on one course may well have little or no feeling of being part of, say, a Coursera community simply because there is no open and enduring outlet for such a sentiment–all her contributions and efforts are locked away in the discussion forums of a completed course. If anything, Saylor is at the opposite extreme; course discussions are relatively few while the focus of activity (such as it is!) is on the community boards. If we disregard mere numbers, this situation could bode well for Saylor’s future.

        • January 26, 2015

          Sean Connor Reply

          Thanks, Paul. My growing sense is that we need to simplify. The word always evokes Thoreau for me, which leads us to this sentiment: “So simplify the problem of life, distinguish the necessary and the real. Probe the earth to see where your main roots run.”

          Our de facto community, if it is to be the community we would wish for, should place itself first. That is, if we attend to the community side of things rather than the strictly course-oriented side of things — and persist — the community will soon enough be able to articulate and resolve more mundane “problems” related to the courses. Put another way, we create a critical, comfortable mass and then help that gathering to find its own necessary structure.

Leave a Reply

Your email address will not be published. Required fields are marked *