Way back in June we launched a survey asking Saylor Academy users to provide an open-ended response to one question. The subject of this question is one we are always striving to learn more about, as it is at the center of our mission. The question we asked, and got many response to, was:
As it relates to your use of Saylor Academy materials, what does the term “open” mean to you?
There were over 1,100 thoughtful responses, and we really cannot express enough gratitude to our community of learners who participated.
We had also promised back in June that we would analyze and share the survey results once the survey was closed, which brings us to this long overdue post. While we have not yet fully analyzed all that is to be analyzed (there is a lot of data here), we did put together a presentation of some high-level response results for the Open Education Conference in Richmond, VA, and which I’ve converted into blog format here. Following the summary results, at the end of this post, I’ll discuss a bit more about my own thoughts of what “Open” means, my own motivations for conducting this survey, and what sorts of things I hope we as an organization can do now that we do have a better understanding of our users’ perceptions of open.
Results
In order to apply some analysis to this data, staff at Saylor read through each of the 1,162 free response submissions and created qualitative themes with which we could code each response. Many of the responses ended up falling into more than one theme. Some responses, either in part or whole, were also coded in an “other” category based on that portion of the response not fitting into any of the other more prevalent themes.
Word Cloud composed of all free response answers.
In total, there were 12 coded themes that emerged from the response data, plus the additional “other” category of responses:
- Free/Affordable
- Available Anytime; 24/7
- Available Anywhere
- Available to everyone/anyone
- No pre-reqs/previous qualifications required
- Available regardless of age, gender, race, religion, etc
- No limitations/regulations on its use
- Accessible
- OER: Openly licensed; shareable; modifiable; open source
- Transparent (eg. with regards to practices/policies of Saylor)
- Not closed
- “Textbook” Definition
- Other
As can one might guess from the word cloud above, the most common response theme among the participants of this survey was that “Open” means free or affordable.
Overall 532 of 1,162 individual responses indicated that Open means being free, or affordable/more affordable as compared to other options for educational materials, which equates to 45.78% of surveyed Saylor Academy users. While this does not represent a “majority,” this line of response was by far the most common, with 15% more respondents than the second most common response, which was that open means available to everyone/anyone. This result was not much of a surprise to us. Based on the history of more informal communications we have had with learners over the years, the tuition-free aspect of Saylor Academy courses in particular has seemed to play a very important role in attracting and keeping users engaged with our site, and has played a large role in our messaging and communication about Saylor offerings, even though we internally may also attribute other characteristics of our site and courses to our being open.
As a personal aside, I must say that I was happy to see that a full 30% of individuals related the openness of Saylor materials to being available to anybody/everybody. Considering the question asked about “your use,” such a high rate of responses of this type shows, I think, that the essence of openness extends beyond the individual. Even when describing and contextualizing the use and benefits of open materials to oneself, there does seem to be an understanding that the way in which those materials being open have provided access and opportunity for one, means that others could benefit as well.
Other Hypotheses
Beyond anticipating that the “free/affordable” response would be highly prevalent, we did have some other expectations for the sorts of responses we might see, and as such, explored these as part of the first wave of analysis done for our Open Education Conference presentation.
Hypothesis 2: Beyond being free, the second largest theme in regards to the meaning of open will be that Saylor Academy materials are available anytime; specifically, our courses do not have starting or ending dates.
195 (16.78%) respondents included mention on not being restricted by time, or having the right to learn at one’s own pace (or other wording closely related), in their answer to what open means to them. This was not the second largest theme, as hypothesized, but rather, was the 4th most common answer type (not counting the “other” category).
However, the two themes that did have the 2nd and 3rd highest response rates were more “catch-all” type themes: Available to everyone/anyone and No limitations/regulations on its use. Particularly for the latter of these two, the more specific response such as “Available anytime; 24/7” may in fact have been an important aspect in making up the common and more broad definition of open as having no limitations or restrictions on use. Of the 5 predominant themes that emerged having mention of one specific genre of restriction being lifted as a result of Saylor materials being open, (cost; time; place; previous qualifications; and age, gender, race, religion) the removal of time restriction was in fact the second most mentioned.
Hypothesis 3: A minority of the Saylor audience will place value in the open license of our courses as a whole or the OER nature of the resources available within.
Only 10.76% of all respondents, in their explanation of what Open meant to them, specifically placed importance on the ability to share resources, open licensing, or one or more of the 5Rs of Openness. It is possible that some respondents, when indicating more broadly that open meant having restriction free access to and use of materials, meant having the ability to remix, reuse, revise, etc, but if those sorts of use cases were not stated explicitly, responses were not coded with the OER theme.
This hypothesis did seem to be validated by the results, as 10.76% of respondents is certainly a minority.
Hypothesis 4: Among those identifying themselves as teachers, a higher proportion of respondents will mention the ability to reuse materials found on the Saylor site within their own academic setting.
332 of 1162 respondents indicated that they were either a Teacher or Both a teacher and student. While teachers’ responses didn’t often mention specific reuse of Saylor materials within their own teaching settings, the proportion of this subset of respondents who had an answer that was aligned to the OER theme more broadly was indeed higher. Compared to the 10.67% of all respondents addressing OER-related aspects of “Open,” 15.96% of the teacher population responded in such a way (only 8.64% of the student only group mentioned an OER/open licensing related aspect of open).
15.96% itself is not that high of a response rate, but such a low number was not entirely surprising. From conversations with teachers over the years, we know that many who come across Saylor curated content and our courses are not fully aware of the OER nature of those materials, or were not seeking such materials because they were looking to take full advantage of the rights allowed as a result of their having open licenses. Rather, they use them in their own academic settings largely because they are a) free; and b) available in full and therefore able to be reviewed for quality.
What was somewhat surprising was the fact that teachers were even more more likely than students to associate “Open” with free.
Hypothesis 5: Populations in the Global South would have an even higher likelihood of expressing openness as meaning free.
It turned out that this was not the case. As mentioned above, 45.78% of all respondents associated “Open” with being free or very affordable. In contrast however, of the 413 of 1,162 respondents from countries that make up the Global South, only 36.56% associated open with free. “Free/Affordable” was still the theme with the most associated responses, but the percentage of respondents from this subset matching this theme was much lower than the those respondents who identified as living in a country that makes up the Global North (50.87% of responses coded to the “Free/Affordable” theme).
In hindsight, I have one possible explaination as to why we were so off on this hypothesis. Part of the reason we thought that free would be an important aspect of “Open” for many in the Global South was based on the types of inquiries we have received from students from the Global South over the past 5 years or so. Many such students will ask us about the costs to take one of our courses (we of course respond by telling them there are no costs other than the time spent taking them). In establishing our hypotheses, I believe we confused the prevalence of this question with the significance of it. Rather than this sort of inquiry signifying that it is vitally important to our users from the Global South that our courses/content be completely free, I think perhaps that this type of inquiry may underscore the idea that for some, a small cost to gain remote access to something that they otherwise may not have physical access to may be expected.
Compared to the Global North, access to higher education from a physical space perspective is often more limited in the Global South. While cost may indeed be an issue for those in the Global South, it does make sense that cost might not be associated with factors of “openness” at rates comparable to the Global North, where prospective learners otherwise have greater physical access to institutions of higher education and for whom being more selective about where they get their education, based on variables such as cost, is perhaps more feasible. This may also explain why those from the Global South responded by saying “Open” meant “available anywhere” more so than those from the Global North (8.23% vs. 7.10%).
Word Cloud of “Other” responses from Global South population (32.93% of all sub-group responses).
Hypothesis 6a: Length of time being a Saylor student will have positive correlations to a user’s expressing open meaning something related to OER.
This does appear to be the case. 157 respondents indicated that they have been Saylor users for more than 3 years. 14.01% of those respondents had a response coded to the OER theme (as opposed to the overall rate of 10.76%). Similarly, users joining 6 months to 1 year ago and 1-3 years ago also expressed a connection between “Open” and OER at a rate (12.63% and 10.22% respectively) higher than those users who have just joined Saylor within the last 6 months (8.96%).
Also of note is the fact that users who have been with Saylor for longer are much more likely to correlate “Open” with Free. It was not particularly surprising that so many users of longer than 3 years expressed this idea, but it was a bit of surprise in comparison with newer users. In particular, those who have been with Saylor for less than 1 month mentioned free/affordable as an aspect of “Open” only 30.49% of the time, compared with the overall rate of 45.78%, and the rate for users of more than 3 years, which was 57.32%. (I have a hypothesis as to why this might be the case, which I plan to look into and address in a future post).
Much like users from the Global South, newer users were more likely to have responses or part of their response that did not fit into one of the coded themes, but rather fell into the “other” category.
Word Cloud of “Other” responses from newest Saylor Academy users.
Hypothesis 6b: Teachers however would be more likely to relate “Open” to OER even upon initial sign up and use of Saylor materials.
The foundation for Hypotheses 6a and 6b was our belief that the majority of our visitors may not initially come to use Saylor with an awareness that we use, promote, and provide OER, and therefore those newer users would not associate “Open” with aspects more specifically related to OER. Over time however, by becoming a part of the Saylor community and being exposed to the licenses we display in our courses and our mention of OER in blogs, newsletters, social media, etc, we believed students who have been with Saylor for longer would be more likely to associate “Open” with OER. As discussed in the information above, this did seem to be the case.
However, just as we believed teachers overall would mention OER in relation to “Open” more often (Hypothesis 4) we did also believe that teachers, in comparison to our student only users, were more likely to be motivated to use Saylor initially because of the openly licensed courses and content found on our site. Therefore, we hypothesized that even if a teacher was brand new to Saylor, they would be more likely than the average user to relate “Open” with OER related themes and ideas.
Based on survey results however, this would not in fact seem to be the case. Survey respondents who reported being just Teachers, or Both Teachers and Students, and also said they joined Saylor within the past six months (n=74), gave responses related to OER 10.81% of the time. This is slightly higher than the overall response rate for that theme (by just .05%) but quite a bit less than the entire teacher cohort (15.96%), suggesting that even for the teacher population of users, it is the prolonged use and exposure to Saylor materials that helps to solidify the idea of OER and open licenses as core characteristics of the openness of Saylor Academy.
Further comparison of the length of use variables for Student/Teacher user respondents does provide some additional validation of this point. While teachers who have known about Saylor for less than six months are more likely to mention OER in relation to openness than are students who have known about Saylor for the same length of time (10.81% compared to 7.41%), those teachers who have known of Saylor Academy for 3 or more years (n=75) described “Open” in terms of materials being openly licensed, sharable, modifiable, etc, 20.00% of the time. For both students and teachers then, those who have known of Saylor for more than 3 years are almost twice as likely as their peers who are very new to Saylor to relate “Open” to OER.
What “Open” (and this survey) Means to Me
The main aim for this survey was to gain insight into Saylor Academy user’s thoughts about a topic that is near and dear to myself and other Saylor Academy staff: openness. If you have happened to find yourself on Saylor’s About Page in the past few years, you’ll have probably noticed that the mission of Saylor is to “open education to all.” The word ‘open‘ is used there quite deliberately, rather than for instance, provide; supply; deliver; etc, because our own collective understanding and application of the word open is ingrained in what can be found on our site.
For me personally, opening education to all is not just about having a platform that delivers free educational content. Many platforms do that, and many of them are great at what they do. What I strive for however, and much of the reason why I work at Saylor Academy specifically, is to give anyone who needs it access, opportunity, and choice. Part of the way Saylor currently tries to do this is by building and offering openly-licensed courses, which are free for anyone to access, allow for access anytime and from anywhere, and while designed in a complete and very structured way, are open for users (whether those users are students, teachers, or both) to use them however they would like, towards whatever goals they may have.
However, I have also long believed that my own definition of “Open” may not have been the same as those who regularly use Saylor materials. Or perhaps more importantly, I suspected that in the past we at Saylor may not have always succeeded in letting our own users know what open means to us and how that meaning shapes what we do – which of course has an effect on what they as users can potentially gain from participating on our site. As a way to try to confirm and reconcile this, I had the idea to put this question of what “Open” means to our users. Not only was I simply interested in what the range of responses would be, given the large and diverse set of users who come to us for many different reasons, but I hoped to be able to act on this better understanding of our user’s perceptions of “Open”.
Even a high-level analysis of the survey responses does seem to confirm that many Saylor Academy users do not think of “Open” in all of the ways that I do, or rather, place an even higher value on select open practices. As a whole, I do believe this survey was enlightening and successful, in that the results do provide me with additional clarity with which to make informed decisions in the future. For instance, though it was already partly expected, the additional validation that not many Saylor users think about, or consciously associate the ability to reuse, revise, remix, retain, and redistribute, Saylor materials with the concept of being “Open” sparks in me the motivation to think more about why this might be the case, and what Saylor can potentially do to change that in the future. To be clear, it is not my goal that all Saylor Academy users walk away from our site touting the fact that our courses are OER above and beyond everything else. If the most important way that one of our courses is open to an individual, regardless of its reusability and adaptability, is the fact that it was free to access or available anytime, then that is the message they should share with their like minded peers. That being said, as long as Saylor as an organization continues to believe that there is value from a resource being openly licensed, then we should challenge ourselves to make that value proposition more understandable to our users.
In addition to these survey results potentially shaping new strategies at Saylor, I also believe that some of the responses will result in our maintaining the way that we already do some things. For example, a somewhat common response from our users was that “Open,” as it relates to their use of Saylor Academy, means transparent. As part of an organization that I think prides itself with being open about how we design our courses, how we communicate expectations with students, and why we have made some of the decisions and changes that we have made over the years (like our moving of 200+ courses to our legacy website), I was happy to see that respondents of this survey also value transparency. Even if responses of this sort were a minority overall, I hope that their presence will motivate Saylor to continue to operate with transparency.
Ultimately, my desire for the lasting result of this survey is that it causes us to continually challenge what we think we know, and to reflect on the ways that we discuss what it is we do at Saylor as well as how we do things, in an effort to be more than simply a provider of higher education, but rather, as our mission states, an organization that can open education to all.
Whats Next?
As mentioned at the top, there is plenty more analysis to be done with this survey data, and we at Saylor hope that the more analysis we do, the more use we can make of this data in an effort to be better actors within the community of open educational learners and providers. As it stands now, I do feel that I have a better understanding of what “Open” means to the members of the Saylor Academy community. However, there is a lot more specific insight to be gained. Most of the discussion of responses in this particular post focused on those responses at a very high level, or with perhaps one layer of context added (eg. difference between student/teacher responses; new/longtime user responses). While not required of them, most respondents of this survey did answer all of the six optional demographics related questions, which does mean that many very specific user groups can be analyzed in this data. While there may not actually be much difference in the responses between say, female, male, and transgendered students who are all highly educated and longtime users of Saylor, that is something that may be interesting to look at that more closely.
I also hope that others outside of Saylor Academy participate in this sort of research in the future. I mean that in both the sense that I encourage outside review and analysis of the open dataset from our own survey of what open means (which is attached at the end of this post), but also in the sense that I hope more surveys or projects like this are conducted and shared with the open community. As insightful as the results of this one survey may be to us at Saylor Academy, the responses reflect the opinions of just those individuals who have decided to seek educational opportunities through saylor.org, which has its own unique bells and whistles. Responses to our survey therefore may not be of much use to the likes of other open education providers, such as OpenStax or OLI. I do believe however that new groups and organizations seeking to participate in Open Education, or those of us looking to do better in the space, can find real value in a collective body of knowledge that together paints a picture of how different organizational approaches to delivering open education and being “Open” may impact the perception that others have of openness, and the self-perceived benefits reported by those users of open educational resources.
Attachments