Earlier this week, I shared a post in three places on Facebook: My own page, the AERC page, and Endurance Riders America. I will paste my original post and my edit below, but first I want to say, this is not science! My purpose was twofold. First, I wanted to give everyone an opportunity to express their opinions about the Tevis Cup. Emotions ran high after the death of two horses this year. (Read about it on the page linked above.) In general, being given the opportunity be heard relieves tension (though venting is a mixed bag). Second, I intended to blog about the issue, and wanted to understand others’ perspectives.
I did not and do not intend to carry out any sort of statistical analyses on the data provided. I did not and do not claim that the survey will provide any sort of scientific evidence of anything at all.
Jump to:
The Facebook post and the survey itself
Some of the critical responses I received on the AERC page
On my qualifications as a scientist and why they matter (or not)
The importance of including everyone
What about the “results”?
This is not science! But should we do science in the future?
The Facebook post and the survey itself
I should have been much clearer in my intentions. Clearly, I should have specifically stated that this is not science. No, I was not doing scientific research. I have no intention of analyzing these data with any sort of statistical test. I do not have IRB approval nor do I have any intention of submitting anything for publication based on anything I learn from this opinion poll.
Sadly I did not do all of that 😉 Instead…
This is what I wrote in my original Facebook post:
Hi everyone!
I included a very short opinion poll in my blog post about this year’s Tevis results. I didn’t get that many responses, so I thought I would post here. I’d like to get enough to write up a meaningful report!
After receiving some great feedback, I added:
EDIT: I’ve had some very insightful feedback from people worrying about misinterpreting any findings from this poll and about the ethics of self-report research, etc. I am not doing science here; I am just testing the waters for an eventual blog post…
I am a researcher and COULD get a study approved by an ethics board. Jelena Woehr has suggested I offer to work with WSTF. I will reach out when I have time. Semester is just starting now
The thing is, to do this properly we’d need qualitative as well and quantitative methods. I am mostly a quantitative researcher. Anyone want to help out with the qualitative aspect?
Here is the blog post to which I referred:
Summary of the 2024 Tevis Cup Results
And here is the google form I am using to collect data:
(If you have already completed it, please do not do so again. Or if you have the overwhelming desire to do so, make a note of it in the comments section.)
Some of the critical responses I received on the AERC page:
And by the way, “critical response” is a compliment! I want people to think critically about what I say and do and give me constructive feedback! Most of these were very welcome. The few that were hard to swallow showed that the commenter had not actually read what I had already written and had made assumptions about who I am.
Some great comments:
From Jelena Woehr:
I don’t think it is appropriate to poll on such a sensitive subject and release results without any kind of scientific controls or ethical review process. Sampling a blog audience and posting on social media is not going to get a representative sample, nor are the questions in this poll making a meaningful attempt to determine the respondent’s level of expertise in wilderness trail safety. (Asking about riding Tevis isn’t enough; riding on trails does not make a person an expert in how to build and manage trails in a mountainous environment on protected land.)
I would encourage you to offer your assistance to WSTF as a volunteer in case they want to conduct any surveys of past & aspiring riders, rather than conducting an amateur poll on a personal blog that is unlikely to produce results of any statistical significance, but that certainly could help to direct unwanted attention to the sport. (I can just see some PETA intern stumbling across a blog post and then running Facebook ads saying something like “90% of endurance riders think horses should die in races!”)
Jelena brought up some excellent points here.
First, researchers should always consider whether the sample represents the population they want to make inferences about. Jelena assumed I wanted to sample people who have experience with Tevis, and of course, if I was trying to do so, I would be getting a lot of data from those who have no experience. Below I will talk about why I am interested in the opinions of those who have no experience, but this does not invalidate Jelena’s point. I did not make any attempt to verify experience or level of expertise, and had I wanted to obtain data from a specific subset of people with opinions about Tevis (those with relevant experience), then I was going about it the wrong way.
Second, Jelena mentioned the possibility of working with the Western States Trail Foundation (WSTF).
This is a great idea, and if I can find time, I will contact them. I really had no intention of turning this into anything more than me trying to understand where people are coming from with their opinions. But as a researcher, I totally understand that there is value in conducting good research on the topic.
Note that I was not trying to “produce results of any statistical significance.” Again, this is not science! Horses and endurance are my hobby, not my job. However, I would be happy to work with WSTF to do science, if they are interested (and if I can get some funding).
Finally, PETA.
At risk of getting stoned, I will say here that PETA (People for the ethical treatment of animals) too often does more harm than good. I have not yet been stoned, so I assume they have not yet discovered my Using a whip is not horse abuse, it’s communication post. But still, they are a menace, and I am very aware of how much damage they can do. If I use any of the opinions people share in my poll, it will be carefully curated to ensure there are no misunderstandings.
When reporting results, any good scientist is careful to qualify all claims. (Anyone who claims that what they say is “the truth” is not a scientist. This is part of how we know PETA is not science.)
My response to Jelena:
I am a statistician and published research scientist. I’m not going to make any inappropriate claims based on bad data. If you want me to do so I can readily obtain IRB approval to carry out a study that could be published. I don’t really want to do that now; no time. What I am intending to do is give people the opportunity to express their views anonymously (this is a good thing it decreases tension) and with the feedback offer a curated summary. So far there has been nothing objectionable stated anyway
I am curious to hear how you would obtain a more representative sample, if the population I’m interested in are those who participate in two endurance web pages…?
Jelena’s reply:
I’d say a more representative sample can’t be obtained without first conducting a broader review of the target populations to create a baseline for comparison—or at least working with admins of the pages you’re interested in to see if they’ll share any demographic data they get from Facebook on their audiences. I’ll admit I’m skeptical of opinion polling in general, so honestly I’d be hard to satisfy. But at a minimum I’d like to know basic demographics of both the overall population and the sample self-selecting into answering the survey, and I have hesitations about the self-selection element too—intuitively I suspect the most knowledgeable people are least likely to respond to an anonymous survey that is asking very general questions which may not encompass their entire opinion about the subject. (For example, there’s not an option that would fit someone who believes that the ride should be made more safe to the extent that is possible, but also feels that the tragedies in 2024 were non preventable on the ride management side and should not be the focus of trail safety efforts.)
I appreciate Jelena’s feedback and thanks to her am seriously considering making this more scientific.
From David Lewis:
The problem is that this is an opinion poll that is going to attract all of the Couch Critics, I hope it won’t be used for anything because it isn’t structured in a way that would be useful for the WSTF. There are going to be vastly different opinions from the Couch Critics compared to those of us who attend the Tevis as Riders, Officials, Crew, or Volunteers. There’s nowhere in the Opinion Survey to discover if the Opinion is from a non-rider random street person or crew/volunteer of Tevis. Just if it’s a rider or not.
My answer:
I should have included a crew or volunteer option for sure. Fortunately most people select other and volunteer the info.
I really should have given people more options. When I created it, I only wanted to know if they had ridden or not, but the comments show that people who volunteer and crew have a lot of wisdom to offer.
I should have added in my response to David that most of the “couch” participants are not in fact critics. There is a lot of positive mystique too.
I got many other useful comments, some very critical, some less so. I will address most of them below. There is one more I’d like to share:
The most forcefully criticizing comment:
Hey Jessica, I am a scientist (PhD in Animal Biology from UC Davis) that always loves data! There is no time to analyze bad data. Survey data are tricky as are the questions and responses – responses MUST be collected in ways that can be assessed unbiasedly for statistical analyses. The questions you proposed are NOT scientific or statistically assessable. Folks, don’t waste your time answering this survey. Work with experienced scientists with survey expertise if you want real answers.
To which I replied:
I am not using these data for analyses. I am an experienced scientist and a statistician. And paid psychometrician. But perhaps you have not read my previous answers to comments.
The response:
Jessica, being a statistician or scientists without survey development skills or experience doesn’t equal developing a quality, unbiased survey. Statistician know the art of mathematically tweeking sound science to have the data say what they want or what the funders want is great skill. I no statistician. I only have survey development and analysis skills. As a scientist, you known how formulate research questions, and this “survey” is an opinion piece.
On my qualifications as a scientist and why they matter (or not)
I generally keep my hobby (horses) separate from work. What do I do for a living? By training, I am a research psychologist. I currently teach at Bakersfield College. I spent most of my time as a graduate assistant working as a data specialist on a K20 Center team dedicated to developing and testing educational games. I didn’t do the game development; I did the experimental testing. This means I developed surveys to assess student learning. My job as a data specialist included working with teachers, the College Board, and Oklahoma State Board of Education to set benchmarks.
However, this job was part time. I spent far more time designing my own studies and collecting and analyzing my own data and that of my lab. I developed many scales to measure different psychological constructs (such as moral identity, moral agency, imaginative resistance, cognitive engagement with fiction). That is the meaning of psychometrician. (someone who creates and administers tests… mainly surveys nowadays… to measure psychological constructs and abilities). Psychometrics refers to the measurement of mental things: things hidden in the darkness behind the eyes.
(To learn more about my career in psychology and for links to my publications go to my Research About Me.)
So am I qualified to design a study to measure people’s attitudes towards Tevis?
Most experts would say yes, based on my record of research and publication in psychology. Although I do not work as a psychometrician anymore, I do still consult on study design and statistical analyses. I could also obtain IRB approval to carry out a study that would be publishable in a peer reviewed journal.
But I do not have the knowledge to do this properly and I don’t have time to do the research needed to write something up for publication in a peer-reviewed journal
This is not my area of expertise! In fact, I am not even sure where it would be published. I guess it could fit into social psychology or health psychology or some sort of animal science journal. I could certainly turn it into a study publishable in psychological journals. Doesn’t matter because I don’t want to add this to my too long list of stuff to be written up for publication in an academic journal.
(It’s been more than three years since I collected data for my helmet use study, and I still have not written up the results for publication! Because so many people volunteered on Facebook, especially from the AERC page, I did write up preliminary findings in this blog post: Attitudes towards Helmet Use Study.)
However, I would be very happy indeed to collaborate with someone who wanted to do this. I’d also be happy to work with WSTF or AERC if they are interested in understanding people’s experience and opinions. I am not sure where they’d go with the data.
What were my intentions with my little poll then?
I was planning to write a blog post addressing a few issues brought up by Tevis fatalities. These issues include:
- Inherent dangers of riding endurance.
- What is and is not preventable.
- The shared responsibilities of organizers, owners, vets, and riders.
- Specifics of Tevis and any equine event with historical significance.
- Including the meaning of sports and sportsmanship
- Public opinion.
(PS: This blog post is not the one I was going to write!)
The importance of including everyone
Yes public opinion matters. I got a lot of feedback that boiled down to: “You shouldn’t be doing this poll this way because a lot of people who have no experience with Tevis (“keyboard warriors”) will respond.”
Some people pointed out that I had no way of verifying who was a keyboard warrior and who was someone whose opinion was substantiated by direct experience with Tevis.
I wanted both and I didn’t need verification!
Again, this is not science! I was simply gathering people’s opinions so that I could be more informed about other people’s opinions before I started writing. True, the fact that the form was anonymous means I cannot quote a specific person. But my intention was simply to better understand what others were thinking and feeling.
I do mean all others. Anyone willing to take the time to respond to my request to complete the short survey has potentially valuable information to offer me.
I believe that it’s very important to include opinions of people who have little or no experience with Tevis because they represent public opinion. Public opinion matters. Yes, it may be poorly informed. Many people do leap to conclusions without a solid basis for understanding. But these people can have an outsized influence on policy and the opinion of people even further from “ground zero.”
Witness the power of PETA.
It would be shortsighted for the endurance community to ignore the opinions of those who care enough about the sport to voice their opinion even when they do not have any “Tevis” or “endurance” credentials.
What about the “results”?
As stated above, I am using that little google form to get a feeling for what people are thinking about this year’s Tevis Cup. I never intended to do any statistical analyses on the data gathered. (In truth, there are few I could do without extensive coding.)
The response to my poll suggests that people want some sort of scientific study to be carried out. Maybe I can make that happen. It’s more likely that I do so if someone volunteers to help.
At the moment, I am just planning to use the responses to write an opinion piece. Maybe I will include some data (as in, how many respondents had experience with Tevis and how many believed it should be unchanged, changed, etc.). I hadn’t intended to, but perhaps I should. We’ll see.
It was really just for my own learning. It was never intended to be “official” (whatever that means) or scientific (for that I would submit an IRB application and have an intended scientific outlet).
Above: From the Teviscup.org page. Pictured are Reyna Mero and Chndakas Eklipse, who placed second in Tevis 2024 and won the Haggin Cup, and Reyna’s mother Jeanette Mero, DVM, and Ozark Kaolena SWA, who won Tevis 2024.
This is not science! But should we do science in the future?
As I said before, if anyone is interested in collaborating with me on doing a study to investigate the associations of opinions regarding the safety and future of the Tevis Cup with other variables (experience, sex, age, whathaveyou), please let me know. I would be happy to collaborate. Similarly, I am happy to volunteer my time and expertise to AERC or the WSTF if they have specific research questions they wish to explore. I will share this post with both groups.
If any reader has an idea and wishes to put time and effort into seeing it turn into research, just let me know!