A Professor’s Comments on the Budget Survey

Dear Superintendent McMinimee and Members of the JEFFCO Board of Education ,

My name is Danielle Varda and I am an Associate (Tenured) Professor at the University of Colorado Denver. I am writing in regards to my concerns of the JEFFCO budget survey that is currently in the field. In my role as professor at UCD, I am the Director of a Research Center. In that capacity I am often hired to develop and administer surveys, with the intent to translate the data into practice and actionable steps. I am the author of 18 peer reviewed journal articles and have won over $2.5M in research funding. I am probably most well-known as the author of PARTNER. PARTNER is a validated survey used by community groups in over 600 communities across the United States and in 30 countries. I also teach both quantitative and qualitative methods to our Master’s and PhD students. From this perspective, I have three primary concerns with the current budget survey.

1)      The survey lacks reliability. Surveys must be reliable to be interpretable. Reliability requires that questions be vetted to show that each respondent will understand the question. The wording of the questions on the budget survey are such that any two people will not interpret them the same way. The percentage scales are most problematic, with insufficient context to explain the choices, and a scale that has no certainty that when one person picks their split it is meant to be interpreted the same way as another person’s split. The questions also present single-response only options for non-mutually exclusive options, making them uninterpretable.

2)      The survey lacks validity. A survey should include a valid sampling methodology. The JEFFCO budget survey is using a convenience sample, at best, and a completely invalid approach at worst. There is nothing keeping people from answering the survey multiple times (I’ve answered it twice in confusion getting multiple invitations) and appears to have little control over who is taking it. Without a valid sampling methodology, the survey itself is not valid.

3)      The survey potentially violates ethical standards of data collection. The survey does not provide a written or oral consent to participate. The survey does not advise the respondent of any risks for taking the survey, nor informs the respondent of any guarantee to anonymity or confidentiality. While not required that the survey be reviewed by a Human Subjects Protection Committee, because the survey is not administered through a university, it is standard practice to protect all human subjects from risks of harm, discomfort, or adverse consequences such as retaliation. Given the well-documented reports on social media of the discomfort, concerns for retaliation, and overall feelings of risk by the responding community, this survey would likely not pass muster if reviewed by any Human Subjects Protection Committee.

These issues concern me, given the statements by the district that the results will be used to guide the budgetary process. As a researcher and teacher, this survey would not pass if presented by a student, and would certainly never be administered with my name on it. I hope that you also have high standards for the work you do, and I believe that you and the district have been cheated out of what you paid for – a high quality survey.  The firm that authored and administered this survey for the district should be immediately dismissed and I would suggest that the district refuse to pay for such a poor quality product. I am concerned, and would think at this point you should be too, that the data cleaning and analysis will be botched in the same fashion that the survey was designed.

More importantly to me, as a JEFFCO parent, I want the best for my family. When I know that a budget survey could be a powerful tool for providing data to guide decision-making, and could potentially guide the district through a process informed by its constituency, it is very difficult to watch such an opportunity pass. My kids, your kids, our kids, all need these types of processes to happen in good faith, with the best science, analysis, and tools to guide the process.

While I imagine the district must be in a “no going back” ditch at this point, I urge you to take a stance for our district and denounce this survey.  The district paid for a high quality survey, and this is not what you got. I would not put my name next to the survey that is currently going around, as it not only represents an invalid, unreliable, potentially unethical survey, but also implies that you endorse invalid, unreliable, and unethical practices. I hope that is not the Superintendent that you mean to be.

I would be happy to discuss this issue with you in greater detail.

Danielle Varda, PhD