Quality in Education

Jay

Another excellent, thought provoking topic. It is also one that has concerned me for some time. If you don?t mind indulging a rather lengthy comment I will point out why.

As school systems have invested in the rapidly improving technology that is available, many have become data rich. Unfortunately, I don?t believe that data is sufficient for making decisions. Russell Ackoff, one of the fathers of system?s theory, suggested that there is a hierarchy for the contents of the human mind that moves progressively as follows: Data - Information - Knowledge - Wisdom. Data, in and of itself, has no meaning; it merely is -- a set of scores taken out of context 10, 94, 75, 32, 98 -- it is only when its relational context is supplied that it becomes information. Information in not necessarily meaningful either -- knowing that the scores are those of a math quiz where 100 represents a perfect score and the mean score was 72 is richer, but it still doesn?t provide you with sufficient meaning to be able to predict how well a particular student will do on the state benchmark. It is only once we move into the category of Knowledge -- experience that is probabilistic and allows us to make accurate predictions about how cause and effect relationships will function in the future -- that we begin to be able to make good decisions.

I would guess that most administrators would respond ?Oh, well put in that context we don?t really mean data, we mean knowledge?. Unfortunately, I am not sure they really do. Here is an example from my recent past. In the state of Maryland, we have had a data system that for years has captured information about whether or not a student was a ?University System Completer?, meaning that they had completed the minimum course requirements for admission to the University System of Maryland schools. This data collection system was quite robust and many systems had institutionalized this information as a performance measure -- the percentage of graduates that were USM completers -- on school improvement plans, scorecards, etc. Over time, however, increasing academic standards and expectation had eroded the value added component of being a USM completer to the point where it had become almost identical to meeting the state graduation requirements, with the exception of completing 2 credits foreign language. When I got access to the complete data set for the state, I calculated a correlation coefficient that showed there was virtually no correlation between being a USM completer and actually going on to enroll in college (r = 0.22). On the other hand, another little used piece of data in the same set called the Rigorous Math Indicator -- student completes a math course beyond Algebra II with a grade of B or better -- has very strong (r = 0.81) correlation with going on to college.

In the above scenario, a data driven decision might have been for a principal, under pressure to increase his percentage of USM completers, to work with guidance counselors to enroll a few more students in two Spanish courses. A knowledge driven decision on the other hand would be to communicate to parents and students the critical nature of going on and taking that extra math course in their senior year even though they don?t need it to graduate.

Sorry for the length of this response. Feel free to delete it, but I thought it was an interesting follow-up to the item on selecting in-process measures.

Jeff,

Thanks for your comments. Don't apologize for the length of your response as it gets at the heart of issues with data driven decision making.

I relate with your comment "It is only once we move into the category of Knowledge -- experience that is probabilistic and allows us to make accurate predictions about how cause and effect relationships will function in the future -- that we begin to be able to make good decisions."

Our challenge is help educators understand how to use data in a meaningful way that elicits knowledge. To take that a step further, our responsibility is to also help students understand how to obtain knowledge from the data they collect.

As always, your comments are appreciated.

Jay

I do agree with you completely. We have to make use of data that is of the outcome of knowledge and experience for decision making. This has also got to be both ways teacher and student.

It can be seen that the need for data driven decision making has its necessary purpose however I do not believe that the data is reliable or structured enough to be sole mean of educational productivity. Once there is a clarification of meaningful test and surveys, then as educators we have to be able to analyze data effectively followed by being able to use the data that is collected properly. Remember knowing is only have the battle. So even though I am against the use of DDDM, I would still consider its use in limitation.

Data driven instruction is a very interesting and highly debatable topic. In discussion with a few colleagues in graduate school, we’ve reached the consensus that data driven decision making in theory seems to be an effective way for teachers, principals and other administrators to improve students’ learning in the classroom. However, there are many obstacles that exist that make data driven decisions not the most accurate and useful means of driving instruction.

One of the most important obstacles my colleagues and I discussed is the added pressures it places on teachers. Teachers are not trained to be statisticians and should not necessarily be responsible for analyzing data and test scores and their implications upon a curriculum without proper and extensive training. This type of training and support is frequently not provided in schools. Another question for debate is should teachers even be ones analyzing the data? Teacher already face the pressures of NCLB and moving their students towards proficiency levels to obtain those higher test scores, but to what end? Is it truly improving student learning or does it become a matter of teaching to the test? Colleagues with first-hand experience with DDDM in the schools report being provided little direction in obtaining or handling and interpreting of the data. This in turn provides a feeling of dissatisfaction and dissolution towards the test taking process and its results therefore teachers are less likely to make real or effective changes in the teaching of curriculum.

One type of data that does have greater potential of influencing effective change in instructional learning of students in the use benchmark testing which helps with instructional decisions for principals and teachers. Benchmark tests are administrated at the beginning of a school year and provide a basis for students’ prior knowledge. This data in turns helps teachers set the pacing and goals of the year based on the scope and sequence of the curriculum. This benchmark is repeated at intervals during the school year to check for improvements and help to guide future planning and to find areas where additional support is needed for students.

So in answer to the question, “are we data driven”, our consensus says no, decision making based on data has been implemented in the ways that promote effective change in the process of student learning.

Group B's DDDM Agruments

According to Doyle, "DDDM also promises real economics of operation, getting it right the first time saves money because it means an end to remediation improved performance across the board." Isn't this a good thing? In today's financial climate (academia is a business), doesn't this make sense? Also according to the author, "...with NCLB data will have to be used, not just collected. It will be used to plot progress (or lack thereof); plan and execute instructional interventions; report results; as well as hold students, teachers, administrators and school systems accountable." Isn't this being done now?

1. How the data is being judged/viewed...

Dennis Doyle noted in his writings that only if data is seen as as valuable can it be embraced. If we view the process of collection and the work required to implement proper actions, then we are starting out the process on the wrong foot.

2. Data must be continuously be used...

Jay Marino pointed out that being data driven, changes should be made mid-stream, before the end product is reached. As highlighted in the RAND article the problem with most state tests is that they are at the end of the year so there is little time, once the results are received, to make any adjustments. Most schools seem to adapt and use teacher/district measurements throughout the year so that they can adjust their teaching to reflect the needs of the students.

3. Is the data meaningful & used...

In the NCLB driven state tests the information arrives too late to be effectively used. The RAND article pointed out it is important to use data that can be meaningful enough that it can be used to guide teaching. It is important when creating or using measurements they are a valid measure. Another problem is that even when great data is collected, if the teachers or schools are not using them then it is useless.

**Data Driven is meaningless without the implementation of Decision Making**

Schools are struggling with the No Child Left Behind (NCLB) mandates because student data is often stored in forms that are difficult to access and interpret. Fortunately, there are new available computer technologies that allow for efficient organization and access to student data. Assessment of student performance is integral in the planning, implementation, and assessment of instruction. Educators must be able to determine whether or not an assessment is a valid and reliable measure of what is being taught and to know what types of assessments are appropriate for district level vs. classroom or individual student level planning. Teachers and school administrators need to be data literate. This can happen only when the district/school leadership actively encourages and assists educators in understanding the importance of relating valid student assessment information to instructional practice. A culture must be created in which student information is considered essential to improving student results.

Key problems in data-decision making in education.

1. Students don’t always see their results. I think it should be mandated that all students view their result. It’s what we call harsh reality. For some students the test is not as important to them until they see how they are affected by it. For example, participating in some programs, or taken out of some programs. Showing their results can also encourage them.

2. The data the schools are forced to analyze is hard to understand. I have been to several workshops where I think I understand, but when I go to another workshop, I get confused. I ask some question and the presenter doesn’t know as well. My biggest thing is how these tests demonstrate accurately our student’s education. There are too many gaps. There are some solutions but this leads to my next concerns.

3. Curriculum across the states. Every county has their own curriculum. That alone is a mess. This is one of the reasons why some counties are doing well and some are not. Again there are some solutions to this problem.

4. Unfairly testing students. In our county we have a lot of different cultures. It is not fair to test a child who cannot speak English.

5. The quality of the data. There are some software/programs where the data is entered in the program by a human not the computer. Or the way the program receives the information can get messed up. Which leads me back to the comment I stated earlier; are these test accurately showing the students best performance?

Here are a few examples of different education systems run: • When the students reach middle school, they are gearing to their career.

• When students reach high school; during their ninth grade year they look towards a major. By tenth grade their schedule is set up to prepare them for their major in college. So what I am saying is by 10th grade the students are taking some classes that will gear him to his major.

• By the end of 6th grade the students have studied another language. By high school or during high school they should have mastered the language unless they changes.

• By the end of 6th grade a profession has been chosen for the students.

Those are just a few. There are some counties in the US that have adopted the first two systems. But there are countries, large and small, that have all the students using the same curriculum, meaning if a student wants to be a biologist, then all the students who want to be a biologist will be taught the same thing.

In the US case, what I was stating is at least each state should create a curriculum for all counties to follow. This way all counties are respectfully held accountable for their students state scores. This would leave the flexibility for each county to differentiate instruction, but at lease we know what is being taught in Montgomery County is the same in St. Mary’s County. Right now I feel it is not fair. State test are a must but a test score should not solely determine where a child is placed. Several summers ago I took an education class and the teacher brought to our attention numerous students who do not test well. When went over different strategies to assess a student and so on. But it was very interesting. I am glad for most colleges, they not only look at your SAT or ACT scores, and they look at you grades. Again I can go on and on about this; but that is one of my solution.

For the students who just come to this country. If they have been in the country 3-6 months, they must take the test. It is sad. I had to proctor a class and some of them couldn’t understand the questions so they blindly guessed, not an educated guess. It broke my heart and of course it also lowered the school AYP.