What do we say about our students? Do our values align with the words we use? Do they reflect what parents think is important about what happens in the classroom?
In this data story, we take a closer look at 3,694 comments written about 2,862 K - 5 students on their winter report cards.
The Data Similar to last time, there wasn't anything especially fancy in terms of getting the data. We have reports in our student information system that will gather the information and spit it out into a spreadsheet. After that, I added student demographics and program information from another student file using trusty old INDEX/MATCH.
The big challenge part was getting the data clean...or, at least, cleanish. You see, I didn't want student names. Why not? In part because I wanted to make some of the data available to others. This means I needed to strip out identifying information from the text. Also, the names interfered with some of the frequency counting and comparisons I wanted to make. (Aside: Do you realize how many kids are go by the name "Maddie"? I didn't.)
I did a first pass using the SUBSTITUTE function in Excel. I had Excel replace any occurrences of a student's first name with "" to blank it out. However, this only worked when a teacher used the actual name of the student. Many kids go by nicknames, shortened versions of their names, first and middle names, etc. I'm sure there must be better ways than looking through things row by row, but that's what I ended up doing.
The Analysis After the spreadsheets were all cleaned up and ready for church, I looked at some different options for doing the text analysis. I don't have any real experience with this, and while I looked as some fancy options like Overview, KH Coder, and Emosaic, I just didn't have the time to devote to digging into them right now. Instead, I used the WordCounter and SameDiff options over at DataBasic.io.
The WordCounter provided the basis for the word cloud you see in the picture at the top of this page. I used SameDiff to compare lists of comments for male and female students, for example.
There are also comparisons for students who receive special services (vs. those who don't), students eligible for free/reduced lunch (vs. those who aren't), and students of colour (vs. white).
I also used a couple of pivot tables in Excel to summarize and sort through the data—for example, the total number of comments per grade level or per student population.
The Build Compared to the last few data stories we've built out in the hallway, this one is less complicated. There's a lot of paper and stickers, with some foam to help provide dimension to the word cloud.
I knew I wanted the background to be yellow...something bright for spring, but neutral enough that the black lettering could pop. We put the word cloud in the center of the board. It has the 50 most commonly used words. On the outside, we have the four pairs of lists with words that are only found in comments for students in a particular group. The list for our students who receive special services is particularly depressing.
But wait, there's more...
This is our first data story which uses two boards. On the second board, we have information for our students in secondary grades (6 - 12). There are two middle schools and two high schools. Teachers have a list of "canned" comments at each school that they can assign—two per class per grading period—as opposed to the freeform comments elementary teachers create. For these students, we did some simple counts of how many comments per student and then underneath those charts are lists of the most common and least common comments selected. On the right of the board, we have an area for people to leave comments for us.
This second board isn't as sexy as the one for elementary, but I'm still excited that we have represented something for every school and every K - 12 student (even if they received no comments).
Lessons Learned This is one project where I would have loved to have rejected the null hypothesis: the idea that there isn't any difference between student groups. But even with this very basic analysis, I couldn't. Even though most of the text is pretty much the same across student groups at the elementary level, the bottom line is there are some differences in how we talk about boys and girls...and for students of colour...and students from low-income backgrounds...and those who receive special services.
We may never eliminate bias, but if we don't bring it to light, we can't start to address it. While it's great that our district is taking on several initiatives around inclusion and cultural competency, but these are useless if we only use them to pat ourselves on the back for starting them. If we can't change the system in meaningful ways for students, we are just as complicit as those who built the structures in the first place. This display is one way to raise some awareness of what we're up against.
To see more pictures of this project, or view frequency tables of the comments, please visit the page for this data story. As always, comments welcome!
This (school) year, I have built a story about our high school seniors...and one for our sixth graders...and now I'm moving down to kindergarten. For our sixth data story, we are looking at early learning data. What does it mean to be kindergarten ready?
As usual, I didn't set out to tell this particular tale. I was totally going down a different path, thinking about student absences and creating some sort of strip plot...and as I doodled over dinner on the longest night of the year, The Muse came calling. And in about 10 minutes, I had the whole thing in mind about how to present our early learning data.
Our district has been participating in the state-mandated WaKIDS assessment for three years. Teachers collect observational data about each student's development in six categories: social-emotional, physical, cognitive, language, literacy, and math. There is a 9-level scale for each item, with birth to age 1 being the lowest and third grade being the highest. Only eight of these are shown below, as no child in our district was rated the lowest level in any category.
The Data This story was one of the easiest in terms of managing the data files. The state provides us with a file of everything submitted by teachers, and then I merged in a few other demographic and program pieces. This time around, there were no statistical shenanigans, just total counts for each category and level.
The Build Do you remember these? Usually built with pony beads, they have made various appearances throughout the years to signify friendship, solidarity, remembrance, or another purpose. Perhaps you wore them on your shoelaces or the lapel of a jacket.
If you haven't seen these sorts of things, they are constructed from safety pins and beads. I remembered them when I was still pondering the strip plot idea and veered off into how I might be able to string or hang beads from a line. Once I thought of the safety pins, I made an immediate connection to early learning (even if we don't use safety pins for diapers anymore).
After I had the concept, I knew I could build a safety pin with beads for each of our 486 kindergartners. Some back of the envelope calculations showed that I could fit 6 beads (at .5 - .6 mm each) on a 2" safety pin. This would allow me to show all six data points of the assessment, using beads with colours matching the developmental levels indicated by the teacher.
But what else could I encode, I wondered?
In divining the entrails of the data, I noticed that there were a few student attributes that might be worthy of further attention. First was gender. It is not uncommon to hear parents talking about "red shirting" young boys to give them an additional year to mature...but do the data bear this out? I ordered two types of safety pins to help us look at this: gold for girls and silver for boys. The second piece was a student's birthdate. We have a cutoff of September 1. If a student is not 5 years old by then, they can't enroll. But does that really matter---are older students more "ready"? I decided to encode this using different colours of map pins to attach the safety pins to the display. I picked red (because it was not one of the 8 colours of beads) for students who had a birthdate less than six months prior to the first day of school and white pins for older students. Finally, what about low income status? I didn't want to mark this by individual student, due to privacy issues; but, as I organized the pins by school, I decided to order the schools by their overall percentage of students who have low income status. That would give a general comparison. I did look at and consider race; however, I did not represent it with this display because (a) it was actually not as influential a factor as the others for this particular data set and (b) I couldn't represent it as accurately as it deserves. By this, I mean that with student privacy laws, by the time I made a pin showing race and gender, it could become very easy to associate the data with a particular child...especially as most of our schools might only have only one kindergarten student of a particular race. (Yes, we are very white.)
So here, is the final display:
The pins are organized in the space for each school by those kindergartners who were reported as most ready (all purple---or better---in the six categories) to least ready. As you can see in this broad shot, the school with the greatest percentage of low income students (PGS) has a lot lower proportion of all purple pins as compared to BLE, our school with the lowest percentage of low income students.
Here you can see the red vs. white map pins. Do you notice how many white are at the top and how many red are toward the bottom? This seems to tell us that age does matter a bit. Older kids are more ready. And I like seeing this, because while we might be able to talk about potential bias when it comes to gender, teachers don't have birthdays memorized or have an obvious way to connect them while working with students.
It's harder to see in the picture at the left (but you can click to embiggen), but gender also seems to play a role in how students are viewed in terms of readiness. Remember, gold pins represent girls...and by the time we get down to the bottom two rows, there is a lot of silver showing. I do wonder whether bias factors in here. I also noticed when I put the pins together that many of the girls were not rated as highly in math as they were in other areas. Hmmm.
Finally, you might notice the labels beneath each board. These are actually little booklets for each school with charts that show aggregate data. Viewers can look at the distributions for each category or for the demographics of the school. If you would like to see these charts, additional photos, or explore the web-based data workbook, please visit our district web site for this display.
Lessons Learned One of my colleagues has said that this is his favourite story that I've built. I am very happy with it---we encoded a lot of information into a small space and were able to include every child. The pins jingle and move. The paper sparkles and feels sandy. Light refracts through the beads to make them glitter in the light. I think folding in elements of touch and sound is a critical piece of this work. I know that those aspects don't represent anything in particular about the data, but they invite people to ponder...and that's what I'm after: Engagement.
I also think it's interesting to compare schools and see how they used the scale for this assessment. One school (LRE) only used three levels---all of their pins only have purple, blue, or green beads. Another school (PGS) used eight levels and really differentiated. One school (MTS) is very large (106 kinders), but only 2 were rated as being kindergarten ready in all six areas, while all the other schools had a much larger proportion.
As we get ready to work with our community about registering kindergartners for next year, it will be interesting to think about how this display impacts our conversations. I already had one co-worker spend some time looking at it as she thinks about whether or not to enroll her son with a July birthday in school next year.
I am more than halfway through this project of building large-scale, analog, interactive data displays. My goal is to build ten...and I have four more to dream up and construct. The Muse will be back. I don't know when or what she'll bring, but I will be ready.
Last spring, I was invited to put together a 3.5 hour workshop for our state assessment conference that happens every December. I was told that the topic could be anything I wished. And, of course I agreed to do just that and hastily wrote a description in early June...and by the time my summer holiday started in July, I was already panicking. What the hell had I written? How was I going to deliver on this description?
Public narratives about schools often focus on measures of student performance. However, students are more than the sum of their test scores, demographics, and program participation. Join our conversation about how to repurpose and leverage data walls to create, inspire, and communicate with audiences about what matters most. As part of this interactive session, we will share examples of innovative data stories from both school and district levels, as well as tools and strategies for constructing new narratives about student and school outcomes.
Basically, I said I would facilitate some learning around how to build the sort of data (story) walls we've been doing recently. This sounds great on the surface. I truly believe that we have change the types of conversations we are having with and about data in education. In practicality, however, it turns out to be really challenging to develop guidance on how to listen critically for opportunities to transform these interactions...much less how to tap into and be confident about being creative with displays. As Isaac Asimov said, "The world in general disapproves of creativity...and to be creative in public is particularly bad."
I made a lot of notes this summer based around my own questions about using data, and this fall, I've been chipping away at organizing all my notes from the last 18 months into some sort of outline. Combined with the data academy work I'm guiding this fall, I have felt overwhelmed by the amount of adult learning I need to generate...all while trying to do my "regular" work.
But, at long last, I think I'm ready for this workshop...which is a good thing, because it will happen on Wednesday. I don't have a lot of time left to fuss with it. It's time to fish or get off the pot, so to speak.
The slide deck is 150 slides (and 90 MB). I have around 10,000 words in my speaker notes. I've incorporated references to King Kong, Frankenstein, Pink Floyd, ETA Hoffman, and The Inferno. And, I've built a companion site on GitHub for all of the other links, references, handouts, and structure.
We'll see how this goes. Three and a half hours is a long time to spend with 40'ish people on one topic. The whole premise---building these oversized, experiential data installations---requires a different sort of investment and risk-taking on their part. I am not foolish enough to believe that they will all go back to their schools and districts and put up these displays. But let's say three of them do. What a wonderful start...and how I would love to know of others who are doing this work. It all has to start somewhere. I will plant seeds now and see what sprouts.
Student achievement data is among the most public data sets that exist for education. Scores from annual summative tests wind up in all sorts of places, from the front page of the newspaper to Zillow. Because of that, I've been shying away from using these data as the source of a data story. They already get enough attention, right?
But then I started hearing some conversations in my district that piqued my interest. (You can read about that on my other blog.) So I pulled scores for our current sixth grade students. I looked at their scale scores and performance levels from third, fourth, and fifth grade in the area of English language arts. I was interested in a few different things:
What is the average growth students made between grades 3 and 5?
What are the common characteristics of students who are persistently low-performing (or high-performing)? What about the ones in the middle (a/k/a "the bubble")?
What do we mean when we refer to groups of students as red kids or bubble kids?
The Data Calculating the growth was easy: just subtract the 2015 score from the 2017 score. But understanding the significance of that change was a bit more challenging. Is it better to gain 75 points vs the 70 necessary to show two grade levels of growth? Probably, but how much better? I wasn't ready to just declare an arbitrary cut. I was going to need some stats to guide me.
Now, all you stats lovers out there can probably think of several ways to represent this (and maybe a few will leave comments to tell me how I can do it mo betta)...but in the end, I decided to do something very simple. I transformed the values representing the amount of change in scores to z-scores. I normalized them. Why? Because this gave me a quick idea of how many students had performance that was relatively typical to their peers (within one standard deviation) and which ones had even larger amounts of change (two or three deviations from the mean).
The Build Once I had the various groups (-3, -2, -1, 1, 2, 3 SD), I thought more about how to represent this. But to back up a little bit, I have to say that my favourite starting idea was to cut different diameters of dowel rods into lengths that represented the amount of change for each students performance. Since we often talk about bubble kids, I wanted the data to look like bubbles. However, I don't have a saw that would make this simple work...and thinking about all the measurement involved and sorting things out made me a little queasy. I wanted a round shape, though. Lucky for me, Amazon sells bags of wooden disks in 1", 2", and 3" sizes. A-ha! Now I had something that corresponded to the calculated SDs and could easily paint them to match the level of student performance.
I still wanted to work in the concept of change. In other words, do students tend to stay at one particular level of academic performance? To represent this, I decided to stack the disks. If a student had spent only one out of the last three years at the most recent performance level, they got one disk. Two years at a level was shown by two, and three years with three disks.
Another challenge was to represent negative change. The raw difference between 2015 scores and 2017 scores ranged between -101 and +226. Although I could still place the disk on the display according to the 2017 score, I decided to paint the disks black to represent the "hole" in performance over time. Then, a roofing nail was attached to the back and the raw score written on the edge for placement reference.
The other materials for the build included five shades (orange, yellow, green, blue...and black) of shiny enamel paint in sample sized cans, wood glue and super glue, .75" roofing nails (leftover from the last story), fabric with small circles to reflect the bubble theme, and glittery grey paper for the lettering. My assistant and I punched 1" discs in the paper and used an enlarged dot matrix font to guide the creation of the letters. We slid the template for the words behind the fabric on the bulletin board, which was enough for us to see where to peel and stick each dot before sliding the template back out.
The "bubbles" were pushed into the board like oversized push pins, using the y-axis of the board as a guide for placing each disk in position (axis ranged from 2200 - 2800, with 2502 = met standard). There was no particular value assigned to the x-axis. We placed disks in that direction purely based on aesthetics of the overall display.
Lessons Learned When I get an idea for one of these, it comes together very quickly. This one took a week from inception of idea to final product. And yes, that includes a lot of weekend time to pull, clean, organize, and transform the data. But I am always grateful to The Muse, when she deigns to visit, and willing to give in to the compulsion to create these works. I am also grateful to have an enthusiastic and supportive working environment for these. I try to keep them cost effective (the last two have included ~$100 in materials) and am always sure to keep up with my regular work while cranking out this other stuff.
These builds include a lot of trial and error. I am embarrassed to share how long it took us to figure out that we could just slide a patterned piece of paper behind the fabric on the board to use a template for the letters. We had googled for ideas for paper-to-fabric transfer. We plotted different ideas using a measuring stick and T-square. We drew things in PowerPoint. And then, duh, we figured it out. We did try different types of glue to see which was best for getting the non-pointy end of a nail to stick to wood. And so on. The lesson here is that even when you have to kill your darlings during the ideation phase, you can't give up during the build. Keep going.
This time around, we are not looking at one story, but 605 of them. This untidy display shows the pathway all of our incoming seniors are taking to graduation. Will we put a nice little bow on the K - 12 career...or will we leave them hanging this year?
The display is a Sankey diagram made of nails, mason line, foam board, ribbon, paper, and an untold number of swear words. Starting on the left side of the photo you see above, there are four blocks of colour, each one representing one of the high schools in our district. Each block is labeled with the name of the high school and the number of students/strands leading off to the right. The strands then weave through three different graduation requirements. First is passing the state math test, then the state English test, and finally being on track for earning at least 22 credits by the end of this year. Students who have met a particular requirement are grouped at the top...and those who have missed one or more requirements run along the bottom of the display.
I just put the story up on Tuesday, and already I've had a lot of questions and interest in it...more than any other story (so far). It is a bit of a mess, I realize, but so is the process of learning and making one's way through high school. And if you've tried to make 600+ individual strands behave, you'd probably agree that I've managed to do a pretty good job. Inspiration
Honestly, I hadn't planned on this particular story. I've been thinking about one related to transportation for months...but my Muse wasn't having it. And then I got pissed off. You see, our state legislature and superintendent of public instruction worked out a deal to weaken graduation requirements...starting with those pesky tests that some students struggle to pass (along with the alternatives). But kids in our district don't finish high school mainly because of credit issues. Among last year's senior class, only 3 were in danger of not graduating because they hadn't met the state standards. I can't change the law, but I can call some attention to the facts.
I pulled the data for our incoming seniors and the numbers reflected what I'd noticed last year. More kids have met the math requirement than any other. We hear all the time how students struggle in math, but these kids are all right.
I wish I understood more about how my Muse works. I seem to go weeks (or even months) at a time where I can't quite capture magic in a bottle...and then I get the right idea and feel compelled to complete it. I did most of the work for this particular story while I was on vacation in late July.
My dining table is five feet long, so I used it as a template for
the strands of mason line. I wound them around and then cut each end. From there, I tied two strands to 1.75" roofing nails. I chose these nails because they had a sizeable head to keep the line in place and because they were shiny. I wanted something that looked nice.
Next, I used a nail and some grid paper to punch the requisite number of holes for each school into foam board. After removing the paper, I used a hammer to place the nails in, ensuring that all the string was pulled in the same direction.
I used picture hangers on the back of each foam board that was at the top of a section and white ribbon to connect the various areas, such as the schools or the "yes/no" for meeting standards or credit requirements.
I grouped the strands for each school using the data...sorting the correct number of strands into each category. Then, I tied each group so that I could transport it easily to work.
Lessons Learned These projects are exercises in solving one problem at a time. Some problems are related to the data. Others are engineering issues, for example, "How I am going to attach this to a bulletin board without incurring the wrath of the facilities administrator?"
As I've noted with previous stories, I have to let go of my tendency to want everything perfect. At some point, it's more important to just get something out into the world. I have a million other things I need to do (at least it feels that way), but also manage this compulsion to put out this particular data tale. I am very excited about two more ideas I have in the pipeline. My original goal was to produce ten of these. I have a ways to go, but I'm learning more with each one.
I still have to build the companion web page for this display, but can work on that over the weekend. I want to share some data related to which students are not being successful and are in danger of not graduating in June. This, plus what's on the bulletin board outside my office will make for good conversation starters as we gear up for the school year ahead.
Read Full Article
Scroll to Top
Separate tags by commas
To access this feature, please upgrade your account.