There’s something so satisfying (for programmers anyway) about obtaining a huge chunk of raw, frenzied data and pounding it into a shape that integrates nicely into a normalized database, and cranking out some slick display algorithms for data visualization. Generally gone are the days when a client emerges with such a simple, sweet, organizational pleasure.
Enter OUSD (Oakland Unified School District)!
I admit it; I recently got my fix. They came to us with the ceremonial Excel file on which the livelihood of an entire organization is founded. “With great data comes great responsibility,” wrote the great Stan Lee. (Or was it “with great power…”?) At any rate, our responsibility is clear: make-data-look-useful. Specifically: Learn what the data means. Store it in a straightforward and flexible manner. Handle manipulation of the data according to its prescribed relationships. Explore the various possible means of generating a visual display of said data and its structure so that human beings can interpret it.
This task presented some interesting aspects. The original data source itself is a subset of an hierarchy of implied relationships that must be extrapolated from its attributes. It’s a bit like building a genealogy tree that goes back a hundred years by gathering the knowledge of one household’s family members. One datum needs to be “expanded” into a chain of nodes, merged with those to which it is related, where each node becomes a virtual owner of other pieces of data. The other interesting aspect is in the display routines, which must be aware of patterns in the data, such that one node must know of its particular incarnation in another context and the new set of attributes it has there.
In the end, what’s it all for? Well, if we did our job right, someone has overlooked the incredible complexity of the data with its mind-numbing rules and exceptions in order to spend the absolute least amount of time necessary to examine the output generated and leave the computer to get back their simplified duties. Of course, that’s the beauty of it! Less hassle and increased efficiency through the clarification of an otherwise overwhelming wealth of information.
You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.