In the business analytics universe, the discipline of "business intelligence" is often frowned upon. Business intelligence is primarily generating reports on business metrics, tracking them over time, and producing ad-hoc analyses explaining these trends.
People often complain that such work is not challenging and not sexy. There is a stigma that BI work is data dumping. In reality, good BI work is rare and extremely valuable. Horrible BI work is commonplace and frequently leads to bad decisions.
Design thinking is very important to good BI work. Analytics reports should be designed in such a way that it facilitates managers making the right decisions. When a report is designed poorly, it causes bad decisions.
I just started a new course at Columbia called Applied Analytics Methods and Frameworks. After students did Quiz #1 (worth 5 points out of 100 points in the grading scheme), I have an unusual number of worried students saying they will fail the course. I have taught for over 10 years, and have never encountered something like this.
Eventually, thanks to one persistent student, I found the root cause of the anxiety. It is the terrible design of the "Gradebook" report on our online course website.
The Gradebook is just a spreadsheet with one column for each graded component of the course, and a column at the end of the spreadsheet labeled "Total". Because this is an online website, everything must be in "real time". So after the assignment (Quiz #1), the "Total" column shows the percentage score of the Quiz #1, and it assigns a letter grade to that score, as if this is your projected grade for the course.
Since Quiz #1 is worth 5 points, if a student scored 3/5, the "Total" shows up as 60%, and a letter grade of D is printed next to it.
Students see D and even F in that column, and are immediately demotivated from the first week of the course! Talk about bad design causing emotional harm.
Upon learning this, I sent the following note to my students.
Subject: Gradebook Idiocy: An example of terrible analytics report design
It has come to my attention that the way Canvas's gradebook presents the "Total" grade is idiotic, and causing unnecessary consternation.
You should ignore the letter grade that is being printed on the Grade report. This "real time" grade has no meaning.
For example, if you did 80% on Quiz #1, it assumes that you will do 80% on all of your future assignments, and it "projects" that you have a B-.
This "projection" is terrible on several levels:
- each Quiz is different and your percentage score will vary
- the skills needed to do well on Quizzes are very different from skills needed to do well on Projects and Presentations, and therefore, your performance on one Quiz does not give much information about your performance on other assignments
- Projects and Presentations together account for 70% of the total grade. Taking the first quiz and projecting the percentage score as your "Total grade" is equivalent to giving 100% weight on Quiz #1 and 0% weight on everything else.
Since the software does not allow me to suppress this useless column, I'm asking you to ignore it.
Here is how you should think about your total grade: the Project assignments are 60% of your grade and therefore, if you do well on them, you will almost surely do well on the class. You will do well on Class Participation by attending the live sessions. The Presentation assignment tests a completely different skill set from the other assignments. The Quizzes account for a maximum of 20 out of 100 points in the course. Even if you only score 50% on your quizzes, you would have lost 10 points out of 100 points. If you do well on the other components, you can still score up to 90 points out of 100.
This Gradebook is an example of terrible design of an analytics report. After taking this course, you should know how to spot these issues!