#BbWorld17 – Course Design Patterns: Latent Class Analysis with Opensource BbStats

Session Description: Fostering the adoption of teaching and learning technologies requires taking stock of how faculty use the LMS through a process that is reproducible, reliable, and affordable. Studies in student activity patterns, such as the recent Patterns in Blackboard Learn tool use: Five Course Design Archetypes, help us to understand evidence of use. But what about the ways that faculty design courses before any student activity takes place? This session will explore a case study at University of Illinois at Chicago in which course design patterns in Blackboard Learn were derrived using BbStats and Latent Class Analysis.


  • Elizabeth Romero Fuerte, University of Illinois at Chicago
  • Alena Steffen, University of Illinois at Chicago 
  • Szymon Machajewski, University of Illinois at Chicago, Grand Valley State University


  • BbStats is an opensource and free Building Block for institutions using Blackboard. It provides a wide array of usage data.  More information and a download package is available on Ocelot.
  • Here are some sample BbStats reports:

  • Data collected: announcements, assessments, assignments, content, discussion forums, gradebook columns, blogs/wikis
  • Sample size: n=2,561


  • Content was present in nearly all courses.
  • Colleges with more online courses fell in group 1 (holistic tool use).
  • Half of coures show a complementary tool use pattern.
  • Most F2F classes using Blackboard fell into group 2 (complimentary tool use) and group 3 (content repository).

Next Steps

  • LMS governance board
  • UIC innovation center
  • PD for faculty
  • LMS roadmap for UIC

#BbWorld17 – Machine Learning: Past, Present, and Future

Session Description: Blackboard is pleased to welcome Hilary Mason, noted Machine Learning expert and Data Scientist to the 2017 DevCon Keynote. In this session, Hilary will take us on a journey, beginning with the origins of Machine Learning and weaving through to the present state of affairs in the discipline, before embarking on a wild journey to where Machine Learning and Artificial Intelligence will go in the years to come. Attendees should expect to leave this session with knowledge of Machine Learning and how it is applied today, and excitement and insight into the future of smarter education through data-driven design and implementation.

Speaker: Hilary Mason, Fast Forward Labs, @hmason, fastforwardlabs.com

  • We are able to build things now that 5 years ago we thought were never possible via Machine Learning!
  • First big data project was census, in 1888 a patent was filed for counting people. A machine was developed “tabulating machine company” ended up becoming IBM. The punch cards persisted into the 1900’s.
  • Teenager Teaches A.I. to Rap like Kanye

Fred Brooks – “In 1969, I went to te Provost, Prof. Charles J. Morrow, and I said, “We are ready and have the necessaries to build an intelligence amplifying system in which the mind and matching will cooperate on tackling hard problems. Who on this faculty most deservices to have his intelligence amplified?”

  • 1952 Bell Labs, Claude Shannon, Machine Learning Mouse 
  • In 2017, the world is becoming computable. 
  • In many ways, what we are doing now isn’t unique but rather we can do it faster and easier.
  • Artificial intelligence hold great potential for both students and teachers – but only if we used wisely. – The Conversation
  • How AI is Already Changing Business – Harvard Business Review
  • Scientific American – How people read about information on the social web. Based on clicks.
  • AI? > Maching Learning > Data Science > Analytics > “Big Data”
  • Going from counting to predicting with data = data science.
  • “Big Data” – once you build th stack, it’s not just one thing, you now have a bunch of things = business intelligence, analytics, business intelligence, product data science, R&D 
  • Your data can offer the opportunity to see the future, and a lot of people ignore this…
  • Google Maps – A map with traffic data is an example, you can make a decision after seeing the data and you don’t have to think at all about the technology behind the scenes… you just use it.
  • When we as community build great products and that data fades into the background.
  • The biggest opportunities may surprise you.
  • What higher value problem than we have than education.
  • Data science is a new professional role and a “team sport”.
  • Executives and product managers must embrace data, too.
  • We are augmenting human capabilities – not replacing a human with Machine Learning…
  • Strategies: first, invest in automation to support an existing process, then, invest in scaling to allow entirely new products and applications.
  • Often there are ethic issues with data that engieers are often not trained in.
  • Data culture is vital! “Data Driven – Creating a Data Culture” Free book.

Tips for Machine Learning

  1. Research Activity
  2. Change in Economics
  3. A Capability becomes a Commodity
  4. New Data is Available (Wikipedia)
  5. Plus a bit of adventure, some coffee, and bad ideas…

Natural Language Generation

  • AP’s robot journalists are writing their own stories now – The Verge
  • The real impact will be in making complex data simple.
  • Deep Learning – Tools like image analysis makes rich media computable.
  • Summarization is another example, finding similar content across large about of text.
  • Small messy data + intuition is useful!

It’s “intuition” AND “data” it’s not one or the other.

#BbWorld17 – Devcon: @Blackboard Technology State of the Union

Session Description: Come hear Tim Tomlinson @timtomlinson, Chief Product Officer for Blackboard, outline Blackboard’s product development philosophy and progress. Tim will outline Blackboard’s overall approach to product development and highlight key achievements and development priorities. We’ll also announce the winners of the 2017 Hackathon.



  • Blackboard “Your Partner in Change”
  • Sponsors Amazon Web Services and Amazon Alexa
  • 375 developers, 250 institutions, 18 countries at DevCon
  • 20 years of Blackboard – Founded in 1997

Product Development

  • Product Excellence and Quality – 100s of bug fixes with focus on not releasing new bugs, updated technology, simplified releases
  • Quality – New focus on initial quality and maintenance with a new build-and-test automation to increase almost 600% in fix requests delivered, installer size reduced 33% and run time is down 20%, with managed hosting 80% of update procedures are automated, average time of resultion down to 6.5 days, new test automation technology and higher coverage, updated performance testing technology
  • Innovation – Innovating in products and in the way products are delivered (self-hosted, managed hosted, Saas). Recognized by Ovum and PC Magazine. Ally was introduced to assist in providing accessibility for students.  Blackboard Instructor mobile app is brand new for the needs of the faculty persona. REST API, IMS Standards (LTI, Caliper, Common Cartridge) Support, new architectural paradigm through micro-services, and more.

“The widespread availability of business-grade “as-a-service” (aka cloud) offerings is affecting all industry sectors, and higher education is no exception. However, while the institutional discussion regarding cloud has moved from “if” to “when,” most organizations are still challenged as to the practicalities of how they get to the future state. At the recent Blackboard Teaching and Learning Conference (TLC) in Sydney, Australia, Blackboard laid out a transition plan to help institutions manage the move while minimizing disruption.” – OVUM

‘Blackboard’s interface was dated and counterintuitive, and I worried that the company simply couldn’t both satisfy long-time customers and create an interface that would attract new ones. The latest version, Blackboard Learn 9.1, has allayed my concerns, bringing a fresh look and feel—and responsive design—to existing Original courses and a sleek Ultra view for new ones.” – PCMAGAZINE

  • Transparency – Blackboard Community is growing strong to involve clients in the direction of the product and the needs for clients – and to build community among developers, users, faculty, and system admins.
  • Accountability – Tracking via report cards along with building community as a partner to align needs of institutions with solutions that Blackboard provides. Additional Learn and Collaborate REST APIs will continue, additional LTI placements points for Building Blocks in the Ultra experience, activity stream support including Caliper, fully committee to delivery product experiences that conform to global accessibility concerns (W3C, Section 508 Compliance, BSI) with Learn and Collaborate being the most accessible platforms in the education market today. 200 partners and over 3,000 extensions. 
  • NEW! Open Innovation Initiative – A FREE open license for developers for the ability to test integrations.


  • 203 clients on SaaS
  • 68 additional migrations underway
  • 52 more evaluations and pilots
  • Awarded: aws competency status via Amazon

Blackboard is a partner for you to provide flexibility in deployment for your needs:

#CampusTech – Making the Most of Multimedia

Session Description: Technological advances have put multimedia directly into the hands of faculty and students. It is not uncommon to find video lectures, interactive modules, audio/video assignment feedback, and other multimedia tools used in both online and face-to-face courses. Learn how to take your multimedia content to the next level. This presentation explains how to use audio, images and video effectively to make your presentations and courses more engaging. Additionally, you’ll see examples of mini-lecture videos, audio feedback and screencasts from faculty at the University of Cincinnati.

Speakers: JP Long, Nikki Holden, University of Cincinnati


  • What is multimedia? Well… it’s anything digital eg.: webcam videos, screen recordings, lectures, guest speakers, presentation slides, interactive applications, etc.
  • Online Learning at CECH: 8,163 faculty created videos in CECH were produced, 13,927 student created videos, 23,015 faculty created videos in all of UC.
  • Benefits of multimedia: add instructor presence, demonstrate processes, promote UDL, increase retention, reinforce learning objectives, etc.
  • UC faculty use self-produced videos such as screencasts, lectures, software demos, assignment feedback, student video assignments, web conferencing tools, interactive modules, etc. 

Four Best Practices of Design

  1. Contrast
  2. Repetition
  3. Alignment
  4. Proximity

Design tips from presentationzen:

#CampusTech – Creative Leadership: A Human-Centered Approach to Building Technology Strategies

Session Description: Student and stakeholder engagement throughout the strategic development process is essential to creating technology that is relevant, effective and forward-looking. Taking a human-centered approach, the Annenberg School for Communication and Journalism partnered with innovation and strategy firm MO Studio to redesign an annual Student Technology Survey. Initially designed as a tool to collect basic usage data, it is now being used as a platform to increase student engagement. By building student empathy, focusing on collaborative design and generating insight, the school is able to better understand how students learn and faculty teach and to use this understanding to drive better technology decisions. The team will share the story of the approach, outcomes and impact of this process on the school’s ongoing strategic effort to develop an innovative technology roadmap.

Speakers: James Vasquez, CIO, University of Southern California and Sue Tan and Jeff Scheire, MO Studio


  • Survey… it all begin with a tech based survey. What tech are students using… now >>> asking about the student experience and not focused on technology hardware/software.
  • Map a users journey – the best services are designed around a broad understanding of the different interrelated moments along an end-to-end experience that balances, supports, and evolves to support academic priorities: awareness > consideration > commitment > experience > reflection
  • What keeps you up at night?

  • A survey can be more than just a tool to collect basic usage data – desire for deeper use: 1) how might we create a holistic survey that provides insight to prepare use to go from practical to strategic, 2) how to design a survey that is engaging, and 3) how can we use a survey to experiment with new ways of engaging students for feedback and ideas.
  • It’s about how the technology adds value to teh learning experience and not about th technology.
  • Vision > Value > Student Experience > Facilities and Resources
  • Design Thinking approach: 1) understand the problem, 2) reimagine the solution, 3) execute and deliver
  • Survey has 16 questions with 3 open ended… “how might we questions”.
  • Uncover insights (why) from the opinions (what) shared by users.

The Annenberg Digital Lounge was an outcome and provided the insight of importance of communicating with students that such a valuable resource was available.

The Digital Lounge is a creative makerspace where members of the Annenberg community can learn to experiment and play with the digital tools they have received as part of the Digital Literacy Initiative. We are located on the third floor of our new, state-of-the-art building, Wallis Annenberg Hall, located right in the heart of campus and literally designed to foster creativity, collaboration and media. Through workshops, events, helpdesk support and Adobe Certification Courses, we aim to promote a DIY culture that empowers our students to create and learn, regardless of their background or prior experience. From audio to video, to interactivity and the web, we want to make these tools approachable and fun for all!

In closing…

#CampusTech – The Decade Ahead for #HigherEd

Session Description: Higher education is on the cusp of far-reaching changes over the next decade as technology plays a larger role, and as students, parents and educators ask what colleges should teach and how learning should be measured in an era of shifting needs in the economy. Drawing on research from his bestselling book, College (Un)Bound, and his follow-up report for The Chronicle of Higher Education, Jeffrey Selingo will discuss the attributes of a new era of higher education, demographic changes coming to campuses in the next decade, emerging learning pathways, and the roles of technology and the physical campus in the future directions of higher education.

Speaker: Jeffrey Selingo, Washington Post, @jselingo


  • There is life after College [Book]
  • Education will be facing the impact of technology as other fields have… Education is behind when compared to consumer, business, etc.

  • The Growth Era: 1968-1990 every year more and more people came to higher ed.
  • The Tech Era: 1991 – 2010 – Growth of LMS, online learning.
  • The Collaboration Era: 2011 – Current – Enrollment is slowing, changes in economy, different jobs

What we learn?

  • 50% of jobs are under thread from automation
  • Accountants, Technical writers, Commercial pilots, economists, chemical engineers, athletic trainers leading to job losses…

  • Burning Glass Technologies scraped job adds and in 3 of 4 job ads, just 25 skills appeared: communication/writing, organizational skills, planning/detailed oriented, problem solving/customer service, and…. MS Excel

“We need to rethink the concept of college; of highered in general.” #Aftercollege

“In many ways the modern work world looks a lot like a pre-school classroom where curiosity, sharing, and negotiating are front and center.” #Aftercollege

  • Workplace of Old: Showing up on time, following a list of tasks, just a like a class, we follow the syllabus
  • The New Workplace: Mashup of a variety of projects /tasks and the unknown, no one is telling you what to do next

“People know how to take a course, but they need to learn how to learn.” – John Leutner, Head of Global Learning at Xerox #Aftercollege

  • STEM is fine as long as they are there with SOFT SKILLS

Many students wait for college to happen to them:

  • 33% of students had no learning gains
  • 40% of college seniors fail to graduate with complex reasoning skills
  • 50% of college seniors said they talked often with a faculty member abou their career plans

“There are things you’re taught and then there are things you learn… A lot of what college comes down to is not what happens in the classroom. It’s about navigating life and building relationships.” – Rick Settersten, Oregon State

Who Learns

  • Ed was designed to educate a small number of people. Apprenticeships trained people for jobs. This pathway was never designed to serve 18 milllions students. Learners that are entering are changing and changing drastically with financial concerns. Nearly 1/2 of states more than 50% of families in K12 make under $40,000 each year.

  • Age of financial independence for college graduates is 30 in 1983 it was 26. (Georgetown Center on Education adn the Workforce)
  • Students are sprinters (determination, jump right in), wanderers (take their time, takes them a few years to get going and many students have switched majors), stragglers (press pause, the 30th Birthday is their prompt for getting something done)
  • Most will change jobs 8 times in their 20s30s.
  • 3 factors determine success: debt (how much), internships (how many), credential (the value of)
  • 43% of sprinters had less than $10,000 worth of debt at graduation, far short of the $37,000 average for the class of 2016.
  • Internships are crictical are an important cog in the hiring wheel – 75% of new hires come from their internship pool. 79% of sprinters had internships

  • Credentials, those in their 20s make up the largest share (12 million) of the 31 million adults in the US who left college without a degree.

Where and when we learn?

  • Learning is continual… we think of school as something that happens to young people, the future is about lifelong learning.
  • Education is not about being episodic. 
  • Searches on YouTube, 100M+ was the number one search for how to do something.
  • 73% of Gen Xers watch YouTube videos.
  • A lot of growth of “non-organized” educational opportunities: Khan Academy, Lynda, Codeacademy, Skillcrush, treehouse, General Assemply, Skillshare, edX, Coursera, LinkedIn Learning, Facebook Learning
  • Growth of credentials and nano-degrees. Georgia Tech – Masters Degree for $7,000 for Computer Science.
  • Large growth in stackable credentials.
  • Stanford Open Loop University – Design Exercise Idea (could be a game changer…)

How we communicate learning?

  • “The signal of the degree” 
  • Employers now pay more than 22million in education for employees aka Starbucks, JetBlue
  • 1990’s Best Colleges for US News and World Report Rankings are based on inputs.
  • Now The Economist, Forces, Best Colleges Money, WSJ College rankings are now looking at outputs.

“Busy workers don’t have time to distinguish between colleges and universities. We’re doing that work for them and eliminating some of the complexity.” – Bonny Similar, President Jet Blue

Collaboration Examples

  • Coalition for Access, Affordability, and Success
  • American Talent Initiave
  • University Innovation Alliance

“The decade ahead will be about developing platforms for success – both for students & campuses – through institutional alliances and throughout the lifecycle of a students’ education.”

#CampusTech – Measuring Student Success: What #HigherEd can learn from @Fitbit

Session Description: What gets measured gets improved. We are tracking activity like never before using wearables to measure steps and sleep. This data leads to greater awareness, which, in turn, perpetuates positive changes in user behavior. What if we applied this same concept by measuring activity in the classroom? Research shows that the more a student is engaged in class, the better he or she does in the course. Hear how behavioral data from the classroom, like learning management system engagement during the first weeks of class, note-taking and reviewing recorded lectures, can offer real-time insights about student success and transform the teaching and learning experience.

Panel Speakers: Travis Thompson – University of South Florida, Jenna Talbots – Whitehead Advisors, Kristin Eshleman – Davidson College, Mark Milliron – Civitas Learning, Fred Singer – echo360


What can higher ed learn from wearables?

  • Last leg of the journey… data analysis and revealing insight
  • Ability to analyze large streams of data
  • Realizing simple basic things from large amounts of data
  • Giving data to users for the benefit of the user
  • When people interact with their own data they change their behavior 
  • People wear Fitbit for the data – get hooked
  • Using data to chart the way forward
  • Getting data to the people in the way that is useful to them
  • The experts are “us” if we have the data
  • Education is far behind in data gathering when compared to other industries eg. health care, athletics, insurance, finance, etc.
  • In the classroom, can we track what is happening there? Yes we have grades, but what about the information DURING class? 
  • Data is there to empower faculty not to replace them…
  • Data can provide self-awareness and intrinsic motivation
  • Fitbit gives data about the environment, in the classroom what about the environment (lighting, classroom space, temperature, active learning interactions, etc.) – how can we understand student engagement more fully with data?

What are we learning from data?

  • Meaningful stories about data – move the needle, they are compelling as a continuous improvement movement
  • Putting the data in terms familiar to users – this engages them, especially if there are incentives
  • Retention – these data are needed BEFORE the semester is over BEFORE it’s too late
  • Early detection is helpful
  • In class, asking in realtime questions
  • Engagement in class via active learning resulted in lower video lecture watching, why is this… unknown as of yet
  • Awareness allows the change in behavior
  • Engagement data outpaces demographic data quickly – what students DO is far more predicative than WHO they are…
  • Data: 1) relative variables (usage vs other users), 2) consistency (when a pattern is broken there is an alert), 3) Min/Max, 4) Averages across user activity 
  • Pinpoint data is helpful as far as the basics (logging into LMS), but beyond that we need more sophisticated analysis – pinpoint is still valuable as you can see heatmaps of engagement – you can see “cramming” behaviors for example…
  • Nudge campaigns can help with variables of data
  • If learning is a change agent, why aren’t we also learning along side students using data
  • Most faculty don’t have access to the data they need about their students
  • The data about learning is signal processing – in a face to face small classroom this is done extremely well by faculty (body language, eye contact, in realtime) 
  • With data and predictive modeling and simple nudge campaigns (you have done great so far, you can do it, we are here to help!) 
  • Analytics don’t tell you why but the signal is there and it can start the conversation and social engagement between faculty an students
  • Micronarratives can tell you a lot and give you insight of the “weak” signals – with the goal of helping students to thrive and feel connected
  • The integration of a variety of sources provides the strongest light of data
  • Design thinking is helpful in how to manage the data to interact with the users
  • Bridging the people and technology under a shared why is key
  • Extractive data is important, but this is our students data – so how can we go where they are and find out what matters to them and how can we give back to them – to help them 
  • Let’s use all of our sensors (data gather devices) to begin a movement of infrastructure of analytics to create student success scientist…

What about you? Use the information in front of you to start!