Tag Archives: Big data

Recommended: Microsoft is creating an oracle for catching biased AI algorithms

This page is moving to a new website.

Artificial Intelligence (AI) algorithms that are used for crime detection, loan approvals, and employee evaluations are considered by many to be objective, but they can sometimes have many of the same prejudices and biases that human evaluators have. Given the opacity of many black box approaches to AI, this could lead to serious problems with fairness and equity. This article discusses an admittedly imperfect approach by Microsoft to evaluate these AI algorithms using (surprise!) an AI algorithm. It flags situations where an algorithm appears to have problems with unfair differential treatments  based on race, gender, or age. Continue reading →

Recommended: Statistical and Machine Learning forecasting methods: Concerns and ways forward

This page is moving to a new website.

At first glance, you might think that this article looks like a vindication of traditional statistics. Classical time series models (methods that were available in the 1960′s) outperform newer machine language forecasting models. Then, you might worry that the comparisons were unfair. But neither viewpoint is accurate. The classical time series models have certain structural advantages for certain types of problems, but you might be better off with machine learning if you use classical time series as a preprocessing step, such as de-seasonalizing your data. If nothing else, this article provides a nice overview of some of the major machine learning methods. Continue reading →

PMean: My work on a CTSA grant

This page is moving to a new website.

I’m on a Clincal and Translational Science Award (CTSA) research grant (5UL1TR000001-05, formerly 1U54RR031295-01A1), which is pretty cool. My name is even mentioned a few times in the grant. I thought that as I plan what I would do for this grant, I would see what the grant promised and write down what, exactly, that those promises mean. As I talk with various people (especially Russ Waitman, who is supervising my work on this grant), I will revise and update my plans. Still, I thought it would be valuable to put some thoughts down now, both to help me focus on what I should be doing and to offer an early draft of those ideas to the various people that I will end up interacting with. Continue reading →

Recommended: The Origins of ‘Big Data’

This page is moving to a new website.

I’m not a big fan of the term “big data” but I’ve been applying for a couple of jobs that ask for expertise in big data instead of expertise in Statistics. So in one of the cover letters, I wrote that I was doing big data analysis before the term was even coined. That forced me to do a quick fact check, and it looks like the term first came into wide use in the late 1990s. Here’s an article on the person who first coined the term “big data.” Continue reading →

Recommended: Can A.I. be taught to explain itself

This page is moving to a new website.

This is a nice article in the popular press that talks about some of the problems with “black box” models (in particular deep neural nets) used extensively in many big data projects. It is a bit shy on technical details, which is understandable for a paper like the New York Times. Even so, the stories are quite intriguing. This is a wake up call for those people who fail to recognize the serious problems with many big data models. Continue reading →

Recommended: beanumber repository

This page is moving to a new website.

This is the github repository of Ben Baumer. He is one of the co-authors of “Modern Data Science with R” and the data and code from that book is available here. He also provides code and data for OpenWAR, an open source method for calculating a baseball statistic, Wins Above Replacement. Finally, there is an R library for extracting, transforming, and loading “medium” sized datasets into SQL. Medium here means multi-gigabyte sized files. Related to this are a couple of “medium” sized data sets from the Internet Movie Database and from the NYC CitiBike dataset. Continue reading →