When you are running a non-parametric test, like the Wilcoxon-Mann-Whitney test, you can only be 100% of the properties of that test (including Type I and Type II error rates) if the data are continuous. If there are ties in the data, the properties of the test are unknown. This paper shows four commonly used approaches for settings where values might be tied and runs simulations to measure Type I and Type II error rates for both the two-sample t-test and the Wilcoxon-Mann-Whitney test under a range of tied values and a range of distributions. The results are, at least to me, quite surprising. Continue reading

# Category Archives: Recommended

# Recommended: When life gives you coloured cells, make categories

A lot of people use formatting to denote important information in an Excel spreadsheet. In particular, they will use the color of a cell to designate a particular category. Pretty much all formatting information is lost when you import from Excel to R or any other statistical package. But rather than ask people to go back and fix things, there are a few tricks that you can use to recover this information, as is shown in this blog entry. Continue reading

# Recommended: Census Geocoder

This website will take an address, either on a form or as a batch of up to 10,000 addresses (CSV format) and provide latitude, longitude pairs as well as U.S. Census tract information. Continue reading

# Recommended: Data Science Has Become About Lending False Credibility To Decisions We’ve Already Made

A rather harsh and cynical take on data science, but still worth reading. Let me share a story about this. Back in my college days (that would be the 1970s), someone found a New Yorker cartoon and shared it with me. It showed a politician, obviously a very powerful politician because his office had a view of the Washington Monument. He was speaking to his aide “That’s the gist of what I want to say. Now go and find me some statistics to base it on.” So the issues that this person brings up are no different than those from four decades ago. There’s no easy solution to the problem. You can’t say, “I’ll only work with people who have a commitment to the truth, no matter where it might lead” because even people without strong overt biases still have subtle biases that can profoundly skew the results. Requiring a priori specifications and reserving a hold out sample for a final quality check can help, but mostly it is just being careful and detail oriented and transparent in all your work. Continue reading

# Recommended: Harvard University Program on Survey Research

This is a series of guides on survey research, written for the beginning student. It is written from the perspective of Political Science, but the advice works for other areas as well. Continue reading

# Recommended: Webinar Series, Congressionally Directed Medical Research Programs

We live in a golden age of learning, where you find find just about anything you’d ever need to learn from on the Internet. One example of this is a series of webinars about who to get research funding through the Congressionally Directed Medical Research Programs (CDRMP). I have not listened yet to any of these webinars, but they look like they would be very helpful for anyone seeking funding through this program. Continue reading

# Recommended: R Markdown Basics

This is actually a nice “peek under the hood” approach with lots of practical advice about getting that last tweak in to make your results go from good to great. Continue reading

# Recommended: LaTeX/Mathematics

You can incorporate very nice looking mathematical formulas in R Markdown fairly easily. The system relies on LaTeX for displaying formulas and is surprisingly easy to learn. But every once in a while you want to do something a bit exotic, like placing a “hat” in your equation. I’ve typically just done a quick Google search on something like “LaTeX hat symbol” and each different search yields a different website. Recently, I stumbled up a fairly comprehensive guide to displaying mathematical formulas in LaTex. It is published as an eBook.

Note: Some of the examples require additional libraries like amsmath and I haven’t figured out yet how to take advantage of these libraries in R Markdown. Continue reading

# Recommended: Stop Saying ‘Exponential.’ Sincerely, a Math Nerd.

This is a brief plea to avoid using the word “exponential” when you really mean “a lot.” Continue reading

# Recommended: 12 things I wish I’d known before starting as a Data Scientist

This article was recommended to me at a webinar I attended. The author offers very personal and practical advice. The author’s third point “You’ll never have to know all the tools” is quite reassuring. Continue reading