If you want to understand big data, you need to understand Hadoop. Hadoop is the technology underlying many big data efforts. But most of the descriptions of Hadoop are jargon laden and impenetrable to newcomers. Well, maybe just impenetrable to this newcomer. But one great revelation to me was a historical note as to WHY there was a need to develop Hadoop. It was all those pages that had to be indexed by search engines at Google and Yahoo. So I went out to try to find more details. This article, with a ton of references throughout, is an excellent introduction to the precursors to Hadoop, the development of Hadoop itself, and the explosion of systems that used Hadoop as their foundation.

Marko Bonaci. The history of Hadoop. Medium, April 11, 2015. Available at https://medium.com/@markobonaci/the-history-of-hadoop-68984a11704.

A greedy algorithm is an algorithm that tries to maximize some multi-step process by taking the biggest first bite out of the problem, looking at what’s left and taking the largest bite out of it and so forth. The hope is that after all the possible bites are taken, you’ve eaten as much as possible. You can also talk about greedy algorithms in the context of minimization, and that would mean taking the smallest bites possible at each step.

Greedy algorithms are simple and they work well at least some of the time. Here’s a simple example of a greedy algorithm.

Suppose you wanted to make change and you wanted to use as few coins as possible. The greedy algorithm would be to pull out the largest coin needed, look at what remains and pull out the largest coin for that and so forth. So if you needed 32 cents in change, pull out a quarter. What remains is 7 cents, so pull out a nickel. What remains is 2 cents. Pull out a penny. What remains is 1 cent. Pull out another penny and you’re done. 32 cents in 4 coins, a quarter, nickel, and two pennies.

Suppose, though, that you had no nickels in your cash register. The greedy algorithm would hand back eight coins, one quarter and seven pennies. You could do better if you weren’t so greedy on your first step. Hand out a dime on your first step rather than a quarter. Then when all the dust settles, you’d end up with five coins, three dimes and two pennies.

Greedy matching is a practical example worth examining. Suppose you an equal number of subjects in a treatment group and a control group. You want to match them in pairs so that the ages in each pair are as close as possible. The treatment group ages are 58, 63, and 66. The control ages are 55, 60, and 64.

A greedy match would first pair the 63 year old treatment subject with the 64 year old control subject. You’re left with ages 58 and 66 in the treatment group, and 55 and 60 in the control group. Match the 58 years old treatment subject with the 60 year old control subject. That leaves the 66 year old treatment subject, the oldest one in the treatment group, to be matched with the 55 year old control subject, the youngest one in the control group. You would have been a lot better off if you hadn’t been so greedy with the first match. Matching the 66 year old control subject first would have left you in much better position for the later matches.

These are very simple examples, but show that a greedy algorithm may optimize the first step at the expense of the entire process.

There’s another context in which greedy is used. I was interested in using regular expressions to match certain expressions in a string so I could remove them. Here’s an example:

"Carcinoma in situ [47,232 facts]"

I want to remove the square brackets and everything between them. This would involve the sub or gsub functions in R. To match the left square bracket, you need to use an escape sequence because the left square bracket has a special meaning in regular expressions.

"\\["

Then you need to match one or more characters of any type. A dot means any character and a plus means look for one or more.

"\\[.+"

Then you need to match the right square bracket.

"\\[.+\\]"

The R command, then, would be

gsub("\\[.+\\]", "", string_vector)

but there’s a potential problem. In regular expressions like this, there may be more than one match possible. Consider, for example, the following string

"[274.0] Carcinoma in situ [47,232 facts]"

The regular expression could match the  “[274.0]” or the “[47,232 facts]” but it could also match the entire string. The string starts with a left square bracket, has one or more characters, and ends with a right square bracket.

The default match in regular expressions is a greedy match. You try to match as big a piece of the string as possible. So the sub command would not change the above string to something just a little bit shorter. It would change it to an empty string! For this particular problem, that’s not what we want.

There’s a simple change you can make, though. If you follow the dot with a question mark, the regular expression tries to find the shortest match possible. So the command that works is

gsub("\\[.+?\\]", "", string_vector)

and it would change “[274.0]Carcinoma in situ [47,232 facts]” to “Carcinoma in situ “. Note that you now have a trailing blank, but a second gsub command

gsub(" $", "",  string_vector)

will remove that trailing blank.

The first thing you need is a name for your repository. All my repositories (so far) are public, so I want a reasonably descriptive name. The githum site playfully suggests names like “furry-parakeet” and I will stick by that convention. Most of my repositories (so far) are two or three simple words connected with dashes (understanding-lasso, sparse-matrices).

If you use github, they will tell you that every repository should have three files:

1. .gitignore
2. README.md
3. LICENSE

I’m not thrilled about including the third file because it has no extension, but it’s a mistake for a beginner like me to deviate from well established standards.

The .gitignore file tells git that there are certain types of files that might exist on your local computer but which are not needed or wanted in the repository. This is important, because you don’t want a repository that is excessively cluttered. Here are the files that I put in .gitignore.

• .Rproj.user. This is a subdirectory associated with RStudio. The files in it are set up by RStudio itself. Any outsider using RStudio will have these files, so they don’t need (or want) your copy.
• .RHistory. This file is great for when you need to recapture something that you wrote a long time ago. But it is big and unwieldy and unnecessary if your code works.
• *.RData. Files ending in this extension represent things in R that you decided to save for later use. You could include these files, but I prefer not to. I want my R code or my R markdown code to create these files automatically as part of running the program. If there is something that you cannot easily create in your program, such as a model data set, try to download that file as part of your R code, or give instructions on how others can download it in the README.md file. If all else fails, you would be better off to store your data set in a format that is human readable (like json or csv). No one likes buying a pig in a poke.
• *.RProj. Your visitors will get files with this extension when they use RStudio on their own system, so there is no need for you to include your copy.

Many people who peruse your github repository will look for and expect to see a README.md file. What should go in this file? It depends, but the general recommendation is to give genereal instructions on what your visitors need to have in order to use the code in your repository. Here’s a draft boilerplate of what I have written, though I expect I will change this to the special needs and requirements of an individual repository.

This is a single file stand-alone application using R Markdown.

It was run and tested using RStudio and the knitr package.

Use the file with the .Rmd extension as input, or examine the file with the .html extension to see what the output is like.

This application does not need any special files beyond the ones listed above.

You need to decide what license you want for your repository. If you are making your code publicly available, you are already committed to a high level of sharing, so you should consider a fairly liberal license. I decided on the Creative Commons Public Domain license. It’s a bit more liberal than what I have for the rest of my blog and web site, but I’m fine with that.

I wanted to test some features of the object.size function in R. I created a vector of a million observations and looked at its size. Depending on how I looked at the size, the results seemed inconsistent.

Here’s the R code.

> tst &gt- 1:1000000
object.size(tst)
4000040 bytes
> format(object.size(tst), "auto")
[1] "3.8 Mb"

So how come when you list the size in byes, it is slightly over 4 megabytes, but when you use the format function, the size shrinks to 3.8 megabytes?

Well, the answer is that a megabyte (2^20 bytes) is not a million bytes but rather 1,048,576 bytes. Divide 4,000,040 by 1,048,576 and you’ll see that the result is approximately 3.8.

I should have known that, but the reflexive divide by a million to convert bytes to megabytes is only an approximation to the truth.

You can set up version control entirely on a single computer, but there are lots of advantages to run version control on the cloud. So I went to www.github.com and signed up for a free account. If you have really huge requirements or if you need private repositories, you might think about getting a paid account.

There’s a brief tutorial in github that has you create a simple repository with a single very brief file. You make some changes, and save it as a new branch. Then you issue a pull request to merge the branch back with the main version. I’m not totally new to version control, but the brief tutorial was still helpful. I also needed to install a version control program on my computer, git, that interfaces with github (and with R Studio). I went to https://git-scm.com/downloads to do this.

Then I logged into github and found the “+ New repository” button. After clicking on it, I named the repository (single-imputation), and selected the Creative Commons Zero v1.0 license. There are other licensing options, mostly of the open source or public domain variety, but you don’t have to commit to any particular license as this step is optional.   Then I uploaded a program to the repository (impute.Rmd). There were a few other files, .gitignore, LICENSE, and README.md, that github placed in the repository for me. I made some changes to README.md, just because. You can put anything in this file, but the recommendation is that you should specify what your program does and what it needs to run properly in your README.md file. Then I opened up R Studio and tried to open a new project (File | New Project from the menu) and specified that I should create the new project from version control.  It then asked me which program I was using for version control.  It didn’t work the first time because I needed to restart R Studio after I installed the git software.

This was easy to fix, but it didn’t work the second time either. I didn’t put the https:// in front of my git repository. Then I forgot to include the subdirectory. Phew! Nothing serious, just tedious, and it’s actually working right now. Hooray! When you make a change and want to update version control, you click on the git tab on the window in the upper right corner.

Click on the file that was modified, peek at the differences between the two files, and then briefly describe the changes in the commit window before clicking on the commit button.

Before I could commit the changes, I had to modify the global configuration file to list my proper name and email address. It was easy on my first computer, not so easy on my second. For some reason, I got stuck in a Unix-like editor called vim, which is kind of like one of those awful phone trees that you get trapped in when you call an 800 number.

A major reason for version control, as I understand it, is that once you have a working program, you don’t want to break it with new enhancements. Test out your enhancements on a separate branch, and then if you are happy, merge the enhancements back in with the original program.

The other big reason for version control is a bit more obvious. If you make changes and don’t like the changes, you can easily revert to the earlier code. Or you can compare the new code and the old code to figure out where you started programming in La-La land. Since version control is in the cloud, you can easily switch from computer to computer. I had been using Dropbox and Amazon Web Services to facilitate work across multiple computers, but it doesn’t hurt to have another option at your disposal.

Finally, github allows you to easily share your code with anyone who might be interested. Your work might be such a narrow niche that only two or three other people in the entire world would be interested in your code, but for those two or three people, github is a godsend.

Now before I used version control, I did an internal numbering system. My first program was something01.txt and when I had anything other than a trivial change, I saved it as something02.txt, and so forth. If I was working with a client, I would joke about how when we got to something99.txt, we would have to quit and publish. I don’t think I ever got above 60, to be honest.

I’m still getting used to version control, but it’s not all that hard even for an old dog like me.

The key publication about i2b2 is

Murphy SN, Weber G, Mendis M, Gainer V, Chueh HC, Churchill S, Kohane I. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). Journal of the American Medical Informatics Association 2010. 17(2); 124-130. Available at https://jamia.oxfordjournals.org/content/17/2/124.full.

The i2b2 database has five tables. Across the five tables are only few dozen fields (a statistician might use the word “variables” here). How can the i2b2 system store something as complex and heterogenous as the medical record on an individual with so few fields? Many other data bases, such as the National Ambulatory Medical Care Survey, published by the CDC, will have hundreds or even thousands of fields.

The i2b2 system uses a model called the Entity Attribute Value (EAV) model. Pretty much any data table, no matter how many fields, can be restructured into a table with only three fields. Here’s a simple example.

id sex race age  bmi
01   M    W  72 26.7
02   F    B  45 24.2
03   F    W  19 37.1
...

This is a fairly standard format. There is a primary key, id, and four fields: sex, race, age, and bmi. In the EAV format, the primary key becomes the entity, the field names become the attribute, and the data within the various field names become the values.

entity attribute value
    01       sex     M
    01      race     W
    01       age    72  
    01       bmi  26.7
    02       sex     F
    02      race     B
    02       age    45
    02       bmi  24.2
    03       sex     F
    03       race    W
    03       age    19
    03       bmi  37.1
    ...

The first time I saw an example of the EAV model, I was a bit bemused. Why would anyone store their data in such a strange way? There’s huge variation in the data stored in the value field: some strings of various types and a bunch of numbers, with potentially a huge number of units of measurement among those numbers.

The EAV format is often called the long and skinny format (I like tall and thin, myself). If your original table has hundreds of fields, then the EAV format will have records by the thousands. But no matter how many fields you have the EAV format will only have three.

There are many advantages of the EAV format, but the most critical one from the perspective of i2b2 is that it represents an efficient storage model for sparse data.

Think about your typical encounter with a doctor. It often starts with a measurement of weight and blood pressure, but then it could go off in many different directions. You might have some blood drawn or maybe not. If you do have blood drawn, it might be to measure hormone levels, or enyzme markers of kidney and liver function, or maybe cholesterol values. You might provide a urine sample instead of or in addition to the blood draw. You might not get either of these, of course. You might leave with a prescription. If you do, it might be for one of thousands of drugs available by prescription. The dosing frequency of the drug, the dosing concentration, and the mode of delivery might change.

The point is that there are thousands of things that might happen, but typically only a few dozen of them actually do happen. If you tried to have a field for every possible thing that could happen, you’d have thousands of fields. But it’s worse than this. The things that might happen in a medical context might happen multiple times. So you might have multiple blood draws across different visits and you might leave with the same prescription each time, but probably not.

Each of the various things that end up happening in a health care setting are documented as separate attributes. So your visit might have a weight attribute, a blood pressure attribute (actually two attributes for systolic and diastolic pressures), a cholesterol level attribute (again probably two attributes for your good cholesterol and your bad cholesterol levels), and a prescription attribute (again multiple attributes, defining the drug, dosage, route, frequency of administration, etc.). If something didn’t happen to you during your visit (for example, they didn’t measure your weight), then that is just an attribute that is not reported for you.

Now my description of the EAV format is a gross oversimplification. You need more than three fields in the EAV format. But the general concept is the same. The entity may be a series of primary keys, for example, and the attributes often require a supporting table to provide important details about the attributes. The medical data stored in i2b2 has to distinguish distinct patient encounters.

Other key references include

Dinua V, Nadkarnia P. “Guidelines for the Effective Use of Entity-Attribute-Value Modeling for Biomedical Databases.” Int J Med Inform. 2007; 76(11-12): 769–779. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2110957/.

Prakash Nadkarni. “An Introduction to Entity-Attribute-Value Design for Generic Clinical Study Data Management Systems.” Center for Medical Informatics, Yale University Medical School, n.d. Available at: http://ycmi.med.yale.edu/nadkarni/Introduction%20to%20EAV%20systems.htm.

Wikipedia. Entity-attribute-value model. Wikipedia, n.d. Available at: https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80%93value_model.

I have been on the receiving end of lots of programs and data sets that are poorly documented and lots that are well documented. I also have helped set up and run a working group on secondary data analysis, so I have lots of experience seeing how the professionals at groups like the Centers for Disease Control and Prevention set up clear and easy to follow documentation for some pretty complex data sets.

So if you’re like me and you’re creating programs and data sets that others statisticians will use, you need (I will argue at my meeting) formal written programming standards. What are some of the things that you should put in your programming standards document?

Variable naming convention

One of the simplest things you need is a variable naming convention.This may seem trivial, and sometimes it is, but at a minimum a consistent naming convention will make a program easier to read. It also greatly simplifies program maintenance.

The story I tell when I talk about variable naming conventions is a true one, actually. There was a group called Writer’s Exchange that wanted to set up a website. They named it www.writersexchange.com, which seems like a pretty good name except that you can accidentally read it as “writer sex change. So to avoid confusion, you need some way to easily separate words in a variable name. You can’t use blanks: most programs won’t allow it, so the three common approaches are camelCaseNames, underscore_separated_names, and dot.separated.names. The actual choice varies. Microsoft likes camelCaseNames (and a closely related variant they call PascalCaseNames). Google likes underscore_separated_names for some of their programming languages, camelCaseNames for others, and a mix depending on the type of variable for still others. Their R programming style guide recommends dot.separated.names but says that camelCaseNames are acceptable. The Google Style Guide for R recommends AGAINST underscore_separated_names, but other style guides recommend AGAINST dot.separated.names instead.

I like underscore_separated_names, but would be fine with any convention. R itself is terribly inconsistent. So, for example, the argument for handling missing values in the mean function is dot separated (na.rm). But in the table function, it is camel case (useNA).

But don’t you be like R itself. If you adopt a naming convention, you should follow it religiously. Here’s some of the reasons why.

First, a mix of naming conventions makes your code fragile. Suppose one programmer is cumulating events in event.counter, and a second programmer starts accumulating the events in event_counter, then the best you can hope for is that you get an error code that one of the variables is uninitialized. If you don’t get that message, then some of your events will not get counted properly.

Second, a mix of naming conventions messes up an alphabetical listing. If you have variables named

• vowel_A
• vowel.E
• vowel_I,
• vowel.O
• vowel.U

when you list them with ls(), they come out listed as

• vowel.E
• vowel.O
• vowel_A
• vowel_I
• vowel_U

Finally, variable names in R can be manipulated as easy as data sets. If you name variables consistently, then you can more easily loop across variables, pull out subsets of variables, and re-order your variables.

Another convention that a lot of R programmers use is that they use nouns for variables and verbs for functions. An experienced R programmer will recognize a function because it is usually followed by a parenthesis, while most variables (except scalars) are followed by one or two square brackets. But less experienced R programmers and even experienced R programmers who are sleep deprived will get confused. So “age.in.years” might be a vector of patient ages and “calculate.age” might be a function that calculates an age from a exam date and a birth date. This also kind of makes sense because a function takes an action on one or more variables. Here’s an example:

age.in.years <- calculate.age(event.date,birth.date,units=”years”).

It almost reads like an English sentence.

Some programmers use all upper case for constants, but I dislike this approach. It especially bad to have upper case and lower case versions of the same name. R is case sensitive, but if you start relying on this, then you will get in trouble when you switch to coding in a program like SAS which is not case sensitive.

Finally, avoid acronyms and abbreviations. A few acronyms like bmi for body mass index might be okay, but try to spell out anything that is not instantly recognizable. You should also avoid abbreviations. They can sometimes be misread. So, does “temp” refer to temperature or a temporary variable? There’s also more than one way to abbreviate. Your temperature variable could easily by “temp” or “tmp” or even “t”.  This can lead to the same sort of problems that you might have with “event.counter” and “event_counter”.

One last word. If you decide to adopt a naming convention, don’t settle for 90% consistency or 95% consistency or even 99% consistency. The closer you get to 100% consistency, the more the exceptions will bother you. There’s a quote from one of the earlier Star Wars movies where Yoda says “Do or do not. There is no try.” A partial attempt at enforcing a naming convention is worse than not having a naming convention at all.

Now when I said 100% consistency, I didn’t really mean it. There might be a few exceptions to variable naming conventions. First, don’t go back and rename all the variables in a program that already works. That’s a fool’s errand and will probably end up breaking the program. Second, external demands might require you to deviate from a naming convention. This could occur if you are inserting code from a different language. So if you are running SQL statements within R, don’t feel that you have to abandon the norms of SQL. Dots in SQL have a special meaning, so dot.separated.names are at a minimum, going to cause confusion. You might also need to deviate from a naming convention if some of the data sets you produce need to be read by a different statistical package with conflicting requirements or different naming conventions. The point is that you only deviate for a very good reason, and then you document that deviation explicitly in the program.

Documentation headers

Every function in R should have have a documentation header. This is a brief explanation of what the function does. It also explains what form the input takes and what the output looks like.

The documentation header is written before the code, or sometimes, it is written simultaneously with the code. But it is never written after the code is written. If you defer writing the documentation header, you will end up skipping it, or skimping on it.

In R, a function often can take different types of input. So the sqrt function, for example, can accept either a scalar, a vector, a matrix, a data frame as input, but not a list. This can save you a lot of time by avoiding various loops, but if you don’t know this, you can’t take advantage of it.

All small files are readable by a human

There are some files that are stored in binary format for speed or storage efficiency. But for small files, the gains are not worth it. You should strive to use formats that are easily readable by a human. Some of the people who read your code will not have the tools to open a binary file and inspect it. Examples of machine readable files are comma separated values, json, yaml, and xml (some might argue that xml files are too dense to be easily read).

Every file should, if possible, follow the same naming conventions. So use nouns for files that contain data and verbs for files that contain programs. Take the time to spell out abbreviations and avoid acronyms. Don’t mix comma.separated.file.names, underscore_separated_file_names, and camelCaseFileNames.

Place a brief comment header on the top line of every file. In R, the default for reading in comma separated value files is to ignore any line that starts with a pound sign (though you can override this if you need to). For a program file, note any other program or data files that this program depends on. Also explain briefly what the program does.

The brief comment helps a lot because many file management programs offer a preview of the first few lines of a file. This can help an outsider quickly browse through a large set of files and pick the ones that are most relevant for further study.