« A Half-day in the Life of a Data Scientist | Main | Gelman and I on the Daily Beast »

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Ken

Some years ago I had a data set of medical treatment where some patients moved centres, but because of privacy concerns we didn't get names or even id numbers. So i resorted to dates of births. Then I found that someone hadn't implemented a check for valid date of birth, so some had day and month transposed, others were missing the 19 at the start of the year. I ended up with a lot of merges to test each possibility if the correction wasn't obvious. It was a constant stream of e-mails back to the source of the data asking them what was correct. Eventually we ended with checking for patients that suddenly disappeared and then seeing if a similar patient suddenly appeared at another site.

I think there is now software for medical health data that will determine how close the match is to perfect, and will allow a decision on how close they need to be to be included.

Nate

@Ken - those might have been european folks with the day and month transposed. I try to impress on people the need to use a standard date field. YYYYMMDD. It's sortable, easy to remember, and easy to enter.

The comments to this entry are closed.

NEW BOOTCAMP



Part-Time Immersive
Fall 2019


Link to Principal Analytics Prep

See our curriculum, instructors. Apply.
Kaiser Fung. Business analytics and data visualization expert. Author and Speaker.
Visit my website. Follow my Twitter. See my articles at Daily Beast, 538, HBR.

See my Youtube and Flickr.
Numbers Rule Your World:
Amazon - Barnes&Noble

Numbersense:
Amazon - Barnes&Noble

Search3

  • only in Big Data

Next Events

Aug: 15 NYPL Analytics Resume Review Workshop, New York, NY

Past Events

Jun: 5 NYPL Public Lecture on Analytics Careers, New York, NY

Apr: 2 Data Visualization Seminar, Pasadena, CA

Mar: 30 ASA DataFest, New York, NY

See more here

Courses

R Fundamentals, Principal Analytics Prep

Numbersense: Statistical Reasoning in Practice, Principal Analytics Prep

Applied Analytics Frameworks & Methods, Columbia

The Art of Data Visualization, NYU

Signed copies at McNally-Jackson, NYC

Excerpts: Numbersense Ch. 1, 7, 8. NRYW

Junk Charts Blog



Link to junkcharts

Graphics design by Amanda Lee

Community