Monday, July 20, 2015

Misusing The ACS: A Lesson in What NOT to Do

Recently an article appeared in the local paper citing a website created by the Economistsoutlook Blog which purports to compare recent gains in what they term "housing wealth", which is derived from rising home prices over the last several years. As the basis for their measurement they located, and compared, the FIVE YEAR ESTIMATES of the American Communities Survey from 2011 and 2013.

According to the local news article:
"Median home prices in the Utica zip codes of 13501 and 13502, increased from $91,500 to $94,800 and from $92,300 to $92,600 during the years examined, and saw housing wealth gains of $5,950 and $2,973 respectively.
The Herkimer zip code of 13350 saw the biggest housing wealth gain of $5,983 and also saw the largest jump in home prices – from $82,300 to $85,900."


Here's just a few reasons why...

First, there is substantial overlap in the samples periods. The first period (the five year estimate for 2011) covers the following years: 2007, 2008, 2009, 2010, 2011 The second period (five year estimates for 2013) covers these years: 2009, 2010, 2011, 2012, 2013. Three out of five of these years are overlapping, a HUGE thing to avoid when comparing ACS estimates. 

Second, there is no mention of whether these differences are statistically significant or could be due to random variation. A simple comparison of the margins of error would tell you this, yet they fail to provide this information. So in fact, the small differences may be insignificant.

Third, the Census Bureau urges caution because of changes in the housing values provided by respondents from 2007 to 2008. Specifically they say that caution should be used when comparing ACS data on value from the years 2008 and after with pre-2008 ACS data. In 2007 and previous years, the ACS home value question included a series of categorical response options, and then a with a write-in for values over $250,000. Beginning in 2008, the response option became solely a write-in value for ALL homes. 

Fourth, it is important to realize that the value is the respondent's perception of what the house would sell for, and not an objective appraisal of fair market value. 

And finally there has been no adjustment for inflation between the vintage years of 2011 and 2013. In fact, many of the values cited may simply be functions of inflationary pressures, not a true increase in wealth.

In hindsight, it is important to recognize that the ACS can be tricky to deal with sometimes, and especially when it comes to comparing varying years.