READY, FIRE, AIM: The Need for More Accurate Building Energy MODEL Accuracy Assessments

Evan Mills and Danny Parker - April 9, 2012

Mills is a Staff Scientist at the U.S. Department of Energy's Lawrence Berkeley National Laboratory

Parker is a Principal Research Scientist at the Florida Solar Energy Center

Response to "Energy Modeling Isn't Very Accurate"

There is an old expression: "Models model modelers," and it couldn't be truer in the case of building energy models. On the one hand, these models mirror their authors' view of the world, plus, often more importantly, their users' skill in assigning inputs and interpreting the outputs. The results of a four-year-old Energy Trust of Oregon analysis referenced in Martin Holladay's recent post are more a product of how the models were used than what the models were capable of.

The Oregon study has been repeatedly invoked to make a series of points. But, citing a flawed study over and over doesn't make it true. In fact, such repetition does a distinct disservice to the building energy community—creating a mythology of misinformation.

Truly useful accuracy assessments couple a rigorous and transparent methodology with constructive forensics to help understand the sources of inaccuracies and provide fodder for improving the models. While deficient in these respects, the Oregon study also did not "accurately" characterize the building energy models. The authors chose not to heed review comments from developers of the REM/Rate tool pointing out specific deficiencies in the methodology and analysis. The final product was insufficiently documented to allow independent validation of the results. After publication, the Energy Trust of Oregon opted not to respond to requests for more transparent documentation to help identify the sources of asserted inaccuracies. What we can tell from the document is that the study hamstrung at least one of the tools—the Home Energy Saver (HES)—in multiple ways. Other problems with the experimental design are too numerous to go through here. We have now rerun HES against these same Oregon homes with greater quality control of the input data and full inclusion of known operational and behavioral factors. The median result agreed within 1% of the measured energy use, and with much lower variance between actual and predicted use than suggested in the Oregon study. For those who prefer not to consider occupant behavior ("asset analysis"), HES still predicts the central tendency of this set of homes much better than represented in the Oregon Study (but there is more spread in the results and many more outliers).

Unfortunately, the Oregon study (and more recent derivatives) has become the thing of urban legend: 'No reason to bother with detailed simulation. It is not accurate, nor worth the trouble.' However, in writing this response we wish to very strongly contest this conclusion and to let readers know that our recent research indicates the opposite is true. Not only do detailed simulations work well, they work better than simple calculations and provide greater predictive ability, especially when the more detailed operational level characteristics are considered.

Oddly, even taking the study at face value, perhaps its most important charts (not among those included in the GBA article), and other metrics found in the report show HES performing better than the other tools, including as defined by symmetry in the distribution of errors around the line of perfect agreement. This fact was de-emphasized in the study, and instead the reader's attention was directed primarily to average (rather than more appropriate median) results, coupled with a focus on "absolute" errors (obscuring problems of asymmetrical errors in some of the tools and inability to track realistic envelopes of energy use). All of this points readers toward a misinterpretation of the relative accuracy of the tools.

The refrain about simpler models producing better results is a red herring. Indeed, as show in the Oregon study, the more highly defaulted version of HES ("HES-Mid") had far less predictive power than the "HES-Full" version. Moreover, while HES offers many possible inputs (ways to tune the model inputs to actual conditions), many of them were skipped in the "HES Full" case, in lieu of set default values that were not representative of the individual subject homes. Additionally unclear to the reader, the Home Energy Saver does not in fact require any inputs other than ZIP code, thus leaving tradeoffs regarding time spent describing the house up to the professional using the model rather than to some remote third party. Of course, some inputs are more influential than others and analysts should focus on the ones that matter most for the job at hand.

That said, the amount of time that running a model "should" take (and the role of operational versus asset attributes) is a function of the purpose to which the results will be put and the definition and level of accuracy required. There is certainly no one-size-fits-all solution. This fact is rather glossed over in the article. User interface design also has much to do with the ease of model use and hence the cost. Moreover, thanks to long-term support from the U.S. Department of Energy, the Home Energy Saver is available at no cost to all users, which helps reduce the ultimate cost of delivering energy analysis services in the marketplace.

On the other hand, it is wishful thinking to suggest that simplified assumptions can capture the complex reality of estimating home energy and the potential for savings, and doing so results in a hazardous folk tale. Indeed, in a new peer-reviewed study of HES accuracy to be presented at the 2012 ACEEE Summer Study, we show that simulation is a very powerful means to improve predictions of how buildings use energy. Our analysis, however, heralds the uneasy conclusion that the importance of household occupants and their habits is on a par with that of building components and equipment.

Whether or not open source, it is important that these models not be "black boxes" and that the user community is free to discover what is happening under the hood. Extensive documentation of Home Energy Saver is open to public and peer review and suggestions on improving the methodology are always welcome. Public documentation of the other tools examined in the Oregon study is much more patchy.

All other things aside, many building energy models are in a state of constant improvement. Dredging up a dated analysis that was flawed in 2008 produces an even more flawed impression four years later as these tools have evolved. Indeed, this may have been the most useful result of that the Oregon study: to draw critical attention to model predictions in a number of areas—infiltration, hot water estimation, HVAC representation and interactions, thermostat uniformity etc. Suffice it to say, that observations from the microscope are now reflected in more powerful and robust simulations four years later. And that was a positive impact. Moreover, no one in the simulation community is standing still; further improvements are being made as our attention is drawn to further phenomenon. Did you suspect that interior walls might influence heat transfer substantially in poorly insulated homes? Or that basements are seldom ever conditioned to the same levels as the upstairs? We did. And addressing these shortfalls are bringing the space heating predictions of HES into ever-closer correspondence with actual consumption.

Beyond Oregon, we are looking at high-quality data sets that span the gamut of geographical variation and housing types: from cold Wisconsin to hot and humid Florida, often looking at fine-grain measurements of end-use loads from monitoring studies that allow further insight.

We fully agree that many nuances in building science may not be well reflected in a given model, and that bad inputs will yield bad outputs ("garbage-in; garbage-out"). These are areas of intense ongoing research and improvement in the Home Energy Saver tool at least. All would also no doubt agree that models are no panacea. The map is not the territory, but it can still help you get to where you want to go.

Simple models can provide rudimentary insight and that should not be under valued. However, the most detailed tools of today can help one to fathom the deeper influences that determine energy use in our homes. Yet, they teach this with a price for admission, suggesting that understanding comes not from simplicity, but rather from its opposite. "If you are out to describe the truth," Einstein famously said, "leave elegance to the tailor."

Evan Mills, Lawrence Berkeley National Laboratory
Danny Parker, Florida Solar Energy Center

Discussion continued here.