A Preliminary Summary Report of
the NIST-DOD Workshop on V&V
of Computer Models of High-consequence Engineering Systems
Nov.
8-9, 2004
,
Gaithersburg
,
MD
Jeffrey
T. Fong, Chairman, Workshop Committee
NIST, Gaithersburg, MD 20899-8910 fong@nist.gov
1.
Attendance:
The workshop was attended by 50 pre-registered and
invited researchers, of which 19 came from NIST.
A final list of attendees is attached as Appendix D of Vol. 2 of the
workshop documents.
Among the non-NIST attendees were Bill
Oberkampf
and Jon Helton from Sandia who pioneered some of the most
significant V&V research early in the 1990s; Chris
Freitas
and Ben Thacker of Southwest Research Institute, Len Schwer
of Schwer Engineering & Consulting Services, and Hans Mair of Institute for Defense Analysis, who have been key
contributors of ASME V&V guidelines; Roger Logan
of Livermore Lab, and Robert
Eberth of
Marine Corps Warfighting Lab, who are both key contributors to the V&V of
weapons programs; and Simone Youngblood
of Defense Modeling and Simulation Office, Jamileh Soudah of DOE V&V Program Office, and George
Hazelrigg
of NSF Directorate for Engineering, who all manage V&V programs in DOD,
DOE, and NSF, respectively.
Dr. Shashi Phoha,
Director of NIST Information Technology Laboratory, gave the opening remark in
welcoming the workshop attendees. A
draft transcript of her remark is attached at the end
of this page.
2.
Technical Highlights:
The workshop attendees heard a total of 11 invited
papers and 7 short presentations of candidate benchmark problems.
Of the 11 papers, 5 (Oberkampf,
Rainsberger, Helton, Freitas, and Wortman)
were invited from outside NIST, and 6 came from NIST (Sedransk, Fong, Lozier, Filliben, Guthrie,
and deWit).
Two workshop keynote lectures were entitled “Verification,
Validation, and Predictive Capability in Computational Engineering and Physics
(Oberkampf),” and “Uncertainty and Sensitivity Analysis for
Models of Complex Systems (Helton).” Three
survey lectures were entitled “Parametric Finite Element Modeling Across
Many Simulation Codes (Rainsberger),” “Verification, Validation and Uncertainty -
An ASME Perspective (Freitas),” and “Assessment of Uncertainty of Computer
Models using Decision Theory (Wortman).”
A total of 45 discussions-plus-responses were recorded in 5 videotapes,
and their transcripts plus those of two opening and two closing remarks will
be sent to each speaker for review and final approval for inclusion in a
workshop proceedings to appear in the spring of 2005.
NIST researchers addressed issues of statistics in metrology (Sedransk), reference benchmarks in V&V (Fong),
measuring error in mathematical computations (Lozier), role of experimental design in V&V (Filliben),
Bayesian approach to combining results from multiple methods (Guthrie for
Liu and Zhang), and an example application of error budgeting and uncertainty
analysis (deWit).
The workshop concluded with an extremely cogent
remark by Norm Abramson of Southwest Research Institute, the draft transcript
of which is attached.
3.
Accomplishments & Shortcomings:
The workshop began with a multi-disciplinary group of
researchers, but ended as inter-disciplinary because of the opportunities to
interact, both formally at the single-session workshop and informally during
social hours and coffee-breaks.
A day-and-half-long workshop was definitely too short
for a fuller discussion of the outstanding issues in V&V.
However, as stated in the organizers’ message in Vol. 2 of the
workshop documents (see attached page iii at the end of this summary), the
fully transcribed proceedings of 18 presentations and 45 discussions could
yield sufficient evidence that there were enough exchanges of ideas in
advancing the state of knowledge of V&V in spite of a lack of time for
more in-depth discussion.
Sample
Page of Transcript for Speaker’s Approval
Workshop
Proc. 5.02-G2 (Abramson)
- Draft 1
Transcribed & edited by Fong,
11-16-04
Dr.
H. Norman Abramson, Exec Vice President (retired), Southwest Research
Institute, San Antonio, TX:
I am interested in
validation. My background is more
experimental than mathematical. Validation
to me is a kind of a guarantee that something that you had is going to do what
is supposed to do. There are two
ways to validate something. One
way is to compare with something that is exact, and all you mathematicians
know how to do that. The other is
to compare with reality, and to me reality means nature or experiment.
In order to do that, we need to have some kind of benchmarks.
I consider a benchmark as something of value that is
generally accepted as one that serves as a basis of comparison.
We have 25 presented at this workshop and they are all terrifically
interesting, but taken together they do not seem to provide a unifying theme
for the kind of V&V we are talking about.
So we need to select good benchmarks and we need criteria to select.
I have no idea what the right criteria are, but I have jotted down a
few by which you might try to select for certain kind of analysis:
(1)
Reproducibility.
We need a benchmark or a set of benchmarks that refer to a level of
reproducibility. That is
you can reproduce it time after time in your own lab, and then if it’s good
science, you should be able to reproduce in other labs essentially the same
result as in yours recognizing there are some uncertainty and variability.
Then at least you have something you can call it a real benchmark.
(2)
Complexity. Another set of
benchmarks relates to a level of complexity.
For example we have a benchmark on the bending of a cantilever beam.
As Prof. Brinson said in his discussion of that benchmark, even such a
simple example has some problems if you want to do it correctly.
Compare that with a complex aircraft structure undergoing a dynamic
response. That is another level of
complexity you may want to have a certain benchmark that relates to that level
of complexity.
(3)
Inherent Variability.
Another set to deal with needs to be at a level of inherent
variability. If you
want to test a cantilever beam made of aluminum, that is one thing. But if you
want to have a test involving a soil sample, I cannot do the same test twice
because there is no second soil sample that is exactly like the first one.
Then there is another set of problem that I won’t
call a benchmark criterion but it appears in some of those 25 presented at
this workshop. It is a lack of
knowledge on fundamental behavior or characteristics which are subjects of
active research aimed at understanding the phenomena that are observed.
Examples of such phenomena are non-linearity, instability, etc. which
are not yet understood, and we don’t want those as benchmarks.
We may still want to have models to predict those phenomena, but they
are more in the realms of research.
What this workshop showed me was that we need to put a
great deal of emphasis on how to select benchmarks to insure that these very
sophisticated analysis you folks have been developing in so many years can be
used to verify & validate.
Another
Sample Page of Transcript for Speaker’s Verification & Approval
Workshop
Proc. 1.01 (Phoha) - Draft 1
Transcribed & edited by Fong,
11-16-04
Dr. Shashi Phoha, Director, NIST Information Technology
Laboratory:
It is indeed my great
pleasure to welcome you all to this workshop on measurable V&V for
high-consequence engineering systems.
I think we have all seen the tremendous advances in
computation that have made human beings to think of very large complex
systems. Large is not necessarily
in physical scale, but it could be in micro- or nano-scales, but the
complexity nevertheless is very high. So
numerical simulation is one way of measuring the reconsideration or redesign
of these complex systems.
The difficulty with the physical systems is that they
are complex and they don’t have very good numerical interfaces that people
could interpolate in scientific analysis.
The numerical simulation world gives ample opportunity to do that in
order to assess the design space and the performance space for these very
complicated systems. The problem
with the numerical simulation space is that they live in a digital abstraction
of the cyber space, and they tend to lose touch with the physical world.
I see MV&V or measurable V&V as an attempt to confluence the
physical space with the numerical space, the measurement space, and the
computational space, and to bring the cyber and physical spaces together.
What we have done in the past in dealing with a V&V
problem as soon as the computational development of a system became possible
and took over a real system was like this.
The general response of the scientific world has been defining
requirements, specifying some logical specifications and then getting into the
issue of theorem proving for the correctness of the system as required.
That’s a very difficult task.
Going through this book, I think that the MV&V is
really an alternative way where you confluence the two spaces, the cyber space
and the physical space, because through these tools you have developed,
namely, the strong-sense benchmarks (SSB), the experimental design, and the
modeling of capturing the uncertainties, you have three pragmatic tools to
assess the numerical models to see if they have truly captured the causal
architecture of the physical space. It
is not in writing requirements and specifications, because if we can do them
perfectly, we wouldn’t be writing incorrect programs.
It’s really how you capture the causal architecture that generates
data in the physical space and how you capture them adequately in the
numerical space.
So I think MV&V has a lot of
promise to really bring the abstract cyber space to the physical world
and really put some science behind the programs we write and the computations
we develop which are supposed to emulate the real world.
At least we can see how far away we are.
I think the charge here is very difficult, the problem is very
fundamental, and the approach is very new, but it has the root to grow into a
very effective method for really changing the value of numerical simulation in
the scientific world. Looking at
all of you and looking at the draft book over the weekend, I have great faith
in this workshop. You will help us
bring about some fundamental changes in computational sciences and put them
into a scientific basis with other sciences so we could see how good numerical
simulations are in the physical world. Thank
you.