CS6310: Design Studies

Designers often build and compare physical models in order to better understand a problem. Examples are building architects and automobile designers. For the purposes of this course, a design study is a systematic comparison of multiple versions of a program with the intent of better understanding it.

You can think of a design study as an empirical scientific experiment. That is, there are research questions, subjects of study, experimental conditions, methods, tools, metrics, independent and dependent variables, data collection, statistical analyses, and conclusions. An overall goal is repeatability. That is, someone else should be able to take your study report, use it to recreate the study and reach the same conclusions. It is one of the goals of this class that you learn the skills to produce and present an industrial-quality design study.

Here are the topics you should consider in planning a design study. They should appear as separate labeled sections in the resulting report.

• Context: What is your study examining? For CS6310, this means determining the role of various non-functional constraints on the quality of a simulation program. Give enough background so that a reader unfamiliar with this class and its projects can understand what you are talking about. Also, in this section, define any specialized vocabulary that you will be using in the report.
• Research questions: You should formulate specific research questions that you will study. For example, What is the effect of increased precision on execution time? There should be questions discussing each tradeoff among the problem constraints
• Subjects: A subject is something that you are studying, for example, a program. A design study compares multiple subjects. Each subject should be briefly described, differentiating it from the other subjects
• Experimental conditions: What is the environment in which the study will take place. This includes elements such as machines, operating systems, programming languages, virtual machines, network (where relevant), build and execution parameters, input files, and confounding factors (other users or processes). Remember that the goal is repeatability
• Independent variables: Explicitly describe factors that you can vary, such as precision. Give a distinct name to each variable
• Dependent variables: Explicit statement of what factors you expect to be affected by variations in the independent variables, such as performance. Give a distinct name to each variable
• Metrics: For each variable (independent and dependent), state how will you measure it. This should include the unit of measurement, such as seconds, and a precise definition, such as a line of code is a maximal sequence of non-line-break characters that includes some non-whitespace content occurring in a Java source file". For some factors, such as time, this is straightforward. For others, such as maintainability, you must think carefully about the factor and invent an appropriate way to measure it
• Method: A description of how you will conduct the experiment. This includes number of trials, measurement devices and tools, randomization technique (where appropriate), significant digits, etc. This should also include an explicit statement of which subjects will be run with the arguments used. For example, if you were studying the relation of performance to grid size, you would want to specify what different grid sizes you will be using
• Analysis techniques: How will you analyze the data? What statistics will you use? For example, means vs. medians? Are you performing regression analysis or computing correlations? What will you do about outliers? How will you judge significance? How will you report error bounds? Are you doing any interpolation or curve fitting?
• Presentation: How do you plan to present the results? Graphs? Tables?
• Results: The data collected (empirical results) and the results of analysis (analytical results)
• Discussion: Reflection on the experimentation itself, including explanations for anomalous phenomena and suggestions for improving the study process. In particular, any anomalous results should be thoroughly explored and explained
• Conclusions: Specific answers to the research questions

A design study is presented in a report. The report is a carefully formatted presentation of the items listed above. It may include charts, tables, graphs and screen shots as well as descriptive text. It is not a narrative, but a dispassionate description of a systematic exploration. The report should be professional in quality. This means that you should treat it like you would if you were preparing to show it to customers or submit it for publications. Its spelling and grammar should be checked, and it should be carefully proofread by a team member other than the author.

Here is an MS Word file providing a starting point for your report.