Writing and responding to reviews is the bread and butter of any academic and especially in AI research, PhD students are confronted with both rather early compared to other displicines. Unfortunately, I found that drafting reviews and rebuttals is a skill that is rarely properly taught as part of PhD programs. Thus, in this article, I want to share some of the lessons I learned throughout the past 7 years. The post Some Lessons on Reviews and Rebuttals appeared first on David Stutz ..read more

Watermarking AI-generated content has the potential to address various problems that generative AI threatens to aggravate — misinformation, impersonation, copyright infringement, web pollution, etc. However, it is also controversial with many researchers and users worrying about reduced quality and questioning whether watermarking actually works and helps. In this article, I want to share some of my thoughts on how well watermarking works and whether it can actually help with our problems. The post Thoughts on Watermarking AI-Generated Content appeared first on David Stutz ..read more

With the goal of deploying generative AI systems, rater studies are becoming increasingly common and important. This means more and more researchers and engineers face the challenge of actually planning and conducting rater studies for AI systems. In this article, I want to share some lessons I learned over the past year. The post Thoughts and Lessons for Planning Rater Studies in AI appeared first on David Stutz ..read more

Dealing with rater disagreement is becoming more important in AI, especially for LLMs and in specialized domains such as health. In the past year, I helped open source two datasets allowing to study rater disagreement in the health domain: a relabeling of MedQA, a key benchmark for evaluating medical LLMs, and a dataset including differential diagnosis ratings for skin condition classification. Both are available on GitHub. The post Open-Sourcing Relabeled MedQA and Dermatology DDx Datasets appeared first on David Stutz ..read more

In this article, I want to share some thoughts on the difference between research ideas and technology, particularly in machine learning. This distinction is have been contemplating since starting my PhD. After joining Google DeepMind and being involved in product releases such as SynthID, I realized that it can be useful to distinguish between research ideas and technology in many projects, both in industry and academia. The post Thinking About Research Ideas vs. Technology appeared first on David Stutz ..read more

When working in AI, especially when doing an empirical PhD, the engineering required for effective experimentation is incredibly important. I procrastinated on writing this article for a while now, but it is getting more and more apparent these days when a large portion of research shifted to working with large foundation models. But even ~7 years ago, when I started my PhD, I quickly realized that running experiments effectively will be crucial. This is because effective experimentation means that research hypotheses can be tested quickly and provide insights into the next hypotheses to test ..read more

Are code and data available? Yes, code and data are on GitHub. Code includes both Monte Carlo conformal prediction as well as the plausibility regions from v1 of the paper. Can you derive the conformal $p$-values used in the paper? The connection of conformal prediction and $p$-values is scattered across the literature and there is, to the best of my knowledge, no good reference to understand this. So we added a thorough derivation in Appendix B of the paper. How do you get the plausibilities $\lambda$ in practice from different formats of annotations? In a nutshell, this is a modeling choice ..read more

Introduction In the beginning of my PhD, I had a series of conversations with PhD students and academics about how to keep track of everything during a PhD. Some of these discussions were motivated by the 2018 Workshop of being a Good Citizen of CVPR. This inspired me to start thinking properly about how to organize and eventually document my PhD work. In retrospective, I think this was one of my better decisions. Only recently, at the Heidelberg Laureate Forum 2023, I found that many successful academics also document, for example, who they talk to during conferences to stay on top of things ..read more

A short disclaimer is necessary before diving in: the below is a rather personal opinion on the subject — driven by my personal experiences in AI research. As such, it is not meant to blame, contradict or discredit anyone or anything. Instead it is an attempt to add color. I think the project track in question is rather specific; I am sure much thought has gone into it and NeurIPS will iterate on it in future instances of the conferences. This being said, I think many arguments raised on X are not necessarily about NeurIPS' decision to have such a track in specific. Instead, many arguments can ..read more

Introduction By construction, a PhD has a clear end. Depending on the program, country and field, a PhD is supposed to be done within 3-6 years when it is usually awarded after an official defense of the research work. This is in contrast to most other careers and jobs, especially in industry but also in the public sector. Even though a PhD is often considered as a qualification for independent research and thereby acts as the entry to an academic career, it is commonly assumed that most PhD graduates do not continue in academia. This also matches my impression and surveys among PhD students i ..read more