tag:blogger.com,1999:blog-1208196493518969640.comments2015-08-10T11:33:19.185-07:00Reproducible Scientific ComputingDouglas Thainhttp://www.blogger.com/profile/10046446527813216338noreply@blogger.comBlogger21125tag:blogger.com,1999:blog-1208196493518969640.post-55819776034742933542015-08-10T11:33:19.185-07:002015-08-10T11:33:19.185-07:00I had to look it up, but I wholly approve of the i...I had to look it up, but I wholly approve of the idea!<br /><br />In a similar way, the Go language is very particular about source code dependencies. As in most languages, you can't use a library unless you do "import X". But just as importantly, you aren't allowed to "import X" unless you actually use something from X! This cuts down on the problem of the every-growing set of include statements found in many C and C++ programs.<br /><br /><br /><br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-6102183515275148802015-08-10T10:54:32.159-07:002015-08-10T10:54:32.159-07:00I'm reminded of 'implicit none' from F...I'm reminded of 'implicit none' from FORTRAN - scary...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-20531026959106537102015-08-10T09:10:31.266-07:002015-08-10T09:10:31.266-07:00Well, first a conference steering committee needs ...Well, first a conference steering committee needs to decide to mandate artifact evaluation. I am optimistic that this will happen sooner or later in a venue where common artifacts are valued but unusual hardware is not required. For example, this would be much easier to accomplish in ICSE or KDD where the concern is the algorithms and the output, as opposed to HPDC or SC, where the concern is usually the performance on unusual hardware.<br /><br />If you really want to make it stick, the PC could require artifact evaluation to happen *before* technical evaluation. e.g. require brief instructions on how to start a VM, checkout the code, download the data sets, and run an example. If that can't be done, then return without review...<br /><br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-33728768256404398832015-08-10T09:03:58.281-07:002015-08-10T09:03:58.281-07:00Gordon, I think you hit upon something really impo...Gordon, I think you hit upon something really important: the preservation tools need to be part of the work from the beginning. If you only start to think about preservation at the end, once the publication is made, then everyone has moved on and there will be little interest (time, funding) to do preservation tasks.<br /><br />It follows that the preservation tools have to be used every single time, and they better not put up roadblocks to the work that you actually want to do.<br /><br />I like the approach of interposing on the grid submission system, which is an example of "preserving the mess" that I posted today. And, it also has the advantage that, in order to run remotely, you need to already specify all the dependencies necessary to create the environment.<br /><br />If every important task gets run in a batch system instead of locally (does it?) then that could be the right solution for HEP...<br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-19001720548003857792015-08-08T04:06:34.132-07:002015-08-08T04:06:34.132-07:00Sorry, I didn't read the linked post earlier. ...Sorry, I didn't read the linked post earlier. I do understand the reasoning for your choice, and shouldn't have said "dumping", but I still see something wrong with the idea of having 2 groups (classes) of reviewers, one that reviews the ideas/papers and one that reviews the artifacts.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-8058812390827031472015-08-08T04:04:46.614-07:002015-08-08T04:04:46.614-07:00This comment has been removed by the author.Daniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-43714524655014499622015-08-07T18:58:18.545-07:002015-08-07T18:58:18.545-07:00Shriram Krishnamurthi has sent in the following co...Shriram Krishnamurthi has sent in the following comment:<br /><br />Thanks for the very nice article. You've certainly gotten to the<br />essence of the design.<br /><br />Responding to @Dan: I deeply dislike this characterization of<br />“dumping” work onto students and new postdocs. (By the way, who said<br />students and postdocs aren't capable of at least as much creativity as<br />superannuated professors?) The justification for using younger people<br />was clearly articulated in the first essay I wrote about artifact<br />evaluation:<br /><br />http://cs.brown.edu/~sk/Memos/Conference-Artifact-Evaluation/<br /><br />Look for “AEC Composition”. It lays out several points in detail.<br /><br />But yes, the upcoming Dagstuhl meeting (which I'm co-organizing) will<br />indeed be glad to talk about this if there's interest from the<br />attendees. You're welcome to bring it up.Anonymoushttps://www.blogger.com/profile/17687217255267977457noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-41923564548081486102015-07-25T04:56:48.708-07:002015-07-25T04:56:48.708-07:00I've been pleased to see artifact evaluation c...I've been pleased to see artifact evaluation catch on in this area, but the fact that we dump the work on students and new postdocs seems to say that it's just a box we need checked, not something we see as an area for creativity and innovation. I hope the upcoming Dagstuhl meeting (http://www.dagstuhl.de/en/program/calendar/semhp/?semnr=15452) will think about this.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-28064131886488877072015-07-16T00:55:21.055-07:002015-07-16T00:55:21.055-07:00Replicating someone else's computational work ...Replicating someone else's computational work is indeed of no interest. Reproducing, i.e. reimplementing the ideas from scratch, is mainly of interest for gaining better understanding, and therefore as much part of education and training as of doing science.<br /><br />The big advantage that computational scientists have over experimentalists is that replication of computations is a purely technical issue (see https://khinsen.wordpress.com/2014/08/27/reproducibility-replicability-and-the-two-layers-of-computational-science/ and http://dx.doi.org/10.12688/f1000research.5773.3 for an explanation). In the long run, it can and should be delegated to technology. For reasonably short computations, a replication server could accept submissions (complete records of computational work), replicate them, and issue some certificate. Once certain tools for ensuring replicability are trusted by the community, explicit replication could be reduced or even given up. A trusted tool chain would also solve the problem of huge computations: they would be considered replicable because of the use of trusted tools.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-90428265669717798332015-06-29T11:17:18.086-07:002015-06-29T11:17:18.086-07:00Kristin, my guess is that most academics would agr...Kristin, my guess is that most academics would agree at least in principle with this statement of values. But, it seems that our review and publication practices don't reward this level of investigation. Reviewers largely assume that a publication is submitted without deception, and largely focus on the novelty and correctness of the written component. Do we need some additional incentives to look at correctness and buildability of the technology described by the paper?<br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-11793910021844144422015-06-29T11:00:16.079-07:002015-06-29T11:00:16.079-07:00Eric, I share your concern about the focus of our ...Eric, I share your concern about the focus of our profession. Computer science as an academic discipline spends a lot of effort producing technology, training students, and burning coal that ultimately supports nothing more than the optimization of advertising within social networks. Our educational systems could do a better job of demonstrating more important applications of computing.<br /><br />But there is the converse problem: we have seen many times that specialized areas of computing cannot take advantage of economies of scale, and are better off riding the commodity wave. Specialized supercomputers have largely been supplanted by commodity clusters. GPU technology driven by video games has far better energy/performance density than any bespoke parallel hardware. Soldiers in the gulf war preferred using personal smartphones to military-issue GPS units. More examples abound.<br /><br />So, can we advance science, sustainability, governance, and public health using commodity technologies? Or do they need something fundamentally different?<br /><br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-18950163486377830362015-05-31T14:35:48.439-07:002015-05-31T14:35:48.439-07:00Dan,
Thanks for the pointer to ORCID.
I see that...Dan,<br /><br />Thanks for the pointer to ORCID.<br /><br />I see that NSF's Fastlane includes the option to list one's ORCID identity.<br /><br />Do you know if this has existed for a while (and I have just noticed it) or whether this is a new feature?<br /><br />If it has been around for an extended period, has there been any analysis of its uptake / impact?Ashish Gehanihttps://www.blogger.com/profile/01703256916114179943noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-5687972912148583582015-05-26T11:56:25.278-07:002015-05-26T11:56:25.278-07:00You can think of ORCID just as a unique person ID....You can think of ORCID just as a unique person ID. Or, you can use it to store a profile. You don't have to do the latter to take advantage of the former.Daniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-60936399071816524222015-05-26T07:14:32.241-07:002015-05-26T07:14:32.241-07:00ORCID is a good idea, and they have a nice infrast...ORCID is a good idea, and they have a nice infrastructure, but I was surprised that after creating a profile (http://orcid.org/0000-0001-5218-1956) it became *my* job to upload papers to their system and associate them to my ID. (This is now the fourth or fifth website where I'm expected to maintain my publications!) The ecosystem will work better when publishers collect ORCIDs at submission time and take on the role of associating ORCIDs to DOIs.<br /><br /><br /><br /><br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-69498728226118425922015-05-26T07:04:44.425-07:002015-05-26T07:04:44.425-07:00I wonder why you decided to start with reconstruct...I wonder why you decided to start with reconstruction. One might equally well choose to start with a new run of the same detector, or even with a new detector. The question of where reproducibility starts seems like something that should always be asked.Daniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-39114064948936335682015-05-26T06:02:55.843-07:002015-05-26T06:02:55.843-07:00There are certainly cases in CS where the datasets...There are certainly cases in CS where the datasets are small, and so dumping everything into one VM is a perfectly good solution. And you are quite right about incentives.<br /><br />But if you look at the real sciences, the data situation is different. In bioinformatics, you frequently have thousands of analyses performed on a common dataset of tens to hundreds of gigabytes. You can squeeze 100GB into a VM, but you wouldn't want to duplicate it once for each analysis code, so you need a way to archive the data independently of the code, be able to refer to it, mount it at runtime, and verify that you have the right thing. Same thing in astronomy at larger scales (100s of TB) and high energy physics (100s of PBs) and many other fields as well.<br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-6604581509676492352015-05-25T14:47:43.573-07:002015-05-25T14:47:43.573-07:00I've had some experience with this in a variet...I've had some experience with this in a variety of contexts.<br />I don't think the problem is tooling, because apt and conda cover a wide variety of situations for deploying software.<br />Sometimes data is a problem, but for most computer science papers I read the data sets aren't that large.<br />For many of the papers (and some textbooks) I've read, reproducibility could be solved with a virtual machine snapshot. The fact that this isn't done very often is telling. The real problem is the lack of incentive to make work reproducible.<br /> Robert Zehhttps://www.blogger.com/profile/12647206624571931529noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-60482080942106181162015-05-25T05:08:04.820-07:002015-05-25T05:08:04.820-07:00You should look at ORCID - http://orcid.orgYou should look at ORCID - http://orcid.orgDaniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-7124582162713105662015-05-25T05:07:24.423-07:002015-05-25T05:07:24.423-07:00This comment has been removed by the author.Daniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-27581473565192022502015-05-19T07:22:22.723-07:002015-05-19T07:22:22.723-07:00One way that reproducibility is sometimes pushed i...One way that reproducibility is sometimes pushed is that I need to make sure my work is reproducible by the future me. This seems appealing to me.Daniel S. Katzhttps://www.blogger.com/profile/07436481749560349407noreply@blogger.comtag:blogger.com,1999:blog-1208196493518969640.post-45278594700621797152015-05-19T06:34:51.358-07:002015-05-19T06:34:51.358-07:00In talking with colleagues in a variety of scienti...In talking with colleagues in a variety of scientific fields using computation, I find that there is almost no interest in reproducing another PI's published work simply for the purpose of validating or disproving it, in a competitive sense. If validated, nothing new is learned. If not validated, then one merely enters into a complex argument about whether the reproduction was correct.<br /><br />(The value judgement is probably different in non-computational science: multiple trials will improve the confidence interval on the question or whether a particular antibiotic works or not, and so reproducibility has an easily quantified value.)<br /><br />But, there is considerable interest in reproducibility for the sake of efficient collaboration! That is, if I have a student that sets up some software and system to run a particular workload efficiently, then I want to be able to pass that setup off to other people and hope they have a good chance of getting it running in the next day, rather than the next month. If my algorithm is easy to reproduce, then someone is likely to use it as a comparison point for new work, resulting in more interest and citations. I see a lot of people who would like that.<br /><br />If the cost of reproducibility between collaborators becomes low, perhaps interest in reproducibility between competitors would become of more interest?<br />Douglas Thainhttps://www.blogger.com/profile/10046446527813216338noreply@blogger.com