Peer Review In the Age Of Viral Video

Recently, a YouTube video has been making the rounds online which shows a rather astounding comparison between two printed models of the US Capitol. Starting with the line “3-D PRINTERS CAN NOW PRINT TWICE AS FAST”, the video shows that one print took four hours to complete, and the other finished in just two hours by virtue of vibration reducing algorithms developed at the University of Michigan. The excitement around this video is understandable; one of the biggest limitations of current 3D printer technology is how long it takes to produce a model of acceptable quality, and if improvements to the software that drives these machines could cut total print time in half, the ramifications would be immense.

In only a few weeks the video racked up tens of thousands of views, and glowing articles popped up with headlines such as: “How to cut 3D print times in half by the University of Michigan” and “University of Michigan professor doubles 3D printing speeds using vibration-mitigating algorithm“. Predictably, our tips line lit up with 3D printer owners who wanted to hear more about the incredible research that promised to double their print speed with nothing more than a firmware update.

The only problem is, the video shows nothing of the sort. What’s more, when pushed for details, the creators of the video are now claiming the same thing.

A Sub-Standard Standard

The first indications that the video might not be quite what it seems can be found in the YouTube comments, where several viewers mentioned that the state of their “Standard Printer” was questionable at best. Intense vibrations are clearly visible in the printer’s belts, and issues with bed adhesion on the final printed model hint at either a poorly leveled bed or incorrectly calibrated Z axis. For those who have spent a bit more time than they may care to admit tuning and calibrating 3D printers, these issues are red flags that the machine is in a poor state.

It comes as no surprise when the video then goes on to show how prints on this machine fail as the acceleration settings are gradually increased, eventually ending up in a comically lopsided building. The severe layer shifting shown is again an obvious indicator that the printer’s belts were not correctly tensioned at the beginning of the test.

To anyone who has printing experience, it’s clear that the “Standard Print” demonstrated in the video is a poor metric by which to measure the performance of desktop 3D printing.

Replicating The Experiment

If the condition of the printer used in the “Standard Print” is in question, the next logical step is to attempt to replicate the experiment shown in the video and find how long it takes on a properly tuned 3D printer. Unfortunately, no information was given as to the model used, the scale it was printed at, the layer height, or infill percentage. If the condition of the printer itself was the first red flag in the video, the omission of this critical information is certainly the second. Had the video simply listed this information in the description, anyone with a 3D printer at home could have independently verified the claims made.

Luckily the model itself was not very difficult to identify. A quick search on Thingiverse found “The Capitol – Legislative“, part of MakerBot’s “Structures of Government” collection. But the size of the model was clearly much larger than what was shown in the video; the next step would be finding the scale factor used during printing.

Learning that the printer used in the video was a HICTOP brand clone of the Prusa i3, determining scale was simply a matter of comparing the apparent size of identifiable parts of the machine to the printing model. Knowing the heater block on the HICTOP printer is 10 mm thick, a particularly close view of the print from the side presented the perfect opportunity.

As the height from the top of the steps to the peak of the front portico appeared to be approximately 10 mm, the scale of the model could be adjusted in the slicer until the desired dimension was reached. In this way, it was determined that the scale of the model in the Michigan video was not more than 60%.

Infill was then estimated to be approximately 15 – 20% by comparing the slicer’s layer view at varying infill percentages with that of the in-progress print in the video.

20% infill in slicer, versus print in Michigan video

With the model, scale, and infill now known within a reasonable margin of error, the only question left was which layer height was used. Judging by the rather rough surface quality, .2 mm layers seemed likely. The model was then printed with these settings, and the process timed to see how it would compare to the two examples given in the Michigan video.

Scale: 60%, Infill: 20%, Layers: 0.2 mm, Print Time: 1 hour 20 minutes

The model was printed in 1 hour and 20 minutes, nearly half the time required for the “accelerated” print demonstrated in the video, and nearly 4 times faster than what was given for the “Standard Print”. Something clearly didn’t add up.

Looking For Answers

Chinedum Okwudire

There was certainly a margin of error in the estimations of scale and infill, but not enough to account for such a huge discrepancy between our experiment and what was shown in the original video. With our video evidence and real-world testing in hand, we reached out to Professor Chinedum Okwudire to shed some light on the situation.

Professor Okwudire agreed that transparency and peer-review were critical aspects of the scientific process, and true to his word he provided us with a highly detailed statement as to the nature of the original YouTube video, the team’s research, and how they believe it fits into the current state of the art in desktop 3D printing.

At the core of his argument Professor Okwudire asserts that despite the clear language used in the video and associated press coverage, the demonstration was only meant to portray the results his team achieved within the narrow scope of their research; and not to be taken as a claim that this technology would necessarily produce the same performance improvements under different circumstances:

“A few people seem to misconstrue our work (and the 4 hr case study presented in our video) as showing the fastest printing speed of the Capitol part by ‘standard’ desktop printers. This is a very wrong understanding. Print time highly depends on the printer and the parameters used for the print.”

Further, Professor Okwudire says that the “Standard Print” was not meant to be taken as a best case scenario. In fact, quite the opposite. In the context of this research, “Standard Print” is taken to mean a poorly maintained machine, printing at minimal speed, and operated by a novice user:

“The reason is because we found out that the average non-technical user of 3D printers tends to be conservative about print speeds and accelerations. We found out from many blogs that, in order to avoid failed parts or poor surface quality, people commonly use speeds around 40 mm/s and/or accelerations around 1 m/s^2 on highly-vibratory printers like HICTOP Prusa i3. So even though, with trial and error, parts may be printed at higher speeds and accelerations without failure, many non-technical users mention on blogs that they prefer to be conservative in order to ensure that they always get reliable prints.”

Repeating the Test

Scale: 50%, Infill: 25%, Layers: 0.1 mm, Print Time: 1 hour 55 minutes

In response to our request, Professor Okwudire provided us with the scale, infill, and layer height used in the video. Our estimates of scale and infill were both within 10%, certainly not enough of an error to create a significant difference in print time. However, the layer height used in the video was given as .1 mm compared to the .2 mm we tested at, which would have a significant impact on print time. In the name of due diligence, we re-ran the print with the precise settings provided.

Using the same parameters as the test video, the print completed within minutes of their “accelerated” print, further strengthening or initial impression of the video. The “Standard Print” is anything but, and the “Accelerated Print” is the performance you should expect from a properly configured and maintained printer.

A more accurate description of what the University of Michigan has demonstrated in their video is a vibration compensation algorithm able to overcome the shortcomings of flimsy or improperly configured printers. In fairness, this may well be a valuable avenue of research as manufacturers try to drive the cost of 3D printers ever lower, but it is surely not an algorithm that doubles the speed of 3D printing.

Sensationalism Vs Science

To the credit of Professor Okwudire and his team, Hackaday received in a very timely manner an exceptionally thorough document detailing the conditions of their experiments and scope of their research. From these documents it’s clear that the original video posted to YouTube is not false in the sense that it does indeed show that under very specific circumstances the time required to print the model is reduced by half.

That being said, the clearly misleading wording used in the video, the intentionally hindered software and hardware conditions used as a baseline, and the absence of independently verifiable parameters for their “Standard Print” control are impossible to ignore. It’s hard to believe that the creators of this video did not intentionally craft it in such a way as to sensationalize the results achieved in the exceptionally narrow scope covered by this experiment.

As a more widely relatable example, imagine a YouTube video that purported to show a method of doubling a car’s fuel economy. The video starts by showing a decades old clunker lurching down the road, backfiring and belching smoke with an indicated efficiency of 8 miles per gallon; followed by the same vehicle after a tune-up and repairs, running visibly better and now managing 16 miles per gallon. In both examples, the demonstration starts from an unreasonably poor position, introduces improvements, and yet still ends with a result that is no better than a more modern or better constructed specimen.

Professor Okwudire maintains that the algorithm being developed at the University of Michigan can potentially double the print speeds of a 3D printer, assuming the printer is functioning poorly to begin with. It has yet to be demonstrated that similar gains are possible on machines which are not currently suffering from mechanical issues, though there is certainly the potential for improvement. We, along with the 3D printing community as a whole, eagerly await an expanded and more transparent study of this technology and its potential use under more reasonable conditions.

If you find the concept of peer review fascinating, consider lending a hand as a Peer Reviewer of our own journal. Yesterday we announced the Hackaday Journal of What You Don’t Know and we’re looking for submissions, Associate Editors, and Peer Reviewers.

Source link

Leave a Reply

Your email address will not be published. Required fields are marked *