I found a serious, embarrassing error in a highly cited Nature Computational Science paper on quantum machine learning:
>The power of quantum neural networks (QNNs) https://www.nature.com/articles/s43588-021-00084-1
Things go like this: In Fig. 3b, the authors claim a training advantage of QNNs over classical neural networks (NNs) on the Iris dataset. I checked the GitHub repo and noticed that I am apparently not the first person to find the classical baseline suspicious. Someone already opened an issue pointing out that the authors used a strange classical NN architecture:
Screenshot of this open issue on 2022
For an 8-parameter classical NN, the authors use 4 layers with neurons 4->1->1->1->2. This means the 4-dimensional Iris input is immediately compressed into one scalar. That is an extremely poor classical baseline.
Actually, the simplest classical NN baseline one can think of — a single linear layer from 4 inputs to 2 outputs — already has 8 parameters, as pointed out in the pull request.
The ridiculous definition of 4->1->1->1->2 by the Nature paper
So I tried the same experiment using the original GitHub code: https://github.com/amyami187/effective_dimension/blob/master/Loss_plots/generate_data/classical_loss.py, but change the definition of classical NN to 4->2.
After this change, the classical NN converges much faster and reaches much lower loss than the quantum NN. So the training advantage shown in original Fig. 3b collapses completely once the classical baseline is changed to the obvious 8-weight linear layer.
This is not a subtle quantum ML issue. This is basic ML benchmarking. The claimed “advantage” appears to come from comparing the QNN against an extremely weak classical NN, a ridiculous baseline that would be unacceptable even in an undergraduate ML final project.
Since this is the ONLY experiment in this paper to support the claim, I believe this is a serious issue and retraction should be discussed.
The codebase is public, so everyone can try it: https://github.com/amyami187/effective_dimension
I can now truly feel “the power” of quantum neural networks!