@Ken_GlassWire, this topic is timely - thanks @euclidean - as I was about to post on this very topic because of the likelihood of database problems occurring in GlassWire.
I recently had reason to use some results of research into real-world DRAM errors. This got me thinking about GlassWire and the likelihood of DRAM errors causing database corruption.
The basic point of the research is that DRAM errors are much more prevalent than we expect.
We already know that the GlassWire database can be corrupted by disk and CPU errors. The relevance of this research for GlassWire is that GlassWire’s logging of history is, compared with other software, exceptionally exposed to memory errors that will look just like the problem reported in this topic: an unexplainable fault that requires the database to be deleted.
The solution is that GlassWire needs to have the ability to correct database errors and eliminate the undesirable requirement to delete the database to resolve corruption problems. I consider this to be the most important feature that GlassWire needs right now. Personally, I don’t care about my GlassWire data history, but so many users do want to retain their history and this feature is promoted as one the main benefits of the GlassWire graph view.
I looked at two papers:
A study published in 2009 looked at DRAM errors in Google datacentres using predominantly ECC DRAM. They found that in any one year about one-third of computers and more than 8% of DRAM had errors.
PDF DRAM Errors in the Wild : A Large-Scale Field Study
A 2011 study of hardware failures in one million consumer PCs based on Windows error reports (WER) uploaded to Microsoft. The DRAM is mostly non-ECC so the errors were from the small part of Kernel memory (averaging about 1.5% of total memory available to Windows) where Windows error correction can detect changed data…These errors are only from computers that could still operate because they had to be able to upload the error report to Microsoft.
Cycles, Cells and Platters: An Empirical Analysis of Hardware Failures on a million consumer PCs
The research is particularly weighty because of the scale of both studies. The researchers had data from so many more computers than earlier studies. The datacentre study used hundreds of parallel processors just to preprocess the many terabytes of data:
Each one of many ten-thousands of machines in the [Google] ﬂeet logs every ten minutes hundreds of parameters
The implications for GlassWire
I consider that GlassWire is particularly susceptible to errors in the logged history:
- GlassWire doesn’t usually run on hardware and operating systems where ECC can detect and correct such errors.
- GlassWire keeps a database of logged history which becomes more susceptible to memory errors. In other words, the probability of database corruption keeps increasing with time:
- The file size increases over time which, as an aside, also increases the likelihood of a disk error corrupting it.
- Continuous processing increases the likelihood that an error will not be avoided.
- The scope of the impact increases with time. In other words, the more days data I have then the more days that will be affected by an error. And the more data I have collected then the more I will care about it if that history is lost.
If we calculated a rough likelihood of such errors occurring then I expect that we’d all be surprised how significant it would be for GlassWire users.