Cassandra EOFException thrown on some queries after a power failure

`INFO 11:44:29,874 Listening for thrift clients...
ERROR 11:47:01,471 Exception in thread Thread[ReadStage:33,5,main]
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept
ion: java.io.EOFException: EOF after 2 bytes out of 29545
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE
xception: EOF after 2 bytes out of 29545
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:89)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea
der(SSTableSliceIterator.java:68)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS
TableSliceIterator.java:44)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera
tor(SliceQueryFilter.java:101)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q
ueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationC
ontroller.java:275)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat
ionController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa
milyStore.java:1363)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1220)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1132)
at org.apache.cassandra.db.Table.getRow(Table.java:355)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm
and.java:70)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr
ow(StorageProxy.java:1052)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1578)
... 3 more
Caused by: java.io.EOFException: EOF after 2 bytes out of 29545
at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:
350)
at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU
til.java:382)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:71)
... 16 more
ERROR 11:47:12,900 Exception in thread Thread[ReadStage:35,5,main]
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept
ion: java.io.EOFException: EOF after 2 bytes out of 29545
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE
xception: EOF after 2 bytes out of 29545
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:89)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea
der(SSTableSliceIterator.java:68)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS
TableSliceIterator.java:44)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera
tor(SliceQueryFilter.java:101)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q
ueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationC
ontroller.java:275)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat
ionController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa
milyStore.java:1363)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1220)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1132)
at org.apache.cassandra.db.Table.getRow(Table.java:355)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm
and.java:70)
at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr
ow(StorageProxy.java:1052)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1578)
... 3 more
Caused by: java.io.EOFException: EOF after 2 bytes out of 29545
at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:
350)
at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU
til.java:382)
at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:71)
... 16 more`

And in Hector i get a `HTimedOutException` which I am assuming is because of the SSTable corruption in cassandra because it was working well before the power failure. How do I fix this? Also this is development database and I have had this problem before and ended up dropping some Column Families but I don't want this to be the only solution every time power cuts or system crashes. I am using cassandra 1.2.2

Instead of deleting the whole CF, you can just delete the corrupt SSTable then run repair to recover the data from the replicas. The problem is the error doesn't print the filename in the log so it's hard to find out which file is corrupt. You can get it though by running Cassandra in debug mode (edit log4j-server.properties). See this other SO post:

[To see links please register here]

for how they did it.

Perhaps a first thing to try is to `nodetool scrub` this cf (see [this post][1] for v.1.2 for instance).


[1]:

[To see links please register here]