Slashing event in era 660

A slashing event occurred in era 660 (29.01.2022):

Validator affected - CHORUS ONE
Nominators affected: 200
Total CFG: 13,475.8139 CFG
The validator was informed about slashing. Hope he could explain the slashing reason.

Updated:
To confirm the incident timeline:

  • block height 9461658 timestamp (2022-01-28 16:20:36 (+UTC)) matches our logs where the event was triggered on “2022-01-28 16:19:45.334”.
  • the latest height presented in logs (after reboot) points to #9460394 block height (2022-01-28 14:11:54), so that confirms ~2h downtime

We have contacted the centrifuge devs to help us identify the root cause of the slashing event. We are working on this.

3 Likes

thank you for reply!

1 Like

Hi all,

we have been trying to figure out the root cause of the slashing event and while we found the signal that triggered the slashing, the root cause is still unknown.

TL;DR

The server running the ChorusOne validator node became unresponsive and we had to power-cycle it. After reboot, our node came back online and reported Prevote Equivocation.

Timeline

  • 2022-01-28 14:11:54 - our node went down (we know this from log pointing to last known height #9460394)
  • 2022-01-28 16:19:45 - our node came back online (we also know this from the logs)

In short, the validator was down between #9460394 - #9461658 block heights, for approximately 2 hours.

Investigation (so far)

After reviewing our logs, this is the only unusual entry we found:

{"log":"2022-01-28 16:19:44.959 main INFO babe  👶 Starting BABE Authorship worker\n","stream":"stderr","time":"2022-01-28T16:19:44.959191737Z"}
{"log":"2022-01-28 16:19:45.334 tokio-runtime-worker WARN afg  Detected prevote equivocation in the finality worker: Equivocation { round_number: 36853, identity: Public(72bd245ae66dc28b153ec00437e83dda704323152e18278ffb25ab95f0d48ea4 (5Ef9W87R...)), first: (Prevote { target_hash: 0x0a2c9f73700e697d312d36cb48ecefecbd5f03884f117ce5d89ed9d6cd506962, target_number: 9460393 }, Signature(6e5d0abe49e626cb8e44d7879b0b0ee493dc19e284fdd10cb635dd2e120a2f1245fb29ec50ef01bbd1a682099c8ede2c6eefd197485dd30c30f0b4c50d18d50a)), second: (Prevote { target_hash: 0xc7656ad3ac6328ed6a12ac54aff828b79765d219d029be743ac05badce8276c8, target_number: 9460392 }, Signature(96615f4a37398e22836c0dc044d40e5cb6506f978deb4f5dd8b623691575a3c7f089628120a978d5f146adcc49adcc833816ec63de7a44d087dc9296eecb760f)) }\n","stream":"stderr","time":"2022-01-28T16:19:45.334347276Z"}

Docker Image: centrifugeio/centrifuge-chain:20201022093419-da56ac5 (v2.0.0-rc6.0).

Following the trail of the log we found the relevant code paths:

Now with the substrate JS API we found the block containing the grandpa.reportEquivocationUnsigned extrinsic (#9461658): Polkadot/Substrate Portal

equivocation_proof: GrandpaEquivocationProof
{
  setId: 698,
  equivocation: {
    Prevote: {
      roundNumber: 36,853,
      identity: 4dwBxAs1RqLPVwiCvQsoDYEQbuCzwiCBLHdqVag4dG66s6Dh,
      first: [
        {
          targetHash: 0x0a2c9f73700e697d312d36cb48ecefecbd5f03884f117ce5d89ed9d6cd506962,
          targetNumber: 9,460,393
        },
        0x6e5d0abe49e626cb8e44d7879b0b0ee493dc19e284fdd10cb635dd2e120a2f1245fb29ec50ef01bbd1a682099c8ede2c6eefd197485dd30c30f0b4c50d18d50a
      ],
      second: [
        {
          targetHash: 0xc7656ad3ac6328ed6a12ac54aff828b79765d219d029be743ac05badce8276c8,
          targetNumber: 9,460,392
        },
        0x96615f4a37398e22836c0dc044d40e5cb6506f978deb4f5dd8b623691575a3c7f089628120a978d5f146adcc49adcc833816ec63de7a44d087dc9296eecb760f
      ]
    }
  }
}

What is the root cause?

In our setup nothing has changed, the node has been running successfully for a long time, therefore it’s possible that we hit a bug. Interestingly, almost at the same time our second node went down and came back just fine.

Taken the above information, I think it would be fair to propose a governance vote to revert the slashing (current state: “unapplied slash”).

Of course, if anyone has a better idea of what could cause the equivocation, please let us know!

4 Likes

Thank you for your deep feedback.

If voting will be launched I will definitely vote for the revert slashing event.

2 Likes

We’ve just created a proposal to cancel the slash:

thank you all for all the help!

2 Likes

Thank you for clarifying what happened - I will also vote for reverting the slashing event.

2 Likes

Thanks @mkaczanowski for reporting it openly and diving deeper on it. Since our version of substrate is fairly old, I think you indeed faced a substrate bug.
It is possible that is related to this: GRANDPA Equivocation and sysinfo Process Collection Results In Slashing on Kusama Network: a Post-Mortem.

I would vote as well on reverting the slashing event.

3 Likes

Hello, I voted no as I did not see any slash to take effect. The referendum failed because I voted no because Supermajority was necessary.

Thanks @TurkValidator for proactively participating on the on chain governance. I would like to clarify that the slashing event was canceled through the Council a few weeks back. So technically that referendum open was a no-op. More information here
All the best,

4 Likes