Slashing event in era 660

Engineering Collator Introduction CFG
1

chorusone
chorusone1275×293 36.1 KB

A slashing event occurred in era 660 (29.01.2022):

Validator affected - CHORUS ONE
Nominators affected: 200
Total CFG: 13,475.8139 CFG
The validator was informed about slashing. Hope he could explain the slashing reason.

Updated:
To confirm the incident timeline:

  • block height 9461658 timestamp (2022-01-28 16:20:36 (+UTC)) matches our logs where the event was triggered on “2022-01-28 16:19:45.334”.
  • the latest height presented in logs (after reboot) points to #9460394 block height (2022-01-28 14:11:54), so that confirms ~2h downtime
2

We have contacted the centrifuge devs to help us identify the root cause of the slashing event. We are working on this.

3 Likes
3

thank you for reply!

1 Like
4

Hi all,

we have been trying to figure out the root cause of the slashing event and while we found the signal that triggered the slashing, the root cause is still unknown.

TL;DR

The server running the ChorusOne validator node became unresponsive and we had to power-cycle it. After reboot, our node came back online and reported Prevote Equivocation.

Timeline

  • 2022-01-28 14:11:54 - our node went down (we know this from log pointing to last known height #9460394)
  • 2022-01-28 16:19:45 - our node came back online (we also know this from the logs)

In short, the validator was down between #9460394 - #9461658 block heights, for approximately 2 hours.

Investigation (so far)

After reviewing our logs, this is the only unusual entry we found:

{"log":"2022-01-28 16:19:44.959 main INFO babe  👶 Starting BABE Authorship worker\n","stream":"stderr","time":"2022-01-28T16:19:44.959191737Z"}
{"log":"2022-01-28 16:19:45.334 tokio-runtime-worker WARN afg  Detected prevote equivocation in the finality worker: Equivocation { round_number: 36853, identity: Public(72bd245ae66dc28b153ec00437e83dda704323152e18278ffb25ab95f0d48ea4 (5Ef9W87R...)), first: (Prevote { target_hash: 0x0a2c9f73700e697d312d36cb48ecefecbd5f03884f117ce5d89ed9d6cd506962, target_number: 9460393 }, Signature(6e5d0abe49e626cb8e44d7879b0b0ee493dc19e284fdd10cb635dd2e120a2f1245fb29ec50ef01bbd1a682099c8ede2c6eefd197485dd30c30f0b4c50d18d50a)), second: (Prevote { target_hash: 0xc7656ad3ac6328ed6a12ac54aff828b79765d219d029be743ac05badce8276c8, target_number: 9460392 }, Signature(96615f4a37398e22836c0dc044d40e5cb6506f978deb4f5dd8b623691575a3c7f089628120a978d5f146adcc49adcc833816ec63de7a44d087dc9296eecb760f)) }\n","stream":"stderr","time":"2022-01-28T16:19:45.334347276Z"}

Docker Image: centrifugeio/centrifuge-chain:20201022093419-da56ac5 (v2.0.0-rc6.0).

Following the trail of the log we found the relevant code paths:

Now with the substrate JS API we found the block containing the grandpa.reportEquivocationUnsigned extrinsic (#9461658): Polkadot/Substrate Portal

equivocation_proof: GrandpaEquivocationProof
{
  setId: 698,
  equivocation: {
    Prevote: {
      roundNumber: 36,853,
      identity: 4dwBxAs1RqLPVwiCvQsoDYEQbuCzwiCBLHdqVag4dG66s6Dh,
      first: [
        {
          targetHash: 0x0a2c9f73700e697d312d36cb48ecefecbd5f03884f117ce5d89ed9d6cd506962,
          targetNumber: 9,460,393
        },
        0x6e5d0abe49e626cb8e44d7879b0b0ee493dc19e284fdd10cb635dd2e120a2f1245fb29ec50ef01bbd1a682099c8ede2c6eefd197485dd30c30f0b4c50d18d50a
      ],
      second: [
        {
          targetHash: 0xc7656ad3ac6328ed6a12ac54aff828b79765d219d029be743ac05badce8276c8,
          targetNumber: 9,460,392
        },
        0x96615f4a37398e22836c0dc044d40e5cb6506f978deb4f5dd8b623691575a3c7f089628120a978d5f146adcc49adcc833816ec63de7a44d087dc9296eecb760f
      ]
    }
  }
}

What is the root cause?

In our setup nothing has changed, the node has been running successfully for a long time, therefore it’s possible that we hit a bug. Interestingly, almost at the same time our second node went down and came back just fine.

Taken the above information, I think it would be fair to propose a governance vote to revert the slashing (current state: “unapplied slash”).

Of course, if anyone has a better idea of what could cause the equivocation, please let us know!

4 Likes
5

Thank you for your deep feedback.

If voting will be launched I will definitely vote for the revert slashing event.

2 Likes
6

We’ve just created a proposal to cancel the slash:

thank you all for all the help!

2 Likes
7

Thank you for clarifying what happened - I will also vote for reverting the slashing event.

2 Likes
8

Thanks @mkaczanowski for reporting it openly and diving deeper on it. Since our version of substrate is fairly old, I think you indeed faced a substrate bug.
It is possible that is related to this: GRANDPA Equivocation and sysinfo Process Collection Results In Slashing on Kusama Network: a Post-Mortem.

I would vote as well on reverting the slashing event.

3 Likes
9

Hello, I voted no as I did not see any slash to take effect. The referendum failed because I voted no because Supermajority was necessary.

10

Thanks @TurkValidator for proactively participating on the on chain governance. I would like to clarify that the slashing event was canceled through the Council a few weeks back. So technically that referendum open was a no-op. More information here
All the best,

4 Likes