Filecoin chain halted after height 336458
Incident Report for filecoin
Postmortem

Please see this post on *Resolving the Dec 19 Chain Halt: Cause, Impact, & Take Aways *for cause and impact of this outage. More specific post mortems WIP from the actors & infra teams to describe specific learnings and mitigations for the future.

Posted Jan 16, 2021 - 23:13 UTC

Resolved
If you’re just catching up: check the upgrade and recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) to get back in sync.

Miners have upgraded & the chain is now recovering smoothly. 🎉🙏💪 We don’t expect further action is required, but will continue monitoring.

Expect degraded chain quality. We should expect the network to be somewhat degraded for the next few hours as miners continue to upgrade and power recovers (currently ~34% upgraded). We recommend pausing all non-critical messages until chain capacity is restored to normal levels - BaseFee may be volatile as the network recovers.

Full recovery ETA. Conservatively, we will deem the network to be fully recovered & stable 2x finality from when the chain resumed. This means 1,400 epochs (11.6 hrs) from 337120 (2020-12-19 23:20 UTC). So, conservatively, full recovery at epoch 338520 (2020-12-20 11:00 UTC).

For custodians, exchanges, wallets: The full recovery ETA is very conservative, and the chain is expected to be fully functional long before. Our recommendation for you would be to keep transfers out disabled until that time or until you gain comfort with chain status -- at least for larger amounts. Your own internal transfers, trading, and other fully off-chain activity should be fine.
Posted Dec 20, 2020 - 00:03 UTC
Update
We are continuing to monitor for any further issues.
Posted Dec 19, 2020 - 23:27 UTC
Monitoring
The network has restarted and progressed to the expected block height (337120 @ 3:20pm PT). As more power joins the upgraded chain, the network will continue to stabilize. Check back for an update here when >50% of power is in sync.
Posted Dec 19, 2020 - 23:26 UTC
Update
Lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) has been released. Please upgrade ASAP and see the recovery instructions (https://github.com/filecoin-project/lotus/discussions/5232) for more information/troubleshooting.
Posted Dec 19, 2020 - 22:16 UTC
Identified
The Filecoin network is halted with most nodes stuck at height 336459 with mismatched state compute error due to indeterminate behavior. We have found the root cause and are testing the fix.

Next Step: Miners will need to upgrade to lotus v1.4.0 (https://github.com/filecoin-project/lotus/releases/tag/v1.4.0) with the fix when it is released. Once miners upgrade, the chain will start to catch up.
Posted Dec 19, 2020 - 21:29 UTC
This incident affected: Mainnet (Chain Height).