Node out of sync / partial network outage
Incident Report for filecoin
Resolved
Lotus patches are out now (https://github.com/filecoin-project/lotus/releases/tag/v1.28.3; https://github.com/filecoin-project/lotus/releases/tag/v1.29.1) and the impacted nodes are recovering. Implementers are considering this incident as resolved now, and we will keep monitoring the network. Thank you to the Filecoin node operators for all your support for a speedy recovery!
Posted Sep 16, 2024 - 19:43 UTC
Identified
We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.
Posted Sep 16, 2024 - 18:39 UTC
Update
We have identified the issue: the recent BLS library dependency update in Lotus was causing BLS signature validation failure. The Lotus team is preparing the patches against the latest stable release for v1.28.x and v1.29.x, and we recommend that all node operators update to the newest release as soon as possible! For more details and ongoing updates, please see https://github.com/filecoin-project/lotus/issues/12467.
Posted Sep 16, 2024 - 18:39 UTC
Investigating
Many Filecoin nodes are not syncing due to "failed to verify beacon". Lotus implementers are investigating the issue with the Drand implementers, and we will provide updates in #fil-network-announcement and https://github.com/filecoin-project/lotus/issues/12467 as soon as we have more information.
Posted Sep 16, 2024 - 17:25 UTC
This incident affected: Mainnet (Chain Height).