Sequencer

DA network issues

Not Connecting to DA Light Client

• Error Displayed:

  time="Sep 19 08:45:15.953" level=error msg="Failed to sync Celestia DA[attempt 1 error RPC client error: sendRequest failed: Post \"http://localhost:26658\": dial tcp 127.0.0.1:26658: connect: connection refused]" module=celestia

• Solution:
  The DA light client is not accessible at the specified URL http://localhost:26658. Please verify that the Celestia light client is up and running. If it is not running, you can start it with the following command:

  roller da-light-client start

DA Not Synced

• Error Displayed:

  time="Sep 19 09:03:18.539" level=info msg="Celestia-node syncing[height 28161 target 2737972]" module=celestia

• Solution:
  Sequencers need to connect to a synced Celestia light client. If the Celestia light client is not in sync (e.g., due to being down for an extended period), the RollApp will have to wait until the client catches up.

DA Unauthorized

• Error Displayed:

  level=error msg="Failed to sync Celestia DA[attempt 2 error RPC client error: sendRequest failed: request failed, http status 401 Unauthorized]" module=celestia

• Solution:
  Roller will configure RollApp with the authorization token from the DA light client. If there are modifications, the auth_token in da_config in the dymint.toml configuration file will need to be updated. The authorization token can be obtained using the following command:

  celestia light auth admin --node.store <da-light-client-store-path> --gateway --p2p.network <celestia-network>

Not submitting batches to Dymension

Sequencer Out of Funds

• Error Displayed:
  roller rollapp status shows unhealthy and the balance will be shown as 0.

• Solution:
  The active proposer account must be funded to submit state updates to the hub, which will include batches of block information (e.g., DA info and state roots). If the account has insufficient funds to send transactions to the hub, state updates will not be submitted, and DA batches will be disregarded.

Celestia Light Client Account Not Found

• Error Displayed:

  time="Sep 19 09:30:32.071" level=error msg="Submit blob.[error do rpc submit: querying account my_celes_key: rpc error: code = NotFound desc = account celestia1tcql03snla6eh94zkg3tz9dc2sk0u05cqlrndf not found]" module=celestia

• Solution:
  The account celestia1tcql03snla6eh94zkg3tz9dc2sk0u05cqlrndf is not found because it has never been funded. Transfer funds to the account to enable batch submission to Celestia. If using the Mocha testnet, you can obtain funds from the Celestia Discord faucet.

Celestia Light Client Account with Insufficient Funds

• Error Displayed:

  time="2024-08-29T05:44:36Z" level=error msg="Submit blob.[error do rpc submit: tx failed with code 5: 31810utia is smaller than 95022utia: insufficient funds: insufficient funds]" module=celestia

• Solution:
  Transfer funds to the account to enable batch submission to Celestia. If using the Mocha testnet, you can obtain funds from the Celestia Discord faucet.

Failing to Submit Blob to DA

• Error Displayed:

  time="2024-09-05T14:23:26Z" level=error msg="Submit blob.[error do rpc submit: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 65.109.16.220:9090: connect: connection refused\"]" module=celestia

• Solution:
  Issues with submitting blobs may be caused by network problems within Celestia. First, try restarting the Celestia light client. If the problem persists, it may be related to the Celestia full-node that the light client is connected to. If using a public node, trying a different one could help. A list of public nodes is available in the Celestia documentation.

Failing to Check Availability of Blob Submitted to Celestia

• Error Displayed:

  time="2024-09-16T15:48:04Z" level=error msg="Blob submitted not found in DA. Retrying availability check.[]" module=celestia

• Solution:
  Availability check issues for submitted blobs are often due to performance issues within the Celestia network. First, try restarting the Celestia light client. If the problem persists, it may be related to the performance of the Celestia full-node connected to the light client. If using a public node, trying a different one could help. A list of public nodes is available in the Celestia documentation.

Block Production Stopped

• Error Displayed:

  [31mERRO[0m[48759] Enough bytes to build a batch have been accumulated, but too many batches are pending submission. Pausing block production until a signal is consumed.[]  [31mmodule[0m=BlockManager

• Solution:
  Sequencers that fail to submit to the DA and/or to the Hub, and accumulate max_skew_batch without submission, will stop block production, causing the node to become unhealthy. No new blocks will be generated until batches are successfully submitted. Resolving the issues listed above will reactivate block production.

Transactions Not Received

• Error Displayed:
  (No specific error message provided)

• Solution:
  If transactions are not arriving at the sequencer, it could be due to several reasons, with the most likely cause being that RPC nodes are not synced or not connected to the sequencer through the P2P network.

  To check P2P connectivity in the sequencer, follow these steps:

  1. Run the following command:

     rollappd dymint show-p2p-info --home=<rollapp_path>

  2. If the output displays something like:

     Host ID: 12D3KooWSUAAXqyz8To4xQwwBZyojSwWZMngSVKMEk215KX7wB3h
     Listening P2P addresses: [/ip4/127.0.0.1/tcp/26656 /ip4/192.168.1.6/tcp/26656]
     0 peers connected

     This means the sequencer is not connected to any node. Check that RPC nodes are up and running. Additionally, ensure that there is a p2p_bootstrap_nodes field in the dymint.toml configuration file, with a P2P address in the following format:

     /ip4/<ip_address>/tcp/<port>/p2p/<host_id>

     where <host_id> is the one obtained using rollappd dymint show-p2p-info from the sequencer or the node you want to connect to, and the default P2P port is 26656.