Polkastats Crawler is not working

[gituser]

Describe the bug

I've tried following the docker-compose process on the latest develop branch:

npm run docker:build:westend
npm run docker:westend

and getting this on the crawler container:

2021-02-22 20:27:57        API/INIT: RPC methods not decorated: system_addLogFilter, system_resetLogFilter
2021-02-22 20:27:57        METADATA: Unknown types found, no types for BlockWeights, ConsumedWeight
2021-02-22 20:27:57        REGISTRY: Unable to resolve type BlockWeights, it will fail on construction
2021-02-22 20:27:57        REGISTRY: Error: Cannot construct unknown type BlockWeights
    at new DoNotConstruct (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/primitive/DoNotConstruct.js:21:11)
    at new <anonymous> (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/primitive/DoNotConstruct.js:27:9)
    at new Type (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/registry.js:300:16)
    at initType (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/createType.js:49:19)
    at createTypeUnsafe (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/createType.js:68:12)
    at createType (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/createType.js:83:10)
    at TypeRegistry.createType (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/registry.js:257:39)
    at /usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/consts/fromMetadata.js:27:30
    at Array.reduce (<anonymous>)
    at /usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/consts/fromMetadata.js:24:73
2021-02-22 20:27:57        API/INIT: Error: FATAL: Unable to initialize the API: createType(BlockWeights):: Cannot construct unknown type BlockWeights
    at EventEmitter.value (/usr/app/polkastats-backend-v3/node_modules/@polkadot/api/base/Init.js:87:25)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
2021-02-22 20:27:57        API/INIT: Error: createType(BlockWeights):: Cannot construct unknown type BlockWeights
    at createTypeUnsafe (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/createType.js:71:11)
    at createType (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/createType.js:83:10)
    at TypeRegistry.createType (/usr/app/polkastats-backend-v3/node_modules/@polkadot/types/create/registry.js:257:39)
    at /usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/consts/fromMetadata.js:27:30
    at Array.reduce (<anonymous>)
    at /usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/consts/fromMetadata.js:24:73
    at Array.reduce (<anonymous>)
    at constantsFromMeta (/usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/consts/fromMetadata.js:15:36)
    at expandMetadata (/usr/app/polkastats-backend-v3/node_modules/@polkadot/metadata/decorate/index.js:45:49)
    at ApiPromise.injectMetadata (/usr/app/polkastats-backend-v3/node_modules/@polkadot/api/base/Decorate.js:182:56)
(node:19) UnhandledPromiseRejectionWarning: Error: FATAL: Unable to initialize the API: createType(BlockWeights):: Cannot construct unknown type BlockWeights
    at EventEmitter.value (/usr/app/polkastats-backend-v3/node_modules/@polkadot/api/base/Init.js:87:25)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(node:19) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:19) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I've tried both polkadot v0.8.27 and polkadot v0.8.28-1 (latest) - same result.

Also another question is: crawler should be crawling blocks into the postgres and who is serving /api/ endpoint? Is it graphql-container?

Thanks.

@mariopino

Expected behavior

Screenshots

Additional context

[gituser]

After bumping polkadot-api to 3.9.3, the crawler seems to be working (not sure), used this commit - d6df56b

But I still do not see where should point /api/v3 endpoint, I've tried putting it into 8082 port (hasura/graphql) but it responds with 404 error..

Particularly these endpoints in the polkastats config, what do I specify If I'm using docker (my local ones):

nodeWs: "wss://westend.polkastats.io/rpc",
    backendWs: "wss://westend.polkastats.io/api/v3",
    backendHttp: "http://westend.polkastats.io/api/v3",

EDIT:
it seems I've figured it out:

wss://westend.polkastats.io/api/v3 -> points to /v1/graphql which is on port 8082, so in nginx (if you're using one) you need to either rewrite /api/v3 to /v1/graphql or specify graphql endpoint, e.g.:

location /api/v3 {
	     rewrite /api/v3 /v1/graphql break;
             proxy_pass  http://127.0.0.1:8082;
             proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
             proxy_set_header X-Forwarded-Proto https;
             proxy_redirect off;
             proxy_buffering off;
             proxy_http_version 1.1;
    	     proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";
        }

Thought transfers seems to be empty for now on my instance:

Also it seems https://westend.polkastats.io is stuck too on 12th Jan for some reason.

Another issue:
on local deployment when visiting any address, e.g. http://localhost:3000/account/5ENpP27BrVdJTdUfY6djmcw3d3xEJ6NzSUU52CCPmGpMrdEY loading data takes forever and nothing is displayed, in console I can see this:

commons.68c530e.js:2 TypeError: address.substring is not a function
    at f.shortAddress (index.049fb00.js:1)
    at fn (index.049fb00.js:1)
    at r (commons.68c530e.js:2)
    at In (app.5745132.js:2)
    at f.normalizeSlot (app.5745132.js:2)
    at f.renderTbodyRowCell (app.5745132.js:2)
    at app.5745132.js:2
    at Array.map (<anonymous>)
    at f.renderTbodyRow (app.5745132.js:2)
    at app.5745132.js:2
te @ commons.68c530e.js:2
commons.68c530e.js:2 Error: Invalid character
    at n (7a82ef42.6730359.js:2)
    at c (7a82ef42.6730359.js:2)
    at h._parseBase (7a82ef42.6730359.js:2)
    at h._init (7a82ef42.6730359.js:2)
    at new h (7a82ef42.6730359.js:2)
    at formatAmount (index.049fb00.js:1)
    at f.<anonymous> (_id.3e11fbb.js:1)
    at f.t._render (commons.68c530e.js:2)
    at f.r (commons.68c530e.js:2)
    at _n.get (commons.68c530e.js:2)

it seems same issue here as well - https://polkastats.io/account/15j4dg5GzsL1bw2U2AWgeyAk6QTxq43V7ZPbXdAmbVLjvDCK

Can you clarify please @mariopino?

Thanks.