out of memory

[lastmjs]

First off, my JavaScript environment is very custom and different than the web or Node.js. We are running in a wasm32-wasi environment with QuickJS, using wasmedge-quickjs, and Wasmtime on the Internet Computer.

That being said, the strange thing is that I had this working about a week ago, but now it doesn't work. We've been doing a lot of low-level changes to our JS environment (we're working at the engine/platform level), but I just don't understand what could have caused this.

The simple problem is that we get an error just trying to initialize the database: out of memory

Here's our code:

import { Server } from 'azle';
import express from 'express';
import initSqlJs from 'sql.js/dist/sql-asm.js';

export default Server(async () => {
    const SQL = await initSqlJs({
        locateFile: (filename) => `/dist/${filename}`
    });

    const db = new SQL.Database();

    const app = express();

    app.get('/', (req, res) => {
        res.send('Hello world');
    });

    return app.listen();
});

We have to use asm as our environment does not have a Wasm interpreter. I've tried sql-asm-memory-growth.js as well and I get the same error. I've even gone back multiple versions to maybe 1.7.0 and the error is always there.

Any ideas on what could cause this? I've also tried adding:

            TOTAL_STACK: 4_000_000_000,
            TOTAL_MEMORY: 4_000_000_000,
            FAST_MEMORY: 4_000_000_000

to the initSqlJs options.

[lovasoa]

Do you have an easy and minimal way to reproduce your issue?

@kripken , any idea?

[lovasoa]

Are you sure /dist/${filename} is what you need? Did you check if the files were loaded correctly in your custom environment.

[lovasoa]

Compiling sqlite to js and quickjs to wasm to run one on the other strikes me as bizarre and inefficient. Why not compile sqlite to wasm directly if you have access to a wasm runtime ?

[lastmjs]

Do you have an easy and minimal way to reproduce your issue?

@kripken , any idea?

I could get you a reproduction maybe after our next release, but it's not going to be "easy" because you would have to setup our dev environment.

[lastmjs]

Are you sure /dist/${filename} is what you need? Did you check if the files were loaded correctly in your custom environment.

I don't think this matters in our case at all since we are using sql-asm.js, am I correct? I've tried changing that path and even removing it entirely and the behavior is the same.

Though I'm not really sure what that path is supposed to do, when I logged it it was looking for a .wasm file I believe, so seemed not applicable.

[lastmjs]

Compiling sqlite to js and quickjs to wasm to run one on the other strikes me as bizarre and inefficient. Why not compile sqlite to wasm directly if you have access to a wasm runtime ?

We're basically building a Node/Deno for the Internet Computer which is a Wasm runtime, and we are trying to allow developers to npm install any package and just use it. So we would like SQL.js to work without modifications.

It was working...alas this bug that I can't seem to get around now.

[kripken]

@kripken , any idea?

Sorry, no - an internal out of memory error is too general.

I'd recommend the usual ideas, though, as before:

• Try a debug build (-sASSERTIONS)
• Try a sanitizers build (say, -fsanitize=undefined,address)
• Bisect on emscripten releases

[lovasoa]

@lastmjs , what does the debug build output for you ?

[lastmjs]

I think I tracked down the source of the issue, though I don't fully understand yet, my understanding is deepening. This seems to have to do with the order of some of our imports, particularly our importing of a polyfill of TextEncoding and TextDecoding. It's relatively project-specific, so not sure if this has anything to do with sql.js itself.

It would be nice if the error could be better...I'm still investigating. I was able to reproduce and fix is the good part.