database object caching is too greedy

[Dimitar Misev]

The ObjectBroker caches database objects, and does so even when there is no free ram, forcing the server to go into swap memory.

This seriously hampers performance, as free ram is occupied by (arguably) outdated, unused database objects, instead of the objects which are needed for evaluating the current query. It can even block the machine completely, when all the ram+swap is used up.

The proposed solution now is to limit the ObjectBroker to evicting old objects when free ram reaches a critical threshold, instead of only when total ram is filled up. I've set an arbitrary safe threshold of 100MB free ram that rasdaman's cache should not cross.

[Dimitar Misev]

Right now it would be best to even evict the whole cache after each query. The reason is that the cache is not shared among rasserver processes. If one query that runs on rasserver N1 fills up the cache, another query on N2 will not be able to run too well as N2 cannot manage the cache of N1.

The ultimate solution is to have a shared rasserver cache.

[Dimitar Misev]

Disregard my previous comment as the cache is cleared at the end of every transaction. The recommendation for a shared cache is still valid though.

[Dimitar Misev]

Deferred for now, will be addressed with Veranika's work.

[Veranika Liaukevich]

The comment for ObjectBroker states "The ObjectBroker can be configured to serve as a cross transaction cache". Is it done by default? I tried some simple queries (selects, updates), but I don't see any calls to ObjectBroker::registerDBObject, the function which inserts objects to be stored by object broker.

[Dimitar Misev]

I think the caching is done directly in the load* methods, I guess registerDBObject is not used.

[Peter Baumann]

seems like work on this should be synced with Veranika's cache mgr, pls keep her in the loop. Maybe best participate in the next thesis meeting (Mon 10:30).

[Veranika Liaukevich]

Dimitar, thanks for the note, registerDBObject is not used indeed. However, objectBroker reports everything freed after each transaction is commited/aborted. Is it possible, that some other component silently caches data? postgres driver?