Page tree
Skip to end of metadata
Go to start of metadata

(lightbulb) We give here basic recommendations what to set on a server running openBIS concerning memory usage. There might be better memory settings for your specific use case. As an alternative please have a look at this webpage http://pgtune.leopard.in.ua/ which allows you to calculate your memory settings given on your server specs.

We distinguish between four classes of servers, based on available RAM.

NOTE: MaxPermSize support was removed in Java 8.

Java settings for openBIS Application Server:

<openBIS_HOME>/servers/openBIS-server/jetty/etc/openbis.conf

Server RAM

-Xmx

-Xms

XX:MaxPermSize

8G

4G

512M

256M

16G

6G

1G

512M

32G

16G

2G

512M

Configuration of table data caching

In openBIS AS UI can show table which are kept on server side for quick paging, sorting and filtering. In big openBIS instance these tables can be quite large. Therefore the table data is kepted in a cache. If there isn't enough space in the cache the less-recently-used entries are removed. 

The maximum cache size is by default 25% of maximum memory available to the JVM. With the system property ch.ethz.sis.openbis.generic.client.web.tabledatacache.size other percentage values (like 34%) or absolute values (like 512m and 2g) can be specified.

Java settings for openBIS Data Store Server:

<openBIS_HOME>/servers/datastore_server/etc/datastore_server.conf

Server RAM

-Xmx

-Xms

XX:MaxPermSize

8G

512M

128M

128M

16G

1G

256M

128M

32G

4G

512M

128M

Xmx: maximum java heap size
Xms: initial java heap size

-XX:MaxPermSize=64m Size of the Permanent Generation. (5.0 and newer: 64 bit VMs are scaled 30% larger; 1.4 amd64: 96m; 1.3.1 -client: 32m.)

 

Out of memory

In case you do a unspecific search, you might end up with a lot of result data which might overload the server due to lack of memory. The log message in the openBIS Application Server looks like this:

2017-08-03 13:28:20,426 ERROR [qtp1435804085-36 - /openbis/openbis/common] OPERATION.GWTRPCServiceExporter - Unexpected throwable
java.lang.OutOfMemoryError: GC overhead limit exceeded

In the GUI the response will look like this.

The easiest and quickest way to solve this problem is to increase the memory settings in <openBIS_HOME>/servers/openBIS-server/jetty/etc/openbis.conf.

 

Postgres settings:

Please have a look here for possible memory settings

Descriptions of each parameter:

shared_buffers (integer)
Sets the amount of memory the database server uses for shared memory buffers. The default is typically 32 megabytes (32MB), but might be less if your kernel settings will not support it (as determined during initdb). This setting must be at least 128 kilobytes. (Non-default values of BLCKSZ change the minimum.) However, settings significantly higher than the minimum are usually needed for good performance. Several tens of megabytes are recommended for production installations. This parameter can only be set at server start.

effective_cache_size (integer)
Sets the planner's assumption about the effective size of the disk cache that is available to a single query. This is factored into estimates of the cost of using an index; a higher value makes it more likely index scans will be used, a lower value makes it more likely sequential scans will be used. When setting this parameter you should consider both PostgreSQL's shared buffers and the portion of the kernel's disk cache that will be used for PostgreSQL data files. Also, take into account the expected number of concurrent queries on different tables, since they will have to share the available space. This parameter has no effect on the size of shared memory allocated by PostgreSQL, nor does it reserve kernel disk cache; it is used only for estimation purposes. The default is 128 megabytes (128MB).

work_mem (integer)
Specifies the amount of memory to be used by internal sort operations and hash tables before switching to temporary disk files. The value defaults to one megabyte (1MB). Note that for a complex query, several sort or hash operations might be running in parallel; each one will be allowed to use as much memory as this value specifies before it starts to put data into temporary files. Also, several running sessions could be doing such operations concurrently. So the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, and hash-based processing of IN subqueries.

maintenance_work_mem (integer)
Specifies the maximum amount of memory to be used in maintenance operations, such as VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY. It defaults to 16 megabytes (16MB). Since only one of these operations can be executed at a time by a database session, and an installation normally doesn't have many of them running concurrently, it's safe to set this value significantly larger than work_mem. Larger settings might improve performance for vacuuming and for restoring database dumps.

Note that when autovacuum runs, up to autovacuum_max_workers times this memory may be allocated, so be careful not to set the default value too high.

Example output of memory settings:

postgres=#  show shared_buffers; show effective_cache_size; show work_mem; show maintenance_work_mem;
 shared_buffers
----------------
 512MB
(1 row)

 effective_cache_size
----------------------
 1GB
(1 row)

 work_mem
----------
 1MB
(1 row)

 maintenance_work_mem
----------------------
 16MB
(1 row)

 

 

Since Postgres 9.5 it is possible to have a single file with all changes compared to postgresql.conf. This file is called postgresql.auto.conf. Here we have an example of /var/lib/pgsql/9.6/data/postgresql.auto.conf, which is used on a server with 8 GB of RAM (Note: We have activated PITR!):

# Do not edit this file manually!
# It will be overwritten by ALTER SYSTEM command.
archive_command = '/usr/local/bin/archive_wal.sh %p %f'
archive_mode = 'on'
wal_level = 'replica'
archive_timeout = '600'
checkpoint_timeout = '60min'
effective_cache_size = '2GB'
maintenance_work_mem = '512MB'
shared_buffers = '1GB'
work_mem = '256MB'
max_parallel_workers_per_gather = '2'
shared_preload_libraries = 'pg_stat_statements'
temp_buffers = '128MB'

 

 

  • No labels