• [COMPUTER]: Defines the cluster hosts.

• [NDBD]: Defines the cluster's data nodes.

• [MYSQLD]: Defines the cluster's MySQL server nodes.

• [MGM] or [NDB_MGMD]: Defines the cluster's management server node.

• [TCP]: Defines TCP/IP connections between nodes in the cluster, with TCP/IP being the default connection protocol.

• [SHM]: Defines shared-memory connections between nodes. Formerly, this type of connection was available only in binaries that were built using the --with-ndb-shm option. In MySQL 5.1-Max, it is enabled by default, but should still be considered experimental.

• Each executable has its own command-line option which enables specifying the management server at startup. (See the documentation for the respective executable.)

• It is also possible to set the connectstring for all nodes in the cluster at once by placing it in a [mysql_cluster] section in the management server's my.cnf file.

• For backward compatibility, two other options are available, using the same syntax:

  1. Set the NDB_CONNECTSTRING environment variable to contain the connectstring.

  2. Write the connectstring for each executable into a text file named Ndb.cfg and place this file in the executable's startup directory.

  However, these are now deprecated and should not be used for new installations.

• Id

  This is an integer value, used to refer to the host computer elsewhere in the configuration file.

• HostName

  This is the computer's hostname or IP address.

• Id

  Each node in the cluster has a unique identity, which is represented by an integer value in the range 1 to 63 inclusive. This ID is used by all internal cluster messages for addressing the node.

• ExecuteOnComputer

  This refers to one of the computers defined in the [COMPUTER] section.

• PortNumber

  This is the port number on which the management server listens for configuration requests and management commands.

• LogDestination

  This parameter specifies where to send cluster logging information. There are three options in this regard: CONSOLE, SYSLOG, and FILE:

  • CONSOLE outputs the log to stdout:

    CONSOLE
    

  • SYSLOG sends the log to a syslog facility, possible values being one of auth, authpriv, cron, daemon, ftp, kern, lpr, mail, news, syslog, user, uucp, local0, local1, local2, local3, local4, local5, local6, or local7.

    Note: Not every facility is necessarily supported by every operating system.

    SYSLOG:facility=syslog
    

  • FILE pipes the cluster log output to a regular file on the same machine. The following values can be specified:

    • filename: The name of the logfile.

    • maxsize: The maximum size (in bytes) to which the file can grow before logging rolls over to a new file. When this occurs, the old logfile is renamed by appending .x to the filename, where x is the next number not yet used with this name.

    • maxfiles: The maximum number of logfiles.

    FILE:filename=cluster.log,maxsize=1000000,maxfiles=6
    

    It is possible to specify multiple log destinations as shown here, using a semicolon-delimited string:

    CONSOLE;SYSLOG:facility=local0;FILE:filename=/var/log/mgmd
    

    The default value for the FILE parameter is FILE:filename=ndb_node_id_cluster.log,maxsize=1000000,maxfiles=6, where node_id is the ID of the node.

• ArbitrationRank

  This parameter is used to define which nodes can act as arbitrators. Only MGM nodes and SQL nodes can be arbitrators. ArbitrationRank can take one of the following values:

  • 0: The node will never be used as an arbitrator.

  • 1: The node has high priority; that is, it will be preferred as an arbitrator over low-priority nodes.

  • 2: Indicates a low-priority node which be used as an arbitrator only if a node with a higher priority is not available for that purpose.

  Normally, the management server should be configured as an arbitrator by setting its ArbitrationRank to 1 (the default value) and that of all SQL nodes to 0.

• ArbitrationDelay

  An integer value which causes the management server's responses to arbitration requests to be delayed by that number of milliseconds. By default, this value is 0; it is normally not necessary to change it.

• DataDir

  This sets the directory where output files from the management server will be placed. These files include cluster log files, process output files, and the daemon's pid file. (For log files, this can be overridden by setting the FILE parameter for [NDB_MGMD]LogDestination as discussed previously in this section.)

• Either ExecuteOnComputer or HostName.

• The parameter NoOfReplicas

• Id

  This is the node ID used as the address of the node for all cluster internal messages. This is an integer between 1 and 63. Each node in the cluster has a unique identity.

• ExecuteOnComputer

  This refers to one of the computers (hosts) defined in the COMPUTER section.

• HostName

  Specifying this parameter has an effect similar to specifying ExecuteOnComputer. It defines the hostname of the computer the storage node on which is to reside. Either this parameter or ExecuteOnComputer is required in order to specify a hostname other than localhost.

• ServerPort (OBSOLETE)

  Each node in the cluster uses a port to connect to other nodes. This port is used also for non-TCP transporters in the connection setup phase. The default port is allocated dynamically in such a way as to ensure that no two nodes on the same computer receive the same port number, so it should not normally be necessary to specify a value for this parameter.

• NoOfReplicas

  This global parameter can be set only in the [NDBD DEFAULT] section, and defines the number of replicas for each table stored in the cluster. This parameter also specifies the size of node groups. A node group is a set of nodes all storing the same information.

  Node groups are formed implicitly. The first node group is formed by the set of data nodes with the lowest node IDs, the next node group by the set of the next lowest node identities, and so on. By way of example, assume that we have 4 data nodes and that NoOfReplicas is set to 2. The four data nodes have node IDs 2, 3, 4 and 5. Then the first node group is formed from nodes 2 and 3, and the second node group by nodes 4 and 5. It is important to configure the cluster in such a manner that nodes in the same node groups are not placed on the same computer, as in this situation a single hardware failure would cause the entire cluster to crash.

  If no node IDs are provided then the order of the data nodes will be the determining factor for the node group. Whether or not explicit assignments are made, they can be viewed in the output of the management client's SHOW command.

  There is no default value for NoOfReplicas; the maximum possible value is 4.

• DataDir

  This parameter specifies the directory where trace files, log files, pid files and error logs are placed.

• FileSystemPath

  This parameter specifies the directory where all files created for metadata, REDO logs, UNDO logs and data files are placed. The default is the directory specified by DataDir. Note: This directory must exist before the ndbd process is initiated.

  The recommended directory hierarchy for MySQL Cluster includes /var/lib/mysql-cluster, under which a directory for the node's filesystem is created. This subdirectory contains the node ID. For example, if the node ID is 2, then this subdirectory is named ndb_2_fs.

• BackupDataDir

  It is also possible to specify the directory in which backups are placed. By default, this directory is FileSystemPath/BACKUP. (See above.)

• DataMemory

  This parameter defines the amount of space available for storing database records. The entire amount is allocated in memory, so it is extremely important that the machine has sufficient physical memory to accommodate this value.

  The memory allocated by DataMemory is used to store both the actual records and indexes. Each record is currently of fixed size. (Even VARCHAR columns are stored as fixed-width columns.) There is a 16-byte overhead on each record; an additional amount for each record is incurred because it is stored in a 32KB page with 128 byte page overhead (see below). There is also a small amount wasted per page due to the fact that each record is stored in only one page. The maximum record size is currently 8052 bytes.

  The memory space defined by DataMemory is also used to store ordered indexes, which use about 10 bytes per record. Each table row is represented in the ordered index. A common error among users is to assume that all indexes are stored in the memory allocated by IndexMemory, but this is not the case: only primary key and unique hash indexes use this memory; ordered indexes use the memory allocated by DataMemory. However, creating a primary key or unique hash index also creates an ordered index on the same keys, unless you specify USING HASH in the index creation statement. This can be verified by running ndb_desc -d db_name table_name in the management client.

  The memory space allocated by DataMemory consists of 32KB pages, which are allocated to table fragments. Each table is normally partitioned into the same number of fragments as there are data nodes in the cluster. Thus, for each node, there are the same number of fragments as are set in NoOfReplicas. Once a page has been allocated, it is currently not possible to return it to the pool of free pages, except by deleting the table. Performing a node recovery also will compress the partition because all records are inserted into empty partitions from other live nodes.

  The DataMemory memory space also contains UNDO information: For each update, a copy of the unaltered record is allocated in the DataMemory. There is also a reference to each copy in the ordered table indexes. Unique hash indexes are updated only when the unique index columns are updated, in which case a new entry in the index table is inserted and the old entry is deleted upon commit. For this reason, it is also necessary to allocate enough memory to handle the largest transactions performed by applications using the cluster. In any case, performing a few large transactions holds no advantage over using many smaller ones, for the following reasons:

  • Large transactions are not any faster than smaller ones

  • Large transactions increase the number of operations that are lost and must be repeated in event of transaction failure

  • Large transactions use more memory

  The default value for DataMemory is 80MB; the minimum is 1MB. There is no maximum size, but in reality the maximum size has to be adapted so that the process does not start swapping when the limit is reached. This limit is determined by the amount of physical RAM available on the machine and by the amount of memory that the operating system may commit to any one process. 32-bit operating systems are generally limited to 2-4GB per process; 64-bit operating systems can use more. For large databases, it may be preferable to use a 64-bit operating system for this reason. In addition, it is also possible to run more than one ndbd process per machine, and this may prove advantageous on machines with multiple CPUs.

• IndexMemory

  This parameter controls the amount of storage used for hash indexes in MySQL Cluster. Hash indexes are always used for primary key indexes, unique indexes, and unique constraints. Note that when defining a primary key and a unique index, two indexes will be created, one of which is a hash index used for all tuple accesses as well as lock handling. It is also used to enforce unique constraints.

  The size of the hash index is 25 bytes per record, plus the size of the primary key. For primary keys larger than 32 bytes another 8 bytes is added.

  Consider a table defined by

  CREATE TABLE example (
    a INT NOT NULL,
    b INT NOT NULL,
    c INT NOT NULL,
    PRIMARY KEY(a),
    UNIQUE(b)
  ) ENGINE=NDBCLUSTER;
  

  There are 12 bytes overhead (having no nullable columns saves 4 bytes of overhead) plus 12 bytes of data per record. In addition we have two ordered indexes on columns a and b consuming roughly 10 bytes each per record. There is a primary key hash index on the base table using roughly 29 bytes per record. The unique constraint is implemented by a separate table with b as primary key and a as a column. This table will consume an additional 29 bytes of index memory per record in the example table as well as 12 bytes of overhead, plus 8 bytes of record data.

  Thus, for one million records, we need 58 MB for index memory to handle the hash indexes for the primary key and the unique constraint. We also need 64 MB for the records of the base table and the unique index table, plus the two ordered index tables.

  You can see that hash indexes takes up a fair amount of memory space; however, they provide very fast access to the data in return. They are also used in MySQL Cluster to handle uniqueness constraints.

  Currently the only partitioning algorithm is hashing and ordered indexes are local to each node. Thus ordered indexes cannot be used to handle uniqueness constraints in the general case.

  An important point for both IndexMemory and DataMemory is that the total database size is the sum of all data memory and all index memory for each node group. Each node group is used to store replicated information, so if there are four nodes with 2 replicas, then there will be two node groups. Thus, the total data memory available is 2*DataMemory for each data node.

  It is highly recommended that DataMemory and IndexMemory be set to the same values for all nodes. Data distribution is even over all nodes in the cluster, so the maximum amount of space available for any node can be no greater than that of the smallest node in the cluster.

  DataMemory and IndexMemory can be changed, but decreasing either of these can be risky; doing so can easily lead to a node or even an entire MySQL Cluster that is unable to restart due to there being insufficient memory space. Increasing these values should be acceptable, but it is recommended that such upgrades are performed in the same manner as a software upgrade, beginning with an update of the configuration file, then restarting the management server followed by restarting each data node in turn.

  Updates do not increase the amount of index memory used. Inserts take effect immediately; however, rows are not actually deleted until the transaction is committed.

  The default value for IndexMemory is 18MB. The minimum is 1MB.

• MaxNoOfConcurrentTransactions

  For each active transaction in the cluster there must be a record in one of the cluster nodes. The task of coordinating transactions is spread among the nodes: the total number of transaction records in the cluster is the number of transactions in any given node times the number of nodes in the cluster.

  Transaction records are allocated to individual MySQL servers. Normally there is at least one transaction record allocated per connection that using any table in the cluster. For this reason, one should ensure that there are more transaction records in the cluster than there are concurrent connections to all MySQL servers in the cluster.

  This parameter must be set to the same value for all cluster nodes.

  Changing this parameter is never safe and doing so can cause a cluster to crash. When a node crashes one of the nodes (actually the oldest surviving node) will build up the transaction state of all transactions ongoing in the crashed node at the time of the crash. It is thus important that this node has as many transaction records as the failed node.

  The default value for this parameter is 4096.

• MaxNoOfConcurrentOperations

  It is a good idea to adjust the value of this parameter according to the size and number of transactions. When performing transactions of only a few operations each and not involving a great many records, there is no need to set this parameter very high. When performing large transactions involving many records need to set this parameter higher.

  Records are kept for each transaction updating cluster data, both in the transaction coordinator and in the nodes where the actual updates are performed. These records contain state information needed in order to find UNDO records for rollback, lock queues, and other purposes.

  This parameter should be set to the number of records to be updated simultaneously in transactions, divided by the number of cluster data nodes. For example, in a cluster which has 4 data nodes and which is expected to handle 1,000,000 concurrent updates using transactions, you should set this value to 1000000 / 4 = 250000.

  Read queries which set locks also cause operation records to be created. Some extra space is allocated within individual nodes to accommodate cases where the distribution is not perfect over the nodes.

  When queries make use of the unique hash index, there are actually two operation records used per record in the transaction. The first record represents the read in the index table and the second handles the operation on the base table.

  The default value for this parameter is 32768.

  This parameter actually handles two values that can be configured separately. The first of these specifies how many operation records are to be placed with the transaction coordinator. The second part specifies how many operation records are to be local to the database.

  A very large transaction performed on an 8-node cluster requires as many operation records in the transaction coordinator as there are reads, updates, and deletes involved in the transaction. However, the operation records of the are spread over all 8 nodes. Thus, if it is necessary to configure the system for one very large transaction, then it is a good idea to configure the two parts separately. MaxNoOfConcurrentOperations will always be used to calculate the number of operation records in the transaction coordinator portion of the node.

  It is also important to have an idea of the memory requirements for operation records. These consume about 1KB per record.

• MaxNoOfLocalOperations

  By default, this parameter is calculated as 1.1 * MaxNoOfConcurrentOperations which fits systems with many simultaneous transactions, none of them being very large. If there is a need to handle one very large transaction at a time and there are many nodes, then it is a good idea to override the default value by explicitly specifying this parameter.

• MaxNoOfConcurrentIndexOperations

  For queries using a unique hash index another, temporary set of operation records is used during a query's execution phase. This parameter sets the size of that pool of records. Thus this record is only allocated while executing a part of a query, as soon as this part has been executed the record is released. The state needed to handle aborts and commits is handled by the normal operation records where the pool size is set by the parameter MaxNoOfConcurrentOperations.

  The default value of this parameter is 8192. Only in rare cases of extremely high parallelism using unique hash indexes should it be necessary to increase this value. Using a smaller value is possible and can save memory if the DBA is certain that a high degree parallelism is not required for the cluster.

• MaxNoOfFiredTriggers

  The default value of MaxNoOfFiredTriggers is 4000, which is sufficient for most situations. In some cases it can even be decreased if the DBA feels certain the need for parallelism in the cluster is not high.

  A record is created when an operation is performed that affects a unique hash index. Inserting or deleting a record in a table with unique hash indexes or updating a column that is part of a unique hash index fires an insert or a delete in the index table. The resulting record is used to represent this index table operation while waiting for the original operation that fired it to complete. This operation is short lived but can still require a large number of records in its pool for situations with many parallel write operations on a base table containing a set of unique hash indexes.

• TransactionBufferMemory

  The memory affected by this parameter is used for tracking operations fired when updating index tables and reading unique indexes. This memory is used to store the key and column information for these operations. It is only very rarely that the value for this parameter needs to be altered from the default.

  Normal read and write operations use a similar buffer, whose usage is even more short-lived. The compile-time parameter ZATTRBUF_FILESIZE (found in ndb/src/kernel/blocks/Dbtc/Dbtc.hpp) set to 4000*128 bytes (500KB). A similar buffer for key info, ZDATABUF_FILESIZE (also in Dbtc.hpp) contains 4000 × 16 = 62.5KB of buffer space. Dbtc is the module which handles transaction coordination.

There are additional [NDBD] parameters in the Dblqh module (in ndb/src/kernel/blocks/Dblqh/Dblqh.hpp) which affect reads and updates. These include ZATTRINBUF_FILESIZE, set by default to 10000*128 bytes (1250KB) and ZDATABUF_FILE_SIZE, set by default to 10000*16 bytes (roughly 156KB) of buffer space. To date, there have been neither any reports from users nor any results from our own extensive tests suggesting that either of these compile-time limits should be increased.

The default value for TransactionBufferMemory is 1MB.

• MaxNoOfConcurrentScans

  This parameter is used to control the number of parallel scans that can be performed in the cluster. Each transaction coordinator can handle the number of parallel scans defined for this parameter. Each scan query is performed by scanning all partitions in parallel. Each partition scan uses a scan record in the node where the partition is located, the number of records being the value of this parameter times the number of nodes. The cluster should be able to sustain MaxNoOfConcurrentScans scans concurrently from all nodes in the cluster.

  Scans are actually performed in two cases. The first of these cases occurs when no hash or ordered indexes exists to handle the query, in which case the query is executed by performing a full table scan. The second case is encountered when there is no hash index to support the query but there is an ordered index. Using the ordered index means executing a parallel range scan. The order is kept on the local partitions only, so it is necessary to perform the index scan on all partitions.

  The default value of MaxNoOfConcurrentScans is 256. The maximum value is 500.

  This parameter specifies the number of scans possible in the transaction coordinator. If the number of local scan records is not provided, it is calculated as the product of MaxNoOfConcurrentScans and the number of data nodes in the system.

• MaxNoOfLocalScans

  Specifies the number of local scan records if many scans are not fully parallelized.

• BatchSizePerLocalScan

  This parameter is used to calculate the number of lock records which needs to be there to handle many concurrent scan operations.

  The default value is 64; this value has a strong connection to the ScanBatchSize defined in the SQL nodes.

• LongMessageBuffer

  This is an internal buffer used for passing messages within individual nodes and between nodes. Although it is highly unlikely that this would need to be changed, it is configurable. By default it is set to 1MB.

• NoOfFragmentLogFiles

  This parameter sets the size of the node's REDO log files. REDO log files are organized in a ring. It is extremely important that the first and last log files (sometimes referred to as the “head” and “tail” log files, respectively) do not meet; when these approach one another too closely, the node will begin aborting all transactions encompassing updates due to a lack of room for new log records.

  A REDO log record is not removed until three local checkpoints have been completed since that log record was inserted. Checkpointing frequency is determined by its own set of configuration parameters discussed elsewhere in this chapter.

  The default parameter value is 8, which means 8 sets of 4 16MB files for a total of 512MB. In other words, REDO log space must be allocated in blocks of 64MB. In scenarios requiring a great many updates, the value for NoOfFragmentLogFiles may need to be set as high as 300 or even higher in order to provide sufficient space for REDO logs.

  If the checkpointing is slow and there are so many writes to the database that the log files are full and the log tail cannot be cut without jeopardizing recovery, then all updating transactions will be aborted with internal error code 410, or Out of log file space temporarily. This condition will prevail until a checkpoint has completed and the log tail can be moved forward.

• MaxNoOfSavedMessages

  This parameter sets the maximum number of trace files that will be kept before overwriting old ones. Trace files are generated when, for whatever reason, the node crashes.

  The default is 25 trace files.

• MaxNoOfAttributes

  Defines the number of attributes that can be defined in the cluster.

  The default value for this parameter is 1000, with the minimum possible value being 32. There is no maximum. Each attribute consumes around 200 bytes of storage per node due to the fact that all metadata is fully replicated on the servers.

  When setting MaxNoOfAttributes, it is important to prepare in advance for any ALTER TABLE commands that you might wish to perform in future. This is due to the fact, during the execution of ALTER TABLE on a Cluster table, 3 times the number of attributes as in the original table are used. For example, if a table requires 100 attributes, and you wish to be able to alter it later, then you need to set the value of MaxNoOfAttributes to 300. A good rule of thumb is that, if you can create all desired tables without any problems, then add 2 times the number of attributes in the largest table to MaxNoOfAttributes to be sure. You should also verify that this number is sufficient by trying an actual ALTER TABLE after doing so. If this is not successful, increase MaxNoOfAttributes by another multiple of the original value and test it again.

• MaxNoOfTables

  A table object is allocated for each table, unique hash index, and ordered index. This parameter sets the maximum number of table objects for the cluster as a whole.

  For each attribute that has a BLOB data type an extra table is used to store most of the BLOB data. These tables also must be taken into account when defining the total number of tables.

  The default value of this parameter is 128. The minimum is 8 and the maximum is 1600. Each table object consumes approximately 20KB per node.

• MaxNoOfOrderedIndexes

  For each ordered index in the cluster, an object is allocated describing what is being indexed and its storage segments. By default each index so defined also defines an ordered index. Each unique index and primary key has both an ordered index and a hash index.

  The default value of this parameter is 128. Each object consumes approximately 10KB of data per node.

• MaxNoOfUniqueHashIndexes

  For each unique index that is not a primary key, a special table is allocated that maps the unique key to the primary key of the indexed table. By default there an ordered index is also defined for each unique index. To prevent this, you must use the USING HASH option when defining the unique index.

  The default value is 64. Each index consumes approximately 15KB per node.

• MaxNoOfTriggers

  Internal update, insert, and delete triggers are allocated for each unique hash index. (This means that 3 triggers are created for each unique hash index.) However, an ordered index requires only a single trigger object. Backups also use three trigger objects for each normal table in the cluster.

  Note: When replication between clusters is supported, this will also make use of internal triggers.

  This parameter sets the maximum number of trigger objects in the cluster.

  The default value for this parameter is 768.

• MaxNoOfIndexes

  This parameter is deprecated in MySQL 5.1; you should use MaxNoOfOrderedIndexes and MaxNoOfUniqueHashIndexes instead.

  This parameter is used only by unique hash indexes. There needs to be one record in this pool for each unique hash index defined in the cluster.

  The default value of this parameter is 128.

• LockPagesInMainMemory

  For a number of operating systems, including Solaris and Linux, it is possible to lock a process into memory and so avoid any swapping to disk. This can be used to help guarantee the cluster's real-time characteristics.

  By default, this feature is disabled.

• StopOnError

  This parameter specifies whether an NDBD process is to exit or to perform an automatic restart when an error condition is encountered.

  This feature is enabled by default.

• Diskless

  It is possible to specify MySQL Cluster tables as diskless, meaning that tables are not checkpointed to disk and that no logging occurs. Such tables exist only in main memory. A consequence of using diskless tables is that neither the tables nor the records in those tables will be preserved after a crash. However, when operating in diskless mode, it is possible to run ndbd on a diskless computer.

  Important: This feature causes the entire cluster to operate in diskless mode.

  When this feature is enabled, backups are performed but backup data is not actually stored.

  This feature is enabled by setting Diskless to either 1 or Y. Diskless is disabled by default.

• RestartOnErrorInsert

  This feature is accessible only when building the debug version where it is possible to insert errors in the execution of individual blocks of code as part of testing.

  By default, this feature is disabled.

• TimeBetweenWatchDogCheck

  To prevent the main thread from getting stuck in an endless loop at some point, a “watchdog” thread checks the main thread. This parameter specifies the number of milliseconds between checks. If the process remains in the same state after three checks, it is terminated by the watchdog thread.

  This parameter can easily be changed for purposes of experimentation or to adapt to local conditions. It can be specified on a per-node basis although there seems to be little reason for doing so.

  The default timeout is 4000 milliseconds (4 seconds).

• StartPartialTimeout

  This parameter specifies the time that the cluster will wait for all storage nodes to come up before the cluster initialization routine is invoked. This timeout is used to avoid a partial Cluster startup whenever possible.

  The default value is 30000 milliseconds (30 seconds). 0 means eternal timeout; in other words, the cluster may start only if all nodes are available.

• StartPartitionedTimeout

  If the cluster is ready to start after waiting for StartPartialTimeout milliseconds but is still possibly in a partitioned state, the cluster waits until this timeout has also passed.

  The default timeout is 60000 milliseconds (60 seconds).

• StartFailureTimeout

  If a data node has not completed its startup sequence within the time specified by this parameter, then the node startup will fail. Setting this parameter to 0 means that no data node timeout is applied.

  The default value is 60000 milliseconds (60 seconds). For data nodes containing extremely large amounts of data, this parameter should be increased. For example, in the case of a storage node containing several gigabytes of data, a period as long as 10-15 minutes (that is, 600,000 to 1,000,000 milliseconds) might be required in order to to perform a node restart.

• HeartbeatIntervalDbDb

  One of the primary methods of discovering failed nodes is by the use of heartbeats. This parameter states how often heartbeat signals are sent and how often to expect to receive them. After missing three heartbeat intervals in a row, the node is declared dead. Thus the maximum time for discovering a failure through the heartbeat mechanism is four times the heartbeat interval.

  The default heartbeat interval is 1500 milliseconds (1.5 seconds). This parameter must not be changed drastically and should not vary widely between nodes. If one node uses 5000 milliseconds and the node watching it uses 1000 milliseconds then obviously the node will be declared dead very quickly. This parameter can be changed during an online software upgrade but only in small increments.

• HeartbeatIntervalDbApi

  Each data node sends heartbeat signals to each MySQL server (SQL node) to ensure that it remains in contact. If a MySQL server fails to send a heartbeat in time it is declared “dead,” in which case all ongoing transactions are completed and all resources released. The SQL node cannot reconnect until all activities initiated by the previous MySQL instance have been completed. The three-heartbeat criteria for this determination are the same as described for HeartbeatIntervalDbDb.

  The default interval is 1500 milliseconds (1.5 seconds). This interval can vary between individual data nodes because each storage node watches the MySQL servers connected to it, independently of all other data nodes.

• TimeBetweenLocalCheckpoints

  This parameter is an exception in that it does not specify a time to wait before starting a new local checkpoint; rather, it is used to ensure that local checkpoints are not performed in a cluster where relatively few updates are taking place. In most clusters with high update rates, it is likely that a new local checkpoint is started immediately after the previous one has been completed.

  The size of all write operations executed since the start of the previous local checkpoints is added. This parameter is also exceptional in that it is specified as the base-2 logarithm of the number of 4-byte words, so that the default value 20 means 4MB (4 × 2²⁰) of write operations, 21 would mean 8MB, and so on up to a maximum value of 31, which equates to 8GB of write operations.

  All the write operations in the cluster are added together. Setting TimeBetweenLocalCheckpoints to 6 or less means that local checkpoints will be executed continuously without pause, independent of the cluster's workload.

• TimeBetweenGlobalCheckpoints

  When a transaction is committed, it is committed in main memory in all nodes on which the data is mirrored. However, transaction log records are not flushed to disk as part of the commit. The reasoning behind this behavior is that having the transaction safely committed on at least two autonomous host machines should meet reasonable standards for durability.

  It is also important to ensure that even the worst of cases — a complete crash of the cluster — is handled properly. To guarantee that this happens, all transactions taking place within a given interval are put into a global checkpoint, which can be thought of as a set of committed transactions that has been flushed to disk. In other words, as part of the commit process, a transaction is placed in a global checkpoint group; later, this group's log records are flushed to disk, and then the entire group of transactions is safely committed to disk on all computers in the cluster.

  This parameter defines the interval between global checkpoints. The default is 2000 milliseconds.

• TimeBetweenInactiveTransactionAbortCheck

  Timeout handling is performed by checking a timer on each transaction once for every interval specified by this parameter. Thus, if this parameter is set to 1000 milliseconds, then every transaction will be checked for timing out once per second.

  The default value for this parameter is 1000 milliseconds (1 second).

• TransactionInactiveTimeout

  If the transaction is currently not performing any queries but is waiting for further user input, this parameter states the maximum time that the user can wait before the transaction is aborted.

  The default for this parameter is zero (no timeout). For a real-time database that needs to ensure that no transaction keeps locks for too long a time this parameter should be set to a much smaller value. The unit is milliseconds.

• TransactionDeadlockDetectionTimeout

  When a node executes a query involving a transaction, the node waits for the other nodes in the cluster to respond before continuing. A failure to respond can occur for any of the following reasons:

  1. The node is “dead”

  2. The operation has entered a lock queue

  3. The node requested to perform the action could be heavily overloaded.

  This timeout parameter states how long the transaction coordinator will wait for query execution by another node before aborting the transaction, and is important for both node failure handling and deadlock detection. Setting it too high can cause a undesirable behavior in situations involving deadlocks and node failure.

  The default timeout value is 1200 milliseconds (1.2 seconds).

• NoOfDiskPagesToDiskAfterRestartTUP

  When executing a local checkpoint the algorithm flushes all data pages to disk. Merely doing as quickly as possible without any moderation is likely to impose excessive loads on processors, networks, and disks. To control the write speed, this parameter specifies how many pages per 100 milliseconds are to be written. In this context, a “page” is defined as 8KB; thus, this parameter is specified in units of 80KB per second. Therefore, setting NoOfDiskPagesToDiskAfterRestartTUP to a value of 20 entails writing 1.6MB in data pages to disk each second during a local checkpoint. This value includes the writing of UNDO log records for data pages; that is, this parameter handles the limitation of writes from data memory. UNDO log records for index pages are handled by the parameter NoOfDiskPagesToDiskAfterRestartACC. (See the entry for IndexMemory for information about index pages.)

  In short, this parameter specifies how quickly local checkpoints will be executed, and operates in conjunction with NoOfFragmentLogFiles, DataMemory, and IndexMemory.

  The default value is 40 (3.2MB of data pages per second).

• NoOfDiskPagesToDiskAfterRestartACC

  This parameter uses the same units as NoOfDiskPagesToDiskAfterRestartTUP and acts in a similar fashion, but limits the speed of writing index pages from index memory.

  The default value of this parameter is 20 index memory pages per second (1.6MB per second).

• NoOfDiskPagesToDiskDuringRestartTUP

  This parameter in a similar fashion as NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, only it does so with regard to local checkpoints executed in the node when a node is restarting. As part of all node restarts a local checkpoint is always performed. During a node restart it is possible to write to disk at a higher speed than at other times, because fewer activities are being performed in the node.

  This parameter covers pages written from data memory.

  The default value is 40 (3.2MB per second).

• NoOfDiskPagesToDiskDuringRestartACC

  Controls the number of index memory pages that can be written to disk during the local checkpoint phase of a node restart.

  As with NoOfDiskPagesToDiskAfterRestartTUP and NoOfDiskPagesToDiskAfterRestartACC, values for this parameter are expressed in terms of 8KB pages written per 100 milliseconds (80KB/second).

  The default value is 20 (1.6MB per second).

• ArbitrationTimeout

  This parameter specifies the time that the data node will wait for a response from the arbitrator to an arbitration message. If this is exceeded, then it is assumed that the network has split.

  The default value is 1000 milliseconds (1 second).

• UndoIndexBuffer

  This buffer is used during local checkpoints. The NDB storage engine uses a recovery scheme based on checkpoint consistency in conjunction with an operational REDO log. In order to produce a consistent checkpoint without blocking the entire system for writes, UNDO logging is done while performing the local checkpoint. UNDO logging is activated on a single table fragment at a time. This optimization is possible because tables are stored entirely in main memory.

  The UNDO index buffer is used for the updates on the primary key hash index. Inserts and deletes rearrange the hash index; the NDB storage engine writes UNDO log records that map all physical changes to an index page so that they can be undone at system restart. It also logs all active insert operations for each fragment at the start of a local checkpoint.

  Reads and updates set lock bits and update a header in the hash index entry. These changes are handled by the page-writing algorithm to ensure that these operations need no UNDO logging.

  This buffer is 2MB by default. The minimum value is 1MB, and for most applications the minimum is sufficient. For applications doing extremely large or numerous inserts and deletes together with large transactions and large primary keys, it may be necessary to increase the size of this buffer. If this buffer is too small, the NDB storage engine issues internal error code 677 Index UNDO buffers overloaded.

• UndoDataBuffer

  The UNDO data buffer plays the same role as the UNDO index buffer, except that it is used with regard to data memory rather than index memory. This buffer is used during the local checkpoint phase of a fragment for inserts, deletes, and updates.

  Because UNDO log entries tend to grow larger as more operations are logged, this buffer is also larger than its index memory counterpart, with a default value of 16MB.

  For some applications this amount of memory may be unnecessarily large. In such cases it is possible to decrease this size down to a minimum of 1MB.

  It is rarely necessary to increase the size of this buffer. If there is such a need, then it is a good idea to check whether the disks can actually handle the load caused by database update activity. A lack of sufficient disk space cannot be overcome by increasing the size of this buffer.

  If this buffer is too small and gets congested, the NDB storage engine issues internal error code 891 Data UNDO buffers overloaded.

• RedoBuffer

  All update activities also need to be logged. This log makes it possible to replay these updates whenever the system is restarted. The NDB recovery algorithm uses a "fuzzy" checkpoint of the data together with the UNDO log, and then applies the REDO log to play back all changes up to the restoration point.

  This buffer is 8MB by default. The minimum value is 1MB.

  If this buffer is too small, the NDB storage engine issues error code 1221 REDO log buffers overloaded.

• LogLevelStartup

  The reporting level for events generated during startup of the process.

  The default level is 1.

• LogLevelShutdown

  The reporting level for events generated as part of graceful shutdown of a node.

  The default level is 0.

• LogLevelStatistic

  The reporting level for statistical events such as number of primary key reads, number of updates, number of inserts, information relating to buffer usage, and so on.

  The default level is 0.

• LogLevelCheckpoint

  The reporting level for events generated by local and global checkpoints.

  The default level is 0.

• LogLevelNodeRestart

  The reporting level for events generated during node restart.

  The default level is 0.

• LogLevelConnection

  The reporting level for events generated by connections between cluster nodes.

  The default level is 0.

• LogLevelError

  The reporting level for events generated by errors and warnings by the cluster as a whole. These errors do not cause any node failure but are still considered worth reporting.

  The default level is 0.

• LogLevelInfo

  The reporting level for events generated for information about the general state of the cluster.

  The default level is 0.

• BackupDataBufferSize

  In creating a backup, there are two buffers used for sending data to the disk. The backup data buffer is used to fill in data recorded by scanning a node's tables. Once this buffer has been filled to the level specified as BackupWriteSize (see below), the pages are sent to disk. While flushing data to disk, the backup process can continue filling this buffer until it runs out of space. When this happens, the backup process pauses the scan and waits until some disk writes have completed and have thus freed up memory so that scanning may continue.

  The default value for this parameter is 2MB.

• BackupLogBufferSize

  The backup log buffer fulfills a role similar to that played by the backup data buffer, except that it used for generating a log of all table writes made during execution of the backup. The same principles apply for writing these pages as with the backup data buffer, except that when there is no more space in the backup log buffer, the backup fails. For that reason, the size of the back up logbuffer must be big enough to handle the load caused by write activities while the backup is being made.

  The default value for this parameter should be sufficient for most applications. In fact, it is more likely for a backup failure to be caused by insufficient disk write speed than it is for the backup log buffer to become full. If the disk subsystem is not configured for the write load caused by applications, the cluster will be likely to be able to perform the desired operations.

  It is preferable to configure cluster nodes in such a manner that the processor becomes the bottleneck rather than the disks or the network connections.

  The default value is 2MB.

• BackupMemory

  This parameter is simply the sum of BackupDataBufferSize and BackupLogBufferSize.

  The default value is 2MB + 2MB = 4MB.

• BackupWriteSize

  This parameter specifies the size of messages written to disk by the backup log and backup data buffers.

  The default value is 32KB.

• Id

  This value is used as the address of the node by all cluster internal messages, and must be an integer between 1 and 63. Each cluster node must have a unique identity within the cluster.

• ExecuteOnComputer

  This refers to one of the computers (hosts) defined in a [COMPUTER] section of the configuration file.

• ArbitrationRank

  This parameter is used to define which nodes can act as arbitrators. Both MGM nodes and SQL nodes can be arbitrators. A value of 0 means that the given node will never be used as an arbitrator, a value of 1 gives the node high priority as an arbitrator, and a value of 2 gives it low priority. A normal configuration uses the management server as arbitrator, setting its ArbitrationRank to 1 (the default) and those for all SQL nodes to 0.

• ArbitrationDelay

  Setting this parameter to any other value than 0 (the default) means that responses by the arbitrator to arbitration requests will be delayed by the stated number of milliseconds. It is usually not necessary to change this value.

• BatchByteSize

  For queries that are translated into full table scans or range scans on indexes, it is important for best performance to fetch records in properly sized batches. It is possible to set the proper size both in terms of number of records (BatchSize) and in terms of bytes (BatchByteSize). The actual batch size will be limited by both parameters.

  The speed at which queries are performed can vary by more than 40% depending upon how this parameter is set. In future releases, MySQL Server will make educated guesses on how to set parameters relating to batch size, based on the query type.

  This parameter is measured in bytes and by default is equal to 32KB.

• BatchSize

  This parameter is measured in number of records and is by default set to 64. The maximum size is 992.

• MaxScanBatchSize

  The batch size is the size of each batch sent from each data node. Most scans are performed in parallel in order to protect the MySQL Server from receiving too much data from many nodes in parallel; this parameter sets a limit to the total batch size over all nodes.

  The default value of this parameter is set to 256KB. Its maximum size is 16MB.

• NodeId1, NodeId2

  To identify a connection between two nodes it is necessary to provide the node identities for both of them in the [TCP] section of the configuration file.

• SendBufferMemory

  TCP transporters use a buffer to store all messages before performing the send call to the operating system. When this buffer reaches 64KB its contents are sent; these are also sent when a round of messages have been executed. To handle temporary overload situations it is also possible to define a bigger send buffer. The default size of the send buffer is 256KB.

• SendSignalId

  In order to be able to retrace a distributed message diagram it is necessary to identify each message. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default.

• Checksum

  This parameter is also a boolean (Y/N or 1/0) parameter, and is disabled by default. When it is enabled, checksums for all messages are calculated before they placed in the send buffer. This feature ensures that messages are not corrupted while waiting in the send buffer, or by the transport mechanism.

• PortNumber (OBSOLETE)

  This formerly specified the port number to be used for listening for connections from other nodes. This parameter should no longer be used.

• [TCP]ReceiveBufferMemory

  Specifies the size of the buffer used when receiving data from the TCP/IP socket. There is seldom any need to change this parameter from its default value of 64KB, except possibly in order to save memory.

• NodeId1, NodeId2

  To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.

• ShmKey

  When setting up shared memory segments, a node ID is used to identify uniquely the shared memory segment to use for the communication. This is expressed as an integer which does not have a default value.

• ShmSize

  Each SHM connection has a shared memory segment where messages between nodes are placed by the sender and read by the reader. The size of this segment is defined by ShmSize. The default value is 1MB.

• SendSignalId

  In order to retrace the path of a distributed message, it is necessary to provide each message with a unique identifier. Setting this parameter to Y causes these message IDs to be transported over the network as well. This feature is disabled by default.

• Checksum

  This parameter is also a Y/N parameter which is disabled by default. When it is enabled, checksums for all messages are calculated before being placed in the send buffer.

  This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

• NodeId1, NodeId2

  To identify a connection between two nodes it is necessary to provide node identifiers for each of them, as NodeId1 and NodeId2.

• Host1SciId0

  This identifies the SCI node ID on the first Cluster node (identified by NodeId1).

• Host1SciId1

  It is possible to set up SCI Transporters for failover between two SCI cards which then should use separate networks between the nodes. This identifies the node ID and the second SCI card to be used on the first node.

• Host2SciId0

  This identifies the SCI node ID on the second Cluster node (identified by NodeId2).

• Host2SciId1

  When using two SCI cards to provide failover, this parameter identifies the second SCI card to be used on the second node.

• SharedBufferSize

  Each SCI transporter has a shared memory segment used for communication between the two nodes. Setting the size of this segment to the default value of 1 MB should be sufficient for most applications. Using a smaller value can lead to problems when performing many parallel inserts; if the shared buffer is too small, this can also result in a crash of the ndbd process.

• SendLimit

  A small buffer in front of the SCI media stores messages before transmitting them over the SCI network. By default this is set to 8kB. Our benchmarks show that performance is best at 64 kB but 16kB reaches within a few percent of this, and there was little if any advantage to increasing it beyond 8kB.

• SendSignalId

  In order to trace a distributed message it is necessary to identify each message uniquely. When this parameter is set to Y, message IDs are transported over the network. This feature is disabled by default.

• Checksum

  This parameter is also a boolean value, and is disabled by default. When Checksum is enabled, checksums are calculated for all messages before they are placed in the send buffer. This feature prevents messages from being corrupted while waiting in the send buffer. It also serves as a check against data being corrupted during transport.

• --ndb-connectstring=connect_string

  When using the NDB storage engine, it is possible through setting this option to specify the management server that distributes cluster configuration data.

• --ndbcluster

  If the binary includes support for the NDB Cluster storage engine, the default disabling of the NDB Cluster storage engine (also referred to more simply as the NDB storage engine) can be overridden by using this option. The NDB Cluster storage engine is necessary for using MySQL Cluster.

• --skip-ndbcluster

  Disables the NDB Cluster storage engine. This option is activated by default for binaries where it is included; in other words, the NDB Cluster storage engine is disabled until activated via --ndbcluster. This option applies only if the server was compiled with support for the NDB Cluster storage engine.

• --daemon, -d

  Instructs ndbd to execute as a daemon process (the default behavior). --nodaemon can be used to not start the process as a daemon.

• --initial

  Instructs ndbd to perform an initial start. An initial start erases any files created for recovery purposes by earlier instances of ndbd. It also re-creates recovery log files. Note that on some operating systems this process can take a substantial amount of time.

  An --initial start is to be used only the very first time that the ndbd process is started, as it removes all files from the Cluster filesystem and re-creates all REDO log files. The exceptions to this rule are:

  • When performing a software upgrade which has changed the contents of any files.

  • When restarting the node with a new version of ndbd.

  • As a measure of last resort when for some reason the node restart or system restart repeatedly fails. In this case be aware that this node can no longer be used to restore data due to the destruction of the datafiles.

  This option does not affect any backup files that have already been created by the affected node.

• --nodaemon

  Instructs ndbd not to start as a daemon process. Useful when ndbd is being debugged and you want output to be redirected to the screen.

• --nostart

  Instructs ndbd not to start automatically. When this option is used, ndbd connects to the management server, obtains configuration data from it, and initializes communication objects. However, it will not actually start the execution engine until specifically requested to do so by the management server. This can be accomplished by issuing the proper command to the management client.

• --config-file=file_name, -f file_name,

  Instructs the management server as to which file it should use for its configuration file. This option must be specified. The file name defaults to config.ini.

  Note: This option also can be given as -c file_name, but this shortcut is obsolete and should not be used in new installations.

• --daemon, -d

  Instructs ndb_mgmd to start as a daemon. This is the default behavior.

• --nodaemon

  Instructs the management server not to start as a daemon.

• [host_name [port_num]]

  To start the management client it is necessary to specify where the management server resides, which means specifying the hostname and the port. The default hostname is localhost; the default port is 1186.

• --try-reconnect=number

  If the connection to the management server is broken, then the node tries to reconnect to it every 5 seconds until it succeeds. By using this option, it is possible to limit the number of attempts to number before giving up and reporting an error instead.

• CLUSTERLOG ON

  Turn cluster log on.

• CLUSTERLOG OFF

  Turn cluster log off.

• CLUSTERLOG INFO

  Information about cluster log settings.

• node_id CLUSTERLOG category=threshold

  Logs category events with priority less than or equal to threshold in the cluster log.

• CLUSTERLOG FILTER severity_level

  Toggles cluster logging of events of the specified severity_level.

• Metadata: the names and definitions of all database tables.

• Table records: the data actually stored in the database tables at the time that the backup was made

• Transaction log: a sequential record telling how and when data was stored in the database

• BACKUP-backup_id.node_id.ctl

  A control file containing control information and metadata. Each node saves the same table definitions (for all tables in the cluster) to its own version of this file.

• BACKUP-backup_id-0.node_id.data

  A data file containing the table records, which are saved on a per-fragment basis; that is, different nodes save different fragments during the backup. The file saved by each node starts with a header that states to which tables the records belong. Following the list of records there is a footer containing a checksum for all records.

• BACKUP-backup_id.node_id.log

  A log file containing records of committed transactions. Only transactions on tables stored in the backup are stored in the log. Nodes involved in the backup save different records because different nodes host different database fragments.

• BackupDataBufferSize

  Amount of memory used to buffer data before it is written to disk.

• BackupLogBufferSize

  Amount of memory used to buffer log records before these are written to disk.

• BackupMemory

  Total memory allocated in a database node for backups. This should be the sum of the memory allocated for the backup data buffer and the backup log buffer.

• BackupWriteSize

  Size of blocks written to disk. This applies for both the backup data buffer and the backup log buffer.