Cannot start mysql or mysqld_safe on ubuntu

My Ubuntu runs in docker and I need to use Percona, however, the mysqld cannot be started. I have been trying to fix this for 2 days. I don’t think it’s a permission issue, but I am not sure if any of my configuration is incorrect. The following is my Ubuntu version, and it runs inside docker.

3.13.0-36-generic #63-Ubuntu SMP Wed Sep 3 21:30:07 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

  • Run VirtualBox in Docker to achieve running Genymotion in Docker
  • Is it a docker best practice to use volume for the code?
  • polkit-gnome-authentication-agent-1 fail to start in docker without privileged flag
  • Is it possible to log into Gitlabs container registry without using the CI runner?
  • enable scripting within docker image
  • Docker for Mac: Using Persistent Storage
  • my.cnf

    [client]
    port            = 3306
    socket          = /var/run/mysqld/mysqld.sock
    
    [mysqld_safe]
    pid-file        = /var/run/mysqld/mysqld.pid
    socket          = /var/run/mysqld/mysqld.sock
    nice            = 0
    
    [mysqld]
    user            = mysql
    pid-file        = /var/run/mysqld/mysqld.pid
    socket          = /var/run/mysqld/mysqld.sock
    port            = 3306
    basedir         = /usr
    datadir         = /var/lib/mysql
    tmpdir          = /tmp
    lc-messages-dir = /usr/share/mysql
    explicit_defaults_for_timestamp
    
    # Instead of skip-networking the default is now to listen only on
    # localhost which is more compatible and is not less secure.
    bind-address    = 127.0.0.1
    
    log-error       = /var/log/mysql/error.log
    
    # Recommended in standard MySQL setup
    sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES
    
    # Disabling symbolic-links is recommended to prevent assorted security risks
    symbolic-links=0
    
    # * IMPORTANT: Additional settings that can override those from this file!
    #   The files must end with '.cnf', otherwise they'll be ignored.
    #
    !includedir /etc/mysql/conf.d/
    

    This is the initial error log when mysql tries to init itself.

    /var/log/error.log

    2014-10-17 15:43:27 53 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-10-17 15:43:27 53 [Note] WSREP: wsrep_load(): loading provider library 'none'
    2014-10-17 15:43:27 7fe0e0f2e780 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
    2014-10-17 15:43:27 7fe0e0f2e780 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
    2014-10-17 15:43:27 53 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-10-17 15:43:27 53 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-10-17 15:43:27 53 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-10-17 15:43:27 53 [Note] InnoDB: Memory barrier is not used
    2014-10-17 15:43:27 53 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2014-10-17 15:43:27 53 [Note] InnoDB: Using Linux native AIO
    2014-10-17 15:43:27 53 [Note] InnoDB: Using CPU crc32 instructions
    2014-10-17 15:43:27 53 [Note] InnoDB: Initializing buffer pool, size = 64.0M
    2014-10-17 15:43:27 53 [Note] InnoDB: Completed initialization of buffer pool
    2014-10-17 15:43:27 53 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
    2014-10-17 15:43:27 53 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
    2014-10-17 15:43:27 53 [Note] InnoDB: Database physically writes the file full: wait...
    2014-10-17 15:43:28 53 [Note] InnoDB: Setting log file ./ib_logfile101 size to 128 MB
    InnoDB: Progress in MB: 100
    2014-10-17 15:43:34 53 [Note] InnoDB: Setting log file ./ib_logfile1 size to 128 MB
    InnoDB: Progress in MB: 100
    2014-10-17 15:43:43 53 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
    2014-10-17 15:43:43 53 [Warning] InnoDB: New log files created, LSN=45781
    2014-10-17 15:43:43 53 [Note] InnoDB: Doublewrite buffer not found: creating new
    2014-10-17 15:43:43 53 [Note] InnoDB: Doublewrite buffer created
    2014-10-17 15:43:43 53 [Note] InnoDB: 128 rollback segment(s) are active.
    2014-10-17 15:43:43 53 [Warning] InnoDB: Creating foreign key constraint system tables.
    2014-10-17 15:43:43 53 [Note] InnoDB: Foreign key constraint system tables created
    2014-10-17 15:43:43 53 [Note] InnoDB: Creating tablespace and datafile system tables.
    2014-10-17 15:43:43 53 [Note] InnoDB: Tablespace and datafile system tables created.
    2014-10-17 15:43:43 53 [Note] InnoDB: Waiting for purge to start
    2014-10-17 15:43:43 53 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 0
    2014-10-17 15:43:43 7fe0c246e700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
    2014-10-17 15:43:43 7fe0c246e700 InnoDB: Cannot open './/ib_buffer_pool' for reading: No such file or directory
    2014-10-17 15:43:43 53 [Warning] Bootstrap mode disables GTIDs. Bootstrap mode should only be used by mysql_install_db which initializes the MySQL data directory and creates system tables.
    2014-10-17 15:43:43 53 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
    2014-10-17 15:43:43 53 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
    2014-10-17 15:43:44 53 [Note] WSREP: Service disconnected.
    2014-10-17 15:43:45 53 [Note] WSREP: Some threads may fail to exit.
    2014-10-17 15:43:45 53 [Note] Binlog end
    2014-10-17 15:43:45 53 [Note] InnoDB: FTS optimize thread exiting.
    2014-10-17 15:43:45 53 [Note] InnoDB: Starting shutdown...
    2014-10-17 15:43:45 7fe0c246e700 InnoDB: Dumping buffer pool(s) to .//ib_buffer_pool
    2014-10-17 15:43:45 7fe0c246e700 InnoDB: Buffer pool(s) dump completed at 141017 15:43:45
    2014-10-17 15:43:47 53 [Note] InnoDB: Shutdown completed; log sequence number 1625977
    2014-10-17 15:43:47 221 [Note] WSREP: Read nil XID from storage engines, skipping position init
    2014-10-17 15:43:47 221 [Note] WSREP: wsrep_load(): loading provider library 'none'
    2014-10-17 15:43:47 7fc35fa5c780 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
    2014-10-17 15:43:47 7fc35fa5c780 InnoDB: Warning: Using innodb_locks_unsafe_for_binlog is DEPRECATED. This option may be removed in future releases. Please use READ COMMITTED transaction isolation level instead, see http://dev.mysql.com/doc/refman/5.6/en/set-transaction.html.
    2014-10-17 15:43:47 221 [Note] InnoDB: Using atomics to ref count buffer pool pages
    2014-10-17 15:43:47 221 [Note] InnoDB: The InnoDB memory heap is disabled
    2014-10-17 15:43:47 221 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
    2014-10-17 15:43:47 221 [Note] InnoDB: Memory barrier is not used
    2014-10-17 15:43:47 221 [Note] InnoDB: Compressed tables use zlib 1.2.8
    2014-10-17 15:43:47 221 [Note] InnoDB: Using Linux native AIO
    2014-10-17 15:43:47 221 [Note] InnoDB: Using CPU crc32 instructions
    2014-10-17 15:43:47 221 [Note] InnoDB: Initializing buffer pool, size = 64.0M
    2014-10-17 15:43:47 221 [Note] InnoDB: Completed initialization of buffer pool
    2014-10-17 15:43:47 221 [Note] InnoDB: Highest supported file format is Barracuda.
    2014-10-17 15:43:47 221 [Note] InnoDB: 128 rollback segment(s) are active.
    2014-10-17 15:43:47 221 [Note] InnoDB: Waiting for purge to start
    2014-10-17 15:43:47 221 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 1625977
    2014-10-17 15:43:47 7fc340fd0700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
    2014-10-17 15:43:47 221 [Warning] Bootstrap mode disables GTIDs. Bootstrap mode should only be used by mysql_install_db which initializes the MySQL data directory and creates system tables.
    2014-10-17 15:43:47 221 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
    2014-10-17 15:43:47 221 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
    2014-10-17 15:43:47 7fc340fd0700 InnoDB: Buffer pool(s) load completed at 141017 15:43:47
    2014-10-17 15:43:47 221 [Note] WSREP: Service disconnected.
    2014-10-17 15:43:48 221 [Note] WSREP: Some threads may fail to exit.
    2014-10-17 15:43:48 221 [Note] Binlog end
    2014-10-17 15:43:48 221 [Note] InnoDB: FTS optimize thread exiting.
    2014-10-17 15:43:48 221 [Note] InnoDB: Starting shutdown...
    2014-10-17 15:43:48 7fc340fd0700 InnoDB: Dumping buffer pool(s) to .//ib_buffer_pool
    2014-10-17 15:43:48 7fc340fd0700 InnoDB: Buffer pool(s) dump completed at 141017 15:43:48
    2014-10-17 15:43:50 221 [Note] InnoDB: Shutdown completed; log sequence number 1625987
    

    directory permission

    /var/log/mysql

    [16:35][cwhsu@cwhsu:/var/log/mysql ]$ ls -al
    total 16
    drwxr-xr-x 2 mysql mysql 4096 Oct 17 16:35 .
    drwxr-xr-x 5 root  root  4096 Oct 16 19:26 ..
    -rw-rw---- 1 mysql mysql 6927 Oct 17 15:43 error.log
    [16:35][cwhsu@cwhsu:/var/log/mysql ]$ cd ..
    [16:35][cwhsu@cwhsu:/var/log ]$ ls -al
    total 28
    drwxr-xr-x  5 root     root      4096 Oct 16 19:26 .
    drwxr-xr-x 41 root     root      4096 Oct 17 16:01 ..
    drwxr-xr-x  2 mysql    mysql     4096 Oct 17 16:35 mysql
    drwxr-xr-x  2 www-data www-data  4096 Oct 16 19:26 nginx
    -rw-------  1 root     root      5499 Oct 17 15:43 php5-fpm.log
    -rw-------  1 root     root         0 Oct 16 19:26 php-fpm-slow.log
    drwxr-xr-x  2 cwhsu    developer 4096 Oct 17 15:43 supervisor
    

    /var/run/mysqld/

    [16:41][cwhsu@cwhsu:/var/run ]$ ls  /var/run/mysqld/ -ld
    drwxr-xr-x 2 mysql root 4096 Oct 16 20:15 /var/run/mysqld/
    [16:42][cwhsu@cwhsu:/var/run ]$ ls  /var/run/mysqld/* -ld
    -rw-rw---- 1 mysql mysql 5 Oct 16 20:15 /var/run/mysqld/mysqld.pid
    -rwxrwxrwx 1 mysql mysql 0 Oct 16 20:15 /var/run/mysqld/mysqld.soc
    

    This is the initial condition. Then I try to do service mysql start.

    [16:35][cwhsu@cwhsu:/var/log ]$ service mysql start
     * Starting MySQL (Percona XtraDB Cluster) database server mysqld
     * Please take a look at the syslog.
                                                                              [fail]
    

    So I try to user mysqld_safe to find out what is going wrong.

    [16:38][cwhsu@cwhsu:/var/log ]$ mysqld_safe 
    141017 16:40:56 mysqld_safe Logging to '/var/log/mysql/error.log'.
    cat: /var/run/mysqld/mysqld.pid: Permission denied
    /bin/rm: cannot remove ‘/var/run/mysqld/mysqld.pid’: Permission denied
    141017 16:40:56 mysqld_safe Fatal error: Can't remove the pid file:
    /var/run/mysqld/mysqld.pid
    Please remove it manually and start /usr/bin/mysqld_safe again;
    mysqld daemon not started
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    [16:40][cwhsu@cwhsu:/var/log ]$
    

    Now, I have already tried this many times, so I just sudo chmod 777 /var/run/mysql/mysql.pid to simplify the process.

    [16:44][cwhsu@cwhsu:/var/run ]$ sudo chmod 777 /var/run/mysqld/mysqld.pid 
    [16:46][cwhsu@cwhsu:/var/run ]$ mysqld_safe 
    141017 16:46:28 mysqld_safe Logging to '/var/log/mysql/error.log'.
    /bin/rm: cannot remove ‘/var/run/mysqld/mysqld.pid’: Permission denied
    141017 16:46:28 mysqld_safe Fatal error: Can't remove the pid file:
    /var/run/mysqld/mysqld.pid
    Please remove it manually and start /usr/bin/mysqld_safe again;
    mysqld daemon not started
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    

    If I delete the mysqld.pid and try again.

    [16:47][cwhsu@cwhsu:/var/run ]$ sudo rm -rf /var/run/mysqld/mysqld.pid 
    [16:48][cwhsu@cwhsu:/var/run ]$ mysqld_safe 
    141017 16:49:02 mysqld_safe Logging to '/var/log/mysql/error.log'.
    141017 16:49:02 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    /bin/rm: cannot remove ‘/var/run/mysqld/mysqld.sock’: Permission denied
    141017 16:49:02 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
    141017 16:49:02 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to 
    wsrep_start_position
    141017 16:49:03 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
    

    Now I stuck right here and don’t know what to do. Does anyone have an issue about what’s going wrong?


    EDIT

    I try the solution here https://bugs.mysql.com/bug.php?id=73952.
    I modify the my.cnf and add

    #innodb_buffer_pool_load_at_startup=ON
    innodb_buffer_pool_dump_at_shutdown=ON
    

    then run my start script

    #!/bin/sh
    
    test -x /usr/bin/mysql_install_db && test ! -d /var/lib/mysql/mysql && /usr/bin/mysql_install_db
    mkdir -p /var/log/mysql && chown mysql:mysql /var/log/mysql && chmod 755 /var/log/mysql
    mkdir -p /var/log/nginx && chown www-data:www-data /var/log/nginx && chmod 755 /var/log/nginx
    

    modify my.cnf to

    innodb_buffer_pool_load_at_startup=ON
    innodb_buffer_pool_dump_at_shutdown=ON
    

    run the script again and no luck…

    The mysql daemon still cannot be started and if I try mysqld_safe it shows

    [17:35][cwhsu@cwhsu:/var/lib/mysql ]$ mysqld_safe 
    141017 17:35:34 mysqld_safe Logging to '/var/log/mysql/error.log'.
    141017 17:35:34 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    /bin/rm: cannot remove ‘/var/run/mysqld/mysqld.sock’: Permission denied
    141017 17:35:34 mysqld_safe Skipping wsrep-recover for empty datadir: /var/lib/mysql
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    141017 17:35:34 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    /usr/bin/mysqld_safe: 1: eval: cannot create /var/log/mysql/error.log: Permission denied
    141017 17:35:34 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
    /usr/bin/mysqld_safe: 137: /usr/bin/mysqld_safe: cannot create /var/log/mysql/error.log: Permission denied
    

  • Docker Daemon stop - Timeout for container defaults 10s
  • MeteorD container starting failed
  • How to know the file from command “docker change” is regular file or directory
  • Problems with docker container to container communication on same host when firewalld/iptables is enabled
  • Is it possible to pause a Docker image build?
  • The “--cluster-store” and “--cluster-advertise” don't work
  • Docker will be the best open platform for developers and sysadmins to build, ship, and run distributed applications.