I have an installed Gitlab and on one occasion I turned it off. On the other day when I want to turn it on back it is said that it cannot be accessed.
I checked Gitlab’s log file by running the following command :
gitlab-ctl tail
[root@hostname ~]# gitlab-ctl tail ==> /var/log/gitlab/gitlab-rails/production.log <== ==> /var/log/gitlab/gitlab-rails/application.log <== ==> /var/log/gitlab/redis/state <== ==> /var/log/gitlab/redis/current <== 2016-08-23_07:05:26.06194 [1360] 23 Aug 14:05:26.060 * 10 changes in 300 seconds. Saving... 2016-08-23_07:05:26.06198 [1360] 23 Aug 14:05:26.061 * Background saving started by pid 10152 2016-08-23_07:05:26.06525 [10152] 23 Aug 14:05:26.065 * DB saved on disk 2016-08-23_07:05:26.06549 [10152] 23 Aug 14:05:26.065 * RDB: 4 MB of memory used by copy-on-write 2016-08-23_07:05:26.16135 [1360] 23 Aug 14:05:26.161 * Background saving terminated with success 2016-08-23_07:10:27.03220 [1360] 23 Aug 14:10:27.031 * 10 changes in 300 seconds. Saving... 2016-08-23_07:10:27.03223 [1360] 23 Aug 14:10:27.032 * Background saving started by pid 10660 2016-08-23_07:10:27.09826 [10660] 23 Aug 14:10:27.098 * DB saved on disk 2016-08-23_07:10:27.09859 [10660] 23 Aug 14:10:27.098 * RDB: 4 MB of memory used by copy-on-write 2016-08-23_07:10:27.13244 [1360] 23 Aug 14:10:27.132 * Background saving terminated with success ==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-08-23T14:11:48.095333 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) E, [2016-08-23T14:11:48.095468 #10793] ERROR -- : retrying in 0.5 seconds (4 tries left) E, [2016-08-23T14:11:48.595804 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) E, [2016-08-23T14:11:48.595915 #10793] ERROR -- : retrying in 0.5 seconds (3 tries left) E, [2016-08-23T14:11:49.096240 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) E, [2016-08-23T14:11:49.096354 #10793] ERROR -- : retrying in 0.5 seconds (2 tries left) E, [2016-08-23T14:11:49.596765 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) E, [2016-08-23T14:11:49.596875 #10793] ERROR -- : retrying in 0.5 seconds (1 tries left) E, [2016-08-23T14:11:50.097139 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) E, [2016-08-23T14:11:50.097241 #10793] ERROR -- : retrying in 0.5 seconds (0 tries left)
As we can see from the above output log which is generated under the log named unicorn_stderr.log, while Gitlab is trying to add a listener in 127.0.0.1:8080, the process has failed since it is in use. It is shown in the following line of log :
E, [2016-08-23T14:11:49.096240 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use)
So, in order for Gitlab to be activated once more, we have to find the culprit which is occupying port 8080 and use it to provide another service. Another log file which is implying the same error is shown in a log file named gitlab_error.log shown below :
==> /var/log/gitlab/nginx/gitlab_error.log <==
2016/08/23 12:54:02 [error] 1392#0: *16428 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:04:02 [error] 1392#0: *16431 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:14:02 [error] 1392#0: *16434 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:24:02 [error] 1392#0: *16436 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:34:02 [error] 1392#0: *16439 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:44:02 [error] 1392#0: *16442 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 13:54:02 [error] 1392#0: *16444 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 14:04:02 [error] 1392#0: *16447 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.0", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/" 2016/08/23 14:06:28 [error] 1392#0: *16449 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/", host: "xxx.xxx.xxx.xxx" 2016/08/23 14:11:14 [error] 1392#0: *16452 connect() to unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: xxx.xxx.xxx.xxx, request: "GET / HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket:/", host: "xxx.xxx.xxx.xxx"
As shown above, the connection which is being made is refused. Another information which is taken from unicorn_stderr.log which is also about the listening address of Gitlab service is shown below :
==> /var/log/gitlab/unicorn/unicorn_stderr.log <== E, [2016-08-23T14:11:50.597550 #10793] ERROR -- : adding listener failed addr=127.0.0.1:8080 (in use) /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:185:in `bind': Address already in use - bind(2) for 127.0.0.1:8080 (Errno::EADDRINUSE) from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:185:in `new_tcp_server' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/socket_helper.rb:165:in `bind_listen' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:242:in `listen' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `block in bind_new_listeners!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `each' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:809:in `bind_new_listeners!' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/lib/unicorn/http_server.rb:138:in `start' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/unicorn-4.8.3/bin/unicorn:126:in `<top (required)>' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `load' from /opt/gitlab/embedded/service/gem/ruby/2.1.0/bin/unicorn:23:in `<main>' ==> /var/log/gitlab/unicorn/current <== 2016-08-23_07:11:50.69618 master failed to start, check stderr log for details 2016-08-23_07:11:51.70439 failed to start a new unicorn master 2016-08-23_07:11:51.71238 starting new unicorn master ==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
To find the service, I just try to guess on certain service which is normally using port 8080. Below I am searching process named ‘tomcat’. Normally Apache Tomcat Webserver used port 8080 to process request :
[root@hostname ~]# ps aux | grep tomcat java 2885 0.1 8.4 1996072 333056 ? Sl May17 152:14 /usr/bin/java -Djava.util.logging.config.file=/home/java/tomcat/apache-tomcat-7.0.29/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/java/tomcat/apache-tomcat-7.0.29/endorsed -classpath /home/java/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/home/java/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar -Dcatalina.base=/home/java/tomcat/apache-tomcat-7.0.29 -Dcatalina.home=/home/java/tomcat/apache-tomcat-7.0.29 -Djava.io.tmpdir=/home/java/tomcat/apache-tomcat-7.0.29/temp org.apache.catalina.startup.Bootstrap start root 10905 0.0 0.0 103312 864 pts/0 S+ 14:12 0:00 grep tomcat [root@hostname ~]#
Shutdown the Apache Tomcat Webserver to prove that it is actually blocking Gitlab to perform :
[root@hostname bin]# ./catalina.sh stop Using CATALINA_BASE: /home/java/tomcat/apache-tomcat-7.0.29 Using CATALINA_HOME: /home/java/tomcat/apache-tomcat-7.0.29 Using CATALINA_TMPDIR: /home/java/tomcat/apache-tomcat-7.0.29/temp Using JRE_HOME: /usr Using CLASSPATH: /home/java/tomcat/apache-tomcat-7.0.29/bin/bootstrap.jar:/home/java/tomcat/apache-tomcat-7.0.29/bin/tomcat-juli.jar [root@hostname bin]#
After shutting down Apache Tomcat webserver, try to restart Gitlab service as shown below :
[root@hostname ]# gitlab-ctl restart
ok: run: gitlab-workhorse: (pid 11550) 0s
ok: run: logrotate: (pid 11558) 0s
ok: run: nginx: (pid 11564) 1s
ok: run: postgresql: (pid 11569) 0s
ok: run: redis: (pid 11578) 1s
ok: run: sidekiq: (pid 11583) 0s
ok: run: unicorn: (pid 11590) 0s
[root@VDC07035 bin]# gitlab-ctl tail
==> /var/log/gitlab/gitlab-rails/production.log <==
==> /var/log/gitlab/gitlab-rails/application.log <==
==> /var/log/gitlab/redis/state <==
==> /var/log/gitlab/redis/current <==
2016-08-23_07:16:21.79560 `-._ `-._`-.__.-'_.-' _.-'
2016-08-23_07:16:21.79560 `-._ `-.__.-' _.-'
2016-08-23_07:16:21.79561 `-._ _.-'
2016-08-23_07:16:21.79561 `-.__.-'
2016-08-23_07:16:21.79561
2016-08-23_07:16:21.79569 [11578] 23 Aug 14:16:21.795 # Server started, Redis version 2.8.21
2016-08-23_07:16:21.79615 [11578] 23 Aug 14:16:21.796 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2016-08-23_07:16:21.79655 [11578] 23 Aug 14:16:21.796 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2016-08-23_07:16:21.79719 [11578] 23 Aug 14:16:21.797 * DB loaded from disk: 0.001 seconds
2016-08-23_07:16:21.79721 [11578] 23 Aug 14:16:21.797 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2016-08-23T14:14:19.171253 #11029] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=16
I, [2016-08-23T14:14:19.179651 #11029] INFO -- : master process ready
I, [2016-08-23T14:14:19.189902 #11063] INFO -- : worker=0 spawned pid=11063
I, [2016-08-23T14:14:19.190342 #11063] INFO -- : worker=0 ready
I, [2016-08-23T14:14:19.194092 #11065] INFO -- : worker=1 spawned pid=11065
I, [2016-08-23T14:14:19.194436 #11065] INFO -- : worker=1 ready
I, [2016-08-23T14:16:22.674415 #11029] INFO -- : reaped #<Process::Status: pid 11063 exit 0> worker=0
I, [2016-08-23T14:16:22.674602 #11029] INFO -- : reaped #<Process::Status: pid 11065 exit 0> worker=1
I, [2016-08-23T14:16:22.674838 #11029] INFO -- : master complete
I, [2016-08-23T14:16:25.630441 #11622] INFO -- : Refreshing Gem list
==> /var/log/gitlab/unicorn/state <==
==> /var/log/gitlab/unicorn/current <==
2016-08-23_07:14:05.27640 failed to start a new unicorn master
2016-08-23_07:14:05.28421 starting new unicorn master
2016-08-23_07:14:20.27934 adopted new unicorn master 11029
2016-08-23_07:16:22.57083 Received TERM from runit, sending to process group (-PID)
2016-08-23_07:16:22.57333 Terminated
2016-08-23_07:16:22.57333 forwarding TERM to unicorn master 11029
2016-08-23_07:16:22.57334 wrapper for unicorn master 11029 exiting
2016-08-23_07:16:22.67844 adopted existing unicorn master 11029
2016-08-23_07:16:23.68183 wrapper for unicorn master 11029 exiting
2016-08-23_07:16:23.71273 starting new unicorn master
==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/sidekiq/current <==
2016-08-23_07:16:37.74421 2016-08-23T07:16:37.744Z 11583 TID-54n94 INFO: [Sidetiq] See LICENSE and http://opensource.org/licenses/BSD-3-Clause for licensing details.
2016-08-23_07:16:37.74426 2016-08-23T07:16:37.744Z 11583 TID-54n94 INFO: [Sidetiq] Sidetiq::Supervisor start
2016-08-23_07:16:37.75229 2016-08-23T07:16:37.752Z 11583 TID-v34w0 INFO: [Sidetiq] Sidetiq::Actor::Clock id: 52191580 initialize
2016-08-23_07:16:37.75720 2016-08-23T07:16:37.757Z 11583 TID-v3rcc INFO: [Sidetiq] Sidetiq::Actor::Handler id: 52245220 initialize
2016-08-23_07:16:37.75769 2016-08-23T07:16:37.757Z 11583 TID-v407s INFO: [Sidetiq] Sidetiq::Actor::Handler id: 52256720 initialize
2016-08-23_07:16:41.47730 2016-08-23T07:16:41.477Z 11583 TID-54n94 INFO: Booting Sidekiq 3.3.0 with redis options {:url=>"unix:/var/opt/gitlab/redis/redis.socket", :namespace=>"resque:gitlab"}
2016-08-23_07:16:47.91937 2016-08-23T07:16:47.915Z 11583 TID-54n94 INFO: Running in ruby 2.1.7p400 (2015-08-18 revision 51632) [x86_64-linux]
2016-08-23_07:16:47.91946 2016-08-23T07:16:47.919Z 11583 TID-54n94 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2016-08-23_07:16:47.91952 2016-08-23T07:16:47.919Z 11583 TID-54n94 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2016-08-23_07:16:47.91958 2016-08-23T07:16:47.919Z 11583 TID-54n94 INFO: Starting processing, hit Ctrl-C to stop
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <==
2016-08-23_07:16:21.28992 LOG: aborting any active transactions
2016-08-23_07:16:21.29262 FATAL: terminating connection due to administrator command
2016-08-23_07:16:21.29263 FATAL: terminating connection due to administrator command
2016-08-23_07:16:21.29263 FATAL: terminating connection due to administrator command
2016-08-23_07:16:21.29264 LOG: autovacuum launcher shutting down
2016-08-23_07:16:21.29293 LOG: shutting down
2016-08-23_07:16:21.34178 LOG: database system is shut down
2016-08-23_07:16:21.40427 LOG: database system was shut down at 2016-08-23 07:16:21 GMT
2016-08-23_07:16:21.41061 LOG: database system is ready to accept connections
2016-08-23_07:16:21.41075 LOG: autovacuum launcher started
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
==> /var/log/gitlab/nginx/gitlab_access.log <==
...
...
...
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:16:42 +0700] "GET /static.css HTTP/1.1" 304 0 "http://xxx.xxx.xxx.xxx/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/nginx/gitlab_error.log <==
...
...
...
==> /var/log/gitlab/nginx/state <==
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/logrotate/state <==
==> /var/log/gitlab/logrotate/current <==
2016-08-23_07:16:20.44201 Received TERM from runit, sending to process group (-PID)
==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/gitlab-workhorse/current <==
2016-08-23_07:16:20.06174 2016/08/23 14:16:20 Starting gitlab-workhorse 0.4.1-3-g9123131-20151202.172858
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2016-08-23T14:16:49.411184 #11622] INFO -- : listening on addr=127.0.0.1:8080 fd=11
I, [2016-08-23T14:16:49.411577 #11622] INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket
I, [2016-08-23T14:16:49.411801 #11622] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=14
I, [2016-08-23T14:16:49.421408 #11669] INFO -- : worker=0 spawned pid=11669
I, [2016-08-23T14:16:49.421879 #11669] INFO -- : worker=0 ready
I, [2016-08-23T14:16:49.422264 #11622] INFO -- : master process ready
I, [2016-08-23T14:16:49.429714 #11672] INFO -- : worker=1 spawned pid=11672
I, [2016-08-23T14:16:49.430232 #11672] INFO -- : worker=1 ready
==> /var/log/gitlab/unicorn/current <==
2016-08-23_07:16:50.45284 adopted new unicorn master 11622
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/" for 127.0.0.1 at 2016-08-23 14:17:00 +0700
Processing by RootController#index as HTML
Completed 401 Unauthorized in 5ms (ActiveRecord: 0.0ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
...
...
...
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/users/sign_in" for 127.0.0.1 at 2016-08-23 14:17:00 +0700
Processing by SessionsController#new as HTML
Completed 200 OK in 698ms (Views: 235.3ms | ActiveRecord: 33.2ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
...
...
...
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/users/sign_in" for 127.0.0.1 at 2016-08-23 14:17:18 +0700
Processing by SessionsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 401 Unauthorized in 540ms (ActiveRecord: 86.9ms)
Processing by SessionsController#new as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 200 OK in 125ms (Views: 9.2ms | ActiveRecord: 2.2ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:17:19 +0700] "POST /users/sign_in HTTP/1.1" 200 2942 "http://xxx.xxx.xxx.xxx/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/users/sign_in" for 127.0.0.1 at 2016-08-23 14:17:30 +0700
Processing by SessionsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx.xxxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 401 Unauthorized in 251ms (ActiveRecord: 13.9ms)
Processing by SessionsController#new as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx.xxxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 200 OK in 228ms (Views: 56.0ms | ActiveRecord: 1.6ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
192.168.84.11 - - [23/Aug/2016:14:17:31 +0700] "POST /users/sign_in HTTP/1.1" 200 2948 "http://xxx.xxx.xxx.xxx/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/users/sign_in" for 127.0.0.1 at 2016-08-23 14:17:41 +0700
Processing by SessionsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 401 Unauthorized in 150ms (ActiveRecord: 15.8ms)
Processing by SessionsController#new as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Completed 200 OK in 121ms (Views: 9.8ms | ActiveRecord: 2.0ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:17:41 +0700] "POST /users/sign_in HTTP/1.1" 200 2942 "http://xxx.xxx.xxx.xxx/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started POST "/users/sign_in" for 127.0.0.1 at 2016-08-23 14:18:04 +0700
Processing by SessionsController#create as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"login"=>"xxxxx", "password"=>"[FILTERED]", "remember_me"=>"0"}}
Redirected to http://xxx.xxx.xxx.xxx/
Completed 302 Found in 333ms (ActiveRecord: 182.4ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:04 +0700] "POST /users/sign_in HTTP/1.1" 302 97 "http://xxx.xxx.xxx.xxx/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/" for 127.0.0.1 at 2016-08-23 14:18:04 +0700
Processing by RootController#index as HTML
Cache digest for app/views/shared/projects/_project.html.haml: df6ab45e39dbe0199d2b8c35e9a691e2
Read fragment views/namespaces/3-20151229091853804759000/projects/28-20160516020619918400000/root/index/v2.2/df6ab45e39dbe0199d2b8c35e9a691e2 (0.2ms)
Write fragment views/namespaces/3-20151229091853804759000/projects/28-20160516020619918400000/root/index/v2.2/df6ab45e39dbe0199d2b8c35e9a691e2 (0.4ms)
...
...
...
Completed 200 OK in 2248ms (Views: 1892.8ms | ActiveRecord: 206.4ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:07 +0700] "GET / HTTP/1.1" 200 4683 "http://xxx.xxx.xxx.xxx/users/sign_in" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
...
...
...
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/profile" for 127.0.0.1 at 2016-08-23 14:18:34 +0700
Processing by ProfilesController#show as HTML
Completed 200 OK in 498ms (Views: 402.2ms | ActiveRecord: 71.4ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:35 +0700] "GET /profile HTTP/1.1" 200 4265 "http://xxx.xxx.xxx.xxx/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/profile/keys" for 127.0.0.1 at 2016-08-23 14:18:38 +0700
Processing by Profiles::KeysController#index as HTML
Completed 200 OK in 112ms (Views: 76.9ms | ActiveRecord: 4.5ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:38 +0700] "GET /profile/keys HTTP/1.1" 200 3731 "http://xxx.xxx.xxx.xxx/profile" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
...
...
...
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/profile/emails" for 127.0.0.1 at 2016-08-23 14:18:40 +0700
Processing by Profiles::EmailsController#index as HTML
Completed 200 OK in 112ms (Views: 78.5ms | ActiveRecord: 4.3ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:40 +0700] "GET /profile/emails HTTP/1.1" 200 3727 "http://xxx.xxx.xxx.xxx/profile/keys" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin" for 127.0.0.1 at 2016-08-23 14:18:44 +0700
Processing by Admin::DashboardController#index as HTML
Completed 200 OK in 536ms (Views: 487.9ms | ActiveRecord: 22.7ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:45 +0700] "GET /admin HTTP/1.1" 200 5082 "http://xxx.xxx.xxx.xxx/profile/emails" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/admin/users" for 127.0.0.1 at 2016-08-23 14:18:47 +0700
Processing by Admin::UsersController#index as HTML
Completed 200 OK in 291ms (Views: 224.8ms | ActiveRecord: 39.4ms)
==> /var/log/gitlab/nginx/gitlab_access.log <==
xxx.xxx.xxx.xxx - - [23/Aug/2016:14:18:48 +0700] "GET /admin/users HTTP/1.1" 200 5136 "http://xxx.xxx.xxx.xxx/admin" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/50.0.2661.102 Chrome/50.0.2661.102 Safari/537.36"
[root@hostname ]#
At last, finally Gitlab process has successfully started. It can be inferred from one line of output extracted from the log file named unicorn_stderr.log as shown below :
I, [2016-08-23T14:16:49.411184 #11622] INFO -- : listening on addr=127.0.0.1:8080 fd=11