277d9badcbd723e913b3a41e64e8d2f3d2c80598orange-ant-glusterfs

Continues delivery? Jenkins/cron driven task, Ansible playbooks from Git repos ? So what happens if GitLab goes down ?

Example setup

We deploy and install three servers for all important GitLab application. For easinest start with clean CentOS6 64b machines root access and ssh key from your workstation and galera cluster control.

  • 3x balanced application servers with nginx + ssl
  • 3x database backend (Galera with S9s Cluster Control)
  • 3x redis HA with Sentinel
  • GlusterFS for repositories, keys and satelites

Servers:

database01 192.168.0.10/24
database02 192.168.1.10/24
database03 192.168.2.10/24
redis01 192.168.0.11/24
redis02 192.168.1.11/24
redis03 192.168.2.11/24
gitapp01 192.168.1.12/24
gitapp02 192.168.2.12/24
gitapp03 192.168.3.12/24

Galera

Recommended way is to use SeveralNines solution for full MySQL master-master cluster deploy, they provide easy web UI for installation pack creation: http://www.severalnines.com/galera-configurator3/

Redis

Use version > 2.6

/etc/redis.conf

bind 127.0.0.1 192.168.X.11
slaveof 192.168.0.11 6379 #redis02/redis03 only

/etc/redis-sentinel.conf (redis01/redis02/redis03)

sentinel monitor redis01 192.168.0.11 6379 2
sentinel down-after-milliseconds redis01 60000
sentinel failover-timeout redis01 180000 sentinel parallel-syncs redis01 1

restart redis and redis-sentinel process.

GitLab APP

Haproxy

/etc/haproxy/haproxy.cfg

global
group haproxy
maxconn 10000
stats socket /var/lib/haproxy/stats mode 660 level admin
user haproxy

defaults
log global
maxconn 8000
option redispatch
retries 3
stats enable
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout check 10s

userlist STATSUSERS
group admin users admin
user admin insecure-password XXXXXX
user stats insecure-password MONITOR

listen admin_page
bind 0.0.0.0:9600
mode http
acl AuthOkay_ReadOnly http_auth(STATSUSERS)
acl AuthOkay_Admin http_auth_group(STATSUSERS) admin
stats enable
stats refresh 60s
stats uri /
stats http-request auth realm admin_page unless AuthOkay_ReadOnly
stats admin if AuthOkay_Admin

# Specifies TCP timeout on connect for use by the frontend ft_redis
# Set the max time to wait for a connection attempt to a server to succeed
# The server and client side expected to acknowledge or send data.
defaults REDIS
mode tcp
timeout connect 3s
timeout server 6s
timeout client 6s

# Specifies listening socket for accepting client connections using the default
# REDIS TCP timeout and backend bk_redis TCP health check.
frontend ft_redis
bind *:6378 name redis
default_backend bk_redis

# Specifies the backend Redis proxy server TCP health settings
# Ensure it only forward incoming connections to reach a master.
backend bk_redis
option tcp-check
tcp-check connect
tcp-check send PING\r\n
tcp-check expect string +PONG
tcp-check send info\ replication\r\n
tcp-check expect string role:master
tcp-check send QUIT\r\n
tcp-check expect string +OK
server redis01 192.168.0.11:6379 check inter 1s
server redis02 192.168.1.11:6379 check inter 1s
server redis03 192.168.2.11:6379 check inter 1s

listen mysql
bind 127.0.0.1:3306
mode tcp
balance leastconn
default-server port 9200 inter 15s downinter 10s rise 3 fall 3 slowstart 5s maxconn 4000 maxqueue 512 weight 100
option redispatch
option httpchk
timeout client 120000ms
timeout server 120000ms
server database01 192.168.0.10:3306 check weight 100
server database02 192.168.1.10:3306 check weight 100
server database03 192.168.2.10:3306 check weight 100

Check via IP:9600 if database and master node of redis is online.

GlusterFS

We setup 3 replicated disk for repositories, auth. file and uploads. Install gluster-fs on gitlab app servers and then:

# On all gitlabX servers
mkfs.xfs -i size=512 /dev/sdb1 /dev/sdb2 /dev/sdb3
mkdir -p /data/brick1/data/brick2 /data/brick3
echo '/dev/sdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
echo '/dev/sdb1 /data/brick2 xfs defaults 1 2' >> /etc/fstab
echo '/dev/sdb1 /data/brick3 xfs defaults 1 2' >> /etc/fstab

mount -a && mount
# On one of gitappX server:
gluster peer probe gitapp01
gluster peer probe gitapp02
gluster peer probe gitapp03

# On all gitlabX servers
mkdir /data/brick1/gv0
mkdir /data/brick2/gv1
mkdir /data/brick3/gv2

# On one of gitappX server:
gluster volume create gv0 replica 3 gitapp01:/data/brick1/gv0 gitapp02:/data/brick1/gv0 gitapp03:/data/brick1/gv0 gluster volume start gv0
gluster volume create gv0 replica 3 gitapp01:/data/brick2/gv1 gitapp02:/data/brick2/gv1 gitapp03:/data/brick2/gv1 gluster volume start gv1
gluster volume create gv0 replica 3 gitapp01:/data/brick3/gv2 gitapp02:/data/brick3/gv2 gitapp03:/data/brick3/gv2 gluster volume start gv2

Check status via “gluster volume info” if everything is ok than create user git and mount points (on all gitlab0X servers):

useradd git -d /opt/git
mkdir /opt/git/{uploads,repositories,.ssh}
mount -t glusterfs gitapp01:/gv0 /opt/git/uploads
mount -t glusterfs gitapp01:/gv1 /opt/git/repositories
mount -t glusterfs gitapp01:/gv2 /opt/git/.ssh

GitLab installation and setup

Follow this perfect step-by-step manual

https://github.com/gitlabhq/gitlab-recipes/blob/master/install/centos/README.md

Performance testing

Without GlusteFS (local disks) vs GFS, operation clone (repository with 10K files and 20 branches) and push (1x 10Kb text file). GFS is aprox. 6x slower (all servers running on vmware, in three location with ~10ms network latencies).

Screen Shot 2015-11-28 at 15.08.23

Advertisements