Migrating/Upgrading Gitlab 6.X to latest Gitlab docker

Gitlab is an amazing Git hosting service. With it’s community edition, one can setup an in-house git server in no time.

Those who have been an earlier follower of Gitlab, may still be stuck with the version 6.X. Version 6.X was setup manually by creating DB in MySQL/Postgres, Nginx/Apache entries and few other setup steps.

As time moved on, Gitlab started providing packages and one could install or upgrade Gitlab using a package manager like yum, apt. And now it provides docker images for all its Gitlab versions. Those who didn’t upgrade are really missing some of the latest and the collest features of Gitlab

I was tasked to upgrade Gitlab 6.X at our company to the latest version, so we could use the latest Gitlab CI capabilities. Our existing setup was using MySQL.

The impression was that we can’t upgrade, so we would install a new version and import projects into the same. This would mean loosing all the MR’s that we had and won’t be much of use

So I decided to dig into the same and see if we could do an automated/manual upgrade. Gitlab is a Ruby on Rails (ROR) application, so it uses migrations for the DB upgrade. Using migrations should mean no issues upgrading to any version, but that was not the case with Gitlab.

It took me a week or two to resolve different issues that came up during the migration and that’s when I decided to share the details for others

Below is the list of steps one need to follow for an upgrade

  • Stop the existing server and dump the MySQL database dump
  • Convert the dump from MySQL to Postgres compatible dump
  • Launch a new Dockerized instance of Gitlab
  • Import the Postgres Compatible dump
  • Copy existing repositories
  • Reconfigure and restart gitlab
  • Run the migrations and Resolve any issues
  • Excute gitlab checks
  • Reconfigure and restart gitlab
  • Run the post migration setup steps
  • Upgrading to latest Gitlab

Taking the mysql dump

To take the dump we use the mysqldump command

mysqldump --compatible=postgresql --default-character-set=utf8 -r gitlabhq_production_new.mysql --extended-insert=FALSE -u root gitlab -p

Note: I have used the user root, but if you have a different user, use that

Converting the Dump from MySQL to Postgres

This requires python, so make sure you have python installed.

$ git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab
$ mkdir db
$ python mysql-postgresql-converter/db_converter.py gitlabhq_production.mysql db/database.sql

Launch Gitlab using Docker

You can get all the available tags here. I have assumed 8.11.11-ce.0 as the lowest one. The approach is to upgrade to the lowest available tag first and then move on to the higher tag

So we use the below docker-compose.yml file

version: '2'
services:
  gitlab:
    image: gitlab/gitlab-ce:8.11.11-ce.0
    ports:
      - "22:22"
      - "80:80"
    volumes:
      - ./config:/etc/gitlab
      - ./logs:/var/log/gitlab
      - ./data:/var/opt/gitlab
      - ./db:/db
      - ./scripts:/scripts

Note: Before launching this, you need to update your /etc/ssh/sshd_config and set your ssh port to something different than 22. Else you would need change the SSH port of the gitlab. This would require setting some environment variables in the docker-compose.yml. See this article for more details

Now we launch gitlab and wait for the Postgres DB to be up

# Start the gitlab composition
docker-compose up -d

# Give 2 minutes for the system to start
sleep 120

# Wait for the DB to get up
docker-compose logs -f gitlab | grep -q 'autovacuum launcher started'

Import the Postgres Compatible dump

Now to import the dump, we need an empty Postgres DB, but when we start the new docker image it runs the migrations. This initialize the database according to the latest schema and hence and import would fail. So our first task is to clear the database

echo "Dropping current tables"
echo "select 'DROP table ' || tablename || ' CASCADE;'  from pg_tables  WHERE schemaname='public';" | gitlab-rails db | head -n -2 | tail -n +3 | gitlab-rails db

echo "Dropping current sequences"
echo "select 'DROP SEQUENCE ' || sequence_name || ';'  from information_schema.sequences  WHERE NOT sequence_schema IN ('pg_catalog', 'information_schema') and sequence_schema='public' and sequenc    e_catalog='gitlabhq_production';" | gitlab-rails db | head -n -2 | tail -n +3 | gitlab-rails db

Note: Above commands need to be executed inside the gitlab container. So use docker-compose exec gitlab bash to get inside the container and then execute the commands. All upcoming commands need to be executed inside the container

Now we import our db

echo "Loading current database"
cat /db/database.sql | gitlab-rails db

Copy existing repositories

Before we proceed further, we need to copy over the existing repositories to the ./data/git-data/repositories folder. I would recommend top copy and not move the repositories, as there are changes that are made during the upgrade.

Reconfigure and Restart Gitlab

Now we need to reconfigure and restart the gitlab server


echo "Updating permissions"
update-permissions

echo "Reconfigure gitlab"
gitlab-ctl reconfigure

echo "Restart gitlab"
gitlab-ctl restart

These steps may seem unnecessary but I encountered weird errors if these commands where not executed

Run the migrations and Resolve any issues

This is a important step and it take certain attempts to get past through this. To run a migration we execute the below command

MIGRATION_COUNT=0

run_migrations() {
    ((MIGRATION_COUNT++))
    echo "Running migration $MIGRATION_COUNT"
    gitlab-rake db:migrate 2>&1 | tee -a /tmp/migration.log | grep -A9 ERROR
}

We call the run_migrations function, it runs migration and sends all migration logs to /tmp/migration.log in append mode (-a flag). The grep command makes sure that on screen we only see lines with ERROR in it (+9 next lines).

When we run the above migration, first time. The command would error out with below error

PG::UndefinedColumn: ERROR:  column issues.deleted_at does not exist

We fix this issue by manually altering the TABLE for migration to proceed

echo "Fixing issues.deleted_at migration issue"
echo "ALTER TABLE issues add column deleted_at timestamp;" | gitlab-rails db

Next migration error we would get is

PG::UndefinedColumn: ERROR:  column projects.pending_delete does not exist

The fix is similar to previous one

echo "Fixing projects.pending_delete migration issue"
echo "ALTER TABLE projects add column pending_delete boolean;" | gitlab-rails db

Next migration we get is

NoMethodError: undefined method create_label_from_tagging'

Note: You may or may not face this issue, depending on the state of the database

If the issue is faced, below command will fix it

echo "Fixing nilClass migration issue"
gitlab-rake gitlab:db:mark_migration_complete[20140729152420]

PG::DuplicateColumn: ERROR:  column "pending_delete" of relation "projects" already exists

echo "Fixing projects.pending_delete migration issue"
echo "ALTER TABLE projects DROP COLUMN pending_delete;" | gitlab-rails db

PG::DuplicateColumn: ERROR:  column "deleted_at" of relation "issues" already exists

echo "Fixing issues.deleted_at migration issue"
echo "ALTER TABLE issues DROP COLUMN deleted_at ;" | gitlab-rails db

Execute gitlab checks

Next we execute gitlab checks to validate that everything is ok

echo "Checking issues"
gitlab-rake gitlab:check

Reconfigure and restart Gitlab

Next we reconfigure and restart gitlab

echo "Reconfigure gitlab"
gitlab-ctl reconfigure

echo "Gitlab restart"
gitlab-ctl restart

Run the post migration setup steps

The ssh based access to the any repository works by creating a authorized_keys file with ssh keys of the user. This file gets updated when a user adds a new ssh key through the Gitlab UI. But since we imported all the ssh keys directly to DB, none of keys get added to the file.

We must regenerate the file using a rake task

echo "Setting up SSH KEYS"
LC_ALL=en_US.UTF-8 gitlab-rake gitlab:shell:setup

Upgrading to latest Gitlab

By using docker the upgrade process of gitlab becomes very simple. We update the image in our docker-compose.yml from gitlab/gitlab-ce:8.11.11-ce.0 to gitlab/gitlab-ce:latest

So now whenever we pull the images, it would always pull the latest version.

# This step is only needed if you are upgrading from previous docker setup
docker-compose exec gitlab gitlab-rake gitlab:backup:create

# Pull the latest image
docker-compose pull

# Update docker
docker-compose up -d

# Check logs for any issues
docker-compose logs -f

If you need a fully automated script for migration, refer to tarunlalwani/gitlab-6-migration-latest.git

Home