Setting up GitLab

I like using automation, and I’ve been using Azure DevOps as part of my daily worklife for some time. I decided I want to use something open source in my home lab - yeah, I know, GitHub/GitLab and other services are out there, but here I am sitting on 20 years of antiquated technology, maybe I can find a use for it. Besides, if I just do the same old things all the time, I don’t learn anything. So let’s go learn something.

Goals are for me to be able to:

• Register repositories
• Receive Email notifications of changes/problems
• Setup a CI/CD Pipeline to automate my build and deployment process
• Build this website
• Publish this website

Home Lab Server: Debian + SSH

This is all based on a Dell Optiplex Micro a friend of mine donated some time ago. It came with an aging hard drive which I could have use, but it was so painfully slow, I rumaged through my bits box and found a 450GB Sandisk 2.5” SSD that was still in its packaging. I binned the HDD and dropped in the SSD. Connect up a monitor and keyboard and load the Debian Netinst .iso onto a thumb drive, get the hardware deployed and ssh installed. This was fairly conventional stuff, nothing special here, and once I’d proved the SSH connection was good, I disconnected it and went headless from here on out.

I named it gitlab-server (very original)

ssh-copy-id -i ~/.ssh/id_rsa.pub gitlab-server

Now to logon to the new server and setup a few things

# Connect to gitlab-server and elevate to root
ssh gitlab-server
su -
apt-get install sudo
usermod -aG sudo bob
# exit the root shell
exit
# exit user shell
exit

With my user now in the sudo group and sudo installed, I can do the usual things in Debian

# gitlab-server: install vim and a few helpers that we're bound to need
sudo apt-get install vim-nox curl wget

GitLab

GitLab install documentation takes us through the setup process and helpfully provides scripts to install GPG keys and setup the gitlab sources for apt (helpful with updates). I didn’t encounter any issues and stashed the root password in my keyring. We can ignore the postfix configuration, we’ll be setting up SMTP access.

# gitlab-server
# pre-requisites
sudo apt-get install curl ca-certificates perl
# gitlab-server
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ce

As I don’t have a self hosted server for outgoing mail in the same network, I pointed it at my mail host using the GitLab SMTP instructions. The fact that the credentials for this can be encrypted is doubly useful.

Testing that access is all working and I can email out, I setup a personal user account and checked that the email went out as expected - it did - and I was able to login and access the system in the way you’d expect

I went to my git repository for the website and replaced origin with my new gitlab-server

# Editting machine
git remote set-url origin http://gitlab-server.homelab.lan
git push

Git Credentials and GitLab OAuth

After a few tweaks, pushes and pulls, I rapidly became bored of having to re-enter my credentials all the time. Time to setup GitLab OAuth server - the key one here is to allow “read_repository” and “write_repository” access to the authentication tokens. The git-credential-oauth package is available in Debian already so I configured that to do a bit of caching for me. Please note that the documentation for git-credential-oauth does not put any quotes around the read/write repository scopes - single or double quotes according to the git documentation are needed for a multivar

# Editting machine
sudo apt-get install git-credential-oauth
# configure git-credential-oauth
git credential-oauth configure
git config --global credential.http://gitlab-server.homelab.lan.oauthClientId {applicationId}
git config --global credential.http://gitlab-server.homelab.lan.oauthScopes 'read_repository write_repository'
git config --global credential.http://gitlab-server.homelab.lan.oauthAuthURL /oauth/authorize
git config --global credential.http://gitlab-server.homelab.lan.oauthTokenURL /oauth/token

Now it opens up a web page, allows me to authenticate once, then cache that for a few hours - massively reducing the number of times I need to login.

GitLab-runner

The runner at this point did not exist, and I was getting pings to say my builds had failed. Which was a little strange as I’d also not created the CI/CD pipeline for the project.

I plan to setup k8s later, but for now, I’ll install a default runner and Docker, using the default docker image for a build and go from there - at the very least, it’s going to allow me to also tick off another goal on the list

Starting with Docker engine, consult the Install Docker Engine on Debian documentation to setup GPG keys and the apt sources.list

# gitlab-server
# Add Docker's official GPG key
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add repository to apt sources
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/debian  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Now refer to the GitLab-Runner Install documentation and install the apt repository information necessary for installing gitlab runner

# gitlab-server
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
# Pin the package source
cat <<EOF | sudo tee /etc/apt/preferences.d/pin-gitlab-runner.pref
Explanation: Prefer GitLab provided packages over the Debian native ones
Package: gitlab-runner
Pin: origin packages.gitlab.com
Pin-Priority: 1001
EOF
# Install the runner
sudo apt-get update
sudo apt-get install gitlab-runner

At this point it should autodetect and configure Docker access, but just to be safe:

# gitlab-server
sudo usermod -aG docker gitlab-runner
sudo systemctl restart docker
sudo systemctl restart gitlab-runner

You can follow the registration instructions from gitlab-server > Admin area > Instance Runners: New instance runner and stash the url and token in the runner registration. I’m going with Docker (because) and ruby:3.2 as the default image that should let me build my website

# gitlab-server
sudo gitlab-runner register --url http://gitlab-server.homelab.lan --token {fromGitLabServer} --executor docker --docker-image ruby:3.3 --description "default-runner"

And with that, we should be able to start pushing builds through the pipeline.

Simple pipeline for Jekyll

image: ruby:3.3

pages:
  stage: build
  script:
    - gem install bundler
    - bundle install
    - bundle exec jekyll build -d public
  artifacts:
    paths:
    - public
  only:
  - main

Check that in and watch it push through the pipeline without any issues and generate a build artifact. We should also get an email saying the build pipeline is fixed! For my next trick, I’ll setup an SSH key for the gitlab runner to push through to the main website

Setting up SSH remote access

This is based heavily on the GitLab SSH Keys guide for CI/CD pipelines

We need to do three tasks:

• Setup SSH key for the target server
• Store key in a location the deploying task can get to it
• Share Public key on target servers, associated with an account for running tasks as

Key Gen

Let’s start by creating a SSH key with ssh-keygen - you can do this anywhere, but we want this to be somewhat secure because access to this key will grant the holder access to our target server(s)

# anywhere
ssh-keygen -t rsa -b 4096 -C "GitLab Deployment Key" -f ~/gitlab-deploy-key

When asked, do not enter a password

Stashing information

Stash the Public key in an accessible location - servers with this as authorised key will be able to recieve SSH deployment jobs. You can also add this to other GitLab instances as Deploy keys (those instances will be targets for deployment) Gitlab Deploy keys

Stash the Private key in GitLab CI/CD Variables, ensuring that the pasted value ends with a newline (see documentation)

  - Type: File
  - Flags: (x) Protect variable
           ( ) Mask variable
           ( ) Expand variable reference
  - Description: GitLab Deployment Key
  - Key: SSH_PRIVATE_KEY
  - Value: {contents of ~/gitlab-deploy-key}

Treat this key securely. You should know that if you let someone else have access to them you’re giving them the full power of your deployment pipeline. Destroy the secret key file so that copies cannot be taken that way and limit access to the variable in GitLab. Heed also the warnings in the GitLab documentation!

To protect from Man-In-The-Middle (mitm) attacks, create a known-hosts variable and scan they local server keys and stash them as a variable here as well

  - Type: File
  - Flags: (x) Protect variable
           ( ) Mask variable
           ( ) Expand variable reference
  - Description: GitLab Deployment Key
  - Key: SSH_KNOWN_HOSTS
  - Value: {result of running ssh-keyscan}

Sharing the public key on the target

Let’s start by setting up a “gitlab-deploy” user on our targets. Login to the target and create a user account, add it to our www-data group (we’ll need this permission later)

# target server
sudo adduser --group gitlab-deploy gitlab-deploy
sudo adduser --shell /bin/sh --ingroup gitlab-deploy --disabled-password --home /var/opt/gitlab-deploy gitlab-deploy
sudo adduser gitlab-deploy www-data

Now we can copy the contents of the ~/gitlab-deploy-key.pub (created earlier) to authorised_keys on our target. As we created the user account without password, we’ll need to do that manually for now. Assuming that you copied the key from wherever you created it to the target server, that would look like this:

# target server
cat ~/gitlab-deploy-key.pub | sudo su - gitlab-deploy -c "mkdir ~/.ssh; tee -a /var/opt/gitlab-deploy/.ssh/authorized_keys"

We can test that this is working by logging in with the private key

# anywhere
ssh gitlab-deploy@{target-server} -i ~/gitlab-deploy-key

Use the key as part of the deployment script pipeline

before_script:
  - 'command -v ssh-ageeent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
  - eval $(ssh-agent -s)
  - chmod 400 "$SSH_PRIVATE_KEY"
  - ssh-add "$SSH_PRIVATE_KEY"

  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

  - cp "$SSH_KNOWN_HOSTS" ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts

With this, we are ready to automate the deployment of our build. Add the above lines to a deployment section of the pipeline:

image: ruby:3.1

pages:
  stage: build
  script:
    - bundle install
    - bundle exec jekyll build -d public
  artifacts:
    paths:
    - public    
  only:
  - main

deploy:
  only:
  - main
  stage: deploy
  before_script:
  - 'command -v rsync >/dev/null || ( apt-get update -y && apt-get install rsync -y )'
  - 'command -v ssh-agent >/dev/null || ( apt-get update -y && apt-get install openssh-client -y )'
  - eval $(ssh-agent -s)
  - chmod 400 "$SSH_PRIVATE_KEY"
  - ssh-add "$SSH_PRIVATE_KEY"

  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

  - cp "$SSH_KNOWN_HOSTS" ~/.ssh/known_hosts
  - chmod 644 ~/.ssh/known_hosts
  script:
  - rsync -atv --delete ./public/ gitlab-deploy@{target-server}:/var/www/html

Finally - enjoy setting permissions in places you never knew you needed to, in order to get the deployment working. Just remember, on the remote machine “gitlab-deploy” is the user that needs permission to do the action from deployment pipeline