Continuous Integration / Deployment

Biom3d now has a continuous deployment pipeline that activate when you push a tag.

How to trigger it

It trigger when a tag is pushed, typically :

    git tag vx.y.z
    git push origin tag vx.y.z

The tag name must be in the specific format above, starting with a ‘v’ and with semantic versionning (optional but better). You can also remove a tag by doing :

    git tag - vx.y.z # Remove local tag
    git push origin --delete vx.y.z # Remove remote tag

So if the CI/CD crash or you make a mistake, you can safely delete the tag and create a new one.

Working

Here is the script :

  1name: Build and Publish
  2
  3on:
  4  push:
  5    tags:
  6      - 'v*'
  7
  8env:
  9  APP_NAME: Biom3d
 10  DOCKER_USERNAME: ${{ secrets.DOCKER_USERNAME }}
 11  DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
 12
 13jobs:
 14  read-config:
 15    runs-on: ubuntu-latest
 16    outputs:
 17      configs: ${{ steps.read-config.outputs.configs }}
 18    steps:
 19      - uses: actions/checkout@v3
 20      - id: read-config
 21        run: |
 22          echo "configs=$(jq -c '.configs' .github/workflows/config_docker.json)" >> $GITHUB_OUTPUT
 23
 24  build-docker:
 25    needs: read-config
 26    runs-on: ubuntu-latest
 27    strategy:
 28      matrix:
 29        config: ${{ fromJson(needs.read-config.outputs.configs) }}
 30    steps:
 31      - uses: actions/checkout@v3
 32
 33      - name: Login Docker Hub
 34        uses: docker/login-action@v2
 35        with:
 36          username: ${{ env.DOCKER_USERNAME }}
 37          password: ${{ env.DOCKER_PASSWORD }}
 38
 39      - name: Build and push Docker image
 40        run: |
 41          TAG="${GITHUB_REF#refs/tags/}-${{ matrix.config.architecture }}-torch${{ matrix.config.torch_version }}"
 42          if [ -n "${{ matrix.config.cuda_version }}" ]; then
 43            TAG="$TAG-cuda${{ matrix.config.cuda_version }}-cudnn${{ matrix.config.cudnn_version }}"
 44          fi  
 45
 46          if [ -n "${{ matrix.config.target }}" ] && [ "${{ matrix.config.target }}" != "gpu" ]; then
 47            TAG="$TAG-${{ matrix.config.target }}"
 48          fi
 49          docker build \
 50            --build-arg BASE_IMAGE=${{ matrix.config.base_image }} \
 51            --build-arg TARGET=${{ matrix.config.target }} \
 52            --build-arg PYTHON_VERSION=${{ matrix.config.python_version }} \
 53            --build-arg OMERO_VERSION=${{ matrix.config.omero_version }} \
 54            --build-arg TESTED=${{ matrix.config.tested }} \
 55            -f deployment/dockerfiles/template.dockerfile \
 56            -t ${{ env.DOCKER_USERNAME }}/biom3d:$TAG  .
 57          docker push ${{ env.DOCKER_USERNAME }}/biom3d:$TAG
 58
 59  build-macos:
 60    needs: read-config
 61    runs-on: macos-latest
 62    steps:
 63      - uses: actions/checkout@v3
 64
 65      - name: Extract version from tag
 66        id: get_version
 67        run: echo "version=${GITHUB_REF#refs/tags/}" >> $GITHUB_OUTPUT
 68
 69      - name: Update CFBundleVersion in Info.plist
 70        run: |
 71          /usr/libexec/PlistBuddy -c "Set :CFBundleVersion ${{ steps.get_version.outputs.version }}" deployment/exe/macos/Info.plist
 72
 73      - name: Install Miniforge
 74        run: |
 75          curl -L -o Miniforge3-MacOSX-arm64.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
 76          bash Miniforge3-MacOSX-arm64.sh -b -p $HOME/miniforge
 77          echo "export PATH=$HOME/miniforge/bin:$PATH" >> $GITHUB_ENV
 78
 79      - name: Initialize conda environment for pack.sh
 80        shell: bash
 81        run: |
 82          source $HOME/miniforge/etc/profile.d/conda.sh
 83          conda activate base
 84
 85      - name: Get architecture
 86        id: arch
 87        run: echo "ARCH=$(uname -m)" >> $GITHUB_OUTPUT
 88        shell: bash
 89
 90      - name: Build macOS app
 91        run: |
 92          cd deployment/exe/macos && \
 93          export PATH=$HOME/miniforge/bin:$PATH && \
 94          source $HOME/miniforge/etc/profile.d/conda.sh && \
 95          conda activate base &&\
 96          chmod +x ./pack.sh &&\
 97          ./pack.sh ${{ steps.arch.outputs.ARCH }}
 98
 99      - name: Set REMOTE = True for remote build
100        run: sed -i '' 's/^REMOTE = False/REMOTE = True/' src/biom3d/gui.py
101
102      - name: Build remote MacOS app with minimal Python env
103        shell: bash
104        id: remote
105        run: |
106          python -m venv remote
107          source remote/bin/activate
108          python -m ensurepip --upgrade
109          pip install wheel
110          pip install pyinstaller paramiko pyyaml
111          pyinstaller --clean --onefile \
112            --name Biom3d_MacOS_${{ steps.arch.outputs.ARCH }}_Remote \
113            --icon=deployment/exe/windows/logo.ico \
114            src/biom3d/gui.py
115          cp /Users/runner/work/biom3d/biom3d/dist/Biom3d_MacOS_${{ steps.arch.outputs.ARCH }}_Remote $GITHUB_WORKSPACE
116
117      - name: Prepare artifact folder
118        run: |
119          mkdir artifact_root
120          cp Biom3d_MacOS_${{ steps.arch.outputs.ARCH }}_Remote artifact_root/
121          cp deployment/exe/macos/Biom3d_MacOS_*.zip artifact_root/
122        shell: bash
123
124      - name: Upload MacOS artifact
125        uses: actions/upload-artifact@v4
126        with:
127          name: Biom3d-MacOS
128          path: artifact_root
129
130  build-windows:
131    needs: read-config
132    runs-on: windows-latest
133    steps:
134      - uses: actions/checkout@v3
135
136      - name: Install Miniforge (Windows)
137        shell: powershell
138        run: |
139          Invoke-WebRequest -Uri "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Windows-x86_64.exe" -OutFile "Miniforge3.exe"
140          Start-Process -FilePath .\Miniforge3.exe -ArgumentList "/InstallationType=JustMe", "/AddToPath=1", "/RegisterPython=0", "/S", "/D=$env:USERPROFILE\miniforge3" -NoNewWindow -Wait
141
142      - name: Add conda to path
143        run: |
144          Add-Content -Path $Env:GITHUB_PATH -Value "$env:USERPROFILE\miniforge3\Scripts"
145          Add-Content -Path $Env:GITHUB_PATH -Value "$env:USERPROFILE\miniforge3\Library\bin"
146          Add-Content -Path $Env:GITHUB_PATH -Value "$env:USERPROFILE\miniforge3\bin"
147        shell: powershell
148
149      - name: Initialize Conda
150        run: |
151          conda init cmd.exe
152        shell: cmd
153
154      - name: Get architecture
155        id: arch
156        shell: powershell
157        run: |
158          $arch = $env:PROCESSOR_ARCHITECTURE
159          switch ($arch) {
160            "AMD64" { $arch = "x86_64" }
161            "ARM64" { $arch = "arm64" }
162            "x86" { $arch = "x86" }
163            default { $arch = $arch }
164          }
165          echo "ARCH=$arch" >> $env:GITHUB_OUTPUT
166
167      - name: Build Windows app
168        shell: cmd
169        run: |
170          cd deployment\exe\windows && ^
171          call pack.bat ${{ steps.arch.outputs.ARCH }}
172
173
174      - name: Set REMOTE = True for remote build (Windows)
175        shell: powershell
176        run: |
177          (Get-Content src\biom3d\gui.py) -replace '^REMOTE = False', 'REMOTE = True' | Set-Content src\biom3d\gui.py
178
179      - name: Build remote Windows app with minimal Python env
180        shell: powershell
181        run: |
182          python -m venv remote
183          .\remote\Scripts\activate
184          python -m ensurepip --upgrade
185          pip install wheel
186          pip install pyinstaller paramiko pyyaml
187          pyinstaller --clean --onefile `
188            --name Biom3d_Windows_${{ steps.arch.outputs.ARCH }}_Remote.exe `
189            --icon=deployment/exe/windows/logo.ico `
190            src/biom3d/gui.py
191
192          Copy-Item -Path "dist\Biom3d_Windows_${{ steps.arch.outputs.ARCH }}_Remote.exe" -Destination "$env:GITHUB_WORKSPACE"
193
194
195      - name: Prepare artifact folder
196        run: |
197          mkdir artifact_root
198          copy Biom3d_Windows_${{ steps.arch.outputs.ARCH }}_Remote.exe artifact_root\
199          copy deployment\exe\windows\Biom3d_Windows_*.zip artifact_root\
200        shell: cmd
201
202      - name: Upload Windows artifact
203        uses: actions/upload-artifact@v4
204        with:
205          name: Biom3d-Windows
206          path: artifact_root\
207
208  release:
209    name: Create Release
210    needs: [build-macos, build-windows, build-docker, read-config]
211    runs-on: ubuntu-latest
212    steps:
213      - uses: actions/checkout@v3
214      - name: Download macOS artifact
215        uses: actions/download-artifact@v4
216        with:
217          name: Biom3d-MacOS
218          path: mac
219
220      - name: Download Windows artifact
221        uses: actions/download-artifact@v4
222        with:
223          name: Biom3d-Windows
224          path: win
225
226      # - name: Zip source code # Not necessary
227      #  run: |
228      #    git archive --format zip --output source.zip HEAD
229
230      - name: Extract changelog for version
231        id: changelog
232        run: |
233          VERSION="${GITHUB_REF#refs/tags/}"
234          if grep -q "## \[$VERSION\]" CHANGELOG.md; then
235            awk "/## \[$VERSION\]/ {flag=1; next} /^## \[/ {flag=0} flag" CHANGELOG.md > changelog.md
236          else
237            echo "Version $VERSION not found in CHANGELOG.md" > changelog.md
238          fi
239
240      - name: Create GitHub Release
241        uses: softprops/action-gh-release@v1
242        with:
243          tag_name: ${{ github.ref_name }}
244          name: Release ${{ github.ref_name }}
245          body_path: changelog.md
246          files: |
247            mac/Biom3d_MacOS_*.zip
248            mac/Biom3d_MacOS_*_Remote
249            win/Biom3d_Windows_*.zip
250            win/Biom3d_Windows_*_Remote.exe
251
252        #    source.zip
253        env:
254          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The pipeline is composed of 5 jobs :

  • Retrieve Docker information from a JSON file

  • Create and push Docker images

  • Building on macOS

  • Building on Windows

  • Create a GitHub release

Retrieving Docker info

Reading Json files is natively supported in GitHub Actions, so we can just read it and store it in a GitHub variable. A YAML format was considered to make the file more editable, but it would have introduced unnecessary complexity.

Creating Docker Images

It is run on ubuntu-latest as it has Docker pre-installed. We use the JSON parsed by the previous step to determine build argument and other parameters.

Here is the list of supported keys:

  • architecture: Processor architecture

  • torch_version: PyTorch version to install and use. If using a PyTorch base image, it should match this version. It is used in the name of the image.

  • base_image : Base Docker Image used to build from.

  • python_version : Python version used in the image (3.11 is recommended)

  • omero_version : OMERO version used in the image (5.21.0 is recommended)

  • target : Must be either "cpu" or "gpu", it is mandatory only if "cpu" value to determine the name and prevent creating a symlink for non existing GPU library.

  • cuda_version : CUDA version to use. If using a PyTorch base image, it should match this version. It is used in the name of the image. Don’t use if target:"cpu".

  • cudnn_version : cuDNN version to use. If using a PyTorch base image, it should match this version. It is used in the name of the image. Don’t use if target:"cpu".

  • tested : Either 1 (image has been fully tested and validated) or 0 (not tested, or only partially).

Here is an example Json :

{
  "configs": [
    {
      "architecture": "x86_64",
      "torch_version": "2.7.1",
      "base_image": "ubuntu:22.04",
      "python_version": "3.11",
      "omero_version": "5.21.0",
      "target": "cpu",
      "tested":1
    },
    {
      "architecture": "x86_64",
      "torch_version": "2.3.1",
      "cuda_version": "11.8",
      "cudnn_version": "8",
      "base_image": "pytorch/pytorch:2.3.1-cuda11.8-cudnn8-runtime",
      "python_version": "3.11",
      "omero_version": "5.21.0",
      "tested":1
    }]
}

As shown above, the file consists of a “configs” array, where each object defines one image to build using the parameters described.

Each image defined in the JSON file is built the pushed on DockerHub using DOCKER_USERNAME and DOCKER_PASSWORD (or more precisely access toker) stored in GitHub Secrets.

For security reason, we recommend :

  • Restricting access to those variable to trusted collaborators

  • Change the access token on a regular basis.

Building

Here we will describe the differents steps of building. The overall logic the same accross plateforms, but syntaxe vary depending on the OS. We will go into syntax details only where necessary.

Keep in mind that the build is done in the GITHUB_WORKSPACE, that is a the root of the GitHub repository.

Steps

  • Versioning on macOS : macOS has a additional step, it is to retrieve the release version and modify the CFBundleVersion in Info.plist, so the application metadata reflects the correct version.

  • Conda installation : We install Conda and initialize it. On Windows we use Invoke-WebRequest instead of curl because :

    • In PowerShell, curl is an alias of Invoke-WebRequest.

    • Using a Bash terminal to run curl causes permission issues. We add Conda to the path, on MacOS, we also add it to the GITHUB_ENV variable so it is correctly transfered between steps.

  • Architecture detection : We retrive the runner architecture and store it in a GITHUB_OUTPUT variable. On Windows, we added a switch that replace "AMD64" with "x86_64" to avoid ambiguity.

  • Packaging : We move to deployment/exe/os directory and use the packing script described in the installer documentation. On macOS we have to reactivate Conda.

  • Setting up remote : Once the installer is created, we prepare the remote version by switching the REMOTE=False to REMOTE=True in gui.py. This is intended to let pyinstaller statically determine which imports are required (although it doesn’t fully work), but more importantly, it hides the “Start locally” option in the GUI.

  • Creating a minimal pyvenv : Instead of using the existing Conda environment, we create a minimal venv.
    This significantly reduces the final build size — from around 200MB to 15MB.
    We then use pyinstaller to build gui.py.
    Note: For unknown reasons, pyinstaller places the executable outside the GITHUB_WORKSPACE, so we manually copy it to the root for access in later steps.

  • Creating the build artifact : We create a folder containing both the installer and the remote executable, and place it at the repository root (it simplify the path for future use). This folder is then exported as a GitHub artifact.

This concludes the build process.

Release

This step is straightforward and only runs after all other jobs have completed successfully.

  • It will download MacOS and Windows artifacts.

  • It extracts the corresponding changelog section from CHANGELOG.md, based on the pushed tag.

    Important: CHANGELOG.md must include a section matching the pushed tag, for example:
    ## [v1.0.0] - 2025-July-9 If this part fail, it will create a release with the following body Version $VERSION not found in CHANGELOG.md.

  • The CI/CD contain a step to zip the source code however it was commented as GitHub automatically does it.

  • It create a release associated to the tag, it includes :

    • The changelog as body.

    • The two installers.

    • The two remote executable.

    • Source code archived (in .zip and .tar.gz).