Main Setup

Server Setup

Steps that were taken to install and configure the software to run this service.

1. IS set up a VM in the DMZ named docs.mbari.org.

2. IS created a docsadm local account so that the software running on this machine would be independent of any particular user.

3. We wanted to use the Go-based adnanh/webhook tool, so, we first installed Go:

   Note

   We could have also installed the webhook tool directly from the available binary releases.

   • Logged in as docsadm, in the home directory, we ran:

     (cd Downloads && wget 'https://golang.org/dl/go1.15.5.linux-amd64.tar.gz')
     tar -C .local -xzf Downloads/go1.15.5.linux-amd64.tar.gz
     mkdir gowork
     

     Note this is not a system-wide install but just under .local/go

   • Added the following to ~/.bash_profile:

     export GOROOT=$HOME/.local/go
     PATH=$GOROOT/bin:$PATH
     export GOPATH=$HOME/gowork
     PATH=$GOPATH/bin:$PATH
     

4. With Go in place, we built and installed the adnanh/webhook tool:

   [docsadm@docs ~]$ go get github.com/adnanh/webhook
   [docsadm@docs ~]$ go build github.com/adnanh/webhook
   [docsadm@docs ~]$ gowork/bin/webhook --version
   webhook version 2.7.0
   

5. To get the webhook working so that it could respond to requests from Git repositories, it needs a configuration file and a script to run. This has been done as follows.

6. The directory /home/docsadm/mkdocs/ is designated as the base location for the clones of all documentation sites to be managed by the docs-mbari-org-webhook-doc tool.

   mkdir /home/docsadm/mkdocs
   

7. This "docs-mbari-org-webhook-doc" repo itself has been cloned there:

   cd /home/docsadm/mkdocs
   git clone git@github.com/mbari-org/docs-mbari-org-webhook-doc.git
   

   This bootstrapping mechanism will allow not only to update the documentation of the tool as any other regular site, but also to automatically reflect changes in the generation scripts.

8. The docs-mbari-org-webhook-doc clone contains a hooks.json file, which is passed as an argument to the webhook tool (see cronjob below). This file configures the webhook and links it to the command it should execute when called. It also parses the incoming request, looks for certain payload parameters and then passes those as environment variables to the command.

9. We wrote webhook.sh, the initial core script to dispatch the documentation generation. (This was later on replaced with a python implementation.)

10. With all these pieces in place, the adnanh/webhook tool is started at boot time via cron:

    PATH=/home/docsadm/gowork/bin:/home/docsadm/.local/go/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/docsadm/.local/bin:/home/docsadm/bin
    
    # Webhook service at boot time:
    @reboot webhook -port 9001 -verbose -hooks /home/docsadm/mkdocs/docs-mbari-org-webhook-doc/hooks.json > /var/www/html/internal/_/_log 2>&1
    

    This starts the webhook application pointing it to the hooks.json file we indicated earlier, and redirects all output to /var/www/html/internal/_/_log (so, accessible at https://docs.mbari.org/internal/_/_log).

11. The last installation step in getting all this to work was to set up an ssh key and adding it to a public key on an account in BitBucket. You basically run an 'ssh-keygen' and then put the public portion of the key into an account on BitBucket that can read from all the public and private repos.

12. We then set up a ProxyPass for our Apache server to connect the service to the outside world. This was done in the /etc/httpd/conf.d/ssl.conf file:

    <Location /dochook/>
        ProxyPass        http://localhost:9001/hooks/
        ProxyPassReverse http://localhost:9001/hooks/
    </Location>
    

13. Miniconda was installed to support a python environment with python 3.7, instead of 3.6 per the default installed with the OS. Installation included downloading the miniconda installer from the official website, then making it executable, creating a mkdocs environment, and installing the dependencies used in documentation generation, e.g.

    chmod 0755 ./miniconda_install.sh
    ./miniconda_install.sh
    ~/miniconda3/bin/conda create --name mkdocs python=3.7
    ~/miniconda3/bin/conda init bash
    ~/miniconda3/bin/conda activate mkdocs
    pip install mkdocs mkdocs-material mkdocs-material-extensions
    

    Once this is done, the environment was setup to be activated in the mkdocs service by adding the following to the webhook.sh script:

    log "activating conda environment"
    set -e
    source /home/docsadm/miniconda3/etc/profile.d/conda.sh
    conda activate mkdocs
    set +e
    source /home/docsadm/miniconda3/etc/profile.d/conda.sh
    

    NOTE: Such webhook.sh was later on replaced with a python implementation.

Deployment of this generator

As indicated above, this tool bootstraps itself, not only in terms of documentation, but in terms of the associated scripts so they get updated upon pushes to the docs-mbari-org-webhook-doc repository. Of course, any script changes will actually have effect upon a subsequent execution.

Some other installs

Note that initial installations (e.g., mkdocs itself) were done by IS or via sudo. Later on, we opted to continue using pip along with the --user option to install mkdocs plugins as well as other python dependencies. Since the docs machine is exclusively for documentation purposes, setting a virtualenv was not considered a concern.

• mermaid: pip3 install --user mkdocs-mermaid2-plugin

• jq: From https://stedolan.github.io/jq/

• HTTPie: To support pushing some logging to Slack: pip3 install --user https://github.com/httpie/httpie/archive/master.tar.gz

• filelock: pip3 install --user filelock

• 2021-07-20: Upon noting no effect while trying dark mode in one of the sites, (see this), mkdocs-material upgraded: pip3 install --upgrade --user mkdocs-material (to 7.1.11)

• 2021-08-16: Trying to use conda environment for python 3.7.

• 2021-12-02: Installs toward a more python based generation dispatch: sh and requests. Due to the recent conda environment, these were done as follows:

  (base) [docsadm@docs docs-mbari-org-webhook-doc]$ source /home/docsadm/miniconda3/etc/profile.d/conda.sh
  (base) [docsadm@docs docs-mbari-org-webhook-doc]$ conda activate mkdocs
  (mkdocs) [docsadm@docs docs-mbari-org-webhook-doc]$ conda install sh requests
  

• 2024-05-20: Several upgrades

  • mkdocs conda environment upgraded to Python 3.10.14 (and incompatible Numba 0.46.0 upgraded to 0.59.1): conda install python=3.10.14 numba=0.59.1
  • mkdocstrings-python recipe installed from conda-forge: conda install -c conda-forge mkdocstrings-python
  • Pip packages re-installed manually: pip install pygments==2.12 mkdocs-awesome-pages-plugin mkdocs-jupyter mkdocs-material mkdocs-mermaid2-plugin deepsea-ai

• 2024-05-22: In coordination with the CoMPAS Lab, the mbaridocs GitHub "bot" account has been added to the CoMPASLab/dial repo such that docs generation continues to be supported for Dial.

• 2025-03-10: To facilitate integration of API documentation for python projects: conda install -c conda-forge mkdocstrings mkdocstrings-python

Slack webhook

A Slack webhook has been enabled, which is used to expose some logging from the script to the docs-mbari-log channel: https://mbari.slack.com/archives/C01P32T9PDH. This should facilitate debugging related with doc generation/deployment.

Site listing

A generate_site_list.py script has been added to help generate the index.html file for the two main entry points:

• Public sites: https://docs.mbari.org/
• Internal sites: https://docs.mbari.org/internal/

Search field. As explained in this section, the generated index.html file includes a search field that allows to search across all listed sites.

The generated index.html uses https://docs.mbari.org/css/mbari-site-list.css for styling.

The script is launched for the two listings with the helper generate_site_lists_cronjob.sh, which is run by the main generator script right before exiting (so the listings get updated right after any particular site is updated), and also via cronjob:

*/5 * * * * /home/docsadm/mkdocs/docs-mbari-org-webhook-doc/bin/generate_site_lists_cronjob.sh

SSH Keys

We have enabled support for private Bitbucket and GitHub repositories. This is accomplished by using a corresponding SSH Key for the required access. The setup is explained below.

Bitbucket

TODO

GitHub

Ref: https://docs.github.com/en/developers/overview/managing-deploy-keys#machine-users

• Email address created: docsbot@mbari.org (at the moment as a 'distribution list," with Kevin and Carlos as members)
• GitHub account created: mbaridocs: https://github.com/mbaridocs
• Created and uploaded an SSH key for mbaridocs at GitHub
• Added mbaridocs to the mbari-org organization
• As member of the mbari-org the organization, mbaridocs now has (read) access to all private repos there: https://github.com/orgs/mbari-org/people/mbaridocs
• This completes the setup as needed.