Skip to content

Overview

Overview#

Websites#

We have two web sites, as follows.

  • https://kubestellar.io. This is hosted by GoDaddy and administered by Andy Anderson. It contains a few redirects. The most important is that https://kubestellar.io/ redirects to https://docs.kubestellar.io/.
  • https://docs.kubestellar.io. This is a GitHub pages website based on the github.com/kubestellar/kubestellar/ repository.

Also, a contributor has their own copy of the website, at https://${repo_owner}.github.io/${fork_name}.

GitHub pages#

Our documentation is powered by mike and MkDocs. MkDocs is powered by Python-Markdown. These are immensely configurable and extensible. You can see our MkDocs configuration in docs/mkdocs.yml. Following are some of the choices we have made.

  • The MkDocs theme is Material for MkDocs.
  • MkDocs plugin awesome-pages for greater control over how navigation links are shown.
  • MkDocs plugin macros.
  • Our own slightly improved vintage of the include-markdown MkDocs plugin, allowing the source to be factored into re-used files.
  • Python-Markdown extension SuperFences, supporting fenced code blocks that play nice with other markdown features.
  • Python-Markdown extension Highlight, for syntax highlighting of fenced code.
  • Pygments for even fancier code highlighting.
  • MkDocs plugin mkdocs-static-i18n to support multiple languages. We currently only have documentation in English. If you're interested in contributing translations, please let us know!

Serving up documents locally#

You can view and modify our documentation in your local development environment. Simply checkout one of our branches.

git clone git@github.com:kubestellar/kubestellar.git
cd kubestellar/docs
git checkout main

You can view and modify our documentation in the branch you have checked out by using mkdocs serve from mkdocs. We have a Python requirements file in requirements.txt, and a Makefile target that builds a Python virtual environment and installs the requirements there. You can either install those requirements into your global Python environment or use the Makefile target. To install those requirements into your global Python environment, do the following usual thing.

pip install -r requirements.txt

Alternatively, use the following commands to use the Makefile target to construct an adequate virtual environment and enter it.

( cd ..; make venv )
. venv/bin/activate

Then, using your chosen environment with the requirements installed, build and serve the documents with the following command.

mkdocs serve
Then open a browser to http://localhost:8000/

Another way to view (not modify - this method reflects what has been deployed to the gh-pages branch of our repo) all branches/versions of our documentation locally using 'mike' mike for mkdocs:

git clone git@github.com:kubestellar/kubestellar.git
cd kubestellar
git checkout main
cd docs
mike set-default main
cd ..
make serve-docs
Then open a browser to http://localhost:8000/

Supported aliases for our documentation#

mike has a concept of aliases. We currently maintain only one alias.

The publishing workflow updates these aliases. The latest regular release is determined by picking the first version listed by mike list that matches the regexp release-[0-9.]*.

Publishing from the branch named "main"#

The branch named "main" also gets published as a "version" on the website, but with a different name. This is not done by mike aliasing, because that only adds a version. The branch named "main" is published as the version named "unreleased-development".

Shortcut URLs#

We have a few shortcut urls that come in handy when referring others to our project:

note: You need to join our mailing list first to get access to some of the links that follow (https://docs.kubestellar.io/joinus)

and.. the very important… - https://kubestellar.io/quickstart - our 'stable' quickstart

Jinja templating#

Our documentation stack includes Jinja. The Jinja constructs --- {# comment #}, {{ expression }}, and {% statement %} --- can appear in the markdown sources.

File structure#

All documentation-related items live in docs (with the small exception of various make targets and some helper scripts in hack).

The structure of docs is as follows:

Path Description
config/$language/mkdocs.yml Language-specific mkdocs configuration.
content/$language Language-specific website content.
generated/branch All generated content for all languages for the current version.
generated/branch/$language Generated content for a single language. Never added to git.
generated/branch/index.html Minimal index for the current version that redirects to the default language (en)
overrides Global (not language-specific) content.
Dockerfile Builds the kubestellar-docs image containing mkdocs + associated tooling.
mkdocs.yml Minimal mkdocs configuration for mike for multi-version support.
requirements.txt List of Python modules used to build the site.

Global Variables#

There are many global variables defined in the docs/mkdocs.yml. The following are some very common variables you are encouraged to use in our documentation. Use of these variables/macros allows our documentation to have github branch context and take advantage of our evolution without breaking

- site_name: KubeStellar
- repo_url: https://github.com/kubestellar/kubestellar
- site_url: https://docs.kubestellar.io/release-0.23.1
- repo_default_file_path: kubestellar
- repo_short_name: kubestellar/kubestellar
- docs_url: https://docs.kubestellar.io
- repo_raw_url: https://raw.githubusercontent.com/kubestellar/kubestellar
- edit_uri: edit/main/docs/content/
- ks_branch: main
- ks_tag: latest
- ks_latest_regular_release: 0.23.1
- ks_latest_release: 0.23.1

to use a variables/macro in your documentation reference like this:

{{ config.<var_name> }}

and in context that can look something like this:

bash <(curl -s {{ config.repo_raw_url }}/{{ config.ks_branch }}/bootstrap/bootstrap-kubestellar.sh) --kubestellar-version {{ config.ks_tag }}

note:
    - A more extensive and detailed list is located at mkdocs information
    - We also check for broken links as part of our PR pipeline. For more information check out our Broken Links Crawler

The navigation for the documentation is also configured in docs/mkdocs.yml. The section which begins with nav: lays out the navigation structure and which markdown files correspond to each topic.

Page variables#

A markdown source file can contribute additional variables by defining them in name: value lines at the start of the file, set off by lines of triple dashes. For example, suppose a markdown file begins with the following.

---
short_name: example1
manifest_name: 'docs/content/Coding Milestones/PoC2023q1/example1.md'
---

These variables can be referenced as {{ page.meta.short_name }} and {{ page.meta.manifest_name }}.

Including external markdown#

We make extensive use of 'include-markdown' to help us keep our documentation modular and up-to-date. To use 'include-markdown' you must add a block in your document that refers to a block in your external document content:

In your original markdown document, add a block that refers to the external markdown you want to include:

Include Markdown

In the document you want to include, add the start and end tags you configured in the include-markdown block in your original document:

Included Markdown

for more information on the 'include-markdown' plugin for mkdocs look here

Codeblocks#

mkdocs has some very helpful ways to include blocks of code in a style that makes it clear to our readers that console interaction is necessary in the documentation. There are options to include a plain codeblock (```), shell (shell), console (console - no used in our documentation), language or format-specific (yaml, etc.), and others. For more detailed information, checkout the mkdocs information on codeblocks.

NOTE: the docs-ecutable technology does not apply Jinja, at any stage; Jinja source inside executed code blocks will not be expanded by Jinja but rather seen directly by bash.

Here are some examples of how we use codeblocks.

Seen and executed#

For a codeblock that can be 'tested' (and seen by the reader) as part of our CI, use the shell block:
codeblock: 

```shell
mkdocs serve
```
as seen by reader: 
mkdocs serve

Executed but not seen#

(Think hard before hiding stuff from your reader.)

For a codeblock that should be 'tested', BUT not seen by the reader, use the .bash with the plain codeblock, and the '.hide-me' style (great for hiding a sleep command that user does not need to run, but CI does):
codeblock: 

``` {.bash .hide-me}
sleep 10
```
as seen by reader:

Seen but not executed#

(To avoid confusing readers of the HTML, this should be used only for output seen in a shell session.)

For a codeblock that should not be 'tested' as part of our CI, use the .bash with the plain codeblock, and without the '.hide-me' style:
codeblock: 

``` {.bash}
mkdocs server
```
as seen by reader: 
mkdocs server

Seen but not executed and no copy button#

For a codeblock that should not be 'tested', be seen by the reader, and not include a 'copy' icon (great for output-only instances), use the .bash codeblock without the '.no-copy' style:
codeblock: 

``` {.bash .no-copy}
I0412 15:15:57.867837   94634 shared_informer.go:282] Waiting for caches to sync for placement-translator
I0412 15:15:57.969533   94634 shared_informer.go:289] Caches are synced for placement-translator
I0412 15:15:57.970003   94634 shared_informer.go:282] Waiting for caches to sync for what-resolver
```
as seen by reader: 
I0412 15:15:57.867837   94634 shared_informer.go:282] Waiting for caches to sync for placement-translator
I0412 15:15:57.969533   94634 shared_informer.go:289] Caches are synced for placement-translator
I0412 15:15:57.970003   94634 shared_informer.go:282] Waiting for caches to sync for what-resolver

Other language-specific highlighting#

For other language-specific highlighting (yaml, etc.), use the yaml codeblock
codeblock: 

```yaml
nav:
  - Home: index.md
  - QuickStart: Getting-Started/quickstart.md
  - Contributing: 
      - Guidelines: Contribution guidelines/CONTRIBUTING.md
```
as seen by reader: 
nav:
  - Home: index.md
  - QuickStart: Getting-Started/quickstart.md
  - Contributing: 
      - Guidelines: Contribution guidelines/CONTRIBUTING.md

Codeblock with a title#

For a codeblock that has a title, and will not be tested, use the 'title' parameter in conjunction with the plain codeblock (greater for showing or prescribing contents of files):
codeblock: 

``` title="testing.sh"
#!/bin/sh
echo hello KubeStellar
```
as seen by reader:
testing.sh
#!/bin/sh
echo hello KubeStellar

(other variations are possible, PR an update to the kubestellar.css file and, once approved, use the style on the plain codeblock in your documentation.)

Testing/Running Docs#

How do we ensure that our documented examples work? Simple, we 'execute' our documentation in our CI. We built automation called 'docs-ecutable' which can be invoked to test any markdown (.md) file in our repository. You could use it in your project as well - afterall it is opensource.

The way it works:#

  • create your .md file as you normally would
  • add codeblocks that can be tested, tested but hidden, or not tested at all:
    • use 'shell' to indicate code you want to be tested
    • use '.bash' with the plain codeblock, and the '.hide-md' style for code you want to be tested, but hidden from the reader (some like this, but its not cool if you want others to run your instructions without hiccups)
    • use plain codeblock (```) if you want to show sample output that is not to be tested
  • you can use 'include-markdown' blocks, and they will also be executed (or not), depending on the codeblock style you use in the included markdown files.

The GitHub Workflow:#

The original secret sauce:#

  • The original code that made all this possible is at https://github.com/kubestellar/kubestellar/blob/main/docs/scripts/docs-ecutable.sh
    • This code parses the .md file you give it to pull out all the 'shell' and '.bash .hide-me' blocks
    • The code is smart enough to traverse the include-markdown blocks and include the 'shell' and '.bash .hide-me' blocks in them
    • The Jinja constructs are not expanded by this code.
    • It then creates a file called 'generate_script.sh' which is then run at the end of the docs-ecutable execution.

All of this is invoke in a target in our Makefile 

.PHONY: docs-ecutable
docs-ecutable: 
    MANIFEST=$(MANIFEST) docs/scripts/docs-ecutable.sh

You give the path from that follows the 'https://github.com/kubestellar/kubestellar/docs' path, and name of the .md file you want to 'execute'/'test' as the value for the MANIFEST variable:

How to 'make' our docs-ecutable target
make MANIFEST="'docs/content/Getting-Started/quickstart.md'" docs-ecutable

note: there are single and double-quotes used here to avoid issues with 'spaces' used in files names or directories. Use the single and double-quotes as specified in the quickstart example here.

The new and improved secret sauce:#

  • The newer code for executing bash snippets in documentation is at https://github.com/kubestellar/kubestellar/blob/main/docs/scripts/execute-html.sh
    • This code parses the HTML generated by MkDocs to extract all the fenced code blocks tagged for the "shell" language.
    • This HTML scraping is relatively easy because it does not have to work on general HTML but only the HTML generated by our stack from our sources. The use of the option setting pygments_lang_class: true for the Python-Markdown extension pymdownx.highlight plays a critical role, getting the source language into the generated HTML.
    • Because it reads the generated HTML, invisible code blocks are not extracted.
    • Because it reads the generated HTML, the Jinja constructs have their usual effects.
    • This script is given the name of the HTML file to read and the current working directory to establish at the start of the extracted bash.
    • It then creates a file called 'generated_script.sh' which is then run.

All of this is invoked in a target in our Makefile 

.PHONY: execute-html
execute-html: venv
    . $(VENV)/activate; \
    cd docs; \
    mkdocs build; \
    scripts/execute-html.sh "$$PWD/.." "generated/$(MANIFEST)/index.html"

The make target requires the variable MANIFEST to be set to the directory that contains the generated index.html file, relative to 'https://github.com/kubestellar/kubestellar/docs/generated'. This is the name of the markdown source file, relative to 'https://github.com/kubestellar/kubestellar/docs/content' and with the .md extension dropped.

How to 'make' a docs-ecutable target
make MANIFEST="Coding Milestones/PoC2023q1/example1" execute-html

note: this target has no special needs for quoting --- which is not to deny the quoting that your shell needs.

Important files in our gh-pages branch#

index.html and home.html#

These appear in the branch named gh-pages and redirect from the root to the version named latest. The one named index.html is managed by mike set-default. The other should be kept consistent.

both files have content similar to:

index.html and home.html
<!DOCTYPE html>
<html>
<head>
<title>KubeStellar</title>
<meta http-equiv="content-type" content="text/html; charset=utf-8" >
<meta http-equiv="refresh" content="0; URL=https://docs.kubestellar.io/latest" />
</head>

Do not remove these files!

CNAME#

The CNAME file has to be in the gh-pages root to allow github to recognize the url tls cert served by our hosting provider. Do not remove this file!

the CNAME file must have the following content in it:

CNAME
docs.kubestellar.io

versions.json#

The versions.json file contains the version and alias information required by 'mike' to properly serve our doc site. This file is maintained by the 'mike' environment and should not be edited by hand.

[{"version": "release-0.22.0", "title": "release-0.22.0", "aliases": ["latest"]}, {"version": "release-0.22.0-rc3", "title": "release-0.22.0-rc3", "aliases": []}, {"version": "release-0.21.2", "title": "release-0.21.2", "aliases": []}, {"version": "release-0.21.2-rc1", "title": "release-0.21.2-rc1", "aliases": []}, {"version": "release-0.21.1", "title": "release-0.21.1", "aliases": []}, {"version": "release-0.21.0", "title": "release-0.21.0", "aliases": []}, {"version": "release-0.14", "title": "release-0.14", "aliases": []}]

In case of emergency#

If you find yourself in a jam and the pages are not showing up at kubestellar.io or docs.kubestellar.io, check the following 1) Is the index.html, home.html, CNAME, and versions.json file in the gh-pages branch alongside the folders for the compiled documents? If not, then recreate those files as indicated above (except for versions.json which is programmatically created by 'mike'). 2) Is GitHub settings for 'Pages' for the domain pointing at the https://docs.kubestellar.io url? If not, paste it in and check off 'enforce https'. This can happen if the CNAME file goes missing from the gh-pages branch.

How to recreate the gh-pages branch#

To recreate the gh-pages branch, do the following: - checkout the gh-pages branch to your local system 

git clone -b gh-pages https://github.com/kubestellar/kubestellar KubeStellar
cd KubeStellar
- delete all files in the branch and push it to GitHub 
rm -rf *
git add; git commit -m "removing all gh-pages files"; git push -u origin gh-pages
- switch to the 'release' branch, switch to /docs and run 'mike deploy' for the release branch. Add the alias 'latest' for the latest release 
git checkout release-0.22
git pull
mike deploy --push --rebase --update-aliases release-0.22 latest
- switch back to the gh-pages branch and recreate the home.html, index.html, and CNAME files as needed (make sure you back out of the docs path first before switching to gh-pages because that path does not exist in that branch) 
cd ..
git checkout gh-pages
git pull
vi index.html
vi home.html
vi CNAME
- push the new files into gh-pages 
git add .;git commit -m "add index, home, and CNAME files";git push -u origin gh-pages
- go into the GitHub UI and go to the settings for the project and click on 'Pages' to add https://docs.kubestellar.io as the domain and check the box to enforce https.

  • if the above did not work, then you might have an issue with the GoDaddy domain (expired, files missing, etc.)

How to delete a rendering of a branch#

Use mike delete $branch_name, either acting locally on your checked out gh-pages branch (after pull and before git commit and push) or acting more directly on the remote repo using --remote and --push. See the mike delete command doc.

Publishing Workflow#

All documentation building and publishing is done using GitHub Actions in .github/workflows/docs-gen-and-push.yml. This workflow is triggered either manually or by a push to a branch named main or release-<something> or doc-<something>. This workflow will actually do something ONLY if either (a) it is acting on the shared GitHub repository at github.com/kubestellar/kubestellar and on behalf of the repository owner or (b) it is acting on a contributor's fork of that repo and on behalf of that same contributor. The published site appears at https://pages.github.io/kubestellar/${branch} in case (a) and at https://${repo_owner}.github.io/${fork_name}/${branch} in case (b). This workflow will build and publish a website version whose name is the same as the name of the branch that it is working on. This workflow will also update the relevant mike alias, if necessary.