From dd53c45a2cb46f882a856869de26215942749cba Mon Sep 17 00:00:00 2001 From: Markus Heiser Date: Mon, 30 Mar 2020 18:47:01 +0200 Subject: [PATCH] docs: add utils/lxc.sh docs, normalize filtron, morty & searx docs Signed-off-by: Markus Heiser --- docs/utils/filtron.sh.rst | 25 ++++++------- docs/utils/index.rst | 1 + docs/utils/lxc.sh.rst | 79 +++++++++++++++++++++++++++++++++++++++ docs/utils/morty.sh.rst | 11 +++++- docs/utils/searx.sh.rst | 77 +++++++++++++++++++++----------------- utils/filtron.sh | 2 +- utils/lxc.sh | 7 ++++ utils/searx.sh | 23 ++++++------ 8 files changed, 164 insertions(+), 61 deletions(-) create mode 100644 docs/utils/lxc.sh.rst diff --git a/docs/utils/filtron.sh.rst b/docs/utils/filtron.sh.rst index e6d7d6251..40e57aa3c 100644 --- a/docs/utils/filtron.sh.rst +++ b/docs/utils/filtron.sh.rst @@ -27,8 +27,8 @@ The script adds a ``${SERVICE_USER}`` (default:``filtron``) and installs filtron into this user account: #. Create a separated user account (``filtron``). -#. Download and install Go_ binary in users $HOME (``~filtron``). -#. Install filtron with the package management of Go_ (``go get -v -u +#. Download and install Go_ binary in user's $HOME (``~filtron``). +#. Install filtron with the package management from Go_ (``go get -v -u github.com/asciimoo/filtron``) #. Setup a proper rule configuration :origin:`[ref] ` (``/etc/filtron/rules.json``). @@ -36,6 +36,16 @@ into this user account: ` (``/lib/systemd/system/filtron.service``). +.. _filtron.sh overview: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory +(:ref:`toolboxing common`): + +.. program-output:: ../utils/filtron.sh --help + .. _reverse proxy: Public Reverse Proxy @@ -63,14 +73,3 @@ To install searx in your public HTTP server use: $ sudo -H a2enmod headers $ sudo -H a2enmod proxy $ sudo -H a2enmod proxy_http - -.. _filtron.sh overview: - -Overview -======== - -The ``--help`` output of the script is largely self-explanatory -(:ref:`toolboxing common`): - -.. program-output:: ../utils/filtron.sh --help - diff --git a/docs/utils/index.rst b/docs/utils/index.rst index 1c77cbfcd..088586e43 100644 --- a/docs/utils/index.rst +++ b/docs/utils/index.rst @@ -15,6 +15,7 @@ developers. searx.sh filtron.sh morty.sh + lxc.sh .. _toolboxing common: diff --git a/docs/utils/lxc.sh.rst b/docs/utils/lxc.sh.rst new file mode 100644 index 000000000..68577ea13 --- /dev/null +++ b/docs/utils/lxc.sh.rst @@ -0,0 +1,79 @@ + + +.. _snap: https://snapcraft.io +.. _snapcraft LXD: https://snapcraft.io/lxd +.. _LXC/LXD Image Server: https://uk.images.linuxcontainers.org/ +.. _LXC: https://linuxcontainers.org/lxc/introduction/ +.. _LXD: https://linuxcontainers.org/lxd/introduction/ +.. _`LXD@github`: https://github.com/lxc/lxd + +.. _lxc.sh: + +================ +``utils/lxc.sh`` +================ + +.. sidebar:: further reading + + - snap_, `snapcraft LXD`_ + - LXC_, LXD_ + - `LXC/LXD Image Server`_ + - `LXD@github`_ + +With the use of *Linux Containers* (LXC_) we can scale our tasks over a stack of +containers, what we call the: *lxc suite*. Before you can start with +containers, you need to install and initiate LXD_ once:: + + $ snap install lxd + $ lxd init --auto + +The *searx suite* (:origin:`lxc-searx.env `) is loaded by +default, every time you start the ``lxc.sh`` script (you do not need to care +about). To make use of the containers from the *searx suite*, you have to build +the :ref:`LXC suite containers ` first. But be warned, this +might take some time:: + + $ sudo -H ./utils/lxc.sh build + +A cup of coffee later, your LXC suite is build up and you can run whatever task +you want / in a selected or even in all :ref:`LXC suite containers `. Each container shares the root folder of the repository and the +command ``utils/lxc.sh cmd`` handles relative path names *transparent*:: + + $ sudo -H ./utils/lxc.sh cmd -- ls -la Makefile + ... + [searx-ubu2004] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile + [searx-fedora31] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile + [searx-archlinux] -rw-r--r-- 1 root root 7603 Mar 30 11:54 Makefile + +With this in mind, you can run :ref:`searx.sh` and install packages, needed by +searx:: + + $ sudo -H ./utils/lxc.sh cmd -- ./utils/searx.sh install packages + +And run one of the :origin:`Makefile` targets:: + + $ sudo -H ./utils/lxc.sh cmd -- make test.sh + +You can install a *buildhost environment* into the containers (time for another +cup of coffee):: + + $ sudo -H ./utils/lxc.sh install buildhost + +If you want to get rid off all the containers, just type:: + + $ sudo -H ./utils/lxc.sh remove + +To clean up your local images use:: + + $ sudo -H ./utils/lxc.sh remove images + +.. _lxc.sh --help: + +Overview +======== + +The ``--help`` output of the script is largely self-explanatory: + +.. program-output:: ../utils/lxc.sh --help + diff --git a/docs/utils/morty.sh.rst b/docs/utils/morty.sh.rst index 5ab6ee982..2bdc4d9c8 100644 --- a/docs/utils/morty.sh.rst +++ b/docs/utils/morty.sh.rst @@ -1,6 +1,7 @@ .. _morty: https://github.com/asciimoo/morty .. _morty's README: https://github.com/asciimoo/morty +.. _Go: https://golang.org/ .. _morty.sh: @@ -22,7 +23,15 @@ running the command: sudo -H ./utils/morty.sh install all The script adds a ``${SERVICE_USER}`` (default:``morty``) and installs morty_ -into this user account. +into this user account: + +#. Create a separated user account (``morty``). +#. Download and install Go_ binary in user's $HOME (``~morty``). +#. Install morty_ with the package management from Go_ (``go get -v -u + github.com/asciimoo/morty``) +#. Setup a systemd service unit :origin:`[ref] + ` + (``/lib/systemd/system/morty.service``). .. hint:: diff --git a/docs/utils/searx.sh.rst b/docs/utils/searx.sh.rst index 988c08438..905f765dd 100644 --- a/docs/utils/searx.sh.rst +++ b/docs/utils/searx.sh.rst @@ -29,41 +29,6 @@ into this user account. The installation is described in chapter .. _intranet reverse proxy: -Intranet Reverse Proxy -====================== - -To install searx in your intranet HTTP server use: - -.. code:: bash - - sudo -H ./utils/searx.sh apache install - -.. warning:: - - This setup is **not** suitable **for public instances**, go on with - :ref:`reverse proxy`! - -.. tabs:: - - .. group-tab:: apache - - .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:uwsgi - :language: apache - - .. tabs:: - - .. group-tab:: Ubuntu / debian - - .. code-block:: sh - - $ sudo -H apt install libapache2-mod-uwsgi - - .. group-tab:: Arch Linux - - .. code-block:: sh - - $ sudo pacman -S uwsgi - Overview ======== @@ -71,3 +36,45 @@ The ``--help`` output of the script is largely self-explanatory (:ref:`toolboxing common`): .. program-output:: ../utils/searx.sh --help + + +Intranet Reverse Proxy +====================== + +.. warning:: + + This setup is **not** suitable **for public instances**, go on with + :ref:`reverse proxy`! + +To install searx in your intranet HTTP server use: + +.. code:: bash + + sudo -H ./utils/searx.sh apache install + +.. tabs:: + + .. group-tab:: apache + + .. literalinclude:: ../../utils/templates/etc/apache2/sites-available/searx.conf:uwsgi + :language: apache + + .. tabs:: + + .. group-tab:: Ubuntu / debian + + .. code-block:: sh + + $ sudo -H apt install libapache2-mod-uwsgi + + .. group-tab:: Arch Linux + + .. code-block:: sh + + $ sudo -H pacman -S uwsgi + + .. group-tab:: Fedora / RHEL + + .. code-block:: sh + + $ sudo -H dnf install uwsgi diff --git a/utils/filtron.sh b/utils/filtron.sh index c81a252a5..208166fda 100755 --- a/utils/filtron.sh +++ b/utils/filtron.sh @@ -90,7 +90,7 @@ inspect service show service status and log option set one of the available options -apache : ${PUBLIC_URL} +apache (${PUBLIC_URL}) :install: apache site with a reverse proxy (ProxyPass) :remove: apache site ${APACHE_FILTRON_SITE} diff --git a/utils/lxc.sh b/utils/lxc.sh index b03ce2af8..2526a257d 100755 --- a/utils/lxc.sh +++ b/utils/lxc.sh @@ -324,6 +324,13 @@ lxc_delete_images_localy() { lxc_delete_local_image "$i" done fi + + for i in $(lxc image list --format csv | grep '^,' | sed 's/,\([^,]*\).*$/\1/'); do + if ask_yn "Image $i has no alias, do you want to delete the image?" Yn; then + lxc_delete_local_image "$i" + fi + done + echo lxc image list local: } diff --git a/utils/searx.sh b/utils/searx.sh index a6f2d5ee6..71ab978a1 100755 --- a/utils/searx.sh +++ b/utils/searx.sh @@ -44,30 +44,33 @@ SEARX_UWSGI_SOCKET="/run/uwsgi/app/searx/socket" SEARX_PACKAGES_debian="\ python3-dev python3-babel python3-venv uwsgi uwsgi-plugin-python3 -git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev" +git build-essential libxslt-dev zlib1g-dev libffi-dev libssl-dev +shellcheck" BUILD_PACKAGES_debian="\ -shellcheck graphviz imagemagick texlive-xetex librsvg2-bin +graphviz imagemagick texlive-xetex librsvg2-bin texlive-latex-recommended texlive-extra-utils ttf-dejavu" # pacman packages SEARX_PACKAGES_arch="\ python python-pip python-lxml python-babel uwsgi uwsgi-plugin-python -git base-devel libxml2" +git base-devel libxml2 +shellcheck" BUILD_PACKAGES_arch="\ -shellcheck graphviz imagemagick texlive-bin extra/librsvg +graphviz imagemagick texlive-bin extra/librsvg texlive-core texlive-latexextra ttf-dejavu" # dnf packages SEARX_PACKAGES_fedora="\ python python-pip python-lxml python-babel uwsgi uwsgi-plugin-python3 -git @development-tools libxml2" +git @development-tools libxml2 +ShellCheck" BUILD_PACKAGES_fedora="\ -ShellCheck graphviz graphviz-gd ImageMagick librsvg2-tools +graphviz graphviz-gd ImageMagick librsvg2-tools texlive-xetex-bin texlive-collection-fontsrecommended texlive-collection-latex dejavu-sans-fonts dejavu-serif-fonts dejavu-sans-mono-fonts" @@ -76,23 +79,21 @@ case $DIST_ID in ubuntu|debian) SEARX_PACKAGES="${SEARX_PACKAGES_debian}" BUILD_PACKAGES="${BUILD_PACKAGES_debian}" + APACHE_APT_PACKAGES="libapache2-mod-uwsgi" ;; arch) SEARX_PACKAGES="${SEARX_PACKAGES_arch}" BUILD_PACKAGES="${BUILD_PACKAGES_arch}" + APACHE_APT_PACKAGES="uwsgi" ;; fedora) SEARX_PACKAGES="${SEARX_PACKAGES_fedora}" BUILD_PACKAGES="${BUILD_PACKAGES_fedora}" + APACHE_APT_PACKAGES="uwsgi" ;; esac # Apache Settings - -APACHE_APT_PACKAGES="\ - libapache2-mod-uwsgi \ -" - APACHE_SEARX_SITE="searx.conf" # shellcheck disable=SC2034