diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..9065b9333 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1 @@ +Follow the style guides in `docs/develop/style/` when editing Salt states, Pillar files, Jinja templates and configuration files. diff --git a/docs/deploy/deploy.rst b/docs/deploy/deploy.rst index 5ecf58b26..abe44550f 100644 --- a/docs/deploy/deploy.rst +++ b/docs/deploy/deploy.rst @@ -29,6 +29,18 @@ The ``state.apply`` function often completes within one minute. ./run.py --state-output=changes 'mytarget-dev' state.apply pillar='{"python_apps":{"myapp":{"git":{"branch":"BRANCH_NAME"}}}}' +.. _deploy-norestart: + +.. tip:: + + To not restart Scrapyd (for example, if spiders are running), run, for example: + + .. code-block:: bash + + ./run.py --state-output=changes 'registry' state.apply pillar='{"scrapyd_norestart":true}' + + This skips all changes to Scrapyd, which the next deployment without ``scrapyd_norestart`` will apply. + Deploy part of a service ~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/docs/deploy/setup.rst b/docs/deploy/setup.rst index 323ca8141..b76211d6d 100644 --- a/docs/deploy/setup.rst +++ b/docs/deploy/setup.rst @@ -1,12 +1,12 @@ Setup ===== -Before performing any deployment task, run the *Setup* tasks below. Once done, run the *Cleanup* tasks below. If you haven't already, please follow the :doc:`../develop/get_started` guide. +Before deploying, run the tasks below. If you haven't already, follow the :doc:`../develop/get_started` guide. 1. Update deploy repositories ----------------------------- -Ensure the ``deploy``, ``pillar/private`` and ``salt/private`` repositories are on the default branch and are up-to-date. You can run this convenience script to run the appropriate ``git`` commands: +Ensure the ``deploy``, ``pillar/private`` and ``salt/private`` repositories are up-to-date and on the default branch: .. code-block:: bash @@ -16,21 +16,29 @@ Check the output in case there are any issues switching to the default branch or .. _check-if-kingfisher-is-busy: -2. Check if Kingfisher is busy +2. Check if the server is busy ------------------------------ .. note:: - Skip this step unless you're working on Kingfisher. + Skip this step for servers other than ``kingfisher-main`` and ``registry``. -#. :ref:`Access Scrapyd's web interface`, click *Jobs* and look under *Running*. If any spiders are running, don't deploy without the consent of data support managers. -#. :doc:`SSH<../use/ssh>` into ``kingfisher-main`` as the ``root`` user. -#. Check if any :ref:`long-running tasks` are running, by attaching to each session in ``tmux`` to see which commands are running. If any commands would be interrupted by the deployment, don't deploy without the consent of the data support managers, who should be identified by the session names. +#. Check whether any spiders are running in Scrapyd, by clicking *Jobs* and looking under *Running*: - To list all sessions: + - ``kingfisher-main``: :ref:`Access Scrapyd's web interface` + - ``registry``: https://collect.data.open-contracting.org/jobs - .. code-block:: bash + If any spiders are running, :ref:`deploy without restarting Scrapyd`. On ``kingfisher-main``, you can alternatively get the consent of the data team. - for i in root $(ls -1 /home); do echo $i; su $i -c "tmux ls"; done +#. On ``kingfisher-main`` only: -#. If the ``postgres`` service would be restarted by the deployment (for example, due to a configuration change or a package upgrade), check if any :ref:`long-running queries` are running. If there are queries with a ``state`` of ``active`` and a ``time`` greater than an hour, don't deploy without the consent of the data support managers, who should be identified by the ``usename``, ``client_addr`` or comment at the start of ``query``. + #. :doc:`SSH<../use/ssh>` into ``kingfisher-main`` as the ``root`` user. + #. Check if any :ref:`long-running tasks` are running, by attaching to each session in ``tmux`` to see which commands are running. If any commands would be interrupted by the deployment, don't deploy without the consent of the data team, who should be identified by the session names. + + To list all sessions: + + .. code-block:: bash + + for i in root $(ls -1 /home); do echo $i; su $i -c "tmux ls"; done + + #. If the ``postgres`` service would be restarted (for example, due to a configuration change or a package upgrade), check if any :ref:`long-running queries` are running. If there are queries with a ``state`` of ``active`` and a ``time`` greater than an hour, don't deploy without the consent of the data team, who should be identified by the ``usename``, ``client_addr`` or comment at the start of ``query``. diff --git a/docs/develop/style/language.rst b/docs/develop/style/language.rst index e3a2a1ae3..0e4fa45db 100644 --- a/docs/develop/style/language.rst +++ b/docs/develop/style/language.rst @@ -48,10 +48,21 @@ To concatenate strings, use the ``~`` operator, which converts its operands to s \+ ['"]|['"] \+ +Optional boolean keys +~~~~~~~~~~~~~~~~~~~~~ + +To test whether an optional boolean is true, use the ``.get()`` method: + +.. code-block:: jinja + + {% if pillar.parent.get('enabled') %} + +Add a new boolean key to the regular expression under *Optional mapping keys*. + Optional mapping keys ~~~~~~~~~~~~~~~~~~~~~ -To test whether a key is present in a mapping, use the ``in`` operator: +To test whether a non-boolean key is present in a mapping, use the ``in`` operator: .. code-block:: jinja @@ -59,11 +70,11 @@ To test whether a key is present in a mapping, use the ``in`` operator: .. note:: - Maintainers can check this style rule with this regular expression: + Maintainers can check this style rule with this regular expression, which excludes boolean keys: .. code-block:: none - if .*get\(.(?!(?:autoremove|compilemessages|enabled|public_access|replication|smartmon|summarystats)\b) + if .*get\(.(?!(?:ENABLE_CHECKER|autoremove|cardinal|enabled|filter|public_access|replication|reports|scrapyd_norestart|site_logs|smartmon|summarystats|syslog_logging|write)\b) To iterate over an optional mapping: @@ -100,15 +111,6 @@ Or: {% for value in salt['pillar.get']('parent:child', []) %} -Optional boolean keys -~~~~~~~~~~~~~~~~~~~~~ - -To test whether an optional boolean is true, use the ``.get()`` method: - -.. code-block:: jinja - - {% if pillar.parent.get('enabled') %} - Optional keys ~~~~~~~~~~~~~ diff --git a/docs/develop/update/wordpress.rst b/docs/develop/update/wordpress.rst index 71331ccdd..3dd530ddd 100644 --- a/docs/develop/update/wordpress.rst +++ b/docs/develop/update/wordpress.rst @@ -182,7 +182,7 @@ WordPress sites: USERNAME: checks: - enabled: + scheduled: - integrity - updates - silent diff --git a/pillar/coalition.sls b/pillar/coalition.sls index 09c1017b0..eef2e2633 100644 --- a/pillar/coalition.sls +++ b/pillar/coalition.sls @@ -53,7 +53,7 @@ wordpress: context: FATHOM_ANALYTICS_ID: LNRZMMVR checks: - enabled: + scheduled: - integrity - updates - silent diff --git a/pillar/corporate.sls b/pillar/corporate.sls index fa89137ec..13fbb8e12 100644 --- a/pillar/corporate.sls +++ b/pillar/corporate.sls @@ -52,7 +52,7 @@ wordpress: GTM_CONTAINER_ID: GTM-WMV9C5F ITEMS_PER_PAGE: 10 checks: - enabled: + scheduled: - integrity premium_plugins: - acfml diff --git a/salt/aws/backup.sls b/salt/aws/backup.sls index e0fd403e8..ed98907c5 100644 --- a/salt/aws/backup.sls +++ b/salt/aws/backup.sls @@ -21,7 +21,7 @@ include: - file: /home/sysadmin-tools/bin/site-backup-to-s3.sh {% for directory, options in pillar.backup.directories|items %} -{% if options and options.get('exclude') %} +{% if options and 'exclude' in options %} set BACKUP_EXCLUDE setting for {{ directory }}: file.keyvalue: - name: /home/sysadmin-tools/aws-settings.local diff --git a/salt/cms/init.sls b/salt/cms/init.sls index 48907af53..cd97e7d6b 100644 --- a/salt/cms/init.sls +++ b/salt/cms/init.sls @@ -110,14 +110,14 @@ set {{ constant }} in {{ user }} wp-config.php: {% endfor %} {% set checks = entry.checks|default({}) %} -{% set enabled = checks.enabled|default([]) %} -{% set parts = ['updates', 'silent']|select('in', enabled)|list %} +{% set scheduled = checks.scheduled|default([]) %} +{% set parts = ['updates', 'silent']|select('in', scheduled)|list %} {% set wp = '/usr/local/bin/wp --no-color --path=' ~ userdir ~ '/public_html' %} # The must-use plugins above have no checksums at wordpress.org. {% set premium = checks.premium_plugins|default([]) %} {% set exclude = premium + mu_plugins|map('regex_replace', '^', 'opencontracting-')|list %} -{% if 'integrity' in enabled %} +{% if 'integrity' in scheduled %} /usr/local/lib/wp-cli/integrity-check-{{ user }}.sh: file.managed: - mode: 755 @@ -140,7 +140,7 @@ set {{ constant }} in {{ user }} wp-config.php: {% endif %} WordPress integrity check for {{ user }}: - cron.{{ 'present' if 'integrity' in enabled else 'absent' }}: + cron.{{ 'present' if 'integrity' in scheduled else 'absent' }}: - name: /usr/local/lib/wp-cli/integrity-check-{{ user }}.sh - identifier: WORDPRESS_INTEGRITY_CHECK - user: {{ user }} diff --git a/salt/docker_apps/files/.env b/salt/docker_apps/files/.env index 13f2055f1..33ace650d 100644 --- a/salt/docker_apps/files/.env +++ b/salt/docker_apps/files/.env @@ -1,5 +1,3 @@ -{%- if entry.get('env') %} -{%- for key, value in entry.env|items %} +{%- for key, value in entry.env|default({})|items %} {{ key }}='{{ value }}' {%- endfor %} -{%- endif %} diff --git a/salt/grafana/init.sls b/salt/grafana/init.sls index 384ecedf7..9870332db 100644 --- a/salt/grafana/init.sls +++ b/salt/grafana/init.sls @@ -36,10 +36,10 @@ grafana: http_addr: '127.0.0.1' root_url: 'https://{{ pillar.apache.sites.grafana.servername }}' users: - allow_sign_up: true + allow_sign_up: True auth.proxy: - enabled: true - auto_sign_up: false + enabled: True + auto_sign_up: False whitelist: 127.0.0.1 - watch_in: - service: grafana diff --git a/salt/kingfisher/collect/init.sls b/salt/kingfisher/collect/init.sls index b892712d0..9c9c8037f 100644 --- a/salt/kingfisher/collect/init.sls +++ b/salt/kingfisher/collect/init.sls @@ -45,6 +45,12 @@ allow {{ userdir }} access: - user: {{ pillar.kingfisher_collect.user }}_user_exists - user: {{ pillar.kingfisher_collect.group }}_user_exists +# Skip the states that restart Scrapyd, so that a later deployment applies their changes and restarts Scrapyd. +{% if pillar.get('scrapyd_norestart') %} +scrapyd-norestart: + test.show_notification: + - text: Skipped Scrapyd states, because scrapyd_norestart is set. +{% else %} {{ directory }}/requirements.txt: file.managed: - source: https://raw.githubusercontent.com/open-contracting/kingfisher-collect/{{ pillar.kingfisher_collect.ref|default('main') }}/requirements.txt @@ -87,6 +93,7 @@ allow {{ userdir }} access: - service: scrapyd {{ systemd({'service': 'scrapyd', 'user': user, 'group': group, 'appdir': directory}) }} +{% endif %} {{ set_cron_env(user, 'MAILTO', 'sysadmin@open-contracting.org', 'kingfisher.collect') }}