1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
|
<!-- Do not edit this file directly, edit its companion .md instead
and regenerate this file using nixos/doc/manual/md-to-db.sh -->
<chapter xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xml:id="module-services-garage">
<title>Garage</title>
<para>
<link xlink:href="https://garagehq.deuxfleurs.fr/">Garage</link> is
an open-source, self-hostable S3 store, simpler than MinIO, for
geodistributed stores. The server setup can be automated using
<link linkend="opt-services.garage.enable">services.garage</link>. A
client configured to your local Garage instance is available in the
global environment as <literal>garage-manage</literal>.
</para>
<para>
The current default by NixOS is <literal>garage_0_8</literal> which
is also the latest major version available.
</para>
<section xml:id="module-services-garage-upgrade-scenarios">
<title>General considerations on upgrades</title>
<para>
Garage provides a cookbook documentation on how to upgrade:
<link xlink:href="https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/">https://garagehq.deuxfleurs.fr/documentation/cookbook/upgrading/</link>
</para>
<warning>
<para>
Garage has two types of upgrades: patch-level upgrades and
minor/major version upgrades.
</para>
<para>
In all cases, you should read the changelog and ideally test the
upgrade on a staging cluster.
</para>
<para>
Checking the health of your cluster can be achieved using
<literal>garage-manage repair</literal>.
</para>
</warning>
<warning>
<para>
Until 1.0 is released, patch-level upgrades are considered as
minor version upgrades. Minor version upgrades are considered as
major version upgrades. i.e. 0.6 to 0.7 is a major version
upgrade.
</para>
</warning>
<itemizedlist spacing="compact">
<listitem>
<para>
<emphasis role="strong">Straightforward upgrades (patch-level
upgrades).</emphasis> Upgrades must be performed one by one,
i.e. for each node, stop it, upgrade it : change
<link linkend="opt-system.stateVersion">stateVersion</link> or
<link linkend="opt-services.garage.package">services.garage.package</link>,
restart it if it was not already by switching.
</para>
</listitem>
<listitem>
<para>
<emphasis role="strong">Multiple version upgrades.</emphasis>
Garage do not provide any guarantee on moving more than one
major-version forward. E.g., if you’re on
<literal>0.7</literal>, you cannot upgrade to
<literal>0.9</literal>. You need to upgrade to
<literal>0.8</literal> first. As long as
<link linkend="opt-system.stateVersion">stateVersion</link> is
declared properly, this is enforced automatically. The module
will issue a warning to remind the user to upgrade to latest
Garage <emphasis>after</emphasis> that deploy.
</para>
</listitem>
</itemizedlist>
</section>
<section xml:id="module-services-garage-advanced-upgrades">
<title>Advanced upgrades (minor/major version upgrades)</title>
<para>
Here are some baseline instructions to handle advanced upgrades in
Garage, when in doubt, please refer to upstream instructions.
</para>
<itemizedlist spacing="compact">
<listitem>
<para>
Disable API and web access to Garage.
</para>
</listitem>
<listitem>
<para>
Perform
<literal>garage-manage repair --all-nodes --yes tables</literal>
and
<literal>garage-manage repair --all-nodes --yes blocks</literal>.
</para>
</listitem>
<listitem>
<para>
Verify the resulting logs and check that data is synced
properly between all nodes. If you have time, do additional
checks (<literal>scrub</literal>,
<literal>block_refs</literal>, etc.).
</para>
</listitem>
<listitem>
<para>
Check if queues are empty by
<literal>garage-manage stats</literal> or through monitoring
tools.
</para>
</listitem>
<listitem>
<para>
Run <literal>systemctl stop garage</literal> to stop the
actual Garage version.
</para>
</listitem>
<listitem>
<para>
Backup the metadata folder of ALL your nodes, e.g. for a
metadata directory (the default one) in
<literal>/var/lib/garage/meta</literal>, you can run
<literal>pushd /var/lib/garage; tar -acf meta-v0.7.tar.zst meta/; popd</literal>.
</para>
</listitem>
<listitem>
<para>
Run the offline migration:
<literal>nix-shell -p garage_0_8 --run "garage offline-repair --yes"</literal>,
this can take some time depending on how many objects are
stored in your cluster.
</para>
</listitem>
<listitem>
<para>
Bump Garage version in your NixOS configuration, either by
changing
<link linkend="opt-system.stateVersion">stateVersion</link> or
bumping
<link linkend="opt-services.garage.package">services.garage.package</link>,
this should restart Garage automatically.
</para>
</listitem>
<listitem>
<para>
Perform
<literal>garage-manage repair --all-nodes --yes tables</literal>
and
<literal>garage-manage repair --all-nodes --yes blocks</literal>.
</para>
</listitem>
<listitem>
<para>
Wait for a full table sync to run.
</para>
</listitem>
</itemizedlist>
<para>
Your upgraded cluster should be in a working state, re-enable API
and web access.
</para>
</section>
<section xml:id="module-services-garage-maintainer-info">
<title>Maintainer information</title>
<para>
As stated in the previous paragraph, we must provide a clean
upgrade-path for Garage since it cannot move more than one major
version forward on a single upgrade. This chapter adds some notes
how Garage updates should be rolled out in the future. This is
inspired from how Nextcloud does it.
</para>
<para>
While patch-level updates are no problem and can be done directly
in the package-expression (and should be backported to supported
stable branches after that), major-releases should be added in a
new attribute (e.g. Garage <literal>v0.8.0</literal> should be
available in <literal>nixpkgs</literal> as
<literal>pkgs.garage_0_8_0</literal>). To provide simple upgrade
paths it’s generally useful to backport those as well to stable
branches. As long as the package-default isn’t altered, this won’t
break existing setups. After that, the versioning-warning in the
<literal>garage</literal>-module should be updated to make sure
that the
<link linkend="opt-services.garage.package">package</link>-option
selects the latest version on fresh setups.
</para>
<para>
If major-releases will be abandoned by upstream, we should check
first if those are needed in NixOS for a safe upgrade-path before
removing those. In that case we shold keep those packages, but
mark them as insecure in an expression like this (in
<literal><nixpkgs/pkgs/tools/filesystem/garage/default.nix></literal>):
</para>
<programlisting>
/* ... */
{
garage_0_7_3 = generic {
version = "0.7.3";
sha256 = "0000000000000000000000000000000000000000000000000000";
eol = true;
};
}
</programlisting>
<para>
Ideally we should make sure that it’s possible to jump two NixOS
versions forward: i.e. the warnings and the logic in the module
should guard a user to upgrade from a Garage on e.g. 22.11 to a
Garage on 23.11.
</para>
</section>
</chapter>
|