aboutsummaryrefslogtreecommitdiff
# SPDX-FileCopyrightText: 2020 pukkamustard <pukkamustard@posteo.net>
#
# SPDX-License-Identifier: CC-BY-SA-4.0
#+TITLE: Encoding for Robust Immutable Storage (ERIS)
#+PROPERTY: header-args:scheme :session *eris-repl* :eval never-export

ERIS is an encoding for arbitrary content into uniformly sized encrypted blocks that can be reassembled only in possession of a short /read capability/.

This repository contains the reference Guile implementation.

* Demo

ERIS describes a scheme for splitting up content into uniformly sized encrypted blocks. For reassembling (decoding) the content a special reference is required (the /read capability/).

ERIS does not define storage and transport layer. It relies on a block storage that allows block to be stored and referenced via the hash code of the block itself (content-addressing).

** Encoding

To encode some content with ERIS we use ~eris-encode~:

#+BEGIN_SRC scheme :exports both
(use-modules (eris))

(eris-encode "Hello world!")
#+END_SRC

#+RESULTS:
: #<<read-capability> block-size: 1024 level: 0 root-reference: #vu8(63 254 3 75 10 5 103 7 208 238 26 103 0 126 217 124 236 105 205 75 136 116 101 176 189 63 118 210 40 169 217 105) root-key: #vu8(19 230 50 100 252 2 228 177 6 63 67 102 128 217 107 51 19 246 52 165 17 175 247 177 68 0 59 165 100 99 44 219)>
: ((#vu8(63 254 3 75 10 5 103 7 208 238 26 103 0 126 217 124 236 105 205 75 136 116 101 176 189 63 118 210 40 169 217 105) . #vu8(37 178 171 171 251 214 252 73 148 204 97 93 18 173 141 67 201 218 238 91 176 234 82 10 143 71 63 18 164 32 238 88 19 159 199 99 178 38 150 10 116 192 49 61 44 121 231 148 180 150 229 92 163 134 77 0 113 84 102 208 29 122 104 60 130 160 70 10 45 207 142 165 244 79 16 145 141 87 188 150 77 73 150 162 138 161 99 203 147 230 131 232 60 108 175 44 133 76 33 18 205 252 251 45 111 107 209 218 238 42 180 12 221 47 203 89 58 111 207 61 45 94 12 80 238 36 184 179 30 42 149 240 38 192 190 68 122 117 170 250 22 136 250 170 84 212 253 119 72 5 237 79 76 232 211 132 190 21 207 117 195 250 28 35 86 143 12 146 96 118 94 53 233 171 217 134 135 140 228 40 196 171 68 99 189 249 73 232 58 187 95 111 211 136 16 37 159 129 81 84 0 220 233 49 180 249 8 230 190 44 10 105 203 174 30 240 28 234 198 30 243 146 179 129 181 91 74 59 221 235 63 148 58 137 33 172 45 107 35 239 7 156 101 25 244 254 114 252 207 67 64 92 10 26 204 240 235 155 90 251 24 54 99 218 91 174 61 24 49 96 80 238 124 198 30 168 234 197 176 178 162 211 131 128 66 249 187 241 103 82 55 238 29 114 243 220 150 147 119 207 118 243 252 37 56 37 95 206 23 156 192 179 220 75 141 142 85 105 113 211 43 46 102 151 93 54 204 131 74 189 105 43 89 201 222 79 72 162 94 211 191 249 125 124 248 71 41 158 119 5 247 251 205 108 22 197 43 55 213 116 179 146 195 1 176 136 237 135 60 164 236 15 228 209 13 17 250 220 102 135 195 242 255 88 226 226 42 229 224 163 5 26 235 46 53 142 254 21 207 164 231 43 84 250 227 134 65 45 94 126 54 174 205 197 52 164 9 129 234 49 40 108 103 182 78 78 163 71 147 221 225 148 120 140 74 165 244 204 248 124 218 212 141 13 66 47 224 71 226 201 178 79 226 35 252 108 26 229 136 228 5 10 75 159 185 189 46 239 37 61 122 157 115 47 145 11 140 39 234 250 142 46 228 184 83 239 66 3 10 48 58 7 169 197 223 107 243 99 65 172 148 172 119 2 248 242 74 42 150 247 52 120 150 123 71 61 243 53 56 99 251 113 119 124 148 150 204 204 105 163 129 72 214 110 234 104 17 214 21 171 95 201 22 62 72 185 192 158 112 18 64 243 47 197 249 239 53 37 159 247 115 115 231 176 79 71 43 239 1 105 76 231 118 36 4 166 251 135 159 116 31 232 112 137 156 114 55 33 154 116 94 231 8 89 131 65 66 210 9 45 249 90 11 70 243 176 174 207 9 8 93 112 4 78 210 32 208 50 49 97 84 122 164 24 157 148 39 109 34 104 64 63 174 201 185 118 222 150 64 64 122 81 195 238 23 171 212 110 59 40 116 200 218 106 182 221 182 246 131 21 39 187 132 134 6 126 60 207 46 48 193 159 140 72 133 156 95 109 226 230 48 180 129 22 46 160 254 91 136 183 168 252 235 3 129 242 198 81 223 139 121 120 127 247 93 13 98 22 219 89 91 47 154 92 69 204 178 233 73 255 170 130 43 32 105 254 175 40 243 120 42 166 190 204 3 173 198 93 197 186 57 192 101 94 206 186 26 196 165 113 39 21 253 31 188 67 162 226 203 133 31 97 139 218 73 177 248 70 224 92 120 28 159 36 139 137 112 47 69 151 127 247 88 255 157 182 57 151 22 19 140 59 146 3 66 219 62 153 194 149 85 117 235 154 181 56 112 15 180 140 139 242 218 198 231 18 176 221 136 201 146 4 26 20 204 60 164 48 206 40 46 194 39 213 139 136 105 174 140 166 51 56 178 17 124 84 208 4 162 120 199 110 248 242 227 136 251 215 144 117 117 23 190 105 23 19 72 25 205 239 91 31 95 17 56 161 83 46 106 60 186 198 233 253 174 43 215 60 208 43 180 150 191 162 185 107 218 143 15 60 20 127 130 65 255 208 199 75 13 41 154 128 41 239 169 89 94 59 109 154 173 28 146 236 84 179 124 92 55 184 74 101 124 50 242 168 242 58 160 9 193 116 119 62 4 58 217 18 237 137 5 217 53 176 78 64 34 178 229 64 52 101 111 62 208 207 242 149 138 0 124 147 55 180 96 212 151 64 110 135 62 248 219 17 243 5 108 224 141 237 155 39 175 205 93 127 42 165 147 88 34 78 171 19 210 72 55 95 131 116 124 37 174 74 195 78 130 247 87 233 54 183 154 160)))

This returns the read capability and the encoded blocks in an alist.

The read capability can be encoded as an URN (string):

#+BEGIN_SRC scheme :exports both
(use-modules (eris read-capability))

(define read-capability
  (call-with-values (lambda () (eris-encode "Hello world!"))
    (lambda (read-capability _) read-capability)))

(read-capability->string read-capability)
#+END_SRC

#+RESULTS:
: urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M

This is very useful for verifying that some content was encoded properly and can be done directly with ~eris-encode->urn~:

#+BEGIN_SRC scheme :exports both
(eris-encode->urn "Hello world!")
#+END_SRC

#+RESULTS:
: urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M

Note that ~eris-encode~ and ~eris-encode->urn~ can encode a string, a bytevector or the content in a port:

#+BEGIN_SRC scheme :exports both
(use-modules (rnrs bytevectors))

(eris-encode->urn (string->utf8 "Hello world!"))
#+END_SRC

#+RESULTS:
: urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M

#+BEGIN_SRC scheme :exports both
(use-modules (rnrs io ports))

(eris-encode->urn (open-bytevector-input-port (string->utf8 "Hello world!")))
#+END_SRC

#+RESULTS:
: urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M

** Decoding

Given a read capability and access to the blocks the content can be decoded into a bytevector with ~eris-decode->bytevector~:

#+BEGIN_SRC scheme :exports both
    (use-modules (ice-9 receive)
                 (rnrs bytevectors))
    
    (define blocks
      (receive (read-capability blocks)
          (eris-encode "Hello world!")
        blocks))
    
    (utf8->string
     (eris-decode->bytevector
      "urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M"
      (lambda (ref) (assoc-ref blocks ref))))
#+END_SRC

#+RESULTS:
: Hello world!

Note that we need to specify how the blocks can be de-referenced. This is because we want to be able to decode with blocks in arbitrary storage.

** Storing blocks

By default ~eris-encode~ returns an alist of references and blocks. Under the hood the SRFI-171 ~rcons~ reducer is used to collect blocks. Any reducer can be given to change what happens to the blocks. For example we can use ~rcount~ to count the number of blocks (but not store the blocks anywhere):

#+BEGIN_SRC scheme :exports both
(use-modules (srfi srfi-171))

(eris-encode "Hello world!"
             #:block-reducer rcount)
#+END_SRC

#+RESULTS:
: #<<read-capability> block-size: 1024 level: 0 root-reference: #vu8(63 254 3 75 10 5 103 7 208 238 26 103 0 126 217 124 236 105 205 75 136 116 101 176 189 63 118 210 40 169 217 105) root-key: #vu8(19 230 50 100 252 2 228 177 6 63 67 102 128 217 107 51 19 246 52 165 17 175 247 177 68 0 59 165 100 99 44 219)>
: 1

The string "Hello world!" is encoded in a single block.

*** IPFS

IPFS can be used to store and transport block of ERIS encoded content. The ~(eris ipfs)~ module provides the necessary interface to the IPFS HTTP API.

After starting the IPFS daemon (with ~ipfs daemon~), content can be encoded with ERIS and stored on IPFS:

#+BEGIN_SRC scheme :exports both
(use-modules (eris)
             (eris ipfs))


(eris-encode "Hello world!"
             #:block-reducer (ipfs-block-reducer))
#+END_SRC

#+RESULTS:
: #<<read-capability> block-size: 1024 level: 0 root-reference: #vu8(63 254 3 75 10 5 103 7 208 238 26 103 0 126 217 124 236 105 205 75 136 116 101 176 189 63 118 210 40 169 217 105) root-key: #vu8(19 230 50 100 252 2 228 177 6 63 67 102 128 217 107 51 19 246 52 165 17 175 247 177 68 0 59 165 100 99 44 219)>
: done

#+BEGIN_SRC scheme :exports both
  (utf8->string
   (eris-decode->bytevector
    "urn:eris:BIAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M"
    ipfs-block-ref))
#+END_SRC

#+RESULTS:
: Hello world!

See the [[./examples/ipfs.org][IPFS write-up]] for details on how IPFS can be used as storage and transport layer.

** Block size and convergence secret

By default content is encoded into blocks of size 1 KiB. Alternatively a block size of 32 KiB can be used. This is useful (and more efficient) for encoding large content.

A convergence secret can also be specified. See the specification for more details.

** Streaming with Transducers

The functions in the module ~(eris)~ (~eris-encode~, ~eris-encode->urn~ and ~eris-decode->bytevector~) use a streaming interface to the encoding provided in ~(eris encode)~ and ~(eris decode)~.

This lower level interface provides an SRFI-171 transducer for encoding content from any stream of data (e.g. a network socket).

* Acknowledgments

guile-eris was initially developed as part of the [[https://openengiadina.net][openEngiadina]] project and has been supported by the [[https:/nlnet][NLNet Foundation]] trough the [[https://nlnet.nl/discovery/][NGI0 Discovery Fund]].

* License

[[./LICENSES/GPL-3.0-or-later.txt][GPL-3.0-or-later]]