Package com.ultikits.ultitools.interfaces.impl.data.json

Class JsonJournal

java.lang.Object
com.ultikits.ultitools.interfaces.impl.data.json.JsonJournal
final class JsonJournal extends Object
The redo journal that makes a multi-file JSON commit crash consistent (20.4-RESEARCH.md section 12.7). One journal serves one identity: the directory whose sub-directories are that identity's tables, each holding one <key>.json file per entity.

Records. A record lists entity after-images (or deletes) for any number of tables. It is written to <seq>.tmp in the journal directory, forced to disk, and renamed atomically to <seq>.commit; the rename is the commit point. A crash before it leaves only the temporary record, which recovery deletes without applying; a crash after it leaves a committed record, which recovery applies whole. Line 1 of a record is its JSON, line 2 the SHA-256 of line 1; the file name is the zero-padded sequence number.

Temporary files never enter a table directory. An operator loads every file of its table directory as an entity, so entity files are written in the journal's own tmp/ directory, forced, and moved into place with an atomic rename. Recovery empties that directory first.

Recovery deletes temporary records and files, applies the committed records in ascending order, forces the touched table directories, then deletes the applied records in ascending order, forcing the journal directory after each deletion. Every record carries full after-images and every entity file is replaced atomically, so a recovery interrupted at any step and run again ends at the same files. A damaged committed record stops recovery before any file changes; it is never deleted, skipped or applied.

Checkpoint writes its own record first, the entity files only after that record is committed, then deletes every record below its number, forces the journal directory, and only then deletes its own record, so no interruption lets a replay bring an older value back over a newer one.

Why deletions are forced one step at a time. Deleting in ascending order is not enough on its own: a file system may make a later unlink durable before an earlier one, so after a power loss an older record could survive while the record that supersedes it is gone, and replay would bring the older values back. The journal therefore forces the journal directory, as a barrier, before it deletes a record whose absence would expose an older one: before the checkpoint's own record, which supersedes every record below it, and, in recovery, before each next record, because a recovered record supersedes only the keys it carries. PostgreSQL keeps the same order: it removes old WAL segments only after the checkpoint that supersedes them is durable, and its durable_unlink (fd.c) fsyncs the parent directory after an unlink because the unlink alone is not durable.

Directory forces. After a rename or a deletion the journal forces the directory that holds it, so the change survives a power loss. A directory the journal creates (the journal directory, a table directory, and any missing parent of either) exists on disk only once the directory holding its entry is forced, so the journal remembers each one and forces its parent at the same points: after a record's rename, and before a checkpoint or recovery relies on its table files. One whose parent force fails stays remembered and is forced again at the next of those points. The journal's own tmp/ directory is not remembered: its files only ever move into a table directory, and the forced table directory is what keeps them. A failed force has one of three outcomes:

  • a directory that cannot be opened for a force (AccessDeniedException, which Windows gives, or UnsupportedOperationException) is skipped, logged once at FINE;
  • any other failure is a real I/O error and is logged at WARNING naming the directory, every time. After a commit record's rename, the commit point, the commit still succeeds: the transaction survives a server crash, but may not survive a power loss or operating-system crash until the next checkpoint completes;
  • everywhere else a real failure stops the operation where it happens and is thrown: a checkpoint never truncates the journal, and a recovery never deletes an applied record, behind a directory that was not forced. The next checkpoint or recovery repeats the work.

Moves. On Windows a move another program briefly holds (an antivirus scanner, an indexer, a backup tool) is retried for at most 150 ms before it fails; elsewhere a move is attempted once.

Threads. nextSequence() may be called from any thread. Several writeRecord(long, Kind, List) calls may run at once if their sequence numbers differ: each touches only its own two record file names. A checkpoint may run alongside them, because it deletes only records numbered at or below its own number, which its caller has captured; the caller runs one checkpoint at a time. recover() runs alone, before the identity is used.

Since:
6.3.0

  • Field Details

    • DIRECTORY_NAME

      static final String DIRECTORY_NAME
      Name of an identity's journal directory.
      See Also:

    • TEMP_DIRECTORY_NAME

      static final String TEMP_DIRECTORY_NAME
      Name of the directory, inside the journal directory, that holds temporary entity files.
      See Also:

    • COMMITTED_SUFFIX

      static final String COMMITTED_SUFFIX
      See Also:

    • TEMPORARY_SUFFIX

      static final String TEMPORARY_SUFFIX
      See Also:

    • FORMAT_VERSION

      static final int FORMAT_VERSION
      Version written into every record's first line.
      See Also:

    • PHASE_RECOVERY

      static final String PHASE_RECOVERY
      Entity-file phase of recover().
      See Also:

    • PHASE_CHECKPOINT

      static final String PHASE_CHECKPOINT
      Entity-file phase of a checkpoint.
      See Also:

    • RECORD_WRITTEN

      static final String RECORD_WRITTEN
      Record steps, prefixed with the record kind's label.
      See Also:

    • RECORD_FORCED

      static final String RECORD_FORCED
      See Also:

    • RENAMED

      static final String RENAMED
      See Also:

    • DIRECTORY_FORCED

      static final String DIRECTORY_FORCED
      See Also:

    • ENTITY_TEMP_WRITTEN

      static final String ENTITY_TEMP_WRITTEN
      Entity-file steps, prefixed with the phase.
      See Also:

    • BETWEEN_ENTITY_FILES

      static final String BETWEEN_ENTITY_FILES
      See Also:

    • TABLE_DIRS_FORCED

      static final String TABLE_DIRS_FORCED
      See Also:

    • RECOVERY_TEMPS_DELETED

      static final String RECOVERY_TEMPS_DELETED
      See Also:

    • RECOVERY_BETWEEN_RECORDS

      static final String RECOVERY_BETWEEN_RECORDS
      See Also:

    • RECOVERY_RECORDS_APPLIED

      static final String RECOVERY_RECORDS_APPLIED
      See Also:

    • RECOVERY_BETWEEN_DELETIONS

      static final String RECOVERY_BETWEEN_DELETIONS
      See Also:

    • CHECKPOINT_BETWEEN_TRUNCATIONS

      static final String CHECKPOINT_BETWEEN_TRUNCATIONS
      See Also:

    • LOGGER

      private static final Logger LOGGER

    • RECORD_NAME

      private static final Pattern RECORD_NAME

    • GSON

      private static final com.google.gson.Gson GSON

    • NO_STEP_HOOK

      private static final JsonJournal.StepHook NO_STEP_HOOK

    • DEFAULT_DIRECTORY_FORCER

      private static final JsonJournal.DirectoryForcer DEFAULT_DIRECTORY_FORCER

    • DEFAULT_MOVER

      private static final JsonJournal.Mover DEFAULT_MOVER

    • MOVE_RETRY_PAUSES_MILLIS

      private static final long[] MOVE_RETRY_PAUSES_MILLIS
      Pauses between the attempts of a retried move, in milliseconds: at most 5 attempts and 150 ms of waiting in all (20.4-DECISIONS.md question 10 A).

    • DEFAULT_MOVE_RETRY

      private static final boolean DEFAULT_MOVE_RETRY
      Whether a failed move is retried on this platform; decided once from os.name.

    • stepHook

      private static volatile JsonJournal.StepHook stepHook
      Test seam: called with a step name at the points named above. A no-op in production; a test installs a callback that pauses, fails or simulates a crash at one step and resets it afterwards.

    • directoryForcer

      private static volatile JsonJournal.DirectoryForcer directoryForcer
      Test seam for the directory force; the default opens the directory and forces it.

    • mover

      private static volatile JsonJournal.Mover mover
      Test seam for every file move the journal makes; the default is Files.move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...).

    • moveRetry

      private static volatile boolean moveRetry
      Test seam: whether a failed move is retried; the default follows the operating system.

    • identityRoot

      private final Path identityRoot

    • directory

      private final Path directory

    • tempDirectory

      private final Path tempDirectory

    • unforcedNewDirectories

      private final Set<Path> unforcedNewDirectories
      Directories this journal created whose entry in their parent directory is not yet known to be on disk. Also the lock that makes creating a directory and remembering it one step.

    • directoryCannotBeOpenedLogged

      private final AtomicBoolean directoryCannotBeOpenedLogged

    • atomicMoveFallbackLogged

      private final AtomicBoolean atomicMoveFallbackLogged

    • moveRetryLogged

      private final AtomicBoolean moveRetryLogged

    • next

      private long next
      The next sequence number to hand out; 0 until seeded from the journal directory.

  • Constructor Details

    • JsonJournal

      JsonJournal(Path identityRoot, Path directory)
      Parameters:
      identityRoot - the directory whose sub-directories are the identity's tables
      directory - the journal directory; created by the first record write

  • Method Details

    • identityRoot

      Path identityRoot()

    • directory

      Path directory()

    • nextSequence

      long nextSequence() throws IOException
      Hands out the next sequence number. The first call seeds the counter from the highest record number in the journal directory (temporary or committed) plus one, so numbers keep increasing across a reopen while records remain. Writing a record is separate (see writeRecord(long, Kind, List)).
      Returns:
      a number greater than every number handed out or on disk before
      Throws:
      IOException - if the journal directory cannot be listed

    • writeRecord

      void writeRecord(long sequence, JsonJournal.Kind kind, List<JsonJournal.Entry> entries) throws IOException
      Writes and commits one record: <seq>.tmp, forced, then renamed atomically to <seq>.commit (the commit point), then the journal directory forced, together with the parent of every directory the journal created and has not yet forced into its parent. Creates the journal directory if it is missing.

      An IOException before the rename deletes the temporary record and propagates: nothing was committed. Any other throwable propagates and leaves the temporary record for recovery, as a crash would. A real failure of any of these directory forces after the rename is logged at WARNING; for a JsonJournal.Kind.COMMIT record this method still returns, because the record is committed, and for a JsonJournal.Kind.CHECKPOINT record the failure propagates, so the checkpoint writes no entity file.

      Parameters:
      sequence - the record's number, from nextSequence()
      kind - what the record stands for; its label prefixes the record steps
      entries - the record's changes, in the order they are to be applied
      Throws:
      IOException - if the record could not be committed, or a checkpoint record's directory could not be forced

    • listCommitted

      List<JsonJournal.Record> listCommitted() throws IOException
      Reads every committed record in ascending sequence order, verifying each.
      Returns:
      the committed records, oldest first; empty if the journal directory is missing
      Throws:
      DataAccessException - if a record is damaged; the message names the file
      IOException - if a record cannot be read

    • recover

      void recover() throws IOException
      Brings the identity's table files to the state of its committed records: empties the journal's tmp/ directory and deletes temporary records, verifies every committed record, applies them in ascending order through the entity-file writer, then deletes the applied records in ascending order, forcing the journal directory after each one it actually deleted, so the records left on disk after any power loss are always a suffix of the applied ones and replay to the same files. A missing journal directory is an empty journal; nothing is created. A real failure to force a table directory stops recovery before any applied record is deleted.

      A damaged committed record (question 8 option A of 20.4-DECISIONS.md) makes recovery throw before any table file or record is changed. The record is never deleted, skipped or applied: recovery keeps refusing until the server operator moves the file out of the journal directory, and then applies the remaining records.

      A committed record holding an id this server's file system cannot name (a JSON data folder copied from Linux to Windows with an id containing :, for example) is refused the same way, before any table file or record is changed, with an error naming the record, the table and the id (question 12 A of the Phase 20.4 decisions).

      Throws:
      DataAccessException - if a committed record is damaged, or holds an id this file system cannot name; the message names the file
      IOException - if a file cannot be read, written, moved or deleted, or a directory cannot be forced to disk

    • checkpoint

      void checkpoint(long sequence, List<JsonJournal.Entry> entries) throws IOException
      Writes a checkpoint, in this order: commits checkpoint record sequence (K) holding every entry's after-image; only after K's rename, writes the entity files and deletes the files of deleted entries, then forces the touched table directories; then deletes every record, committed or temporary, numbered below K (and a temporary K, if any), in ascending order, forces the journal directory, and only then deletes K and forces the journal directory again. Records above K are left alone.

      Why this order: replay applies records in ascending order and the last record touching a key decides its value. An entity file on disk before K is committed could be overwritten by an older surviving record at replay; deleting K before an older record could leave that older record without K, and a file system may make K's unlink durable before an older one's unless a directory force separates them. With this order, every interruption, power loss included, leaves records whose replay ends with K's values once K is renamed, and the older values before it.

      Truncation deletes records by number without reading them. The caller guarantees that every record numbered below K is already reflected in entries or in the table files.

      A real failure to force a directory stops the checkpoint where it happens: after K's rename no entity file is written, after the table directories no record is deleted, after the older records' deletion K is kept, and after K's deletion the failure is still reported. The caller keeps its entries pending, and the next checkpoint writes them again.

      Parameters:
      sequence - K, from nextSequence()
      entries - the captured after-images and deletes
      Throws:
      IOException - if an entry's id names a file this file system cannot name (checked before record K is written, so nothing is written), a record or entity file cannot be written, moved or deleted, or a directory cannot be forced to disk

    • writeEntityFiles

      void writeEntityFiles(String phase, List<JsonJournal.Entry> entries, Path recordFile) throws IOException
      Applies entries to the table files, in order: a put writes <table>/<key>.json by writing and forcing a file in the journal's tmp/ directory and moving it into place atomically, replacing the old file; a delete removes the file if present. Then forces the parent of every directory the journal created and has not yet forced into its parent (a new table directory and its new parents among them), and every touched table directory; a real failure to force one propagates, after the files are in place, so the caller deletes no journal record. No file other than <key>.json is ever created in a table directory.
      Parameters:
      phase - PHASE_RECOVERY or PHASE_CHECKPOINT; prefixes the steps
      entries - the changes to apply
      recordFile - the committed record the entries come from, named in a failure
      Throws:
      IOException - if an entry's id names a file this file system cannot name (named with the record, the table and the id; never an unchecked InvalidPathException), a file cannot be written, moved or deleted, or a table directory cannot be forced to disk

    • entityFileOf

      private Path entityFileOf(JsonJournal.Entry entry)
      The entity file <identity root>/<table>/<key>.json of entry.
      Throws:
      InvalidPathException - if this file system cannot name it, or the name does not resolve to a file named <key>.json directly in the table directory (see JsonFileNames.entityFile(Path, String))

    • unnameable

      private static String unnameable(JsonJournal.Entry entry, InvalidPathException invalid)
      Names entry's table and id for a message about a file name this file system rejects.

    • requireNameable

      private void requireNameable(List<JsonJournal.Record> records)
      Refuses recovery, before any file changes, if a committed record holds an id this file system cannot name: the same remedy as a damaged record, so the error names the record and where to move it.

    • writeTemporaryEntityFile

      private Path writeTemporaryEntityFile(String afterImage) throws IOException
      Throws:
      IOException

    • deleteTemporaries

      private void deleteTemporaries() throws IOException
      Deletes every file in the journal's tmp/ directory and every temporary record.
      Throws:
      IOException

    • listRecordFiles

      List<JsonJournal.RecordFile> listRecordFiles() throws IOException
      The journal's record files in ascending order of number, a temporary record before a committed one of the same number; empty if the journal directory is missing.
      Throws:
      IOException

    • recordFileName

      static String recordFileName(long sequence, String suffix)
      The file name of record sequence: the number zero-padded to 19 ASCII digits, then suffix. Formatted with Locale.ROOT: under a default locale with other digits (Arabic-Indic, for example) the name would not match RECORD_NAME, and numbering and recovery would never see the record.

    • encode

      private static byte[] encode(long sequence, JsonJournal.Kind kind, List<JsonJournal.Entry> entries)

    • decode

      private JsonJournal.Record decode(JsonJournal.RecordFile file) throws IOException
      Reads and verifies one committed record; any defect is reported as a damaged record.
      Throws:
      IOException

    • member

      private static com.google.gson.JsonElement member(com.google.gson.JsonObject object, String name)
      A required member of a record object; a missing or null member makes the record unreadable.

    • damaged

      private DataAccessException damaged(Path file, String reason, Throwable cause)

    • indexOf

      private static int indexOf(byte[] bytes, byte value, int from)

    • writeFully

      private static void writeFully(FileChannel channel, byte[] bytes) throws IOException
      Throws:
      IOException

    • move

      private void move(Path source, Path target) throws IOException
      Renames a record atomically; falls back to a plain move only if the platform refuses.
      Throws:
      IOException

    • moveReplacing

      private void moveReplacing(Path source, Path target) throws IOException
      Moves an entity file into place atomically, replacing the old one.
      Throws:
      IOException

    • moveWithRetry

      private void moveWithRetry(Path source, Path target, CopyOption... options) throws IOException
      Every file move the journal makes goes through here. On Windows another program (an antivirus scanner, an indexer, a backup tool) can hold a journal file open for a moment, and the move then fails with an AccessDeniedException or with a plain FileSystemException (the JDK's mapping of a sharing or lock violation). Such a move is retried: at most 5 attempts, pausing 10, 20, 40 and 80 ms between them. Every subclass of FileSystemException other than AccessDeniedException is a definite answer and is not retried, and off Windows a move is attempted once, because there an open file does not stop a rename and an access error is permanent. An interrupt during a pause stops the retries and keeps the interrupt flag. The failure thrown in the end carries the earlier ones as suppressed.
      Throws:
      IOException

    • isTransientMoveFailure

      private static boolean isTransientMoveFailure(IOException failure)
      An access-denied failure, or exactly a FileSystemException: what a transient Windows lock gives.

    • withSuppressed

      private static IOException withSuppressed(IOException last, List<IOException> earlier)

    • logAtomicMoveFallback

      private void logAtomicMoveFallback(AtomicMoveNotSupportedException unsupported)

    • forceDirectory

      private IOException forceDirectory(Path target, String consequence)
      Forces a directory's entries to disk and classifies a failure (see the class javadoc). A directory that cannot be opened for a force is skipped and logged once at FINE; any other failure is logged at WARNING, naming the directory and stating consequence, and returned.
      Returns:
      the real failure, or null if the directory was forced or cannot be opened

    • requireDirectoryForced

      private void requireDirectoryForced(Path target, String consequence) throws IOException
      forceDirectory(Path, String), throwing a real failure.
      Throws:
      IOException

    • createDirectories

      private void createDirectories(Path target) throws IOException
      Creates target and every missing parent, as Files.createDirectories(java.nio.file.Path, java.nio.file.attribute.FileAttribute<?>...) does, and remembers each directory it created for forceNewDirectoryParents(String). Creating and remembering are one step under a lock, so a caller that finds target already present also finds it remembered while its parent is still unforced, and forces that parent before it relies on the directory.
      Throws:
      IOException

    • forceNewDirectoryParents

      private IOException forceNewDirectoryParents(String consequence)
      Forces, once each, the parent of every directory in unforcedNewDirectories and forgets the directories whose parent was forced or cannot be opened for a force. A real failure is classified and logged as forceDirectory(Path, String) does; the directories under that parent stay remembered and are forced again at the next call.
      Returns:
      the first real failure, or null

    • requireNewDirectoryParentsForced

      private void requireNewDirectoryParentsForced(String consequence) throws IOException
      forceNewDirectoryParents(String), throwing the first real failure.
      Throws:
      IOException

    • forceDirectoryChannel

      private static void forceDirectoryChannel(Path target) throws IOException
      Throws:
      IOException

    • sha256Hex

      private static String sha256Hex(byte[] bytes)

    • fire

      private static void fire(String step) throws IOException
      Throws:
      IOException

    • installStepHook

      static void installStepHook(JsonJournal.StepHook hook)

    • resetStepHook

      static void resetStepHook()

    • installDirectoryForcer

      static void installDirectoryForcer(JsonJournal.DirectoryForcer forcer)

    • resetDirectoryForcer

      static void resetDirectoryForcer()

    • installMover

      static void installMover(JsonJournal.Mover replacement)

    • resetMover

      static void resetMover()

    • installMoveRetry

      static void installMoveRetry(boolean retry)

    • resetMoveRetry

      static void resetMoveRetry()

    • retriesMovesOn

      static boolean retriesMovesOn(String osName)
      Whether a journal move that fails is retried on the operating system named osName (the os.name system property).
      Parameters:
      osName - the operating system's name; may be null
      Returns:
      whether a failed move is retried there