Corruption during sync process breaks plugin files locally and in remote DB

[izosozi]

EDIT: solved (partially)

It turns out that there was a failure to decrypt some of the .obsidian files. I will leave more details in a comment.

Abstract

Some (but not all) plugins are failing to load on start. The plugin folders appear intact in the filesystem, but not within the app.

I'm not 100% certain that LiveSync did cause this issue; It's very possible that this was totally my own error or another plugin caused it. However, it happened at the same time that I was messing with various LiveSync settings, and I hadn't made any real changes to other plugins or Obsidian in the several hours before the issue happened.

Expected behavior

• Plugins visible in the Community Plugins interface match the plugins in .obsidian/plugins
• Installed plugins don't disappear without a reason

Actually happened

• Over half of the community plugins are not visible in the Obsidian Settings sidebar or the Community Plugins settings pane
• Plugin functionality is not loading for these plugins
• Some plugins remain functional, unclear why

Reproducing procedure

I have no idea what caused it, but I'll provide environment details below. The issue is persistent.

Report materials

Went over the length limit...

report-data.txt

Screenshots

Server config check:

Other information, insights and intuition.

Obsidian plugins

In GUI

The plugins shown in the Community Plugins GUI in Obsidian:

In JSON

The plugins listed in .obsidian/community-plugins.json:

[
  "github-embeds",
  "obsidian-excalidraw-plugin",
  "obsidian-minimal-settings",
  "obsidian-style-settings",
  "obsidian-kanban",
  "obsidian-tasks-plugin",
  "dataview",
  "execute-code",
  "templater-obsidian",
  "recent-files-obsidian",
  "obsidian-mind-map",
  "email-block-plugin",
  "obsidian-outliner",
  "omnisearch",
  "obsidian-linter",
  "nldates-obsidian",
  "better-word-count",
  "highlightr-plugin",
  "table-editor-obsidian",
  "obsidian-vimrc-support",
  "vim-yank-highlight",
  "obsidian-relative-line-numbers",
  "vim-toggle",
  "colored-tags-wrangler",
  "obsidian-icon-folder",
  "media-extended",
  "obsidian-link-embed",
  "dataview-serializer",
  "obsidian-advanced-uri",
  "meld-encrypt",
  "obsidian-auto-link-title",
  "links",
  "obsidian-local-images",
  "wikipedia-search",
  "obsidian-dangling-links",
  "quick-preview",
  "boost-link-suggestions",
  "obsidian-full-calendar",
  "calendar",
  "obsidian-text-format",
  "obsidian-regex-replace",
  "copy-as-html",
  "find-and-replace-in-selection",
  "remember-cursor-position",
  "obsidian-cursor-location-plugin",
  "advanced-cursors",
  "smart-connections-visualizer",
  "homepage",
  "advanced-canvas",
  "canvas-mindmap",
  "obsidian-raindrop-highlights",
  "janitor",
  "random-numbers-generator",
  "obsidian-dice-roller",
  "improved-random-note",
  "share-note",
  "obsidian-livesync"
]

In local filesystem

The content of .obsidian/plugins/:

.
|-- advanced-canvas
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- advanced-cursors
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- better-word-count
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- boost-link-suggestions
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- calendar
|   |-- data.json
|   |-- main.js
|   `-- manifest.json
|-- canvas-mindmap
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- colored-tags-wrangler
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- convert-url-to-iframe
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- copy-as-html
|   |-- main.js
|   `-- manifest.json
|-- dataview
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- dataview-serializer
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- email-block-plugin
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- execute-code
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- find-and-replace-in-selection
|   |-- main.js
|   `-- manifest.json
|-- github-embeds
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- heading-level-indent
|   |-- data.json
|   |-- main.js
|   `-- manifest.json
|-- highlightr-plugin
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- homepage
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- improved-random-note
|   |-- data.json
|   |-- main.js
|   `-- manifest.json
|-- ink
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- janitor
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- link-exploder
|   |-- main.js
|   `-- manifest.json
|-- link-favicon
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- links
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- media-extended
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- meld-encrypt
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- nldates-obsidian
|   |-- main.js
|   `-- manifest.json
|-- numerals
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-advanced-uri
|   |-- main.js
|   `-- manifest.json
|-- obsidian-auto-link-title
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-cursor-location-plugin
|   |-- main.js
|   `-- manifest.json
|-- obsidian-dangling-links
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-dice-roller
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-excalidraw-plugin
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-full-calendar
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-git
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-icon-folder
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-importer
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-kanban
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-link-embed
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-linter
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-livesync
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-local-images
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-math-plus
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-mind-map
|   |-- main.js
|   `-- manifest.json
|-- obsidian-minimal-settings
|   |-- data.json
|   |-- main.js
|   `-- manifest.json
|-- obsidian-outliner
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-raindrop-highlights
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-regex-replace
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-relative-line-numbers
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-style-settings
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-tasks-plugin
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-text-format
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- obsidian-vimrc-support
|   |-- main.js
|   `-- manifest.json
|-- omnisearch
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- quick-preview
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- random-numbers-generator
|   |-- main.js
|   `-- manifest.json
|-- recent-files-obsidian
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- remember-cursor-position
|   |-- cursor-positions.json
|   |-- main.js
|   `-- manifest.json
|-- share-note
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- smart-connections
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- smart-connections-visualizer
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- solve
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- system3-relay
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- table-editor-obsidian
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- tag-wrangler
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- templater-obsidian
|   |-- data.json
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
|-- vim-toggle
|   |-- main.js
|   `-- manifest.json
|-- vim-yank-highlight
|   |-- main.js
|   |-- manifest.json
|   `-- styles.css
`-- wikipedia-search
    |-- data.json
    |-- main.js
    `-- manifest.json

Potential causes

• Multiple rebuilds, merges, and fetches while I was troubleshooting other issues
• Quitting Obsidian during data operations
• Briefly enabling Customization Sync for LiveSync's own plugin data files
• Toggling options before I understood what they did
• A couple days ago I accidentally put self-referencing symlinks in the .obsidian folder, and LiveSync hit a recursion error trying to parse them. I removed these and LiveSync seemed to recover fine.
• Laptop and phone both running LiveSync simultaneously
• General recklessness

Attempts to fix

• LiveSync > rebuild local db and remote db from local files
• LiveSync > fetch from remote db
• Disabling LiveSync and relaunching obsidian
• Disabling LiveSync and duplicating the vault locally
• Uninstalling LiveSync and relaunching
• Uninstalling LiveSync and duplicating the vault locally
• Reinstalling Obsidian and opening both vaults
• Created a vault with only the .obsidian directory and no other files
• Verified file permissions for all files in .obsidian/

While LiveSync seems to have caused the problem initially, the problem seems persistent in the vault contents, probably

Other observations

• Multiple syntax errors in the Obsidian console, these might each correspond to an individual failure to load a plugin?
• An extraneous [] in syncIgnoreRegEx: \.DS_Store|[]|^raindrop\/ in livesync-config
• The size of the remote went from ~600MB to ~150MB around the same time, but I can't say for sure if this is related, since I was implementing ignore patterns which might have removed files separately from this issue.

The end

I do have backups and nothing was lost in this case thankfully!

Thanks so much for your hard work on this plugin; It's certainly the best open source option for Obsidian syncing I've seen, and I'd be interested in contributing once I get it all figured out.

EDIT: Solved partially

See comment below

[izosozi]

Within 30 seconds of finishing writing the initial bug report, I finally decided to actually look inside the vault files in .obsidian, and voila, It looks like around half the files in .obsidian were left encrypted at some point, for example:

./plugins/homepage/manifest.json: 
ewoJImlkIjogImhvbWVwYWdlIiwKCSJuYW1lIjogIkhvbWVwYWdlIiwKCSJ2ZXJzaW9uIjogIjQuMC43IiwKCSJtaW5BcHBWZXJz[...]

./plugins/better-word-count/main.js: 
J3VzZSBzdHJpY3QnOwoKdmFyIG9ic2lkaWFuID0gcmVxdWlyZSgnb2JzaWRpYW4nKTsKdmFyIHN0YXRlID0gcmVxdWlyZSgnQGNv[...]

./plugins/obsidian-raindrop-highlights/main.js: 
LyoKVEhJUyBJUyBBIEdFTkVSQVRFRC9CVU5ETEVEIEZJTEUgQlkgRVNCVUlMRAppZiB5b3Ugd2FudCB0byB2aWV3IHRoZSBzb3Vy

It seems likely that the direct cause had to do with interrupting LiveSync at the wrong time, probably by quitting Obsidian. I don't really know how the decryption process is handled though.

Some (uninformed) ideas on how to address this:

• Safety checks in place to prevent unencrypted data from being overwritten by encrypted data, possibly by decrypting in a temporary file (or in memory?)
• A way for interrupted processes to resume decryption gracefully
• A user option to request a scan of the vault for leftover encrypted files to decrypt
• Warning text next to the end-to-end encryption setting of this risk (if it isn't easily reversible)

Let me know if there's a straightforward way to decrypt these files? If so I might just keep the setting on and fix it if it breaks. Otherwise I'll disable encryption and rebuild the vault, and should be smooth sailing from there on.

edit

Those could potentially be from compression rather than encryption?

edit 2

Never mind, I hadn't enabled compression at any point

[vrtmrz]

Thank you for opening this issue! And so sorry for this!

It seems that the files had been decrypted successfully. However, it had not been decoded as a binary files.
(The file content that your have pasted indicate this situation! Thank you for your cooperation!)
I still wonder why it had been occurred, however, once it had been done, it cannot be resolved automatically.
Would you mind if I ask you to uninstall these plug-ins (from all devices) and install again?