No files in sub directories created from sync on new device

[phxyz12]

Abstract

I added a new device to sync. Sync seems to fetch all chunks from the remote and processes all remote files. Nevertheless, all files in subfolders are not created on the local device. Also, the folder structure is not created.

Expected behaviour

• Synchronisation creates all missing files and folders on the new device.

Actually happened

• Synchronisation seems to run completely without errors
• Synchronization does not create files/folders in the local Obsidian vault.

Reproducing procedure

1. Configure LiveSync on the new device using the setup URI also used for all other devices.
2. Click the replication button on the ribbon.
3. Synchronising has begun.
4. Quite a while later, after lots of chunks have been fetched and the file list from the remote has been processed, sync ends with message "completed".
5. Only two files have been created locally, namely ones that are stored in the vault's root folder (not in a subfolder).

Aditional observations

• When I create a new file on the new device, it's synced to the remote, but the other devices don't get it.
• I made several attemps to add this new device, rebuilt everything on a working (master) device, rebuilt the database on remote. All attemps to add the new device failed in the same way.
• I am not able to add this new device by any means. All existing devices sync without problems.

[jasonlaguidice]

Okay so it's not just me. I got a new phone and tried to set it up using a URI. The sync APPEARS to work properly but it never makes any files or directories.

I tried again from a desktop client setting up a new sync and same thing - it parses the folder and file names from the server as you can see it doing this but it never actually makes any files on the new device.

[phxyz12]

@jasonlaguidice Yes, that's exactly my observation. The current version 0.24.2 works on all devices that already existed and were just updated to this version. Adding a new device does result in the behaviour you (and me) described.

[vrtmrz]

Thank you for opening the issue, and sorry for your inconvenience!

Unfortunately, I have not reproduced it yet. Would you mind if I ask you to share the report of the new device, and logs? How to dump them is documented on this doc.

[jasonlaguidice]

Report:

---- Obsidian info ----
Navigator: Mozilla/5.0 (Linux; Android 14; SM-F956U1 Build/UP1A.231005.007; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/131.0.6778.39 Mobile Safari/537.36
FileSystem: sensitive
---- remote config ----
cluster:
  n: "1"
cors:
  credentials: "true"
  headers: accept, authorization, content-type, origin, referrer
  max_age: "3600"
  methods: GET,PUT,POST,HEAD,DELETE
  origins: app://obsidian.md,capacitor://localhost,http://localhost
chttpd:
  bind_address: 127.0.0.1
  enable_cors: "true"
  max_http_request_size: "4294967296"
  port: "5984"
  require_valid_user: "true"
admins: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
vendor:
  name: The Apache Software Foundation
feature_flags:
  partitioned||*: "true"
chttpd_auth:
  authentication_redirect: /_utils/session.html
  hash_algorithms: sha256, sha
  require_valid_user: "true"
log:
  file: /var/log/couchdb/couchdb.log
  level: info
  writer: file
indexers:
  couch_mrview: "true"
prometheus:
  additional_port: "false"
  bind_address: 127.0.0.1
  port: "17986"
httpd:
  WWW-Authenticate: Basic realm="couchdb"
  bind_address: 0.0.0.0
  enable_cors: "true"
  port: "5986"
smoosh:
  state_dir: ./data
couch_httpd_auth:
  authentication_db: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  secret: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  authentication_redirect: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchdb_engines:
  couch: couch_bt_engine
couchdb:
  database_dir: ./data
  max_document_size: "50000000"
  single_node: "true"
  uuid: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
  view_index_dir: ./data

---- Plug-in config ---
version:0.24.2
remoteType: ""
useCustomRequestHandler: false
couchDB_URI: self-hosted(HTTPS)
couchDB_USER: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_PASSWORD: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
couchDB_DBNAME: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
liveSync: true
syncOnSave: true
syncOnStart: true
savingDelay: 200
lessInformationInLog: false
gcDelay: 0
versionUpFlash: ""
minimumChunkSize: 20
longLineThreshold: 250
showVerboseLog: false
suspendFileWatching: false
trashInsteadDelete: true
periodicReplication: false
periodicReplicationInterval: 60
syncOnFileOpen: true
encrypt: true
passphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
usePathObfuscation: false
doNotDeleteFolder: false
resolveConflictsByNewerFile: false
batchSave: false
batchSaveMinimumDelay: 5
batchSaveMaximumDelay: 60
deviceAndVaultName: ""
usePluginSettings: false
showOwnPlugins: false
showStatusOnEditor: true
showStatusOnStatusbar: true
showOnlyIconsOnEditor: false
usePluginSync: false
autoSweepPlugins: false
autoSweepPluginsPeriodic: false
notifyPluginOrSettingUpdated: true
checkIntegrityOnSave: false
batch_size: 25
batches_limit: 25
useHistory: true
disableRequestURI: true
skipOlderFilesOnSync: true
checkConflictOnlyOnOpen: false
showMergeDialogOnlyOnActive: false
syncInternalFiles: false
syncInternalFilesBeforeReplication: false
syncInternalFilesIgnorePatterns: \/node_modules\/, \/\.git\/, \/obsidian-livesync\/
syncInternalFilesInterval: 60
additionalSuffixOfDatabaseName: /storage/emulated/0/Documents/Personal
ignoreVersionCheck: false
lastReadUpdates: 23
deleteMetadataOfDeletedFiles: false
syncIgnoreRegEx: ""
syncOnlyRegEx: ""
customChunkSize: 50
readChunksOnline: true
watchInternalFileChanges: true
automaticallyDeleteMetadataOfDeletedFiles: 0
disableMarkdownAutoMerge: false
writeDocumentsIfConflicted: false
useDynamicIterationCount: false
syncAfterMerge: true
configPassphraseStore: ""
encryptedPassphrase: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
encryptedCouchDBConnection: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
permitEmptyPassphrase: false
useIndexedDBAdapter: true
useTimeouts: false
writeLogToTheFile: false
doNotPaceReplication: false
hashCacheMaxCount: 300
hashCacheMaxAmount: 50
concurrencyOfReadChunksOnline: 30
minimumIntervalOfReadChunksOnline: 25
hashAlg: xxhash64
suspendParseReplicationResult: false
doNotSuspendOnFetching: false
useIgnoreFiles: false
ignoreFiles: .gitignore
syncOnEditorSave: true
pluginSyncExtendedSetting: {}
syncMaxSizeInMB: 50
settingSyncFile: ""
writeCredentialsForSettingSync: false
notifyAllSettingSyncFile: false
isConfigured: true
settingVersion: 10
enableCompression: false
accessKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
bucket: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(0 letters)
endpoint: Not configured or AWS
region: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷(4 letters)
secretKey: 𝑅𝐸𝐷𝐴𝐶𝑇𝐸𝐷
useEden: false
maxChunksInEden: 10
maxTotalLengthInEden: 1024
maxAgeInEden: 10
disableCheckingConfigMismatch: false
displayLanguage: ""
enableChunkSplitterV2: false
disableWorkerForGeneratingChunks: false
processSmallFilesInUIThread: false
notifyThresholdOfRemoteStorageSize: -1
usePluginSyncV2: true
usePluginEtc: false
showLongerLogInsideEditor: false
sendChunksBulk: false
sendChunksBulkMaxSize: 1
useAdvancedMode: false
usePowerUserMode: false
useEdgeCaseMode: false
enableDebugTools: false
configPassphrase: ""
preset: ""
syncMode: LIVESYNC
dummy: 0

When choosing "set up as a subsequent device" I cannot capture the logs because the log to file feature gets toggled off as soon as the synchronization starts. What should I do to remedy this?

[GrandyBoy]

Okay so it's not just me. I got a new phone and tried to set it up using a URI. The sync APPEARS to work properly but it never makes any files or directories.

I tried again from a desktop client setting up a new sync and same thing - it parses the folder and file names from the server as you can see it doing this but it never actually makes any files on the new device.

Same issue here 😢

[UBBRDW]

Same here, only part of the files are sync to the new device

The log of getting files from db to the chunk(? I'm not sure) seems good as the number of files are correct as ~10000,
but after present of logs of "processing xxx", the process stop without having all the files in the edatabase

[khominhvi]

Same issue. Setting up Obsidian/livesync on new devices doesn't pull files from the remote DB. Fetching the database from remote seems to complete without any errors. But no files appear.

Tried rebuilding local and remote databases from local files and it uploads to the remote DB ok. But when trying to fetch to new devices its still the same, process completes but no files show up. Tried setting up a fresh Obsidian install on Windows 11 24H2 and Debian 12. Same problem on both devices.

[vrtmrz]

I am so sorry for being absent!
As following your information and other issues, it is possibly a processors problem.

I have rewrote the library used in Self-hosted LiveSync recently. I will be going to fix the problem with this.

Sorry for your inconvenience!

[phxyz12]

I am so sorry for being absent! As following your information and other issues, it is possibly a processors problem.

I have rewrote the library used in Self-hosted LiveSync recently. I will be going to fix the problem with this.

Sorry for your inconvenience!

No worries, no need to be sorry. I am appreciating your work and efforts to a great extent, take whatever time it needs. There are more important things in life.

[tomaszduda23]

I rollback to 0.23.20 to make it works. 0.23.23 didn't work for me.

[UBBRDW]

The newest 0.24.4 resolved the issue for me, really appreciate for all the work and contribution

[phxyz12]

Tried again with new vault and plugin version 0.24.5. Still the same behavior: No files and directories are created although the sync "seems" to work when looking into the log.

[ekovarski]

What does the sync icon look like? Mine wasn't defaulting to LiveSync after pulling in the remote config - I had to manually go into setup and select LiveSync.

Also noticed that empty folders are not replicated, once you add content it replicates the folder.

[phxyz12]

What does the sync icon look like? Mine wasn't defaulting to LiveSync after pulling in the remote config - I had to manually go into setup and select LiveSync.

Also noticed that empty folders are not replicated, once you add content it replicates the folder.

New test with new Obsidian vault and newly installed plugin, this time with version 0.24.6. I checked sync setting, all good. Behaviour as before: The sync process goes over all files and folders but does not create any of them locally.

I do not have empty folders, I have > 300 files in dozends of folders. All gets "scanned", but nothing created.

[jasonlaguidice]

I've managed to get it working (I think?) but it's convoluted.

I THINK this summarizes the steps I took to get mine working properly:

On an older already-synced device with a working copy of your vault:

1. Make a backup copy of the vault files, just in case
2. Open Obsidian, navigate to Settings -> Community Plugins -> Check for updates
3. Download the latest version of the Self-Hosted Live Sync plugin
4. Restart Obsidian
5. Navigate to Settings -> Self-Hosted Live Sync -> Maintenance Tab
6. Under the "Rebuilding Options (Remote Only)" section, click send for "Overwrite remote"

Some number of Obsidian restarts will likely be required to get this to stabilize and the database will say it's locked (if I remember correctly) until you manually unlock it

Now, on the new device you want to sync (and all other subsequent devices)... If you have already attempted a sync, remove the vault and delete all files associated. On Android, clear storage for the app. On other platforms - do what is necessary to start fresh

1. Open Obsidian and navigate to Community plugins, enable plugins, install Self-Hosted Live Sync
2. Accept and enter your URI and passphrase
3. Set the device up as a new or subsequent device and follow through the setup procedure. It may take a few Obsidian restarts for everything to sync successfully. At this point all of your files should show up.

Return to the original device and unlock the database. Change the sync preferences to Live Sync

Return to the new device and change the sync preferences to live sync.

It seems as if the detection of the database migration doesn't work properly and the database ends up split. You need to update the plugin from an existing old copy and force-overwrite the contents of the remote/couchdb database... then you can successfully join new clients in.