immich-go not uploading most of my 202Gig Takeout #650

phands · 2025-01-24T17:43:28Z

phands
Jan 24, 2025

Hi all,

Sorry if this is a daft question.
I have a large-ish (just over 202Gigs) Takeout directory (extracted from the compressed archives to a directory structure), and I'm trying to use immich-go to upload it all.

It seems to work partially, but I only see a tiny fraction of the files in the immich server. There are tens of thousand of photos and videos. but less than 25 in Immich.

I know that the uploader is changing rapidly, and I'd love to help with testing if I can.
My Takeout directory is structured as shown in the screenshot...

Immich-go says it's seeing 31553 images...see this screenshot...

my iimmich-go command is ....
/immich-go upload from-google-photos /local-12T/Google\ Photos/Takeout --api-key 853oJrfwOpiGB8XhASoAraVsQ8USDS156Eu1nLaNWZY --server http://10.0.0.37:2283

The log file seems OK, but shows that while it scanned 31553 images, it only uploaded 35 the first time, and now they are duplicates. Here are the last few lines of 102900...

2025-01-24 12:31:34 INF Tagged file=Takeout:Photos from 2024/Simon Durrant and Anna and Alan Kidd and wife.jpg tags=[Takeout]
2025-01-24 12:31:34 INF added to an album file=Takeout:Photos from 2024/Simon Durrant and Anna and Alan Kidd and wife.jpg Album=TATC 11th Entry
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Input analysis:
2025-01-24 12:33:39 INF ---------------
2025-01-24 12:33:39 INF scanned image file : 31553
2025-01-24 12:33:39 INF scanned video file : 2926
2025-01-24 12:33:39 INF scanned sidecar file : 33311
2025-01-24 12:33:39 INF discarded file : 0
2025-01-24 12:33:39 INF unsupported file : 468
2025-01-24 12:33:39 INF file duplicated in the input : 7
2025-01-24 12:33:39 INF associated metadata file : 42
2025-01-24 12:33:39 INF missing associated metadata file : 34437
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Uploading:
2025-01-24 12:33:39 INF ----------
2025-01-24 12:33:39 INF uploaded : 0
2025-01-24 12:33:39 INF upload error : 0
2025-01-24 12:33:39 INF file not selected : 0
2025-01-24 12:33:39 INF server's asset upgraded with the input : 0
2025-01-24 12:33:39 INF server has same asset : 35
2025-01-24 12:33:39 INF server has a better asset : 0
2025-01-24 12:33:39 INF
2025-01-24 12:33:39 INF Check the log file: /home/phands/.cache/immich-go/immich-go_2025-01-24_12-31-01.log

Can anyone point me at where to start debugging or what is wrong with my immich-go command? I'm also available for live debugging (USA Eastern time).

Thanks,

Paul

simulot · 2025-01-24T18:41:52Z

simulot
Jan 24, 2025
Maintainer

The JSON files have an unusual name:

photo.jpg-supplemental-metadata.json

This explain this missing associated metadata file : 34437

You're the first to report such case.
Could you grab the list of files? See the page https://github.com/simulot/immich-go/blob/main/docs/how-to-send-debug-data.md

The takeout isn't easy to parse. Long file names are truncated using unclear rules. So, I'm surprised to have much longer name for the JSON files. I'm particularly curious how are named JSON files attached to photos with very long names...

I have requested a fresh takeout, to see if something has changed.

7 replies

phands Jan 24, 2025
Author

Here's another view of the list...

list2.txt

simulot Jan 25, 2025
Maintainer

Thank you for the list. This is new for me.

Some weird cases

Screenshot_20231027_123303_Facebook(1).jpg
Screenshot_20231027_123303_Facebook.jpg.supple.json
Screenshot_20231027_123303_Facebook.jpg.supple(1).json
Screenshot_20231027_123303_Facebook.jpg
Screenshot_20231027_123303_Facebook-edited.jpg
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_.json
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_2.jpg
original_c576ba15-9661-4bd4-b0bf-a7bf2ce3d354_2(1).jpg

I'll add the support for those case very soon

simulot Jan 25, 2025
Maintainer

Issue #652

Could you provide the content of some of the supplemental-metadata.json files ?

phands Jan 25, 2025
Author

I'll get on it today...do you want the metadata for the particular files you list above, or just any metadata?

simulot Jan 25, 2025
Maintainer

any metadata is good.

heylookltsme · 2025-02-03T17:51:00Z

heylookltsme
Feb 3, 2025

Hello! I'm having this same issue. Must be a recent change with Google Takeout because I first started playing with immich-go a couple weeks ago and everything worked as expected. But a couple days ago I wiped my immich installation to start fresh and used a new takeout export and now I'm seeing this exact issue.

Here's the summary of a dry run:

Input analysis:
---------------
scanned image file                      :   11990
scanned video file                      :     453
scanned sidecar file                    :   14386
discarded file                          :       0
unsupported file                        :      16
file duplicated in the input            :      14
associated metadata file                :      75
missing associated metadata file        :   12368

Uploading:
----------
uploaded                                :      61
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :       0
server has a better asset               :       0

Here's the contents of one of the supplemental-meta.json files:

{
  "title": "PXL_20221003_181216799~2.jpg",
  "description": "",
  "imageViews": "1",
  "creationTime": {
    "timestamp": "1670067700",
    "formatted": "Dec 3, 2022, 11:41:40 AM UTC"
  },
  "photoTakenTime": {
    "timestamp": "1664820736",
    "formatted": "Oct 3, 2022, 6:12:16 PM UTC"
  },
  "geoData": {
    "latitude": 43.190594399999995,
    "longitude": -75.1767833,
    "altitude": 301.33,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "geoDataExif": {
    "latitude": 43.190594399999995,
    "longitude": -75.1767833,
    "altitude": 301.33,
    "latitudeSpan": 0.0,
    "longitudeSpan": 0.0
  },
  "people": [{
    "name": "Kinsey"
  }, {
    "name": "Christie Seaver"
  }],
  "url": "https://photos.google.com/photo/AF1QipM4SRG7vwLtVXUeBw2hUz1Hu39sXQ7sBhlZu0JP",
  "googlePhotosOrigin": {
    "mobileUpload": {
      "deviceFolder": {
        "localFolderName": ""
      },
      "deviceType": "ANDROID_PHONE"
    }
  }
}

Please let me know if there's any other information that would be helpful for you. Thanks for your great work on this project!

3 replies

simulot Feb 3, 2025
Maintainer

I'll release a new version in a couple of hours

heylookltsme Feb 4, 2025

Things look much improved with the latest version! Thanks much!!

Input analysis:
---------------
scanned image file                      :   55837
scanned video file                      :    4022
scanned sidecar file                    :   59818
discarded file                          :       8
unsupported file                        :      47
file duplicated in the input            :   29851
associated metadata file                :   59674
missing associated metadata file        :     181

Uploading:
----------
uploaded                                :   29819
upload error                            :       0
file not selected                       :       0
server's asset upgraded with the input  :       0
server has same asset                   :       0
server has a better asset               :       0

Are these results expected? I'm trying to figure out how the numbers add up.

Is this what I should expect?

scanned images + scanned videos - discarded - unsupported - duplicates - missing metadata = files uploaded

If so, the numbers don't seem to add up so I think I may be misunderstanding. I just want to make sure nothing is missed. Thank you!

simulot Feb 4, 2025
Maintainer

I try to do the math correctly... but it's more challenging than expected. some problems are counted twice and some CLI options may change the behavior. unsupported aren't image or video.

phands · 2025-02-03T19:18:55Z

phands
Feb 3, 2025
Author

Thank you! As soon as it's available, I'll test it! Paul

…

On Mon, Feb 3, 2025 at 1:46 PM Jean François CASSAN < ***@***.***> wrote: I'll release a new version in a couple of hours — Reply to this email directly, view it on GitHub <#650 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCC5NBRJXC5QQ6VXB5PDED2N62RLAVCNFSM6AAAAABV2FPVYWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTEMBUGYYTMNQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

1 reply

simulot Feb 4, 2025
Maintainer

you may haven't noticed I have released a new RC this morning.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

immich-go not uploading most of my 202Gig Takeout #650

{{title}}

Replies: 3 comments 11 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

immich-go not uploading most of my 202Gig Takeout #650

phands Jan 24, 2025

Replies: 3 comments · 11 replies

simulot Jan 24, 2025 Maintainer

phands Jan 24, 2025 Author

simulot Jan 25, 2025 Maintainer

simulot Jan 25, 2025 Maintainer

phands Jan 25, 2025 Author

simulot Jan 25, 2025 Maintainer

heylookltsme Feb 3, 2025

simulot Feb 3, 2025 Maintainer

heylookltsme Feb 4, 2025

simulot Feb 4, 2025 Maintainer

phands Feb 3, 2025 Author

simulot Feb 4, 2025 Maintainer

phands
Jan 24, 2025

Replies: 3 comments 11 replies

simulot
Jan 24, 2025
Maintainer

phands Jan 24, 2025
Author

simulot Jan 25, 2025
Maintainer

simulot Jan 25, 2025
Maintainer

phands Jan 25, 2025
Author

simulot Jan 25, 2025
Maintainer

heylookltsme
Feb 3, 2025

simulot Feb 3, 2025
Maintainer

simulot Feb 4, 2025
Maintainer

phands
Feb 3, 2025
Author

simulot Feb 4, 2025
Maintainer