The Photos app's filter-based "smart albums" build their file listing using the search configuration (photosSourceFolders) of the user viewing the album, rather than the album owner's configuration. When an album owner shares a smart album with another user, that user's own folder configuration is used to determine which of the owner's files are searched — allowing them to discover files (name, file ID, and other metadata) in folders the album owner never intended to include in the shared album.
This requires the album owner to have shared a filter-based smart album with the attacker; it does not allow access to arbitrary users' files without such a share.