Skip to content

Attachments and observations missing from the cloud after sync #80

Open
0 of 6 issues completed
Open

Description

@gmaclennan

Partners using the cloud API report photos and audio that cannot be fetched: GET /projects/:id/attachments/... returns a 404 for some attachments, a 500 with {"message":"fetch failed"} for others, and for a third group the download starts and then stops part way ("Response ended prematurely"). Some partners also see observations that exist on a phone but never appear on the cloud. The missing media stays missing for weeks, and it is seen across several deployments on cloud 0.4.0 with phones on core 7.5.2.

Cause(s): when a phone's upload to the cloud is interrupted (the user leaves the Exchange screen, presses Stop, loses signal, or the cloud restarts) the cloud is left with the index entry for an attachment but only some of its blocks. The gap only repairs itself when that phone comes back and stays long enough to finish. Which error the API returns depends on which block is the first missing one. Observations go missing the same way; it is just invisible because a missing record produces no error, and a photo is far more exposed because it spans dozens of blocks where an observation spans one. Separately, a phone reconnecting to the cloud while its previous connection is still open (on the server side) can have its sync state wiped by the old connection's late close, so nothing further syncs until the next visit.

The sub-issues below cover each underlying problem and link the pull requests that address it. Most need both a cloud release and a core or mobile release to be fully fixed; the cloud-side changes can ship first and stand on their own.

Until fixes ship, two things help: run docker logs <cloud> 2>&1 | grep -c 'BlobStore error' on a deployment to rule a downloader stall in or out, and ask the connector to log the body of each 500 and the bytes received before a truncated response.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions