Skip to content

ARTIFACT-APPENDIX.md

Latest commit

 

History

History
187 lines (124 loc) · 7.01 KB

ARTIFACT-APPENDIX.md

File metadata and controls

187 lines (124 loc) · 7.01 KB
Raw

Artifact Appendix (Required for all badges)

Paper title: Mapping Android and iOS Permissions Across Space and Time

Requested Badge(s):

  • Available
  • Functional
  • Reproduced

Description (Required for all badges)

This codebase is the artifact related to the paper Mapping Android and iOS Permissions Across Space and Time (PETS 2027) Authors: Magdalena Steinböck, Jakob Bleier, Florian Draschbacher, Christine Utz, Tobias Urban, Martina Lindorfer

The artifact contains the relevant data and scripts to reproduce the Figures and Tables provided in the paper.

Security/Privacy Issues and Ethical Concerns (Required for all badges)

The artifact raises no ethical concerns or security/privacy issues.

Basic Requirements (Required for Functional and Reproduced badges)

Hardware Requirements (Required for Functional and Reproduced badges)

  1. Can run on a laptop (No special hardware requirements)
  2. The experiments were conducted using a MacBook Pro M1, 16GM RAM, 500GB storage, and verified on a Linux machine with similar specs.

Software Requirements (Required for Functional and Reproduced badges)

  1. We tested the artifact on macOS Sequoia 15.7.4 as well as on Ubuntu 24.04.
  2. The artifact was tested using Python 3.12.4. The required python packages are provided in the script files themselves through the uv tool.
  3. Datasets required to run our artifact are provided in permission_mapping/permission_analysis/data.

Estimated Time and Storage Consumption (Required for Functional and Reproduced badges)

  • The overall human and compute times required to run the artifact are about 10 minutes.
  • The overall disk space consumed by the artifact is below 1GB.

Environment (Required for all badges)

Accessibility (Required for all badges)

Our Artifact is available at GitHub under https://github.com/SecPriv/PermiOSAn/.

Set Up the Environment (Required for Functional and Reproduced badges)

To set up our artifact, first clone it from GitHub:

git clone git@github.com:SecPriv/PermiOSAn.git

Install uv, which will manage python and python dependencies for our scripts.

To remove the dependencies installed by uv, run uv cache clean and rm -r "$(uv python dir)". To remove uv itself, either run rm ~/.local/bin/uv ~/.local/bin/uvx or run the appropriate package manager command if one was used to install it.

Alternatively, a combined requirements.txt file is provided that can be used to install the dependencies in other environments.

Testing the Environment (Required for Functional and Reproduced badges)

All scripts are stand-alone, the easiest way to check it works is by running the following commands:

cd permission-mapping
./calculate_cohens_kappa.py

This will run /usr/bin/env -S uv run --script as interpreter for the file install dependencies declared in the script block at the top of the file. The expected output is:

344
344
0.8630337465564738

Artifact Evaluation (Required for Functional and Reproduced badges)

Main Results and Claims

Main Result 1: Reproducing Tables and Figures

By executing Experiment 1, our Figures 2, 3, 4, and 5, as well as our Tables 2, 3, and 6. The respective functions in the script each describe a Figure or Table.

Main Result 2: Get Adaptation of Android-only and iOS-only Permissions

By executing Experiment 2, our Tables 4 and 5 can be verified, as the script generates the usage statistics of Android-only and iOS-only permissions per year. For our results presented in the paper, only 2025 is relevant.

Main Result 3: Verifying the Jaccard Similarity

Our paper claims that, across all XPPCs, we observe an average Jaccard similarity of 0.59 (median: 0.50), indicating a moderate degree of similarity between the two platforms’ permission-granting entities. This claim is reproducible by executing our Experiment 3. In this Experiment we calculate the Jaccard similarity of all XPPCs based on our mapping results (from Table 11).

Main Result 4: Verifying Cohen's Kappa

Our paper claims that In 306 (88.4%) cases, the two researchers assigned the same group to permissions accross Android and iOS and we measured an inter-rater reliability using Cohen’s Kappa (𝜅 = 0.86). This claim is reproducible by executing our Experiment 4. In this Experiment we calculate the Cohen's Kappa based on the provided group assignments of both coders.

Experiments

Experiment 1: Reproducing Tables and Figures

  • Time: < 10m
  • Storage: <10GB

To verify the numbers and measurements presented in the paper's tables and figures, run the following scripts in order from the permission_analysis/ directory:

cd permission_analysis
./permission_analysis.py
./process_json.py
./filter_permission_diffs.py
./permission_analysis_statistics.py

Note: Scripts must be run sequentially, as each step depends on the output of the previous one.

This generates all tables and figures that were used in the paper. The expected result is that the generated figures match the figures from the paper provided in permission_mapping/permission_analysis/plots_28_07_2026.

Experiment 2: Get Adaptation of Android-only and iOS-only Permissions

  • Time: < 5m
  • Storage: <10GB

To get the numbers of Android-only and iOS-only permissions and verify Table 4 and Table 5

cd permission_analysis
./get_permission_stats_no_mapping.py

This generates a json file per year with the respective usage stats for Android-only and iOS-only permissions, in files named ./no_mapping_permission_stats_{year}.json.

Experiment 3: Verifying the Jaccard Similarity

  • Time: < 5m
  • Storage: <10GB

To reproduce the Jaccard Similarity scores measuring the degree of overlap between XPPCs with shared entities, run the following from the jaccard_similarity/ directory:

cd jaccard_similarity
./main.py

This generates the value of the Jaccard Similarity we provide in Chapter 3.3 Underlying Permission-Granting Mechanisms.

Experiment 4: Verifying Cohen's Kappa

  • Time: < 5m
  • Storage: <10GB

To reproduce the Cohen's Kappa inter-rater reliability score for our codebook-based permission mapping approach, run the following from the permission-mapping/ directory:

cd permission-mapping
./calculate_cohens_kappa.py

This generates the value of the Cohen's Kappa we provide in Chapter 3.2.

Limitations (Required for Functional and Reproduced badges)

We provide all extracted data and code to reproduce the figures and numbers in the paper. The app packages (downloaded from the iOS app store and the Google Play store respectively) are not public, but can be shared on request together with their static analysis scripts.

Notes on Reusability (Encouraged for all badges)

Our main contribution that can be reused by other researchers is our permission mapping of permissions on Android and iOS, which we provide in a machine-readable json format in permission_mapping/permission_analysis/data/permission_mapping.json.