Latest

fdroidserver UX Testing Report – Guardian Project

We ran consumer exams of fdroidserver, the instruments for developers to create and handle F-Droid repositories of apps and media. This check was set as much as gather usability suggestions concerning the instruments themselves and the related documentation. These exams have been put collectively and run by Seamus Tuohy/Prudent Innovation.

Individuals accomplished a pretest demographic/background info questionnaire. The facilitator then explained that the period of time taken to complete the check activity can be measured and that exploratory conduct inside the app should happen after the duties are completed.

The participant was then offered a laptop computer with a browser window open to the F-Droid documentation. The facilitator let the participant know that if they felt they need something external that they might ask the facilitator for that.

At the beginning of every process, the participant was offered the duty, and proven the assets that have been out there for them to make use of on a USB stick offered by the facilitator. The participant was then instructed to read the duty description from the printed copy and start the task. Time-on-task measurement began when the participant began the duty.

The facilitator instructed the participant to ‘think aloud’ so that they might seize their otherwise inner thoughts about interactions with the F-Droid server software. The facilitator observed and entered participant conduct, participant comments, and system actions into their report of the event.

After every process, the participant spoke with the facilitator concerning the activity. In any case process situations have been attempted, the participant accomplished a post-test satisfaction questionnaire.

Each participant was asked to finish six specific tasks. Initially of each process, the participant was learn the duty, and shown the assets which might be obtainable for them to use. The participant was then instructed to read the task description from the printed copy and begin the task.

Tasks

  1. Set Up an F-Droid software repository with the purposes on this USB Drive.
  2. Hook up with that repository utilizing the F-Droid shopper.
  3. Group the purposes beneath your repositories identify in the F-Droid interface.
  4. Obtain the barcode scanner software utilizing the F-Droid shopper.
  5. Replace your F-Droid repository with an update to the barcode scanner app.
  6. Download the update utilizing the telephone.

Tasks are marked as “complete” after the participant says they’ve completed the task or after the check facilitator has enough proof that the duty has been completed. Tasks are marked as “uncompleted” after the participant says they can’t full the task and requests assistance.

Check time will begin after the participant says they’ve understood the task and will begin. Testing time will end after the participant says they have completed the task or after the check facilitator has sufficient evidence that the task has been completed. Within the case of a essential error, check time will cease after the participant says they can’t complete the task and request assistance.

Usability Metrics

Usability metrics refers to participant performance when completing the assigned tasks. That is consists of completion success rates, error charges, time to activity completion and subjective evaluations/interviews.

  • Important Errors: Important errors are reported as errors that end in failure to finish the task. Members might or will not be aware that the task objective is wrong or incomplete. Unbiased completion of the duty is the aim; help from the check facilitator or others is to be marked as a crucial error.
  • Non-critical Errors: Non-critical errors are errors that the participant recovers from alone and will not be such that the participant can not full the task. They will embrace errors comparable to extreme steps taken to complete a activity or initially utilizing an incorrect perform but recovering from that incorrect step.
  • Exploratory Conduct: Errors which might be off process from the primary process trying to be completed shall be marked as exploratory conduct.

It must be famous that most of the errors that have been encountered in duties two, 4, and 6 have been the results of configuration selections and non-critical errors in tasks one and 5. Extra details about why that is the case could be found in the “Task Specific Findings.”

Activity 1: Set Up an F-Droid software repository with the purposes on this USB Drive.

  • Common Time: 53 Minutes
  • Number of Members with Crucial Errors: 1

Docs needed for configuring the distant server

The members have been offered a distant server that was pre-configured to host F-Droid purposes. The analysis group did this because establishing a web-server to host an F-Droid repository seemed out of scope, and it might improve the length of the UX session to an unacceptable length.

Throughout this activity the members have been persistently confused by the documentation associated to establishing the F-Droid host server (despite the fact that the server was pre-configured for them.) Documentation round file permissions on the remote server have been especially troublesome.FOOTNOTE: Footnote

Suggestions:

  • The participant was confused about the place the documentation begins to start out deploying the server.
  • The participant said that there is not sufficient info on tips on how to configure the remote server.
  • It’s lacking in the documentation that before doing “fdroid server update” the permissions of the information to be served might be modified regionally since they’re transmitted to the distant server with the same local permissions.
  • Within the documentation, the details about the file permissions and configuration of the server at Fdroid degree is lacking. The documentation should say what permissions include the generated information, (or what permissions they should include).
  • It is missing within the documentation that earlier than doing “fdroid server update” the permissions of the information to be served might be changed regionally since they’re transmitted to the distant server with the identical local permissions.
  • The participant didn’t use the command fdroid server replace to push the apps to the remote server, as an alternative of that, he used the command SCP to send the info of manually.
  • The participant was confused concerning the location of the information hosted on the internet server. Doubt resolved after reviewing the documentation.

Complicated ordering of documentation

The individuals felt that the documentation was confusingly ordered. While a couple of noted that the individual items of documentation have been right, and even clear as soon as a participant is acquainted with themFOOTNOTE: Footnote, the general organization and ordering of documentation made it made it troublesome to use.

Suggestions:

  • There have been many doubts relating to the order of the steps, there was additionally a sensation of ordered record of steps when in follow the knowledge have been associated to totally different actions (eg. Native Demo and Real World setup)
  • The documentation just isn’t understandable, not when it comes to language however in the best way of explaining, it ought to be more detailed. It feels difficult to understand for entry degree individuals.
  • The best way info is introduced and the way it is structured seems to be disconcerting and overloaded.
  • In outlines, poorly structured, getting started ought to have the part “set repo” and “install”, shouldn’t make emphasis on FAQs.
  • The participant considers that documentation is confusing as to the step-by-step, he ends up diverting from the task acquired.
  • The participant stated that the content material of the listing in the overview section shouldn’t be detailed in subsequent sections in the Setup an Fdroid App Repo page.
  • It seems extra logical for him if in the primary page of the documentation are positioned first the tutorials and in the long run the FAQs.
  • The participant had to make use of the Ctrl + F perform several occasions to seek out key phrases within the index file of the documentation. It seemed unclear and messy to him.

MetaData

All however one of the members left their app and repository metadata empty during this activity. There’s a risk that it’s because it was not explicitly said within the duties necessities, as a result of it isn’t clearly marked in the directions, or because the general problem of the other elements of this activity distracted them from this element of the duty.

Feedback:

  • The participant didn’t complete the repository or software metadata.
  • The metadata of the apps and the server was not configured, nevertheless the server was operating with the given purposes.
  • The participant didn’t place the metadata of purposes or repository.
  • The participant generated the skeleton metadata and did not update the knowledge.
  • The participant modifies the metadata information within the visual surroundings of the local machine (Nautilus and Gedit) because it was extra snug.

Docs to assist handle errors

When individuals did encounter errors there was little assistance to be found in the present documentation. The current documentation assumes success. Members who encountered errors desired that the documentation also included steerage about what might have gone mistaken once they encounter errors.

Feedback

  • The participant felt annoyed as a result of he didn’t find a place in the documentation where he might seek the advice of errors.In lots of occasions, he needed to use Google to assist with the debug course of however would not find relevant info there both. The participant needed a piece in the documentation displaying the way to validate the steps of the method and again a piece with widespread problems within the implementation and configuration.
  • Documentation ought to have a guide of error messages which may be acquired, e.g. those associated to the metadata.
  • The participant says that it could possibly be useful a piece in the documentation for Troubleshooting

Android Dependencies

A number of individuals have been unclear about what android associated dependencies have been required and tips on how to acquire them.

Suggestions:

  • The participant seeks for info on easy methods to download Android Studio
  • The participant was confused about the need to set up the dependencies related to Android with a purpose to complete the duty. After reviewing the documentation, the participant decided to not install these dependencies.
  • Documentation must be more evident relating to to dependency administration. e.g. If the binary set up consists of Android dependencies.
  • The participant doubts whether to do the setup to create app builds, and if the installation of the corresponding dependencies is roofed with the set up itself of Fdroid server.

Process 2: Hook up with that repository using the F-Droid shopper.

  • Average Time: 13 Minutes
  • Variety of Individuals with Essential Errors: 2

HTTPS Defaults on Repository Handle

Regardless that the builders thought the shopper was comparatively straightforward to make use of some encountered issues associated to the assumptions that F-Droid makes around the safety of a repository.

When including a new repository to the shopper software if the participant leaves the protocol clean (e.g. no HTTP or HTTPS) the shopper defaults to utilizing HTTPS. If the server doesn’t help HTTPS the repository will fail to set up. A repository that doesn’t help HTTPS is clearly undesirable. But, even so, F-Droid’s implicit assumption, without accompanying documentation and/or errors to alert the developer and participant, induced points for individuals.

Suggestions:

  • The error the participant was experiencing was that the shopper by default adds the HTTPS protocol to URLs that don’t explicitly state HTTP(s), and the server used for the duty didn’t help HTTPS.
  • It should have had documentation on the way to input the URL of the repository to be properly acknowledged.
  • The participant acquired errors when setting the repository tackle incorrectly with HTTPS as was instructed by default on the shopper software on the android telephone. Then the participant observed the state of affairs and altered the tackle to HTTP.
  • The shopper is intuitive to use.
  • The solution to the difficulty introduced doesn’t seem complicated to the participant but isn’t documented and is liable to make mistakes.

Interconnected Documentation Needed

When switching from building a repository to using the repository the members had to seek for shopper documentation in different elements of the F-Droid web site and on-line. Multiple members advised that shopper documentation related to using the F-Droid server being set up ought to be more easily accessed from the server setup documentation.

It must be famous that the individuals have been utilizing a improvement construct of the new documentation, and, as such it was in a roundabout way linked to the F-Droid web site the place the shopper documentation is out there.

Feedback:

  • In the documentation, there was no information about how setup the shopper’s app.
  • The shopper documentation ought to be accessible in the server-setup documentation. For example, including methods to set up and setup the shopper.
  • The participant claimed the task was completed, nevertheless the connection was not profitable because he wrongly placed the URL repository in the Android shopper ([email protected]:/var/www/html /). The participant was then assisted to move on to the subsequent exercise.He explains that within the shopper FAQ does not appear the best way to fill the URL of latest repositories within the shopper
  • Very confused about how one can get began.
  • Inside the app was straightforward to finish the task, since it’s an intuitive software. It turns into essential to have a step-by-step record of the way to set the repositories in the shopper, especially the URLs.
  • A link to house of fdroid.org could possibly be placed in the documentation.

Process three: Group the purposes beneath your repositories identify within the F-Droid interface.

  • Average Time: 1 Minute
  • Variety of Individuals with Essential Errors: zero

MetaData

Individuals had quick success and no important errors during this process. But, the members achieved this by circumventing the goal of the check. As an alternative of appropriately setting the metadata of their app, and looking for it inside the Shopper, they disabled all different repositories to make it so that solely their apps confirmed in the shopper. This speedy and common adoption of this specific technique of circumventing the need to set, and use, metadata is particularly troubling.

Feedback:

  • The participant felt confused about what was expected within the process. Nevertheless, he thought that disabling other repositories on the shopper record could possibly be enough.
  • There ought to be an choice to disable all repositories included within the software.
  • Initially, it was confusing to know which switches had to be disabled and which ones to enabled in an effort to see the purposes of the personal repository. It was simple a couple of seconds later however it ought to be specified in the documentation.
  • There ought to be within the documentation an instructive guide on using the shopper to manage repositories.
  • The participant discovered very straightforward to disable other repositories in an effort to see only the purposes of the server used

Activity four: Obtain the barcode scanner software utilizing the F-Droid shopper.

  • Common Time: 15 Minutes
  • Variety of Individuals with Important Errors: 2

The issues that the individuals faced in this process have been truly related to earlier tasks with “critical issues” that have been undetectable till this level. With out the file permission points, described under, downloading apps was seen as an easy activity. This highlights the overall feeling by the developer members that the shopper is usually straightforward and intuitive to make use of.

Server Permissions

The members encountered a number of troublesome to diagnose points related to file permissions on the distant server. These points solely appear once a participant is trying to download an software from a F-Droid repository. As such, permission points can simply slip previous a repo maker who merely updates their repository and checks to make sure that it may be related to.

Suggestions:

  • The participant acquired within the Android shopper the error “Unsuccessful Non-critical error download”, understood at the time as a permissions error on the (min 1) distant server, which he modified immediately on the time by accessing by means of SSH before re-testing the obtain.
  • Extremely snug and quick if the operation of permissions on the Subjective remote server is understood. The participant works with this permissions analysis management every single day and says that’s why it turned really easy, however maybe for an additional individual without such familiarity on net servers could possibly be extra difficult.
  • “It’s easy once you know how to configure file permissions.”
  • The participant did not find any documentation about distant server configuration, especially associated to file permissions on the internet server. The participant suggests that for many people this is able to be essential.
  • In the documentation, the information about the file permissions and configuration of the server at Fdroid degree is lacking. The documentation ought to say what permissions include the generated information, (or what permissions they should include).
  • The participant understood methods to resolve the permissions drawback on the remote server and corrected it by putting 775 permissions on the remote server’s information. Nevertheless other actions meant that the exercise could not be completed efficiently.
  • The shopper displayed the error “Connection refused”, the participant concluded that it was because of a problem of permissions on the remote server, nevertheless decided to deal with it by modifying the nginx configuration file, ensuing within the lack of ability to read and download information from the distant server.

Process 5: Replace the F-Droid repository with an update to the barcode scanner app.

  • Common Time: 9 Minutes
  • Variety of Members with Important Errors: zero

Updating the Server

Some members had continued problem working with the server throughout updates. While a few of this was associated to problem differentiating duties that ought to be carried out on the shopper vs. people who must be carried out on the server when reading the documentation, others have been related to the conduct of the software program.

Suggestions:

  • The upgrade process is best documented in comparison with different elements of the documentation, however still might be extra detailed.
  • The terminal was giving an error on the local machine when operating the replace command saying “CRITICAL: Failed to get repo pubkey!”, later the participant found that the replace was pushed anyway to the server saying that the task was completed.
  • The participant felt confused about which instructions have to be executed within the native machine and which instructions in the remote analysis server. The participant says that if he needs to configure an solely native occasion he would not know the right way to do it. The participant says that it could possibly be helpful a piece within the documentation for Troubleshooting
  • The participant does not know exactly the place to put the update information.
  • The participant isn’t clear whether or not it is essential to run the command fdroid replace and then fdroid server update, or just by operating fdroid server update would both of the commands run routinely.
  • The participant can’t read the replace from the shopper and doubts whether to run a command on the remote server as a part of the replace process, then he solves by doing update of the local repository before pushing the remote server.
  • The participant was not clear if Fdroid update, truly updates the remote server, then after operating it he was capable of entry by way of SSH to the distant server with a view to verify if the updated software was present, when seeing that it was not up to date, the participant understood that he additionally had to run Fdroid server replace

Documentation is Intuitive As soon as Understood

Once members obtained snug with the documentation, server setup, and/or updating course of they felt that it was relatively straightforward to use.

Suggestions:

  • Very nicely designed software to deal with the updates, it finally ends up being very straightforward to use.
  • The duty was straightforward once you perceive the file construction in the repository.
  • The task was very pleasant to complete on the software program degree.
  • Documentation is obvious but yet complicated at first studying.
  • The upgrade course of is best documented compared to different elements of the documentation, but still might be extra detailed.

Activity 6: Download the update using the telephone.

  • Common Time: 7 Minutes
  • Number of Members with Important Errors: 1

Permissions Cont.

Permissions points that went unaddressed in process 5 brought about points in the course of the, otherwise very straightforward activity, of downloading updates to an software.

Suggestions:

  • Once understood the difficulty of file permissions on the remote server, is straightforward to deal with the download of the update.
  • It’s straightforward once you understand how to configure file permissions.
  • Replaying of the same challenge of permissions and the error “Download unsuccessful”, then resolved by understanding how permissions are handled on the remote server.
  • It’s lacking in the documentation that before doing “fdroid server update” the permissions of the information to be served could be changed regionally since they’re transmitted to the remote server with the identical native permissions.
  • Very nice on the Android shopper aspect, nevertheless it must be documented the kind of permissions that the information on the remote server should include.
  • The replace is loaded in the remote repo nevertheless it does not present on the
  • Fdroid shopper, then the participant suspects it is because of the file permissions. After 12 minutes the participant says that he doesn’t know find out how to repair the difficulty and stopped the duty. The participant suspects before notifying the failure within the activity completion that the difficulty could possibly be related to some instructions and information created as root that creating conflicts with the duties triggered by the server update command.

Miscellaneous Suggestions and Requests

Complexity of Language within the Documentation

  • One participant used Google Translator for a greater understanding of some terms inside the documentation. This participant was additionally the only participant who speaks English as their main language. All other individuals converse Spanish as their main language.

Documentation requests

  • One participant needed instance configuration information within the Gitlab repository of F-Droid-server.

More specific documentation

  • The documentation must specify that a set of information has been created when fdroid init runs
  • It should have specified that commands with Fdroid must be executed in the primary directory of the repository (e.g. ~/fdroid) and never in other directories inside it (e.g. ~/fdroid /repo).