ARDUOUS2024: 8th International Workshop on Annotation of useR Data for UbiquitOUs Systems

This is the latest in a series of successful ARDUOUS (Annotation of useR Data for UbiquitOUs Systems) workshops. In this workshop, we will explore key topics surrounding the role of user data, scoring and ground truth data throughout the lifecycle of machine learning and artificial intelligence systems. This is particularly timely in light of the new EU AI Act, which will come into force in the following months and years. The use of user data to explore the efficacy and performance of AI systems is well understood, but as the regulatory landscape takes into account the risks to participants of automated decision-making and data analysis, there is an increasing need to better understand and characterise issues which may affect the fundamental rights of those whose data is processed using these systems. These include but are not limited to characterisation of model bias, robustness, sustainability issues, and detection of security and privacy issues such as malicious training data, data poisoning or leakage of source data. As full-lifecycle monitoring of AI systems becomes a priority for practitioners and system deployers, methods drawn from academic research must be translated into accessible real-world policy and practice. Transparency in AI, too, benefits significantly from the availability of ground truth data that enables us to concretely understand and characterise the performance of automated systems in real-world use cases, and is in scope for this workshop.  

Submission Guidelines

Format: For your submission you should use one of the following dedicated templates:

• .doc format
• LaTeX

All accepted papers will be published in the GI-Edition Lecture Notes in Informatics (LNI) under a Creative Commons BY-SA 4.0 license which allows re-publication of any part of the paper.

The paper length should be proportional to their contribution. The absolute minimum page limit is 3 pages and the maximum limit is 12 pages. Generally the paper length should not exceed the following guideline page limits, including references:

• Full paper: 12 pages
• Short paper: 8 pages
• Position paper: 3 to 5 pages
• Industry papers: 3 to 5 pages

Submission: through the EasyChair submission system https://easychair.org/conferences/?conf=arduous2024 .

Review process: the review process will be single-blind

All papers must be original and not simultaneously submitted to another journal or conference.

List of Topics

• experiences of the development, validation and sharing of data annotation protocols, both in academia and in industry
• ensuring compliance with GDPR (DSGVO) and AI EU Act when annotating data
• human-centred and human-in-the-loop approaches to designing and deploying AI systems
• characterising forms of annotation appropriate for specific sectors – for example, digital health, logistics, text mining for the digital humanities
• Low-resource annotation workflows
• Annotation metrics: inter-indexer consistency, similarity, bias and subjectivity

• Further general annotation-related topics include but are not limited to:

• methods and intelligent tools for annotating data
• methods for standardisation and normalisation in annotation practices
• influence of interface on annotation
• processes of and best practices in annotating data
• methods towards automation of the annotation process
• improving and evaluating the quality of annotations
• beyond the labels: ontologies for semantic annotation of user data
• high-quality and re-usable annotation for publicly available datasets
• impact of annotation on a system’s performance
• building classifier models that are capable of dealing with multiple (noisy) annotations and/or making use of taxonomies/ontologies
• the potential value of incorporating modelling of the annotators into predictive models
• evaluating the efficacy of transfer learning via existing annotated datasets
• handling semantic and temporal shift and drift in the applications of annotated datasets

Committees

General Workshop Chairs

• Kristina Yordanova, University of Greifswald, DE
• Emma Tonkin, University of Bristol, UK
• Gregory Tourte, University of Oxford, UK

Organizing committee

• Teodor Stoev, University of Greifswald, DE
• Dipendra Yadav, University of Greifswald, DE
• Fernando Moya Rueda, Motion Miners GmBh, Technical University of Dortmund, DE
• Nilah Ravi Nair, Technical University of Dortmund, DE

Venue

The conference will be held at the Informatik Festival 2024, Wiesbaden, Germany.

Contact

All questions about submissions should be emailed to organizers@arduous.eu