Category: Interaction Design

As I got to know hand tracking from the HoloLens2 and a little bit trough one of my lectures “Interaction Design”, I wanted to try out the tracking with the LeapMotion Controller. The Leap Motion Controller from the company ultraleap is an optical hand tracking module which captures the movements of hands and fingers even if they move unparallel to each other. It is used for development and deployment of various applications.

Set up

I borrowed out a leapmotion controller at the mediacenter from the FH and downloaded the accompanied software from the website.

You plug in the controller per USB and install it as well. There are various possibilities you can choose from to try out the LeapMotion. As my topic of learning sign language deals with viewing or tracking – not only – but mainly hands, I wanted to see how I can interact and which movements can be tracked preciously. With the information for setting up and trying out with the Unity engine given trough one of the many helpful YouTube videos, and after the installation, you can open Unity. In the program you install the XR Plugin Management under the package manager to prevent problems when downloading the packages for the LeapMotion controller. You import the unity packages that are provided by ultraleap into the assets folder and can try out the different examples given.

Try out and conclusion

It was very helpful for me to try out specific movements for a better understanding of how systems are tracking and which parts of the hand brings the most valid information for the systems to possibly translate it into sign language.

I got to know that the parallel tracking of both hands works very well but there are some difficulties for the controller to track preciously when the signs are to complex and have many overlapping fingers or else. After just randomly moving and changing the positions of my hands and the fingers, I tried to spell my first name consisting of five letters. The controller tracked three of them without any problems but with the other two, it could not recognize the position of the thumb or index finger and this is why it showed a different gesture than the one I was doing. I tried to readjust it by turning or giving the controller itself another angle to read the gesture as well as turning the hands around to view it from the other side but it always set back to the wrongly tracked gesture.

Heuristics for LeapMotion Interaction

As already mentioned a couple of years ago from the LeapMotion developers, it is important to evolve applications with the focus on the following points which I can now relate to and understand better after trying out the LeapMotion:

1. consistency in tracking: working constantly on accuracy and consistency of tracking by having multiple people perform actions, motions and gestures

2. ease of detection: create a concept of how easy the motions are detected and what is important to detect as obvious conditions which define the motion but are separated from other things around which could also be detected

3. occlusion: the controller should be able to detect the environment but if the sensor is covered by physical objects, tracking is not accurate or not possible at all. If motions involve occluded parts, the system can not visibly see a part of the hand and makes assumptions trough available data. A diagnostic visualizer can help to test a variety of detectable areas around the field of view to prevent occlusion.

4. ergonomics: improving posture and working environments but not on a physical object. A consideration of affordances and guestures adjusted to the movements of the humans and users body without harming or straining is necessary by having the environment and comfortable positions on mind.

5. transitions: creation of a concept for the interaction with the application. Every interaction should be defined and differ so that the gesture can be detected from the system as well as be more memorable for the user. If actions have similar results the usage of slightly similar interactions is acceptable, otherwise they should differ. It depends on the beginning and ending of a motion in the space in front of the body if the user can perform the action in the air easily. It therefor should be looked for minimizing awkward switches by implementing an “initialization” or “resting” pose which initializes actions.

6. feedback: consideration of how providing a feedback from the application to the user who is performing the gesture because a lack of hardware-based physical feedback in motion-based interactions can leave the user unknowing of the current state of the application and state of the system. The user interface should communicate three things regarding the users interaction: Where am I now? Where do I need to be to complete? How fare and in what way do I need to move?

With the programm Unity I can in the end evolve the whole setup of a possible application as I already informed myself in the last semester as well as in this one about the developement of the final product. There are many possibilities how to do so, so that it is necessary to first develop a concept and designing the experience and interactions beforehand.

source:

https://medium.com/@LeapMotion/6-principles-of-leap-motion-interaction-design-98bddff77a53

Woche 6 (24.06. – 01.07.)

Das Ziel von Woche 6 ist es wieder Follower zu gewinnen. Deshalb fokussiere ich mich in dieser Woche vor allem auf den Punkt Reels. Reels sind kurze Video Postings (maximal 30 Sekunden), die von TIkTok abgekupfert wurden. Weitere Infos über Reels findest du in einem separaten Blogeintrag. Ebenfalls wird es wieder Stories geben, wenn auch nicht täglich, jedoch super oft, um Follower wieder zurück auf mein Profil zu bringen.

In Woche 3 fragte ich meine Follower ob sie gerne eine Anleitung für das Erstellen eines Tierportraits in Photoshop hätten. 90% stimmten für ja und so entschied ich mich dafür eine Anleitung in Form eines Reels zu posten. Auch der Post in Woche 3 erhielt sehr viel Reichweite und konnte insgesamt 64 Likes ergattern. Das war der reichweitenstärkste und meist gelikteste Post, den es bis jetzt auf meinem Profil gibt.

Ich postete dass Reel am 25.6 gegen 21 Uhr (eigentlich schon viel zu spät für meine Followerschaft) und er hielt über Nacht 1248 aufrufe. Das ist für mein Profil mit nun 131  Follower sehr stark. Weiters erhielt das Reel 34 Likes und insgesamt 6 Kommentare (Stand 26.6). 

Update (Stand 30.06.): 1.347 Aufrufe, 40 Likes, 6 Kommentare

Aufgrund des großen Erfolges des Reels am Vortag, entschied ich mich dazu am darauffolgenden Tag noch einmal ein Reel zu posten. Dieses zeigt die Arbeit mit Acryl Tusche in Wasser welches einen schönen Effekt ergibt. Das Video wurde im Zuge eines Uni-Kurses produziert.

843 Aufrufe, 46 Likes, 4 Kommentare (Stand 15.07.)

When discussing the mind we also need to discuss our body. Interactions are not only processed in our minds but always use our body as the vehicle. In this article I refer to a paper that discusses the body and its cultural conjunction as well as and interesting gesture controller design that shows a designed outcome of of a philosophical point of view.

Sara Sithi-Amnuai has developed a custom built MIDI glove interface called NAMI that is designed for live electro-acoustic musical performances. It is a tool that extends her own multicultural background. Nami utilizes a force-sensitive-resistor, flex sensors, buttons, hall effect sensors, and a photoresistor. The purpose is to allow the user to use their own gestural vocabulary while being culture general and flexible valuing cross-cultural exploration and accommodating a variety of cultural gestural language.

In Sithi-Amnuais paper “Exploring Identity Through Design: A focus on the cultural body via Nami” she discusses the cultural body, its consideration in existing gestural controller design, and how cultural methods have the potential to extend musical/social identities and/or traditions within a technological context. She starts off by describing the metaphysics of agency, identity, and culture. The identity of a human-being is deeply linked to culture and sustained through creation and performance of music and dance. The agency describes an “… active entity constantly intervening in the course of events ongoing around [them]”. Our mind is inseparable from our body, while the body being the physical material through which we have access to the world. Sithi-Amnuai refers to a study by Steven Wainwright and Bryan Turner that shows that the identity of former dancers in Royal Ballet in London is deeply rooted in their own body. Once their career has ended, it is exceedingly challenging to reinvent themselves outside the company.

Sithi-Amnuais take on gestural controllers is very philosophical. Once again, I thought how important it is to have philosophical and meaningful foundation for the execution of a design. But also, the gadget itself is very interesting and makes up for an inspiring controller that enables users to put more of themselves into the music. She sparked my curiosity to explore and extend my own cultural body. In that regard, cultural etiquettes, movements, and experiences through sense, sight, and feel are affecting our instruments and tools as well.

Reference:
https://zenodo.org/record/4813188#.YOAqVS335QI

For this blog entry I chose to examine the paper “Collaborative Learning with Interactive Music Systems” from Adnan Marquez-Borbon. It is presented on the website of the International Conference on New Interfaces for Musical Expression (NIME-20). It is a bit off-topic but relevant for the lecture ‘Interaction Design’.

The paper draws a new perspective on how to learn a new instrument – more specifically an interactive digital music system. The designers of those instruments are often the only performers and do not have many copies of their specific instrument. In general they often do not have instructional informations aside from a technical document (if existent) what makes it hard to learn it in a traditional way. The ‘traditional way’ means to first learn the musical notation and then the according application to the instrument. This way means a very linear and structured process of learning. The teacher transmits knowledge, evaluates the technical proficiency, musical accuracy and the appropriateness of style of the student. But this approach ignores the diversity of alternative musical practices and approaches. In an interactive digital music system the musical text and notation are not obligatory central to the practice. It lives from variable and numerous practices and the form of documentation applies to that. The learning process is therefore more complex and probably not tangible with the traditional way to learn an instrument.

The author describes an open-ended, exploratory and collective learning approach to learn a technologically mediated instrument and with that to overcome the traditional way of learning an instrument. The so called ‘socially mediated learning’ process centres collaborative learning and should provide a flexible and adaptable learning environment. This space is unlimited in musical exploration, creativity and bringing in additional musical skills of students. Learning by imitating other learners can lead to the extension of the own capabilities of students. The challenge of collective learning is to keep up the motivation and let the students take responsibilities in their freedom of learning (like setting subjectives). The teacher becomes a guide and is not longer an authority. 

As a method the author developed a new instrument with a 3D printed case, two outward facing speaker, four push buttons controlling pitches organised in a one-octave chromatic scale, two linear soft potentiometer (controlling the pitch & other) and a force-sensing resistor controls the output volume. The volunteers have been three people with traditional music education and extensive performance experience but with no experience in performing with musical technologies or interactive musical systems. They learned to play the instrument in group sessions over a period of six months. 

As a result they actually started to learn from each other, imitated and/or extended each others findings and were learning effectively together. Furthermore they started to make up their own learning structures by noting down individual musical notations as a learning aid. They even started to come up with group exercises and created own compositions in order to rehearse and perform together. The participants developed their own style in playing but also found a way to play together. Even without fixed learning structures they started to self-organise their capacities and activities as well as learning goals which motivated and oriented them. 

Due to the flexible exchange of ideas and techniques the participants found a unified conceptualisation and usage of the instrument. Both, the whole group and the individual learner benefit from the rich learning opportunities in the open and flexible learning structure.

My resume: Personally, I never thought about how we learn to play instruments – even though I myself learned the musical notation and how to play the flute and the piano. Looking back I realise that even as a kid the linear learning structure is very hard to accomplish in the beginning. I am sure I would have loved to learn with such an open and collaborative learning approach that Adnan Marquez-Borbon suggests. 

Keeping in mind, that the approach is aimed for more complex and often unique technologically mediated instruments I am sure, that this structure can be helpful also in other fields of computer-mediated channels like new and unknown software. Since a lot of daily tasks are getting more complex day by day I think we should see a collaborative learning approach as a serious opportunity to enhance not only our learning processes but also ideation processes in terms of innovative design. I currently started my summer job in the area of ‘making’ – means building, tinkering around and hacking all kind of old electronic rubbish in order to create new stuff, new ideas and appreciate creativity from a new, more practical perspective. This kind of practical process also nourishes from collaborative exchange of ideas and knowledge which is in my opinion a really fruitful way of creation. 

So far I have analyzed some problems and circumstances not only in our society but also in our minds that lead to a rather negative state of mind or even mental diseases. With this article I want to focus on something incredibly positive, something that is even worth living for: the flow.
I’ve read the term “flow” in the context of psychology for the first time in the book “Thinking, Fast and Slow” by Daniel Kahneman. It striked me because it was absolutely congruent with what I have experienced. By referring to Mihály Csíkszentmihályi, Kahneman wrote that the flow is a curtain state of mind of effortless concentration which is so deep that you lose the sense of time, yourself and your problems. The joy that you experience is immense and Csíkszentmihályi says with good reason that the flow is the ideal experience of a human-being.

A lot of activities can set you in a flow whether it is making music, writing a book, or participating at a Formula-1 race. Driving a race car at 260 km/h or playing world class chess are without a doubt very exhausting activities. But what’s interesting about flowing is that you don’t need any self-control to maintain your focus on those activities, whereby resources are released that can be used for the activity. [2]

Mihaly interviewed various creative people and scientists to find out what makes them happy, and what makes them strive, sometimes without fame or fortune, and I would like to share two quotes with you here.
A well-known musician said: “You are in an ecstatic state to such a point that you feel as though you almost don’t exist. I have experienced this time and again. My hands seem devoid of myself, and I have nothing to do with what is happening. I just sit there watching it in a state of awe and wonderment. And the (music) just flows out of itself.”
A poet said: “It’s like opening a door that’s floating in the middle of nowhere and all you have to is go and turn the handle and open it and let yourself sink into it. You can’t particularly force yourself through it. You just have to float. If there’s any gravitational pull, it’s from the outside world trying to keep you back from the door.” [3]

And this is just exactly what I’ve felt, and what just made me incredibly happy. In fact, that is a major reason why I’ve started to pursue a creative career.

Regardless of the culture, education, or the like, Csíkszentmihályi claims that there seem to be seven conditions occurring when being in a flow:

1. Total involvement in what we are doing – engaged, concentrated and focused
2. A sense of ecstasy – of being outside everyday reality
3. Inner Clarity – knowing what needs to be done and measuring our achievements resourcefully.
4. Knowing that the activity is doable – that you have the skills you require for the task at hand.
5. A sense of serenity – unconscious competence, present and a feeling of growing behind the boundaries of the ego.
6. Timelessness – thoroughly focused on the present and in a zone of the fullness of being
7. Intrinsic motivation – whatever produces flow becomes its own reward.

Owen Shaffer proposed seven clear and short steps that are necessary to get into the flow:

1. Knowing what to do
2. Knowing how to do it
3. Knowing how well you are doing
4. Knowing where to go (if navigation is involved)
5. High perceived challenges
6. High perceived skills
7. Freedom from distractions [4]

Being a designer I am interested in getting myself into a flow. But I am also very interested in taking other people with me into a sort of flow. I wonder if it could be possible to enhance experiences with my knowledge about the flow. Is it something that can be triggered through design? I know that the flow is something really powerful not only regarding mental health. It is something worth considering for my master thesis.

PS: While writing this blog-post I was [almost] in a flow.

Ressources:
[1] https://upload.wikimedia.org/wikipedia/commons/8/8f/South-Goodwin.jpg
[2] Kahneman, Daniel (2011), Thinking, Fast and Slow
[3] Mihaly Csikszentmihalyi, TED2004
[4] Schaffer, Owen (2013), Crafting Fun User Experiences: A Method to Facilitate Flow, Human Factors International